The former, and then only with spells that have a somatic component. It's cumbersome. If you think about it, druids with their divine casting are not actually hampered in any way by the heaviness of their armor, just the material interference. Arcane casters are the exact opposite. They can wear metal, they just cast less effectively the more they are burdened. Arcane casters require special training (still spell) to cast in it, because of the movement required.
Edit - to clarify, the training is to learn to prepare spells without a somatic component, not to train in making the gestures in armor.