It is supposed to happen if you enter a "light" area.
Drows are normally not used to the sunlight, and hence get the negative modifiers just as you describe.
To get around this, you have to either make sure only travelling during the night hours, or to select the Daylight Adaption feat.