Direct gaze - that is, eye contact with another person - causes the subcortical system to overload, and the parts of the brain that deal with arousal and calming are failing to strike a balance. The end result is that direct eye contact triggers a physiological response which makes it physically uncomfortable to maintain eye contact.
For me (am ASD) it always feels a bit overstimulating. A bit like staring at the sun. Or even more, like trying to do complex math in your head while also maintaining a conversation with someone. I am bound to lose my train of thought while keeping eye contact. I have to break it away to actually use my brain