Every LOL he's faced has affected his thinking instead of his body, so he tends to get freaked out when a very different thought suddenly presents itself in his head. Fujimiya logic also differs from human logic and snake logic, so explaining it can be rough....

He tends to figure that most people understand other people better than he does, so if they completely fail to understand him they're either toying with him or hating him for being incomprehensible. It hurts him, so he becomes more guarded around these people, and thinking that they hate him gives him an excuse to keep his distance. Does that make any sense?