Joysticks are different than a mouse, the move more of a diagonal direction, whereas a mouse is a 1d surface.
Well, I guess it depends on who you're talking to, some people may think it's quite different, I personally don't feel these two are very different in a context where I'm controlling some sort of object in space.
The first time I picked a FPS game where you could actually look up and down (it was "Outlaws"), I had headaches trying to play it for like a week until I started to fiddle with the options. I found the "revert Y axis" option (I wasn't very bilingual at the time so I didn't understand all the options quite well), and when I toggled it, it was immediate relief. So, like I said, it's innate I think. I wonder what's the percentage of people who are in my situation, I'd be curious to see if it was correlated with the % of people who are more "ears"-focused and those who are more "eyes"-focused.