I'm almost positive that's how it works based on multiple observations using Catherine to monitor what the AI is upset about. You can have a -5 penalty with someone and have that be "Neutral" and then a few turns later they will become "Unhappy." That's because that -5 isn't the leader's actual attitude, its the rate of change of the attitude. It's also why when you give the AI a gift you can see the values change on the screen, but they dont actually become Happy for several more turns.
This does actually make so much sense, not sure why I didn't think of it?
Also, I've had up to +5 first impression (and ranging in between so probably multiple factors here), still no idea why, I guess it could be that the "nice" answer is not always the right one and some leaders would think more highly of you for being more defensive, or busy (isn't there a "don't have time" answer)?
Or that it depends on how you meet, is it in your turn (you found them) or in their turn? Also what units met? Some civs might like an explorer while others like military? That does seem unlikely though.
I know I got -5 when Rome found one of my cities (no units in sight) with a frigate.
It just might be that this diplomacy system is deeper and more complex than initially thought.
My first game everyone pretty much hated me for no reason, then I realized that it's probably standard behavior when I din't give them any reason to like me.
In my current game I offered small gifts, sent trade routes to Scythia and changed my government to theirs. Gave that like five turns to work and she went from about 25% to 60% thus becoming neutral. After that she agreed to a joint war with India and even though I got -24 for warmongering and her slider went down she stayed neutral and seemed quite happy with me. Note that she wouldn't let me send an envoy at all.
After conquering all four Indian cities (and wiping them out) my warmonger penalty with here had actually dropped to -16 (she had denounced India before the war) and the net change was once again positive, it took like 10 more turns for here to become friendly and she came to me asking to accept her envoy. Since the warmonger penalty is still dropping the slider just keeps on going up, about 70% atm, won't let me establish permanent embassy though, guessing it will be at about 75-80%. Not able to declare friendship either, possible Scythia is a bad example here since here agenda is tied to alliances...
Since I just met Rome (hiding on another continent by himself) I tried the same steps as with Scythia, trade route, favorable deal, and once his agenda (large empire) kicked in his attitude just sky rocketed and I was able to establish permanent residence after just like 10 turns, even with differing governments and a bad first impression.
So, it seems like this system is designed with some inertia, much like religious pressure, a few turns (even with big changes like warmongering -24) doesn't actually change their attitude that much even though there are "big" numbers displayed, especially if they are already friendly.
The slider might not be to scale either, it seems to move faster closer to the middle and civs aren't that keen on allowing you to establish embassies or send delegations if they have reasons to dislike you (may not even show up as a moddifier), no matter what the attitude tells you.