In my last game, while on friendly relations and with open borders agreement with the Bannor, their rangers kept attacking my thanes of Kil without declaring war on me. Of course, I couldn't attack them back unless I wanted to declare war. What's the mechanic behind this?

Perhaps they were hidden nationality from being built in a city with the council of essus.

Yeah, there is anything special about the Bannor Ranger that would allow this. Either the rangers or your thanes must have had hidden nationality. I suspect it was your thanes if you weren't able to attack the rangers back.