It's not that hard in principle - higher difficulties always choose the best option, while easier difficulties randomly take one of the best three options. Like a chess AI that doesn't (always) take the highest rated move. Vox Populi did it that way for a long time (though they got rid of it in newer releases, don't know why).
For the divide of NPC AI vs competitive player AI, there's a divide of behaviors towards the player. They actually tried to add it in a sloppy way with the "they hate us because we are winning modifier", but basically a competitive AI could turn around and nuke your capital after a game-long alliance just because you are ahead. A NPC AI would not.
They could add that as a "slider", the more you dial up competitive, the more the AI refuses to do behaviors that benefit the player and more it's attitude gets worse towards the player when they lead. Not sure how many players actually want a truly antagonistic AI though.
Also the way the win conditions are done in civ could lead to another variant - how much the AI targets the steps of a particular win condition (ie steps) vs just following flavors (gaining science)