Proposal vetoed.
Reason: After considering
@ilteroi,
@azum4roll and
@balparmak's points as well as how much work it would require to implement this proposal, I've decided it's not feasible to implement within a reasonable time frame. However, there is a clear consensus that players do not want to make the AI stupider based on difficulty, as both this poll and the previous one had a majority in favor of making this change.
Instead what I'm going to do is this on all difficulty levels:
CityNumOptionsConsidered: 90 (%)
TechNumOptionsConsidered: 90 (%)
PolicyNumOptionsConsidered: 90 (%)
Rather than pick using weighted randomization from the top X proposals, the AI will consider all options which have a score of at least 90% of the top score, and pick using weighted randomization from those options. This should adequately serve to prevent the AI from doing stupid things since an option which scored 90%+ of the top score should be good enough, allow for some randomization from game to game for the players who want that, and still allow players to perceive any policy trees which are chosen too often.
If anyone wants to turn this off (to experiment with AI scoring for debugging), they can set it to 100, which will only pick options that have the top score.
In addition, if anyone really wants to make the AI stupider, they can always set it to 50 or whatever.
I think this is a good compromise that can be done immediately to improve AI intelligence.