I believe there's a disconnect between the Civ series and single-player AI design.
If we just look at "Some single-player 'builder' challenge with AI rivals", then there are lots of points that can be made. Having a complex field of choice, having AIs that can make good choices a lot of the time, but for which that field's nuances, situational variance, and mental-contest potential, will of course escape the AI, creates a game where the Human's ability to look at what is happening and think about it inventively will win him the day. That's a good game. That's a pass.
But civ games don't have that freedom to be such a game, to afford the conditions to an AI. The economics and tile exploitation and strategies that span a hundred turns to flower, there isn't an AI that plays competently short of one that plays amazingly. It just doesn't make sense, with what a strategy game is, a builder game is. The inescapable conditions of a 4X (e.g. snowballing - and the toolbox "negative feedback" is orthogonal to the snowballing that occurs in a builder game).
And civ games have a motive to be something different on the AI side - the historical leader facet. Or, just the general playing characteristically facet, in place of minmaxxing. This is Beyond Earth, but the 'characterizing behaviour' thing still exists, so you've got this voice to make them specifically not good players, but flavourful ones. (The trouble, see, is how sacrificing 'good' does not run it off the rails into abysmal whole cloth.) Now, a personality can be said to be a kind of default-state strategy, but nonetheless the AI is not a participant in the game the same way a Human will be. They'll have their personality and/or their strategy and that's that. They will always be a scribble at the bottom of a Civ game's potential for exploratory play.
Take for a moment that I am too quick to be sure such an AI is impossible - a good one. Let's suppose it could work. There would be some restrictions, and you would commit to making the game and AI together - I think that's the dialogue you were inviting in the OP? It's troublesome because these conditions in many places conflict with that which makes the game exciting were we to discount the PvAI mode:
Opportunity for bold advances stimulates creativity in exploiting the game, and also allows for the potential of multiple competing routes to success. Big numbers and swingy decisions, the audacity of multiple available slingshots keeps things in balance and depth for at least a long tail while the metagame is built up after release, ideally it in fact stabilizes a competitive and deep game in perpetuity. But meanwhile, slingshots are not friendly to an AI, which wants a flattened progression diagram, one that grows in requirements at a smothering rate, one that 'branches' rapidly and, whether it allows or heavily rewards long plunges or not, it must require breadth or the AI just will be left behind. If the AI is tinkered with to use the slingshots, then the developers would have to know the slingshots, which (A) defeats the excitingness of the bold strokes in the first place, and (B) defeats the excitingness of 'finding' them over just watching a computer use them, and (C) is improbable given well-known balance testing limitations.
Also the tactical combat. This is a laborious point, I'm sorry. Tactical AIs exist and are pretty darn good. But I claim it is the tactical system that screws up Civ's AI. 1upt makes the combat tactical, but 1upt is not the problem. The combat should be strategic. Or tactical. But at present it is trying to be both, and making a subsystem of the AI be good at a potpourri is ludicrous. Strategic combat would be about using the right terrain and fighting at the right time for the right geography (to protect or obtain territorial assets). The tactical combat is trying to use positioning, sight, mobility, surprise and all that to play a tactical game, which would be fine except you don't get to choose to not have a strategic combat game there, because it is there, and what those tactical games have in common where the AI challenge works is that everything else stays still and the grand strategy is outside the frame. a 4X AI has to be good at fighting war, and also good at figuring out how good a war might be, and also good at figuring out how much to want a war to fiddle with other things dozens of turns before that war would actually happen. That's just the basic economic reasoning of the genre.
As soon as you wash over that much complexity with "We'll let the AI spam resources", that's hands in the air and given up on what Ryika is asking. you need an (artificial reasoning) tool that , might not be good, but at least -tries- to do those things, and somehow, you need a game where an imbecile can make a decent go at those decisions, and yet you want to excite your player to tackle the same strategic difficulty to 'beat' it, meaning those decisions matter.
You somehow need to have a game where arbitrary decisions won't work, where random decisions are 95% confident not to work, (i.e., choices matter) and yet a rule-following machine can keep up with that same game.
So like I said. A single player , AI-driven challenge is totally doable. You can have it. It can be tough and fun. But it basically pulls in 180 degree opposite direction from making a Civ game you'd actually want to have.
The Civ 5 and Civ:BE did not just have an AI that "kept up" with some handicaps to shove it to keep tie with you. Those AIs are given literally everything, so that they don't play the game. They don't make choices. Not merely 'middling good' ones - they don't make any, they scoop up everything and are kept at bay by an artificially pacifist diplobalance.
As your experiment discovered, Ryika, once you take away some bold advance potential - flattening out the progression by making technology so much further away, and greatly expanding the requirements for advancement to include nearly every dimension - the AI got really good, and pure material won hard.
So the work cut out for even experimenting with meeting the AIs to the game's progression design, tying this back to your actual request for input, is to let the AIs make choices: Get them to where they are choosing things, and the Human is choosing things, and the AI isn't that good at seeing the future, or interpreting you, or managing gambles, but it is at least putting one foot in front of the other. Then you get to play along, do smart stuff in a string and beat the AI down and be proud of yourself. The AI gets cheaty to stand back up , and then you outsmart it again, and you do that throughout the ages and finally you prove your ultimate superiority. Good game. At present they are doing nothing strategic, so the task of tricking them into acting that way, combined with swapping the game around, is a tall order indeed.