Goals for v39?
- Thread starter Fermongu
- Start date
- Joined
- Mar 17, 2007
- Messages
- 9,345
Since it seems like as good of a place as any to ask, I'll jump on as well:what's the latest in terms of really big maps in C2C, and are any changes likely? I started up a game of the latest version last fall, and was pleasantly surprised by the stability and speed, though I chose a more sane map size than I did back in the day. Eventually got distracted by shiny things, and haven't finished it... but I also recall that in areas such as viewports, C2C has pushed some frontiers in making more out of the resources available to Civ4 than traditionally thought possible.
The thing that triggered the thought was the realization that, with Threadripper CPUs (with 8, 12, or 16 cores today, and up to 32 cores in a few months), we're at the point where a multithreaded Civ AI could make a serious dent in AI turn times. Taking a 5-minute turn time to a minute and 15 seconds on a quad is good, but still kinda slow. Taking it to less than 20 seconds on a 16-core, and it's definitely in the playable range. Even after applying Amdahl's Law, Threadripper rips open some new possibilities. But I don't know how much progress has been made in this area.
Of course there's also the Memory Allocation Failure problem, Civ4's Achilles heel. Even if the recent mods can cut through AI turns like a hot knife through butter, it's not much use if it leads to quicker MAFs. I recall C2C made significant strides towards reducing the occurrence of these... but I'm also not sure if we're at the point where I could safely increase the mapsize without dooming my game 20 hours in.
It would seem that being able to work around both of these issues would open the possibility for an ultimate empire-scale Civ game, of the type that 2008 me would have found the most appealing game possible, and 2018 me would still be interested in. As it has been, Civ5/6 have 1 UPT which kinda kills it for me, Civ4 has MAFs, and Civ3 doesn't have those but does have a hard city limit and slow, single-threaded AI. That leaves advanced Civ4 mods, and maybe the Civ2 ToT Patch Project, as the remaining possibilities for breaking the playable empire-scale-map title, which still IMO belongs to Civ3.
(N.B. The most recent CPU I have is a quad-core Intel one from 2011)
The thing that triggered the thought was the realization that, with Threadripper CPUs (with 8, 12, or 16 cores today, and up to 32 cores in a few months), we're at the point where a multithreaded Civ AI could make a serious dent in AI turn times. Taking a 5-minute turn time to a minute and 15 seconds on a quad is good, but still kinda slow. Taking it to less than 20 seconds on a 16-core, and it's definitely in the playable range. Even after applying Amdahl's Law, Threadripper rips open some new possibilities. But I don't know how much progress has been made in this area.
Of course there's also the Memory Allocation Failure problem, Civ4's Achilles heel. Even if the recent mods can cut through AI turns like a hot knife through butter, it's not much use if it leads to quicker MAFs. I recall C2C made significant strides towards reducing the occurrence of these... but I'm also not sure if we're at the point where I could safely increase the mapsize without dooming my game 20 hours in.
It would seem that being able to work around both of these issues would open the possibility for an ultimate empire-scale Civ game, of the type that 2008 me would have found the most appealing game possible, and 2018 me would still be interested in. As it has been, Civ5/6 have 1 UPT which kinda kills it for me, Civ4 has MAFs, and Civ3 doesn't have those but does have a hard city limit and slow, single-threaded AI. That leaves advanced Civ4 mods, and maybe the Civ2 ToT Patch Project, as the remaining possibilities for breaking the playable empire-scale-map title, which still IMO belongs to Civ3.
(N.B. The most recent CPU I have is a quad-core Intel one from 2011)
Forster
Prince
Why?
JosEPh_II
TBS WarLord
Civ IV engine is a single core engine. The attempts at multi-threading some of the inputs before being given to the single core engine input chain was done by koshling. koshling is the Modder and author of the Viewports and Graphical Paging options that Caveman2Cosmos offers that most other Mods do not have. His work helped C2C achieve the size and complexity that the mod employs now. He retired as an active modder several years ago.
In one of the Global Define files there is a line for setting how many multi-threads the mod can use. Current Team Modders, Thunderbrd included, do not have a good grasp or skill levels of what koshling did in his coding off these aspects of the Mod. The only one that does is alberts2 who is a part time modder When he can find the time from RL. Iirc alberts2 is the one that wants the multi-threading option for inputs to the engine to be discontinued. But the rework will be very extensive and time consuming, both of which he can not commit to doing and possibly has no desire to do as well.
AIAndy developed and introduced the Property system that is unique to C2C. He is also inactive as a modder but does pop in from time to time to help with the Property System fundamentals. Examples and corrections when needed.
These 3 professional C++ coders are light years above what the team can do right now. ThunderBrd is a self taught C++ coder with help from these 3. If T-brd had ceased to be active at anytime over the last 3+ years this mod's development would have dropped to a slow crawl. Or even come to a complete stop.
So there you have some reasons and answers to your question of "why?".
Thunderbrd
C2C War Dog
Which I have basically done for a bit at this point. Just taking a break - I'll be back up to full swing probably in winter.
The multithreading proved to take as much time for processing to setup for the functions it would work with (or more) than it would save. The core EXE is untouchable to us and it is single threaded and this is a synchronized environment in multiplayer, both of which limit the effectiveness of multithreading, which can thus only be turned on in very limited cases. It just wasn't helping more than harming (there is overhead in getting it started and it didn't have wide enough realms of processing to cross to make it worth it.)
Thunderbrd
C2C War Dog
I do have one important goal for this next version and that is to complete my trait set. Beyond that, I'll probably put anything more aside from critical bug fixes onto the v40 cycle. There are important things to review on buildings and units but I've got to finish the traits or I've invested a LOT of time into something that continues to falter only just at the point where it's put to XML.
I can understand input to a single threaded core would be prone to too much overhead.
Would it make sense to have some of the mod features be run on other threads than the normal features. If properties take up a decent amount of work, then maybe multithreading those would make sense, as the game seems to be mostly interested in the total for a city rather than the details of its production. It might have the side-effect of delaying the effect of properties by a turn or so, if the calculations arent done in the end turn step.
Would it make sense to have some of the mod features be run on other threads than the normal features. If properties take up a decent amount of work, then maybe multithreading those would make sense, as the game seems to be mostly interested in the total for a city rather than the details of its production. It might have the side-effect of delaying the effect of properties by a turn or so, if the calculations arent done in the end turn step.
Thunderbrd
C2C War Dog
We did that. But you have to understand that you have to setup multithreading, then break it down to single threading before it returns to the main stream where the game actually records what takes place in a synchronized environment. And even if you bypass concern for synchronization, making the multi-threaded portions only function for single player games, you still have a major problem with the EXE expecting things to report to it in an orderly fashion and IT is ONLY ever going to be single threaded, so you can only open up multiple threads to handle things for small amounts of time, such as processing properties, spawns, city queues etc but then it has to quickly be reduced back to single threading for the game to continue. The overhead in time to setup and break down the multi-threading for these segments proved to be as much processing cost or more than what was saved.
I think multithreading is still somewhat in action on single player games in the city queues if I'm not mistaken. But Alberts2 was going to revert it back after test showed it to slow things down further than just single threading would've been. We've not seen Al in a while.
If you could keep the extra threads alive but asleep, you could bypass some of the overhead by simply waking them and locking some property until they are done. This deals with the synchonization, though there is the problem of leaking threads on changing games if not handled properly. If we can use outside libraries, there are some concurrency methods with very little overhead, such as the CSP library for at least python, though I cannot recall if there was also one for c++.
If the game strait up forces closure of extra threads, it does become hard to not have it die to overhead though, since there is only so much work that can be done at a time before touching the core.
If the game strait up forces closure of extra threads, it does become hard to not have it die to overhead though, since there is only so much work that can be done at a time before touching the core.
Yeah, I think if anything, multithreading is simply not an option that in and of itself can reduce turn time significantly, for those reasons. Where it could theoretically be useful is if there were other calculations to run that didn't need to be updated on a turn by turn basis. If you knew it was a slower paced game, maybe it could make sense to spawn off threads to calculate an AI grand strategy, where they'd take a snapshot of their empire and their known world and crunch for a few turns about what they want to prioritize on a high level (e.g., get that Iron before Monty does, then fix our broken economy). The problem is that it'd be kind of difficult to spin something like this off, spend ten turns coming up with it and then the data coming back at a time when they've been declared upon by a couple of other neighbors, etc. Well, and of course that synchronization and threading is headache inducing even on a good day.
Thunderbrd
C2C War Dog
Still very important. I'm not saying C2C is complete yet. Just that I'm taking a rest on it. A while back I got burned out bad on another project and I got sucked in here. Now I have a total reignition on that project and C2C has me burnt out (probably mostly due to conflicts over stuff.) Eventually I'll feel the groove again and be back. I think it might be impossible not to unless the entire computer industry suddenly goes through a complete revolutionary shift, which could happen of course.
Sounds nasty, be sure to put health before fun and creation.
V38,5 feels definitely the best version so far without even remote gamebreakers in sight at the dawn of modern era. AI is still quite pitiful in fight though but I know that it is still unfinished as well so no problem there. Can't think of pretty much anything else right now that the mod needs besides these two things.
Btw, is diplomatic victory and elections disabled?
Thunderbrd
C2C War Dog
Not that I'm aware of but we've had numerous people over the years say they think there's a problem there. Haven't found anything in the codes to suggest there is but it's hard to track down something that is NOT happening that should be. Much easier to fix things that shouldn't be happening but are.
Dancing Hoskuld
Deity
Yes people have. There is even a thread on it in the main Civ4 forums somewhere. The exe is licensed not owned by the creators of Civ 4. It is similar in nature to the modern game engines like Unreal etc. but not as well documented for use by us.
