CCTP Working Version Progress

Tritous · Mar 24, 2014

GAMEBREAK: \CCTP\Technologies\Technology_PrereqTechs.sql circular dependence between TECH_PHILOSOPHY, TECH_DEMOCRACY, TECH_CODE_OF_LAWS. and TECH_GOVERNMENT. Trying to research any of these or any tech which depends on these will cause the game to be unable to resolve which to research first and the infinite loop crashes the game.

Quick fix for those who are hurt by this:

Apply the first hotfix given in post http://forums.civfanatics.com/showpost.php?p=13131589&postcount=455

Open \CCTP\Technologies\Technology_PrereqTechs.sql using any simple text editor (e.g. notepad++, wordpad, you don't need special software for this but because of how long the lines are using simple notepad may struggle to show the lines). Turn off word wrap / line wrap

go to line 393. This should read "SELECT 'TECH_DEMOCRACY', 'TECH_GOVERNMENT' UNION ALL"

put "--" at the start of the line. If using notepad++ this line should now appear greyed out as a commented line.

Open Community Call To Power V.4.04 (v 1).modinfo using any simple text editor (e.g. notepad++, wordpad, you don't need special software for this but because of how long the lines are using simple notepad may struggle to show the lines). Turn off word wrap / line wrap

go to line 2896. This should end with "CCTP/Technologies/Technology_PrereqTechs.sql</File>"

remove the checksum after the md5= tag and replace it with

- 4ba5d4b3b1285291b0e1c123e2561ab0 if you also used the previous hotfix

- 2f06e6ea059abe5e2a92bf6a8a3eebf6 if you did not use the previous hotfix (only possible if you start the game at classical era or later)

Save the changes. you will have to fully exit the game and open it again before it takes effect.

When you reopen the game make sure you re-enable the mod as modifying the files will turn off the mod by default as the game detects the change

hotfix seems to break games, I'm looking into why this would break it when the other hotfix worked. I'm hoping it's not related to this being a cross tech tree change

Note: this is compatible with my previous hotfix as no line numbers are changed, but note the two md5 numbers
Note 2: This part of the tech tree looks wrong: why is democracy needed for philosophy and why is government after democracy? Why is government not part of the tree towards taxes? Why is alphabet needed for democracy? Something illogical here. The hotfix will get the game running but doesn't balance anything

DarkSides · Mar 24, 2014

@rottijumbo: Thx for the advice, it worked :goodjob:

. I doesnt know this mod before.

@Tritous: I am exited to know whats the solution is

Edit: @Tritous this was the first thing i was trying, and this results in a none loading game for me. For you not?

Tritous · Mar 24, 2014

Did you completely close the game and exit to windows (or whatever os you run)? You have to restart the game in order for it to identify the changed files and load them correctly

Next set of testing to see what I can do will be this evening, I have work to go to again. I'm now officially annoyed that the sql is not doing what I tell it to do, and fully motivated to beat it into submission [pissed]

(my usual approach to software design)

edit: and now my testing is completely blocked because steam is down and wont even start in "offline mode"

skitsopsycho · Mar 25, 2014

hey trit, with the 2nd hotfix blacked out do I still need to do the first? I did the manpower fix

Tritous · Mar 25, 2014

the first hotfix has worked. The reason the second hotfix hasn't worked I do not know but I have suspicions, Unfortunately with steam down I'm blocked from testing anything.

It's not like I actually know sql or Civ V modding, I'm having to learn as I go along

Tritous · Mar 26, 2014

Trying to figure out why I can't get the game to work now even when I roll back to the original files. something has gone wrong somewhere.

On the bright side I have some new findings to be fixed thanks to the game debugger. Don't need the game to work to be able to analyse it

:

Log data

Spoiler :

Analysis

Spoiler :

Tritous · Mar 26, 2014

follow up, some of the issues tracked down:

\CCTP\Technologies\Technology_Flavors.sql
Line 75 - no comma between the flavour and the value (most likely the syntax error, no union between selects
Line 122 - no union between selects
Line 190 - no union between selects
line 394 - no union between selects
line 586 - no union between selects
line 1000 - no union between selects

[40502.430] near "(": syntax error
\CCTP\Resources\Resource_Flavors.sql line 68 - ; at end instead of ,

[40502.508] 3 values for 4 columns
\CCTP\Improvements\CCTP_Improvements.sql line 19 only has 3 columns instead of 4. The yield is missing

CCTP/Technologies/Technology_PrereqTechs.sql - Refers to column "TechType" in table technologies
CCTP/Technologies/CCTP_TechnologyUpdate.sql - Refers to column "TechType" in table technologies
CCTP/Technologies/Technology_Flavors.sql - Refers to column "TechType" in table technologies
BUT
CCTP/Technologies/Technologies.sql - creates table technologies with a field "Type"

Update: prereqtechs and flavours have a translation mapping from techtype to technologies(Type). The issue is technology update: where it refers to the technologies table it should use Type, where it refers to prereqs it should use TechType

general comment: whoever made these sql files knew nothing about making code readable! crazy mix of misaligned tabbing and space characters - not so debug friendly

rottijumbo · Mar 26, 2014

Tritous, can't you launch steam in Offline mode and start Civ5?

Taa · Mar 26, 2014

Awesome work Tritous, You should really team up with Gilga and get this mod organized. You have done so much already and think Gilga could use some help

Tritous · Mar 26, 2014

rottijumbo said:
Tritous, can't you launch steam in Offline mode and start Civ5?

Steam can't be started in offline mode until you log in. you can't log in offline (or when steam's servers are offline). you can log in then go offline and stay in offline mode for as long as you don't shut down your PC, but not start your PC then start steam in offline mode. Sends me spare.

@Taa I'm working heavy hours at work, have to travel quite a lot so often am not around, and come home 95% dead. I can't commit to support a mod as I can't guarantee I have that level of energy. I game to try and unwind and calm down so I don't have much time on top of that.

What I've done has just been a minute here and there investigating a bug I see, paying attention, and this morning cross referenced a couple of files: 30 minutes work while drinking coffee and slowly waking up. I'm glad it helps, but I can't promise more than that in a structured way.

If we all did the same: noting down the bugs, seaching the xml for the matching entries, and noting it down we could do a massive amount collectively while doing very little individually.

edit
Ok, now I'm officially miffed at this issue with the tech tree. I've got half a dozen other files fixed and I can make it work, but I cannot split this tree at the cross-branch line. I spent half a day learning sql but all I learned is there is nothing wrong in the change I made...

iNeatProphet · Mar 26, 2014

Hey Tritous, are you certain that Steam is down? I had a nasty bug with Steam recently, where I couldn't log in, but it ended up being a client-side issue. The TCP stack apparently needed to be reset.

Tritous · Mar 27, 2014

iNeatProphet said:
Hey Tritous, are you certain that Steam is down? I had a nasty bug with Steam recently, where I couldn't log in, but it ended up being a client-side issue. The TCP stack apparently needed to be reset.

Steams's servers were down for about a day (or at least the EU ones) for "scheduled maintenance". They didn't tell us their schedule, typical steam....

Well I'm close to giving up on the PreReqTechs. I have made some fixes for some of the other things I found so I would have a shorter debug log but the second I touch the prereqs it breaks. My suspicion is it's the graphical mapping but that makes no sense as the algorithm on the tech tree drawing is pretty generic. This may need someone who actually knows the CIV V engine better to analyse it or at least to pass some of the knowledge on.

Tritous · Mar 27, 2014

[40502.430] , Update Database - CCTP/Resources/CCTPFontIconAtlas.xml, 0.000879
cause
[40502.430] column IconFontTexture is not unique

Last 4 entries (ICON_RES_ELECTRICITY, ICON_RES_OLIVES, ICON_RES_OLIVE_OIL, ICON_RES_PLATINUM) are duplicates and need to be removed. Lines 88 to 91 are duplicates and need to be removed (commented out for now)

CCTP/Resources/Resources.sql - commented out line 33 as it's an empty (and not closed) table

CCTP\Technologies\Technology_Flavors.sql - Lines 1053, 1054, 1006, 982-985, 456, 412, 418, 342, 334, 313, 158, 75 are missing a union and line 559 has a double union. Note: this may relate to the reports of people of not very good AI with techs. if there are breaks in the flavours that could do it!

I have corrected these and everything in post 467. this doesn't have the tech tree fixes as I can't get those to work.
Gilgamesch you may want to use these updated versions of the files. After your next release this weekend I'll have a try at putting all the files into a normalised indenting and spacing policy. The result will be smaller files (up to 40% smaller), possibly slightly faster loading, and it will be easier for me to use tools and searches to hunt down bugs.

The attached files should work for everyone else too, but backup first of course

Note: I've also set the affects saves to true in my modinfo, as it is true!

Gilgamesch · Mar 29, 2014

Sorry for the late response, was very busy in rl.

Ok lot of news i have to read before i restart working.

Good news i still worked on CCPT and i hope that i will upload 404 V2 today.

For he loading problem, i figured out it was the prerequested technologies.
So i reorganized the early techtree and the loading time is back to normal.

Thanks fo all the reports and tipps, you rock:=)

Gilgamesch · Mar 29, 2014

@Tritous thanks for all the news, in the current update the log list is much lesser then in V1 :=)

Gilgamesch · Mar 29, 2014

Tritous said:
general comment: whoever made these sql files knew nothing about making code readable! crazy mix of misaligned tabbing and space characters - not so debug friendly

Cant tell you how many times i banged :wallbash:

...my head because of damn sql problems, and yes the files are old and not user friendly.

Tritous · Mar 29, 2014

when you release this version I'll give them a tidy up and teach you a couple of surprisingly easy tricks to handle them. It's very very clear to me that the SQLs have been created by 2-3 people, with 2 different people modifying them over the years. When I'm done one of the easiest ways to open them will be with excel as tab delimited text. Then you will see everything in nice neat columns and can filter and search easily. tab aligned tables were thrown out in the 80s because they are inconsistent and almost impossible to machine read in a sensible way. I will put it to a simple tab delimited text layout as it should be.

I'm half tempted to learn some LUA and have a look again at the algorithm for the techtree piping. It looks like it may be O(n^2) and with quite a few passes. It would take me a long time but my instinct tells me there is a way to optimise the algorithm to take less processing time based on my first look through. I may also have a look around at other SQL modifications. I'm fairly sure that straight value insertion is faster than building a big select union as it avoids database duplication in parallel. just a different way of coding exactly the same thing that will run faster - and have smaller file sizes

Gilgamesch · Mar 29, 2014

Tritous said:
when you release this version I'll give them a tidy up and teach you a couple of surprisingly easy tricks to handle them. It's very very clear to me that the SQLs have been created by 2-3 people, with 2 different people modifying them over the years. When I'm done one of the easiest ways to open them will be with excel as tab delimited text. Then you will see everything in nice neat columns and can filter and search easily. tab aligned tables were thrown out in the 80s because they are inconsistent and almost impossible to machine read in a sensible way. I will put it to a simple tab delimited text layout as it should be.

I'm half tempted to learn some LUA and have a look again at the algorithm for the techtree piping. It looks like it may be O(n^2) and with quite a few passes. It would take me a long time but my instinct tells me there is a way to optimise the algorithm to take less processing time based on my first look through. I may also have a look around at other SQL modifications. I'm fairly sure that straight value insertion is faster than building a big select union as it avoids database duplication in parallel. just a different way of coding exactly the same thing that will run faster - and have smaller file sizes

Great

Dumb question, did you ever used modbuddy.
I could send you a modbuddy version of CCTP so that you could get open it as a dev.

There are several files (mostly sql) that are done by different peoples over the years.
I tried to make them more userfriendly, but as the progress of growing goes biggger and bigger, i decided to focus on the important parts first.

For the techpipes.
The plan is to include at least 2 new eras to the game. (Stone Age and one for the latere eras)
Will need lua changes, and new pics for the menue.
Also this would make the techtree looks more optimal then now.

One other point/problem with the current techtree is, adding new features like for example the priest specialist, brakes the whole techtree.

Not to mention other usefull lua tools, that also are incompatible with the techtree.

Tritous · Mar 29, 2014

I'm an oldschool programmer and I prefer to go directly into the code in a basic text editor rather than trust other tools, especially ones aimed at non experienced programmers. anything that generates code has to either be 1: so generic that it is massively inefficient and makes bulky code or 2: unreliable. There are many different tools to help search, analyse and organise code but there remains only 1 tool for producing optimised and intelligent code: and that's a human (although expert programmers may push the definition of human a little). Because I work directly from the code I can learn new languages in ~1 week for syntax and semantics, <1 month for optimisation and advanced techniques: ultimately all languages have 1 thing in common which is they are for controlling a machine. Understand the machine and the language is easy.

If we are going to rely on lua tools there is always going to be a penalty in code size, loading time, efficiency of the code and ultimately it stops us from thinking and understanding the code as well as we might do otherwise. getting hands on in the code is the only way to really beat it. Think like a machine, see the database structure in memory, and see the linking keys weaving through the tables and suddenly the whole thing is much simpler.

I will not be here in CW15 (7-11th April) as I am travelling on a business trip again. I may be able to do some optimisation of the SQL before then. The LUA will take a little more learning time as it's a C derivative bolton language which depends entirely on the CIV5 data structure. That I have no reference anywhere to the CIV5 datastructure is what is slowing me down the most as I do not know what data is used for what by the game engine. If you know of any references on the database, libraries or APIs then I should be able to do a lot more with my limited time. Besides, I always like learning a new programming language: keeps me in good practise

Gilgamesch · Mar 30, 2014

Ok one day later....

The last problem before i can upload V2 is, i to have to create prerequest techs for he last two eras.
Reselected all 380 technologies then and hope it will fit better with the game pathing.

Damn a nasty work iam happy when im done.

But the good news is the loading time is back t normal.

When there are no new problems i will upload V2 today.

CCTP Working Version Progress

Chieftain

Chieftain

Chieftain

Chieftain

Chieftain

Chieftain

Chieftain

Chieftain

Warlord

Chieftain

Chieftain

Chieftain

Chieftain

Attachments

Ancient Alien

Ancient Alien

Ancient Alien

Chieftain

Ancient Alien

Chieftain

Ancient Alien

Similar threads