(possibly) IMPORTANT SUGGESTION FOR SDK modders: A plugin interface

primem0ver

Emperor
Joined
Jun 16, 2006
Messages
1,158
Location
Peoria, AZ
I don't know if this is already in the works (as I am new to this forum) but I am going to suggest (or emphasize) it anyway. I do not own Civ4 (yet) but I feel this really needs to be said considering the fact that the source code is available for modification. People (especially mod makers who are not yet familiar with programming) need to be aware of some major problems that can result from the source code being available.

This thread focuses on an important need if people want to avoid countless hassles a mod involving a dll change can create. This may be asking too much, but it will be necessary if people are going to make significant changes to the main dll, especially as part of a mod.

Even with regular (albeit large) mods, the people who developed CTP2 mods found a mod/plugin manager important even (years) before it was possible to change the source code. With the SDK available and mods possibly involving changes in source code, IMO it is even more essential to have such a feature. The problem is that the by changing the dll, you change the way the program runs and quite probably in the long run making your mod incapable of working with other mods with a different version of the dll and even some non-dll mods.

To prevent this from happening, it would be best to have a team develop a managed, all purpose main dll with a plugin interface allowing any future source code additions/changes to be done through a separate plugin dll. Plugin features and access to main program functionality should be documented and any additional features/access desired by mod makers should be suggested to the team that develops the dll.

The main purpose the plugin interface would be to try and enforce a standardized system so that dll changes can work together instead of against each other and that all changes made programmatically to the source code (particularly those that ship as part of a mod) can ship (be downloaded) as a separate dll. This will allow changes to the source code to literally be modular (i.e. available by of the plugin interface) and the central executable will be the same for everyone no matter what mods you download (as long as you download the plugin version of the main dll. A control panel applet (or just a central managing program) can also control which plugins you want to use during runtime, allowing you to configure the mods you want to use (be active) during a certain run of the game. The dll can be updated regularly or however the team sees fit.

My suggestion for how this should be done? If anyone is familiar with Windows shell programming, a feature provided through the Windows32 API is IMO the best paradigm to follow for this “plugin” managing main dll: Windows Hooks. Essentially what this means is that you can create “hooks” so that any time a certain event is triggered within windows, your “hook” allows your program to “intercept” the event before it is passed to other windows (or even windows itself. An example of this is a cd-player application “hooking” (intercepting) the “new cd inserted” windows event, allowing the cd-player to check the cd before windows does. This way, if it is a music cd, the cd-player can start playing it automatically and prevent windows xp and other versions from popping up that “what do you want to do?” dialog or “autoplaying” the cd. Many real life cd-playing applications do this.)

A Civ “hook” example (assuming Civ4 works like most other civ games): Lets say you completely change the way a map is created (lets say you want to add a special type of tile to the map (or change the way Civ does it’s tile system completely). This mod will hook the “map creation” event and the “tile improvement” event so that your plugin can handle map creation completely on its own and then check the “tile improvement” event to see if the improvement being made is to one of your special kind of tiles. If so, it can handle the event. If not, it passes the event along through the normal Civ code. This kind of mod might also want to hook the “settle” and “movement” events to handle such events appropriately depending the kind of tile you are adding.

That being said, I need to mention one other thing here: I am a fairly busy person and while I may (if I decide to actually buy Civ 4) participate on this team, I am very limited in the time I can contribute since I am a high school teacher who also happens to also be working on starting an educational software business.

-primem0ver
 
Excellent idea. This would also allow my Civ4Modder to combine mods with DLLs.

I have no experience with the Civ4 SDK (and probably will not have, due to the lack of a WinXP machine), so unfortunately I'm not able to provide direct help. However, I have pretty solid programming skills in general, including plugin systems on Linux, so I can offer some general hints if needed. The hook idea sounds good to me.
 
I'm not quite sure what you mean by "the C++ program", but I assume you mean a compiler. There's no integrated compiler in Civ4 for sure. AFAIK, Microsoft's compiler is needed, and it doesn't run under Linux.
 
tdb said:
I'm not quite sure what you mean by "the C++ program", but I assume you mean a compiler. There's no integrated compiler in Civ4 for sure. AFAIK, Microsoft's compiler is needed, and it doesn't run under Linux.

There are other compilers out there besides Microsoft. You don't need to use theirs necessarily. I've gotten by just fine using a freeware compiler. Borland is also a possibility as well.
 
Potential problem:

Mod A captures the hook put on a unit's doTurn() call. Mod A adds one line of code to support an entire mod (new code within the DLL). Mod B also captures the hook put on a unit's doTurn() call. Mod B also adds one line of code to support the entire mod (also new code).

So while the new code has no conflicts, what about the unit's doTurn() call? Which takes precedence? Do both run (thereby allowing the unit to have two turns for the price of one)? If only one runs, what determines which one runs, and what happens to the other mod which now doesn't have its call?

Dale
 
The Civ4 Core Comunity Project is aiming to create the "Universal Dll" but were doing it the old fashioned way with a SourceForge CVS repository and people adding their mods to the code, many mods are controled by XML, others have config files to turn them on and off. Everything must to togglable and default to off. I dont realy see any better way of doing it, though I am admitedly ignorant of the hooks method you describe.
 
Dom Pedro: You mean you've compiled a working DLL for Civ4 with some compiler other than Microsoft's? I do know there are other compilers - I'm using GCC myself, on both Linux and Windows. But AFAIK, DLLs from different compilers tend to have some compatibility problems.

Dale: Maybe the hooking interface could have some way for the mod DLL to tell that it wants to "own" the hook? So if some mod wants to own it, no other mods could use the same hook, and if they try, the system would bail out with an error. I'd imagine many mods will only make small changes that can be used together with no ill effects.
 
Hm, I guess a combined DLL will work as well, if the features are controllable. And since that project is already underway, I guess it might be better to wait for a while and see how that turns out.
 
Hooks
A hook is a point in the system message-handling mechanism where an application can install a subroutine to monitor the message traffic in the system and process certain types of messages before they reach the target window procedure.

This overview describes hooks and explains how to use them.

About Hooks
Using Hooks
Hook Reference
For more information, see WinEvents.

I am not sure that 'hooks' can be used. What the CvGameCoreDLL.dll does is develop its own 'hooks' into using the Python programming which calls the "xml" file.

The CvGameCoreDLL.dll file then has to be "Compiled" then, and it must be compiled to work.

Have to look at it further and Shell 'hooks' in the first place, but it is not one large file, it is many small files all 'compiled' together at the end into the CvGameCoreDLL.dll file.
 
Ok...to help this discussion I am going to be more specific about what I mean by “hook.”

I am not sure that 'hooks' can be used...[and the rest of what was said]

I am using "hooks" as an example of the type of approach I am suggesting. I don't mean to suggest that using windows hooks is what we should do. It is simply that the hook model (stylistic approach/concept of hook) is best for addressing the needs of all possible mods. This is what I am suggesting:

1. A developing team comes up with a list of game events (and characteristics) that it would be useful for modders to modify.

2. The core (main) dll has a "hook" class that is called just before all events on the list are called by the game. The following changes would need to be made to the source code to accomplish this: a) addition of the hook (or equivalent) class. b) for every event affected, insert a call to the hook class’s main loop (with an event id code) before the call to the event.

3. The hook class checks to see if any of the mods being used (set by a control panel applet or other central control that be can configured by a user) have a function to change the way that event is handled (i.e. have a callback procedure or hook to be called to handle the particular event).

4. If it finds a hit, it calls that mod’s procedure. Then it moves onto the next one (back to step 3--if the hook procedure finds it beneficial… you can change functionality of hooks to meet any number of different situations).

5. At the end of the check, it finally returns control to the normal flow to the original program so that the game itself makes the call (if the situation requires it… in cases where you are making a separate procedure that handles the ENTIRE functionality of the event , such as in the creation of a map, you could supply an if statement that controls whether the original game’s code is invoked or not).

This provides a built in solution to Dale’s potential problem as well:

So while the new code has no conflicts, what about the unit's doTurn() call? Which takes precedence? Do both run (thereby allowing the unit to have two turns for the price of one)? If only one runs, what determines which one runs, and what happens to the other mod which now doesn't have its call?

If we use the model I have suggested, the mod dll does not call doTurn(). The hook handling routine calls your mod’s procedure so that Mod A can do its stuff (without calling doTurn()). Then the hook class looks for the next mod that modifies the event (in your situation: Mod B) and so on. Finally control is returned to the main program. The main program is what calls doTurn() AFTER all the mods being used have made their changes.

Hope this make things a little more clear. Of course I am not yet familiar with the actual code, I am just explaining the basic idea of how this “plugin” interface would/could work.

Lastly…

Hm, I guess a combined DLL will work as well, if the features are controllable. And since that project is already underway, I guess it might be better to wait for a while and see how that turns out.

I don’t want to derail any work already been or being done. Depending on how a “combined” dll works it might be a good thing and a good suggestion. However, if by “combined dll” you mean that there is a dll that all mods get added to, the upkeep on such a “combined” dll would be astronomical as people are always going to be creating new mods. Even if the developing team provides a way to submit new mods to streamline the process, they still have to constantly change the dll and people always have to download a newer (and bigger) main dll. The plugin interface allows anyone to add functionality without ever having to change the plugin interface (i.e. the main dll). The only condition that an update would be necessary to the plugin interface is if we missed a usable event and people want that or other new events to be accessible for modification.

I am not familiar with the uses of XML in programming but I can imagine it being used to implement something similar to what I have suggested. If this is already in the works, then maybe we should wait until the “universal” mod is done. I just want to bring these issues to everyone’s attention to make sure they are being dealt with. In the end, I am trying to turn a potentially messy situation (including a constantly growing universal dll) into a much simpler solution using a plugin/hook approach (but not WINDOWS hooks).
 
Sorry, still not convinced.

Here's some real examples:

Mod A: stack attack combat
This mod changes combat to be similar to CTP's stack attack with combined arms. It replaces the GroupAttack() function with a new one. Units in the stack are placed into an array and attackers and defenders are selected based on their location within the array.

Mod B: missiles mod
The missiles mod allows units with the xml flag to act as missiles, being destroyed after hitting the target. It modifies GroupAttack() by grabbing the missile before normal combat and performing a special combat round with that unit.

From what I understand of your explanation the game calls the "hook" combat. Which GroupAttack function is run first? If Mod A's is called first then the missile unit will be caught into the array and a stack attack using the missile performed, thus stopping it from acting like a missile and dieing on impact. If Mod B's is called first then the stack of units will perform a vanilla stack attack using the 1-on-1 method.

As you can see, both mods will fail to function as they should if the other mods GroupAttack() function is called first. This is what I was trying to point out above.

Dale
 
primem0ver said:
If we use the model I have suggested, the mod dll does not call doTurn(). The hook handling routine calls your mod’s procedure so that Mod A can do its stuff (without calling doTurn()). Then the hook class looks for the next mod that modifies the event (in your situation: Mod B) and so on. Finally control is returned to the main program. The main program is what calls doTurn() AFTER all the mods being used have made their changes.

Just a quick note - I think by "doTurn() call" Dale meant that the core DLL calls the doTurn hooks registered by the mods. Also, note that mods might do things that don't work too well together or are mutually exclusive. I can't come up with an example right away, maybe someone else can?
 
As it stands right now, I do not know enough about the source code to give you an "informed" answer yet... but I see a few possibilities based on the purpose of a “standard”.

First though, what is a "vanilla" attack?

I suppose with the possibility of XML coding that was not part of the original software, a solution would be difficult. I am assuming that your MISSILE flag is not already part of the Game definitions which is the only reason I can see this being an issue. This is why a team of modders should come up with the plugin code for the dll, so that all current possibilities can be accounted for and a standard be set. If a designer doesn't follow the standard, well then obviously its not going to work with other mods. People familiar with making mods should know what kinds of possibilities to expect. I am not saying this is going to be a be all end all solution. People are still free to (and probably will) develop mods that are incompatible with other mods, including those that use the plugin. But that is the designer's decision. If the designer wants to have his design work with other mods then he should follow the standard. If the standard does not allow for certain circumstances then he should let his desires be known and if the team decides to, they can set a standard for handling the request (such as one for units with as yet undefined special attacks should follow.)

From my perspective the situation in Mod A in your example is purely a coding matter for the mod designer (as long as all calls to hooks follow a certain standard that makes Mod A possible). Mod B
is the one with a real issue.

I don't know enough about XML to know specifically how XML would represent this so this may be a naïve answer but here is a possibility I can see:

(Assuming that the developing team hasn't already thought of this possibility ahead of time, they come up with the following special attack standard that is handled by the hook calling procedure)

Any type of "special attack" should follow this standard: Any newly defined tags that call for special consideration for attack should define when and where that attack is to take place using the following XML tags:

BEGIN SPECIAL ATTACK SPEC:

Code:
<! - The following tag structure should be used to define any type of attack that is non-standard>
< SPECIAL_ATTACK_FLAG_TAG TagName=&#8221;NAME&#8221;><!-or however you can class a tag to follow another tag&#8217;s standard in XML>
	<POSITION=&#8221;SEIGEWEAPON&#8221; or &#8220;INCOMING&#8221;&#8230;><!-this establishes where the type of attack originates from>
	<TIME=&#8221;Pre&#8221; or &#8220;During&#8221; or &#8220;Post&#8221;><!-defines when the call should be made>
	<LIFE=&#8221;Disposeable&#8221; or &#8220;CounterDisposeable&#8221; (Counter = countermeasures can be taken) or &#8220;Standard&#8221; or &#8220;CounterStandard&#8221; etc&#8230;><!-defines how the hook proc should handle disposal of this unit or allow it to remain for regular attack>
	<EFFECT=&#8221;SingleUnit&#8221; or &#8220;NoBuildings&#8221; or &#8220;MultipleUnits&#8221; &#8220;AllUnits&#8221; or &#8220;CityDestroyed&#8221; or &#8220;PopulationDestroyed&#8221; or &#8220;UnitsDestroyed&#8221;>
	<!Effect can be defined more than once&#8230; equivelant of an inclusive or to combine effects)
</ SPECIAL_ATTACK_FLAG_TAG >

END SPECIAL ATTACK SPEC:

Example Implementations defining several types of missiles that follows designer&#8217;s standard:

Code:
<SPECIAL_ATTACK_FLAG_TAG Name="Missile">
	<POSITION="Incoming">
	<TIME="Pre">
	<LIFE="Disposeable">
	<EFFECT="MultipleUnits">
</SPECIAL_ATTACK_FLAG_TAG>

<SPECIAL_ATTACK_FLAG_TAG Name="ChemicalMissile">
	<POSITION="Incoming">
	<TIME="Pre">
	<LIFE="Disposeable">
	<EFFECT="AllUnits">
</SPECIAL_ATTACK_FLAG_TAG>

<SPECIAL_ATTACK_FLAG_TAG Name="NuclearMissile">
	<POSITION="Incoming">
	<TIME="Pre">
	<LIFE="CounterDisposeable">
	<EFFECT="CityDestroyed">
</SPECIAL_ATTACK_FLAG_TAG>

<SPECIAL_ATTACK_FLAG_TAG Name="NeutronMissile">
<!-I don&#8217;t know if they actually exist&#8230; but I have heard about a bomb version of this weapon which eradicates population but leaves buildings intact.  Illustrates multiple effects>
	<POSITION="Incoming">
	<TIME="Pre">
	<LIFE="Disposeable">
	<EFFECT="UnitsDestroyed">
	<EFFECT="NoBuildings">
	<EFFECT="PopulationDestroyed">
</SPECIAL_ATTACK_FLAG_TAG>



<!-I don&#8217;t know what tags are used by Civ4 but assuming that the above is possible in XML&#8230; all of the above tags can then be used as a flag such as in the example below)
<UNIT DEF >
	<TYPE = "AIR">
	<HP=100>
	<Name="Chemical Missile">
	<ChemicalMissile>
</UNIT DEF>

As I said earlier, this would all depend on a team coming up with a standard that would need to be followed by the mod developer if it was going to be usable with other mods. It is up to the designer team to come up with the dll code that handles interpereting flag definition and handling the unit definition appropriately.

We don't have to do anything if we don't want to... I am just saying that it would be nice to be able to combine mods (without having to regularly download a supermod that combines them all manually) and that a plugin/standard approach is the best way I know of that this can be done. Dale, do you have an alternative suggestion for getting mods that alter source code to work together?
 
To be honest, I don't have a solution.

But the one you propose I don't really see working. You state you haven't looked over the DLL code (or played the game). I HIGHLY suggest you do before continuing discussion.

If you look through my combat mod thread you'll quickly realise that what you raise above wouldn't work for the mod.

I think the biggest problem here is that too much can be done with the DLL. No team will ever think of all the possibilities. And if you do get a team together to do it, why should a modder have to bow to the wishes of them? Who decides if a requested feature is added or not? Remember, we're just scraping the top of the barrel with the DLL. And when Warlords comes out that will just increase the possibilities.

Not to put a downer on your idea, I think it's admirable. But you REALLY need to get into the code to realise the difficulty. The example I gave above just wouldn't work as an add-in as both mods change the exact same lines of code. It's not one then the next, it's both at the same time.

Dale
 
This is just a picture and not even all of it of what you are dealing with in the SDK. What is needed is each Mod to have its own CvGameCoreDLL.dll file (if it needs it) to be in the MOD folder the player is using for his mod.

http://dimensionalcitizen.tripod.com/SDKScreen1.html

That also has to include the Python programming files and XML files also needed plus the WorldBuilder map or Map for the Mod or Scenario, since it is all "Text".

Example of WorldBuilder Map:

BeginPlayer
Team=13
LeaderType=LEADER_MONTEZUMA
LeaderName=Montezuma
CivDesc=Aztec Empire
CivShortDesc=Aztec
CivAdjective=Aztec
FlagDecal=Art/Interface/TeamColor/FlagDECAL_AztecCalendar.dds
WhiteFlag=0
CivType=CIVILIZATION_AZTEC
Color=PLAYERCOLOR_GREEN
ArtStyle=ARTSTYLE_SOUTH_AMERICA
PlayableCiv=1
MinorNationStatus=0
StartingGold=54
StartingX=19, StartingY=37
StateReligion=RELIGION_CHRISTIANITY
StartingEra=ERA_MODERN
CivicOption=CIVICOPTION_GOVERNMENT, Civic=CIVIC_DESPOTISM
CivicOption=CIVICOPTION_LEGAL, Civic=CIVIC_BARBARISM
CivicOption=CIVICOPTION_LABOR, Civic=CIVIC_TRIBALISM
CivicOption=CIVICOPTION_ECONOMY, Civic=CIVIC_DECENTRALIZATION
CivicOption=CIVICOPTION_RELIGION, Civic=CIVIC_PAGANISM
AttitudePlayer=1, AttitudeExtra=25
Handicap=HANDICAP_MONARCH
EndPlayer
BeginPlayer
Team=14
LeaderType=LEADER_FRANKLIN_ROOSEVELT
LeaderName=Roosevelt
CivDesc=American Empire
CivShortDesc=America
CivAdjective=American
FlagDecal=Art/Interface/TeamColor/FlagDECAL_Star.dds
WhiteFlag=0
CivType=CIVILIZATION_AMERICA
Color=PLAYERCOLOR_BLUE
ArtStyle=ARTSTYLE_EUROPEAN
PlayableCiv=1
MinorNationStatus=0
StartingGold=2000
StartingX=28, StartingY=45
StateReligion=RELIGION_CHRISTIANITY
StartingEra=ERA_MODERN
CivicOption=CIVICOPTION_GOVERNMENT, Civic=CIVIC_DESPOTISM
CivicOption=CIVICOPTION_LEGAL, Civic=CIVIC_BARBARISM
CivicOption=CIVICOPTION_LABOR, Civic=CIVIC_TRIBALISM
CivicOption=CIVICOPTION_ECONOMY, Civic=CIVIC_DECENTRALIZATION
CivicOption=CIVICOPTION_RELIGION, Civic=CIVIC_PAGANISM
Handicap=HANDICAP_MONARCH
EndPlayer
BeginMap
grid width=124
grid height=68
top latitude=90
bottom latitude=-90
wrap X=1
wrap Y=0
world size=WORLDSIZE_HUGE
climate=CLIMATE_TEMPERATE
sealevel=SEALEVEL_MEDIUM
num plots written=8432
num signs written=0
EndMap

### Plot Info ###
BeginPlot
x=0,y=0
FeatureType=FEATURE_ICE, FeatureVariety=0
TerrainType=TERRAIN_OCEAN
PlotType=3
TeamReveal=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,
EndPlot
BeginPlot
x=0,y=1
FeatureType=FEATURE_ICE, FeatureVariety=0
TerrainType=TERRAIN_OCEAN
PlotType=3
TeamReveal=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,
EndPlot
BeginPlot
x=0,y=2
TerrainType=TERRAIN_OCEAN
PlotType=3
TeamReveal=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,
EndPlot

This Blue Marble Earth Map with 18 Civilizations being changed is the Map you would see in the game, and all the AI players and everything.
The map is the Blue Marble Earth Map that is 128 x 64 or so. It is not a graphic file, it is a text file, and already (well since this is about the biggest one could go) is already about 1.25mB file.

The Mods or Scenarios usually may have a -- Map, Python files, XML files, and may include changes to the CvGameCoreDLL.dll file -- but all of that should be in the MOD folder for the Mod or Scenario.

Right now, the CvGameCoreDLL.dll file goes in the Assets folder but I just change the name of the original one to/like CvGameCoreDLL.dl_ and the new ones compiled (which right now is the exact original one trying out the Microsoft Toolkit 2003 thingy is being used to see if it works, and it should since it is the same darn file. Although changes can be done to it, there are players talking "Starcraft" mods, and whatever type Mods, which will make graphic changes to a lot of things, and probably end up being as large as the patch, if they do not watch it -- 46.8mB or so, I guess.!

I mean it is "Out in Space" and "Out in Time" like in --- "Out There" in the Twilight Zone, the Upload Zone, the Ozone, the Loading Zone, the Download Zone, all kinds of Zones, I guess.
:goodjob:

I mean they have to invent rewrite DVD disks for all of it probably.
:lol:
 
Dale:

Actually I have played the game, although admittedly I just bought it this weekend and haven't finished. Not that that means that I shouldn't do what you mention before trying to suggest an actual solution. Apparently you feel it is more trouble than it is worth which is a possibility.

But I wanted to raise the issue to see what people thought... because changes to the main dll that are necessary for a mod to work would be impossible to get to work together without some sort of plugin interface (or an inconvenient and constantly changing combined mod dll) and it would be nice if one COULD be made.

I still think it can... at least to some extent. But let me see if I understand the situation:

Are you saying that ModA and ModB are using two different (and incompatible) approaches to execute the attack? With my limited understanding of the source code that is the only situation in which I can see your problem being an issue. Otherwise I think their could be a standard that allows the mods to work together. I am not trying to impose a specification on people who want to use the SDK. I am just making a suggestion, hoping to encourage people who want to make compatible mods... because I think that would make game play more fun and certainly more interesting if it were possible. If you want to develop a mod that doesn't follow the spec that is not a problem... it just may not work with other mods that use the plugin interface.
 
Top Bottom