Possible Future Direction (personal view)

[primem0ver]

Database stuff is probably my biggest knowledge gap when it come to programming. So I am curious. The thing that worries me about using database stuff and queries is that the queries have to be translated and then functions called. Is it possible for standard queries to be inlined and exported to the executable during compilation?

Let's use AIAndy's example:

As an example a building effect that should apply to all cities of the team.
That can be expressed by an SQL query that looks somewhat like this:
SELECT E.city
FROM buildings A
INNER JOIN cities B ON A.city = B.city
INNER JOIN players C ON B.owner = C.player
INNER JOIN players D ON C.team = D.team
INNER JOIN cities E ON D.player = E.owner
WHERE A.building = THIS_ENTITY;

Let's assume for this example that this query is contained in a method in a C# class: ApplyBuildingEffectOnAllCities(THIS_ENTITY, E) (or whatever parameters are necessary)

Within this method is the code above. Assuming all the variables above are variables accessible from the C# code, can the resulting query algorithms be compiled directly into the ApplyBuildingEffectOnAllCities(...) method or does a running sql engine have process the script and then perform the necessary tasks?

 

[Laskaris]

I know we've had a lot of discussion here on the tile methods we could/should employ. And I realize I'm probably not saying anything that hasn't been said in a manner that I may not have understood . But my proposition is a bit like the above.

[...]

The problem I see with this is that, in arranging square tiles in such a brick-like manner, you are basically throwing away the two main advantages that a (regular) square grid has over a hex grid: 1) being able to move in 8 rather than 6 directions from any one tile, and 2) movement being more intuitive to some players than on a hex map because you can move in the four cardinal directions in a straight line. So, I would argue that your proposal probably would not satisfy many of those players who prefer a square grid.

Also, as primem0ver has pointed out, you can't actually make a globe from such a square grid, either.

The question I ask myself is, should we really try to pacify both the square and the hex tile camps? There is something to be said for that, but there is also something to be said for just deciding on one system: less work, and purity of concept. If we do try to have the engine work with squares as well as hexes, I suggest using a regular square tiling system, however, because that is what proponents of square tiles actually want.

 

[AIAndy]

Database stuff is probably my biggest knowledge gap when it come to programming. So I am curious. The thing that worries me about using database stuff and queries is that the queries have to be translated and then functions called. Is it possible for standard queries to be inlined and exported to the executable during compilation?

Let's use AIAndy's example:


Let's assume for this example that this query is contained in a method in a C# class: ApplyBuildingEffectOnAllCities(THIS_ENTITY, E) (or whatever parameters are necessary)

Within this method is the code above. Assuming all the variables above are variables accessible from the C# code, can the resulting query algorithms be compiled directly into the ApplyBuildingEffectOnAllCities(...) method or does a running sql engine have process the script and then perform the necessary tasks?

If we use something like SQlite which runs in the process itself, then you can prepare/compile the query in advance so it is only done once at startup of the program. Using a parametrized query you can still change the entity you actually want to query when you run it.

And to deal with relations that are not explicitely available in table form we would need to parse the SQL ourselves before that anyway.

 

[Dancing Hoskuld]

Database stuff is probably my biggest knowledge gap when it come to programming. So I am curious. The thing that worries me about using database stuff and queries is that the queries have to be translated and then functions called. Is it possible for standard queries to be inlined and exported to the executable during compilation?

Let's use AIAndy's example:


Let's assume for this example that this query is contained in a method in a C# class: ApplyBuildingEffectOnAllCities(THIS_ENTITY, E) (or whatever parameters are necessary)

Within this method is the code above. Assuming all the variables above are variables accessible from the C# code, can the resulting query algorithms be compiled directly into the ApplyBuildingEffectOnAllCities(...) method or does a running sql engine have process the script and then perform the necessary tasks?

In general I have found this not to be necessary these days. The sql engines are fast enough, I even demonstrated a number of cases where calling the sql engine for two simpler queries that did the same thing as a complex was much faster than doing the full complex query inline. True, we were only processing and matching about 100,000 entities on about 10 criteria over 3 data sources and doing statistical functions on 30-500 properties of those entities - used about 15 different sql statements as opposed to the c program which did it all inline - the sql was significantly faster in that case and much easier to maintain when the client wanted to change the matching or add extra statistical functions. Which of course they did.

If we use something like SQlite which runs in the process itself, then you can prepare/compile the query in advance so it is only done once at startup of the program. Using a parametrized query you can still change the entity you actually want to query when you run it.

And to deal with relations that are not explicitely available in table form we would need to parse the SQL ourselves before that anyway.

And if it is part of the conversion of the data storage model to the use models then it would also be set up/complied before hand as part of the system anyway.

 

[ls612]

If we use something like SQlite which runs in the process itself, then you can prepare/compile the query in advance so it is only done once at startup of the program. Using a parametrized query you can still change the entity you actually want to query when you run it.

And to deal with relations that are not explicitely available in table form we would need to parse the SQL ourselves before that anyway.

Like primem0ver, I am also not very familiar with databases in general and SQL in particular, that is one of my major shortcomings as a programmer IMO. I do like SQLlite though, as that is what Tortoise uses for it's stuff IIRC.

Being in the position though of probably designing all new content & mechanics for this I do like the options and flexibility it gives us for relationships. Also, when you say parse the SQL ourselves do you mean with C# code in the engine?

 

[AIAndy]

Like primem0ver, I am also not very familiar with databases in general and SQL in particular, that is one of my major shortcomings as a programmer IMO. I do like SQLlite though, as that is what Tortoise uses for it's stuff IIRC.

Being in the position though of probably designing all new content & mechanics for this I do like the options and flexibility it gives us for relationships. Also, when you say parse the SQL ourselves do you mean with C# code in the engine?

Yeah, the database engine can only run SQL queries on explicit relations which means those that are available in table form.
But we will also have relations that are too expensive to have as table like maybe for each plot all plots that are within range X (with X having different values).

An SQL statement on that still has meaning, but we have to execute it ourselves in C# code:

SELECT B.unit
FROM PlotsInRange(10) A
INNER JOIN units B ON A.targetplot = B.plot
WHERE A.sourceplot = THIS_ENTITY;

That should select all units that are in range 10 of a certain plot but of course PlotsInRange(10) is not a table.

 

[Koshling]

Yeah, the database engine can only run SQL queries on explicit relations which means those that are available in table form.
But we will also have relations that are too expensive to have as table like maybe for each plot all plots that are within range X (with X having different values).

An SQL statement on that still has meaning, but we have to execute it ourselves in C# code:

SELECT B.unit
FROM PlotsInRange(10) A
INNER JOIN units B ON A.targetplot = B.plot
WHERE A.sourceplot = THIS_ENTITY;

That should select all units that are in range 10 of a certain plot but of course PlotsInRange(10) is not a table.

IMO we should define relationships in abstract (at the definition level) with our own (XML) syntax. Whether they map directly onto SQL selects, or some code-based construction using zero or more actual SQL-level SELECTs then becomes an implementation issue there is no need to expose.

In terms of C# functions to add a building to very city - there won't be one! The whole point of doing things declaratively in the XML is that all you need to do is define the relationships and some event triggers and let the rest 'just happen'. For this example:

When building A is built add building B to all cities (with a completely random made-up pseudo-syntax for illustrative purposes since it's not yet decided at all):

Event is

CREATION_OF_ENTITY MATCHING 'buildingType==BUILDING_TYPE_A'

Triggers:

CREATE_ENTITY type=EntityTypeBuilding INSTANTIATION_VALUES:
buildingType=BUILDING_TYPE_B
IN_PARENTS MATCHING
'entityType == EntityTypeCity' &&
RELATION(isSameOwner, RELATION(parentCityOf(TRIGGER_ENTITY))

where the relationships 'isSameOwner' and 'parentCityOf' are themselves declaratively defined by the XML of the game ruleset.

The nearest likely C# method would be one to explicitly process a specified event (so effectively you could have events with no declarative triggers and those would be invoked by code, however I doubt many such are need except for UI actions)

 

[ls612]

@Koshling:

Do you know when you will post the blog post about the software layering? Because it seems that we are discussing that now, it would be good to have a baseline to discuss.

 

[Koshling]

@Koshling:

Do you know when you will post the blog post about the software layering? Because it seems that we are discussing that now, it would be good to have a baseline to discuss.

Hopefully tomorrow, but no promises

 

[primem0ver]

Hopefully tomorrow, but no promises

I also take it you don't have a formal opinion yet on our map discussion.

 

[AIAndy]

IMO we should define relationships in abstract (at the definition level) with our own (XML) syntax. Whether they map directly onto SQL selects, or some code-based construction using zero or more actual SQL-level SELECTs then becomes an implementation issue there is no need to expose.

I am not intending to expose. If we actually pass it through directly does not matter but there is no real gain from implementing our own language that in the end has the same feature set as SQL.
Consider the selects as working on abstract relations, not tables. So you are not querying database tables, you write down expressions of relational algebra:
http://en.wikipedia.org/wiki/Relational_algebra

So either we write our own XML based query language and then translate it into one or more database SQL queries or internal operations on non explicit relations or we use SQL in XML tags and then translate it into database SQL queries or internal operations.
I don't see the gain from implementing our own language. Unless you don't like the syntax or think that there are missing language features?

 

[Koshling]

I also take it you don't have a formal opinion yet on our map discussion.

I do actually. I'll respond in due course - just very busy atm. Next post but two I'm now planning to do a reprise of the comments and suggestions made and how it probably modifies/makes choices on things.

 

[Koshling]

I am not intending to expose. If we actually pass it through directly does not matter but there is no real gain from implementing our own language that in the end has the same feature set as SQL.
Consider the selects as working on abstract relations, not tables. So you are not querying database tables, you write down expressions of relational algebra:
http://en.wikipedia.org/wiki/Relational_algebra

So either we write our own XML based query language and then translate it into one or more database SQL queries or internal operations on non explicit relations or we use SQL in XML tags and then translate it into database SQL queries or internal operations.
I don't see the gain from implementing our own language. Unless you don't like the syntax or think that there are missing language features?

Fair point. I'm not really a SQL guy. What we need is likely a subset, and given that we are not exposing a table-based schema (at least explicitly), but an entity-relation one, it isn't (yet) clear to me that things like JOINs will necessarily directly map onto the concepts we need when defining relations. However, it may well be that it does all map directly (at least the subset we need, which is likely not full SQL), in which case there would indeed be no point re-inventing the wheel and we can use existing SQL syntax.

 

[AIAndy]

Fair point. I'm not really a SQL guy. What we need is likely a subset, and given that we are not exposing a table-based schema (at least explicitly), but an entity-relation one, it isn't (yet) clear to me that things like JOINs will necessarily directly map onto the concepts we need when defining relations. However, it may well be that it does all map directly (at least the subset we need, which is likely not full SQL), in which case there would indeed be no point re-inventing the wheel and we can use existing SQL syntax.

If we have a relation that connects cities with their owners and a relation that connects players with their teams, then the INNER JOIN of those relations ON player = owner is a new relation that connects cities with their teams.
Or you self join the city/owner relation on the owner and get a relation between a city and all cities of the same player.
A join of the improvement/plot relation with the city/working plot relation results in a relation between a city and the improvements belonging to it.

 

[Laskaris]

@ primem0ver: After thinking some more about the idea of using points instead of tiles, I must say that I still have reservations about it. On a tile grid, it is clear and obvious which tile has which terrain, but with points and edges between two different terrains, it is not clear, and that to me is a major drawback. It is true that technical solutions for this could be found, such as the ones you proposed:

However, this won't affect much when it comes to the AI since we can assume that humans are intelligent and can pick the side of the boundary that suites their purpose. For example, in the case of traveling along an edge between woods and bare plains, if speed is what is important, the lower movement cost is used (i.e. the units travel on the side of the line that is the plains rather than the side that is the woods). For stealth units, the AI would be forced to pick whatever movement cost offers the best stealth (i.e. the woods value).

To solve this problem there are a few things to keep in perspective. First, don't forget that the resolution when using vertices (points) is around twice as precise. The distance between two points is on average half the radius of a tile. So the shoreline itself should be considered land and the half "tile" away point is considered sea. This could be applied to all shoreline points except where cities and edifices are built that support land and sea units occupying the same space. This same logic could be applied in other circumstances where ambiguity becomes a problem.

As far as the graphics are concerned, we would need to make sure that beach graphics extend over the line just a touch which isn't too hard to accomplish. We already see this happening in tile based graphics where one tiles texture sometimes reaches over into the next tile.

However, while these solutions might be technically valid, I would argue that this aspect of a points-and-edges-based map will remain counter-intuitive to players. Players will always have to ask themselves: "Okay, here I am, on the point between a forest and a plain. So what terrain was this again?" My worry is that we will be turning the map display, i.e. something that should be intuitive and effortless to use, into something that requires a lot of conscious attention from the player.

If we do points on an ISEA grid instead of using tiles, the "tiles" between the points are more or less equilateral triangles. ALL rendering is based on triangles so creating graphics for an ISEA grid would be very simple both in terms of creation and rendering.

One of the main advantages of your proposed point-based map is the graphical rendering. My question: would it be possible to use triangles for rendering, while using hex (and a couple of pentagon) tiles, composed from these triangles, for the actual map tiles? In other words, would it be possible to use the graphical advantage of the point-based map while still using a tile-based map?

 

[ls612]

One of the main advantages of your proposed point-based map is the graphical rendering. My question: would it be possible to use triangles for rendering, while using hex (and a couple of pentagon) tiles, composed from these triangles, for the actual map tiles? In other words, would it be possible to use the graphical advantage of the point-based map while still using a tile-based map?

We can probably make the rendering somewhat independent from the tiles, so that graphically the terrain looks more natural, while still making each tile have specific and limited terrains.

 

[AIAndy]

We can probably make the rendering somewhat independent from the tiles, so that graphically the terrain looks more natural, while still making each tile have specific and limited terrains.

Yeah, I guess we should go for some procedural terrain graphics generation that is based on the gameplay assignment of terrain types, height maps and the like but does not follow it strictly.

 

[masaykh]

Interesting!

What exactly formats Gamebryo using?
NIF and KFM format can be ported using http://sourceforge.net/projects/niftools/files/
DDS microsoft format so can be used too

I`m trying to say that atleast some of assets can be safely ported to new project isnt?

one more question - Is it not possible to make freeciv port \ mod?

 

[ls612]

Interesting!

What exactly formats Gamebryo using?
NIF and KFM format can be ported using http://sourceforge.net/projects/niftools/files/
DDS microsoft format so can be used too

I`m trying to say that atleast some of assets can be safely ported to new project isnt?

one more question - Is it not possible to make freeciv port \ mod?

No. Technically we can't transfer anything from C2C assets-wise, or code-wise for that matter.

 

[masaykh]

If you can please tell me why?

3d\2d models can be opened\used via niftools so if it is not troublesome to you - can you give brief description?