The ChatGPT thread reminded me of something I experimented with a few months ago but never shared: using Stable Diffusion to generate Civ 3 graphics. For anyone who doesn't know, Stable Diffusion is an ML model that generates images given some text prompt. It was trained on hundreds of millions (at least, maybe billions) of image + caption pairs scraped from the internet. There are several models like it out there but what makes Stable Diffusion unique is that it's readily available and small enough to run on a PC with a strong graphics card.
The most success I had with it was with generating leaderhead images. Here are some examples:
Prompt: zwx portrait of Barack Obama
Prompt: zwx portrait of Donald Trump
Prompt: zwx portrait of Elon Musk on Mars, in the background is a futuristic colony on Mars
I tried a more complex prompt for this one but it didn't quite work, notice there's no colony in the background. In general, SD can't construct scenes. It's best to give it a short 5-10 word description of a single object, beyond that it mostly ignores what you've entered.
To create these images I fine tuned the model using a technique called "dreambooth". It's a way to teach the model about a subject or style using a set of examples. I simply used all 33 static, smiling leaderhead images from the game's files as examples. As part of the process, the model associates what it's learning with a special phrase. I chose "zwx portrait", which is why that phrase is in all the prompts, without it the model wouldn't know to use the Civ 3 style. Without the fine tuning, I wasn't able to get good results.
The subject doesn't have to be a person, just something the model knows about, for example:
Prompt: zwx portrait of a ((dog))
The parenthesis here cause the model to put more emphasis on the word "dog". Without the emphasis the model would mostly produce pictures of people or weird human/dog hybrids. Presumably that's because the training has biased it toward producing images of people with the Civ 3 style since that's what all the training examples contained.
Prompt: zwx portrait of a pale skin (zombie)
However, the results won't be as good if the subject is not something the model knows well. It's seen Obama, Trump, and Musk probably hundreds of thousands of times during training and so can reproduce them very well. I also tried some lesser known celebrities and the results were disappointing, the figures in the pictures were often not recognizable. Another note on quality, all of the above images were cherry picked. Each is the best one out of 10 or 20 generations.
I also played around with generating tech icons and building sprites. It's easy to get good tech icons out of Stable Diffusion, "good" meaning icons that look like they could have been part of the original game. That's because the original icons are simple and abstract and don't have a distinctive style outside of being late '90s CG. I tried to get the model to employ some creativity when producing these, for example prompting it with "an illustration of the concept of Feudalism", hoping it would come up with some interesting details on its own. The results never impressed me. In general Stable Diffusion is not that imaginative. A better approach would be to ask ChatGPT to come up with a description for an image.
Stable Diffusion is decently capable of producing sprites out of the box, however I wasn't able to get it to generate game ready sprites for Civ 3. It tends to generate entire sprite scenes without much detail instead of single detailed buildings. It might just be a matter of finding the right prompt, or maybe it's necessary to fine tune on the Civ 3 building graphics. I tried using a little "few shot" learning setup, where I created a grid of four sprites, three taken from the game and the last slot left open for SD to fill in. Unfortunately that did not work at all. The model either copied one of the example sprites or generated an ugly blob.
The most success I had with it was with generating leaderhead images. Here are some examples:
Prompt: zwx portrait of Barack Obama
Prompt: zwx portrait of Donald Trump
Prompt: zwx portrait of Elon Musk on Mars, in the background is a futuristic colony on Mars
I tried a more complex prompt for this one but it didn't quite work, notice there's no colony in the background. In general, SD can't construct scenes. It's best to give it a short 5-10 word description of a single object, beyond that it mostly ignores what you've entered.
To create these images I fine tuned the model using a technique called "dreambooth". It's a way to teach the model about a subject or style using a set of examples. I simply used all 33 static, smiling leaderhead images from the game's files as examples. As part of the process, the model associates what it's learning with a special phrase. I chose "zwx portrait", which is why that phrase is in all the prompts, without it the model wouldn't know to use the Civ 3 style. Without the fine tuning, I wasn't able to get good results.
The subject doesn't have to be a person, just something the model knows about, for example:
Prompt: zwx portrait of a ((dog))
The parenthesis here cause the model to put more emphasis on the word "dog". Without the emphasis the model would mostly produce pictures of people or weird human/dog hybrids. Presumably that's because the training has biased it toward producing images of people with the Civ 3 style since that's what all the training examples contained.
Prompt: zwx portrait of a pale skin (zombie)
However, the results won't be as good if the subject is not something the model knows well. It's seen Obama, Trump, and Musk probably hundreds of thousands of times during training and so can reproduce them very well. I also tried some lesser known celebrities and the results were disappointing, the figures in the pictures were often not recognizable. Another note on quality, all of the above images were cherry picked. Each is the best one out of 10 or 20 generations.
I also played around with generating tech icons and building sprites. It's easy to get good tech icons out of Stable Diffusion, "good" meaning icons that look like they could have been part of the original game. That's because the original icons are simple and abstract and don't have a distinctive style outside of being late '90s CG. I tried to get the model to employ some creativity when producing these, for example prompting it with "an illustration of the concept of Feudalism", hoping it would come up with some interesting details on its own. The results never impressed me. In general Stable Diffusion is not that imaginative. A better approach would be to ask ChatGPT to come up with a description for an image.
Stable Diffusion is decently capable of producing sprites out of the box, however I wasn't able to get it to generate game ready sprites for Civ 3. It tends to generate entire sprite scenes without much detail instead of single detailed buildings. It might just be a matter of finding the right prompt, or maybe it's necessary to fine tune on the Civ 3 building graphics. I tried using a little "few shot" learning setup, where I created a grid of four sprites, three taken from the game and the last slot left open for SD to fill in. Unfortunately that did not work at all. The model either copied one of the example sprites or generated an ugly blob.
)
