To be fair, I only recently discovered how to make smooth-looking icons. Turns out all my previous work looked good by accident, and in many most recent mods for Civ 5, unit flags and civilization symbols have jagged edges as well.
The method is very basic and you can approach it twofold. The main principle is to avoid stark contrast between the icons and the background in the alpha icon atlas - otherwise, the alpha channel will cut the icon out very abruptly, and pixels on the edges will assume the colour of the background, creating a jagged border or a blackened contour. Interestingly, Firaxis made the exact same mistake when it comes to civilization symbols in Civ 6 - look at some city banners, for example Arabia's or Kongo's - you'll notice a darker outline around their civ symbol, and that is true even in the civilopedia. However, equally plausibly, this might have been done on purpose.
First approach is to create your icon atlas in one colour, preferably white - that's right, white background and white icons. This does not matter to the game. What matters is that the alpha channel ought to have black background and white icons, which will then be precisely cut out from the fully white picture in-game.
Second approach, which is essentially the same in principle, but possibly looks better, is to have both the main diffuse and alpha layers have black background and white icons, but the standard diffuse layer ought to have its icons enlarged or broadened in relation to the alpha channel, which needs to have the icons in the exact shapes and dimensions you desire. What you need to do is to select the white icons with a magic wand tool (make sure you select all the white bits if they're disjointed), enlarge the selection by about 5-10 pixels and fill it out with white (on a new layer above, then merge down).
You want to turn this:
into this:
The one below will be your new diffuse layer, while the original atlas should remain as your alpha channel/layer. The result is that the game will now cut out the exact unit flag from a larger white one, thus eliminating any jagged edges. That's how it should look when previewing the alpha channel in Photoshop:
The last crucial thing is to save the atlas in .DDS format using DXT3 or DXT5 encoding method. Those encoding methods require the pictures to have their dimensions as products of the multiplication of 4, so you might need to expand the work area of the picture - at some sizes - by a couple pixels on the right and at the bottom (in this example, the 38px size of the picture would have 72x38px dimensions, meaning that the height has to be expanded by 2 pixels at the bottom, making it 72x40 and thus both values will be divisible by 4). NEVER expand the working area to the left and at the top - that's where the game starts to read the file. This is not to be confused with resizing the entire picture, otherwise the contents of it will be resized as well. Only expand the borders of the picture with a dedicated tool.
I think I've exhausted the topic - hope I was clear enough and I hope it helps!
