1. We have added the ability to collapse/expand forum categories and widgets on forum home.
    Dismiss Notice
  2. Photobucket has changed its policy concerning hotlinking images and now requires an account with a $399.00 annual fee to allow hotlink. More information is available at: this link.
    Dismiss Notice
  3. All Civ avatars are brought back and available for selection in the Avatar Gallery! There are 945 avatars total.
    Dismiss Notice
  4. To make the site more secure, we have installed SSL certificates and enabled HTTPS for both the main site and forums.
    Dismiss Notice
  5. Civ6 is released! Order now! (Amazon US | Amazon UK | Amazon CA | Amazon DE | Amazon FR)
    Dismiss Notice
  6. Dismiss Notice
  7. Forum account upgrades are available for ad-free browsing.
    Dismiss Notice

XML text cleaner

Discussion in 'Civ4 - Utility Programs' started by j_mie6, Apr 6, 2012.

  1. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    The XML text cleaner! (V3)



    The XML text cleaner is a Python built utility that searches through a mods xml files and makes sure that all Descriptions etc have text tags in the Assets/XML/Text directory

    for example say I had made a new civilization like this:

    Code:
    		<CivilizationInfo>
    			<Type>CIVILIZATION_VENICE</Type>
    			<Description>The Venetian Empire</Description>
    			<ShortDescription>Venice</ShortDescription>
    			<Adjective>Venetian</Adjective>
    			<Civilopedia>Venice is a city in northern Italy bla bla bla...</Civilopedia>
    
    while running the cleaner would convert it to this:

    Code:
    		<CivilizationInfo>
    			<Type>CIVILIZATION_VENICE</Type>
    			<Description>TXT_KEY_VENICE_DESC</Description>
    			<ShortDescription>TXT_KEY_VENICE_SHORT_DESC</ShortDescription>
    			<Adjective>TXT_KEY_VENICE_ADJECTIVE</Adjective>
    			<Civilopedia>TXT_KEY_VENICE_PEDIA</Civilopedia>
    
    with some text tags of

    Code:
    	<TEXT>
    		<Tag>TXT_KEY_VENICE_DESC</Tag>
    		<English>The Venetian Empire</English>
    		<French>The Venetian Empire</French>
    		<German>The Venetian Empire</German>
    		<Italian>The Venetian Empire</Italian>
    		<Spanish>The Venetian Empire</Spanish>
    	</TEXT>
    	<TEXT>
    		<Tag>TXT_KEY_VENICE_SHORT_DESC</Tag>
    		<English>Venice</English>
    		<French>Venice</French>
    		<German>Venice</German>
    		<Italian>Venice</Italian>
    		<Spanish>Venice</Spanish>
    	</TEXT>
    
    etc...

    Installation and intructions for use!

    usage is very easy, first install python 2.7 from www.python.org (this is required for either version).

    open the module with IDLE, then edit the options to fit your mod's xml path/documents (see below) and press F5 to run. (this is is using V1 or if the variable bGraphics is set to False in V2+)

    if using V2/V3 simply doubleclick the XML text cleaner.py file and it will open (like image 2 below) then press the options button to enter mod path etc and then press run Cleaner! In GraphicalInterface.py there are some options that allow you to change the colour, font type, font colour and font size of the main box!

    there are many options that the user can use.
    Code:
    ##Options##
    
    #Enter mod xml location (Files are automatically backed up to the path specified in backupPath
    xmlPath = "C:/Program Files/2K Games/Firaxis Games/Sid Meier's Civilization 4 Complete/Beyond the Sword/Mods/[your mod name]/Assets/XML"
    
    #Backed up files will be places in this path in a folder called XMLBackup, MAKE SURE THIS POINTS TO A VALID FILE PATH!
    backupPath = "C:/Users/User/Documents"
    
    #Here add files that should ignore the checks (for example EffectInfos has many wrong references, but it is on purpose!)
    lIgnores = ["CIV4EffectInfos.xml", ]
    
    #The name of the new text xml (if one doesn't exist this will be the name of the new one!)
    textFilename = "CIV4CustomText.xml"
    
    #Language Options (set to True to create language node in xml!)
    bFrench = True
    bGerman = True
    bItalian = True
    bSpanish = True
    
    #Search Options (set to True to enable searching for this node!)
    bDescription = True
    bPedia = True
    bStrategy = True
    bHelp = True
    bShortDescription = True
    bAdjective = True
    bCityNames = True
    bQuotes = True
    
    if I was to set all the Language options to False any text tags created would be in the form of:

    Code:
    	<TEXT>
    		<Tag>TXT_KEY_SOME_TEXT</Tag>
    		<English>some text</English>
    	</TEXT>
    
    and turning search options to false ignores that type of node (e.g setting bPedia to False would mean that the program wouldn't bother to check the Civilopedia tags for incompete text.)

    When entering paths make sure you use either / or \\ eg

    C:/Users/User/Documents or C:\\Users\\User\\Documents

    When the program starts it will backup the mods files in the path specified in options. When the program runs again it will delete the previous backup and create new ones. Make sure that when you are finished you check that there are no xml errors before running the program again!

    Future features include an xml file for options instead of in the .py file (for non graphical setup) and italian and spanish translation fix, also more intelligent text tag management will be implemented to speed up translation time.

    Post any bugs, questions or requests here!

    Enjoy! :D
     

    Attached Files:

  2. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    Very nice work, thanks for taking up my request :D

    I am going to update my mod in a day or two then im going to run this, since my backup will be uploaded here
     
  3. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    good luck!
     
  4. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    oh and truthfully, this project was a lot of fun and a good learning curve :p Well spent 12 straight hours of programming :rolleyes: :D
     
  5. The_J

    The_J Say No 2 Net Validations Retired Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    29,835
    Location:
    Germany / Netherlands
    Congrats :goodjob:.
    Sounds like an useful thing for everyone who started with the lazy way :D.
     
  6. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    Ok I have finished work on the graphical interface :D

    to run with the graphical interface you just need to double click on the XML text cleaner.py file and it is ready to go. Will upload later but going to put on a teaser picture :p

    Notes
    Will work on autofill of the options box so that user doesn't have to enter the path every single time they use it
     
  7. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    Very interested to see the interface
     
  8. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    Going to upload it by the end of the day (and update the original post, as you can use one of the options in the .py file to turn off graphics (means you don't have to enter in the paths every time, but I hope to fix that soon with my good old fashioned pickle jar (don't ask :p)))

    but the pictures are up now :p

    also you can change the bg colour, font colour, font size and font type of that main box in the GraphicalInterface.py file :D
     
  9. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    Looks really good, this is definatly coming around real nice!
     
  10. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    Version 2 is up!
     
  11. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    Ok I have a method of translation and I am workin gon getting it up and running as soon as possible
     
  12. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    that is nice! how did you figure it out?
     
  13. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    myGengo API :p current having problems with italian and spanish though. The code is telling me they don't work
     
  14. PsiCorps

    PsiCorps FF: Babylon 5 mod team

    Joined:
    Dec 30, 2007
    Messages:
    1,415
    Location:
    Britain
    This Utility program looks like it will be very useful for me however, you said
    Specifically which variation of 2.7? I use Win XP and am not as computer literate as most.

    •Gzipped source tar ball (2.7.3) (sig)
    •Bzipped source tar ball (2.7.3) (sig)
    •XZ source tar ball (2.7.3) (sig)
    •Windows x86 MSI Installer (2.7.3) (sig)
    •Windows x86 MSI program database (2.7.3) (sig)
    •Windows X86-64 MSI Installer (2.7.3) [1] (sig)
    •Windows X86-64 program database (2.7.3) [1] (sig)

    Please help me out.
     
  15. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    Really I should have put python 2.7.? because you just take the latest version ie 2.7.3

    then it is up to you which file you choose however I would say go for the top one on the page:

    &#8226;Python 2.7.3 Windows Installer (Windows binary -- does not include source)
    option as then it will install easily

    hope that helped!
     
  16. PsiCorps

    PsiCorps FF: Babylon 5 mod team

    Joined:
    Dec 30, 2007
    Messages:
    1,415
    Location:
    Britain
    Perfect, just what I needed.

    Thank you.
     
  17. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    any update on the translator?
     
  18. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    haven't got around to that yet, I still want my e-mail reply from the developer on why italian and spanish are both not working
     
  19. j_mie6

    j_mie6 Chieftain

    Joined:
    Dec 20, 2009
    Messages:
    2,962
    Location:
    Bristol (uni)/Swindon (home)
    right the translator is finished! when the cleaner makes a new text tag it auto translates but there is another button on the GUI that allows you to further translate all your mods existing text tags aswell. The danger is that if you have the typical:

    Code:
    <TEXT>
               <Tag>TXT_KEY_ETC</Tag>
               <English></English>
               <French></French>
               <German></German>
               <Italian></Italian>
               <Spanish></Spanish>
    </TEXT>
    
    and only select French translation ALL your text tags will become:

    Code:
    <TEXT>
               <Tag>TXT_KEY_ETC</Tag>
               <English></English>
               <French></French>
    </TEXT>
    
    so make sure you select all the languages you want!!! this will be uploaded soon.

    if the translation fails the tags will be the english equivilents. For the moment the Italian and Spanish translations are not working (something to do with MyGengo) and are typical english.

    How the translator works (for those that are interested):
    Spoiler :

    when the Translate button is clicked/a new tag is made by the program the program then interfaces with the Translator class. In the case of a new tag the translators static get*****() methods are used to collect language *****, it basically sends a request to the MyGengo API and waits for response. With the GUI's Button a new instance of the Translator class is created and passes the path and other options, the __init__ then removes unneeded options. next, doTranslate is called and gets the files from the Text folder, the Text tags are "Replaced" to sort out problems with missing tags (during translation itself) the replaceTags method actually calls the classmethod disectNode() which destroys a TEXT node and throws back its mangled body parts. the replace tags method preceeds to perform life saving operations on the data to reform the tags with the required language tags. now that its all fine doTranslate swaps the language info with translated string and saves the xml (after reformating). finished :p. pretty nice setup I think.
     
  20. dacubz145

    dacubz145 Chieftain

    Joined:
    Jun 21, 2010
    Messages:
    3,730
    Location:
    Windy City
    Can't wait till you upload it:D
     

Share This Page