Correct, there's no way to remove a district. As for alternative uses of faith, two major uses come immediately to mind:
1. Faith purchases of Great People -- let regular Great Person Points accumulate and when you see, e.g., a Great Scientist you really want (Isaac Newton is my favorite - free library and university in one city, and all universities add +2 science -- yes, please!), you can sneak in and buy him for the price of your accumulated Great Scientist points, plus some amount of faith (less faith the closer you are to being able to buy him with Great Scientist points).
2. Faith purchases in your cities after a religion has spread to your city -- depending on the beliefs of a religion, or certain governments or policy cards, you may be able to use faith to buy a religious building in your city (all give some amount of faith, plus another benefit, like housing, religious art slot, science, culture etc.), or city center buildings, or campus and theater buildings, or military units. Later in the game, you can buy Naturalists with faith.