I was thinking about this earlier today, as I kind of love Byzantium even though they are clearly underpowered.
The truth is that, IMHO, the best thing about this civ is the Dromon. Ranged naval combat from the ancient era is fantastic (and beautifully rendered, to boot.) It makes it worthwhile to spend that first point in Honor so that you can explore the world and make up your culture points and then some.
The UA was fun, but below-par and gimmicky even in G+K. With Reformation it has been significantly nerfed. Cataphract has never even really been on my radar with them. (Though that has more to do with my penchant for naval maps, which decrease the value of mounted units even further.)
One of the problems with the UA is that pantheons are more valuable than religious beliefs are, in many ways. The best way I've found to utilize religions is to get the best-suited pantheon (desert folklore, god of the sea, what have you) and then build my religion around supporting the pantheon. Byzantium's UA forces you (more or less) into choosing a faith-producing pantheon so that you can ensure that you get one more of the less-useful beliefs later. It's an uneven trade. You're trading an early "good" for a later "meh." (Or possibly a later "good," because they can pick from among pantheon beliefs as well, but by that point those will have been well-picked-over.)
What I want to see from them runs the risk of making them Overpowered, but that's become less of an issue as the expansions have given more highly specialized super-bonuses to different civs. I think they need to either get their bonus belief at the pantheon stage. Even if it's just a choice among the available pantheons that will be a great improvement.
Another possibility is to give one of their UUs a faith-per-kill bonus, but that steps on the pictish warriors and the Celts are already underpowered themselves. I think a UA aspect which gives faith bonuses to fulfilling CS missions would be the most keyed-in to Byzantium's history and be powerful while still requiring some effort. If this were the case, then I'd support keeping the rest of the UA as is or perhaps granting a second extra belief, now that their more common and all.