The problem is that you cannot raze these cities due to your culture there. If you play in Emperor and allow the Arabians to survive, they'll rapidly outtech you and it will be impossible to recapture Aleppo or Antioch.
What exactly does the Byzantine core consist of? (I thought Iconium was part of it and therefore starting to build walls/church there)
You can raze them usually after the second time they declare independence. So far, I'm pretty sure the core area is the part marked by the Byzantine spawn zone in the reference folder. Your starting cities in Greece, Thrace, Constantinople, and the two Western most cities in Anatolia.
In a recent build I was able to capture Antioch, and about 2 turns shy of Aleppo, so it is certainly possible with catapracts and bombards. With Byzantium your upkeep costs are so high, any city of your core area you need to shed ASAP or it will bring down your research even without reaching point where your techs cost more.