-this is how it works-if it's got too little production for an obelisk, then it's not worth building a wall(90%For Castles down the road, unless that prereq has been removed since I last tested.
Units also = cost. A few surviving units that gain xp while defending behind a wall are way more valuable than lots of green units that continually get wiped out and in doing so give the enemy xp.
I don't focus on culture too much early on with my frontier towns either.

-now if culture based defense started 1 border expansion later then it would be more balanced