I've done this before and it really wrecks your economy early on. I wouldn't recommend. Walls in a border city by Turn 130 is a reasonable expectation but building Walls as your second building isn't.
Ideally you wouldn't want to ever have to do this but sometimes locking down a key chokepoint can be a game changer. And I've never done that more than once per game.
But the bottom line is that some early game neighbors like Persia, Songhai, Greece, etc will make it impossible to have your cake and eat it too.
Sometimes you have to just respect the sheer early game pressure these civs can dish out, especially if with authority.
Pyramids can also help set up a strong early border, and God of All Creation can give you enough early game to survive