Mostly it sets the stage for later diplomatic relations, but there are are side benefits just to being able to see their capital:
You'll know the population and defenses of the capital (good if you're planning an invasion)
You'll see improvements, plus surrounding (knowing if it's on a coast is handy for the above too)
You can place a spy in the capital (which will allow you to actually see without fog around the city)
You'll count as having "discovered" the civ if you have a City-State quest out there
Keep in mind that having an embassy of their in your capital allows all of that stuff, too. In the early game, especially at higher difficulties, the AI players may have vastly higher military strength than you, and seeing that you don't have walls and the location of your territory could prompt them to invade.