If you DO unfortify it, it will loose the bonus. But if you just clicked it it won't be automatically unfortified; just click on another unit, or if none available end turn. At the begining of the next turn, though, you will have this very same fortified unit selected, so be sure to click on...