I have run some tests as well, and I cannot properly make out what is important. I had a brilliant starting position by the way, on a river with a cow. Also within the 9 tile city limits were a BG and a forest on the river.
First of all, using JMK's setup I showed that you do not get the extra commerce from the extra tile.
Second, if I set the governor to emphasize production and let my worker just sit, the governor would choose the forest (1/2/1) upon growth. If I had however set the governor to emphasize food he would choose the BG (2/1/1).
However, when I let my worker put a road on the BG, the governor would pick that tile pver the forest even when put on emphasize production (of course the BG had added value now with an extra trade).
However, if I had irrigated the cow and put a road on the BG, the governor would once again choose the forest over the BG.
My conclusion is that I don't know why the governor chooses what, but that the amount of surplus food definitely plays a part, just as the governor settings, and also that the governor sometimes prefers extra food and commerce over extra production, even if you want him to emphasize production. (This means that if in a settler factory the governor chooses the wrong tile, you could put a road on it to make it more attractive, even if you never actually get the extra commerce from that tile).
Upon expansion to larger cultural values I notice one nice thing. The governor resets the tiles worked and once again you get the food from the old configuration (and probably also the commerce) but the production from the new. In my example, where my city was size 2 at expansion and worked the cow and the BG (and the worker had improved nothing), I would get +3 food and +4 production for that turn, which is the best of both worlds.