Ok, that game just wouldn't recover. Year 30+ and still stuck at under 25 workers with no growth happening.
There are two possible actions that might get you out of that misery:
(1) Reduce the number of available houses to increase the likelyhood of families that live together and breed.
(2) Build a townhall and accept some nomads.
Most efficient crop field size = 4x15 with 1 farmer and right next to a barn / market for 100% efficiency (build a market, surround it with 4x15s of each crop type).
I think you can actually assign even more space. I am mostly using 10x10 fields and they yield 700 food with 1 worker assigned. Biggest problem with small(ish) fields seems to be the single worker: If he is busy with something else during sowing or harvest season, the yield can vary quite a bit...
Most efficient orchard is 4x5, due to the spacing of trees being 3x2 tiles apart. The larger you make them, the more space wasted along the axis of 3 tiles per tree. Better to stick 4x5s in small corners around your settlements, one farmer can still clear them even with a bit of a trek to the nearest storage.
How much does that yield on average per year? Seems a bit like waste of manpower to me...
Biggest food producers = Gatherers Huts and Hunting Lodge, both most efficient with 2 workers in each. For each extra worker in the gatherers hut, I don't even think I saw a 25% increase in the food output, and there's 0 difference with more than 2 hunters (1000 max venison, and usually getting 800-1000 per season with 2 workers in well placed lodges).
Fishermen can also be amazing for the early game if the setup is right (lots of water + nearby house/barn). But gatherer is indeed a no-brainer. Even better: You can make ale from the berries and trade that for other goods.
The advantage to crops and orchards is really just adding lots more health, they aren't very reliable.
Farms are not super efficent in regards of yield per manpower, but they are VERY good in regards of yield per area. A gatherer covers a pretty large area - and as your town grows space and travel distances will actually become a problem.
I don't know what I think of stone houses asap. Its worked well in 3/4 games, then failed miserably in my last one.
Stone houses are a luxury. Early on wooden houses are perfectly fine, better save the stones and get something else for it, like an early trade depot, townhall or market place. Once you have secured a steady source of stone via trade it is, however, a good idea to upgrade houses. It seems to reduce firewood demand quite a bit, which in return means you need less foresters/woodcutters and have more space/manpower for other stuff.
I always build a school first. That way I only ever have 1 uneducated worker. I've not yet found the best opening build plan after that other than houses, but it seems that gatherers huts are the best early food source, and placing two of those in the first year, and then adding a hunting lodge next to both in the second year plus two fisheries, each with 2 workers should work well for food (one fishery with 4 workers = 650 food, same fishery with 2 workers = 430 food, and even less gains on hunters and gatherers as noted above).
Some ways to improve your early game food yield:
(1) Build a forester next to the gatherer, assign 1 guy and set him to "plan" mode. The extra trees should double the yield of your gatherer.
(2) Add a barn + house right next to your gatherer asap. The reduced walking time will also significantly increase yield. The same is also true for fishermen who benefit SIGNIFICANTLY from that.