Started a new game without barbs, there's a lot less desyncs now.
The few I had were marked as so in the log:
[1560.587] DBG: Unit out of sync. Player=1, Name=Scout, id=1395, X=41, Y=24
Variable Out Of Sync : CvUnit::m_iEverSelectedCount
[1563.660] DBG: setTimeoutCheck(false)
[1563.660] DBG...