well i like your point of view, but go farther... when you shifted /lost charismatic you lose all benefits... well the units have a bigger level, but when you shift trait, they have again a huge xp gap to cross before changing level. it was only an "haste spell" that is followed by a "exhausted"...I look at it this way. Many attributes offer benefits that extend forward into the future - i.e. I can build certain buildings faster, or get bonuses on unit creation, or whatever, and benefit from that when the attribute changes. However, Charismatic is the only attribute where you get the benefit applied to past actions (i.e. past XP gain). It doesn't seem quite right to me.
other still keep the empire wide boni, without boomerang effect : the quicker buildings, the earned commerce don't become harder to get than normal. while going from lvl10 (earned with charismatic : 76xp) to lvl 11 (117) without.. is much harder than going from lvl10 to lvl11... 101 to 117...
and while leaving organised you keep those command posts.., not only the xp provided but the actual buildings !!