Basically when I declare independence I'm lucky to have even 1/10 an army the size of the REF, and there is no way to even clear out the initial landing of troops when they first arrive.
This is just poor planning on your part. You don't need to match the size of the REF but it should be entirely possible to fight it well. You will never be close to the REF in size at the beginning of the war of independence, but you have many advantages against it nonetheless.
Normal soldiers don't suffice to attack the REF stacks, you seem to be required to have Veteran Soldiers operating as Dragoons, but since you can no longer educate Veteran Soldiers but must buy them, the quantity is limited.
Normal soldiers are functionally the same as veteran soldiers. The only difference is that veteran soldiers get XP faster. It's good to have some dragoons but you don't need every unit to be a dragoon. Dragoons are especially good against artillery. I don't know about not being able to educate veteran soldiers. I thought it was possible, just very expensive. Even still, you don't need your entire army to be veteran soldiers. They just get an XP boost, which is very helpful, of course, especially if you trained them on the natives and rival colonies a bit in the years beforehand. As far as realism, this represents how there were some american soldiers who were veteran from the french and indians war. Also, militia was important in the continental army. Militia is important still now. You probably don't need those sugar planters and cigar makers as much at this point, so raise them into a militia. Make sure to procure a lot of guns before the war in order to raise a large army (but once again, numbers don't matter as much as you think).
Gunpowder units (foot soldiers) are actually very good though because of the promotions they can get that mounted units can't. For example, the ranger and mountaineer promotion lines are actually probably debatably the strongest in the game. Only gunpowder units can have them. They give bigger bonuses than just taking veteran, and there are a ton of forests, hills, and mountains, and ranger II and mountaineer II double your movement which can be helpful. I have a feeling that you neglect to take these promotions. The civilopedia actually hints at the strength of these promotions in a couple places but I for the life of me can't find where now, but I know it does make some explicit comment about the two of them being key to victory. This is actually realistic because it represents the importance that guerilla warfare played in the revolutionary war. Also, a couple of the military founding fathers give free promotions to only gunpowder units, which can be very beneficial (paul de mais and ethan allen). These are high priority founding fathers.
Is there any way to win without essentially gaming the system? I've read a lot of the strategy posts here and every one seems to advocate for exploiting a game mechanic- like making your key city a one-tile island or producing no liberty bells at all until ~20 turns or so before you want to go independent to minimize the size of the REF. t seems like if you want to play an 'average' game (I know, I know, what does that really mean?) then the game is unwinnable.
You just need to have a coherent strategy, you can't do whatever you want. Unlike regular civ 4, the lowest difficulty is actually still really hard so it's kind of demoralizing for newbies when they can't get a free win like on the lower difficulties of regular civ 4. The one tile island strategy does sound stupid, I wouldn't recommend playing THAT gamey, either. But if your definition of "average" game, is playing however you want with no regard for long term planning aimed at winning independence until the perhaps near the end, then you will lose. Every single decision from the start of the game, has to be a step towards getting independence, not just "creating an empire" however you want, which is a mindset that some newbies have coming from regular civ.
The liberty bell thing is a little more reasonable as a strategy but is not the only strategy you could go with. Some people do believe that strategy is superior but it is a debated point. Even the large guide about REF and rebel sentiment which presented that idea said it is not the only way to play, but IF your strategy is to minimize REF, this is how you do it. But you don't actually need to minimize REF, which is the point I was telling you above. You just need to play well. The REF will always outnumber you by a lot, and yes, it is very strong, but no, it is not impossible circumstances unless you actually are woefully unprepared.
Nearly every change added over the DOS version seems designed to add negative feedback to the game mechanics to punish players for simply enjoying the game.
Well yes, that is what makes a good game, is choices and decisions. You are framing this in a loaded manner though, it's not that you get punished for enjoying the game, it's that there are tradeoffs to every choice and you can't have everything or have it exactly the way you want it. You can't have your cake and eat it too.
Do you enjoy managing a large empire? Well, now anyone outside your towns also needs rebel sentiment.
This doesn't mean you can't have a "large empire", but once again don't fall into the mindset of "I should be able to play however I want with no regard for tradeoffs but I can't so therefore this is a bad game". If you actually want to play with a large population, then nobody is stopping you from doing that, but there are weaknesses to that strategy as much as there are strengths, similar to how playing with a small population has strengths and weaknesses too. The way you win is by recognizing what those strengths and weaknesses are and playing to them accordingly.
Plus, imagine if people outside your towns didn't need to be "converted" to rebels. Then you could just take everyone out of your towns except a single statesmen and get that 50% rebel sentiment pretty much immediately, like 1 or 2 turns. It would be so OP. It actually turns out though that what you want to do is still similar to this. You do want to generate your overall rebel sentiment in smaller towns in order to get that last (or first?) push for the rebel 50% regardless of whether or not you did the strategy where you waited to generate liberty bells. This is because of two things, which you may or may not already know. First of all, overall rebel sentiment is ultimately based on your bells produced per turn in all your colonies divided in some manner by your total population (including outside units). So it doesn't matter where you are producing these bells, and there is essentially going to be an overflow to the overall number even if an individual city is already at 100%. So basically don't worry about wasting liberty bells in small cities with maxed rebel sentiment. In fact, this is actually the best place to produce them due to the second point, which is that cities with high rebel sentiment actually produce liberty bells faster. Larger cities take longer to grow rebel sentiment so it will take them longer to get this bonus +%. So smaller cities are actually really the best place to accelerate and quickly get that rebel sentiment in order to declare independence. Once again, this doesn't mean you have to wait to only produce liberty bells in small cities at the end to game the system, that's just one strategy. You can produce liberty bells in the large cities to help out with production and get a headstart on rebel sentiment if you want throughout the game, but just be aware the REF will be larger. But, due to the fact your liberty bells increased production of food and everything else, you might have a larger army as a result of doing this. But yes, it basically is made so that the REF will almost always be larger than your army, but you just need to manage things well and have a gameplan for the war. One benefit of having a larger population is that once the war starts, you will have more production capability as opposed to a smaller population, and this won't effect the REF. But to take true advantage of this you really need to have managed well enough to be able to roll out 1-2 new units per turn across your colonies during the war (even if they are militia).
Like making money? Well, now the king will just randomly demand any or all of your money.
I don't really think this is a big complaint, if the king demands a boat load of money, you can just deny him. Once again, just need to measure the tradeoffs. It would depend on how much gold he is asking for and how much you need to be on his good side. A happy king will provide you cheap units to use against him later (if you didn't know this, you can buy units from your king in the diplomacy screen).
Do you enjoy wars? Well, now the Continental Army is gone and you can no longer train professional soldiers in your schools.
Again I don't know if this is true but I don't think I've ever trained a veteran soldier in a school although I *thought* I was given to option to at some points where I had a lot of gold. I am pretty sure they are one of the more expensive majors in the universities if they are there. I've heard a couple people on this forum say this though actually, but I really didn't think this is the case, but whatever. Veteran soldiers have the potential to be really OP anyways actually, so it's just a matter of tradeoffs. Once you start racking up kills during the REF landings, veteran soldiers could be getting really OP really quick. You don't need a million of them.
War at sea also much harder to win, etc
War at sea is definitely not easier than war on land in this game but it is still feasible, but most people don't recommend it, because you don't need to destroy their navy to win. Although, I will say that ships of the line have the potential to have 12 strength or very close to it and match the man o' wars through rebel sentiment. If you didn't know this, overall rebel sentiment increases your troops' strength with a +% modifier, including ships, for a maximum of +50% due to +100% rebel sentiment. To eventually get +100% rebel sentiment you need to be generating 4 bells per turn per population. That means 2 elder statesmen in newspaper towns per every 6 citizens. That actually is a lot, but doable, but also not necessary to win, and once again it all comes down to your strategy and weighing your options long term and short term throughout the game. If you have a few liberty bell founding fathers, which there are plenty of, then you actually need less than that, even. You also don't really need to get to 100% to have a fighting chance on the water as long as you stay close to the docks to hide and heal and pick off lone targets, but yeah, the rebel sentiment helps a lot especially with naval warfare where there are not a lot of other bonuses otherwise unlike on land, where there are all kinds of tricks and strategies and bonuses. The only other thing I would say about naval combat is to just accept that you're gonna lose some ships doing that, and try to keep your ships with promotions alive. Another thing you can do is attach a great general to a ship (I've never done this but I've heard it is possible to attach a general to a ship so don't take my word for it), and make that ship really OP.
I do like the cultural borders carrying over from Civ, or I would, if it didn't require Liberty Bells, the production of which make the game unwinnable for me.
Basically the point here is that the game requires some planning and a bit of micromanagement. Liberty bells are actually really good.... this is actually a really good game and I recommend you try to get better at it instead of writing it off as broken because you aren't immediately good at it....
