I may be missing it, but have yet to see a good guide to what can be changed other than UI, icons, maps, scenarios etc.
Kael's modding guide, stickied in the References section, is a good place to start.
For example - would really like to modify some behaviors, for example, always frustrated when a SAM attacks and destroys tanks. Or a mobile rocket launched sinks a nuclear submarine.
Here's the thing: behaviors themselves aren't modifiable, in that we can't just arbitrarily tell the AI "don't do that!". What you CAN do is change the units' stats so that they SHOULDN'T do that, and the AI is smart enough to see the change and account for it.
So let's take your two examples.
1> SAMs vs. Tanks.
It's not that the AI is stupidly doing this, it's that
in the vanilla game, there's no reason not to use SAMs as resourceless tanks, since they can do basically everything a tank can do and also have the extra anti-aircraft abilities. A smart player would do the same thing the AI does.
SOLUTION: In my mod, I added a "Melee Penalty" promotion that gives a unit -50% vs. infantry, armor units, etc. and Cities, but NOT vs. siege, aircraft, or naval units. I then gave that to the AA gun and the SAM launcher. Now, SAMs are no longer able to easily attack tanks or cities, but can still do their air defense role just fine. Alternatively, you can use a promotion that gives a generic penalty to attacks (or disables them entirely).
2> Artillery (especially rocket artillery) vs. naval vessels.
The game has an inherent power imbalance. Naval units' stats are WEAK compared to artillery units. The counterbalance is that air units (especially Bombers) have an inherent "Naval Penalty" promotion (-50% vs. naval units), so air units kill artillery much more easily than they kill battleships.
SOLUTIONS: There are many.
A> Increase the power of naval units across the board, but remove the Naval Penalty promotion from the air units.
B> Give artillery the Naval Penalty promotion as well.
C> Find some other useful ability to give naval units. For instance, I mentioned over in another thread that in my mod I'd created a "Damage Reduction" promotion (your unit takes 1 less damage in every combat) for some sci-fi and fantasy units, and then realized that I could give it to Ironclads and Battleships to make them much stronger when facing multiple weak bombardment attacks (Fighters, outdated artillery, leftover crossbows, etc.)
Now, if you only want to address the submarine thing, then you're missing one point: that submarine was invisible until it was detected, presumably through an enemy sub or destroyer. So that sub should never have been near the shoreline in the FIRST place, since it can't attack land units, and out in the deep water it should have been far superior to artillery in terms of killing enemy ships or sinking transports. Your scenario, then, assumes that two unusual things have already broken in your favor; it's not a fair comparison.
---------
Bottom line: we can't directly alter behavior. But we CAN change an awful lot just using the current XML/Lua setup, and as you get a better idea of what can be done, you'll think of dozens of ways to fix each issue you come across.