He didn't give a genuine counteroffer. If he's trying to occupy my territory he's then the aggressor. Which means you can rightfully stay out of it. A DP simply exists to preserve your partner's land, not to go on a war of Conquest for them, regardless of who said the words "I declare war" first.
Also, you didn't "Agree to a defense pact," you joined a Union which Mathalamus invited him into. That's not the same thing.
Also, I'd be willing, if push came to shove, to give 2 XP per territory, but not 5.