First of all, most of the Serial Events (like SerialEventImprovementDestroyed) only trigger if the active player can SEE the improvement in question at the time it's destroyed. If the improvement is destroyed off in the fog of war, the trigger won't happen until the active player moves a unit to where he can now see that hex. Now, depending on exactly what you need it for, that might be fine. If you're trying to give some sort of extra benefit on the capture of a barbarian camp, then you'll have problems, but if you just want to give the active player an extra little message when he takes out a camp, it's just fine.
As to your other complaint, you can always use the X and Y value to find out what's on a hex. First you use a function (GetPlotFromXY? I can't check at the moment, but it's something like that) to get a Plot structure for the appropriate hex, and then use Plots:GetImprovementType() to find out what improvement is on the hex. The problem is that if the serial event triggers AFTER the improvement is destroyed, then this won't do you much good. It's far more useful if you're going the other direction, triggering when an improvement is created.
The simple fact is, though, there are no good options for MANY of the things we want. Most of the serial events have UI-related issues, and the GameEvents (which are much more reliable to use) are very limited in number, with only a couple dozen made so far. So in a lot of cases, you either have to do without, or find some less direct method. For instance, there's no event that triggers when a Building is completed in a city. Mods like the heavily used "Building Resources" mod component get around this by creating an array storing exactly which buildings are completed in each city, and then at the start of each turn, checking the new values against what was stored on the previous turn. If the two lists are different, then a new building must have been completed; if it's of a certain type, then you can trigger a new effect. It's not instantaneous, since it won't trigger until the start of the next turn, and there are major issues when loading an in-progress savegame, but it does work fairly well.
So you could, at the start of each turn, loop over every hex on the map, recording the number and positions of those containing barbarian camps. If one is missing at the start of the next turn, then just check to see whose unit (if any) is sitting on the hex in question. Chances are, it's whoever defeated the camp. Obviously this adds significant overhead (scanning the entire map), and it's abuseable (kill whatever unit DID clear the camp and then move onto that hex before the next check), but it's about as close as you'll get until we get full DLL access.