When you change any source file, any .cpp file that depends directly or indirectly on that file will need to be recompiled. Since .h files will have more files dependent on them than .cpp files (since you #include .h files, not .cpp files), changing a .h file causes many files to be recompiled.
However, I'm pretty sure there are a lot of unneeded #includes in the source code (look at the #includes under "//jason tests" in CvGameCoreDLL.h). If you really wanted to optimize things you could clean up the includes so that only required files are included, or even split up some files into multiple files (CvInfos.h maybe).
I hear you on the long recompile time though, especially when you just want to test a minor change.