Well, in your case, it's NOT your video card. Your video card must be performing a lot better than the rest of your system, because from what you've described, it's clear that the video card is mostly sitting around doing nothing, waiting for the rest of the system to "deliver the goods". If the rest of your system was keeping the video card busy, you would notice a greater effect on framerate when you change graphics settings.
Here's an analogy. Imagine you and a bunch of people are in a long line at the theater, waiting for the guy to check your ticket. He can't process everyone fast enough, but you and everyone else in line have your ticket in hand, all ready to go. If he could just work faster, you'd all be seated already.
Your video card is waiting on the rest of your system. Whether the bottleneck is the CPU, memory, hard drive, I don't know. Post your system specs and we might figure out what's slowing you down. Include operating system (what version of Windows), CPU speed (how many MHz), size of memory (how many GB), and amount of free hard drive space.
And keep in mind there is always a bottleneck (a pipeline can only move as fast as its slowest component). So if you improve the performance of the rest of the system, it may be the video card that keeps you from getting any better performance. But by that point, performance may be entirely satisfactory.