Because it means nothing to me. It's like swearing on a comic book or store flyer or the phone book. I don't mind promising to keep the information I learn about residents or electors confidential or to do the job I was hired for to the best of my ability while respecting all the rules and laws associated with it (anything learned while conducting the census can't even be revealed to the RCMP, to the annoyance of a cop who asked me about the residents of one house I'd just left as he was pulling up; I told him to contact City Hall and talk to my supervisor).
Why should I swear on a book I don't believe in, or bring in a supernatural being I regard as made-up by humans? It would be essentially lying if I were to do so.
I take my own word seriously, and am glad that there's an option to affirm and sign a piece of paper, rather than swear on the book used by a religion I don't believe in.
The reaction in that room when the City Clerk asked if there was anyone who objected to swearing on a bible and I put up my hand was a collective "GASP!" of disapproval, and the City Clerk hadn't seriously expected anyone to decline to swear on a bible.
It only took an extra 2-3 minutes on his part to get that done for me, but you'd think I'd said something terrible, the way people reacted.