Well the current cost of keeping a horse starts at about £20 per week.
http://www.equine-world.co.uk/buying_horses/cost_horse.asp
The cost of a bicycle is going to be in the region of £3 per week. Service, tyres and repairs.
http://www.halfords.com
The bicycle is also cheaper to buy than a horse.
I do not know the relative rates of inflation over the last hundred years but it seems probable that the relative costs would be similar.
The other problem is the minimum area of land required to keep a horse seems to be in the region of 3000m2 but you would have to give it more feed. So you cannot keep your horses in your terrace house. You have to walk to the land you where you keep your horse. You also have to leave the horse in a place where it has access to food and water when you go somewhere. So a horse is not really a practically means of transport for a person to travel from a terraced house to a factory ignoring the cost.
The bicycle is also faster than the horse.
So yes the bicycle is really just a cheap alternative to the horse but it was a horse that was affordable and useable by the working class.