British colonies in the late 18th century built factories simply as buildings where a large number of workers gathered to perform hand labor, usually in textile production, such as power-loom factories... The first power loom, a mechanized loom powered by a drive shaft, was designed in 1784 by Edmund Cartwright and first built in 1785. It allowed textile making to be done far faster than if a human had done the same work. Though it eventually became one of the most important inventions in the industrial revolution, it was initially limited by its reliance on water power, which required workshops equipped with power looms to be located near a source of running water... only later was steam used.
So... as you can see, factories were well established before steam power. I will make coal-plants come with steam power though.