You need transports to invade. Colony ship will only work, if the planet is uninhabited. The colony ship also has to have the proper environment. What that measn is you cannot use a std colony ship to build a colony on a toxic or any environment other than std.
You can use a "higher level" colony ship though. Say you have a colony ship that has toxic. It can build a colony on all habitable planets other than radiated. Note that some planets cannot have a colony and will say none.
So invade with transports, if pop still on that planet. If all are dead, then use a colony ship with at least the minimal type. Looking at the screen you can see the rank of colony types. Radiated it the highest.