On lower difficulties, yes I don't have to choose settlers, I don't have to choose workers, there is probably no impact.
On higher difficulties, it's just going to make things harder if I don't make use of any advantage I can find.
Is the reply then going to be "I don't have to play at higher...