If you actually turn around to the backseat, sure - that's worse since your eyes actually left the road. Duration of distraction is important here.
Talking with somebody sitting next to you is different in almost all the ways that are important. Driving is a physical activity, like playing music or a sport. It follows a tempo and people in the car will be involved in that tempo as well. Conversations will pause and resume as inputs from the road vary, in the case of sighted passengers they also identify road risks if at a lower rate than the person driving. A conversation on the phone does something important. It takes the mental state of the driver and gives it two separate tempos to try and follow as the speaker on the other end is not an active participant in the drive. This matters. This is where the distraction and increased risk come from, not from the piece of plastic in hand.