As I am approaching an intersection where I have no stop sign and a car is sitting at the stop sign waiting to enter from the right the only prediction I can make is that if I am too close they will wait for me to pass. The reality is that they might not (trust me, tested this one) but there is no predictive system that can tell me, among the hundreds or thousands of similar situations, in which one they will for whatever reason pull out in front of me.
It's not a perfect predictive system, but neither is what happens in the algorithms for chess by human beings. You don't know whether that car is going go pull in front of you, or if it does when. It's still constrained in possibility space. The car won't start flying, roll sideways on a flat surface absent something striking it in that situation, or morph into an elephant and roll towards you at a 45 degree angle. It might or might not pull in front of you, but apparently this is something you anticipate as a possibility after all, while flight and elephant transformations are not reasonable to bother modeling.
Since there is no predictive system the developer of the AI has two choices. Use the demonstrably flawed system that assumes the other car will always "do the right thing" and just proceed at speed, which is a choice with very harsh consequences (again, trust me, tested this one). Or it can approach every such situation frantically calculating corrections based on the acknowledged possibility that the car will pull out. This sidetracks tremendous computational power into a dead end.
If all vehicles are self-driving cars you can get away with something close to 1. If not, you can constrain its computations and force it to somewhat reduce speed, reacting to motion by the stopped vehicle (if the driver in that vehicle really wants to pull out at an awful time, few people behind the wheel could react even knowing it's possible).
When driving, you don't have that many options. You can (maybe) shift gears, accelerate to varying degrees, use the brakes to varying degrees, turn the wheel to varying degrees, and use support systems like signals/headlights (AI should basically always signal per training and use headlights appropriately though). What does a human do in this situation? Varies by human, with a significant subset assuming the other car will do the right thing...or not paying attention and acting that way regardless of considering it. AI can hedge this situation by reducing speed in uncertainty, and has the benefit of reacting much faster than humans to brake or turn the steering wheel (an amount that won't flip the vehicle or lose control, contingent on other vehicles present, something it can compute in a timeframe humans can't).
There was a time when chess AI could be beaten fairly handily by the simple expedient of opening with a rook's pawn. The move was so obviously bad that the AI had no capacity for dealing with it, because it reasonably predicted that it would never happen and did no analysis along the (dead end) lines presented.
Yes, you do want to test your AI past human capability before rolling this out. Even by the early 90's chessmaster programs commercially available had no trouble with rook's pawn opening and would just play d or e pawns and trade material if the rook was still advanced.
One of the major reasons I anticipate self-driving cars will be superior to human drivers is that they will be far better at doing this and will never speed up because they like going fast.
They will also not "feel surprise" at such events, nor will they on average have a 250ms delay just to acknowledge something is happening (average time it takes a human to click on a dot that appears on a screen)...with likely double that time again before inputs of any kind are reliably applied to the vehicle...and the AI can consistently be precise within the vehicles capabilities.
As you say the anticipation of things that require sudden reaction would be a bigger challenge. At the same time, you're trying to illustrate the unpredictable nature of something by *actually predicting* events with known occurrences in the past. These are not good examples, since they're actually anticipated and have enormously different probabilities given visual inputs! You're not going to get deer or children suddenly darting onto the road while driving on a desert road in Nevada with no obstacles in sight. Even as a heuristic, speed can be reduced as the proximity of obstructions to the road has smaller distances.
There are also ways to predict that someone in front of you will stop, but an AI doesn't need this. Unlike a large portion of human drivers, AI programmed to use x following distance at y speed will actually use x following distance, and again it should trivially outperform humans against sudden stoppages. I'd me more concerned in odd scenarios like water running over a road or "what to do if a tornado is coming up on you", because these don't have obvious, consistent safe driving algorithms to avoid issues with them.