Depends if you have to be at the controls or not. There's an obvious reaction difference between
a) having your hands on the wheel and paying attention then realising you have to intervene
b) looking up from your smartphone in time to realise you have to take control and then grabbing the wheel
The former, fine, but in that case why bother with the driverless car? if we're all sat in front of the wheel attending to the traffic on the off chance that something bad might happen, the benefit of the driverless car seems pretty minimal to me.
In the latter case there are plenty of possible accidents where there's no way the passenger could intervene in time.
Well, at one end of the scale, yes. There's a whole continuum of 'driverless' from basically a self driving box you sit in to a pretty normal car with extensive driver aids to massively decrease the accident risk. The latter isn't what most people would consider 'driverless' though.