It's also the nature of the accidents. There are human-causes of accidents that we understand because we are human. We can punish irresponsible drivers more harshly than those who just had a freak accident.
Automated systems on the other hand will fail in completely unexpected non-human ways. We will look at the circumstances of a collision and say something like "it was completely clear [to a human] that the pedestrian was crossing the road, how could the car not see them?" and this will fuel a contempt of the automated car for being incompetent in ways which should disqualify it from driving, as an incompetent human would be, even if the car has a fraction of the accident rate.
We can drive defensively by predicting the mistakes or bad behaviours of other (human) drivers. But when there are drivers on the road that are completely unpredictable and make mistakes in unexpected ways, it makes all of us less safe, and less able to drive safely.