Seeing that data suggest that they're safer compared to normal cars I'd say with a few decades of testing they should be ready.
The data lies.
So far, self-driving cars are kept out of difficult situations, either totally or by turning control over to the driver. And, so far, self-driving cars are well maintained by researchers, rather than being 10-years-old and never having maintenance.
But, self-driving cars are safer than incompetent drivers. And, I hope to see them starting to go mainstream in the next few years.