Although numerous Earthquake Early Warning (EEW) algorithms have been developed we still lack a detailed understanding of how often and under what circumstances useful ground motion alerts can be provided to end-users. Here we analyze the alerting performance of the PLUM, EPIC and FinDer algorithms by running them retrospectively on the seismic strong motion data of the 219 earthquakes in Japan since 1996 that exceeded Modified Mercalli Intensity (MMI) of 4.5 on at least 10 sites (Mw 4.5-9.1). Our analysis suggests that, irrespective of the algorithm, EEW end-users should be prepared that EEW can often but not always provide useful ground motion alerts. A majority of sites with moderate-strong ground motion (MMI 5-6) can generally get at least a few seconds of warning time from all algorithms. If such shaking is caused by a shallow crustal event, around 50% of such sites receive alerts with warning times >5 s. Many sites with severe-extreme ground motion (MMI >=8) can be alerted successfully in the case of very large offshore earthquakes, but less than 20% can be alerted ahead of time if such shaking is caused by a shallow crustal event. Our results provide detailed quantitative insight into the expected alerting performance for EEW algorithms under realistic conditions. The main caveat is that the largest shallow crustal event in our data set has Mw7.0, i.e. the data set does not contain very large strike slip events.