Let me answer this specifically as a SFP (Seller Fulfilled Prime) business.
Yes it will hit your metrics. If you shipped 50 items and two were late during the month, you will lose the right to do SFP. 97% on time delivery is required. Yes, even though this is out of your hands, if the carrier is late, weather, acts of God, anything, you are held accountable.
Yes, on any late order. Even one day late, they get to keep the item and they get a refund. All it takes is a request to Amazon.
No you are not protected. At all. Only adding Sig-Con can protect you. Buying shipping through Amazon, and that is required, gives you no protection from A-Z or bad feedback.