Intermittent website outages
Incident Report for Pronto Marketing
Postmortem

Overview

On September 14th, 2021, clients may have experienced intermittent website outages for sites on our legacy Amazon Web Services platforms, starting at 10:30am. The outage lasted for approximately 67 minutes and was resolved by 11:37am.

The root cause of this outage was a distributed denial-of-service (DDoS) attack on one of the websites which we host, which, due to the shared platform, also impacted the rest of the websites on the platforms. This caused a fluctuation in service from our AWS Varnish servers, so websites were intermittently available as the Varnish servers automatically recovered from the attack, and then became unavailable again.

Follow Up

As part of a long-term plan to improve the reliability of websites on our AWS platforms and to prevent issues like this occurring again, we plan to migrate all websites to our new Google Cloud Platform environment in the near future. We have also identified a number of improvements related to the actions we can take to limit the impact of similar issues in the future, and have begun a process of improving our internal processes related to these issues, allowing us to mitigate attacks like this more effectively in the future.

We sincerely apologise for the downtime caused by this - we understand how important it is that your website is available 100% of the time, and we will continue to aim for this. You can always monitor our platform status by visiting our Status page, and if you do notice any problems with your website, you can use our live chat service at https://www.prontomarketing.com/chat/ to alert our Support team immediately and we will investigate the problem.

Posted Oct 08, 2021 - 22:30 EDT

Resolved
The issue has been resolved and we'll continue to monitor for further issues.
Posted Sep 14, 2021 - 12:46 EDT
Monitoring
The issue has been resolved and we'll continue to monitor for further issues.
Posted Sep 14, 2021 - 12:08 EDT
Update
Our team is still developing a fix and we'll post an update as soon as possible.
Posted Sep 14, 2021 - 11:54 EDT
Identified
The issue has been identified, and our team is developing a fix.
Posted Sep 14, 2021 - 11:08 EDT
Update
We are continuing to investigate this issue.
Posted Sep 14, 2021 - 10:56 EDT
Investigating
We are aware of an issue with some of the websites on our legacy platform being intermittently unavailable.

We are currently investigating this issue and will post a further update as more information becomes available.
Posted Sep 14, 2021 - 10:50 EDT
This incident affected: Pronto Website Platform (AWS - Multi-Site Instance (Legacy)).