Outage Postmortem

This post contains the exact, unedited email that was sent to all users with an account on Plunk after the biggest outage the platform has ever faced. The outage started on the 21st of March and lasted 5 entire days.

Hi,
You are receiving this email since you have an account on useplunk.com.

IMPORTANT: Action Required

Your domain has automatically been unlinked from your account. If you wish to continue using Plunk, kindly re-verify it in your dashboard!

What happened

On the 21st of March, around midday, AWS locked our account. This resulted in a complete halt of what Plunk does best, sending emails.

Why did this happen

Around a week before, Plunk was targeted by a pretty severe cyber attack where automated systems created large numbers of accounts to deliver phishing emails.

This issue was clearly visible in our analytics and AWS also flagged it to us. We resolved the issue and our account was still well within the bounce and complaint boundaries we have with AWS.

However, attacks like these pack an initial punch, but a second wave of bounces and complaints typically arrives a week later when people start opening the emails or their mailserver refuses it.

Once this happened, AWS locked our account instead of linking it to the recently closed ticket.

How did we resolve it

We provided AWS with all of the requested information within 30 minutes of the ticket being opened. Since we experienced a similar issue a year ago, we also knew that their response times on these types of issues are very slow and had thus prepped a backup account.

We switched over to this backup account, meaning that your emails will have been delivered from fallback@useplunk.dev, and later fallback@useplunk.net. Click and open tracking has also been disabled for all projects during this time.

We have also made a change to our pricing. In the past, anyone could open a Plunk account for free and upgrade without any cost. This made it trivially easy for fraudulent actors to upgrade and bypass our free tier limit. Starting the 21st of March, any account that tries to upgrade will have $1 due at the time of upgrading and we will return this $1 as a credit of 1K emails.

What we could have done better

Communication during this outage has been subpar. I am aware of this and strive to do better in the future.

Updates were posted on status.useplunk.com but many people did not find their way to this portal and had to resort to asking questions on Twitter or Discord.

What will happen

The future of Plunk during times like these is very uncertain. I remain a solo-founder and operating a business-critical tool like Plunk has taken its toll on my health, relationships and overall wellbeing.

The following possibilities are on the table and I would appreciate your input.

• We change nothing. The change to the pricing should prevent a lot of headaches in the future + this was the only serious outage in the last 365 days.
• We switch to a license with bring-your-own AWS.
• We phase out the hosted version and focus entirely on open-source and AWS consulting.

Regardless of what is next for Plunk.
I appreciate your business and what happened is a clear violation of the trust you put into me.

Dries