We are experiencing issues with sending emails from workspaces

Closed | Nov 30, 2023 | 16:06 GMT+02:00

During 29.11.2023 12.35 - 30.11.2023 8.45 Howspace had an incident that prevented all email deliveries.


November 27th Spammy account noticed

On Monday 27th our employee noticed from our monitoring, that there was an exceptional spike in email delivery amounts on one of the Free accounts. We immediately investigated the issue and noticed that this account was used to deliver emails that were obviously spam. User limitation was circumvented by always deleting the users and adding new ones. Emails were sent during the weekend, from Friday to Sunday. When we investigated this on Monday, the last emails were sent 6 hours ago.

We immediately closed the account and went to look through if there were similar accounts. We found 3 other accounts that were used for similar activity and closed those too.

In case to have a better view of a situation like this, we implemented some fine-tuned monitoring and alerts for Free email sending and decided to manually pay close attention to every new Free account that is created.


November 29th Email delivery stopped

At 12.35 (GMT) our support received a phone call from our client, that they cannot receive any emails from Howspace. Something that is usually a common support case (email address can be mistyped, the spam filter is working too eagerly, etc) turned much more severe when our technical support noticed that AWS (our hosting provider) had paused our email deliveries. That was immediately escalated to our developers, who then noticed that AWS had created a support issue on Friday 24th, and warned us about the increasing amount of email complaints we are receiving that could lead to pausing our email deliveries. The problem with this issue was that we never received an actual email from that issue, so no one in our company knew about this.

This issue was already updated with the information, that our email sending is paused until we take actions that prevent email complaints from increasing.

We then verified that most of the complaints were caused by the spammy account we identified and closed on Monday. After that, at 14.21, we gave detailed answers about what had happened and the actions we had already taken on Monday. Since email delivery is a critical aspect of our platform (the most common login method is a login link delivered by email), we contacted our Account Manager, who promised to increase the importance of our issue.

We didn't receive any information back from AWS, so we decided to open another support ticket to them at 19.21 (they provide 4-hour response times for issues with "Production system impaired" severity). We highlighted the importance of emails for our platform and asked for a quick resolution. We got a response for this issue on 00.20 where they closed the duplicate issue raised the severity of the original issue - and promised that the engineer will be reaching out to us soon.


November 30th Issue resolved

On 4.26 Thursday morning we still haven't received any update on the original issue, where AWS notified us they had paused our email service and we had already provided the list of actions we had taken to prevent the issue. We asked for an update and 2 minutes later received an (probably automated) update to the issue, that asked for the same evidence again. At 4.38 we provided them again.

Then at 6.10, we received the next update which once again asked for more information about what actions we had taken to prevent this, but this time also said that "your appeal did not provide the information we need to make a decision". So we answered for 3rd time at 6.42 and this time added also new action where we closed our Free form, which can be used to open new Free accounts on our platform.

We started to feel a bit desperate about the poor communications and the lack of urgency we were receiving from AWS, so we started thinking about alternatives - although we knew there wasn't any fast shortcut available. Then at 8.45, we received an update on the issue, that the review period and sending pause was over and email delivery was working again.


Aftermath

Our platform is sending approx. million emails a month and the incident that led AWS to halt our full email deliveries was caused by one spam account, that sent ~2000 spam emails through our platform. We never received any email, phone call, or other notification about that this is leading to halting our whole email delivery. While we are very unhappy about how our hosting partner handled this topic, we can only focus and take responsibility for our actions. Therefore we have decided to improve our email deliveries in the future by:

- Adding hard limitations to the amount of emails Free accounts can send
- Enhance our monitoring and alerting so we get ahead of the issues before they escalate
- Implement an alternative email delivery system, so that in case of failure, we can quickly change the way our platform sends emails. This should also help us to improve the email delivery percentage.

We are deeply sorry for any inconvenience this has caused for your Howspace usage and we do our best to make sure this won't happen again.

Affected components

• Howspace (in.howspace.com)