On May 22, 2024, starting at 14:44 UTC, our system experienced a significant issue where envelopes were not visible to users after being sent. This incident was also caused by a spike in ElasticSearch indexing latency, which led to job queues filling up faster than they could be processed. Thanks to the improvements made the previous day, the impact was not as severe. 

Additional manual scaling of the infrastructure returned latency to normal levels. By 19:15 UTC all job queues were fully processed and normal service resumed. 

During our investigation, we identified the cause of the increased latency to be the ElasticSearch service being throttled after we hit our throughput limit. 

Customer Impact

The incident resulted in the following:

• Delayed syncing to ElasticSearch meant there was a delay before updates to envelopes were showing on the “envelopes” view in the web app.
• Slow processing of bulk sends.
• Delayed envelope expiries and reminders.

This affected 75% of new envelope requests, leading to significant user inconvenience and delays in processing.

Lessons Learned and Improvements

Preventive Measures:

1. Infrastructure Changes: Upgrade the ElasticSearch data node types to improve performance.
2. Dedicated resources for ElasticSearch: Give ElasticSearch more dedicated resources so it does not have to contend with other services. This will ensure that in the event of a latency issue, the impact is isolated.
3. Enhanced Scaling Mechanisms: Implement enhanced auto-scaling procedures for all critical services.
4. Alerting and Monitoring: Set up additional alerts for when message queues exceed predefined thresholds.
5. Improvement in Job Management: Prevent job duplicates to reduce queue size during slowdowns.

We apologise for the inconvenience caused; we are proactively addressing the design of our infrastructure as we grow and appreciate your understanding as we continuously improve.

For any further questions or concerns, please contact our support team at help@signable.co.uk