Thanks for your patience whilst we rectified the issue on Tuesday, to give you a little more information about what happened and how we will prevent it from happening in the future, here is the breakdown. 

What happened? 

Between the hours of 12:11 and 16:42 on Tuesday, 18th January 2022 , about half of all Signable Web App users encountered issues when uploading documents to create envelopes and templates. Public API integrators saw issues with webhooks and envelopes getting stuck in processing and there were issues with email notifications. 

The event was triggered by a hardware network communication issue on one of the AWS EC2 instances in circulation. 

It was detected by our monitoring systems which indicated a high level of jobs taking a long time to process. The intermittent nature of the incident made it hard to diagnose. After an extensive detailed investigation, it was clear that the issue was not related to the Signable app but to a faulty AWS instance. The instance was taken out of circulation which stopped the errors. It was resolved by restarting all instances and forcing the app to run on new hardware within the AWS data centres. 

This priority 1 incident affected ~50% of Web app users and  ~30% of public API jobs.

What are we doing about this? 

We are focusing on transforming the following areas to help us prevent, detect and respond to these types of incidents more efficiently:

• Improve internal monitoring and alerting in the areas of the platform affected by this problem.
• An improved communication process to our app and API users when dealing with incidents.
• Changes to our diagnosing process to try to minimise the impact in future.