Summary: 

From approximately 11:10 am - 11:35 am GMT, Wednesday April 10th, EU hosted communities on the Dynamic Signal Platform experienced a disruption in service caused by an exhaustion of local storage space.  

Impact:

Users accessing the Member app would have encountered a Technical Difficulties page during this time.

Mitigation: 

The issue was determined to be an exhaustion of local storage space on an affected Web Server and was quickly corrected to restore service. Also during this period of time the issue would have been mitigated upon a refresh of the Member app page as the communities would have been redirected to an available Web Server without the storage issue.   

Root Cause Analysis: 

The issue was determined to be rapid storage growth of local storage space on an affected Web Server. This caused those users who initially connected to the affected Web Server which hosts the Member app to be temporarily unavailable.

Recurrence Prevention: 

To prevent a future recurrence, the below actions have been implemented:

1. Log automation removal has been optimized to make sure our production operating limits are tolerant of surges and organic growth over time. 
2. In the event of an unexpected storage increase, alerting has been improved to lower the threshold so it will be triggered as ‘Disaster’ when it gets to a certain percentage and intervention will be taken sooner before a disruption of service occurs.