This incident has now been mitigated for most Communities, by rerouting traffic away from the affected third party provider. This was able to be done in software for some of the communities and physical interface changes for others at the data center. Please reach out to Support at https://support.dynamicsignal.com if you or any of your users are continuing to experience any connection issues. Marking incident as resolved and all effected components to fully operational.
Posted Nov 20, 2021 - 13:11 PST
Update
We have not received any further information at this time regarding service restoration by the thir party provider and have dispatched our own engineers on site to see if there is any way to bypass the upstream provider.
Posted Nov 20, 2021 - 12:28 PST
Update
We have confirmation from the colocation provider at the primary US datacenter that the third party network operator has provided an ETA of 11:30 AM PT to correct the routing problem.
Posted Nov 20, 2021 - 10:33 PST
Update
Scope for this issue seems to include all voicestorm hosted domains, and any custom URLs that are CNAMEd to their voicestorm host equivalents, rather than pointing directly to BGP routable IP. A single ISP seems to be having routing issues that is upstream from the Dynamic Signal platform itself. Increasing severity based on revised scope.
Posted Nov 20, 2021 - 07:47 PST
Monitoring
Reducing incident severity. There appears to be a widerspread Internet issue that is also affecting the platform monitoring tool. There may be some end users who are sporadically having difficulty accessing the service, so keeping at degraded performance until we have a fuller understanding of scope and root cause.
Posted Nov 20, 2021 - 07:04 PST
Investigating
We are in the process of investigating a service disruption that appears to be network related, and is preventing the ability of users to be able to access their communities through the member and manager apps.
Posted Nov 20, 2021 - 06:50 PST
This incident affected: Products (Manager App, Member App, iOS App, Android App, API) and Platforms (US Firstup Dynamic Platform).