Summary: 

On Wednesday, June 24th, 2024, starting at around 05:29 AM PST (14:29 UTC), we received reports of incorrect display of links on posts shared to Facebook. We determined the root cause to be a newly imposed or enforced API request volume restriction by our URL shortener Bit.ly. This led to failures in link shortening and displaying posts on social media platforms.

‌

Impact: 

The scope of this incident primarily affected users who did not have a custom bit.ly configuration. The incident caused sharing links to display incorrectly on Facebook, resulting in posts that did not show the community name, post title, or images properly. 

‌

Mitigation: 

The issue was temporarily mitigated on July 1st when the API monthly quota was reset however the API limit soon exceeded again.

‌

Root Cause:

An exceeded API quota caused disruptions in our URL shortening service, leading to incorrect display of links on posts shared to Facebook. The root cause was traced to API limits recently imposed by our URL shortener Bit.ly, with whom Firstup Dynamic once had a legacy account providing unlimited links and API requests. This led to failures in link shortening and displaying posts on social media platforms. The incident was also prolonged by a company wide bit.ly holiday which delayed the discussion/creation of a new custom account to implement a quota allowance which would match our needs.

‌

Recurrence Prevention: 

Enhanced Monitoring and Alerts: The new Custom API account allows the development of more robust monitoring and alerting mechanisms for API usage quotas to prevent exceeding limits. This includes alerting for when the API usage approaches critical thresholds. 

Exploration of Alternative Solutions: Continuously explore and evaluate alternative solutions to prevent over-reliance on a single provider. This includes investigating other URL shortener services and developing abstractions so that we may keep our options open.