Internal backbone connectivity issues

Uptime Impact: 10 hours
Resolved
Resolved

During this period, we experienced an internal connectivity issue that disrupted data flow between several of our sites. The incident was caused by a software bug in a third-party component, which triggered an invalid recovery script configuration intended to handle this specific scenario. As a result, the automatic recovery mechanism did not behave as expected and contributed to the impact.

We have identified and fully understood the root cause of the issue. Corrective changes have been applied to our infrastructure, including adjustments to the affected configuration and additional safeguards, to ensure this scenario cannot recur. We are also improving monitoring around internal connectivity to allow for faster detection and response in the future.

Avatar for
Began at:

Affected components
  • Market Data API
    • REST
      • Metrics
      • Exchange rates
      • Indexes
      • Trades
      • Quotes
      • Order book
    • WebSocket
      • Trades
      • Quotes
      • Order books
      • Timeseries data (OHLCV)
      • Exchange rates
    • FIX
      • Market Data