Back to overview
Downtime

Gateway is temporarily down

Jun 17, 2026 at 9:45pm UTC
Affected services
Gateway services
Gateway services

Resolved
Jun 17, 2026 at 10:32pm UTC

Respan Security Incident Post-Mortem 6/16

June 18, 2026

We are sharing this update on a security incident we began investigating on June 15. We want to thank you for your patience throughout this process.

Timeline

  • June 15, 11:00 PM PT: Suspicious provider API key activity flagged; investigation began.
  • June 16, 12:31 AM PT: Internal platform credentials rotated across all environments.
  • June 16, 2:00 PM PT: Impacted users notified and asked to rotate provider keys.
  • June 16, 3:09 PM PT: Root cause confirmed: sandbox escape vulnerability in our code evaluator feature.
  • June 16, 3:51 PM PT: Fix deployed across all production clusters (about 40 minutes from confirmation to full rollout).

What Happened

A vulnerability in our code-evaluation sandbox allowed an attacker to escape the sandboxed environment and access environment variables containing sensitive decryption-related configuration, which can be used to decrypt and access user-uploaded provider API keys. The exploit left no error or log trace, which delayed detection. We ruled out all other possible vectors including SQL injection, S3 export theft, and credential table access. The accounts responsible were identified and have been permanently blocked from the platform.

Who Was Impacted

Approximately 4.5% of Respan platform users had provider API keys compromised, and a portion of those users experienced unexpected usage. No other user data, including logs, outputs, or secrets, was accessed. Enterprise environments were not affected. We have contained the identified issue and are continuing to monitor closely as we complete the investigation.

What We Did

  • Replaced the vulnerable sandbox with a WASI-based runtime (wasmtime) with no access to host environment variables or credentials
  • Rotated all internal platform credentials
  • Blocked accounts identified during the investigation
  • Enabled database audit logging (pgAudit) for improved forensic visibility

What You Should Do

If you stored provider API keys in Respan and have not yet rotated them, please do so now. Reach out directly if you notice any suspicious activity.

Moving Forward

This incident highlighted the need to minimize where sensitive credentials exist in our platform. We are accelerating work to support federated identity so users no longer need to store raw API keys in Respan. Expanded audit logging is also being rolled out to close the detection gaps this incident exposed.

Reimbursement

For the unexpected charges, OpenAI will process refunds directly back to the affected users and their original impacted orgs. If you need help submitting or following up on a refund request, please let us know and we can help support the process. We are also actively working with other providers to resolve any related usage or billing issues as quickly as possible.

Created
Jun 17, 2026 at 9:45pm UTC

We’re investigating an issue with the gateway services that is impacting the platform. We’re working to fix the problem as quickly as we can. We’ll share another update shortly.