Incidents reported on status page for Respan AI https://status.respan.ai/ https://d1lppblt9t2x15.cloudfront.net/logos/70b5cfc3c83d2fbc177e44adc14cd668.png https://status.respan.ai/ en https://status.respan.ai/incident/927132 Wed, 17 Jun 2026 22:32:00 -0000 https://status.respan.ai/incident/927132#32350d8a8ebaaf696300e87b4638f084eecc49d5ce89259cc1ba969dca3a06f4 # **Respan Security Incident Post-Mortem 6/16** *June 18, 2026* We are sharing this update on a security incident we began investigating on June 15. We want to thank you for your patience throughout this process. **Timeline** - **June 15, 11:00 PM PT:** Suspicious provider API key activity flagged; investigation began. - **June 16, 12:31 AM PT:** Internal platform credentials rotated across all environments. - **June 16, 2:00 PM PT:** Impacted users notified and asked to rotate provider keys. - **June 16, 3:09 PM PT:** Root cause confirmed: sandbox escape vulnerability in our code evaluator feature. - **June 16, 3:51 PM PT:** Fix deployed across all production clusters (about 40 minutes from confirmation to full rollout). **What Happened** A vulnerability in our code-evaluation sandbox allowed an attacker to escape the sandboxed environment and access environment variables containing sensitive decryption-related configuration, which can be used to decrypt and access user-uploaded provider API keys. The exploit left no error or log trace, which delayed detection. We ruled out all other possible vectors including SQL injection, S3 export theft, and credential table access. The accounts responsible were identified and have been permanently blocked from the platform. **Who Was Impacted** Approximately 4.5% of Respan platform users had provider API keys compromised, and a portion of those users experienced unexpected usage. No other user data, including logs, outputs, or secrets, was accessed. Enterprise environments were not affected. We have contained the identified issue and are continuing to monitor closely as we complete the investigation. **What We Did** - Replaced the vulnerable sandbox with a WASI-based runtime (wasmtime) with no access to host environment variables or credentials - Rotated all internal platform credentials - Blocked accounts identified during the investigation - Enabled database audit logging (pgAudit) for improved forensic visibility **What You Should Do** If you stored provider API keys in Respan and have not yet rotated them, please do so now. Reach out directly if you notice any suspicious activity. **Moving Forward** This incident highlighted the need to minimize where sensitive credentials exist in our platform. We are accelerating work to support federated identity so users no longer need to store raw API keys in Respan. Expanded audit logging is also being rolled out to close the detection gaps this incident exposed. **Reimbursement** For the unexpected charges, OpenAI will process refunds directly back to the affected users and their original impacted orgs. If you need help submitting or following up on a refund request, please let us know and we can help support the process. We are also actively working with other providers to resolve any related usage or billing issues as quickly as possible. https://status.respan.ai/incident/927132 Wed, 17 Jun 2026 21:45:00 -0000 https://status.respan.ai/incident/927132#5f97766b5340f810e8ff4dcd3e323935d9004f55c85bdbb09baee574e16ed717 We’re investigating an issue with the gateway services that is impacting the platform. We’re working to fix the problem as quickly as we can. We’ll share another update shortly. https://status.respan.ai/incident/815692 Sat, 31 Jan 2026 11:15:00 -0000 https://status.respan.ai/incident/815692#6d93d1681d4f565db1178caf0ccf8e718e076a71ca75275d50f5ef55909d6f93 Log ingestion and platform analytics dashboard have resumed. https://status.respan.ai/incident/815692 Sat, 31 Jan 2026 10:05:00 -0000 https://status.respan.ai/incident/815692#945544e1514fd460fc5fb97946c93e1a97e57832b7832ac2440af8accb9144cc Due to a ClickHouse Cloud outage, log ingestion is delayed and analytics dashboard is temporarily unavailable. LLM proxy requests are not impacted and continue to operate normally. Reference ClickHouse status: https://status.clickhouse.com/ https://status.respan.ai/incident/748348 Tue, 21 Oct 2025 17:30:00 -0000 https://status.respan.ai/incident/748348#3d4e20eeda296209049ada9d182330abb937ce6d55c1791b5db935a442de7819 Redis instance overload due to increased log volumes consuming excessive memory and impacting infrastructure performance. Migration work in progress to offload logs from Redis and stabilize the system. https://status.respan.ai/incident/748342 Tue, 21 Oct 2025 16:30:00 -0000 https://status.respan.ai/incident/748342#c299a8e4776202849fea02b89d1e44afbed44def2fcb4933c70ec856fe3efcde Redis instance overload due to increased log volumes consuming excessive memory and impacting infrastructure performance. Migration work in progress to offload logs from Redis and stabilize the system. https://status.respan.ai/incident/748348 Tue, 21 Oct 2025 14:30:00 -0000 https://status.respan.ai/incident/748348#157e831f95d5ef0a8c1147762287ddcc4b2b6637c5d4ba73eae076fe89f8aaba The proxy service has been restored and is now fully operational. All traffic routing and functionality have returned to normal. We are continuing to investigate the root cause of the outage to prevent future occurrences. https://status.respan.ai/incident/748342 Tue, 21 Oct 2025 14:00:00 -0000 https://status.respan.ai/incident/748342#42d9f036513395525ac95c2d505772adfa993e650564eef41e68829f31bc2d20 Our proxy service is currently experiencing a complete outage and is unavailable. The team is actively investigating the root cause and working to restore service as quickly as possible. https://status.respan.ai/incident/705443 Wed, 13 Aug 2025 08:34:00 -0000 https://status.respan.ai/incident/705443#e1eaf4b920e70af720206e8de5c4008ae55b90cb9c1f7e1ea0a357e220e52a2d Resolved: We have deployed a fix for the issue and is investigating the root cause of it. As of Aug 13, 08:33:00 UTC, proxy API calls are functioning normally. The root cause is a migration issue. We will add additional deployment verification steps to confirm valid dependencies and enhance validation pipelines to include more import verification tests. https://status.respan.ai/incident/705443 Wed, 13 Aug 2025 08:03:00 -0000 https://status.respan.ai/incident/705443#d5360127532afcafdfd6aa7d64695d1f2f262137ac59f91f877f20b0ab2c0eb9 Investigating: We are seeing an abnormal rate of server errors for proxy API calls from Aug 13, 07:59:26 UTC. Enterprise platform is not affected. https://status.respan.ai/incident/705635 Wed, 13 Aug 2025 07:58:00 -0000 https://status.respan.ai/incident/705635#b371fd25203a3d429808727376ceef378c17dfa221590c7b504a1b7491f99c65 A fix has been deployed to fix this. https://status.respan.ai/incident/705635 Wed, 13 Aug 2025 07:53:00 -0000 https://status.respan.ai/incident/705635#dfde73b554db8bf517b22ef36019076f0ed7a30f3c7290bdf7ad44cb9bf8dd62 Platform APIs were down due to high traffic.