Incidents and maintenance reported on status page for Respan AI https://status.respan.ai/ https://d1lppblt9t2x15.cloudfront.net/logos/70b5cfc3c83d2fbc177e44adc14cd668.png https://status.respan.ai/ en https://status.respan.ai/incident/1077403 Tue, 29 Sep 2026 17:38:00 -0000 https://status.respan.ai/incident/1077403#16984fa623b7cc41cce6521ef24be451388ed8c01dd5eb56a1b35025f5d72265 Incident This incident is resolved. All services are operating normally. https://status.respan.ai/incident/1077403 Tue, 29 Sep 2026 17:20:00 -0000 https://status.respan.ai/incident/1077403#26effa4ef40c349548515a027ab594e2b68a8417028f36fd33a8272e589d1933 Incident Enterprise API and ingestion traffic fully recovered, and all API traffic is working normally. Delayed request logs are being processed and will appear in the dashboard as the backlog clears. https://status.respan.ai/incident/1077403 Tue, 29 Sep 2026 16:40:00 -0000 https://status.respan.ai/incident/1077403#7283701fad21b388eb1f85e2c8319530071e2a834ad143a13da9c9bbbb34df88 Incident Some requests with valid API keys are being rejected with 403 errors. This comes from a stale read in our caching. We implemented the fix for it. Request logs are still delayed. https://status.respan.ai/incident/1077403 Tue, 29 Sep 2026 16:13:00 -0000 https://status.respan.ai/incident/1077403#4baf702c85fcc240128a3000e804b49043c5a3cb449bbc913a9c08a807a9330e Incident Some API requests are failing with server errors, and request logs are still delayed in the dashboard. We are working on a fix. https://status.respan.ai/incident/1077403 Tue, 29 Sep 2026 15:40:00 -0000 https://status.respan.ai/incident/1077403#b17725a5c4f4fe2dc7a64608caac13904ef228d672f61886480e1ae1a24bc8ff Incident We are investigating delays in request logs appearing in the dashboard. Logs are being received and stored, but processing is delayed due to a capacity issue in our infrastructure caching layer. API requests are not affected. https://status.respan.ai/incident/1074876 Sat, 26 Sep 2026 21:10:00 -0000 https://status.respan.ai/incident/1074876#eb8f42dae765c76f9b681f837a534f705f288769677d62632d54959e29c87ffb Incident The Response API gateway is operating normally. OpenAI and Anthropic requests are no longer affected. https://status.respan.ai/incident/1074876 Sat, 26 Sep 2026 07:23:00 -0000 https://status.respan.ai/incident/1074876#92bb8494398d9a2fd194f26d78cd41bdac0c0342cd92b340a481f19b33ec9a10 Incident ~0.5% of OpenAI and Anthropic requests are affected. https://status.respan.ai/incident/927132 Wed, 17 Jun 2026 22:32:00 -0000 https://status.respan.ai/incident/927132#32350d8a8ebaaf696300e87b4638f084eecc49d5ce89259cc1ba969dca3a06f4 Incident # **Respan Security Incident Post-Mortem 6/16** *June 18, 2026* We are sharing this update on a security incident we began investigating on June 15. We want to thank you for your patience throughout this process. **Timeline** - **June 15, 11:00 PM PT:** Suspicious provider API key activity flagged; investigation began. - **June 16, 12:31 AM PT:** Internal platform credentials rotated across all environments. - **June 16, 2:00 PM PT:** Impacted users notified and asked to rotate provider keys. - **June 16, 3:09 PM PT:** Root cause confirmed: sandbox escape vulnerability in our code evaluator feature. - **June 16, 3:51 PM PT:** Fix deployed across all production clusters (about 40 minutes from confirmation to full rollout). **What Happened** A vulnerability in our code-evaluation sandbox allowed an attacker to escape the sandboxed environment and access environment variables containing sensitive decryption-related configuration, which can be used to decrypt and access user-uploaded provider API keys. The exploit left no error or log trace, which delayed detection. We ruled out all other possible vectors including SQL injection, S3 export theft, and credential table access. The accounts responsible were identified and have been permanently blocked from the platform. **Who Was Impacted** Approximately 4.5% of Respan platform users had provider API keys compromised, and a portion of those users experienced unexpected usage. No other user data, including logs, outputs, or secrets, was accessed. Enterprise environments were not affected. We have contained the identified issue and are continuing to monitor closely as we complete the investigation. **What We Did** - Replaced the vulnerable sandbox with a WASI-based runtime (wasmtime) with no access to host environment variables or credentials - Rotated all internal platform credentials - Blocked accounts identified during the investigation - Enabled database audit logging (pgAudit) for improved forensic visibility **What You Should Do** If you stored provider API keys in Respan and have not yet rotated them, please do so now. Reach out directly if you notice any suspicious activity. **Moving Forward** This incident highlighted the need to minimize where sensitive credentials exist in our platform. We are accelerating work to support federated identity so users no longer need to store raw API keys in Respan. Expanded audit logging is also being rolled out to close the detection gaps this incident exposed. **Reimbursement** For the unexpected charges, OpenAI will process refunds directly back to the affected users and their original impacted orgs. If you need help submitting or following up on a refund request, please let us know and we can help support the process. We are also actively working with other providers to resolve any related usage or billing issues as quickly as possible. https://status.respan.ai/incident/927132 Wed, 17 Jun 2026 21:45:00 -0000 https://status.respan.ai/incident/927132#5f97766b5340f810e8ff4dcd3e323935d9004f55c85bdbb09baee574e16ed717 Incident We’re investigating an issue with the gateway services that is impacting the platform. We’re working to fix the problem as quickly as we can. We’ll share another update shortly. https://status.respan.ai/incident/815692 Sat, 31 Jan 2026 11:15:00 -0000 https://status.respan.ai/incident/815692#6d93d1681d4f565db1178caf0ccf8e718e076a71ca75275d50f5ef55909d6f93 Incident Log ingestion and platform analytics dashboard have resumed. https://status.respan.ai/incident/815692 Sat, 31 Jan 2026 10:05:00 -0000 https://status.respan.ai/incident/815692#945544e1514fd460fc5fb97946c93e1a97e57832b7832ac2440af8accb9144cc Incident Due to a ClickHouse Cloud outage, log ingestion is delayed and analytics dashboard is temporarily unavailable. LLM proxy requests are not impacted and continue to operate normally. Reference ClickHouse status: https://status.clickhouse.com/ https://status.respan.ai/incident/748348 Tue, 21 Oct 2025 17:30:00 -0000 https://status.respan.ai/incident/748348#3d4e20eeda296209049ada9d182330abb937ce6d55c1791b5db935a442de7819 Incident Redis instance overload due to increased log volumes consuming excessive memory and impacting infrastructure performance. Migration work in progress to offload logs from Redis and stabilize the system. https://status.respan.ai/incident/748342 Tue, 21 Oct 2025 16:30:00 -0000 https://status.respan.ai/incident/748342#c299a8e4776202849fea02b89d1e44afbed44def2fcb4933c70ec856fe3efcde Incident Redis instance overload due to increased log volumes consuming excessive memory and impacting infrastructure performance. Migration work in progress to offload logs from Redis and stabilize the system. https://status.respan.ai/incident/748348 Tue, 21 Oct 2025 14:30:00 -0000 https://status.respan.ai/incident/748348#157e831f95d5ef0a8c1147762287ddcc4b2b6637c5d4ba73eae076fe89f8aaba Incident The proxy service has been restored and is now fully operational. All traffic routing and functionality have returned to normal. We are continuing to investigate the root cause of the outage to prevent future occurrences. https://status.respan.ai/incident/748342 Tue, 21 Oct 2025 14:00:00 -0000 https://status.respan.ai/incident/748342#42d9f036513395525ac95c2d505772adfa993e650564eef41e68829f31bc2d20 Incident Our proxy service is currently experiencing a complete outage and is unavailable. The team is actively investigating the root cause and working to restore service as quickly as possible. https://status.respan.ai/incident/705443 Wed, 13 Aug 2025 08:34:00 -0000 https://status.respan.ai/incident/705443#e1eaf4b920e70af720206e8de5c4008ae55b90cb9c1f7e1ea0a357e220e52a2d Incident Resolved: We have deployed a fix for the issue and is investigating the root cause of it. As of Aug 13, 08:33:00 UTC, proxy API calls are functioning normally. The root cause is a migration issue. We will add additional deployment verification steps to confirm valid dependencies and enhance validation pipelines to include more import verification tests. https://status.respan.ai/incident/705443 Wed, 13 Aug 2025 08:03:00 -0000 https://status.respan.ai/incident/705443#d5360127532afcafdfd6aa7d64695d1f2f262137ac59f91f877f20b0ab2c0eb9 Incident Investigating: We are seeing an abnormal rate of server errors for proxy API calls from Aug 13, 07:59:26 UTC. Enterprise platform is not affected. https://status.respan.ai/incident/705635 Wed, 13 Aug 2025 07:58:00 -0000 https://status.respan.ai/incident/705635#b371fd25203a3d429808727376ceef378c17dfa221590c7b504a1b7491f99c65 Incident A fix has been deployed to fix this. https://status.respan.ai/incident/705635 Wed, 13 Aug 2025 07:53:00 -0000 https://status.respan.ai/incident/705635#dfde73b554db8bf517b22ef36019076f0ed7a30f3c7290bdf7ad44cb9bf8dd62 Incident Platform APIs were down due to high traffic.