Incidents reported on status page for BobTheBot https://status.bobthebot.net/ https://d1lppblt9t2x15.cloudfront.net/logos/5355e02f7c3ae4a0d507e7b6b71779d7.png https://status.bobthebot.net/ en https://status.bobthebot.net/incident/900452 Wed, 20 May 2026 05:00:00 -0000 https://status.bobthebot.net/incident/900452#fbaf922d106da1f411442373b715f67790a6abe25a49cc6f5943ec5c1c4eeec4 Resolved May 19, 2026 at 8:41pm CDT Date: May 19, 2026 | ⏱ Duration: 3 hours 25 minutes 🧭 Summary Bob experienced partial failure. While the interface could initiate command responses, critical internal communication between service components and the database failed due to Railway infrastructure issues, resulting in global 404 "Application not found" errors. 🛠️ Resolution Bob achieved full restoration at 8:41pm once Railway restored internal connectivity for our service components. Note that our service was fully functional well before Railway's platform-wide recovery was complete. 🕑 Timeline 5:16 PM – 🚨 404 "Application not found" errors detected. 5:25 PM – 🛑 Incident triggered. 8:38 PM – ✅ Service monitors recovered across all regions. 8:41 PM – 🏁 Incident auto-resolved; Bob fully functional. ~11:58 PM – 🚇 Railway officially declared the upstream incident resolved. Root Cause: Railway's GCP account was erroneously suspended by an automated system. This external dependency failure was outside of Railway's direct control, which significantly extended the time required for resolution. Here is what Railway knows so far: https://station.railway.com/community/what-we-know-so-far-may-19th-2026-86354cdd They will be posting a full post-mortem once they gain a complete understanding of everything that went down. As always, thanks for your understanding! https://status.bobthebot.net/ Wed, 20 May 2026 01:38:16 +0000 https://status.bobthebot.net/#374d4f0e05471b89dc4e30e9d9897e471f485b5dc135086dd7c5a1e8d1224b56 Bob (Bot Service) recovered https://status.bobthebot.net/ Tue, 19 May 2026 22:25:56 +0000 https://status.bobthebot.net/#374d4f0e05471b89dc4e30e9d9897e471f485b5dc135086dd7c5a1e8d1224b56 Bob (Bot Service) went down https://status.bobthebot.net/incident/827751 Wed, 18 Feb 2026 04:06:00 -0000 https://status.bobthebot.net/incident/827751#34899b5e7f41bfccb7a41807e2d51474a40aec01c872ed53e599cbfd89ad6751 **Date:** Feb 17, 2026 | **⏱ Duration:** 27 mins ### **🧭 Summary** Bob the Bot experienced a brief total service outage followed by a prolonged networking desync. While the bot’s internal functions (Discord Shards/Slash Commands) were manually restored and confirmed operational shortly after the start of the incident, Railway’s edge infrastructure continued to report 502 "Bad Gateway" errors for several minutes due to a persistent routing failure. ### **🛠️ Resolution** Service was restored internally through a recovery of the application process. Although logs confirmed the bot was "Ready" and processing commands at **9:43 PM**, the incident remained open as Railway's proxy failed to route traffic to the container's health-check port. The edge network eventually synchronized, and monitors recovered automatically. ### **🕑 Timeline** - **9:38 PM** – 🚨 **Incident started.** Total service failure detected; global 502 errors across all regions. - **9:43 PM** – 🤖 **Application Restored.** Internal logs confirm Shards 0-2 connected and Slash Commands registered. **Bot functionality fully recovered.** - **9:46 PM** – 🛠️ Incident acknowledged by Zach Goodson. Investigation confirmed a "ghost" networking issue as the bot was functional but the URL remained unreachable. - **10:04 PM** – 📶 **Infrastructure Recovery.** Railway's edge proxy successfully re-linked the public domain to the container. - **10:05 PM** – ✅ **Incident resolved.** **Root Cause:** **Application Interruption & Upstream Desync.** An initial disruption took the bot offline; however, a secondary infrastructure failure occurred where Railway's load balancer failed to recognize the healthy application (`upstreamAddress: ""`) for 21 minutes after the bot had successfully resumed operations and port binding. https://status.bobthebot.net/incident/823566 Wed, 11 Feb 2026 21:16:00 -0000 https://status.bobthebot.net/incident/823566#2608b7ea1c70876f3a8986f51cf50c997aeafa14e9bc865c2cb2db3ab13f48d0 **Date:** Feb 11, 2026 | **⏱ Duration:** ~5 mins ### **🧭 Summary** Bob went offline briefly due to **Railway (cloud provider) infrastructure issues**, resulting in global 502 "Bad Gateway" errors. ### **🛠️ Resolution** A manual redeploy successfully restored service. This resulted in a successful redeployment hours before railway had officially resolved the issue. ### **🕑 Timeline** - **8:56 AM** – 🚨 502 errors detected globally. - **8:58 AM** – 🛠️ Manual redeploy; service restored. - **9:01 AM** – ✅ Incident resolved, but closely monitored. - **12:31 AM** - 🚇 Railway Officially resolved the issue on their end. **Root Cause:** Upstream infrastructure failure (Railway) https://blog.railway.com/p/incident-report-february-11-2026. https://status.bobthebot.net/incident/823570 Fri, 21 Nov 2025 01:00:00 -0000 https://status.bobthebot.net/incident/823570#cba23495ccc71f8f57214d3d87bf2148737eccb39678c0a22c4bc3de12fa085c **Date:** Nov 20, 2025 | **⏱ Duration:** ~1 hour ### **🧭 Summary** For about an hour, Bob failed to respond to slash commands due to Discord rejecting global command updates. The bot itself was healthy 🟢, but Discord’s API returned a persistent `50035 Invalid Form Body → BASE_TYPE_REQUIRED` error during registration. ### **🧰 What We Tried** 1. 🔁 Re‑deployed containers and tokens (no change) 2. 🧹 Cleared and re‑registered global commands 3. 🧪 Verified code against the dev bot (worked fine there) 4. 🏠 Registered commands per‑guild – confirmed success ### **🧩 Root Cause** A corrupted record in Discord’s global command database for Bob’s production application ID prevented any bulk overwrite from completing correctly. This was a **Discord‑side issue**, not a code error. ⚠️ ### **🛠️ Resolution** Catching and safely ignoring the benign API exception allowed Bob to continue starting up and attach all interaction handlers normally. Once running, slash commands responded instantly again. ✨ Discord Support has been contacted to clear the bad record for a permanent fix. 📨 ### **🕑 Timeline** - **5:23 PM CST** – 🚨 First 502 monitor alert - **5:30‑6:30 PM** – 🔎 Multiple retries and redeploys; issue isolated to Discord API - **6:50 PM CST** – 🧩 Workaround deployed; bot fully operational - **6:52 PM CST** – ✅ Incident resolved **Bob is stable again across all 2,500 servers.** Root cause: **Discord API corruption**, successfully mitigated by code‑level handling while awaiting an official fix from Discord. Nonetheless, we apologize for the inconvenience.