Service Outage Postmortem - January 9th, 2025 Issue Summary On January 9th, 2025, from 9:58 AM to 10:18 AM PST (17:58 - 18:18 UTC), the Medplum service experienced an outage that caused HTTP 500 errors for most API requests. The incident affected all customers using our API services. The root cause was identified as an error in our logging middleware implementation, which was introduced in a recent deployment. Service was fully restored by reverting the problematic change. Timeline (All times PST) - 9:58 AM - Issue began affecting customers - 10:01 AM - First customer report received - 10:02 AM - Engineering team began investigation - 10:15 AM - Fix deployment initiated - 10:18 AM - Service fully restored - 10:24 AM - All customers notified of resolution Root Cause The outage was caused by an implementation error in our logging middleware that resulted in resource handling conflicts. When the affected code was deployed to production, it caused our API servers to return 500 errors on most requests while appearing healthy to our monitoring systems. Resolution and Recovery Our engineering team identified the root cause and implemented a fix by reverting the problematic changes. The service recovered immediately upon deployment of the fix, and normal operation was restored to all systems. Corrective and Preventative Measures To prevent similar incidents in the future, we are: 1. Implementing enhanced testing procedures for logging middleware components 2. Improving our monitoring systems to better detect similar issues: - Adding more sophisticated health checks that verify API functionality - Implementing better error rate monitoring with refined alerting thresholds 3. Enhancing our deployment process to include additional validation steps 4. Updating our testing infrastructure to better mirror production configurations