https://medplum.com logo
Join Discord
Powered by
Query crashes Fargate task
# support
s
Hey all, I'm running into a weird issue. I have a query that runs with no problem in our dev environment which is using the hosted version of Medplum (the older Dev plan). However, in our test env running it a few times will cause the task to fail health checks and then go offline. I'm not seeing anything glaring in the logs, and I have quite a bit of hardware spun up for it to use (DB = t4g.large x 2 with the proxy enabled, and I've bumped the tasks up to 8 cpu and 32gb of ram.) AuditEvent saves/logs are turned off. Latest version of Medplum (5.0.3) is installed. We also don't have a ton of data in this env right now (only 62 service requests). I'll put the query below, but if anyone has any thoughts, I'd really appreciate the help.
Copy code
{self-hosted-medplum}/fhir/R4/ServiceRequest?
_count=10
_offset=0
_sort=authored
identifier=UP25-41
_include=ServiceRequest:subject
_include=ServiceRequest:specimen
_include=ServiceRequest:performer
_revinclude=DiagnosticReport:based-on
_revinclude:iterate=Provenance:target
_revinclude=Task:based-on
_revinclude=QuestionnaireResponse:based-on
_total=accurate
_elements=id,identifier,masterIdentifier,code,basedOn,status,authoredOn,name,birthDate,type,receivedTime,container,parent,note,request,reference
Adding a quick note to say that I'm seeing an exit code of 137 on the task. I've tried adding the 
max-old-space
flag and setting it a higher value, but still see the same error.
r
@Spencer Smith - which version are you running?
If you look at the stats on your RDS - how does the CPU look?
s
Hey! We're on v5.0.3 and the DB CPU maxes out at about 8% at any given time.
Another quick troubleshooting update: we tried scaling way out on tasks (15) and still saw the same 137 exit code.
r
@Spencer Smith you might be failing a healthcheck. What does you server CPU / mem look like, and how long is it taking for those queries to complete?
s
So the LB healthcheck is definitely what's telling Fargate to kill it. The CPU is sub 10% and the memory is ~6% (I cranked these way up yesterday just to see if that helped). The query completes almost instantly (until it crashes). We have a grand total of 62 service requests right now in this env.
We gave node ~24gigs of RAM and still see the 137. Not sure if that's a red herring at this point though. It's bizarre because we have more data in the hosted dev and it works 100% fine.
i
@Spencer Smith do you have any logging on your RDS instance?
s
Only seeing the error log that's in RDS. Here's an example from one of the time the nodes crashed.
Copy code
2025-11-20 15:33:11 UTC:10.0.137.112(56897):clusteradmin@medplum:[16424]:WARNING:  there is no transaction in progress
2025-11-20 15:33:36 UTC:10.0.137.112(15189):clusteradmin@medplum:[16308]:WARNING:  there is no transaction in progress
2025-11-20 15:34:27 UTC:10.0.179.29(42215):clusteradmin@medplum:[16355]:WARNING:  there is no transaction in progress
2025-11-20 15:34:30 UTC:10.0.137.112(62881):clusteradmin@medplum:[2954]:WARNING:  there is no transaction in progress
2025-11-20 15:34:36 UTC:10.0.137.112(49187):clusteradmin@medplum:[16164]:WARNING:  there is no transaction in progress
2025-11-20 15:34:51 UTC:10.0.179.29(45975):clusteradmin@medplum:[3187]:WARNING:  there is no transaction in progress
2025-11-20 15:34:53 UTC:10.0.137.112(15189):clusteradmin@medplum:[16308]:WARNING:  there is no transaction in progress
2025-11-20 15:35:01 UTC:10.0.179.29(25743):clusteradmin@medplum:[16255]:WARNING:  there is no transaction in progress
2025-11-20 15:36:40 UTC:10.0.137.112(19851):clusteradmin@medplum:[17736]:WARNING:  there is no transaction in progress
2025-11-20 15:47:05 UTC:10.0.179.29(18561):clusteradmin@medplum:[16400]:LOG:  unexpected EOF on client connection with an open transaction
2025-11-20 15:47:25 UTC:10.0.137.112(12767):clusteradmin@medplum:[16367]:LOG:  unexpected EOF on client connection with an open transaction
2025-11-20 15:47:25 UTC:10.0.179.29(18153):clusteradmin@medplum:[16404]:LOG:  unexpected EOF on client connection with an open transaction
2025-11-20 15:47:25 UTC:10.0.137.112(60903):clusteradmin@medplum:[17411]:LOG:  unexpected EOF on client connection with an open transaction
2025-11-20 15:48:05 UTC:10.0.179.29(47853):clusteradmin@medplum:[16211]:LOG:  unexpected EOF on client connection with an open transaction
i
Do you have slow query logging turned on?
s
I haven’t enabled it, we did deploy with the CDK process though. Not sure if it’s enabled as part of that
Copy code
SELECT "ServiceRequest"."id", "ServiceRequest"."lastUpdated", "ServiceRequest"."content" FROM "ServiceRequest" WHERE ("ServiceRequest"."deleted" = $1 AND "ServiceRequest"."projectId" IN ($2) AND "ServiceRequest"."__identifier" @> ARRAY[$3]::UUID[]) ORDER BY "ServiceRequest"."authored" LIMIT 11
$1 = 'false', $2 = 'ff600e8b-234a-49ab-86da-ed0f206c6b42', $3 = '7e6621fe-d182-5aa3-b6d8-71c45980e322'


Limit  (cost=25.33..25.33 rows=1 width=38) (actual time=0.029..0.030 rows=1 loops=1)
  Buffers: shared hit=7
  ->  Sort  (cost=25.33..25.33 rows=1 width=38) (actual time=0.028..0.029 rows=1 loops=1)
        Sort Key: authored
        Sort Method: quicksort  Memory: 25kB
        Buffers: shared hit=7
        ->  Bitmap Heap Scan on "ServiceRequest"  (cost=21.30..25.32 rows=1 width=38) (actual time=0.023..0.024 rows=1 loops=1)
              Recheck Cond: (__identifier @> '{7e6621fe-d182-5aa3-b6d8-71c45980e322}'::uuid[])
              Filter: ((NOT deleted) AND ("projectId" = 'ff600e8b-234a-49ab-86da-ed0f206c6b42'::uuid))
              Heap Blocks: exact=1
              Buffers: shared hit=7
              ->  Bitmap Index Scan on "ServiceRequest___idnt_idx"  (cost=0.00..21.30 rows=1 width=0) (actual time=0.018..0.018 rows=1 loops=1)
                    Index Cond: (__identifier @> '{7e6621fe-d182-5aa3-b6d8-71c45980e322}'::uuid[])
                    Buffers: shared hit=6
Settings: maintenance_io_concurrency = '1', effective_cache_size = '10408168kB'
Planning:
  Buffers: shared hit=602
Planning Time: 0.987 ms
Execution Time: 0.057 ms
Not sure if this helps. I can get slow logging turned on shortly
i
@Spencer Smith , you mentioned you have more data in the medplum hosted development environment, do you have the same shape of data there? Also, how are you executing these queries? Are you directly hitting the API?
s
@Ian Plunkett , the shape of the data is the same yeah. Our dev env is just more used than test so it has more records in it. Our setup is 
custom frontend
-> 
custom backend with Medplum SDK
->`Medplum`. However for this test I took one of the queries that was causing the crash and have been running it directly against the Medplum API with curl in both envs.
i
In your testing, are you running the exact same query every time? Or are you changing anything like the offset? Also, does the query consistently fail after the exact same number of invocations each time or is it more random?
s
A bit of both. We first noticed the crash while using our app, so the queries would've been slightly different. After switching to just hitting the API directly, I've been using the same query over and over again. It seems to be pretty random. I do think we saw negative performance when lowering the max connection count to the DB, but it was pretty late at that point if I'm being honest so my memory might be failing me. 🙂
r
@Spencer Smith we're probably getting to the end of what we can debug remotely - a lot of these SRE type issues require in-depth engagement. But we have seen that response times can slow down as you reduce the number of connections to the DB. Exhausting your DB connection pool could be the reason you're seeing low CPU usage but timeout exhaustion. By limiting DB connections, you rate limit the traffic to the DB, so CPU usage is limited. However, the healthcheck also requires 1 DB connection ( to check that the DB is live). If a query is exhausting that pool, it might take too long for the healthcheck to respond
s
@rahul1 I really appreciate all the help. We talked with Reshma and are going to go the hosted route for now so we can get launched. The engineer in me very much wants to figure this out though when I have the time. 😄 Thanks again to all who helped.
r
Great! If you don't mind sharing, I think it would help the community to know why you decided to start w/ self-hosting, and why you're making the switch to hosted
s
Hey Rahul, sure thing! • Why did we decide to self-host? We actually started with a hosted env on the (now discontinued) dev plan for our development env. When we were ready to setup higher envs I began looking into self-hosting. Most of my career has been in designing/running cloud infra so it seemed like the logical choice. The MP stack is relatively simple (compared to running a full blown k8s system) and actually quite similar to our app's setup (Fargate service behind an ALB). For 3-4 months we've had our self-hosted test environment running without any issues until last week. We made changes to how we were querying for some data (moving to using more 
_include
and 
_revinclude
with 
:iterate
) and that's when we noticed our self-hosted MP servers locking up. However our dev env (hosted by you guys) wasn't having any issues. • Why move to hosted? This was really just the classic 'build vs buy' decision. It came down to what was the best use of my time while we're trying to get our product launched. Like I mentioned above, the engineer side of my brain wants to sit here and figure out what I'm doing wrong and solve the problem, but ultimately that's not what's best for our customers right now. We're drastically better off letting the experts (the MP team) handle hosting while we focus on our app launch. It also means that we get to pay you guys more money. I say 'get to' because Medplum is an amazing tool that we want to see succeed. 🙂 Eventually I'd like to see us switch to self-hosted with a support agreement since there are some things that are self-hosted only for obvious reasons (custom search params, multiple projects, etc.). For now though, hosted is the way to go!
r
Thank you so much @Spencer Smith
> custom search params Stay tuned in 2026. We might have some hosted goodies for you 😉
s
56 Views
PreviousNext
Powered by