Self Host | RDS | Stuck at "Building structure def...
# supports
Sam
06/17/2025, 9:01 PMHello there!
First of all thank you for this awesome open-source project, I've used it in the past and am looking forward to working with it again soon 🙂
I work as a DevOps at a healthtech startup and I'm currently trying to self host Medplum on our own AWS account using the recommended services (ie. RDS, ElastiCache, ECS, EC2, etc).
After finally getting all the different components up and running (backend, frontend & storage), my ECS task running the Medplum Server boots up. However, after executing all the pre-deploy migrations, it seems to be getting stuck on the "Building structure definitions...." step (see attached screenshot). I waited over an hour earlier, but it's like if the process just decided to stop without erroring or anything.
I connected to my RDS instance via our VPN, and all the tables are there. They have been successfully created. Most of them are empty (
UsercreateSuperAdmin161452d9-43b7-5c29-aa7b-c85680fa45c6medplum/medplum-server:4.1.8db.t4g.mediumf
finnbergquist
06/17/2025, 11:30 PMHey @Sam , sorry about this. Looking into it for ya
s
Sam
06/18/2025, 12:21 PMAwesome thank you!
f
finnbergquist
06/18/2025, 9:13 PMOk, so it makes sense that only Project/161452d9-43b7-5c29-aa7b-c85680fa45c6 would exist since that one is created during the pre-deploy migrations.
Would you be able to try again and then when it's hanging, run the following two queries on your database to try to ascertain where it's stuck.
Copy code
-- shows active connections
SELECT application_name, pid, state, age(now(), xact_start) age,
left(regexp_replace(query,E'[\\n\\r]+',' ','g'), 100)
FROM pg_stat_activity WHERE state <> 'idle' ORDER BY age;
-- shows locks
SELECT a.datname, c.relname, l.tuple, l.transactionid, l.mode, l.GRANTED, a.usename,
a.query_start, age(now(),a.xact_start) AS xact_age, age(now(),a.query_start) AS query_age, a.pid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
JOIN pg_class c ON c.oid = l.relation
WHERE a.query_start < NOW() - INTERVAL '1 seconds'
ORDER BY a.query_start;s
Sam
06/18/2025, 10:08 PMJust ran those two, here are the results:
- Showing active connections: only shows the current connection that I'm on (ie. my device -> VPN -> RDS);
- Showing locks: none.
I think it is worth mentioning that today I tried booting up the medplum server locally using
cd medplum/packages/server && npm run devSam
06/19/2025, 12:47 AMAlso probably worth mentioning, my AWS infra is all written with Terraform, I tried making it as close as possible to your guys’s official CDK package, but I might have missed something.
f
finnbergquist
06/19/2025, 5:02 PMAhh, gotcha. Thanks for the clarification there @Sam
s
Sam
06/19/2025, 5:05 PMAnother issue I'm having is I'm completely unable to have the medplum server container healthcheck pass, it always reports as unhealthy. Here's my ECS task definition; the port number I'm using is
8103medplum-server:latest{PRIVATE-IP}:8103/healthcheckSam
06/19/2025, 5:08 PMEven though the logs show a normal boot of the server on port 8103.
https://cdn.discordapp.com/attachments/1384639030817193984/1385305301086896270/Screenshot_2025-06-19_at_1.08.11_PM.png?ex=68559599&is=68544419&hm=db3adc95108993164b2dc4aa210e4e40dc96db769572ebe4c0bafed6f4c8ad69&
Sam
06/19/2025, 5:14 PMThose are all the environment variables I'm using if that's any helpful
https://cdn.discordapp.com/attachments/1384639030817193984/1385306811883847824/Screenshot_2025-06-19_at_1.14.19_PM.png?ex=68559701&is=68544581&hm=db6a9d986bcdef0f494c76d5ed22101a45d8f6796f4c5b317d1468659016febe&
r
reshma
06/19/2025, 5:30 PMhi sam! Quick question - is there a reason the CDK doesn't work for you? Would love to understand if there are some limitations in your organization/environment?
s
Sam
06/19/2025, 5:35 PMHello @reshma ! Thanks for jumping in.
Most (if not all) of our other infra is written with Terraform already; we try to keep it consistent across our different services. However I will be trying out the CDK this afternoon to compare and hopefully find the root of the issues I'm having. Does that answer your question?
r
reshma
06/19/2025, 6:04 PMTotally helps ty!
s
Sam
06/20/2025, 2:01 PMHey @finnbergquist , small update here. I re-ran the medplum server init on a fresh installation (still on ECS), and now whenever it gets stuck on
Building structure definitions....Sam
06/20/2025, 2:03 PMI'm also seeing the active connection of my medplum-server ECS task, showing it's been
idle in transactionSam
06/20/2025, 2:04 PMAny idea what it could mean?
r
reshma
06/20/2025, 4:29 PMWhat does your healthcheck say?
To perform a healthcheck, access the endpoint
/healthcheck{PRIVATE-IP}8103http://{PRIVATE-IP}:8103/healthchecks
Sam
06/20/2025, 6:15 PMHey @reshma , thanks for circling back.
The healthcheck does not return anything, the request just loads / spins for a while before my browser says
This site can't be reachedrunningBuilding structure definitions...Sam
06/20/2025, 6:16 PMNetworking and security groups were configured properly to accept connections from my device.
Sam
06/20/2025, 8:54 PMAfter a lot of debugging, I finally figured out that my issue was the connection to Redis. The [Redis connection initialization](https://github.com/medplum/medplum/blob/main/packages/server/src/redis.ts#L9) was unsuccessful, so I'm suspecting the Node server was just indefinitely waiting for that connection to succeed and wasn't able to process any other code in the meantime.
Thank you @reshmaand @User for helping out! 🙂
f
finnbergquist
06/20/2025, 10:15 PMAhh, got it. Glad you were able to get that figured out!
164 Views