Issues with Self Hosting on AWS
# devu
ute0
04/14/2025, 4:00 PMHi all, I'm working on self hosting with AWS using this guide: https://www.medplum.com/docs/self-hosting/install-on-aws. I'm pretty experienced with AWS and infrastructure, but I am used to terraform and have little experience with cloudformation. That said, I've been following the guide and have run into a few issues that I have not been able to work through.
The first issue that there may not be a workaround for is that I was initially trying to deploy medplum to an existing VPC. The primary issue that I ran into is that the cdk pulls in all of our subnets and does not seem to respect the 1 subnet per AZ that the cdk code is supposed to. This seems to be a known issue for the cdk (https://github.com/aws/aws-cdk/issues/3126) and throws "A load balancer cannot be attached to multiple subnets in the same Availability Zone (Service: AmazonElasticLoadBalancingV2; Status Code: 400; Error Code: InvalidConfigurationRequest;". I'm not optimistic about a resolution for this, but if anyone has a workaround, I would love to try it.
So after more or less giving up on deploying to the existing VPC, I decided to try deploying to a new VPC. I know that medplum recommends using Route53 for DNS, but we are already using cloudflare and moving to Route53 feels like something that we don't want to take on right now. That said, I created certificates for the subdomains that we want to use for medplum in AWS ACM and provided those in the config, but I keep getting errors like this:
11:22:12 AM | CREATE_FAILED | AWS::ElasticLoadBalancingV2::Listener | BackEndLoadBalancerHttpsListener54B76346
Resource handler returned message: "Certificate ARN 'arn:aws:acm:us-east-1:{...}:certificate/{...}' is not valid (Serv
ice: ElasticLoadBalancingV2, Status Code: 400, Request ID: ...) (SDK Attempt Count: 1)" (RequestToken: ..., HandlerErrorCode: InvalidRequest)
The certs are in us-east-1, are in an issued state, and by all accounts appear to be valid.
ute0
04/14/2025, 4:00 PMMy config.json (domains and some other data redacted):
{
"apiPort": 8103,
"region": "us-west-2",
"name": "dev",
"accountNumber": "...",
"stackName": "MedplumDev",
"domainName": "partner.dev.....com",
"apiDomainName": "api.partner.dev.....com",
"baseUrl": "https://api.partner.dev.....com/",
"appDomainName": "app.partner.dev.....com",
"storageDomainName": "storage.partner.dev.....com",
"storageBucketName": "...-medplum-dev",
"maxAzs": 2,
"rdsInstances": 1,
"rdsProxyEnabled": true,
"desiredServerCount": 1,
"serverMemory": 512,
"serverCpu": 256,
"serverImage": "medplum/medplum-server:4.0.4",
"signingKeyId": "...",
"storagePublicKey": "-----BEGIN PUBLIC KEY-----\n...B\n-----END PUBLIC KEY-----\n%",
"apiSslCertArn": "arn:aws:acm:us-east-1:...:certificate/...",
"appSslCertArn": "arn:aws:acm:us-east-1:...:certificate/...",
"storageSslCertArn": "arn:aws:acm:us-east-1:...:certificate/...",
"skipDns": true,
"logLevel": "debug"
}
Any help here tremendously appreciated!
r
reshma
04/14/2025, 6:22 PMlooks possible that the certificate is in a different region than the load balancer - could that be the case? If you are doing everything in us-east-1 does it work?
u
ute0
04/14/2025, 6:25 PMI considered that, but when I had used certs from us-west-2, I got an error about the certs needing to be in the same region as the load balancer so I was assuming the load balancer was deploying to us-east-1. I can try that and see what happens. Not ideal if I need to put everything in us-east-1, but I'd be really happy to get this deployed.
r
reshma
04/14/2025, 6:27 PMthat is what I'd recommend to try
u
ute0
04/14/2025, 8:09 PMRan into a few other issues I was able to resolve, but moving the whole stack to us-east-1 seems to have resolved the ssl cert problems. Thanks for the nudge!
ute0
04/15/2025, 3:39 PMOkay, after a couple of deployment timeouts (during the ECS Fargate Service deployment), I started digging a bit and it looks like load balancer health checks are failing. Any thoughts on why this might be the case?
r
reshma
04/15/2025, 3:59 PMany snippets from cloudwatch?
u
ute0
04/15/2025, 4:48 PMIt looks like I'm getting a type error in the ECS Task:
April 15, 2025 at 12:40 (UTC-4:00)
TypeError: Cannot read properties of undefined (reading 'toString')
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
at F (/usr/src/medplum/packages/core/dist/cjs/index.cjs:1:73210)
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
at addDefaults (/usr/src/medplum/packages/server/dist/config/utils.js:18:62)
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
at loadConfig (/usr/src/medplum/packages/server/dist/config/loader.js:54:44)
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
at async main (/usr/src/medplum/packages/server/dist/index.js:33:20)
MedplumTaskDefinition
April 15, 2025 at 12:40 (UTC-4:00)
{"level":"INFO","timestamp":"2025-04-15T16:40:38.385Z","msg":"Starting Medplum Server...","configName":"aws:/medplum/dev/"}
MedplumTaskDefinition
r
reshma
04/15/2025, 7:40 PMappears to be attempting to read some config which is not present - are your config settings in good shape? https://www.medplum.com/docs/self-hosting/config-settings
u
ute0
04/15/2025, 9:01 PMOh, I'm thinking maybe I missed that I need to manually create config settings in the param store: https://www.medplum.com/docs/self-hosting/config-settings#server-config. Does that sound right?
ute0
04/16/2025, 3:40 PMYes, creating params in the param store solved my issue!
Thank you so much @reshma . I'm sure you are very busy and it is awesome and amazing that you take time to answer questions here!
r
reshma
04/16/2025, 4:06 PMThanks for being part of our community! If you are willing to showcase your implementation - we would be appreciative.
u
ute0
04/22/2025, 2:58 PMHow would one showcase their implementation? I may be able to do that.
r
reshma
04/22/2025, 4:00 PMRecord a video and share it in #1220593360260304926 !