Hi everyone, I'm in the process of upgrading the self-hosted Medplum system from version 2.1.14 to either 2.1.17 or 2.1.20 using the CDK. However, I've run into an error. Could you please provide insights into what might be causing this issue? I've attached a screenshot of the error for your reference.

It seems like your server has gotten in a bad state and is receiving a shutdown singal. Can you expand the log line for "Shutdown signal received" sto share the full message?

@rahul1, I've shared the logs for the last three days. Yesterday, I created a new Medplum server using image 2.1.14, and it's working fine. However, when attempting to update the version to 2.1.17 or 2.1.20, the tasks are failing with the error

exec /usr/local/bin/node: exec format error

. Upon rolling back the update, it starts working again.

@ravindratc i'm working with our eng team to troubleshoot. Could I ask a bit more information about your system? * Are you using AWS? * Did you install medplum using our CLI commands tooling? * Are you on EC2 / Fargate? * Are you deploying our

medplum/medplum:version

docker images from DockerHub? * What machine type are you running on in EC2? An architecture mismatch can cause this error message: https://dev.to/yusadolat/fixing-the-exec-format-error-with-docker-containers-on-aws-ecs-50nc#:~:text=The%20%22exec%20format%20error%22%20when,and%20the%20docker%20image's%20architecture.

@rahul1 - Yes, We are using AWS. - I’ve Installed using CMD with the Medplum CDK. - Yes, We are on fargate service. - I’ve pull the image in my local docker then uploaded it to ECR, pulling from their. - I’ve deployed it using CDK. So, All the resources are created using the Cloud Formation. Let me know if you need any other info.

Hi @ravindratc - can you share which Fargate instance type you are using? Graviton maybe?

@cody I'm a bit confused about the service type. On my AWS, tasks are currently running under ECS, and the launch type is Fargate.

Thanks @ravindratc . A couple questions to follow up: - I see that you are using your own custom docker image, rather than our image on dockerhub. What is your reason for this? Do you have any custom patches? - I think you'll have to find your instance type in the

ServiceDefinition

, rather than task definition. It should be one of these: https://aws.amazon.com/ec2/instance-types/

@rahul1 - last time, We were encountering an error while fetching the docker hub. So, We cloned it in our ECR to resolve the issue. And we’ve not made any changes in the Image. I’ll send you instance type a in 20mins.

ok - you may want to try using the dockerhub image now. We've made improvements to our docker images since you first started your deployment. This error message typically means some incompatibility with a native dependency. To help debug further, it would be helpful to understand which docker image from dockerhub you're using (along with the architectgure). Please let me know which error you're getting with dockerhub, and I can help debug

@rahul1 @cody I attempted to update the server using a Docker image directly instead of using ECR image (Cloned from docker hub) and it worked. So, it seems there's no issue with the service type. I'm curious about what might be causing the problem, especially since we haven't made any alterations to the image.

exec /usr/local/bin/node: exec format error

^ this error message often indicates an incompatibility between compiled binary libraries / executables and the archiecture of the host CPU. In Dockerhub, you'll see that we actually build the docker image for 3 different archectures: - amd64 - arm/v7 - arm64 So it is possible that the architecture of the image you cloned didn't match the CPU architecture of your AWS fargate instances. If you point AWS to dockerhub directly, it chooses the matching architecture automatically

My system has an m2 chipset, and its architecture is ARM64. I've tried to set the system architecture manually to Linux X86_64. This is it got a mismatch.

json
"runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
}

Thanks @rahul1 , Now i understand the issue. Thanks again for your help.