Timeout when launching Fargate
# support
s
I proceeded with the skipDns option. Now, I am stuck at the deploy stage. The issue is related to the creation of the CloudFormation stack. It is failing during the creation of the backend Fargate service. Can you help me to resolve this issue?
r
HI @sanket_89360 , it seems like your Fargate deployment is timing out. I'm trying research the cause of this, but any more information you can provide from your side will be helpful
@sanket_89360 - the best way to debug this would be to look at your ECS logs to figure out why the Fargate Service timed out. See this issue: https://github.com/aws/aws-cdk/issues/14457 You can find the fargate evnets by navigating in the AWS Console: Amazon Elastic Container Service > Clusters > YOUR_ECS_CLUSTER > Services > YOUR_FARGATE_SERVICE. Then click on the "Events" tab
s
ok, i will check
issue is not resolved yet. attaching some screenshots for ref:
p
@rahul1 how we can solve it? We tried from scratch as well. This is blocker for us as we are not able to proceed further to use Medplum for exploring for our use cases. Your help is much appreciated. Happy new year in advance 🙂
r
Hi @Pravin and @sanket_89360 - happy new year! unfortunately it is hard to tell what is going from these screenshots. - In your 3rd screenshot, I see that the Fargate service seems to be running successfully. But in the 6th screenshot, it seems like it has failed. Is that correct? Do you know the cause of the failure? - Can you follow my instructions above to find the "Events" tab. This will have logging information to debug why the fargate service failed
s
Yes, The total time taken for the deployment step is approximately 1.5 hours. I took screenshots in between.
p
@sanket_89360 please check "Can you follow my instructions above to find the "Events" tab. This will have logging information to debug why the fargate service failed" and provide more logs to Rahul?
r
@sanket_89360 you will just need to click on the link in screenshot 6 in the table.
MedplumStaging-BackendFargate...
s
i am working on it
@rahul1 As per your suggestion,I attempted to capture the logs from events in ECS to debug why the fargate service failed. I encountered difficulty in capturing the last event/log, which triggered the rollback process. The overarching issue is that the entire deploy and rollback process consumed nearly three hours. Due to rolling back, I am loosing log access. How to overcome this issue? I also used "--verbose" option, but no valid logs were generated.
@rahul1 Today, I attempted deployment with the "logLevel": "DEBUG" configuration in the config.json file. Unfortunately, I did not receive any fruitful output. Throughout the process, I captured some screenshots of the ECS events and logs. Upon reviewing the ECS events and logs, I noticed a repetitive set of logs or events. I kindly request your assistance in resolving this issue. Your guidance and support would be highly appreciated. Thank you.
p
@rahul1 what do you think what we can do further to find root cause? Should we try something different? Should we try completely new domain instead subdomain(we tried skipdns as well). Your guidance will be really helpful to resolve the setup issue
r
Based on the logs you sent, it seems like there is an unhandled error when connecting to your RDS DB. Specifically, on server startup, this line is failing (https://github.com/medplum/medplum/blob/a0e83131e406eabda1280fdc63bdbea9b15a9032/packages/server/src/database.ts#L31)
The reason these logs are repetitive is that Fargate is continuously trying to: start the server, connect to the db, crash, start a new server process, ...
I would suggest: 1. Confirming that your RDS cluster is up and running 2. Confirm that there is a valid
DatabaseSecrets
key your AWS Parameter Store 3. Confirm DBSecret (in AWS Secrets Manager) with valid DB credentials
p
@sanket_89360 I believe you are checking above
s
@rahul1 thank you for your valuable support. I have deployed infra with the help of medplum. But i have following doubts: 1)A total of three target groups are created, out of which only in one target group, one target is registered . In the remaining target groups, targets are not registered. 2)RDS is deployed by Medplum. Can we access that RDS directly? If yes, then what about credentials? Because during the deployment process, I couldn't find the RDS credentials. 3)Is there any way to crosscheck that the deployed infrastructure is as per the ideal condition or not? 4)Command to rollback the process (i mean if i want to delete the infra which command should i use?)
@rahul1 i deployed medplum with a new domain "healthinnovate.solutions". Now "https://api-stage.healthinnovate.solutions" is working. but "app-stage.healthinnovate.solutions" is not working. wanted to bring your attention to some observations regarding the app: 1)the respective S3 bucket is empty. 2)The permissions for the bucket are set in such a way that both the bucket and objects within it are not public. I kindly request your assistance in resolving this issue. Your guidance and support would be highly appreciated. Thank you. Attaching some screenshots for ref:
p
Any thoughts @rahul1 , seems with new domain everything worked, only FE part remaining now
r
Hi @sanket_89360 @Pravin , sorry for the delay, I'm taking a look now. I'm glafd you were able to make progress!
I just wanted to confirm - what was the output when you ran this step to install the app? https://www.medplum.com/docs/self-hosting/install-on-aws#deploy-the-app
p
@sanket_89360 please check and reply
s
@rahul1 I apologize for any inconvenience. I would like to inform you that I overlooked that step initially. I am pleased to share that I can now successfully access "app-stage.healthinnovate.solutions" as well. Thank you for your understanding.
r
Glad that this is working @sanket_89360 ! For the benefit of our community, could you please share the steps you took to make it work? What changed between your initial deployment at
healthconnect.mindbowser.com
and
healthinnovate.solutions
??
p
@sanket_89360 lets contribute to wiki or community wherever we can
@rahul1 also where we can find super admin credentials?
p
Thanks.
and we can change email and password from profile?
I did changed email but using new email its not allowing to login
@rahul1 any thoughts about how to change super admin email? Rest we are good to close this ticket
r
@Pravin you can use the force set password functionality described here: https://www.medplum.com/docs/self-hosting/super-admin-guide#force-set-password
You can also invite your own users to the Super Admin pproject, and then remove the
admin@example.com
user. This is probably the more straightforward solution
s
@rahul1 as you know that api and app is working smoothly. now i want to configure Medplum Server for SMTP. For that i have to update cofig.json file on local macine for following parameters: "supportEmail": "support@example.com", "smtp": { "host": "smtp.sendgrid.net", "port": 587, "username": "apikey", "password": "YOUR_API_KEY" } Also i have to update parameters store for above 2 parameters. After that i have to restart the server. Can you tell me the command to restart the medplum server? should i use following command: "aws ecs update-service --cluster your-cluster-name --service your-service-name --force-new-deployment"
r
The update server command will be sufficient:
p
but app is showing same error as posted above
seems not pickign smtp config
s
@rahul1 i have followed the above mentioned steps, still it is throwing error. here are some screenshots for ref: I have one more doubt, in smtp what should be the username? acual username or apikey?
r
@sanket_89360 @Pravin - this is a new issue. Do you mind opening a thread in this forum?
s
ok , sure
154 Views