Hey there 👋 I have a question about BullMQ usage for FHIR Subscriptions. The FHIR subscription construct appears to use a library called BullMQ. Super cool library, with a lot of support and adoption, that enables an async queueing implementation on top of Redis. However, I am concerned about how this is hosted in Prod. When looking at the Cloudformation stack provided for self hosting - it uses AWS ElastiCache for Redis as a shared resource for BullMQ and the caching layer for FHIR resource fetching. ElastiCache does not guarantee disc persistence, so when a hardware fault occurs, you can and will lose everything. For the cache that's totally fine, but for subscription queues, I think thats a much bigger deal... What would happen if we lost all queued bot executions? Does production use something different than the CDK package that is provided for self hosting? Here are my sources: > https://www.medplum.com/docs/api/fhir/resources/subscription - Medplum Subscription Docs > https://github.com/medplum/medplum/blob/main/packages/server/src/fhir/repo.ts#L856 - Where FHIR updates trigger subscriptions. > https://github.com/medplum/medplum/blob/08d11217b7b399c6232432f04389d4c66ce85f18/packages/server/src/workers/subscription.ts#L65 - Where the subscription workers are initialized and run. > https://github.com/medplum/medplum/blob/08d11217b7b399c6232432f04389d4c66ce85f18/packages/cdk/src/backend.ts#L134-L148 - Where the self hosted Redis resources are defined. > https://aws.amazon.com/elasticache/faqs/ - FAQ for ElastiCache, contains question about Redis AOF support for persistence.

Hi @Charlie from Imagine - good question. The default Medplum Elasticache config uses Multi-AZ with auto-failover. By default, one replication group, so there are multiple nodes. This is not a perfect solution, but it is comparable if not better than RDB and/or AOF. See https://lipanski.com/posts/persistence-in-elasticache-redis With that said, we are not opposed to enabling RDB and/or AOF. There are some operational considerations (performance, cost, etc). We have also debated adding support for SQS/SNS as a BullMQ replacement. That would provide a proper durable service. There are operational considerations here too. We are open to discussion, if you'd like to discuss more offline, please send a DM and we can schedule some time.

Does multi-az failover account for restarts?

yes. patches and upgrades should happen sequentially on secondary, swap to primary, etc.