https://medplum.com logo
Join Discord
Powered by
🚨 Urgent Medplum Performance Help
# support
m
We are running into significant performance degradation due to high-volume 
_count
requests hitting our write cluster. We need a short-term fix to stabilize performance while we work on a long-term architecture change. The Problem Our application relies on inbox counts for 
ServiceRequest
resources (a ServiceRequest can belong to multiple inboxes). We are polling periodically for these counts. * Impact: CPU spikes to 100% on the cluster, and all requests (not just counts) slow down/fail (Memory/IO are low/fine) * Load: We're generating Users X Inboxes X Polling Rate = hundreds of simultaneous 
_count
requests Technical Details & Questions 1. Read Replica Utilization: Our cluster has one write and one underutilized read replica. All these 
_count
requests are hitting the write replica * Q1: Is there a standard way to force these specific read-only requests (
_count
) to the read replica? Or a recommended way to enable read-request round-robin? 2. Medplum Version: We are on v2.1.4 (yes, we know 😬) * Q2: Can we expect a significant performance gain for 
_count
requests by prioritizing an upgrade? 3. Accuracy vs. Performance * 
_total=accurate
is the culprit * 
_total=estimate
is too inaccurate * Q3: Will the 
accurateCountThreshold
setting help balance speed and accuracy for our specific issue? (How does this feature interact with our count load?) 4. GraphQL Aggregation: * Q4: Can we use a single GraphQL query to perform these count aggregations in one call, reducing the N-query problem to a 1-query problem? > Count Request Example: > 
/fhir/R4/ServiceRequest?_count=0&_filter=authored%20pr%20false&_tag:not=archived,needs-review&_total=accurate&identifier=titan-intake::InboxStatus|registration
We appreciate any help!
r
@mikeheme COUNT queries are known performance bottlenecks in postgres
our immediate suggestion would be to upgrade and set accurateCount threshold to a more manageable number (on production, we have it set to 1M)
other options would be to 1) increase your polling interval (immediate relief) 2) Using GraphQL will actually help because we direct all graphQL to reader instances (it's less about the N query problem)
in the medium term, we highly suggest moving to websockets for your notifications. This will allow you to SIGNIFICANTLY drop your polling interval
m
thanks Rahul, this helps a lot!
I might have more questions about (1), which I assume you mean decrease? but let me talk to my team
r
increase the interval decrease the rate
m
ah, yes, makes sense, thank you!
also, just want to make sure your suggestion (2) applies to our current version v2.1.4 ?
r
no I don't believe it does unfortunately. I believe we implemented that somewhere in the 3.xxx series
m
what would be your suggestion for upgrade? set up new cluster and dump/load data?
since we are so many versions behind?
r
I think going through the standard upgrade path should be fine
in 3.x.x we introduced the concept of a "data migration"
this means that as you enter 4.x.x, you will have to hit specifici versions. However, the server upgrade tool should warn you appropriately
m
got it, thanks.
92 Views
PreviousNext
Powered by