Ingesting 50K x 5MB Observations efficiently
# support
g
Hi, We need to ingest 50,000 Observations within 3 hours, each ~5MB (250GB total). At that size, batching is nearly useless given the 50MB async bundle cap. What's the recommended approach? Should we use FHIR Binary + Attachment references to keep Observations lean? Thanks!
d
Hey @Goswami, thanks for reaching out! I have some follow up questions to frame my understanding: - just to confirm: each Observation resource uploaded contains ~5mb of data? - what's the use case here - are you storing images or something else with the Observation resource to get to ~5mb? - are you self-hosting Medplum? If so, could you please provide a high-level overview of your compute/db setup? - do any duplicates exist within the Observations, such that we need to PUT? Or can we just simply POST? Once I have these, I can help point you in the best direction!
s
We have batch Heart rate data from our device for the whole day We are just storing a plain observation Self hosted - yes No we just need to post
i
@Sarwagya Singh have you seen our documentation on storing Observations with timeseries data? https://www.medplum.com/docs/charting/capturing-vital-signs#using-valuesampleddata-for-timeseries-data
27 Views