Anyone have experience loading large amounts of data into Medplum via the @medplum/core/MedplumClient? https://www.medplum.com/docs/sdk/core.medplumclient.executebatch Ex - a use case where you had 2000-5000 messages per minute. I'm wondering if the executeBatch is the right approach and just any insights or challenges people might have encountered at this example scale.

HI @RyanG , you might run into scale limits and timeouts pretty quickly at that rate. What is the sioruce of this data - ADT feeds?

Hey @rahul1, yeah it would be a combo of ADT, ORM, and ORU feeds

@RyanG , you should take a look at our guide on ingesting HL7 messages using the Medplum On-prem Agent: https://www.medplum.com/docs/agent#preparing-for-scale

thanks for the info @rahul1 after looking at it more, It seems like Observation resources, like Labs/Vitals for X number of hospitals is where the high volume in data could come from. I'm hoping to be able to filter out only certain labs/vitals, but in the scenario where we had to load in large categories of labs/vitals, the data could get pretty large one more quick question I've been exploring, have you had anyone insert into the Medplum PG DB directly (without going through the Medplum Client). Like if we inserted Observation resources directly into the

Observation

table using a typescript PG library, but made sure we handled all the Medplum DB ID's correctly when referencing other resources in this PG Table. This way we could use some of the Postgres bulk insert/update functionality. How bad of an idea do you see this 🤔 ?

It's generally not recommended because there are a number of ancillary operations that get performed via the API layer upon writing to the DB: * Writing to history tables * Pulling out search parameters to write to the appropriate columns (including additional tables, such as the *_Token tables) * Updating resource compartments * Validating resource schemas