I am new to using medplum. We are self hosting it but while trying to insert a lot of records at once, i am running into too many 409 serialization failures. Did anyone else run into this? Is there a way to soften the postgres transaction_isolation_level as it appears it's been forced through the app in the code.

Are you uploading bundles? Some suggestions to migitate: Practical fixes: 1. Prefer Bundle.type = "batch" instead of "transaction" for bulk loads if you do not need all-or-nothing semantics. 2. Avoid conditional creates like ifNoneExist during high-throughput import if you can precompute IDs or de-duplicate before loading. 3. Reduce parallelism and use smaller chunks. For transaction bundles with conditional operations, Medplum already caps strict serializable transaction bundles at 8 entries.

Thanks for responding @reshma . I am hitting those when i am aiming for high throughput on insertion and inserting batches concurrently. What is the standard throughput we should expect. Like how much time medplum needs to safely upsert 100k resources and 1 M resources.

Hey Zeeshan, If you're loading records for the first time and know they're unique, POST is going to be the preferred approach. It's a lot more efficient than PUT since it writes directly without a search phase and has no serialization risk. Are you able to distinguish these records in your bulk load? Like you've said, you'd risk creating duplicate resources with read_committed for conditional PUT which is exactly what conditional PUT is designed to prevent.

We are mostly working with data coming in from other EHRs and using medplum as our staging area before we process that data. All the resources that we ingest in medplum, we don't know beforehand whether they exist in our system or not hence condtional update. What if i conditionally update a million patients? will that break the system? I tested and POST/create seem to work like breeze for even a million patientns but condtional update is horrible even in 100 of ks order

No, you won't break the system - you'll just get a bunch of 409s

Thanks! It appears that there is no way to avoid 409s if we are doing conditional updates concurrently. my concern is the throughput of bulk conditional update which is slower than 10k resources per min locally and if i cannot parallelize that, i cannot use medplum as a staging area for a system which ingests 10s of millions of resources per day from customerse

It's worth noting that PUT /Patient/{known-uuid} will be much less expensive than PUT patient?identifier=system|value, as the former skips the search step. The throughput difference will be significant. Would that approach work for your use case?

That could be a workaround. If i generate a deterministic uuid 5 from identifier's value and system and use that to put on uuid. will that work?

Yes, that's what I was thinking

Because, if i even try to design something of my own, i will run in to the same problem. we cannot use a unique key constraint as a duplication guard because FHIR id system supports an array of identifiers which is why it makes sense to have serialization isolation level. Letme think more about it and see if it works for me. Thanks @Alex for all the help. Really appreciate it

Which version are you running? Want to make sure since we've made updates to make serialization errors less frequent.