Hi, We are trying to migrate data from an old system to medplum and are running into some issues. In our current self-hosted setup we have an external service that creates Task resources in medplum which are scheduled by a subscription and executed by a bot. We are trying to migrate approximately 2.5 million patients and creating many tasks in parallel but are seeing a lot of "could not serialize access due to read/write dependencies among transactions" when upserting patients in the bot. We've tried playing around with the

transactionAttempts

server configuration and controlling how many tasks are in progress at the same time (e.g. 8 or 16) but still see about 40% of the tasks fail with the transaction error. If we only run one task at the time it would take a week or more to finish the migration which is too slow for our needs. Do you have any suggestions for what we could try to speed up our migration and minimize transaction errors?

Hm interesting, it would make sense that there could be transaction conflicts given the scale and the conditional put. Understood the need to make sure that you don't create duplicate patient records, but is there a risk that you would create duplicate patient records? We have had other users use transaction bundles so that the whole bundle rolls back if any of the patients fail, but then ensure that they aren't create duplicate transaction bundles in the first place

The tasks are created by an external service that uses the medplum API. A bot upserts the patients and this is where the transaction error happens. I guess the task part is not that relevant. We would like to be able to re-run the migration in case it fails without having to worry about creating duplicate patients, so we used a conditional PUT for that, which also works fine except for the transaction errors. This is also something recommended by the documentation: https://www.medplum.com/docs/migration/migration-pipelines#using-conditional-updates-for-idempotency. How do other users do it exactly? How do they ensure that there aren't duplicates?