Rate limit when sending > 10 updates for a single ...
# supportc
Charlie from Imagine
10/26/2023, 9:48 PMI am testing a migration script and am getting a too-many-requests response.
r
rahul1
10/26/2023, 10:27 PM@Charlie from Imagine - we typically only rate limit auth requests. Are you getting an access token for every call?
c
Charlie from Imagine
10/26/2023, 10:28 PMNo, just passing in client secret and clientid at the beginning.
r
rahul1
10/26/2023, 10:42 PMThat is surprising. We only hard rate-limit authentication requests, as far as I'm aware right now.
There isn't a super clean way to disable via configuration.
But there are some work arounds you might want to try:
1. Using Basic Auth, to avoid doing an auth handshake for your script
2. Turning on "autobatching" to limit the number of HTTP requests your script is making. You should see better performance from this as well
c
Charlie from Imagine
10/27/2023, 2:50 PMI confirmed we are already doing #1
Charlie from Imagine
10/27/2023, 2:50 PMLooking into #2
r
rahul1
10/27/2023, 4:01 PMAnother thought - is your migration script triggering a call to Okta / Azure auth at all? We've seen cases where the 429 error is actually thrown from the Auth provider
c
Charlie from Imagine
10/27/2023, 4:01 PMIt was because we had >10 updates in our bundles
Charlie from Imagine
10/27/2023, 4:01 PM>10 updates to the same resouces
Charlie from Imagine
10/27/2023, 4:02 PMtriggers some logic in the fhir router to reject it
r
rahul1
10/27/2023, 4:02 PMOh I see! I'm sorry about that - that should be documented
rahul1
10/27/2023, 4:03 PMI'll also talk to the team to see how we arrived at that limit
rahul1
10/27/2023, 4:03 PMWould be able to poiint to a link or line number ?
c
Charlie from Imagine
10/27/2023, 4:10 PMYeah, standby
Charlie from Imagine
10/27/2023, 5:22 PMCharlie from Imagine
10/27/2023, 5:22 PMsorry for the delay - been in some meetings 🙂
r
rahul1
10/27/2023, 5:38 PMHmm, I see you've found our circuit breaker 🔌
This was a safety measure introduced to guard against rogue webhooks. There were medplum users who had
Subscriptionsrahul1
10/27/2023, 5:39 PMWould you mind sharing an example of the kinds of updates the batch request is making for a single resource?
There might be some patterns we can use to consolidate the requests in to a smaller number of
UPDATEc
Charlie from Imagine
10/27/2023, 7:24 PMWe have a caregiver (modeled as a tagged patient) that has 10 children patients. In our bundle each related person linked is done with a patch
Charlie from Imagine
10/27/2023, 7:25 PM Copy code
{
op: 'add',
path: '/link/-',
value: {
type: 'seealso',
other: { reference: relatedPersonUrn },
},
},r
rahul1
10/27/2023, 7:33 PMHave you already tried adding multple PATCH operations to a single PATCH entry in your bundle? If you know the
RelatedPersonc
Charlie from Imagine
10/27/2023, 7:56 PMThe bundles are at the child level unfortunately
Charlie from Imagine
10/27/2023, 7:56 PMBut we had considered that
Charlie from Imagine
10/27/2023, 7:57 PMOur workaround for now is to shuffle our source patient list so that we don't do the same children close together.... hacky but is working
r
rahul1
10/27/2023, 8:39 PMOk I'm sorry about that - i understand it's not the best solution. I'm happy to take a closer look to see if I can offer any help on consolidation.
We can also look into changing that circuit breaker logic for bundles - but you would still see 10 different updates in teh caregiver's history, which may not be ideal
c
Charlie from Imagine
10/27/2023, 8:41 PMThe constraint makes total sense to have really. I think we just need to make sure our script does not have abusive update patterns 😄
175 Views