Bot rate limited even though FHIR quota not used
# support
m
Hi. We have a couple of bots that do a lot of requests, so we increased the FHIR quota to 6.000.000 for those bots using a user configuration. But the bots are still encountering rate limit errors and in the logs we see the following:
Copy code
json
{
  "level": "WARN",
  "timestamp": "2026-01-25T09:51:45.528Z",
  "msg": "User rate limited",
  "limit": 6000000,
  "used": 310585,
  "msToReset": 530,
  "enabled": true,
  "projectId": "9f724bc4-84a2-473e-b1d2-3d0eb09a845c",
  "profile": "Bot/be8c14fd-4fa4-4884-8b87-9c35b6fa3030",
  "requestId": "bb8aff12-3614-4961-af1b-325b1de851d9",
  "traceId": "1-6975e7b1-327ed3ca10d5727b64c03dca"
}
It looks like the limit is set correctly to 6.000.000 but we only used 310.585 points before an error is raised. The bots are triggered from subscriptions. Any idea on how to prevent the rate limit errors?
m
Hi @Mirza ! Are you self-hosting Medplum or using our hosted solution? There are project-level limits that the user configurations cannot override, and those project-level limits can be adjusted on your own for self-hosters, and will have to be changed on our end for those using our hosted solution
Also, 6million as a FHIR quota is quite high! May I ask what your bot is doing?
g
Hi @Maddy We are using a self-hosted solution, and our bot is migrating millions of records in parallel from a legacy database into Medplum and we were getting rate-limited pretty fast. What is the correct approach to increase the FHIR quota in this scenario? Increase the project-level? If so, how? Thanks!
a
Hey @Gonzalo, sorry I dont have a fix for you, but could you please share how you increased the limit? I've been trying to do the same but none of the things I tried works. I tried all 3: increasing at server level, project level, and also at user level...
g
Yes, sure. We created an
UserConfiguration
resource with the FHIR quota option, e.g.:
Copy code
{
  "resourceType": "UserConfiguration",
  "id": "yyy",
  "option": [
    {
      "id": "fhirQuota",
      "valueInteger": 6000000
    }
  ]
}
and then set it then
ProjectMembership
that links the
Bot
to the `Project`:
Copy code
{
  "resourceType": "ProjectMembership",
  "profile": {
    "reference": "Bot/xxx",
  },
  "userConfiguration": {
    "reference": "UserConfiguration/yyy"
  },
  "project": {
    "reference": "Project/zzz",
  },
  "user": {
    "reference": "Bot/xxx",
  }
}
a
This is also one of the things I tried, but I am still getting rate limited and the system doesn't pick up my increased limit. It still shows 50K as the limit 🙂
m
Hi @AP , I assume you are self-hosting as well?
a
Yep!
I am
a
Yep 🙂
I'll add on some more info there, don't want to hijack this thread. But thanks Gonzalo for the advice!
m
This is odd, you had the correct set of settings: Server level: increase the
defaultFhirQuota
parameter Project level: set the
systemSetting
array as the screenshot below, which must be done via the super admin console User level: set the UserConfiguration and add it to the ProjectMembership https://cdn.discordapp.com/attachments/1465321110634627102/1465458295853809932/Screenshot_2026-01-26_at_1.26.27_PM.png?ex=69792de4&is=6977dc64&hm=8ccac2c2f23c7df71cbc34adfe552bc41fdd5033d52b39e8cb1eef44829a665c&
@AP since this seems to be the same issue, I can answer you here. Did you set the systemSetting array at the project level from super admin, and can you confirm that the setting was stored? You shouldn't be able to configure systemSettings on the Project resource from the non-super admin project
a
@Maddy Yes, I tried as super admin. I think it might have taken a while, but now without any more modification from me, it seems that the FHIR quota error has gone away. However, my bot is still giving me a "Failed to fetch" error, but then the logs don't actually show anything. There is no error. I set up cloudwatch on the bot which is failing, and the logs from cloudwatch do not show any error either. But the funny thing is, the logs continue to populate after receiving the "Failed to fetch" error.
My best guess is that it might be a redis issue if you guys cache this parameter in there? But at least my logs do not show 429 error code anymore
I did code in a logic in the bot's code to wait the
_msBeforeNext
number of seconds (this is given to be whenever medplum ran into 429) before retrying because the fhir quota wasn't increasing so maybe thats why the system stopped giving 429? And can this timeout be the reason why my bot gives failed to fetch?
m
Hm I am a bit confused here. Are you saying that there are no logs of the bot runs, or that the logs are just showing a "Failed to fetch" error?
a
The logs on Medplum, under the "Event" tab in Bots page are not populated immediately, but show up after a while. And they do not show a "Failed to fetch" error actually
m
Do you have logic when using
_msBeforeNext
so that you only use the information in that header if you get a 429? Successful requests also have an
_msBeforeNext
header in the response, even if the rate limit has not been hit yet
Regarding the "Event" tab "not populating immediately", these events will only populate once the Bot has completed, which depending on how long the bots take to complete, can have some delay
a
Yes, I only wait if I get 429
I think because of me waiting for too long, my lambda is getting timed out. Just saw that my final output came after 15 mins of lambda execution
So I guess I am getting timed out by Medplum's 50K limit and then waiting for too long, and then finally getting lambda timed out
Which again ties back into the first problem I had, how do I increase the FHIR limit? 🙂
m
Hm, when you set the UserConfiguration against the ProjectMembership, are you using the ProjectMembership tied to the ClientApplication that is triggering this bot? How is this bot being run?
a
So far, I have tried to increase the quota using the Parameter store: /medplum//defaultFhirQuota and set it to 5000000 (5 million just for testing) Also set defaultRateLimit to 5000000/minute In project settings, I logged in as a super admin, went to my project, edit, system settings, and added in totalFhirQuota and defaultFhirQuota, both set to 5000000
m
> I think because of me waiting for too long, my lambda is getting timed out. Just saw that my final output came after 15 mins of lambda execution This is a interesting, are you having multiple lambda executions happen at the same time?
> totalFhirQuota and defaultFhirQuota, both set to 5000000 I assume you meant
userFhirQuota
here. If the
userFhirQuota
is set, you shouldn't need to add a userConfiguration, so it might be worth removing it here and testing at that level
a
Yes, so I basically created a new userConfiguration with the following json:
Copy code
{
  "resourceType": "UserConfiguration",
  "name": "Bot High Quota Config",
  "option": [
    {
      "id": "fhirQuota",
      "valueInteger": 50000000
    }
  ],
  ...
}
And then found the project membership for the bot, and added the user config to it
No, just one. But I think the timeout happens because I am waiting in my code after receiving 429 error for the limit to reset, so I reach the 15 min limit
I did this one before I did the userConfiguration change. But it still didn't work. So I added the user configuration one too, and still doesn't work. I guess I can try removing userFhirQuota and try again
As for running the bot, right now I am manually executing it through the UI, but it is also wired to run once every day via a cronjob
m
Hm ok, are you logging the headers? Are those headers still reflecting "points remaining"/`fhirInteractions` in the 50k range or the 50million range?
a
Not logging them right now, but the 429 errors were in the 50k range
Example output: 2026-01-26T18:01:04.521Z f13fa27c-e38d-4b19-a0a6-61949cfc1c49 INFO Upsert error - Observation: Too Many Requests ({"_remainingPoints":0,"_msBeforeNext":38805,"_consumedPoints":64020,"_isFirstInDuration":false,"limit":50000})
This was the output with the 429 error
m
Hm ok this is odd. The pivotal configuration is the
totalFhirQuota
and
userFhirQuota
in this case. The server one is not relevant
Is the
totalFhirQuota
and
userFhirQuota
being set on the Project that the Bot resides in, or on the super admin project itself?
a
On the project that the bot resides in
m
Hm ok, mind removing the User Configuration resource and the reference to the UserConiguration on the ProjectMembership? Then, just on normal requests in the browser, mind taking a look at the network tab and checking what the what the rate limits showing there are?
a
Sure, let me try that
So I clicked on execute just once, and it seemed to give a 504 error with no response. Then it started a new execution again
All while staying in the "loading" state
It happened 3 times and then I got failed to fetch
But the execution definitely went through, and I can see 3 different log streams being created in CloudWatch, and those are still executing
m
Ok, are there any 429s in the logs?
a
No, because if I get 429, I just wait the _msBeforeNext. Its in a try-catch block
So I just get a timeout from AWS Lambda after 15 mins
m
Hm, the 429 should still show in the Cloudwatch logs in any case, though it may not show up in the Bot AuditEvent
a
Oh okay, let me check again
I don't see any 429s in CloudWatch
But I am not logging it anywhere, I wrapped the entire thing in a try-catch block
So will that still trigger 429 on cloud watch?
m
Also, can you just open up your browser, on your version of app.medplum.com, and see what the headers are showing there? Here you can see that I set an unusually low fhir quota of 10k https://cdn.discordapp.com/attachments/1465321110634627102/1465475869064826923/Screenshot_2026-01-26_at_2.36.56_PM.png?ex=69793e42&is=6977ecc2&hm=87ddb65912b84f59e029839909703d8e21dddf81d9289634a66f2e9cb915b20c&
a
I dont see a ratelimit header on my version of medplum when I go to the $execute request header, but I see it on patietns and it is: "requests";r=49999998;t=60
So I guess it is picking it up. I could remove the try catch block from my code and try again
But then any idea why I am getting 504?
m
What is your bot doing? Is it calling out to some other API server?
a
Its an EHR sync bot. It is calling an EHR's bulk export endpoint
I am pulling over 1000+ patients 🙂
m
Is it possible that that API server is returning a 504?
a
No, I don't think so. Because even though the bot gives 504, the actual execution happens through lambda. The cloudwatch logs show that at least
And out of the 1000 patietns, I could fetch and update about 150
m
Ok, are you adding console.log statements inside the bot itself? Do you know where exactly it gets a 504? Like which API call?
a
I do have them, but the 504 happens only on like the UI side of Medplum, since the bot/cloudwatch doesn't show 504
So the 504 doesn't get logged
Happy to share code privately if you want
m
Can I ask what you mean by the "the UI-side of Medplum"? Are you talking about in the Events tab inside of the UI screen? Can you share a screenshot of that?
m
Hi @Maddy. Do you have an idea why there is a discrepancy between used (~300.000) and limit (6.000.000) in the logs when rate limited?
a
When I execute a bot, I get a 504 error after a few secs which causes the bot to retry up to 3 times. Then it gives me a failed to fetch error with nothing being return in the UI. See screenshot. The events tab is empty when I get the error on Medplum. However, when I navigate to the cloudwatch logs for the Lambda function which is linked to this bot, I can see the bot being executed After digging deeper into the 504, I found out its because of a CORs issue. Apparently requests from my admin console gets rejected by the api server on ECS which is causing the 504 (but the bots still execute). I checked the allowedUrls parameter in the parameter store and saw that my admin console is already under allowedUrls... but it is weird that it is still giving me the error
Based on my current debugging progress, I think the quota limit bug is no longer there. I removed the _msBeforeNext and the try catch block entirely, and it seems to work fine now. No more 429s. But I get a CORs issue just for one bot (others work fine), and it gives me a 504 when I try to execute the bot itself, possibly due to the ALB timing out, since the 504s for each request is consistent with the default ALB timeout
63 Views