https://medplum.com logo
Join Discord
Powered by
Error: File Too Large
# support
j
Hey Medplum team: We're getting to the point where we're storing quite a bit of data in our Medplum server, which is super exciting. 🎉 We are starting to see 
File too large
errors when trying to store large bundles with 
client.executeBatch
- I was thinking of using Node's built in 
zlib
functionality to compress the JSON to a usable size Buffer, but... if I understand correctly, I will run through the 
compression()
middleware [of Medplum's express server] first and then the 
json()
middleware will read the length of the uncompressed bundle and still throw a 
File too large
error. Is my understanding correct? Or am I completely off base? What would you suggest as the best way to handle storing large bundles [other than splitting it into multiple executeBatch commands, which I would love to avoid if possible]?
r
Wow, I think you are teh first user to report this issue. A few questions to help me understand: - Are you storing the 
Bundle
resources, or are you using them as batch requests to create data? Typically, 
executeBatch
is for the latter - Approximately how big are your Bundles (# of entries, and ~KB)
j
1. The latter, we are intending to store several resources, and using executeBatch to do it at once 🙂
2. averaging ~150 entries per executeBatch; I know off the top of my head our largest bundle was about 49 MB and it's nearly 4000 entries
r
Awesome, that makes more sense. The short answer to your question is tht there is a server config setting for the 
maxJsonSize
- have you already tried that? https://www.medplum.com/docs/self-hosting/config-settings
Are you constructing the bundles yourself, or getting complete bundles from another source? If the former, you might want to consider the autobatching feature of the Medplum client. This automatically constructs a Bundle from all the requests you make within a certain time window
That could give you the benefit of smaller bundles, without having to manually break them up yourself
j
That's pretty cool! We are constructing the bundles ourselves from multiple sources and then just passing that straight to the 
medplumClient.executeBatch
method. Do we just set a number for https://www.medplum.com/docs/sdk/interfaces/MedplumClientOptions#autobatchtime when instantiating the client?
True, I did see that [but then started (over)thinking about the express middleware]. so if I set 
MAX_JSON_SIZE='50mb'
you'd expect that to just work?
r
This is not something we've tested, but there's nothing in the Express docs that indicate it wouldn't 🤷‍♂️ http://expressjs.com/en/api.html#express.json
Yes, we have docs on this coming soon. But jst set a number of ms in the constructor
j
nice! Excited for those docs; can you tldr* describe how you're planning for it to work?
r
Copy code
ts
const medplum = new MedplumClient({autobatchTime: 1000}); 

const patientsToCreate = [];

// Main thread continues
patientsToCreate.push(
  medplum.createResource({
    resourceType: 'Patient',
    id: 'john-smith',
    name: [
      {
        family: 'Smith',
        given: ['John'],
      },
    ],
  })
);

// Main thread continues
patientsToCreate.push(
  medplum.createResource({
    resourceType: 'Patient',
    id: 'homer-simpson',
    name: [
      {
        family: 'Simpson',
        given: ['Homer', 'Jay'],
      },
    ],
  })
);

// Both promises are resolved simultaneously
await Promise.all(patientsToCreate);
j
That's cool.. how would it interact with 
executeBatch
? I might be misunderstanding but since we just want to do the one request, we're not queueing up multiple requests [and thus nothing needs to be `autobatch`ed*] right?
Or are you suggesting we iterate through the bundle and call 
createResource
for every entry with 
autobatch
enabled? and that way the request will be as large as is allowed, and as few requests as possible will be made*?
r
Well I thought you were constructing the bundle yourself. Autobatching eliminates teh need to create a bundle in the first place - this does it behind the scenes foryou
But if you already have a bundle, then sticking with 
executeBatch
is fine
Behind the scenes, we keep track of all the requests you make in the time window. When the time window expires, the client creates a bundle, and calls 
executeBatch
on your behalf
j
In theory though, if we stopped using 
executeBatch
and called 
createResource
individually, we could count on the client to make appropriately sized 
executeBatch
requests for us, right? Or would it still hit the file size limit error if we make too many requests in the time window/with bundles** that are too large? Would autobatched requests* respect 
maxJsonSize
?
Don't mind me expanding the scope of your new feature 😅
b
Oh neat.
r
> Or would it still hit the file size limit error if we make too many requests in the time window/with bundles** that are too large? This would still happen . The client won't be aware of server settings, because we try to keep the client decoupled from the server implementation details. You actually can run the MedplumClient against any other FHIR API - i've done it against Epic and Particle health 🙂
We do have a backlog item to allow you to manually control the start / end of autobatching, rather than making it time based. Then at least you can control how many entries are in each batch
j
that sounds good - that way we could limit it to something "reasonable" like ~3000 entries per second (just my first guess)
r
haha yeah
j
> The client won't be aware of server settings, because we try to keep the client decoupled from the server implementation details. I definitely agree this is reasonable, but, I would love if in the manual control of autobatching we could specify the size of the bundle sent
@rahul1 related question though, am I able to pass a gzipped Buffer of my bundle to 
executeBatch
? If I understand correctly 
fetch
that's used under the hood should be able to use that right?
I'm going with the underlying #post method,
Copy code
const stringifiedBundle = JSON.stringify(bundle);

    if (Buffer.byteLength(stringifiedBundle, 'utf-8') > MAX_BUNDLE_SIZE) {
      const compressedBundle = await gzip(stringifiedBundle);
      responseBundle = await medplumClient.post(medplumClient.getBaseUrl().slice(0, -1), compressedBundle, undefined, {
        headers: {
          'Content-Encoding': 'gzip',
        },
      });
    } else {
      responseBundle = await medplumClient.executeBatch(bundle as MedplumBundle);
    }
but I'm curious if you have feedback
r
I think this use of POST would be fine - are you running into any errors?
j
@rahul1 in our tests we found that we're just hitting the underlying 
express.json()
behaviour to check against the inflated (uncompressed) size of the bunde: https://github.com/expressjs/express/blob/2a00da2067b7017f769c9100205a2a5f267a884b/test/express.json.js#L171
Could you share the github issue for this behaviour?
sorry for the `@`pings btw! but I feel like I'm crazy/missing something obvious 😅 . Is the check against the uncompressed size what you would expect?
r
I'm not quite an expert on express's inner workings, but based on their unit tests, it does seem to be the expected behavior: https://github.com/expressjs/body-parser/blob/ee91374eae1555af679550b1d2fb5697d9924109/test/raw.js#L111
Were you have having trouble with the 
maxJsonSize
config setting?
j
Even with that set, we’re hitting the behaviour the middleware enforces.
r
I see. So if I understand: - maxJsonSize = 50mb - You're sending a bundle to the server, compressed size < 50 mb, uncompressed size > 50 mb
We just added a ticket to potentially process large bundles async. This would probably be the "right" solution here https://github.com/medplum/medplum/issues/3222
We don't have a firm commit on this, but I wanted to share how we're thinking about it
168 Views
PreviousNext
Powered by