Denormalization and SQL on FHI
# support
b
Have any Medplum customers started to denormalize FHIR from Medplum to analytics data stores? Have to imagine yes, but the Analytics page (https://www.medplum.com/docs/analytics) only has loose mentions of Bots and the Bulk Export API to explain how they might have done so. With SQL on FHIR gaining momentum, it would be cool to have standard support there
r
Would love your real opinion, is SQL on FHIR going to become a standard? As for what customers do - export to data warehouse is a common use case. Snowflake and BigQuery are popular.
r
Yeah - we've mostly seen bigquery since it's a document store and doesn't require any flattening upon ingestion, so you can bulk-export directly into the DW
@brendankeeler I looked at SQL on FHIR ~4 months ago and it seemed like it had yet to settle. Have you seen it out in the wild / has there been consensus that this is the path forward?
b
Hmmm I do think it will be a popular implementation guide. I doubt it will be regulatorily required (at least in HTI-2) I expect a lot of continued traction. my main barometers here are: 1. Every FHIR server or organization using a FHIR server needs to do something like this (do analytics on stored FHIR data) 2. It's the most active section on chat.fhir.org aside from #implementers
Not super wedded to the spec given how early it is but seems like a clean way to get operational data to analytics format
g
I read about it https://docs.aidbox.app/storage-1/sql-on-fhir and https://build.fhir.org/ig/FHIR/sql-on-fhir-v2/index.html and and wonder to what extent Medplum thinks about implementing this SQL on FHIR standard
r
We're definitely open to it! We've always wanted to support some kind of sql-like syntax for data engineers, and it seems like this spec is starting to gain adoption
The main implementation design point will be around performance and multi-tenancy - how do we implement "view definitions" in a scalable way?
g
Doesn`t this mainly depend a) on the SQL engine in the background i.e. postgres of Aidbox is not a MPP engine and semi-scalable i.e. https://www.starrocks.io would be a totally different kind of scalability - even https://duckdb.org/ as these are vectorized (SIMD) and not row based datastores. b) it is also about updating the rows in a smart way i.e. not recalculating them over and over again i.e. via a materialized view/trigger based on record changes (i.e. there are only so many patients a doctor can see in a day the ingestion volume is rather limited for a single doctor / center of doctors with a couple of them) c) multi-tenancy is a thing. And again I would say it depends on the datastore i.e. BQ https://cloud.google.com/bigquery/docs/row-level-security-intro?hl=de supports row level security where the IAM policy with certain tags will only give specific roles access to some rows. Sadly, I think this is not so convenient in postgres and you would probably have to branch it out i.e. if 1 medplum project 1:1 maps to an organization/tenant then each should have their own table with database user permissions.
Reading https://discord.com/channels/905144809105260605/1169286940999294986/1186757217412710410 it looks like you wuld need some means of a data pipeline (materialized view i.e. the DB takes care of updating; or triggers) or an explicit one like with Airflow or https://dagster.io/ (more complexity in setup, coordination)
r
THanks for the resources @geoheil ! We'll take a look. There is a higher order discussion on whether we'd want to host our own OLAP DB, or focus in integrations with the User's OLAP
The main thing I would be concerned about is the duplication of data between our current OLTP architecture and another data store
g
If you would build on top of postgres -> Alloydb you could make your existing OLTP behave more like OLAP without the synchronization overhead
174 Views