Converting to MEDS

First featured at ML4H 2025

This tutorial was first featured in the MEDS tutorial at ML4H 2025.

In this tutorial, we'll extract a dataset in MEDS format for downstream use. See below for the jupyter notebook tutorial, rendered here, or check it out online on Google Colab or on our GitHub Repository

Converting a Custom Dataset to MEDS

Part 1: Loading the raw data

In this tutorial, we'll use the publicly available MIMIC-IV Demo v2.2 dataset as our fictional "raw data source". Naturally, MIMIC has been used extensively in the public space, so its structure is actually very well understood and widely used; however, for the sake of this tutorial, let's act as though it isn't and we're seeing it for the first time.

The first thing we need to do is load up the raw data (or, really, generally, a small random chunk of the raw data, so we can iterate quickly, though here we'll just use the entire demo dataset given its size) and take a look at it. To do that, we'll go ahead and download the raw files from PhysioNet and store them in the newly created raw_data directory (note this will take some time):

[1]:

%%bash
mkdir -p raw_data

wget \
  --quiet \
  --tries=10 \
  --waitretry=10 \
  --retry-on-http-error=429,500,502,503,504 \
  --no-host-directories \
  --recursive \
  --no-parent \
  --cut-dirs=3 \
  --directory-prefix \
  raw_data \
  https://physionet.org/files/mimic-iv-demo/2.2/

Now that the files have downloaded, what do they actually contain?

[2]:

%%bash
apt-get -qq install tree > /dev/null

tree raw_data

raw_data
├── demo_subject_id.csv
├── hosp
│   ├── admissions.csv.gz
│   ├── d_hcpcs.csv.gz
│   ├── diagnoses_icd.csv.gz
│   ├── d_icd_diagnoses.csv.gz
│   ├── d_icd_procedures.csv.gz
│   ├── d_labitems.csv.gz
│   ├── drgcodes.csv.gz
│   ├── emar.csv.gz
│   ├── emar_detail.csv.gz
│   ├── hcpcsevents.csv.gz
│   ├── index.html
│   ├── labevents.csv.gz
│   ├── microbiologyevents.csv.gz
│   ├── omr.csv.gz
│   ├── patients.csv.gz
│   ├── pharmacy.csv.gz
│   ├── poe.csv.gz
│   ├── poe_detail.csv.gz
│   ├── prescriptions.csv.gz
│   ├── procedures_icd.csv.gz
│   ├── provider.csv.gz
│   ├── services.csv.gz
│   └── transfers.csv.gz
├── icu
│   ├── caregiver.csv.gz
│   ├── chartevents.csv.gz
│   ├── datetimeevents.csv.gz
│   ├── d_items.csv.gz
│   ├── icustays.csv.gz
│   ├── index.html
│   ├── ingredientevents.csv.gz
│   ├── inputevents.csv.gz
│   ├── outputevents.csv.gz
│   └── procedureevents.csv.gz
├── index.html
├── LICENSE.txt
├── README.txt
├── robots.txt
└── SHA256SUMS.txt

2 directories, 39 files

We can see there are a number of data files here, including:

hosp/*.csv.gz
icu/*.csv.gz

as well as a variety of other, likely non-data files. To understand any clinical dataset, you generally should rely on both provided documentation and a local, subject-matter expert who is familiar with both the clinical and operational context of the dataset; however, in practice, we rarely have this. For our purposes, let's take a look at the provided MIMIC-IV documentation to try to understand these various files.

Part 2: MEDS Extraction, conceptually

For now, we'll focus on only a few files, to keep things simple (note that each file below links to its specific data source documentation):

To start understanding how we should think about extracting a MEDS view of this data, let's inspect some of the data using pandas:

[3]:

import pandas as pd
from pathlib import Path

DATA_ROOT = Path("raw_data")

dfs = {}

for fn in [
    "hosp/patients.csv.gz",
    "hosp/admissions.csv.gz",
    "hosp/procedures_icd.csv.gz",
    "icu/icustays.csv.gz",
    "icu/chartevents.csv.gz",
]:
  fp = DATA_ROOT / fn
  df = pd.read_csv(fp)
  print(f"{fn}:")
  display(df.head(2))
  dfs[fn.split(".")[0]] = df

hosp/patients.csv.gz:

	subject_id	gender	anchor_age	anchor_year	anchor_year_group	dod
0	10014729	F	21	2125	2011 - 2013	NaN
1	10003400	F	72	2134	2011 - 2013	2137-09-02

hosp/admissions.csv.gz:

	subject_id	hadm_id	admittime	dischtime	deathtime	admission_type	admit_provider_id	admission_location	discharge_location	insurance	language	marital_status	race	edregtime	edouttime	hospital_expire_flag
0	10004235	24181354	2196-02-24 14:38:00	2196-03-04 14:02:00	NaN	URGENT	P03YMR	TRANSFER FROM HOSPITAL	SKILLED NURSING FACILITY	Medicaid	ENGLISH	SINGLE	BLACK/CAPE VERDEAN	2196-02-24 12:15:00	2196-02-24 17:07:00	0
1	10009628	25926192	2153-09-17 17:08:00	2153-09-25 13:20:00	NaN	URGENT	P41R5N	TRANSFER FROM HOSPITAL	HOME HEALTH CARE	Medicaid	?	MARRIED	HISPANIC/LATINO - PUERTO RICAN	NaN	NaN	0

hosp/procedures_icd.csv.gz:

	subject_id	hadm_id	seq_num	chartdate	icd_code	icd_version
0	10011398	27505812	3	2146-12-15	3961	9
1	10011398	27505812	2	2146-12-15	3615	9

icu/icustays.csv.gz:

	subject_id	hadm_id	stay_id	first_careunit	last_careunit	intime	outtime	los
0	10018328	23786647	31269608	Neuro Stepdown	Neuro Stepdown	2154-04-24 23:03:44	2154-05-02 15:55:21	7.702512
1	10020187	24104168	37509585	Neuro Surgical Intensive Care Unit (Neuro SICU)	Neuro Stepdown	2169-01-15 04:56:00	2169-01-20 15:47:50	5.452662

icu/chartevents.csv.gz:

	subject_id	hadm_id	stay_id	caregiver_id	charttime	storetime	itemid	value	valuenum	valueuom	warning
0	10005817	20626031	32604416	6770.0	2132-12-16 00:00:00	2132-12-15 23:45:00	225054	On	NaN	NaN	0.0
1	10005817	20626031	32604416	6770.0	2132-12-16 00:00:00	2132-12-15 23:43:00	223769	100	100.0	%	0.0

We can see there is a lot of data contained in just these files! How can we hope to go about unifying it all into the simple MEDS format in a reasonable time?

To do so, we'll follow the assumptions of the MEDS-Extract library, which organizes the mapping of EHR data elements into the MEDS format via the following questions. For each row of each input source, we ask

What is happening in this row?
To whom is it happening?
When is it happening?

Once we can answer each of these three questions, we're ready to extract a full MEDS dataset over our inputs.

Part 2.1 Mapping the `hosp/patients` table

To see these in action, let's work through our files in order, starting with hosp/patients.csv.gz:

[4]:

dfs['hosp/patients'].head(2)

[4]:

	subject_id	gender	anchor_age	anchor_year	anchor_year_group	dod
0	10014729	F	21	2125	2011 - 2013	NaN
1	10003400	F	72	2134	2011 - 2013	2137-09-02

We can see that this dataframe clearly captures some static data about the patients in the population, any external date-of-death information present about the subject, and meta-data about how this subject's data is transformed when included in MIMIC via the anchor year group. The latter aspect won't feature in the MEDS representation, so this means we only have the following pieces of information to represent about the patient:

The information in the gender column, which for this dataset we will assign to a static measurement given it is recorded as such within the raw dataset.
The information in the anchor_age column indicating the patient's date of birth (after some transformation).
The information in the dod column, which contains a de-identified date of death for the patient, if applicable. MIMIC-IV takes this date from two sources, the hospital's own records and the Massachusetts state registry of deaths, so it also covers deaths outside the hospital, but only up to one year after hospital discharge: later deaths are censored, so dod is empty for them.

Ultimately, this tells us that for each row of the hosp/patients table, we'll want to construct 3 MEDS events:

A measurement for subject subject_id with a null timestamp with a code indicating the value in the gender column.
A measurement for subject subject_id with a timestamp given by the dod column (if it is not null) and the MEDS_DEATH code (as this is a death event) and no values.
A measurement for subject subject_id with a timestamp given by the difference between the anchor_year and the anchor_age, converted to a date-time, with the MEDS_BIRTH code and no values.

Let's record the information for these events in a simple, declarative format that we'll encode in YAML. For now, just think of this as an approximate format -- it isn't technically precise just yet. But, as we build up our specification, we'll see how we can turn it into a complete description of the extraction process. In particular, we'll have an outer level of the YAML correspond to the file we're talking about (in this case hosp/patients) and then we'll have an inner block for each of the 3 events we've identified, describing what columns they'll use to construct their timestamps and codes (we don't have values for any of these events, but we'll add them in later).

Specification so far:

hosp/patients:
  gender:
    subject_id: subject_id
    code: gender
    time: null
  death:
    subject_id: subject_id
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    subject_id: subject_id
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

Note: The patients table has already revealed two common complications when converting clinical data (to any format, not just MEDS):

The dod column only provides a date level resolution, not a time level resolution. This means that we don't know whether or not the patient died at 12:01 a.m. on that date or at 11:59 p.m. on that date, despite these two times being separated by nearly 24 full hours! This can cause issues with measurement ordering, the validity of temporal prediction tasks (e.g., predicting imminent mortality), etc. Ultimately, some choice needs to be made in how we want to represent this in MEDS. By design, MEDS does not allow you to specify a date-only timestamp, as such a timestamp does not permit a total ordering of measurements across different events. Here, as we know that death is a final event and is often (if not universally) the last event recorded for the patient, it makes sense to place it at the latest possible time within that date (i.e., add an implicit 11:59:59 p.m. onto the end of that timestamp column).
As this dataset records an "age" (via anchor_age) rather than an explicit date of birth, we have a similar, but even greater lack of temporal resolution in the date of birth column of the data. Here, we need to choose when within that year we should assign the patient's date of birth; again, there is no "right" answer, but we need to make a choice. For this event, we'll choose January 1st of that year, to keep things simple.

Part 2.2: The `hosp/admissions` table:

Next, let's inspect the admissions table:

[5]:

dfs['hosp/admissions'].head(2)

[5]:

	subject_id	hadm_id	admittime	dischtime	deathtime	admission_type	admit_provider_id	admission_location	discharge_location	insurance	language	marital_status	race	edregtime	edouttime	hospital_expire_flag
0	10004235	24181354	2196-02-24 14:38:00	2196-03-04 14:02:00	NaN	URGENT	P03YMR	TRANSFER FROM HOSPITAL	SKILLED NURSING FACILITY	Medicaid	ENGLISH	SINGLE	BLACK/CAPE VERDEAN	2196-02-24 12:15:00	2196-02-24 17:07:00	0
1	10009628	25926192	2153-09-17 17:08:00	2153-09-25 13:20:00	NaN	URGENT	P41R5N	TRANSFER FROM HOSPITAL	HOME HEALTH CARE	Medicaid	?	MARRIED	HISPANIC/LATINO - PUERTO RICAN	NaN	NaN	0

Here, we have a lot of additional pieces of data -- records of admissions, discharges, possible competing records of deaths, admission types, locations for both admissions and discharges, patient information at time of admission (e.g., insurance, language, marital status, race), and emergency department (ed*) registration & discharge information. One new piece of complexity here that is worth noting is that many of these events are "interval" style events -- namely, events that present with both a start and an end time (e.g., an admission and discharge, an ED registration and an ED out, etc.). The "MEDS way" to handle such events is to simply include both a separate, appropriately timed start event and an end event -- that way you are representing each interaction separately in its appropriate place in the patient timeline. This comes through naturally when we focus on asking our three questions from above. With this perspective, we can quickly identify a list of measurements these columns represent:

There is (or may be, if the timestamp is not null) a "hospital admission" of type admission_type to location admission_location at the time given by admittime for the subject given in subject_id (Note we are not tracking the admit_provider_id as MEDS does not currently formalize the notion of the treating provider).
At the time of the hospital admission, some patient demographics are collected about subject_id, including their:

- `insurance`
- `language`
- `marital_status`
- `race`

There may be a "hospital discharge" to the location discharge_location at time dischtime for subject_id.
There may be a "death" event at time deathtime for subject_id.
There may be an "ED Registration" event at time edregtime for subject_id.
There may be an "ED Out" event at time edouttime for subject_id.

Given these event descriptions, we can update our specification as follows:

hosp/patients:
  gender:
    subject_id: subject_id
    code: gender
    time: null
  death:
    subject_id: subject_id
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    subject_id: subject_id
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

hosp/admissions:
  admission:
    subject_id: subject_id
    code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
    time: admittime
  language:
    subject_id: subject_id
    code: "LANGUAGE//${language}"
    time: admittime
  marital_status:
    subject_id: subject_id
    code: "MARITAL_STATUS//${marital_status}"
    time: admittime
  insurance:
    subject_id: subject_id
    code: "INSURANCE//${insurance}"
    time: admittime
  race:
    subject_id: subject_id
    code: "RACE//${race}"
    time: admittime
  discharge:
    subject_id: subject_id
    code: "HOSPITAL_DISCHARGE//${discharge_location}"
    time: dischtime
  death:
    subject_id: subject_id
    code: MEDS_DEATH
    time: deathtime
  ed_reg:
    subject_id: subject_id
    code: ED_REGISTRATION
    time: edregtime
  ed_out:
    subject_id: subject_id
    code: ED_OUT
    time: edouttime

Heads up that we're being a bit imprecise with our syntax here, as this is just (for now) a mental aid -- namely, we're using some plain strings to represent column names (e.g., code: gender and subject_id: subject_id) and sometimes we're using strings explicitly indicated with double-quotes to indicate compound codes using python's string interpolation syntax (e.g., code: "HOSPITAL_DISCHARGE//${discharge_location}"). We'll formalize this later, but for now, use context to disambiguate which we mean.

Note that, much like before we've seen some other areas where challenges arise and assumptions need to be made in mapping this table:

Multifactorial measurements: Here, there are several measurements that come with different parts. We have admissions occurring with types and to locations alongside demographic data being measured like language, marital status, race, and insurance type. How should we map all of these to a set of distinct measurements with what codes? In general, this question comes down to a trade-off between more simultaneous measurements vs. more complex codes -- i.e., you can either produce more measurements for each distinct aspect of the code at the same time-point, or you can add more pieces of information into a single code string, thereby increasing the size of your vocabulary. This data extraction step shows both strategies in action, for good reason:

For admission type and location, we include them in the core hospital admission code. This makes sense because every admission has to have a type and a location -- so they are natural "modifiers" to the admission measurement conceptually, as opposed to being distinct measurements. We'd also almost never have a situation where a model would need to know that an admission happened, but not know of what type or to where.
On the other hand, for the patient demographic information, these have each been separated into distinct measurements all at the same point in time -- each aspect of the demographic data is thus recorded separately, so if we wish to later filter out rare or unknown recordings for one aspect of the demographic data in isolation of the others, this will be easy to do at a measurement level. Ultimately, however, it may also be reasonable (or even work better in some modeling tasks) to instead have produced a joint code string across all demographic information (e.g., LANGUAGE//${language}//INSURANCE//${insurance}//...). If you want to try that out yourself, let us know if it works better!

The existence of these multifactorial codes also highlights a convention we'll take in this guide, which is to compose "structured code strings" using the double-slash ("//") as a separator, as this is unlikely to occur in a raw code string. This is not a formal requirement, so feel free to use a different approach in your data -- but what is important to note is that you likely do not want code strings to collide across different measurement sources. So, if you just used code: race and code: language, for example, and UNK was an option for both race and language, your model wouldn't be able to differentiate between those two options unless you use a unique prefix (like we do here).

Competing Measurement Sources: There's another death time in this file, in addition to the dod recorded in the patients table! This is, unfortunately, a common enough problem in EHR data. Luckily, its solution is pretty straightforward -- simply decide which source takes precedence (ideally this will be a universal property, not a data-dependent one) and favor that over the other. Here, as the deathtime in this dataset has full datetime resolution, it will be preferred over the dod in the other file. We'll merely denote that with a comment in our specification for now.

At this point, our specification is also getting pretty verbose. Let's pull out the shared aspects across all event blocks into the upper level categories -- for now this just includes the subject_id specification -- so we can get rid of some wasted space:

subject_id: subject_id
hosp/patients:
  gender:
    code: gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

hosp/admissions:
  admission:
    code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
    time: admittime
  language:
    code: "LANGUAGE//${language}"
    time: admittime
  marital_status:
    code: "MARITAL_STATUS//${marital_status}"
    time: admittime
  insurance:
    code: "INSURANCE//${insurance}"
    time: admittime
  race:
    code: "RACE//${race}"
    time: admittime
  discharge:
    code: "HOSPITAL_DISCHARGE//${discharge_location}"
    time: dischtime
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: deathtime
  ed_reg:
    code: ED_REGISTRATION
    time: edregtime
  ed_out:
    code: ED_OUT
    time: edouttime

While there are other ways we could further condense this (e.g., using a list of objects rather than a dictionary of objects within each data source) that will hurt us more on clarity, so we'll keep that for now.

Part 2.3: The `hosp/procedures_icd` table

Let's move onto our next table: procedures_icd

[6]:

dfs['hosp/procedures_icd'].head(2)

[6]:

	subject_id	hadm_id	seq_num	chartdate	icd_code	icd_version
0	10011398	27505812	3	2146-12-15	3961	9
1	10011398	27505812	2	2146-12-15	3615	9

Here, we have a bit of an easier time -- there's clearly only one measurement being recorded here -- the ICD code itself, recorded for subject_id at the time given by chartdate. However, much like for the dod column in the hosp/patients table, this is only a date, not a full datetime, so we need to decide at what timestamp within the date we should assign this. Here, the situation is not quite so simple; unlike death, which is clearly a "final" event, procedures can happen throughout the day, and we don't know where it would be best to assign the recordings of their ICD codes. Ultimately, as we are more likely to want to predict things that are based on these procedures or heavily indicated by these procedures, it is better to put them later in the day rather than earlier to avoid temporal leakage -- though note that this can still cause leakage in tasks that are attempting to predict these procedure codes themselves! Regardless, we'll assign them the time of 11:59:59 p.m. on the given day here. We'll also want to ensure we capture both the icd_code and icd_version in these measurements, as both are necessary to fully define the assigned ICD code.

Before we show our new specification, note that there is one additional complexity here we should take into account, and that is seq_num. This is actually an important piece of information, as it indicates the relative prioritization of the given codes assigned to the patient (a lower seq_num indicating a higher priority code). This is a common paradigm for diagnostic codes in U.S. healthcare datasets, so we do want to include it; however, it doesn't feel quite right to include it in the code as it is not a real part of the measurement about the patient. Instead, for this example, we'll use the fact that MEDS datasets are permitted to include any other desired columns beyond the required columns, so we can just track it directly as an external column:

subject_id: subject_id
hosp/patients:
  gender:
    code: gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

hosp/admissions:
  admission:
    code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
    time: admittime
  language:
    code: "LANGUAGE//${language}"
    time: admittime
  marital_status:
    code: "MARITAL_STATUS//${marital_status}"
    time: admittime
  insurance:
    code: "INSURANCE//${insurance}"
    time: admittime
  race:
    code: "RACE//${race}"
    time: admittime
  discharge:
    code: "HOSPITAL_DISCHARGE//${discharge_location}"
    time: dischtime
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: deathtime
  ed_reg:
    code: ED_REGISTRATION
    time: edregtime
  ed_out:
    code: ED_OUT
    time: edouttime

hosp/procedures_icd:
  procedure_icd:
    code: "PROCEDURE//ICD${icd_version}//${icd_code}"
    time: chartdate @ 11:59 p.m.
    seq_num: seq_num

Part 2.4: The `icu/icustays` table

Now, let's look at icustays:

[7]:

dfs['icu/icustays'].head(2)

[7]:

	subject_id	hadm_id	stay_id	first_careunit	last_careunit	intime	outtime	los
0	10018328	23786647	31269608	Neuro Stepdown	Neuro Stepdown	2154-04-24 23:03:44	2154-05-02 15:55:21	7.702512
1	10020187	24104168	37509585	Neuro Surgical Intensive Care Unit (Neuro SICU)	Neuro Stepdown	2169-01-15 04:56:00	2169-01-20 15:47:50	5.452662

This table is much like the hosp/admissions table -- we have some "interval" style events being recorded here (namely, ICU stays) which we'll separate into endpoints, resulting in:

An ICU admission event for subject_id at intime to the first_careunit
An ICU discharge event for subject_id at outtime from the last_careunit.

Note two things:

The los here is actually a derived property -- it isn't something we want to record in the MEDS data directly (especially not in the ICU admission event because that could risk future leakage).
We're actually being a bit inconsistent here -- really, we should likely try to find another table in the MIMIC source which captures the sequence of careunits the patient is seen within so that we can record transfers to a careunit universally, rather than having an ICU admission to a careunit and an ICU discharge from a careunit -- but for now, this is outside the scope of our tutorial (but if you're interested, the right table to use for this is the hosp/transfers table, which is actually the ground truth source for the icu/icustays table.).

When we add this to our spec, we obtain:

subject_id: subject_id
hosp/patients:
  gender:
    code: gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

hosp/admissions:
  admission:
    code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
    time: admittime
  language:
    code: "LANGUAGE//${language}"
    time: admittime
  marital_status:
    code: "MARITAL_STATUS//${marital_status}"
    time: admittime
  insurance:
    code: "INSURANCE//${insurance}"
    time: admittime
  race:
    code: "RACE//${race}"
    time: admittime
  discharge:
    code: "HOSPITAL_DISCHARGE//${discharge_location}"
    time: dischtime
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: deathtime
  ed_reg:
    code: ED_REGISTRATION
    time: edregtime
  ed_out:
    code: ED_OUT
    time: edouttime

hosp/procedures_icd:
  procedure_icd:
    code: "PROCEDURE//ICD${icd_version}//${icd_code}"
    time: chartdate @ 11:59 p.m.
    seq_num: seq_num

icu/icustays:
  admission:
    code: "ICU_ADMISSION//${first_careunit}"
    time: intime
  discharge:
    code: "ICU_DISCHARGE//${last_careunit}"
    time: outtime

Part 2.5: `icu/chartevents`:

Finally, let's look at chartevents:

[8]:

dfs['icu/chartevents'].head(2)

[8]:

	subject_id	hadm_id	stay_id	caregiver_id	charttime	storetime	itemid	value	valuenum	valueuom	warning
0	10005817	20626031	32604416	6770.0	2132-12-16 00:00:00	2132-12-15 23:45:00	225054	On	NaN	NaN	0.0
1	10005817	20626031	32604416	6770.0	2132-12-16 00:00:00	2132-12-15 23:43:00	223769	100	100.0	%	0.0

This table clearly has rows that capture a variety of recordings of some more nuanced measurements. Some have numerical results, units of measure, etc. We also have another complexity here in that we have some uncertainty in timestamp, with both charttime and storetime being included. Ultimately, though, there is still just one kind of measurement being recorded here: Namely, a "chart event" (often a lab test), identified via the "Item ID" itemid being recorded at either charttime or storetime, with a value given by the columns within value, valuenum, and valueuom. Let's see how to add that to our specification (for brevity, we'll just show the new bit first, before we put it all together):

icu/chartevents:
  chartevent:
    time: charttime
    code: "CHARTEVENT//${itemid}//${valueuom}"
    numeric_value: valuenum

Note here that we've made a few assumptions:

We've defaulted to favor charttime here -- this is because, according to the data documentation, charttime is the closest proxy to when the data was actually recorded. However, this could benefit from further investigation and empirical validation!
We are omitting the warning column -- this is because we don't know when a warning would actually have been noted by the care-team, as it does not represent an automated process as part of the chart event measurement, but rather is a manual observation by the care team after the data has been recorded

In addition, valueuom is empty for most chart events (it is only recorded for measurements with a unit), so we'll need to decide what the code should be when it is missing. We'll come back to this when we write the final file.

All told, this gives us a final "specification" for the data extraction (at a conceptual level) as follows:

subject_id: subject_id
hosp/patients:
  gender:
    code: gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions
    code: MEDS_DEATH
    time: dod @ 11:59 p.m.
  birth:
    code: MEDS_BIRTH
    time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.

hosp/admissions:
  admission:
    code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
    time: admittime
  language:
    code: "LANGUAGE//${language}"
    time: admittime
  marital_status:
    code: "MARITAL_STATUS//${marital_status}"
    time: admittime
  insurance:
    code: "INSURANCE//${insurance}"
    time: admittime
  race:
    code: "RACE//${race}"
    time: admittime
  discharge:
    code: "HOSPITAL_DISCHARGE//${discharge_location}"
    time: dischtime
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: deathtime
  ed_reg:
    code: ED_REGISTRATION
    time: edregtime
  ed_out:
    code: ED_OUT
    time: edouttime

hosp/procedures_icd:
  procedure_icd:
    code: "PROCEDURE//ICD${icd_version}//${icd_code}"
    time: chartdate @ 11:59 p.m.
    seq_num: seq_num

icu/icustays:
  admission:
    code: "ICU_ADMISSION//${first_careunit}"
    time: intime
  discharge:
    code: "ICU_DISCHARGE//${last_careunit}"
    time: outtime

icu/chartevents:
  chartevent:
    time: charttime
    code: "CHARTEVENT//${itemid}//${valueuom}"
    numeric_value: valuenum

Then, our question becomes, how can we use this model to actually extract the data?

Part 3: Using MEDS-Extract to Automate Extraction

So far, all we've built up is a conceptual map on how to think about extracting data to MEDS. Hopefully, in doing so, you've come to see how the simplicity of MEDS gives rise to likewise simple extraction pipelines -- rather than requiring hours or days to understand the various input files, you can often map the rows of input tables into a conceptual specification for MEDS extraction in minutes, even when presented with more complex cases that require some assumptions to be made.

However, as it turns out, not only is this conceptual specification useful theoretically, it also is very close to a precise technical specification that the MEDS-Extract package can use to extract your data in the MEDS format for you.

The MEDS-Extract library leverages MEDS-Transforms to run a full ETL pipeline, with the secret sauce in the middle being the "MEDS-Extract Specification Syntax YAML" (MESSY) file -- which tells you how to map your messy input data into the MEDS format in alignment with this conceptual model.

This file is (as the name implies) in the YAML format and looks much like our specification above. It consists of blocks mapping input source table name to named measurements within the rows of that table, each measurement block having some sentinel properties which map to a prescribed extraction syntax that controls how the input data is parsed. Every value in a MESSY file is an expression in dftly, a small language for column references, string interpolation, type casts and arithmetic, so nearly everything in our conceptual specification carries over directly; we mostly just need to adjust the syntax. Let's dig in!

Note: this notebook is written for MEDS-Extract 0.8.1. The MESSY syntax changed in version 0.7 (before it, columns were written col(...), and codes were built from lists of parts), so if you're reading an older MESSY file, it may look a bit different.

The MESSY File Format

1. The Outer Structure

First, much like our conceptual specification above, the MESSY file will have a block per input source, within which we'll go through and identify all the measurements we want to extract from that source. In this case, that means we'll have a block for each of the tables we've listed above:

hosp/patients:
  ...
hosp/admissions:
  ...
hosp/procedures_icd:
  ...
icu/icustays:
  ...
icu/chartevents:
  ...

Also, much like our specification above, we can specify shared properties for all tables -- such as which column holds the subject_id. In the MESSY format, these go in a _defaults block, where $subject_id refers to the subject_id column (more on the $ below):

_defaults:
  subject_id: $subject_id
hosp/patients:
  ...
hosp/admissions:
  ...
hosp/procedures_icd:
  ...
icu/icustays:
  ...
icu/chartevents:
  ...

In fact, MEDS-Extract reads the subject ID from a column named subject_id by default, which is exactly what MIMIC-IV uses, so we can leave this block out. (A table whose subject ID is in a different column can set its own _defaults block.)

2. Measurement blocks

Within each table source, we also need to specify all of the measurements we want to extract. Again, our format will look pretty similar, but a bit different. Our conceptual specification had measurements that looked like each of the following prototypical examples:

gender:
  code: gender
  time: null
death:
  code: MEDS_DEATH
  time: dod @ 11:59 p.m.
birth:
  code: MEDS_BIRTH
  time: anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.
admission:
  code: "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
  time: admittime
death:
  code: MEDS_DEATH
  time: deathtime
procedure_icd:
  code: "PROCEDURE//ICD${icd_version}//${icd_code}"
  time: chartdate @ 11:59 p.m.
  seq_num: seq_num
chartevent:
  time: charttime
  code: "CHARTEVENT//${itemid}//${valueuom}"
  numeric_value: valuenum

Let's walk through each to see which features we'll need to change:

Specifying Time Format Strings

A key missing piece here is that we've indicated some strings are "time" columns, but we're not saying how those should be parsed from the (string) input types accessible in our CSV files! While this is not an issue if our inputs were parquets or something else with typed timestamp columns, for CSVs we need to address it. Luckily, this is simple; we can cast a column to a datetime with a format string, using the as operator; e.g., $admittime as "%Y-%m-%d %H:%M:%S". Refer to the chrono crate documentation for how these format strings should be specified. In this case, we want the format string "%Y-%m-%d %H:%M:%S" for most columns.

What if a value doesn't match the format? By default, the cast is strict: the extraction stops with an error, so malformed timestamps can't slip through unnoticed. And what if a column isn't so nicely formatted, and there are multiple format strings in the data? Then you can parse it leniently with each format (a ? before the format string turns a failed parse into a null), and take the first that works with coalesce; e.g., coalesce($charttime as ?"%Y-%m-%d %H:%M:%S", $charttime as ?"%Y-%m-%d").

We'll omit this added detail from our measurement configs for now in the interest of brevity, but see it added in at the end.

Disambiguating column references from string literals

We can see in the gender and death measurements that, in our conceptual specification, we sometimes used strings to refer to column names and sometimes as string literals. The MESSY file disambiguates these by prefixing column references with $: $gender is the value of the gender column, while a bare word like MEDS_DEATH is a string literal. This holds for every value in a measurement block, including extra columns like seq_num and numeric_value. So, we'll need to make some changes to these blocks to account for this (note that as we're making changes iteratively, they won't be fully valid until we're done). In some cases, it isn't clear how to make the change we're describing, so we'll add ??? indicators to those cases.

gender: # We're all done with this format -- this block is complete!
  code: $gender
  time: null
death:
  code: MEDS_DEATH
  time: ??? # dod @ 11:59 p.m.
birth:
  code: MEDS_BIRTH
  time: ??? # anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.
admission:
  code: ??? # "HOSPITAL_ADMISSION//${admission_type}//${admission_location}"
  time: $admittime
death: # We're all done with this format -- this block is complete!
  code: MEDS_DEATH
  time: $deathtime
procedure_icd:
  code: ??? # "PROCEDURE//ICD${icd_version}//${icd_code}"
  time: ??? # chartdate @ 11:59 p.m.
  seq_num: $seq_num
chartevent:
  code: ??? # "CHARTEVENT//${itemid}//${valueuom}"
  time: $charttime
  numeric_value: $valuenum

Watch out for a missing $: it isn't an error, so code: gender would quietly give every subject the code gender!

Note that we can actually now see that in some cases, resolving this piece has "completed" a full block; gender and death are feature complete now, and can be omitted from the later sections for our tutorial pieces.

String interpolation in the code column

Another feature we see a lot of is string interpolation in the code column; e.g., CHARTEVENT//${itemid}//${valueuom}. How can we handle that?

MEDS-Extract supports this directly, with Python-style f-strings: each column (with its $) goes inside braces; e.g., f"CHARTEVENT//{$itemid}//{$valueuom}". Let's see it in action!

death:
  code: MEDS_DEATH
  time: ??? # dod @ 11:59 p.m.
birth:
  code: MEDS_BIRTH
  time: ??? # anchor_year - anchor_age @ Jan 1, 12:00:01 a.m.
admission: # We're all done with this format -- this block is complete!
  code: f"HOSPITAL_ADMISSION//{$admission_type}//{$admission_location}"
  time: $admittime
procedure_icd:
  code: f"PROCEDURE//ICD//{$icd_version}//{$icd_code}"
  time: ??? # chartdate @ 11:59 p.m.
  seq_num: $seq_num
chartevent: # We're all done with this format -- this block is complete!
  code: f"CHARTEVENT//{$itemid}//{$valueuom}"
  time: $charttime
  numeric_value: $valuenum

For the procedure codes, we've written ICD//{$icd_version} rather than the ICD${icd_version} of our conceptual specification. Both would work, but separating the version with // matches the MIMIC-IV ETL, so our codes look like those in the datasets you'll use in the rest of the tutorial.

There is one catch with interpolation: if any column in an f-string is empty for a given row, the whole code is null, and as a MEDS code can never be null, that row is dropped. This matters here, as discharge_location and valueuom are sometimes empty. To keep those rows, we can give a column a fallback with the ?? operator; e.g., {$valueuom ?? 'UNK'} uses UNK ("unknown") wherever valueuom is empty. We'll add these fallbacks in our final file. Whenever rows are dropped, MEDS-Extract logs a warning saying how many, so a missing fallback won't go unnoticed.

Timestamp Resolution and basic arithmetic

Now, we have the last two problems: (a) we need to resolve a date to a specific time of day (e.g., chartdate @ 11:59 p.m.), and (b) we need to perform some simple arithmetic (e.g., anchor_year - anchor_age).

MEDS-Extract can express both almost exactly as our conceptual specification does:

The @ operator sets the time of day of a date: $chartdate as "%Y-%m-%d" @ 11:59:59 p.m. parses chartdate as a date and puts it at 11:59:59 p.m. on that day.
Arithmetic works as you'd expect, and casting a number as year gives January 1 of that year (at midnight, rather than the 12:00:01 a.m. of our conceptual specification, which makes no difference here): ($anchor_year - $anchor_age) as year.

death:
  code: MEDS_DEATH
  time: $dod as "%Y-%m-%d" @ 11:59:59 p.m.
birth:
  code: MEDS_BIRTH
  time: ($anchor_year - $anchor_age) as year
procedure_icd:
  code: f"PROCEDURE//ICD//{$icd_version}//{$icd_code}"
  time: $chartdate as "%Y-%m-%d" @ 11:59:59 p.m.
  seq_num: $seq_num

That leaves one part of our conceptual specification: the death measurement in hosp/patients should be superseded by the one in hosp/admissions whenever a subject has a deathtime. Deciding that for a row of hosp/patients means looking at hosp/admissions, i.e., combining two tables. MEDS-Extract can join tables (see its documentation on joins; the MIMIC-IV ETL uses one for exactly this), but to keep things simple, we'll do it in a "pre-MEDS" step instead, where we have some custom code go through and modify the raw dataframes before we call MEDS-Extract. A pre-MEDS step is a handy place for anything that's easier to write in code than in the MESSY file, such as more involved data cleaning or filtering (though often data cleaning can happen after the MEDS conversion process as well).

Let's write a simple pre-MEDS step we can run here.

Pre-MEDS

Our Pre-MEDS step has a single goal: remove the duplication between the dod column in hosp/patients and the deathtime in hosp/admissions, to favor the latter where both are specified. We'll do this by clearing dod for every subject with a deathtime.

We'll write this using pandas for now, but you can use whatever method you want for your data.

[9]:

def remove_dod_of_in_hospital_deaths(
    patients_df: pd.DataFrame,
    admissions_df: pd.DataFrame,
) -> pd.DataFrame:
  subjects_with_deathtime = (
      admissions_df[~admissions_df["deathtime"].isna()]["subject_id"]
  )

  idx = patients_df["subject_id"].isin(subjects_with_deathtime)
  patients_df.loc[idx, "dod"] = None
  return patients_df

We'll store the output of our pre-MEDS stage in an "intermediate directory" called intermediate_dir -- that way we can always re-use our raw data.

[10]:

INTERMEDIATE_DIR = Path("intermediate_dir")

for name, df in dfs.items():
  if name == "hosp/patients":
    df = remove_dod_of_in_hospital_deaths(df, dfs["hosp/admissions"])

  out_fp = INTERMEDIATE_DIR / f"{name}.parquet"
  out_fp.parent.mkdir(parents=True, exist_ok=True)
  df.to_parquet(out_fp)

[11]:

1 
2

%%bash
tree intermediate_dir

intermediate_dir
├── hosp
│   ├── admissions.parquet
│   ├── patients.parquet
│   └── procedures_icd.parquet
└── icu
    ├── chartevents.parquet
    └── icustays.parquet

2 directories, 5 files

[12]:

pd.read_parquet(INTERMEDIATE_DIR / "hosp/patients.parquet").head(5)

[12]:

	subject_id	gender	anchor_age	anchor_year	anchor_year_group	dod
0	10014729	F	21	2125	2011 - 2013	NaN
1	10003400	F	72	2134	2011 - 2013	NaN
2	10002428	F	80	2155	2011 - 2013	NaN
3	10032725	F	38	2143	2011 - 2013	2143-03-30
4	10027445	F	48	2142	2011 - 2013	2146-02-09

[13]:

1 
2
3

n_raw = pd.read_csv(DATA_ROOT / "hosp/patients.csv.gz")["dod"].notna().sum()
n_kept = pd.read_parquet(INTERMEDIATE_DIR / "hosp/patients.parquet")["dod"].notna().sum()
print(f"Subjects with a `dod`: {n_raw} in the raw data, {n_kept} after the pre-MEDS step")

Subjects with a `dod`: 31 in the raw data, 16 after the pre-MEDS step

The final MESSY File

Now that we've resolved our remaining issues, let's put together our final, complete MESSY file! Besides the blocks above, it adds the time formats and the ?? fallbacks we discussed, and an etl block, which names the dataset and records the version of the raw data it is built from (version 2.2 of the MIMIC-IV demo) in the dataset's metadata.

[14]:

YAML_contents = """
etl:
  dataset_name: ML4H_Tutorial
  raw_dataset_version: '2.2'

hosp/patients:
  gender:
    code: $gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions (see the pre-MEDS step)
    code: MEDS_DEATH
    time: '$dod as "%Y-%m-%d" @ 11:59:59 p.m.'
  birth:
    code: MEDS_BIRTH
    time: ($anchor_year - $anchor_age) as year

hosp/admissions:
  admission:
    code: f"HOSPITAL_ADMISSION//{$admission_type}//{$admission_location}"
    time: '$admittime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  discharge:
    code: f"HOSPITAL_DISCHARGE//{$discharge_location ?? 'UNK'}"
    time: '$dischtime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: '$deathtime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  ed_registration:
    code: ED_REGISTRATION
    time: '$edregtime as "%Y-%m-%d %H:%M:%S"'
  ed_out:
    code: ED_OUT
    time: '$edouttime as "%Y-%m-%d %H:%M:%S"'

hosp/procedures_icd:
  procedure_icd:
    code: f"PROCEDURE//ICD//{$icd_version}//{$icd_code}"
    time: '$chartdate as "%Y-%m-%d" @ 11:59:59 p.m.'
    seq_num: $seq_num

icu/icustays:
  admission:
    code: f"ICU_ADMISSION//{$first_careunit}"
    time: '$intime as "%Y-%m-%d %H:%M:%S"'
  discharge:
    code: f"ICU_DISCHARGE//{$last_careunit}"
    time: '$outtime as "%Y-%m-%d %H:%M:%S"'

icu/chartevents:
  chartevent:
    code: f"CHARTEVENT//{$itemid}//{$valueuom ?? 'UNK'}"
    time: '$charttime as "%Y-%m-%d %H:%M:%S"'
    numeric_value: $valuenum
"""

YAML_fp = Path("MESSY.yaml")
YAML_fp.write_text(YAML_contents)
print(YAML_fp.read_text())


etl:
  dataset_name: ML4H_Tutorial
  raw_dataset_version: '2.2'

hosp/patients:
  gender:
    code: $gender
    time: null
  death: # Superseded by the `death` measurement in hosp/admissions (see the pre-MEDS step)
    code: MEDS_DEATH
    time: '$dod as "%Y-%m-%d" @ 11:59:59 p.m.'
  birth:
    code: MEDS_BIRTH
    time: ($anchor_year - $anchor_age) as year

hosp/admissions:
  admission:
    code: f"HOSPITAL_ADMISSION//{$admission_type}//{$admission_location}"
    time: '$admittime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  discharge:
    code: f"HOSPITAL_DISCHARGE//{$discharge_location ?? 'UNK'}"
    time: '$dischtime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  death: # Takes precedence over the `death` measurement in hosp/patients
    code: MEDS_DEATH
    time: '$deathtime as "%Y-%m-%d %H:%M:%S"'
    hadm_id: $hadm_id
  ed_registration:
    code: ED_REGISTRATION
    time: '$edregtime as "%Y-%m-%d %H:%M:%S"'
  ed_out:
    code: ED_OUT
    time: '$edouttime as "%Y-%m-%d %H:%M:%S"'

hosp/procedures_icd:
  procedure_icd:
    code: f"PROCEDURE//ICD//{$icd_version}//{$icd_code}"
    time: '$chartdate as "%Y-%m-%d" @ 11:59:59 p.m.'
    seq_num: $seq_num

icu/icustays:
  admission:
    code: f"ICU_ADMISSION//{$first_careunit}"
    time: '$intime as "%Y-%m-%d %H:%M:%S"'
  discharge:
    code: f"ICU_DISCHARGE//{$last_careunit}"
    time: '$outtime as "%Y-%m-%d %H:%M:%S"'

icu/chartevents:
  chartevent:
    code: f"CHARTEVENT//{$itemid}//{$valueuom ?? 'UNK'}"
    time: '$charttime as "%Y-%m-%d %H:%M:%S"'
    numeric_value: $valuenum

Using the MESSY File -- how do you run MEDS-Extract?

With the MESSY file specified, running MEDS-Extract is easy. There are two steps. First, install the package. We pin the version this notebook was written for, as the MESSY syntax can change between versions:

[15]:

!pip --quiet install "MEDS-extract==0.8.1"

Next, run meds-extract-run. From just the MESSY file and where to read and write data, it runs the whole extraction pipeline, a series of MEDS-Transforms stages. (It can also download the raw data for you, from a sources block in the MESSY file; as we've already prepared our inputs in intermediate_dir, we turn that off with do_download=false.) Note the ./ before the file name: without it, spec= looks for a packaged ETL of that name, such as spec=MIMIC-IV.

[16]:

%%bash
meds-extract-run \
    spec=./MESSY.yaml \
    output_dir=output_dir \
    do_download=false \
    input_dir=intermediate_dir

[2026-11-11 18:11:16,774][MEDS_extract.config][INFO] - Reading MESSY config from /content/MESSY.yaml
[2026-11-11 18:11:16,777][MEDS_extract.config][INFO] - MESSY config:
etl:
  dataset_name: ML4H_Tutorial
  raw_dataset_version: '2.2'
hosp/patients:
  gender:
    code: $gender
    time: null
  death:
    code: MEDS_DEATH
    time: $dod as "%Y-%m-%d" @ 11:59:59 p.m.
  birth:
    code: MEDS_BIRTH
    time: ($anchor_year - $anchor_age) as year
hosp/admissions:
  admission:
    code: f"HOSPITAL_ADMISSION//{$admission_type}//{$admission_location}"
    time: $admittime as "%Y-%m-%d %H:%M:%S"
    hadm_id: $hadm_id
  discharge:
    code: f"HOSPITAL_DISCHARGE//{$discharge_location ?? 'UNK'}"
    time: $dischtime as "%Y-%m-%d %H:%M:%S"
    hadm_id: $hadm_id
  death:
    code: MEDS_DEATH
    time: $deathtime as "%Y-%m-%d %H:%M:%S"
    hadm_id: $hadm_id
  ed_registration:
    code: ED_REGISTRATION
    time: $edregtime as "%Y-%m-%d %H:%M:%S"
  ed_out:
    code: ED_OUT
    time: $edouttime as "%Y-%m-%d %H:%M:%S"
hosp/procedures_icd:
  procedure_icd:
    code: f"PROCEDURE//ICD//{$icd_version}//{$icd_code}"
    time: $chartdate as "%Y-%m-%d" @ 11:59:59 p.m.
    seq_num: $seq_num
icu/icustays:
  admission:
    code: f"ICU_ADMISSION//{$first_careunit}"
    time: $intime as "%Y-%m-%d %H:%M:%S"
  discharge:
    code: f"ICU_DISCHARGE//{$last_careunit}"
    time: $outtime as "%Y-%m-%d %H:%M:%S"
icu/chartevents:
  chartevent:
    code: f"CHARTEVENT//{$itemid}//{$valueuom ?? 'UNK'}"
    time: $charttime as "%Y-%m-%d %H:%M:%S"
    numeric_value: $valuenum

[2026-11-11 18:11:16,795][MEDS_extract.run.cli][INFO] - Resolved spec='./MESSY.yaml' to /content/MESSY.yaml
[2026-11-11 18:11:16,795][MEDS_extract.run.cli][INFO] - do_download=false: skipping the download stage.
[2026-11-11 18:11:16,796][MEDS_extract.run.cli][INFO] - Wrote synthesized pipeline config to /content/output_dir/.meds_extract_run/pipeline.yaml
[2026-11-11 18:11:16,796][MEDS_extract.run.cli][INFO] - Running: ['/usr/bin/python3', '-m', 'MEDS_transforms.runner', '/content/output_dir/.meds_extract_run/pipeline.yaml']

If we've done things right, then we should see the above cell complete with no errors -- if we haven't, we'll need to debug. Thankfully, MEDS-Extract writes out some nice logs to help with this, which we can find in the output directory output_dir, under output_dir/.logs/pipeline.log. The full log is long, so let's just look at the stages it ran and any warnings:

[17]:

!grep -E "Running stage|WARNING" output_dir/.logs/pipeline.log

INFO:__main__:Running stage: convert_to_parquet
INFO:__main__:Running stage: split_and_shard_subjects
INFO:__main__:Running stage: convert_to_subject_sharded
INFO:__main__:Running stage: convert_to_MEDS_events
[2026-11-11 18:11:19,068][MEDS_extract.config][WARNING] - `hosp/admissions/death`: dropped 226/238 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,068][MEDS_extract.config][WARNING] - `hosp/admissions/ed_registration`: dropped 76/238 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,068][MEDS_extract.config][WARNING] - `hosp/admissions/ed_out`: dropped 76/238 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,075][MEDS_extract.config][WARNING] - `hosp/patients/death`: dropped 65/80 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,091][MEDS_extract.config][WARNING] - `hosp/patients/death`: dropped 9/10 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,094][MEDS_extract.config][WARNING] - `hosp/admissions/death`: dropped 15/17 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,094][MEDS_extract.config][WARNING] - `hosp/admissions/ed_registration`: dropped 10/17 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,095][MEDS_extract.config][WARNING] - `hosp/admissions/ed_out`: dropped 10/17 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,123][MEDS_extract.config][WARNING] - `hosp/admissions/death`: dropped 19/20 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,124][MEDS_extract.config][WARNING] - `hosp/admissions/ed_registration`: dropped 7/20 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,124][MEDS_extract.config][WARNING] - `hosp/admissions/ed_out`: dropped 7/20 rows with null time (unparsable or missing under the configured formats)
[2026-11-11 18:11:19,128][MEDS_extract.config][WARNING] - `hosp/patients/death`: dropped 10/10 rows with null time (unparsable or missing under the configured formats)
INFO:__main__:Running stage: extract_code_metadata
INFO:__main__:Running stage: merge_to_MEDS_cohort
INFO:__main__:Running stage: finalize_MEDS_metadata
INFO:__main__:Running stage: finalize_MEDS_data

The warnings say how many rows each measurement block dropped because their time was missing; they're reported for each shard of subjects, which is why a block can appear several times. These are expected: most subjects have no dod or deathtime (and we cleared dod for those with a deathtime), and only admissions through the emergency department have ED times. If a block dropped all of its rows in every shard, though, that would be a sign that something (like a time format) is wrong!

What do the output files themselves actually look like? Let's see:

[18]:

1 
2

%%bash
tree output_dir

output_dir
├── convert_to_MEDS_events
│   ├── held_out
│   │   └── 0
│   │       ├── hosp
│   │       │   ├── admissions.parquet
│   │       │   ├── patients.parquet
│   │       │   └── procedures_icd.parquet
│   │       └── icu
│   │           ├── chartevents.parquet
│   │           └── icustays.parquet
│   ├── messy.yaml
│   ├── train
│   │   └── 0
│   │       ├── hosp
│   │       │   ├── admissions.parquet
│   │       │   ├── patients.parquet
│   │       │   └── procedures_icd.parquet
│   │       └── icu
│   │           ├── chartevents.parquet
│   │           └── icustays.parquet
│   └── tuning
│       └── 0
│           ├── hosp
│           │   ├── admissions.parquet
│           │   ├── patients.parquet
│           │   └── procedures_icd.parquet
│           └── icu
│               ├── chartevents.parquet
│               └── icustays.parquet
├── convert_to_parquet
│   ├── hosp
│   │   ├── admissions.parquet
│   │   ├── patients.parquet
│   │   └── procedures_icd.parquet
│   └── icu
│       ├── chartevents.parquet
│       └── icustays.parquet
├── convert_to_subject_sharded
│   ├── held_out
│   │   └── 0
│   │       ├── hosp
│   │       │   ├── admissions.parquet
│   │       │   ├── patients.parquet
│   │       │   └── procedures_icd.parquet
│   │       └── icu
│   │           ├── chartevents.parquet
│   │           └── icustays.parquet
│   ├── train
│   │   └── 0
│   │       ├── hosp
│   │       │   ├── admissions.parquet
│   │       │   ├── patients.parquet
│   │       │   └── procedures_icd.parquet
│   │       └── icu
│   │           ├── chartevents.parquet
│   │           └── icustays.parquet
│   └── tuning
│       └── 0
│           ├── hosp
│           │   ├── admissions.parquet
│           │   ├── patients.parquet
│           │   └── procedures_icd.parquet
│           └── icu
│               ├── chartevents.parquet
│               └── icustays.parquet
├── data
│   ├── held_out
│   │   └── 0.parquet
│   ├── train
│   │   └── 0.parquet
│   └── tuning
│       └── 0.parquet
├── extract_code_metadata
│   ├── codes.parquet
│   └── observed_codes
│       ├── held_out_0_hosp_admissions.parquet
│       ├── held_out_0_hosp_patients.parquet
│       ├── held_out_0_hosp_procedures_icd.parquet
│       ├── held_out_0_icu_chartevents.parquet
│       ├── held_out_0_icu_icustays.parquet
│       ├── train_0_hosp_admissions.parquet
│       ├── train_0_hosp_patients.parquet
│       ├── train_0_hosp_procedures_icd.parquet
│       ├── train_0_icu_chartevents.parquet
│       ├── train_0_icu_icustays.parquet
│       ├── tuning_0_hosp_admissions.parquet
│       ├── tuning_0_hosp_patients.parquet
│       ├── tuning_0_hosp_procedures_icd.parquet
│       ├── tuning_0_icu_chartevents.parquet
│       └── tuning_0_icu_icustays.parquet
├── finalize_MEDS_metadata
├── merge_to_MEDS_cohort
│   ├── held_out
│   │   └── 0.parquet
│   ├── train
│   │   └── 0.parquet
│   └── tuning
│       └── 0.parquet
├── metadata
│   ├── codes.parquet
│   ├── dataset.json
│   └── subject_splits.parquet
└── split_and_shard_subjects

42 directories, 61 files

There's a lot here -- thankfully, most of them are internal, partial outputs that MEDS-Extract writes so it can resume after failures on larger datasets. These aren't helpful for us, but are helpful when you're working with hundreds of thousands to billions of measurements!

To see just the final files, we can look in the data and metadata sub-folders:

[19]:

1 
2

%%bash
tree output_dir/data

output_dir/data
├── held_out
│   └── 0.parquet
├── train
│   └── 0.parquet
└── tuning
    └── 0.parquet

3 directories, 3 files

[20]:

1 
2

%%bash
tree output_dir/metadata

output_dir/metadata
├── codes.parquet
├── dataset.json
└── subject_splits.parquet

0 directories, 3 files

Going Forward

While you've just built a great MEDS dataset over the MIMIC demo dataset in this tutorial, you've only looked at a small set of the included files we showed above. In the rest of the tutorial, we'll use the full MIMIC demo dataset, which we'll download as needed from physionet in the other notebooks, rather than the output of this notebook. That dataset is built by the dedicated MIMIC-IV ETL Package, whose analogous MESSY file covers all the sources. It has the same structure and syntax as the file you built here (one block per source table, one block per measurement). A few differences are worth knowing about:

Demographics. To keep it short, our final MESSY file left out the insurance, language, marital_status and race measurements from our conceptual specification. The current MIMIC-IV ETL extracts them just as our conceptual specification does: as separate INSURANCE//..., LANGUAGE//..., MARITAL_STATUS//... and RACE//... measurements at the time of the admission (see the Demographics section of its README).
The pre-built demo uses an older version of the ETL. The MEDS demo dataset that the other notebooks download (version 0.0.1 on PhysioNet) was built before that change, so it keeps these four values as extra columns (insurance, language, marital_status and race) on each HOSPITAL_ADMISSION//... measurement instead. In the later notebooks, you'll find the demographics in those columns, not as codes.
Deaths. Like our file, the MIMIC-IV ETL favors the in-hospital deathtime over the date-only dod, but rather than using a pre-MEDS step, it joins deathtime from hosp/admissions into hosp/patients in its MESSY file and emits a single death event there.

Additional Details and Resources

You can also check out MEDS-Extract's documentation and another example on synthetic data via the included links as well!

Even more importantly, what if you don't like MEDS-Extract and don't want to use it? Then don't! The three guiding questions of the extraction process (What is happening?, To whom is it happening?, and When is it happening?) can be turned into an extraction pipeline in whatever way you like -- the MEDS ecosystem is designed to be data-centric, so it doesn't matter how you got to a MEDS dataset, just that you did, and then tools can run from there!

[20]: