Collaborate
MEDS-DEV is a decentralized benchmark: the community contributes datasets, prediction tasks and model training recipes, and experiments on each dataset are run by the people who have access to it. If you own a dataset, a task or a model, we would like to work with you to bring it into MEDS-DEV.
Collaborating makes your work part of a reproducible benchmark. Tasks are shared as ACES configurations and models as training recipes that, in principle, transfer to any MEDS-compliant dataset, so results can be reproduced and compared. Contributions are listed on the datasets, tasks and models pages, and submitted results appear in the benchmark. Along the way, you can use the MEDS-DEV command-line helpers and the MEDS ecosystem to build datasets, extract tasks, train models and evaluate predictions.
- Dataset owners describe their dataset and add the commands that convert it to MEDS, plus the ACES predicates that MEDS-DEV tasks need. Adding a dataset does not require releasing the data: MEDS-DEV stores configuration files and results, not the data itself. You can then run MEDS-DEV tasks and models on your data locally and contribute the results.
- Task owners add an ACES task configuration with a description. A task can be added before any dataset supports it, so the community can comment on and refine the definition. MEDS-DEV currently supports binary classification tasks only.
- Model owners add a training recipe: the commands that train the model from scratch on a MEDS dataset and write its predictions for evaluation. MEDS-DEV currently compares training recipes rather than pre-trained weights, so others can train and evaluate your model on their own data.
Contributions are made as pull requests to the MEDS-DEV repository, whose README lists the files each one needs. If you already know what you want to add, you can open an issue from one of the MEDS-DEV issue templates. Otherwise, use the form below to start the conversation.
Send a collaboration request
Tick every role that applies to you, for example both dataset owner and model owner. This page does not store your answers or ask for contact details: it opens your request as a pre-filled issue on the public MEDS-DEV GitHub repository, where you can review and post it.