Now Reading: Defining and benchmarking open problems in single-cell analysis
-
01
Defining and benchmarking open problems in single-cell analysis
Code availability
All Open Problems code is publicly available at https://www.github.com/openproblems-bio/openproblems. This code includes data loaders for all datasets used, with associated metadata on where this data came from. Code to reproduce the figures is publicly available at https://github.com/openproblems-bio/nbt2025-manuscript. Detailed information on all datasets is available at https://openproblems.bio/datasets. Documentation for the platform and contribution guides can be found at https://openproblems.bio/documentation.
References
-
Zappia, L., Phipson, B. & Oshlack, A. PLOS Comput. Biol. 14, e1006245 (2018).
-
Heumos, L. et al. Nat. Rev. Genet. 24, 550–572 (2023).
-
Luecken, M. D. & Theis, F. J. Mol. Syst. Biol. 15, e8746 (2019).
-
Donoho, D. J. Comput. Graph. Stat. 26, 745–766 (2017).
-
Sonrel, A. et al. Genome Biol. 24, 119 (2023).
-
Brooks, T. G., Lahens, N. F., Mrčela, A. & Grant, G. R. Nat. Rev. Genet. 25, 326–339 (2024).
-
Buchka, S., Hapfelmeier, A., Gardner, P. P., Wilson, R. & Boulesteix, A.-L. Genome Biol. 22, 152 (2021).
-
Musgrave, K., Belongie, S. & Lim, S.-N. In Computer Vision – ECCV 2020 (eds Vedaldi, A. et al.) Lecture Notes in Computer Science Vol. 12370 (Springer, 2020); https://doi.org/10.1007/978-3-030-58595-2_41
-
Luecken, M. D. et al. Nat. Methods 19, 41–50 (2022).
-
Chazarra-Gil, R., van Dongen, S., Kiselev, V. Y. & Hemberg, M. Nucleic Acids Res. 49, e42 (2021).
-
Tran, H. T. N. et al. Genome Biol. 21, 12 (2020).
-
Mereu, E. et al. Nat. Biotechnol. 38, 747–755 (2020).
-
Cao, Y. et al. Preprint at bioRxiv https://doi.org/10.1101/2023.12.19.572303 (2025).
-
Cannoodt, R. et al. J. Open Source Softw. 9, 6089 (2024).
-
CZI Cell Science Program et al. Nucleic Acids Res. 53, D886–D900 (2025).
-
Dimitrov, D. et al. Nat. Commun. 13, 3224 (2022).
-
Armingol, E., Baghdassarian, H. M. & Lewis, N. E. Nat. Rev. Genet. 25, 381–400 (2024).
-
Efremova, M., Vento-Tormo, M., Teichmann, S. A. & Vento-Tormo, R. Nat. Protoc. 15, 1484–1506 (2020).
-
Hou, R., Denisenko, E., Ong, H. T., Ramilowski, J. A. & Forrest, A. R. R. Nat. Commun. 11, 5011 (2020).
-
Raredon, M. S. B. et al. Sci. Rep. 12, 4187 (2022).
-
Cabello-Aguilar, S. et al. Nucleic Acids Res. 48, e55 (2020).
-
Lance, C. et al. In Proc. NeurIPS 2021 Competitions and Demonstrations Track 162–176 (NeurIPS, 2022).
-
Luecken, M. D. et al. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS, 2021); https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/158f3069a435b314a80bdcb024f8e422-Abstract-round2.html
-
Gigante, S. et al. Openproblems-Bio/Openproblems: V1.0.0. Zenodo https://doi.org/10.5281/ZENODO.13769879 (2024).
Acknowledgements
We received continual support in many ways from Jonah Cool, Ivana Williams and Fiona Griffin from the Chan Zuckerberg Initiative for this project, without whom we would not have come this far. We would also like to thank Mohammad Lotfollahi for early discussions on Open Problems. E.V.B. would like to thank the Caltech Bioengineering Graduate program and Paul W. Sternberg for support. This work was supported by the Chan Zuckerberg Initiative Foundation (grant CZIF2022-007488, Human Cell Atlas Data Ecosystem) and the Chan Zuckerberg Initiative DAF, an advised fund of the Silicon Valley Community Foundation (grant number 2021-235155) awarded to M.D.L., D.B.B., S.G., F.J.T. and S.K. This work was co-funded by the European Union (ERC, DeepCell -101054957, to A.S. and F.J.T.). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. G.P. is supported by the Helmholtz Association under the joint research school Munich School for Data Science and by the Joachim Herz Foundation. Throughout this work, W.L. was supported by the US National Institutes of Health under Continuing Education Training Grants (T15). D.D. was supported by the European Union’s Horizon 2020 Research and Innovation Program (860329 Marie-Curie ITN “STRATEGY-CKD”). M.E.V. is supported by the US National Institutes of Health under a Ruth L. Kirschstein National Research Service Award (1F31CA257625) from the National Cancer Institute. E.D. is supported by Wellcome Sanger core funding (WT206194). This work was supported by the Research Foundation Flanders (FWO) (1SF3822N to L.D.). B.R. is supported by the Bavarian state government with funds from the Hightech Agenda Bavaria. This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen” programme. C.B.G.-B. was supported by a PhD fellowship from Fonds Wetenschappelijk Onderzoek (FWO, 11F1519N). V.K. was supported by Wellcome Sanger core funding. G.L.M. received support from Swiss National Science Foundation grant PZ00P3_193445 and Chan Zuckerberg Initiative grants number 2022-249212 and 2019-002427. D.R. was supported by the National Cancer Institute of the US National Institutes of Health (2U24CA180996).
Ethics declarations
Competing interests
M.D.L. consults for CatalYm GmbH, contracted for the Chan Zuckerberg Initiative and received speaker fees from Pfizer and Janssen Pharmaceuticals. S.G. has equity interest in Immunai Inc. D.B.B. is a paid employee of and has equity interest in NVIDIA. R.C. has equity interest in Data Intuitive BV. L.Z. has consulted for Lamin Labs GmbH. W.L. contracted for Protein Evolution Incorporated. From 2019 to 2022, A.A. was a consultant for 10x Genomics. From October 2023, E.D. has been a consultant for EnsoCell Therapeutics. O.B.B is currently an employee of Bridge Bio Pharma. A.S. consults for Cellarity Inc. and Exvivo Labs Inc. A.B. is a paid employee of and has equity interest in Cellarity, Inc. J.B. has equity interest in Cellarity, Inc. J.S.-R. reports funding from GSK, Pfizer and Sanofi and fees or honoraria from Travere Therapeutics, Stadapharm, Astex, Owkin, Pfizer and Grunenthal. D.W. has equity interest in Immunai Inc. F.J.T. consults for Immunai Inc., Singularity Bio B.V., CytoReason Ltd and Cellarity, and has ownership interest in Dermagnostix GmbH and Cellarity. S.K. is a visiting professor at Meta and scientific advisor at Ascent Bio, Inc. E.d.V.B has ownership interest in Retro Biosciences and ImYoo Inc and is employed by ImYoo Inc. A.T.C. is an employee of Orion Medicines. B.D. is a paid employee of and has equity interest in Cellarity Inc. A.G. is currently an employee of Google DeepMind. Google DeepMind has not directed any aspect of this study nor exerts any commercial rights over the results. R.L. is an employee of Genentech. V.S. has ownership interest in Altos Labs and Vesalius Therapeutics. A.T. has an ownership interest in Dreamfold.
Supplementary information
Source data
Source Data Fig. 1
Methods and metrics used per existing benchmarking repository, including dates of first and last commit.
Source Data Fig. 2
Table of metric results for the cell–cell communication task with metric explanations.
Rights and permissions
About this article
Cite this article
Luecken, M.D., Gigante, S., Burkhardt, D.B. et al. Defining and benchmarking open problems in single-cell analysis.
Nat Biotechnol (2025). https://doi.org/10.1038/s41587-025-02694-w
-
Published:
-
DOI: https://doi.org/10.1038/s41587-025-02694-w
Related Posts
Stay Informed With the Latest & Most Important News
Previous Post
Next Post
Advertisement
