Reproducible Stochastic Simulation

Stochastic simulations are commonly used to quantitatively or semi-quantitatively describe the dynamics of biological systems. At various scales and in multiple applications, stochastic simulation better reflects observed biological processes and robustness. Various methods are widely used to incorporate stochasticity into biological simulation, such as the Gillespie stochastic simulation algorithm for systems biology modeling, stochastic Boolean networks for network modeling, and the Cellular Potts model methodology for multicellular modeling. Proving reproducibility of simulation results is critical to establishing the credibility of a model. To this end, BioModels, the largest repository of curated mathematical models, tests and reports the reproducibility of simulation results for all submitted models when possible. A recent study showed that about 50% of the deterministic ordinary differential equation models on BioModels could not be reproduced when applying criteria for reproducibility to the information provided in their associated publication, reflecting a current crisis of reproducibility. Furthermore, there are no well-accepted metrics or standards for reproducing stochastic simulation results, thus perpetuating the crisis of reproducibility for a broad class of biological models. This talk surveys recent progress to establish an accepted framework for testing the reproducibility of stochastic simulations in biological modeling. The talk provides an overview of the mathematical and statistical foundations for defining quantitative measures to determine whether stochastic simulation results can be reproduced, and when results have been reproduced. Developed information structures for community-level reproducibility are briefly surveyed, outlining the lifetime of a reproducible stochastic model, from publication to curation. Results from select test cases using various modeling methodologies are presented, including stochastic differential equations, agent-based models, and stochastic Boolean networks, collectively demonstrating the broad relevance of newly developed approaches to support reproducible stochastic simulation at scale in multiple biological and biomedical research domains. The talk concludes with a discussion of current challenges and future work, including opportunities in mathematics, statistics, and software development.