2018

Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsis

Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S. et al.

Understand

Sepsis is a life-threatening condition affecting one million people per year in the US in which dysregulation of the body's own immune system causes damage to its tissues, resulting in a 28 - 50% mortality rate.

Clinical trials for sepsis treatment over the last 20 years have failed to produce a single currently FDA approved drug treatment.
In this study, we attempt to discover an effective cytokine mediation treatment strategy for sepsis using a previously developed agent-based model that simulates the innate immune response to infection: the Innate Immune Response agent-based model (IIRABM).
Previous attempts at reducing mortality with multi-cytokine mediation using the IIRABM have failed to reduce mortality across all patient parameterizations and motivated us to investigate whether adaptive, personalized multi-cytokine mediation can control the trajectory of sepsis and lower patient mortality.

Built on

Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 1998
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
Andrew Y Ng, Daishi Harada, and Stuart Russell · 1999
Earlier work this paper cites.
Clinical trials of mediator-directed therapy in sepsis: what have we learned?
JC Marshall · 2000
Earlier work this paper cites.
Anti-tumor necrosis factor therapy in sepsis: update on clinical trials and lessons learned
Konrad Reinhart and Waheedullah Karzai · 2001
Earlier work this paper cites.
Risk and the efficacy of antiinflammatory agents: retrospective and confirmatory studies of sepsis
Peter Q Eichacker, Chantal Parent, Andre Kalil, Claire Esposito, Xizhong Cui, Steven M Banks, Eric P Gerstenberger, Yvonne Fitz, Robert L Danner, and Charles Natanson · 2002
Earlier work this paper cites.
Cytokine modulation in sepsis and septic shock
Sergio Zanotti, Anand Kumar, and Aseem Kumar · 2002
Earlier work this paper cites.
Such stuff as dreams are made on: mediator-directed therapy in sepsis
John C Marshall · 2003
Earlier work this paper cites.
Building hybrid systems with boost. python
David Abrahams and Ralf W Grosse-Kunstleve · 2003
Earlier work this paper cites.
Pharmacoeconomic implications of new therapies in sepsis
Kelly A Wood and Derek C Angus · 2004
Earlier work this paper cites.
In silico experiments of existing and hypothetical cytokine-directed clinical trials using agent-based modeling
Gary An · 2004
Earlier work this paper cites.
Tree-based batch mode reinforcement learning
Damien Ernst, Pierre Geurts, and Louis Wehenkel · 2005
Earlier work this paper cites.
An enhanced agent based model of the immune system response
V Baldazzi, F Castiglione, and M Bernaschi · 2006
Earlier work this paper cites.
Multi-cell agent-based simulation of the microvasculature to study the dynamics of circulating inflammatory cell trafficking
Alexander M Bailey, Bryan C Thorne, and Shayn M Peirce · 2007
Earlier work this paper cites.
Development of a three-dimensional multiscale agent-based tumor model: simulating gene-protein interaction profiles, cell phenotypes and multicellular patterns in brain cancer
Le Zhang, Chaitanya A Athale, and Thomas S Deisboeck · 2007
Earlier work this paper cites.

Similar

Agent-based models in translational systems biology
Gary An, Qi Mi, Joyeeta Dutta-Moscato, and Yoram Vodovotz · 2009
Cited alongside, same era.
Agent-based modeling of host–pathogen systems: The successes and challenges
Amy L Bauer, Catherine AA Beauchemin, and Alan S Perelson · 2009
Cited alongside, same era.
Theoretical and empirical analysis of reward shaping in reinforcement learning
Marek Grzes and Daniel Kudenko · 2009
Cited alongside, same era.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Cited alongside, same era.
Agent-based modeling: a systematic assessment of use cases and requirements for enhancing pharmaceutical research and development productivity
C Anthony Hunt, Ryan C Kennedy, Sean HJ Kim, and Glen EP Ropella · 2013
Cited alongside, same era.
Playing atari with deep reinforcement learning
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013
Cited alongside, same era.
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Cited alongside, same era.
The next generation of sepsis trials: What’s next after the demise of recombinant human activated protein C?
Steven M Opal, R Phillip Dellinger, Jean-Louis Vincent, Henry Masur, and Derek C Angus · 2014
Cited alongside, same era.
Toward modular biological models: defining analog modules based on referent physiological mechanisms
Brenden K Petersen, Glen EP Ropella, and C Anthony Hunt · 2014
Cited alongside, same era.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Cited alongside, same era.
A systematic review and meta-analysis of early goal-directed therapy for septic shock: the arise, process and promise investigators
DC Angus, AE Barnato, D Bell, R Bellomo, C-R Chong, TJ Coats, A Davies, A Delaney, DA Harrison, A Holdgate, et al · 2015
Cited alongside, same era.
Simulating cancer growth with multiscale agent-based modeling
Zhihui Wang, Joseph D Butner, Romica Kerketta, Vittorio Cristini, and Thomas S Deisboeck · 2015
Cited alongside, same era.

Then

Dueling network architectures for deep reinforcement learning
Original
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas · 2015
Later among the works it cites.
The impact of determinism on learning atari 2600 games
Matthew J Hausknecht and Peter Stone · 2015
Later among the works it cites.
The third international consensus definitions for sepsis and septic shock (sepsis-3)
Mervyn Singer, Clifford S Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Jean-Daniel Chiche, Craig M Coopersmith, et al · 2016
Later among the works it cites.
Openai gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Later among the works it cites.
Deep reinforcement learning with double Q-learning
Hado Van Hasselt, Arthur Guez, and David Silver · 2016
Later among the works it cites.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Traffic signal timing via deep reinforcement learning
Li Li, Yisheng Lv, and Fei-Yue Wang · 2016
Later among the works it cites.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Original
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al · 2016
Later among the works it cites.
Determining controllability of sepsis using genetic algorithms on a proxy agent-based model of systemic inflammation
Chase Cockrell and Gary An · 2017
Later among the works it cites.
Optimization and control of agent-based models in biology: a perspective
G An, BG Fitzpatrick, S Christley, P Federico, A Kanarek, R Miller Neilan, M Oremland, R Salinas, Reinhard Laubenbacher, and S Lenhart · 2017
Later among the works it cites.
Sepsis reconsidered: Identifying novel metrics for behavioral landscape characterization with a high-performance computing implementation of an agent-based model
Chase Cockrell and Gary An · 2017
Later among the works it cites.
Simulation-based optimization of radiotherapy: Agent-based modeling and reinforcement learning
Ammar Jalalimanesh, Hamidreza Shahabi Haghighi, Abbas Ahmadi, and Madjid Soltani · 2017
Later among the works it cites.
Continuous state-space models for optimal sepsis treatment-a deep reinforcement learning approach
Original
Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi · 2017
Later among the works it cites.
Deep deterministic policy gradient for urban traffic light control
Original
Noe Casas · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…