Built on

Gattami A (2019) Reinforcement learning for multi-objective and constrained Markov decision processes. arXiv preprint arXiv:1901.08978
Original
1901
Earlier work this paper cites.
Geist M, Scherrer B, Pietquin O (2019) A theory of regularized Markov decision processes. arXiv preprint arXiv:1901.11275
Original
1901
Earlier work this paper cites.
Le HM, Voloshin C, Yue Y (2019) Batch policy learning under constraints. arXiv preprint arXiv:1903.08738
Original
1903
Earlier work this paper cites.
Liu B, Cai Q, Yang Z, Wang Z (2019a) Neural proximal/trust region policy optimization attains globally optimal policy. arXiv preprint arXiv:1906.10306
Original
1906
Earlier work this paper cites.
Shani L, Efroni Y, Mannor S (2019) Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. arXiv preprint arXiv:1909.02769
Original
1909
Earlier work this paper cites.
Wang L, Cai Q, Yang Z, Wang Z (2019) Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150
Original
1909
Earlier work this paper cites.
Liu Y, Ding J, Liu X (2019b) Ipo: Interior-point policy optimization under constraints. arXiv preprint arXiv:1910.09615
Original
1910
Earlier work this paper cites.
Sion M, et al. (1958) On general minimax theorems. Pacific Journal of mathematics 8(1):171–176
1958
Earlier work this paper cites.
Bertsimas D, Orlin JB (1994) A technique for speeding up the solution of the Lagrangian dual. Mathematical Programming 63(1-3):23–45
1994
Earlier work this paper cites.
Singh SP, Cohn D (1998) How to dynamically merge Markov decision processes. Advances in neural information processing systems , 1057–1063
1998
Earlier work this paper cites.
Altman E (1999) Constrained Markov decision processes , volume 7 (CRC Press)
1999
Earlier work this paper cites.
Kakade S, Langford J (2002) Approximately optimal approximate reinforcement learning. ICML , volume 2, 267–274
2002
Earlier work this paper cites.
Harrison JM, Zeevi A (2004) Dynamic scheduling of a multiclass queue in the halfin-whitt heavy traffic regime. Operations Research 52(2):243–257
2004
Earlier work this paper cites.
Mandelbaum A, Stolyar AL (2004) Scheduling flexible servers with convex delay costs: Heavy-traffic optimality of the generalized c μ \mu -rule. Operations Research 52(6):836–855
2004
Earlier work this paper cites.

Similar

Stolyar AL, et al. (2004) Maxweight scheduling in a generalized switch: State space collapse and workload minimization in heavy traffic. The Annals of Applied Probability 14(1):1–53
2004
Cited alongside, same era.
Borkar VS (2005) An actor-critic algorithm for constrained Markov decision processes. Systems & control letters 54(3):207–213
2005
Cited alongside, same era.
Adelman D, Mersereau AJ (2008) Relaxations of weakly coupled stochastic dynamic programs. Operations Research 56(3):712–727
2008
Cited alongside, same era.
Dai JG, Lin W, et al. (2008) Asymptotic optimality of maximum pressure policies in stochastic processing networks. The Annals of Applied Probability 18(6):2239–2299
2008
Cited alongside, same era.
Nedić A, Ozdaglar A (2009) Subgradient methods for saddle-point problems. Journal of optimization theory and applications 142(1):205–228
2009

Then

Chen Y, Wang M (2016) Stochastic primal-dual methods and sample complexity of reinforcement learning. arXiv preprint arXiv:1612.02516
Original
2016
Later among the works it cites.
Achiam J, Held D, Tamar A, Abbeel P (2017) Constrained policy optimization. arXiv preprint arXiv:1705.10528
Original
2017
Later among the works it cites.
Haarnoja T, Tang H, Abbeel P, Levine S (2017) Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165
Original
2017
Later among the works it cites.
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2021

A Primal-Dual Approach to Constrained Markov Decision Processes

Chen, Yi, Dong, Jing, Wang, Zhaoran

Understand

In many operations management problems, we need to make decisions sequentially to minimize the cost while satisfying certain constraints.

One modeling approach to study such problems is constrained Markov decision process (CMDP).
When solving the CMDP to derive good operational policies, there are two key challenges: one is the prohibitively large state space and action space; the other is the hard-to-compute transition kernel.
In this work, we develop a sampling-based primal-dual algorithm to solve CMDPs.

Built on

Gattami A (2019) Reinforcement learning for multi-objective and constrained Markov decision processes. arXiv preprint arXiv:1901.08978
Original
1901
Earlier work this paper cites.
Geist M, Scherrer B, Pietquin O (2019) A theory of regularized Markov decision processes. arXiv preprint arXiv:1901.11275
Original
1901
Earlier work this paper cites.
Le HM, Voloshin C, Yue Y (2019) Batch policy learning under constraints. arXiv preprint arXiv:1903.08738
Original
1903
Earlier work this paper cites.
Liu B, Cai Q, Yang Z, Wang Z (2019a) Neural proximal/trust region policy optimization attains globally optimal policy. arXiv preprint arXiv:1906.10306
Original
1906
Earlier work this paper cites.
Shani L, Efroni Y, Mannor S (2019) Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps. arXiv preprint arXiv:1909.02769
Original
1909
Earlier work this paper cites.
Wang L, Cai Q, Yang Z, Wang Z (2019) Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150
Original
1909
Earlier work this paper cites.
Liu Y, Ding J, Liu X (2019b) Ipo: Interior-point policy optimization under constraints. arXiv preprint arXiv:1910.09615
Original
1910
Earlier work this paper cites.
Sion M, et al. (1958) On general minimax theorems. Pacific Journal of mathematics 8(1):171–176
1958
Earlier work this paper cites.
Bertsimas D, Orlin JB (1994) A technique for speeding up the solution of the Lagrangian dual. Mathematical Programming 63(1-3):23–45
1994
Earlier work this paper cites.
Singh SP, Cohn D (1998) How to dynamically merge Markov decision processes. Advances in neural information processing systems , 1057–1063
1998
Earlier work this paper cites.
Altman E (1999) Constrained Markov decision processes , volume 7 (CRC Press)
1999
Earlier work this paper cites.
Kakade S, Langford J (2002) Approximately optimal approximate reinforcement learning. ICML , volume 2, 267–274
2002
Earlier work this paper cites.
Harrison JM, Zeevi A (2004) Dynamic scheduling of a multiclass queue in the halfin-whitt heavy traffic regime. Operations Research 52(2):243–257
2004
Earlier work this paper cites.
Mandelbaum A, Stolyar AL (2004) Scheduling flexible servers with convex delay costs: Heavy-traffic optimality of the generalized c μ \mu -rule. Operations Research 52(6):836–855
2004
Earlier work this paper cites.

Similar

Stolyar AL, et al. (2004) Maxweight scheduling in a generalized switch: State space collapse and workload minimization in heavy traffic. The Annals of Applied Probability 14(1):1–53
2004
Cited alongside, same era.
Borkar VS (2005) An actor-critic algorithm for constrained Markov decision processes. Systems & control letters 54(3):207–213
2005
Cited alongside, same era.
Adelman D, Mersereau AJ (2008) Relaxations of weakly coupled stochastic dynamic programs. Operations Research 56(3):712–727
2008
Cited alongside, same era.
Dai JG, Lin W, et al. (2008) Asymptotic optimality of maximum pressure policies in stochastic processing networks. The Annals of Applied Probability 18(6):2239–2299
2008
Cited alongside, same era.
Nedić A, Ozdaglar A (2009) Subgradient methods for saddle-point problems. Journal of optimization theory and applications 142(1):205–228
2009

Then

Chen Y, Wang M (2016) Stochastic primal-dual methods and sample complexity of reinforcement learning. arXiv preprint arXiv:1612.02516
Original
2016
Later among the works it cites.
Achiam J, Held D, Tamar A, Abbeel P (2017) Constrained policy optimization. arXiv preprint arXiv:1705.10528
Original
2017
Later among the works it cites.
Haarnoja T, Tang H, Abbeel P, Levine S (2017) Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165
Original
2017
Later among the works it cites.
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Neely MJ (2011) Online fractional programming for Markov decision systems. 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , 353–360 (IEEE)

2011

Cited alongside, same era.

Bhatnagar S, Lakshmanan K (2012) An online actor–critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applications 153(3):688–708

2012

Cited alongside, same era.

Nemirovski A (2012) Tutorial: Mirror descent algorithms for large-scale deterministic and stochastic convex optimization. Conference on Learning Theory (COLT)

2012

Cited alongside, same era.

Turken N, Tan Y, Vakharia AJ, Wang L, Wang R, Yenipazarli A (2012) The multi-product newsvendor problem: Review, extensions, and directions for future research. Handbook of newsvendor problems , 3–39 (Springer)

2012

Cited alongside, same era.

Bubeck S (2014) Convex optimization: Algorithms and complexity. arXiv preprint arXiv:1405.4980

Original

2014

Cited alongside, same era.

Caramanis C, Dimitrov NB, Morton DP (2014) Efficient algorithms for budget-constrained Markov decision processes. IEEE Transactions on Automatic Control 59(10):2813–2817

2014

Cited alongside, same era.

Bertsekas DP, Scientific A (2015) Convex optimization algorithms (Athena Scientific Belmont)

2015

Cited alongside, same era.

Chow Y, Nachum O, Duenez-Guzman E, Ghavamzadeh M (2018) A Lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems , 8092–8101

2018