Sequential Detection Based Quickest Markov Decision Processes： Theory， Algorithms， and Applications

‍ ‍In this paper， joint signal processing and control methods for complex dynamical systems with statistically change point， observation noise， aftereffects， and action latency were investigated to maximize the overall utility of delay-sensitive decision making. A unified framework combining the quickest change detection in statistical signal processing and the Markov decision process in stochastic optimal control was presented along with its potential applications in smart grid， disease control， and hydrology. By leveraging a four-dimensional constrained Markov decision process， the proposed framework maximized the expected reward characterized by the weighted sum of the income and risk， while satisfying various constraints due to operations， feasibility， and environments. In contrast to the conventional layered infrastructure in which an action is launched after the change point is detected， the new architecture enabled a cross-layer cross-disciplinary collaboration between signal processing and control， which implemented real-time decisions much timelier based on instantaneous likelihood estimation. The paradigm-shift idea brought substantial gain for dynamical or stochastic systems that are sensitive to the latency in decision or control， while suffering from huge detection delay and/or strong aftereffects. It was demonstrated that the joint detection and control strategy outperformed the control-after-detection policy in smart grid， disease control， and hydrology with considerable gain observed. Finally， we briefly envisioned the potential applications of sequential detection based quickest Markov decision processes in carbon capture and storage in the seafloor as well as network attack detection and mitigation.