Optimal control for networked control systems with multiple delays and packet dropouts

This article focuses on the problem of optimal linear quadratic Gaussian control for networked control systems with multiple delays and packet dropouts. The main contributions are twofold. Firstly, based on the introduced maximum principle for linear quadratic Gaussian system with multiple input delays and packet dropouts, a nonhomogeneous relationship between the state and costate is obtained, which is the key technical tool to solve the problem. Secondly, a necessary and sufficient condition for the optimal networked control problem is given in virtue of the coupled Riccati equations, and the explicit expression of the optimal controller is presented. Numerical examples are shown to illustrate the proposed algorithm.


Introduction
Networked control systems (NCSs) are spatially distributed systems which transmit information between sensors, actuators, and controllers via a shared communication network. [1][2][3] With the wide use of computer and the rapid development of network technology, NCSs are found in a wide range of applications in various fields, such as mobile sensor networks, building automation surveillance, and reduction of car wiring, which makes the control systems more complicated and has attracted considerable attentions. 4-6 However, the situation of time delay and data loss often occurs in the process of communication. This motivated us to study the optimal control for NCSs with multiple delays and packet dropouts.
As for the optimal control for NCSs problem, many results have been surveyed. 7-14 Cacace et al. 7 studied the linear time invariant linear quadratic Gaussian (LQG) problems with a single input delay and instantaneous state feedback. Yang et al. 10 presented a method for the compensation of a networked state-feedback control system, which possesses a randomly varying transmission delay and uncertain process parameters. Zhang et al. 12 studied the classic linear quadratic regulation (LQR) problem for both continuous-time and discrete-time systems with multiple input delays. Ma et al. 13 designed a state-feedback controller for the closed-loop system with both discrete and distributed input time delays in terms of the solvability of certain Hamilton-Jacobi inequalities. And they 14 studied the nonlinear Markovian jump systems subject to actuator failures and mixed time-delays and proposed the sufficient conditions for the existence of the desired controller. On the other hand, the problem on packet dropout occurred in NCSs is studied widely. Yu et al. 15 obtained the sufficient conditions on the stability and stabilization of the NCSs with packet dropout. Gao et al. 16 considered linear quadratic (LQ) optimal control problems based on state observer for a class of discrete-time system over lossy data network. Gupta et al. 17 investigated the optimal LQG control by decomposing the system into a standard LQR state-feedback controller, applying the method of optimal encoder-decoder. Liang and Xu 18 studied the packet dropout LQG control for NCSs with both local and remote controllers and gave the expression of the optimal controller. Moreover, the more applicability research on NCSs with both packet dropouts and time delay can be shown in the literature. 19-21 Liang et al. 19 obtained the necessary and sufficiency condition for the discrete stochastic LQG control system with multiplicative noises and input delay and presented the suboptimal controller when the separation principle did not hold. Zhang et al. 20 investigated the optimal results of LQR control system in both finitehorizon case and infinite-horizon case, where exist delay and multiplicative noise. What's more, there are many references about the control of NCSs with multiple time delays and packet loss. Fischer et al. 22 derived an optimal solution to the LQG control problem and proved that the separation principle holds, based on the network control system where control inputs and measurements are transmitted over transfer control protocol (TCP)-like network connections that are subject to random transmission delays and packet losses. Liang et al. 23 obtained the optimal LQR controller separately from the state estimator and derived a necessary and sufficient condition for the mean-square stabilization when there are both delays and packet dropouts in the NCSs.
However, there is no LQG control for NCSs with both multiple time delays and packet dropouts in the above literature. Inspired by the work of Liang and Xu 18 and Liang et al. 23 we consider the NCSs with the assumption that one communication channel with large packet dropout probability, while the other communication channel with both small packet dropout probability and time delay. The case is shown in Figure 1, which is composed of a plant, a controller, and two actuators. In the closed loop, obviously, the state x k is sent to the controller, and output as u k . When the signal is transited to actuators over unreliable communication channel in the case of assumption, there will exist packet dropouts and time delays, that is, g k ðiÞ ¼ 0 signifies the packet is lost with the probability of packet dropout p i , and g k ðiÞ ¼ 1 denotes the packet delivered successfully.
Motivated by Li et al., 24 we expend the results of optimal LQR control with multiplicative noises and multiple input delays into NCSs subject to multiple delays and packet dropouts. Moreover, we study the optimal LQG control of NCSs by adding the additive noise. Noting the approach of solving the forward and backward stochastic difference equations (FBSDEs), 19,20,24 this article studies the optimal LQG control problem for NCSs with both multiple time delays and packet dropout. Different from the literature, 13,14 the key technique of this article is to apply the maximum principle of multiple delays system to obtained the solution to FBSDEs, which is the nonhomogeneous relationship of the state and the costate. The main contributions of this article are summarized as follows: (1) A solution to the FBSDEs is obtained based on the derived maximum principle for NCSs with delays and packet loss. (2) In terms of the solution of the FBSDEs, a necessary and sufficient condition is given for the optimal LQG control.
Notation. R n denotes the n-dimensional real Euclidean space. I represents the unit matrix of appropriate dimension. The superscript 0 denotes the transpose of the matrix. O; F; P; F k f g k!0 È É denotes a complete probability space on which random variable n k are defined such that F k f g k!0 is the natural filtration generated by n k , that is, F k ¼ s n 0 ; . . . ; n k f g , augmented by all the P-null sets in F. A symmetric A > 0ð! 0Þ means that it is a positive definite (positive semi-definite) matrix. TrðAÞ represents the trace of matrix A.

Problem formulation
Consider the discrete stochastic LQG system with multiple input delays and packet dropouts, the corresponding plant in Figure 1 is given by for k ¼ 0; :::; N , where x k 2 R n is the state, u k 2 R m is the control input with a constant delay d > 0, g k ðjÞ is an independent identically distributed Bernoulli random noise representing the packet dropouts with the probability of p j 2 ð0; 1Þ, that is, And n k 2 R n is a stochastic sequence, and C; D 0 , and D d are deterministic matrices with compatible dimensions. Denote w k ðjÞ ¼ g k ðjÞ À E½g k ðjÞ ¼ g k ðjÞ À ð1 À p j Þ. Then the system (1) can be written as where w k ðjÞ is a random variable. Besides, w k ðjÞ and n k are uncorrelated and satisfy . . . ; À1 are known. The NCSs with delays and packet loss can be well described as ½C; D 0 ; D d ; pjd conveniently.
The associated cost function for system (2) is given by where Q, R, and T N þ1 are positive semi-definite constant matrices with appropriate dimensions, and N is the horizon length.

Main results
By applying the Pontryagin's maximum principle to the discrete LQG system (2) with the cost function (3) yields that To make further study, we define the following coupled Riccati difference equations where The terminal values are given by It is stressed that the key to solve the optimal LQG control problem is to solve the FBSDEs (2) and (4)-(6). We now give the nonhomogeneous relationship of the optimal costate x kÀ1 and state x k as follows.
Proof. The proof of Lemma 1 is provided in Appendix 1.
Based on the maximum principle and the solution to the FBSDEs from Lemma 1, we can present the optimal control results of Problem 1.
In this context, the optimal controller u k will be given by The associated optimal performance index is given by where O k , L k , L j k , T k , and T j k satisfy the coupled equations (7)-(13) with the terminal value (14).
Proof. The proof of Theorem 1 is provided in Appendix 2.
Now we obtained the special case of NCSs with multiple delays and packet dropouts which is contained with only u k and u kÀd . Remark 1. For stochastic NCSs with multiple delays and packet dropouts, the corresponding plant is given by for k ¼ 0; :::; N , with the initial values x 0 ; u i for i ¼ Àd; . . . ; À1. Besides, the Riccati equations (8), (11), and (13) will be as follows: for j ¼ 0; . . . ; d À 1, where the terminal value is as (14). In this context, there exists a unique F kÀ1 -measurable u k for system (18) subject to the cost function (3) if and only if O k ; k ¼ 0; . . . ; N , are all invertible. The optimal controller u k will be given by with the associated optimal cost function In addition, the relationship of the optimal costate x kÀ1 and state x k is as (15).

Numerical examples
Example 1. Consider the scalar case of LQG control system with multiple delays and packet dropouts (2), and the cost function (3). As the coefficients with By applying Theorem 2 and equations (8)-(15), it yields that Obviously, O k > 0, thus, there is an optimal solution to the NCSs with multiple delays and packet dropouts from Theorem 2. The optimal controller is shown in Figure 2.
Accordingly, the optimal value of equation (18) is Example 2. Consider the LQG control system with multiple delays and packet dropouts (2) with x k 2 R 2 , u k 2 R 2 , and the cost function (3). As the coefficients with

Conclusion
In this article, the optimal LQG control of NCSs with multiple delays and packet dropouts has been studied. A necessary and sufficient condition for the existence of unique optimal controller to the problem is given, which is based on the obtained maximum principle and the solution to the introduced coupled difference equations. Under this condition, the optimal controller and the minimized performance index are represented. In the future works, we expect that the results in this article pave new ways for the stabilization of the NCSs with multiple delays and packet dropouts. Moreover, the optimal output-feedback control for the NCSs with delays and packet dropouts will be studied in our further research.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (61903233)
Assuming that x kÀ1 are as (15) for all k ! n þ 1 with n > N À d, then we will show that equation (15) also holds for k ¼ n. Set u k to be optimal for all k ! n þ 1, with equations (2) and (15), x n can be calculated as Insert x n to equation (6), then equation (6) will become 0 ¼ L n x n þ O n u n þ X dÀ1 j¼0 L j n u jþnÀd Thus, the optimal controller is given by for n ¼ N ; . . . ; N À d þ 1. Using equations (2), (4), and (1C), x nÀ1 yields that which implies that equation (15) holds for k ¼ n; N À d < n N . Then we obtained Analogy with the method, assuming that x kÀ1 are as (15) for all k ! n þ 1; n ¼ 0; . . . ; N À d, and we will verify that equation (2) also holds for k ¼ n. As x n is calculated as (27), then for n ¼ 0; . . . ; N À d, equation (6) will be obtained After inserting equations (2) and (1D), we can summarize that Now, the optimal controller for n ¼ 0; . . . ; N À d is obtained as u n ¼ ÀO À1 n L n x n À O À1 n X dÀ1 j¼0 L j n u jþnÀd In the same way, substituting u n into equation (5), we can also prove that T j n u jþnÀd ; n ¼ 0; . . . ; N À d This completes the proof of the lemma.

Appendix 2
Proof of Theorem 1. "Necessity": Suppose there exists the unique F kÀ1 -measurable u k to make the cost function (3) minimized. We will show by induction that O k ; k ¼ d; . . . ; N , are invertible and the optimal controller can be designed as (16). Define for k ¼ 0; . . . ; N , and when k ¼ N the above equation becomes Using equation (2), we can obviously know that the uniqueness of the optimal controller only depends on whether u N > 0. Then setting x N ¼ 0, and u N Àd ¼ 0, J ðN Þ can be represented as Construct the equation V k ðx k Þ À V kþ1 ðx kþ1 Þ, then we have Adding from k ¼ 0 to k ¼ N on both sides of equation (2B), then we get Tr½T kþ1 Q n k Then the cost function (3) becomes Tr½T kþ1 Q n k As O k > 0, the optimal controller is as (16) and the optimal cost is as (17). The proof of sufficiency is completed.