UAV attitude estimation based on MARG and optical flow sensors using gated recurrent unit

Three-dimensional attitude estimation for unmanned aerial vehicles is usually based on the combination of magnetometer, accelerometer, and gyroscope (MARG). But MARG sensor can be easily affected by various disturbances, for example, vibration, external magnetic interference, and gyro drift. Optical flow sensor has the ability to extract motion information from image sequence, and thus, it is potential to augment three-dimensional attitude estimation for unmanned aerial vehicles. But the major problem is that the optical flow can be caused by both translational and rotational movements, which are difficult to be distinguished from each other. To solve the above problems, this article uses a gated recurrent unit neural network to implement data fusion for MARG and optical flow sensors, so as to enhance the accuracy of three-dimensional attitude estimation for unmanned aerial vehicles. The proposed algorithm can effectively make use of the attitude information contained in the optical flow measurements and can also achieve multi-sensor fusion for attitude estimation without explicit mathematical model. Compared with the commonly used extended Kalman filter algorithm for attitude estimation, the proposed algorithm shows higher accuracy in the flight test of quad-rotor unmanned aerial vehicles.


Introduction
Attitude estimation is the premise to ensure stable and controllable flight for unmanned aerial vehicles (UAV). [1][2][3] In recent years, attitude and heading reference systems (AHRS) are widely adopted in UAV flight control. 4 The typical configuration of AHRS includes magnetometer, accelerometer, and gyroscope (MARG). 5 With the continuous development of microelectro-mechanical system (MEMS) technology, MARG-based AHRS provides a low-cost solution for UAV attitude estimation.
MARG-based AHRS has the advantages of low power consumption and compact size. Theories and techniques for MARG-based AHRS are also well developed. The ellipsoid fitting (also called scalar checking) and multi-position methods can help to calibrate and compensate MARG sensor errors, [6][7][8] while Kalman filtering 4,5 and complementary filtering algorithms 9 can be used to implement data fusion for 1 MARG sensor. But still, the dynamic accuracy, robustness, and redundancy of AHRS needs further enhancement, since MARG-based three-dimensional (3D) attitude estimation is sensitive to external disturbances, for example, motional acceleration and magnetic interference.
Optical flow sensor can detect the movements of its carrier by processing the sequence of images and extract the differences between every two adjacent frames. 10 At present, optical flow sensor has also been applied to UAV flight control. With the aid of optical flow sensor, UAV can achieve tracking flight and spot hover with higher accuracy. [11][12][13] Besides, optical flow measurement can also be used for autonomous obstacle avoidance of UAV. 14 Moreover, optical flow measurement also contains attitude information. [15][16][17][18][19][20] Nevertheless, the attitude information contained in optical flow measurement is mixed with the translational movements in most cases, and thus, it is not easy to be fully utilized. 21,22 Therefore, when using optical flow sensor to augment MARG-based attitude estimation, one of the key issues is to distinguish the angular motion and translational motion that contained in optical flow measurement from each other. In most applications of optical flow sensors on UAV, the angular motion component of optical flow is eliminated with the aid of attitude information provided by IMU, and thus, only the translational motion component of optical flow is utilized. 15 However, to make use of the attitude information contained in optical flow, the translational motion component of optical flow must be compensated using additional position and/or velocity information, for example, position/velocity provided by global navigation satellite system (GNSS) or inertial navigation system (INS).
There are three main purposes of the research presented in this paper: (a) to extract the attitude information contained in optical flow measurement; (b) to implement multi-sensor fusion, including MARG and optical flow sensors; and (c) to compensate sensor errors that probably exists. To achieve the above three goals, a gated recurrent unit (GRU) neural network is used for data fusion between MARG and optical flow sensors, in order to improve the performance of AHRS under dynamic conditions. As a special brunch of artificial neural network, GRU is able to draw time-related information from time series signals, and thus, it can help to extract attitude information from optical flow measurement, without the need of complex mathematical models and additional position/velocity information.
The rest of this article is organized as follows. In section ''Theoretical basis,'' the theoretical basis will be briefly reviewed, including the principles of attitude estimation, data fusion, and optical flow sensor. Then in section ''Attitude estimation algorithm based on GRU,'' the design of GRU-based attitude estimation algorithm will be introduced, which incorporates MARG and optical flow sensors. In section ''Experiment,'' the proposed GRU-based algorithm will be evaluated through experiments on a quad-rotor UAV, and its performance will be compared with that of extended Kalman filter (EKF). After discussions of the experiment results in section ''Discussion'', conclusions will be drawn in the sixth section.

Attitude representation and estimation
The 3D attitude of UAV can be defined by the relationship between the body frame (coordinate system that fixed to UAV) and the reference frame (coordinate system that used as the reference for navigation, e.g. North-East-Down reference frame). The transformation from one coordinate system to another can be decomposed into three consecutive rotations around the three coordinate axes, namely the Euler angles. In the case of UAV, these three Euler angles are called yaw (or heading), pitch, and roll, respectively. Besides, quaternion and direction cosine matrix (DCM) are also commonly used for UAV attitude representation. 23 In traditional INS, especially the strap-down INS (SINS), the 3D attitude are calculated according to the angular velocity measured by gyroscope, and thus, the long-term attitude accuracy is affected by gyro drift. 4,24 In MARG-based AHRS, however, 3D attitude estimation mainly relies on the gravity vector g and geomagnetic vector h, which can be obtained from the accelerometer and magnetometer, respectively. 4,25 The gravity vector g is vertically downward and can be used to determine the tilt angles (i.e. pitch and roll). However, the horizontal component of geomagnetic vector h points to the magnetic north, and thus, it can help to determine the yaw angle. In a word, these two vectors can be directly used to calculate the attitude, and thus, accumulative attitude error caused by gyro drift can be avoided. Besides, the measurement of angular velocity provided by the gyroscope can be used to calculate the change of 3D attitude and thereby augmenting the attitude estimation. Therefore, the measurements of MARG sensor are sufficient for 3D attitude estimation, with appropriate data fusion algorithm.
Kalman filter (KF) is widely used for data fusion in MARG-based AHRS. It is worth mentioning that the mathematical relationship between attitude quaternion and vector observations (i.e. the gravity vector g and geomagnetic vector h) is nonlinear. As a result, nonlinear variants of KF are needed for MARG-based AHRS, such as the extended Kalman filter (EKF) 25 and unscented Kalman filter (UKF). 26 EKF is based on the linearization of the system model using the firstorder derivatives, while UKF makes use of unscented transform to approximate the nonlinear system. So far, EKF are more commonly used for UAV attitude estimation, mainly because its lower computational burden compared to UKF.

Optical flow sensor
Optical flow is the instantaneous velocity of the pixel motion on the imaging plane, which is caused by a spatially moving object. It can be calculated according to the correlation between the previous frame and the current frame in the image sequence. In another word, it contains the motion information of the object between every two adjacent frames. Based on the above principle, optical flow sensor can detect the movement of the UAV relative to the ground, as shown in Figure 1.
As shown in Figure 1(a) and (b), both the horizontal speed and rotation can generate corresponding optical flow. Therefore, in the case that optical flow sensor is attached to UAV and pointed to the ground, as shown in Figure 1(c), the optical flow measurement d can be described by equation (1). In equation (1), v is the horizontal velocity of the UAV, l is the height of the optical flow sensor above the ground, g is the angle between the optical axis and the vertical direction, v is the rotation speed of the optical flow sensor, and f is the scale factor According to equation (1), the optical flow measurement d is related to both the translational and rotational movements. Therefore, if the optical flow sensor is used to monitor ground speed, the optical flow component caused by rotation must be compensated, and vice versa. 12,19 Attitude estimation algorithm based on GRU GRU neural network for attitude estimation GRU is a special version of recurrent neural network (RNN). [27][28][29] RNN is a type of neural network that used to process sequence data. To better process sequence information, RNN has local feedback loop in its hidden layer, in order to recognize the relationship between the current input and the previous ones. 29,30 GRU also has local feedback, but the feedback is controlled by two gates, namely the reset gate and the update gate. In other words, GRU can be viewed as a gated RNN. GRU has achieved great success in natural language processing and speech conversion. The schematic diagram of GRU-based attitude estimation is shown in Figure 2. As shown in Figure 2, the GRU-based neural network takes the measurements from four different sensors as its inputs, namely the gravity vector g = ( g x g y g z ) from accelerometer, the geomagnetic vector h = ( h x h y h z ) from magnetometer, the angular velocity v = ( v x v y v z ) from gyroscope, and the two-dimensional (2D) optical flow d = ( d x d y ) from optical flow sensor. At the kth time step, all the above inputs are collectively denotes as x k . However, the output y k consists of the estimated yaw, pitch, and roll angles. Besides, z k is the hidden state of GRU, while the reset gate and update gate are denoted as f k and g k , respectively. At the kth time step, the reset gate f k and update gate g k are calculated according to the input x k and the previous hidden state z kÀ1 , as described by equations (2) and (3), in which s Á ð Þ denotes the activation function Then, the hidden state z k can be updated using equations (4) and (5) Furthermore, the hidden state z k is transformed into the output y k through the output layer, which can be described by equation (6) y As can be seen in equation (5), the update gate works as a complementary multiplexer, in which larger value of g k will bring more information from the input x k to the current hidden state z k . However, in the reset gate, smaller value of f k means thatz k will further ignore the previous hidden state z kÀ1 , as indicated by equation (4). These two gates can help to handle long-term correlations in time domain, as well as the gradient vanishing problem in RNN training.
In most studies of GRU, the sigmoid function is used for the activation function in equations (2) and (3), which is defined as s(x) = 1=(1 + e Àx ), while hyperbolic tangent (tanh) function is used for the activation function in equation (4). But in order to reduce the computational complexity, in the following experiment, the ''softsign'' function is used to replace tanh function in equation (4), which is defined as s(x) = x=(1 + x j j). Meanwhile, the sigmoid function is also replaced by a biased version of softsign function for equations (2) and (3), which is defined as s(x) = 1 + 0:5x=(1 + x j j). Since the softsign function contains no exponential or trigonometric functions, it can help to promote the processing efficiency on hardware platforms with limited computing power, such as a micro-controller.
As stated above, GRU can deal with time series signals, and it is also able to approximate nonlinear functions. Therefore, it can be used to solve the problem of data fusion between MARG and optical flow sensors for UAV attitude estimation. After sufficient training, GRU neural network can effectively utilize the measurements of MARG and optical flow sensors for attitude estimation, without the need of rigorous mathematical modeling.

GRU neural network training
The training process of GRU network is to determine the parameters in both the hidden layer and the output layer, including Figure 2.
First of all, the cost function for GRU training is defined as equation (7), in which y r, k denotes the target output of GRU network (i.e. the true value of yaw, pitch, and roll) at the kth time step Meanwhile, all the parameters to be estimated can be rearranged as a column vector u, so the training process can be described as an optimization problem in equa- Then, a hybrid algorithm is used for the optimization problem in equation (8), which is a combination of the commonly used Levenberg-Marquardt (L-M) method and extreme learning technique. The L-M method relies on the Jacobian matrix J for all the parameters to be estimated, and the Jacobian matrix J can be constructed according to the following partial derivatives.
(a) Derivatives for W and d in the output layer (e k means the kth row of the cost function e) ∂e k ∂d = 2 y k À y r, k À Á ð9Þ (b) Derivatives for U z and b z , with s 0 Á ð Þ and indicating the derivative of s Á ð Þ and Hadamard product (i.e. element-wise product of matrices), respectively (c) Derivatives for U g , V g , and b g in the update gate (d) Derivatives for U f , V f , and b f in the update gate It is noteworthy that the relationship between z k and y k is linear, as shown in equation (6). Therefore, W and d can be determined directly via least square fitting, and this is the essence of extreme learning. The hybrid training algorithm can be summarized as follows: 1. Initialization: iteration number i = 0, damping factor m i = 0:01, all the parameters ( Calculate the hidden state series z k f g, then solve W and d according to the target output series y r, k È É via direct least square fitting. 4. Calculate the output series y k f g, the cost function e i , and the Jacobian matrix J i = ∂e i =∂u i . 5. Update the parameter vector in which I p 3 p is an identity matrix, and p is the number of parameters in u i . 6. Update the hidden state series z k f g according to the parameters in u i + 1 , then solve W and d again. 7. Update the output series y k f g and the cost function e i + 1 . 8. If e i + 1 j j 2 \ e i j j 2 , accept the updated parameters and let m i + 1 = 0:1m i ; otherwise, rollback all the parameters and let m i + 1 = 10m i . 9. If e i j j 2 À e i + 1 j j 2 \10 À4 e i j j 2 , or i.5000, output all the parameters and then finish; otherwise i = i + 1 and go to 2).

Experiment
Raw data acquisition PX4FLOW optical flow sensor is used in the experimental evaluation of the proposed algorithm, accompanied with PIXHAWK2.4.8 autopilot system. PX4FLOW has the original resolution of 752 3 480 pixels, and it uses a fourfold grading and clipping algorithm to calculate the optical flow. This sensor is connected to PIXHAWK2.4.8 autopilot and attached to the quadcopter platform. Meanwhile, the quadcopter platform also integrates magnetometer, accelerometer, and gyroscope. After debugging, the quadcopter can fly stably in fixed height and hover modes, as shown in Figure 3. The PIXHAWK platform is equipped with 168 MHz Cortex M4 F CPU to process flight data. All the raw data, including the measurements from MARG and optical flow sensors, are recorded in the flight log.
Four datasets are collected using the apparatus shown in Figure 3, and they are denoted as Dataset 1, 2, 3, and 4, respectively. Figure 4 shows the trajectories of UAV during acquiring these four datasets. Dataset 1 is used for GRU training, while the other three datasets are used to evaluate the performance of GRU-based attitude estimation and compare it to EKF. Each dataset contains the raw data from magnetometer, accelerometer, gyroscope, and optical flow sensor, at the output rate of 50 Hz. Figure 5 illustrates the data in Dataset 1.

GRU training
The GRU neural network is trained using Dataset 1 with different numbers of hidden layer dimensions. The training results are evaluated in terms of root mean square error (RMSE), which can be calculated by equation (23) In equation (23), y k is the output of trained GRU network, which contains yaw, pitch, and roll angles. Meanwhile, y r, k is the corresponding true values that based on the fusion of GPS/barometer/MARG sensor, which can be read from the flight log of PIXHAWK autopilot. Figure 6 shows the RMSE and training time of GRU versus different hidden layer dimensions. The training is carried out on Intel Core i5-4590 CPU at 3.50 GHz with 8 GB RAM. It can be seen from Figure 6 that the relationship between GRU training time and hidden layer dimension is almost linear. Besides, the best performance can be achieved when hidden layer dimension is set to 15 (i.e. 15 GRUs in the hidden layer). The RMSE of roll, pitch, and yaw angles are 1.51°, 2.07°, and 0.87°, respectively. Furthermore, Figure 7 shows the estimated 3D attitude compared with the true values (including roll, pitch, and yaw)  when the dimension of hidden layer is 15. Compared to the attitude algorithms based on Elman neural network 31,32 the GRU-based attitude estimation method proposed in this article shows higher accuracy. Therefore, it is a feasible approach for data fusion.

GRU validating
After GRU training, the performance of GRU-based attitude estimation is further evaluated using Datasets 2, 3, and 4. Figure 8 shows the 3D attitude estimated by the proposed GRU-based algorithm in terms of Euler angles, as well as that of traditional EKF algorithm, compared to the ground truth from PIXHAWK flight logs. The root mean square error (RMSE) is still used to evaluate the overall performance for both GRU and EKF, and the results are listed in Table 1. It is worth mentioning that the proposed GRU-based algorithm is evaluated in two cases, that is, working with and without optical flow sensor, while EKF performs typical MARG-based attitude estimation and does not make use of optical flow sensor. According to the results in Table 1, the performance of GRU-based algorithm is better than that of EKF, especially when incorporating optical flow sensor.
To compare the computational complexity of GRU and EKF, the average execution time of both algorithms on STM32F103 micro-controller is listed in Table 2. The hidden layer dimension of GRU is still set to 15. It can be seen that GRU-based algorithm needs shorter execution time than EKF, either with or without the aid of optical flow.

Discussion
The experimental results evidently prove the feasibility of the proposed GRU-based attitude estimation algorithm. According to Figures 7 and 8, the proposed algorithm can effectively solve the problems of sensor data fusion and error compensation in attitude estimation of the quadcopter.
Furthermore, as can be seen in Table 1, the performance of the proposed GRU-based algorithm is comparable to that of EKF, in the case that optical flow sensor is unused. When the measurement of optical flow sensor is adopted, GRU-based algorithm can achieve higher attitude accuracy. Consequently, the    proposed GRU-based algorithm is able to make better use of the 3D attitude information in the measurements of MARG and optical flow sensors.
It is worth mentioning again that the proposed algorithm can implement data fusion for MARG and optical flow sensors, without considering explicit mathematical models of these sensors. It is the very point that the proposed algorithm differs essentially from the commonly used KF-based data fusion technique.

Conclusion
In this article, a 3D attitude estimation algorithm based on GRU network is proposed, which make use of MARG and optical flow sensors. Experimental results on a quadcopter show that the proposed algorithm can provide reliable attitude estimation according to the measurements of MARG and optical flow sensor. Compared with commonly used EKF-based attitude estimation, the proposed algorithm can further enhance 3D attitude accuracy by extracting the attitude information from the optical flow measurement, with no need to redesign the attitude algorithm according to complex mathematical equations. Therefore, it is a novel and feasible approach for the development of AHRS and UAV flight control.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work reported in this article was funded by the National Natural Science Foundation of China (grant numbers 61603107 and 41761087).