A method to keep autonomous vehicles steadily drive based on lane detection

Existing studies on autonomous driving methods focus on the fusion of onboard sensor data. However, the driving behavior might be unsteady because of the uncertainties of environments. In this article, an expectation line is proposed to quantify the driving behavior motivated by the driving continuity of human drivers. Furthermore, the smooth driving could be achieved by predicting the future trajectory of the expectation line. First, a convolutional neural network-based method is applied to detect lanes in images sampled from driving video. Second, the expectation line is defined to model driving behavior of an autonomous vehicle. Finally, the long short-term memory-based method is applied to the expectation line so that the future trajectory of the vehicle could be predicted. By incorporating convolutional neural network- and long short-term memory-based methods, the autonomous vehicles could smoothly drive because of the prior information. The proposed method is evaluated using driving video data, and the experimental results demonstrate that the proposed method outperforms methods without trajectory predictions.


Introduction
The technology of autonomous driving has been attracting amount of attention of both academia and industry over the last several years 1 Generally, the studies on autonomous driving could be divided into three different components including perception, decision-making, and control. 2 The perception component aims to efficiently obtain environmental information for autonomous vehicles, 3 and the control component aims to set appropriate mechanical parameters so that the autonomous vehicles could travel forward by following the planned path. 4 The decisionmaking component tries to determine a perfect path for autonomous vehicles, which could safely navigate the vehicles toward their destinations. 5 Therefore, it is the core component of the three components because it determines all the actions of the autonomous vehicle, such as lane change, brake, and acceleration. Typically, the decisionmaking component in autonomous driving is realized by the decision-making models incorporated by other components, such as navigation system.
To navigate an autonomous vehicle driving to the target place from the current position, the decision-making component should have a best understanding of the environments. 6 For example, the Global Positioning System (GPS) is always used as a good assisted tool for the navigation of the autonomous vehicles. 7 However, there are the following drawbacks of the GPS navigation. First, it is not very accurate for GPS navigation. Second, GPS signals are weak in lots of scenarios, such as the tunnel. Therefore, there should be some methods to make up for GPS navigations. In these methods, the onboard sensors should be used, and the local algorithms should be designed to clearly understand the driving environment so that the autonomous vehicles could make correct decisions.
The road or lane is always the key reference during the driving activities. When a human driver drives a car, he or she keeps the vehicle in a specific position of a lane. For example, the vehicle might be in the middle, and it might be on the left side of the lane. Motivated by this, this article tries to transfer this driving behavior of a human to autonomous vehicles. Therefore, the first task is lane detection which is regarded as one of the challenging tasks in the studies of autonomous driving. 8 Lane detection could identify the lanes on the road so that the vehicle location could be obtained. For example, in the case of a tunnel, GPS signals may be lost, the lane detection tasks could help the navigation system locally percept all kinds of environmental information. 9 Furthermore, when a human drives a car, he or she is always aware of a short path in front of the vehicle. This behavior is meaningful for driving activities. First, the vehicle could be well controlled when the lanes are missing. Second, the path could be planned in advance which is important for safe driving. Finally, it is useful to keep the vehicle steady. In the previous work, a center line is constructed from a set of predefined waypoints to help autonomous driving. 10 In this article, the line might not be the center line. It should be an expectation line, which is used to quantify the behavior of drivers so that their decisions would not be affected by uncertain environments. The drivers should follow the expectation line which is a fictional line. It is shown in Figure 1. The expectation line is between left lane and right lane. Moreover, it must guarantee the vehicle would not be out of the lane. Furthermore, it is not enough for an autonomous vehicle to recognize its current environment because of the time consumption of the calculation tasks. It must also be able to predict the future states of the environment by remembering the past information. 11 Therefore, the autonomous driving method could be divided into the following stages in this article. First, lanes should be detected. The deep learning-based methods are applied for this task. Moreover, it is compared with traditional OpenCV-based methods. Second, an expectation line is constructed. Finally, because of recurrent neural networks (RNNs) that have been used in end-to-end scene labeling systems, 12 the future trajectory of the vehicle is predicted by long short-term memory (LSTM) networks which is a kind of RNN. The proposed method is evaluated using autonomous driving data sets.
The remaining of this article is structured as follows: In the second section, the detailed related work about the proposed method is described. In the third section, the study problems are defined. The architecture of system is introduced in the fourth section. The fifth section is about the detailed autonomous driving method using lane detection. Experimental results are analyzed in the sixth section. The final section provides a brief summary of the article.

Decision-making in autonomous driving
In autonomous driving, the vehicle should select the right route and navigate to the destination. During the process of driving, it should correctly interact with the environment, such as the traffic signals and other vehicles on the road. However, there are a large number of uncertainties over the intentions of other traffic participants in real-world driving. Therefore, decision-making component should be robustly designed so that the autonomous vehicles could handle different uncertainties.
Generally, the decision-making component of autonomous driving could be hierarchically decomposed into the following components. First, a route toward the destination is planned at the highest level. Second, the motion planning module would select a continuous path through the environment to accomplish a locally navigational task. Finally, the control system reactively corrects errors in the execution of the planned motion. 13 In decision-making of autonomous driving, the intentions of other traffic participants should be well predicted, such as the future trajectories of the other vehicles. The machine learning-based methods have been proposed to solve the above problems, such as Gaussian mixture models, 14 Gaussian process regression. 15 Moreover, some model-based approaches are proposed to directly estimate the intentions from collection of onboard sensors. 16 In addition, the probabilistic methods are always used to make decision for autonomous vehicles in complex uncertain environments. For example, Markov decision processes (MDPs) and their extended methods are always used to model the decision-making problems of autonomous driving. Brechtel et al. applied the MDP framework to study the behavioral decision-making problem. 17 Furthermore, they modeled the unobserved driving scenarios and pedestrian intentions explicitly using a partially observable Markov decision process (POMDP) framework. 18 In their work, specific approximate solution strategies were proposed.
With the development of onboard computation and wireless communication technology, it is possible for autonomous vehicles to exchange information with each other or interact with the road infrastructure. Recently, the studies on connected autonomous vehicles are attracting more and more attention. 19 By the information shared by connected vehicles, the driving safety could be greatly improved.

Lane detection
In fact, lane detection is one of the fundamental problems in autonomous driving. The autonomous vehicle must reliably detect the boundaries of its current lane for accurate localization to guarantee the driving safety. However, the simple appearance of a lane is difficult to provide the distinctive features for lane detection tasks. Moreover, diverse lane patterns, such as solid, broken, splitting, and merging lanes, make independent lane modeling difficult.
Traditionally, algorithms based on highly specialized handcrafted features were used to solve the lane detection problems. 20 In these algorithms, the handcrafted features include color-based features 21 and the structure tensor. 22 Furthermore, the features are combined with Hough transform to evaluate a binary edge image by discretizing all possible lines in the image into an accumulator matrix and counting the edge pixels falling on each line. 23 There are some assumptions on lanes in existing lane detection methods, such as lanes are parallel 24 and lanes are straight or close to straight. 25 However, these assumptions are not always valid especially in urban situations. Recently, methods based on deep neural networks were applied to detect lanes for autonomous vehicles. 26,27 Especially, the convolutional neural networks (CNNs) had achieved promising results.
There were studies about the lane detection task and the control of the autonomous vehicles. Mcfall et al. proposed an effective lane detection algorithm employing the Hough transform and inverse perspective mapping to estimate distances in real space. Moreover, the estimated distances could be utilized to send steering control commands to an autonomous vehicle. 28

Path planning
The path planning is also an important task for autonomous driving. Generating the future sequence of driving actions could enable the autonomous vehicles travel successfully and safely.
Montemerlo et al. developed an efficient method for autonomous vehicles to plan path. 29 In their method, the lateral offset was defined as the perpendicular distance to a fixed baseline which was parallel to the driving path of an autonomous vehicle. Moreover, a cost function was optimized to identify the traveling path. The cost function considering the following two aspects. The one was the distance from the central road. The other was collision obstacle. To avoid obstacles on roads, Chu et al. proposed an approach to solve the path planning problem in autonomous vehicles. 30 Phung et al. proposed an improved path planning method for autonomous vehicles using discrete particle swarm optimization. 31 Although these methods could plan a smooth path for autonomous vehicles in a safe way, they could only treat static obstacles. Li et al. had applied discrete optimization to design a path planning method. 9 In their method, an optimal path could be selected from a collection of possible paths. Moreover, the appropriate vehicle speed and acceleration could be simultaneously determined. Zuo et al. considered the navigation problem for mobile robot in complex environments and proposed a hierarchical path planning method. 32 To construct a path to the destination avoiding collision, rapidly exploring random trees (RRTs)-based methods were widely applied. Cui et al. proposed an adaptive path planning algorithm to estimate the scalar field over a region of interest. 33 Devaurs et al. developed two efficient samplingbased approaches based on RRT to solve a complex path planning problem. 34 To effectively plan paths, the environment could be mapped to the cells set. In the set, each cell was used to represent an obstacle which was appeared at that position in the environment. 35 Furthermore, the optimal search algorithms including A* and D* could be applied to discover a global optimized path from current position to the destination. Moreover, the obstacles on the path could be avoided.
Recently, some improved cell-based methods were proposed to apply in unstructured environments, such as U-turns on a blocked road and parking. 36 Recently, LSTMor RNN-based methods had also been applied to analyze the vehicle trajectory. 26,27 Khosroshahi et al. applied LSTM to identify the intention of drivers based on the trajectory data. 37 Kim et al. had applied LSTM to predict the location of the vehicle after several seconds using the past trajectory data. 38 Moreover, other deep learning-based methods were also studied for trajectory prediction. Park et al. proposed a deep learning-based vehicle trajectory prediction technique which could generate the future trajectory sequence of surrounding vehicles in real time. 39 In addition, Gaussian distribution, dynamic Bayesian network, and fuzzy methods are also used in path planning. [40][41][42][43][44][45][46] In this article, a driving method for autonomous vehicles would be designed. To solve this problem, the popular lane detection and trajectory prediction method would be used.
Moreover, the vehicle model should be investigated to obtain the historical trajectory data of vehicles.

Problem definition
In real world, a human could perfectly drive a car in a separate lane. Motivated by this, the goal of this article is to design an autonomous driving approach to keep the vehicle steadily drive, which is similar to human drivers. Specially, we assume that there are no obstacles in the same lane with the autonomous vehicle. Therefore, there are totally three problems that should be defined, including lane model, vehicle model, and trajectory prediction model. The lane model defines the structure used in this article. The vehicle model describes the attributes and dynamic behavior of the autonomous vehicle. The trajectory prediction model introduces the ways to know the future vehicle trajectory.

Lane model
In this article, we assume that a lane includes the right side boundary (right lane, RL) and the left side boundary (left lane, LL). Moreover, assuming that LL is the parallel shift of RL. Each side could be modeled by a polynomial model, which could fit the lane boundary edges. The polynomial model could be defined as follows where x and y are the coordinates of the polynomial curves. The values of A ¼ ða n ; a nÀ1 ; . . . ; a 1 Þ T could be obtained by providing a set of n equations formed by edge pixels . .
The expectation line (EL) is used to measure the appreciate path of the autonomous vehicle. It could be defined as follows The integer number N means that the whole lane would be divided into two parts, the one is 1=N of the whole lane width, and the other is ðN À 1Þ=N of the lane width. When N equals to 2, the expectation line is the central line of the lane. To prevent the autonomous vehicle out of the boundary of the lane, the following constrains should be defined as where w is the width of the vehicle.

Vehicle model
In this article, the autonomous vehicle model is described in the two-dimensional plane, which is shown in Figure 2.
A vehicle V could be described as follows where P ¼ ðx; yÞ represents the coordinates of the location of centroid of the vehicle and V ¼ ðV x ; V y Þ is the velocity of the vehicle. V x and V y , respectively, refer to the longitudinal velocity and the lateral velocity at the center of mass and Y is the yaw angle, which indicates the orientation of the vehicle. The yaw angle is identified as the angle between the vector of the longitudinal velocity and the tangent to the expectation line. Given discrete time Dt, the autonomous vehicle dynamics could be described as follows where aðtÞ is the acceleration of the autonomous vehicle. That is, the autonomous vehicle state at t þ 1 is determined by the current state and future control input.

Trajectory prediction model
In fact, the conditions of the lane are uncertain. Therefore, the prediction of the future trajectories is meaningful for the safe driving. Generally, the trajectory would not dramatically and frequently change for the comfortable driving. Therefore, the historical trajectory data could be used to predict the future status of the trajectory. Assuming that the historical trajectory data are HTD ¼ fp 1 ; p 2 ; . . . ; p n g at time T, where p i ð1 < i < nÞ is the passed position of the vehicle. The future positions Dd ahead of the vehicle should be estimated by the trajectory prediction model. The set of future positions DHTD could be defined as follows That is, a function f should be designed so that it could learn the patterns from the historical data. Moreover, the position data in future Dd could be predicted. Generally, f could be implemented using machine learning-based methods.

System overview
The proposed method could be used in autonomous driving systems from perception, decision-making to control. The system overview of the proposed autonomous driving method using lane detection is shown in Figure 3.
The whole system includes three layers including perception layer, decision-making layer, and control layer. In perception layer, the lane data are collected by the sensors, such as the camera, which is located on the autonomous vehicle. Generally, it is image or video data. In decisionmaking layer, the lanes in each captured image would be detected using methods such as CNN. Moreover, the historical trajectory data could be collected. That is, the trajectory could be constructed at a specific time. Furthermore, the LSTM network could be used to predict the future trajectory of the vehicle. In control layer, a steering command would be generated according to LSTM network, and the command could be sent to the control system of the vehicle. By deploying this system on the autonomous vehicles, the driving behavior of human who drives a car in lane could be modeled.

Autonomous driving method using lane detection
In this article, the proposed method includes two steps. The one is lane detection, and the other is trajectory prediction. The structure of the method is shown in Figure 4.

Lane detection using CNN
Although traditional OpenCV methods were always used to detect lanes from images, the CNN-based methods had achieved better performance for lane detection task. Assuming that the training data D ¼ ff 1 ; f 2 ; . . . ; f n g, where f i ð1 < i < nÞ is the continuous frame obtained from the driving video, and each frame is a separate image. The CNN is applied to detect the lane of the current autonomous vehicle.
The detailed steps of the lane detection method are shown in Algorithm 1. The training data D should be preprocessed, and the lanes in the data set are labeled. The input of the algorithm is D, and output of the algorithm is the detected lane in each image.

Prediction trajectory using LSTM
By the vehicle model, the observation of autonomous vehicle at current time t could be obtained O t ¼ ðx t ; y t ; v xt ; v yt ; Y t Þ. Moreover, the lane L t ¼ ðLL t ; RL t Þ has been detected using  CNN method. The future trajectory of the vehicle would be predicted using LSTM.
First, the expectation line EL t should be constructed using L t . That is, the vehicle would go ahead tracking the expectation line EL t . In LSTM component, the fully connected (FC) layer is followed by rectified linear unit (ReLU) activation function. The FC layer could transform the five-dimensional input data O t into the same dimensional feature needed by LSTM cell. Moreover, it extends the network capacity so that complex structure of the trajectory data could be handled. The output of the FC layer is fed into the LSTM stack which is constructed with 256dimensional cell memory. The output of LSTM is sent to another FC layers which is followed by a Softmax layer. The output of the FC layers is passed through the Softmax function, and the future trajectory would be predicted.
To train the LSTM trajectory prediction model, the loss function could be defined as follows x tþDt À z t;x 2 þ y tþDt À z t;y 2 þ lO ðwÞ (8) where w is the parameter of the LSTM network, N is the total number of the training samples, z t;x and z t;y are the predicted values of LSTM for a specific training sample, and O ðwÞ is the regularization term with the parameter l.
Assuming that the trajectory observation set at time T is

Experiment setup
There are two important deep learning-based methods used in this article. The one is CNN and the other is LSTM. Both methods need plenty of computation resources for the training period. Therefore, the experimental environment with GPU configurations is needed.
In this article, the Colab platform provided by Google is used for GPU resources. Moreover, the TensorFlow framework is used as the deep learning platform. Moreover, the OpenCV development environment is configured so that the experimental results obtained by deep learning methods could be compared. Furthermore, the advantages of the deep learning-based methods could be demonstrated.

Data set
The data set includes the multi-frame images sampled from the driving video. The resolution of the video is 720 pixels where the vertical axis is 720 pixels and the horizontal axis is 1280 pixels. The frame rate is 30 fps.
The images are analyzed. However, the quality of part of images is poor because of lighting and brightness. This also illustrates the importance of lane prediction. The sample rate of the training images is very important. If the rate is higher, the continuous images might be very similar, which is useless for the model. Therefore, only 1 out of 10 images is selected into the training set.
The binomial curve y ¼ ax 2 þ bx þ c, where a, b, and c are coefficients of the curve, is used to fit the vehicle lane. Therefore, the number of training images with different coefficient labels could be counted. The label of the second coefficient b is studied, and the results are shown in Figure 5, where the horizontal axis stands for the coefficient labels and the vertical axis stands for the image number. It is obvious that most of the images are with the coefficient label "0." This shows that lanes are lacking of diversity.
To improve the recognition performance of the lane detection model, the training data set should be extended. That is, more images with the curve lane should be added into the training set. First, more extra images with curve lane are collected from video. Second, the images with less number are collected. Moreover, these images are processed to generate some new images. For example, by rotating the images, the images with curve lane would be extended. The extended image numbers with different coefficient labels are shown in Figure 6. The proportion of curve lanes has been significantly improved.

Experimental results
Traditionally, the computer vision methods could be used to detect lanes. The steps are as follows: (1) Moreover, the accurate metric to evaluate the lane detection algorithms could be defined as follows where the TP is true positive value, TN is the true negative value, FP is the false positive value, and FN is the false negative value.
The experimental results are shown in Figure 7, where the horizontal axis stands for the video time (in seconds) and the vertical axis stands for the accuracy value at that time.   where y min < y < y max is the region used to measure the lane deviation.
Similarly, the two lane detection algorithms are evaluated using LPD metric. The experimental results are shown in Figure 9 The third metric is to evaluate the computation efficiency and detection accuracy. This metric Q CEA could be defined as follows where ACC is the accuracy value mentioned above and FPS is the number of the frames recognized in each second. Evenly, there are about 4.5 frames recognized by OpenCV method, and there are about 30 frames recognized by CNN method. The Q CEA values are shown in Figure 10, where the horizontal axis stands for video time (in seconds) and the vertical axis stands for the Q CEA values at each time step. The results show that the CNN method outperforms the OpenCV method considering both accuracy and efficiency. Moreover, the CNN method has a higher recognition efficiency in the case of the same accuracy value with OpenCV method. From the above experiments, the following conclusions could be obtained. The deep learning-based method outperforms the traditionally OpenCV-based method. Especially, when the driving environment is challenging because of the brightness and lightness.
In general, the types of the roads could be divided into the straight road and the curved road. Therefore, the expectation line should be straight or curved. In this article, these two lines are extracted from the detected lanes. Moreover, LSTM is used to predict the future trajectories of the lines.
The expectation line is the middle line of the detected lane, and it is constructed by 100 data points. In both the straight case and the curved case, the first 90 data points are used as training data set, and the last 10 data points are used as test data set. The splits of the data points are shown in Figures 12(a) and 13(a), where the horizontal axis stands   for the successive time steps and the vertical axis stands for the data point position.
The detected lane with expectation line is shown in Figure 11. In this figure, the expectation line is the middle of the two detected lanes. The structure of LSTM is as follows: an LSTM layer with 50 hidden states, a dropout layer, and a dense layer. Moreover, the "ReLU" activation function is applied. The predicted results are respectively shown in Figures 12(b) and 13(b). The r2_score is used to evaluate the prediction result, and the values are respectively 0.67 and 0.71 in this experiment. The definition of r2_score is as follows where n samples is the number of the samples, y i is the ith sample, y 0 i is the ith prediction result, and y is the mean of y.
The deviation angle which is the angle deviating from the center line is used to evaluate the effectiveness of LSTM. Total five rounds of experiments are carried out. In each round, the average deviation angle is calculated. The results are shown in Figure 14, where the horizontal axis stands for five rounds and the vertical axis stands for the values of average deviation angle. Obviously, the average deviation angle values in the case of LSTM prediction are lower than those of without LSTM prediction. This shows that there are much more deviation in uncertain environments, such as lane missing and calculation timeconsuming. The results demonstrate the effectiveness of LSTM prediction during the process of driving.

Conclusions
The decision-making component is the core part of autonomous driving. Several studies have worked on the    decision-making methods, such as the data fusion methods based on the perception data by onboard sensors. Recently, imitation learning is also one of the important research directions for autonomous driving.
In this article, the idea is motivated by the human drives a car in a lane. Both the two sides of the lane and his or her expected driving directions are important for the safe driving. This experience could be transferred to autonomous vehicle for its robust driving. First, the CNN method is applied to detect lanes in video frames captured by camera. Second, the expectation line is defined to direct the driving direction. Finally, LSTM-based method is used to predict the future trajectory of the expectation line to handle the challenges including the lane missing. The experimental results have demonstrated the effectiveness of the proposed method.
In the future, the dynamic model of autonomous vehicle would be further investigated. Moreover, the driving environment is uncertain in real traffic. Therefore, the proposed method would consider much more environment uncertainties in the future work. For example, the expectation line prediction method should be improved in the case of obstacles.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: