Physical Activity Recognition Utilizing the Built-In Kinematic Sensors of a Smartphone

Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. Although body-worn sensors are well suited to collect data on activity patterns for long periods of time, users may forget to wear special microsensors. On the contrary, more and more people take smartphone with them almost anytime. At present, most popular smartphones have three built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) which could be utilized to recognize PA. This study utilized three built-in kinematic sensors in a smartphone to recognize PA and found out which features derived from the three sensors were significant to different PA. We used a combined algorithm of Fisher's discriminant ratio criterion and J 3 criterion for feature selection. A hierarchical classifiers system including fourteen classifiers was proposed and employed to recognize fifteen activities. The optimal features derived from the built-in kinematic sensors of the smartphone were selected from 140 features. The results indicated that the accelerometer was significant to PA recognition, while gyroscope and orientation sensor were effective to recognize the change of body posture and detect falls, respectively. The total classification accuracy of 95.03% demonstrated the feasibility of utilizing the built-in kinematic sensors of the smartphone to recognize PA.


Introduction
Physical activity (PA) recognition has recently become important in activity monitoring for the public healthcare. The progressive decline in the PA level due to the adoption of sedentary lifestyles has been associated with the increasing incidence of obesity, diabetes, and cardiovascular diseases [1,2]. In addition, a "long lie" caused by a fall may result in negative psychological and physical outcomes [3]. The fear of falling can lead to a further decrease of physical activity. Thus, a reliable PA recognition system in daily life would not only encourage people to do more outdoor activities but also be an assessment of activities of daily living for chronic treatment.
Body-worn sensors [4,5] are well suited to collecting data on activity patterns for long periods of time. In contrast to other approaches, such as laboratory-based systems [6] or video analysis [7,8], they can be used under conditions of free living with minimal inconvenience to the user [9]. However, users may forget to wear these special microsensors.
At present, smartphone is a fast-growing device which people take with it anytime. Most of state-of-the-art smartphones have the built-in kinematic sensors (such as triaccelerometer, gyroscope, and magnetic sensor), Global Positioning System (GPS), Wi-Fi, Bluetooth, camera, proximity sensor, microphone, and so forth.
The built-in kinematic sensors of smartphones are used to recognize PA, which not only reduces the cost of hardware but also exploits existing communication module and ubiquitous monitor. The previous studies have demonstrated the effectiveness of using a smartphone to detect a fall [10]. Acceleration was measured by two devices, a smartphone and an independent accelerometer, to confirm that fall detection using a smartphone is a feasible and highly attractive technology.
Most of the previous studies [10][11][12] only used accelerometers to research PA recognition. In our earlier study [13], we also utilized accelerometer built in a smartphone to detect falls and raised the alarm by Multimedia Messaging Service (MMS). However, acceleration only reflects the change of PA  in linear acceleration, but angular velocity and orientation information were not reflected by an accelerometer.
There are several studies using the combination of accelerometer, gyroscope, and magnetic sensor. Zhang et al. [14] evaluated a continuous activity recognition system, using three small wearable wireless sensors and a smartphone to store, transmit, analyze, and update data. Three sensors, each composed of a triaxial accelerometer and a triaxial gyroscope, are attached to the chest and both thighs. The recognition algorithm was developed using a decision tree based on time series data and spectrum analysis. Bahle et al. [15] found a particularly interesting result that adding the magnetic disturbance features significantly improves recognition based on the vector norm of accelerometers and gyroscopes. Chiang et al. [16] measured essential movements of human body through wireless sensor network body motion sensors that involve the accelerometer and gyroscope. Activities are recognized by the fuzzy algorithm. Jeannet et al. [17] used a monitor containing a 3D accelerometer and a gyroscope fixed on the subject's tee-shirt, on the chest. Multiple sensors systems could identify physical activity in a more refined way by recognizing subactivities such as jogging or rowing, due to the fact that a bigger amount of information on the body motion can be collected. However, the more the sensors were employed to record physical activity, the higher the interference of the measuring system with the spontaneous behavior of the user [18]. Therefore, the development of simple, small and light-weight systems to monitor physical activity in daily life should be recommended.
The aims of this study were to utilize the built-in kinematic sensors (triaccelerometer, gyroscope, and magnetic sensor) of a smartphone to recognize PA and find out which features derived from the built-in kinematic sensors were significant to recognize PA. This work will make the assessment of activities of daily living and fall detection more ubiquitous and portable. The rest of the paper is organized as follows: Section 2 provides the method of the PA recognition. The experimental results and discussion are presented in Sections 3 and 4. Finally, Section 5 concludes the paper.

Methods
The block diagram of the main data processing scheme was described in Figure 1. The built-in accelerometer, gyroscope, and the orientation sensor of a smartphone collected information which reflected acceleration, angular velocity, and orientation of PA. And then features were extracted from the information, and the optimal features were selected. Subsequently, all activities were classified, and if a fall was detected, the alarm information about the location of the fall would be sent to the preset person by MMS.

Device and Data Collection.
A smartphone (Samsung I9023 Nexus S, 125 × 63 × 11.2 mm, 129 g, Android operating system version 2.3.6, Samsung, Korea) was placed inside an adjustable band attached to the chest of a subject, as shown in Figure 2. The activity recognition system requires firm attachment of the smartphone to the subject.  The smartphone has built-in triaxial accelerometer (STM KR3DM) with 19.6 m/s 2 maximum range and 0.019 m/s 2 resolution, triaxial gyroscope sensor (STM K3G) with 34.9 rad/s maximum range and 0.0012 rad/s resolution, and triaxial magnetic field sensor (Asahi Kasei AK8973) with 2000 T maximum range and 0.0625 T resolution. An application software of PA recognition was developed and installed on the smartphone. This application measured the acceleration, angular velocity, and orientation of the smartphone.
The standard sensor coordinate system of the smartphone is defined relative to the screen. As shown in Figure 2, theaxis ( ) is horizontal and points to the right. The -axis ( ) is vertical and points up, and the -axis ( ) points toward the outside of the screen. The , , and of smartphone are according to the mediolateral, vertical, and dorsoventral directions of the subject, respectively.
The coordinate system of the accelerometer is the same as the standard sensor coordinate system. The accelerometer measures acceleration along ( ), ( ), and ( ), respectively. The low pass filter with 0.25 Hz cutoff frequency was employed to separate acceleration force to gravity acceleration (GA) and linear acceleration (LA). So was separated into GA and LA , the same as and . The coordinate system of the gyroscope is the same as the one used for the accelerometer. Rotation is positive in the counterclockwise direction. Gyroscope measures the rate of rotation around ( ), the rate of rotation around ( ), and the rate of rotation around ( ) in radian/s. The orientation sensor is software-based and derives its data from the accelerometer and the geomagnetic field sensor. The orientation sensor monitors the position of the smartphone relative to the earth's frame of reference and measures azimuth ( ), pitch ( ), and roll ( ).
is the angle between magnetic north of the earth and the positive of the smartphone. When is aligned with magnetic north, 4 International Journal of Distributed Sensor Networks the degree of asymmetry of the distribution over the window.
the degree of peakedness of the distribution over the window. Interquartile range IR IR = 3 − 1 , where 3 , 1 is the 75th and 25th percentiles over the window, respectively. Percentage of decline PD The percentage of point decline in the entire window.

Features Description Avg
The average value over the window of . GA Med The median value over the window of GA .

LA Std
The standard deviation value over the window of LA .

SK
The skewness value over the window of 3 . K The kurtosis value over the window of . IR The interquartile range over the window of . PD The percentage of decline in the entire window of . the value is 0, and when is pointing south, the value is 180.
is the angle between the horizontal plane of the earth and the positive .
is positive when the positive rotates toward the positive , and the range of values is from 90 degrees to −90 degrees.
is the angle between the horizontal plane of the earth and the positive . is positive when the positive rotates toward the positive , and the range of values is from 180 degrees to −180 degrees. In Android operating system, four different sampling frequencies (fastest, game, normal, and UI) of sensors can be selected. The values of the four frequencies are not constant and depend on the computational workload of the smartphone. The fastest sampling frequency is selected and can reach 50 Hz normally. The acceleration, gyroscope, and orientation signals were sampled at 25 Hz and stored on a Secure Digital (SD) card in the smartphone. Table 1 showed the descriptions of activities performed by a subject. Ten healthy volunteers (6 males, age: 25±5 years, body mass index (BMI): 23.2 ± 2.7 kg/m 2 and 4 females, age: 23 ± 3 years, BMI: 21.5 ± 2.2 kg/m 2 ) participated in the study.

Feature Extraction.
The sliding window approach was employed to divide the sensor signal into smaller time windows (Figure 3). At a sampling frequency of 25 Hz, each window with 50% overlap represents 1.6 seconds. The features were extracted from the sliding windows signals for activity recognition.   Table 1; for example, "R" represents "Running. " (1) Signal Magnitude Vector LA 3 is the signal magnitude vector of linear acceleration. LA 3 can be represented as Likewise, 3 and 3 have used the same computational method.
(2) Tilt angle (TA) of body trunk TA is defined as the angle between the positive and the gravitational vector, as follows: Twenty signals derived from the acceleration, angular velocity, and orientation are listed as follows: , , , 3 , GA , GA , GA , GA 3 , LA , LA , LA , LA 3 , , , , 3 , , , , and TA.
As shown in Table 2, seven kinds of statistical features were defined and described. And all statistical features of each signal were calculated at each window. So 140 features were extracted from the acceleration, angular velocity, and orientation measured by the built-in kinematic sensors of the smartphone. LA Avg represents the average value of LA over the window, and the remaining 139 features are defined similarly (Table 3).

Feature Selection and Classification.
A combined algorithm of Fisher's discriminant ratio (FDR) criterion and 3 criterion was used for feature selection [19]. Firstly, normalize the features to have zero mean and unit variance. Then the features were ranked by the scalar feature selection, which employs FDR criterion and a cross-correlation measurement between pairs of features. FDR is commonly employed to quantify the discriminatory power of individual features between two classes. The FDR is defined as where 1 , 2 are the mean values of features and 1 , 2 are the standard deviation of features in the two classes, respectively. Then the 10 highest-ranked features were selected. And the exhaustive search method with the 3 criterion was employed to select the best combination of two features. The where is the within-class scatter matrix and is the between-class scatter matrix. The respective definitions are where denotes the a priori probability of th class Ci and is the respective covariance matrix of class Ci. Moreover, is the global mean vector. Furthermore, large values of 3 indicate that classifiers have small within-class variance and large between-class distance. A hierarchical classifiers system was shown in Figure 4. According to level of intensity of physical activity, Classifier 1 was used to differentiate between static and dynamic activity. Likewise, Classifier 3 and Classifier 7 utilized the level intensity of activity to recognize high level of intensity of physical activities, such as falls, jumping, and running. In addition, Classifier 2 employed tilt angle (TA) of body trunk derived from accelerometers to identify lying from sitting/ standing. Moreover, Classifier 5 was employed to classify jumping and falls. Falls are said to have occurred if at least two consecutive peaks in the signal magnitude vector above a defined threshold are recorded [20]. According to the orientation, falls were divided into the categories of right-side fall (FR), left-side fall (FL), backward fall (FB), or forward fall (FF).
Furthermore, there were no obvious features to classify some certain activities, such as walking, walking downstairs and walking upstairs. In a classifier, all possible combinations splitting the activities were exhaustively formed, and for each combination its 3 (3) (1) (3) with the largest 3 value was chosen to be the scheme of Classifier 6. Likewise, Classifiers 4, 10 used the same method to choose the combination. Fourteen classifiers for two classes were employed to recognize fifteen activities.

The Features
Selected. The scatter graphs of part classifiers were shown in Figure 5. The symbols of pluses and circles indicated points into the two classes of classifiers, respectively. In addition, Figure 5 pointed out the combinations of selected features resulted in well separated classes in the 2dimensional feature space. As shown in Table 4, twenty-eight well selected features for fourteen classifiers included nineteen features derived from accelerometer, six features derived from gyroscope and three derived from the orientation sensor.
In these three sensors, accelerometer was dominant to PA recognition. In addition, gyroscope served well in recognizing the change of body posture such as transitions, and the orientation sensor was effective to detect falls. As illustrated in Figure 6, can be used to distinguish between forward fall and backward fall. Likewise, can be used to distinguish between left-side fall and right-side fall. Figure 6 illustrated the application of PA recognition implemented in the smartphone when a subject changed the motion from motionless to walking. The green line indicated the SVM value of the accelerometer in real time. Five common activities (jumping, running, walking, lying, and motionless) can be represented by the blue line.

The Recognition Results for All Activities.
We used holdout method [21] to estimate the classifiers, which partitions the data into two mutually exclusive subsets: a training set 8 International Journal of Distributed Sensor Networks  and a test set. A set of experiments was performed with 10 subjects, with each performing a set of 16 different activities as described in Table 1. 600 windows were randomly divided into two parts: a training set (400 samples) and a test set (200 samples). The confusion matrix in Table 5 was a representative of recognition errors for the test set. From the confusion matrix, we can see that walking may be misclassified as walking upstairs or walking downstairs, because these activities contain similar movements and frequencies at the lower limbs. In addition, falls may be recognized as sit-to-lie, because the posture of the two activities were lying at last. The accuracies of static, transitions, dynamic, and falls were 98.75%, 94.625%, 91.8%, and 97.63%, respectively. A total recognition rate of 95.03% achieved that our proposed PA recognition system based on the smartphone could accomplish high recognition rates for the fifteen activities. The application of PA recognition was displayed in Figure 7. The green line indicated the SVM value of the accelerometer real time, and the blue line presented the PA.

Discussion
A PA recognition system based on a smartphone was developed. The optimal features derived from the built-in kinematic sensors of the smartphone were selected from 140 features for each classifier.
In the previous studies [22,23], the recognition rates of walking, walking upstairs, and walking downstairs were low, caused by the similarity of these three activities (Figure 8).
International Journal of Distributed Sensor Networks  Figure 9: The scatter graphs of three activities, employing a combination of GyStd, A3aStd, and G3aStd. Pluses, circles, and dots indicated points from walking, walking upstairs, and walking downstairs, respectively.
In this study, the three classes were well separated. Figure 9 presented the scatter graphs of three activities (walking, walking upstairs, and walking downstairs), employing a combination of Std, 3 Std, and 3 Std. Pluses, circles, and dots indicated points from walking, walking upstairs, and walking downstairs, respectively. The PA recognition system based on the smartphone provided several advantages, such as existing communication module and ubiquitous monitor. The previous studies have used Short Messaging Service (SMS) [10] or Multimedia Messaging Service (MMS) [13] to send information of a fall to a guardian. In an earlier study, we warned falls of the elderly to a guardian by MMS including time, map of suspected fall location, and GPS coordinate. When a fall was detected, an automatic MMS would be sent to preset people. According to the convenient communication module of smartphones, more information of PA could be sent in real time. A remote PA recognition system based on a private cloud platform [24] will be developed in our future work.
The main limitation of our approach is that the smartphone must be worn on the chest. People put phones in the pocket of shirts or pants according to their respective habits, which would cause loose attachment between the body and smartphone resulting in the invalidness of the algorithm eventually. Khan et al. [25] used linear discriminant analysis to recognize PA based on a single triaxial accelerometer that allowed subjects to carry the sensor freely in any pocket without firm attachment. In future work, we plan to develop a PA recognition system based on a smartphone according to wearing habits of users.

Conclusion
For this study, the optimal features derived from the built-in kinematic sensors of the smartphone were selected to recognize PA. The results of selected features suggested the accelerometer was significant to PA recognition, and gyroscope and orientation sensor were effective to recognize the change of body posture or detect falls, respectively. Despite the limitations of this system which requires firm attachment of the smartphone to a subject, the experimental results demonstrated the feasibility of utilizing the built-in kinematic sensors of the smartphone to recognize PA.