Driver emotion recognition of multiple-ECG feature fusion based on BP network and D–S evidence

: Driving emotion is considered as driver's psychological reaction to a change in traffic environment, which affects driver's cognitive, judgement and behaviour. In anxiety, drivers are more likely to get engaged in distracted driving, increasing the likelihood of vehicle crash. Therefore, it is essential to identify driver's anxiety during driving, to provide a basis for driving safety. This study used multiple-electrocardiogram (ECG) feature fusion to recognise driver's emotion, based on back-propagation network and Dempster–Shafer evidence method. The three features of ECG signals, the time–frequency domain, waveform and non-linear characteristics were selected as the parameters for emotion recognition. An emotion recognition model was proposed to identify drivers’ calm and anxiety during driving. The results show after ECG evidence fusion, the proposed model can recognise drivers’ emotion, with an accuracy rate of 91.34% for calm and 92.89% for anxiety. The authors’ findings of this study can be used to develop the personalised driving warning system and intelligent human–machine interaction in vehicles. This study would be of great theoretical significance and application value for improving road traffic safety.


Introduction
With the rapid development of transportation industry, the number of private cars increases dramatically, and the injury and deaths due to traffic accidents continue to increase annually. According to statistics, more than 90% of traffic accidents are due to human factors, either directly or indirectly, 70% of which are caused by vehicle drivers [1][2][3][4][5]. Driving emotion is considered as driver's reaction to a change in the environment, which affects driver's cognitive, judgement and behaviour. For example, drivers have a poor focus on driving in anxiety, increasing the chance of vehicle accident. Therefore, it is of great significance to recognise driver's emotion during driving, to provide a basis for driving safety [6][7][8][9][10][11].
Transportation scholars have explored driver's emotion recognition using different types of physiological data [12][13][14][15]. Wan used the threshold discriminant method to identify driver's tension and calm moods [16], according to face geometric features such as the area ratio between eye and face. Scott-Parker [17] recognised driver's emotion using the frequency, amplitude and power spectrum of audio signals. Kessous et al. [18] identified driver's happiness and sadness, based on gesture features such as the number of body contractions, and the speed and acceleration of hand movement. Minhad and Md Ali [19] applied the factor decomposition model to distinguish driver's emotion, by physiological characteristics including blood flow pulse, skin conductivity, respiratory rate and skin temperature. Wan et al. [20] developed a detection model for angry driving, based on four physiological characteristics of blood volume pulse, skin conduction, δ wave percentage and β wave percentage.
Among physiological signals, electrocardiogram (ECG) was considered as one of the most effective tool to recognise and detect human emotion. Guo [21] used K-means algorithm to identify subject's calm and happy emotions, based on the two ECG features of heart rate (HR) and HR variability. Yu et al. [22] used the three classifiers of K-means, linear discriminant analysis and multi-layer perceptron to recognise the four human emotions of happiness, calmness, anger and sadness, according to the eight ECG features such as time domain and frequency domain. Hyun-Min [23] applied the fast Fourier transform and support vector machine to classify and identify human emotions using their ECG signals.
Since ECG signals generally exhibit strong non-linearity and chaos, several approaches in non-linear analysis are widely used to process ECG data, including fractal dimension, complexity, approximate entropy and maximum Lyapunov exponent. Guo and Li [24] combined the neural network (NN) and Dempster-Shafer (D-S) evidence theory to recognise the motion pattern of lower limbs, using the fusion of the electrical signals on limb surface and hip joint signals. The results show that the proposed method of evidence synthesis is more reliable and accurate in the recognition, than those with a single evidence. Qu used the entropy weight grey correlation and D-S evidence theory to identify fatigue driving behaviour [25], and achieved a high recognition accuracy in highway. Ling et al. [26] proposed a method for monitoring fatigue driving based on D-S theory and fuzzy NN using driver's physiological signals, and obtained pretty good recognition results.
It was observed that ECG is one of the most effective tool to recognise and detect human emotion. Since ECG signals generally exhibit strong non-linearity and chaos, the non-linear analysis combining the fusion of multiple evidence is needed to process ECG data. However, only a few research related to driver's emotion recognition have been conducted, using multiple-ECG signals and considering its non-linear characteristics. This study will use the back-propagation (BP) NN and D-S evidence theory to identify drivers' calm and anxious emotions during driving based on various ECG signal features, including the time-frequency domain, waveform and non-linear characteristic parameters.

Research method
Uncertainties widely exist in human psychological or physiological information. In this study, we will use the evidence theory to increase the uncertainty estimates, because it is generally considered as a powerful tool for epistemic uncertainty analysis.
can be synthesised; if K = 1, propositions A and B appear to contradict each other and cannot be synthesised. Multiple evidence combination rules: M 1 , M 2 , …, M n are the basic confidence intervals on a same recognition framework, and the corresponding elements are A 1 , A 2 , …, A n . The combination formula for n evidence is as below: A 1 is the judgement result, where ε 1 and ε 2 represent the preset thresholds.

D-S basic probability distribution based on BPNN:
In evidence theory, the most critical step is to assign a basic probability for each element based on existing evidence. BPNN has strong self-learning and parallel processing ability. The trained BPNN can basically replace experts on the aspect of knowledge. Therefore, BPNN can be used to assign the basic probability of each element. The structure of BPNN is as follows.
The number of input nodes is determined by the number of input signals. Assuming there are n input ECG signals in timefrequency domain, waveform and non-linear characteristics, so the number of input nodes in the NN is n. The number of hidden nodes is set to be adjustable. The number of output nodes is determined by the types of driving emotions. This study covers only two types of driver emotions: calm and anxiety, so the number of output nodes is 2. The output [1,0] corresponds to calm and the output [0, 1] corresponds to anxiety. The output y i i = 1, 2 of the trained BPNN is normalised as the basic probability of each element, as follows: where m i represents the normalised result of BPNN output, that is, the basic probability of each element; y i represents the BPNN output; and η represents the recognition accuracy of the trained NN for the model library. Here, 1 − η represents the uncertainty of NN for drivers' emotion recognition. The identification process of driver's emotions is shown in Fig. 1 year). Subject's driving propensity was determined by the driving propensity questionnaire [1]. All the subjects were categorised into extroverted drivers. In this study, if a subject drove more than 10,000 km, she would be defined as an experienced drive; otherwise, a novice driver. All the participants drove <10,000, whose average mileage was about 6700 km. Therefore, they were classified into novice drivers. All the subjects were healthy and had no history of emotional psychosis, cardiovascular and cerebrovascular diseases. Before the experiment, they were asked not to take any drugs that affect the nervous system within 1 week, and were asked not to take the foods and beverages (e.g. tea, coffee and wine) that affect the human mental state within 48 h. In addition, they were asked not to do any intensity exercise and to have a good rest. Prior to the experiment, researchers introduced the experimental environment and procedure to participants in detail.

Emotional induction materials: The International Affective
Picture System (IAPS) and the Chinese Affective Picture System (CAPS) were used as emotional induction materials. IAPS is an emotional material, which is generally acknowledged around the world [27,28], and CAPS is an emotional material that adapts to the social and cultural context of China [29,30]. Various anxiety- inducting materials were used in the experiments, including visual materials (e.g. words and pictures, light variation in driving environment), auditory materials (e.g. noisy and irregular sounds), multi-channel materials (e.g. videos and movies), olfactory materials (e.g. cigarettes and durian) and taste materials (e.g. balsam gourd and liquorice). In addition, participants were also asked to finish difficult tasks under stress; as a result, their anxiety emotion was induced. For example, subjects were asked to complete a mobile game or solve math problems in a limited time.
They would receive unexpected punishment if they failed to do. Parts of the anxiety-induction materials are shown in Fig. 2.

Real driving experiment:
In the real driving experiments, the selected route starts from the West gate of Shandong University of Technology, passes through Qingnian Road, Beijing Road, Xincun West Road and Nanjing Road, and ends at the West Gate (as shown in Fig. 3, the total length of 4.426 km). The experiments were conducted on sunny days and favourable road conditions. In addition, an unmanned aerial vehicle (UAV) was used for recording the experimental process. The experimental equipment includes a comprehensive experimental vehicle, jamming vehicle, 32-channel lidar, BTM300-905-200 laser ranging sensor, global positioning system (GPS) high-precision positioning system, SG299GPS noncontact multi-function speedometer, X5000 vehicle recorder, PSYLAB human factor equipment, WTC-1 pedal force manometer, high-definition camera, laptop and UAV. Parts of the experimental equipment are shown in Fig. 4. Screenshots of the real experimental scenes (in Xincun West Road) are shown in Fig. 3.

Driving simulation:
Using real driving experiments to collect data is time-consuming, expensive and difficult to organise. It is difficult to obtain a large amount of real driving experimental data. Driving simulation can be used as a supplement to real vehicle experiment, because it is safe, low-cost and easy to control. The Road Builder and UC-win/Road software were used to construct the simulation-based experiment platforms of the humanvehicle-environment comprehensive road system and the multiperson multi-machine interactive environment, based on the road attributes, traffic volume and other parameters of the field driving experiments. The Road Builder and UC-win/Road software provided by Japanese company FORUM 8 allow users to construct three-dimensional traffic environment and engage interactive experience. The virtual driving experiment equipment are shown in Fig. 5. The wearable wireless ECG sensors, the simulation-based experiment route and street view are shown in Fig. 6.

Experimental process:
Considering participants with insufficient driving experience, high-traffic-volume situations easily result in their emotional fluctuations. Therefore, the weekends from 6:00 to 8:00 am (with low traffic) were selected to conduct the real driving experiments in calm, in order to ensure the participants in calm while driving. The experimental process is shown in Fig. 7. The real driving experiments in anxiety were carried out during morning peak hours of 7:00-9:00 and evening peak hours of 17:00-19:00 from Monday to Friday. The experimental process is shown in Fig. 8.

2.2.6
Assessing the level of the induced anxiety: It is necessary to assess if subjects' anxiety is induced to a certain level of arousal, because too little or too much arousal can adversely affect subjects' performance in experiments. During the driving experiments, the facial expression, action, road conditions, driving speed and pedal strength were recorded in real time with the video monitoring system, speedometer and pedal dynamics instrument.
Subjects were asked to describe their self-perception of emotion in the conversation with experimenters. After the driving experiment, each subject was asked to watch the video immediately and report her emotional experience. The data segments of subjects' anxiety were determined through the Beck Anxiety Inventory (BAI), selfperception, facial expression and behavioural action. The data segments of subjects' calm were determined through the selfperception reporting method. The selected data segments were used for the subsequent processing and analysis. The anxiety induction is regarded to be successful, if subjects have a score of 26 points or more for anxiety symptoms [6], as shown in Table 1. Therefore, only the moderate and severe anxiety were selected for analysis in this study.
2.2.7 ECG time-frequency domain and waveform feature extraction: Each subject was involved in several driving experiments, in which they were induced to feel either calm or anxious. A total of 1327 groups of effective data were obtained. The variables and symbols for ECG signals are shown in Table 2.
Parts of the experimental data are shown in Fig. 9 and (see Table 3) for table of results.
The paired T-test was used to determine whether there is a difference between calm and anxiety in the ECG characteristic indicators, and the results are shown in Table 4.
The results show that there is a significant difference between calm and anxiety in the seven ECG indicators AVHR, AVNN, PNN50, RWAVE, RMSSD, Q and S (p < 0.05). There is no significant difference between calm and anxiety in the other indicators. The results indicate that compared with the calm state, female drivers have a faster HR, a shorter heartbeat interval, a more frequent heartbeat fluctuation, a longer conduction time of the heart chamber and a more obvious manifestation of myocardial in anxiety. Therefore, the four time-domain indicators AVHR, AVNN, PNN 50 and RMSSD, as well as the three waveform where i = 1, 2, …, N, N = n − m − 1 τ.
The correlation dimension D 1 is the most common method for describing the fractal dimension of geometric objects in phase space, which can be used to quantitatively analyse the fractal features of physiological signals. In the solution process, the associated integral C t r is found by where r represents the threshold parameter and H represents the Heaviside function, which is defined as  Table 1 have four levels of induction. The score of each symptom can be expressed as '1 point' means 'none'; '2 points' means 'mild, no major annoyance'; '3 points' means 'moderate, feel uncomfortable but still tolerable'; and '4 points' means 'heavy, can only barely endure'. The total score of 21 symptoms is 15-25 points for mild anxiety, 26-35 points for moderate anxiety and more than 36 points for severe anxiety.
If r → 0, we have

Largest Lyapunov exponent:
The largest Lyapunov exponent is expressed as the average rate of convergence or divergence between two trajectories in phase space. If it is >0, it indicates that the time series has chaotic characteristics. For a time series x 1 , x 2 , …, x t , …, x n , a set of space vectors is constructed by embedding dimension v and time delay τ where i = 1, 2, …, N, N = n − v − 1 τ.
In the reconstructed phase space, its nearest neighbour x j^ of each reference point x j on the trajectory line is defined as where P represents the average cycle length of time series. The distance d j k of each point X j on the basic orbit to adjacent points in k discrete time steps is calculated. For each k, the average value of ln d j i is obtained by where q represents the number of non-zero d j k . The least-square method is used to fit a line X k , and its slope is the largest Lyapunov exponent.

Sample entropy:
Sample entropy is used to measure the complexity and irregularity of a time series. The greater the sample entropy, the more complex the sequence, and the less obvious the periodicity. For the time series x 1 , x 2 , …, x t , …, x N , the sample entropy is determined as follows: (i) Vectors u m i with length m and u m + 1 i with length m + 1 are defined as where i, j = 1, 2, …, N − m + 1, i ≠ j.
(iii) For a given threshold r r > 0 , the number of j meeting the condition d u i , u j ≤ r is n i m . Subsequently, two sequences with length m are randomly selected from the signal, and the unconditional probability of meeting the condition d < r is C i m = n i m / N − m . The average probability is calculated by (iv) Similarly, for the vectors with length m + 1, C i m + 1 r and φ m + 1 r can be determined following the steps above. The sample entropy of the sequence is determined by It can be seen that the value of sample entropy is related to m and r.
In general, with M = 1 or 2, r = 0.1-0.25 Std (Std is the standard deviation of the raw data), the sample entropy would be reasonable. The larger the value of m, the greater the amount of calculation. The larger the value of r, the more loss the detailed information of time series. After considering comprehensively, we chose m = 2 and r = 0.2 Std in the sample entropy algorithm.

Non-linear parameter feature extraction:
According to (4)- (14), the three parameters of ECG signals under calm and anxiety were calculated, the correlation dimension, the maximum Lyapunov exponent and the sample entropy. About 18 sets of sample data are randomly selected for each chaotic parameter, and the calculation results are shown in Fig. 10. The increased level of driver's anxiety is positively associated with increased correlation dimension and largest Lyapunov exponent, which means enhanced randomness, non-linearity and chaos of ECG signals.
It was noted that the calm and anxiety can be distinguished, using the largest Lyapunov exponent or the correlation dimension lines. In this figure of sample entropy, the lines overlap for calm and anxiety. It would be difficult to identify the two emotions using sample entropy. Therefore, the largest Lyapunov exponent and the correlation dimension were selected as the input parameters in the model to identify the non-linear characteristics of ECG signals.

Model calibration
In this study, 108 sets of ECG signal data from driving simulation were selected as training samples and 54 sets of ECG signal data from real driving experiment were used as testing samples. The NN toolbox from MATLAB was used to train a BPNN, combining additional momentum with adaptive learning rate. Through 3287 training sessions, the number of nodes in hidden layer of the three sub-networks was determined. Next, according to the four timefrequency-domain characteristics of ECG signals AVHR, AVNN, PNN 50 and RMSSD, the first sub-network of 4 × 10 × 2 was built. According to the three waveform characteristics of ECG signals RWAVE, Q and S, the second sub-network of 3 × 7 × 2 was established. According to the largest Lyapunov exponent and correlation dimension, the third sub-network of 2 × 5 × 2 was developed. The expected outputs of the network were set to calm [1,0] and anxiety [0, 1]. Parts of the output results of BPNN are shown in Table 5.
According to (2), the normalised outputs of the BPNN were calculated, that is, the basic probability of each element and the uncertainty ratio 1 − η of the NN to driver's emotion recognition. The results are shown in Table 6.

Model results
On the basis of the multi-sources information fusion theory, different emotion recognition rates could be obtained by fusing different ECG signal parameters. Therefore, the time-frequency domain, waveform and non-linear characteristic parameters of the ECG signals were combined in various ways, according to the basic probability determined by the BPNN normalised output after training, as well as Dempster's rule of combination [see (1)]. The threshold values of ε 1 = 0.65 and ε 2 = 1 were selected. The results of driver's emotion recognition using multiple evidence fusion for table of results are shown in Table 7.
From Table 5, it can be seen that before evidence fusion, the first sub-network with the four characteristic parameters AVHR, AVNN, PNN 50 and RMSSD in the time-frequency domain is with the highest recognition accuracy for calm mood after training NN. The recognition accuracy can reach 86.64%. The second subnetwork with the three characteristic parameters RWAVE, Q and S in waveform is with the highest recognition accuracy for anxiety after training NN. The recognition accuracy can reach 86.51%. Tables 5,7 and Table 8 show that the evidence fusion of BP subnetwork can provide a higher emotion recognition ability and a

Conclusion
This study used multiple-ECG feature fusion to recognise driver's emotion, based on BP network and D-S evidence method. An emotion recognition model was proposed to identify driver's calm and anxiety emotions during driving. The main findings of this study are summarised as follows:  (ii) Using multiple-ECG signal, fusion can achieve a higher recognition rate for drivers' emotion than using a single ECG signal.
(iii) After ECG evidence fusion, the proposed model based on BPNN and D-S evidence theory can recognise drivers' emotion, with an accuracy rate of 91.34% for calm and 92.89% for anxiety.
Our findings of this study suggest that multiple-ECG evidence fusion can increase the accuracy rate of driver's emotion recognition. It can be used to develop the personalised driving warning system and intelligent human-machine interaction in vehicles. This study would be of great theoretical significance and application value for improving road traffic safety. Further studies are required to explore different methods to synthesise drivers' ECG features, to recognise drivers' emotion more efficiently. In addition, further studies are also needed to use multiple-ECG signals to detect more types of drivers' emotions.