GB2597701A - Method for robust reorientation of smartphone sensor data for vehicles - Google Patents

Abstract

A method of updating and correcting measurement data from smartphone sensors after the phone has been moved or used whilst driving a vehicle. Data is classified into usage and non-usage segments, and then non-usage segments have their roll, pitch and yaw angles determined to reorient the data. Straight road segments are detected using gyroscope measurements and standstills are detected based on GPS data. When all the non-usage segments have been processed, the operation finishes.

Description

Method for robust reorientation of smartphone sensor data for vehicles State-of-the-art Smartphones are ubiquitous. Most drivers have their smartphone in their vehicle while driving. They are equipped with devices providing powerful sensing, processing, and communication capabilities. This enables us to deploy smartphones in modern transportation technologies to collect driving data. Such data can be exploited to get insights regarding the safety-relevant patterns as well as the driving behaviours of the drivers. However, to utilize the sensory data provided by a smartphone properly, it must be reoriented with respect to the vehicle frame of reference. Reorienting the smartphone relative to the vehicle reference is a challenging task as the orientation of the smartphone can vary at any time during the trip due to external factors like user activities or dispositioning of the phone because of harsh brakes.

In the intelligent transportation applications literature, the most basic approach for orientation is to position the smartphone in a predetermined orientation fixed on the dashboard [1] or on another surface with a holster [2], or tape [3], or placed on the floor of the vehicle in order to make the smartphone have the same orientation with the vehicle [4]. A closer approach to natural behaviour is to allow locating the smartphone in an arbitrary position and orientation in the vehicle and later try to calibrate the orientation of the phone. In this direction, Bhoraskar et. al [5] orientates the accelerometer data by transforming them first from smartphone's coordinate system to the geometric coordinate system, then from geometric coordinate system to vehicle's coordinate system. The authors report high accuracy on correctly detecting potholes over reoriented accelerometer readings. Marenz et. al [7] proposes an approach based on analysing the distribution of the accelerometer data during an acceleration and deceleration phases. They cluster the acceleration and the deceleration data according to the directions along which the measurements are seen to indicate. This helps them to determine the lateral and horizontal plane of the vehicle.

As the orientation of the smartphone can change over time during the drive, the adaptively reoriented data can be more reliably used in further analyses. Paefgan et. al [8] for example assumes a fixed position during the recording and requires a calibration phase before any recording starts. Every time the orientation of the phone changes, recalibration is necessary. There exist some studies in the literature, which already propose algorithms dealing with changing positions of the smartphone [6], [9], [10], [11]. It is done by either observing the gyroscope [9] or continuously estimating the angles and compare with the previous angles, if the difference is above a threshold [6], [10], [11]. As to how the angles were obtained, the methods slightly differ. Khaleghi et. al [9] used averages of the pitch and roll angles over the recent history and a Gaussian mixture model using the magnetometer heading and the GPS heading to obtain the yaw angle. Padmanabhan et. al [6], [10] on the other preferably extracted the angles for pitch and roll during standstills or segments of constant speed, the mean over a window was just added as a fallback. The yaw angle was calculated by searching for acceleration or deceleration events in the data using GPS without any significant curve in its path. Cordova et. al [11] obtained the angles by applying principal component analyses (PCA) to the measured data.

But all these described methods do not consider the fact, that environmental influences can also contribute to changes in the orientation. Especially changes in the inclination of the road can happen very often and can cause measurable deviations to the correct angle. A study [12] in the physical activity identification domain reports a method that orientates the accelerometer data from an arbitrary position to a predetermined three-dimensional position in order to improve accelerometer-based activity (standing, sitting, lying, walking) identification. They apply quaternion rotation transformation to the raw data to correct accelerometer axis orientation. The method they propose does not provide high accuracy on accelerometer-based activity identification. Moreover, it is not robust to the changes over time on the smartphone orientation.

References [1] Eriksson, J., Giroid, L., Hull, B., Newton, R., Madden, S., Andbalakrishnan, H. The Pothole Patrol: Using a Mobile Sensor Network for Road Surface Monitoring. In the sixth Annual International conference on Mobile Systems, Applications and Services (MobiSys 2008) (Breckenridge, U.S.A., June 2008).

[2] Fazeen, M., Gozick, B., Dantu, R., Bhukhiya, M., Gonzalez, M.C.: Safe driving using mobile phones. IEEE Trans. Intel!. Trans. Syst. 13(3), 1462-1468 (2012) [3] Douangphachanh, V., Oneyama, H.: A study on the use of smartphones for road roughness condition estimation. In: Proceedings of the Eastern Asia Society for Transportation Studies (10), 1551-1564 (2013) [4] Gonzalez, L.C., Martinez, F., Carlos, M.R.: Identifying roadway surface disruptions based on accelerometer patterns. Latin Am. Trans. IEEE (Revista IEEE America Latina) 12(3), 455-461 (2014) [5] Bhoraskar, R., Vankadhara, N., Raman, B., Kulkarni, P. Wolverine: Traffic and road condition estimation using smartphone sensors. In Communication Systems and Networks (COMSNETS), 2012 Fourth International Conference on (jan. 2012), pp. 1-6 [6] Mohan, P., Padmanabhan, V. N., Ramjee, R. Nericell: rich monitoring of road and traffic conditions using mobile smartphones. In Proceedings of the 6th ACM conference on Embedded network sensor systems (New York) NY, USA, 2008), SenSys'08, ACM, pp. 323-336.

[7] Marenz, M., Meissner, C. 2020 March 19, System and method for determining accelerometer orientation, WO 2020/05344441.

[8] Paefgen, J., Kehr, F., Zhai, Y., Michahelles, F.: Driving behavior analysis with smartphones: Insights from a controlled field study. Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, MUM 2012. 36:1-36:8. 10.1145/2406367.2406412.

[9] Khaleghi, B., El-Ghazal, A., Hilal, A. R., Toonstra, J., Miners W. B., Basir, 0. A.: Opportunistic calibration of smartphone orientation in a vehicle, 2015 IEEE 16th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM), Boston, MA, 2015, pp. 1-6, doi: 10.1109/WoWMoM.2015.7158210.

[10] Padmanabhan, V., Ramjee, R., Mohan, P. System for sensing road and traffic conditions, WO 2009/099680 [11] Cordova, B., Sahoo, S.: Methods and systems for sensor-based vehicle acceleration determination, US 10,067,157 B2 [12] M. D. Tundo, E. Lemaire and N. Baddour, Correcting Smartphone orientation for accelerometer-based analysis, 2013 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Gatineau, QC, 2013, pp. 58-62.

Description of the invention

For a better understanding of the method Fig. 1 showing the flowchart of the invention and a list of definitions are provided.

The most relevant definitions in the description of the invention comprise: * Segments: time history of recorded smartphone data * Smartphone: mobile device capable of measuring and providing position data (via GPS) and six degrees of freedom motion data (via 3D gyroscope and via 3D accelerometers) * Smartphone usage segments: segment where the smartphone is actively operated by driver/passenger * Non-smartphone usage segments segment where the smartphone is not actively operated by driver/passenger * Curved road segments: segments classified due to significant gyroscope measurements caused by curves * Straight road segments: segments classified due to significant gyroscope measurements caused by straight road sections * Standstill: segments with zero velocity recorded of the vehicle, caused by non-moving vehicle * Euler angles: a method for describing rotations of a rigid body in a 3D coordinate system A smartphone in a vehicle could be in an arbitrary orientation with respect to the vehicle. This orientation can change over time as the smartphone is moved around or rotated. One aim of the invention is to solve the challenge of accurately estimate the Euler angles of rotation for each axis to virtually align the Cartesian reference frame of the smartphone (represented by xp, y" zp) with the Cartesian reference frame of the vehicle (represented by x" y" zv). In the following, we describe a method for obtaining the Euler angles. The presented method provides an accurate estimation of the Euler angles and is robust to the orientation changes of the smartphone over time during the drive. The flowchart in Fig. 1 shows the phases and the individual steps of the proposed method.

The aim in step 1 is to find the segments in the data where the smartphone remains in a fixed position and orientation. To find these segments, the method first identifies the segments where the smartphone is moved or rotated, then it eliminates the identified segments and takes what remained as the segments with no move or rotation. These movements usually happen fast and involve rotations; therefore, they can be detected by using the gyroscope of the smartphone. This step is done by summing the measurements of all three axes of the gyroscope and comparing the result with a threshold. This threshold is used to distinguish between the changes in the orientation caused by external factors (e.g. curves, changes in the road inclination) and caused by active usage of the smartphone. It is obtained from the data segments, where the phone was used under varying external conditions.

The steps 2-11 are repeated until each segment with no move or rotation in the data is reoriented to the same reference frame. In step 2, the method selects an unprocessed segment with no move or rotation. As the segments can be treated independently, they can also be processed in parallel.

At the time range where the vehicle is in a steady motion the accelerometer measures only the gravity vector along the axis zv. In step 3 the method makes an approximation of the gravity vector and identifies the axis zi, accordingly. The approximation of the gravity vector is [mean(aPr),mean(ayP),mean(azP)if where aPr, ayP, ayP are the accelerometer measurement vectors in the reference frame xp, yp, zp.

An approximation of the roll (r.p1) and pitch (01) angles are calculated with the following relations: (pi = arcsin 7 mean(q) vimean(aP)2 + mean(aP'2 01 = arcsin mean (a) p \ 2 \\ITT/CO.771.0.x) mean(aP)2 where mean(aP,), mean(ay),mean(azP) are the means over aP, , ayP, azP.

Based on the estimated pitch and roll angles, the method provides a reorientation in step 4 by applying the following relations: cos(01) 0 _sin(01) a aty = sin(p1) sin(01) -cos(q1) sin(p1)cos (0,) aPyPx al cos(q1) sin(01) sin(w1) cos(w1) cos(01) After this first reorientation tep, the plane spanned by the x and y axes of the cu rent accelerometer data segment of the reference frame of the smartphone becomes parallel to the plane spanned by the x and y axes of the reference frame of the vehicle. Therefore, the accelerometer data in the current segment is partially reoriented, as the correct heading of the z-axis is still missing. In the next steps, the new reference frame of the smartphone is referred to as the intermediate reference frame (represented by xi, yi, To estimate the yaw angle (0), the method utilizes the highest force measured (e.g. the force measured during braking) by the accelerometer, which could be better detected in straight road segments. For this reason, before the yaw (ip) estimation the method first detects the curved road segments and eliminates them. The curved road segments can be detected by using the reoriented gyroscope measurements. Similar to the reorientation of accelerometer data in step 4, the gyroscope measurements can be reoriented in step 5 with the following relation: = cos(p1) sin(01) g + sin(cpi)dy + cos(90cos(e1)d where gP,,gyP,d are the gyroscope measurement vectors in the reference frame xp, yp, zp and gzu is the gyroscope measurements in the reference frame z. The curved road segments are detected by checking if the z-axis of the reoriented gyroscope measurements is above a certain threshold. This threshold is used to reduce the effect of the noisy measurements of the gyroscope. It is obtained from the segments, where curves are driven with different radii and speeds. Then, the detected curved road segments are eliminated from the accelerometer data, and the remained straight segments are used to estimate the yaw angle (t.P) in the next step.

In step 6 the method searches over all the straight road segments of the accelerometer data for the moment where the accelerometer measures the highest force. This force gives a strong indication of the yaw angle. It is calculated with the following relation: ay' 11, = arctan al} where axt and ayt are the measurements of the accelerometer in the intermediate reference frame of the x and y axes at the time point where the highest force measured.

Based on the estimated yaw angle, in step 7 the accelerometer data is reoriented again as follows in a similar way as in step 4: a'31, av [cos(p) sin 040 oi ce" ac aiz [ z = sia(q)) -cos(4) 0 0 0 1 Although the accelerometer data has been rotated w.r.t all the th ee Euler angles determined so far to get aligned with the vehicle frame, the method still needs to compensate disorientation stemming from changes in the inclination of the road.

Similar to the determination of 91, in step 8 the method applies the following relation to obtain a new roll angle 92: 92 = arcsin mean(a) vimeart(aly')2 + mean(a)2/ where me an(aD, mean(c4,7), mean(a) are the means over ct, a.131" a. The only difference is that data during curve segments are excluded from the calculation.

Step 9 involves searching for standstill segments over avx and avy in the straight road segments by observing the rate of change within a time window. Standstill segments are used in the next step to obtain a new pitch angle.

Caused by the approximation of the pitch angle in step 3, the whole segment has now a constant pitch angle, which is just a mean of the overall encountered pitch angles. Especially due to changes in the road inclination, changes in the pitch angle can occur quite often.

Reorientation with an inaccurate pitch angle leads to an incorrect split of the components of the gravity vector in x,, and zy. The actual gravity force is out of the range of a typical acceleration, however as the component of perceived gravity force in xv might be in the range of typical acceleration, it could be perceived as a typical driving maneuver. Nevertheless, it is possible to distinguish the component of the perceived gravity force in xv from a forward acceleration by looking into short segments, and therefore it is possible to use the component of the perceived gravity force in xv to identify the error in pitch angle.

By integrating the forward acceleration and comparing the result with a known speed difference (obtained from G PS), it is possible to reduce the impact of changes in the road inclination to the pitch angle (step 10). By calculating drift d in the speed, which is caused by the component of the perceived gravity force in x" it is possible to determine a more accurate time course of the pitch angle. d is calculated as follows: (2 d= ($,62" -s) -41, L, where sct2ps,sctip5 are two GPS speed measurements taken in consecutive timepoints t1 and t2 (t1 < t2). This determination is repeated over all ex and for each consecutive GPS speed measurements. As the GPS sensor is error-prone at low speeds, (42,5 -sbips) is set to Din standstill segments. Given the drift d and the accelerometer measurements of the z-axis in the vehicle frame azv, the new pitch angle e2can be determined with the following relation: ( 02 = arcsin jd2 + After obtaining these correction angles, we can reorient av for the last time 11.

[avx-"d v end ay-av_encl z All steps of the method can be implemented as hardware realisation on elect onic control units with access to smartphone data.

The proposed method is adaptive to the orientation changes over time and robust against road inclinations. It enables to distinguish between lateral and forward accelerations 8-10. It uses the forward acceleration and the GPS data to determine the pitch angles over each window ranged in two consecutive GPS measurements, as the pitch angle changes quite often over time and is sensitive to the road inclinations.

cos(02) 0 -sin(O2) = sin(p2) sin(02) -cos(92) sin(p2)cos (02) a; [ COS42)sin(02) sin(92) cos(92)cos(e2) (4

Claims (1)

1. Claims 1. Method for reorienting measurement data from smartphone sensors during driving operation where the position and orientation of the smartphone is not constant, comprising in a first step a detection of smartphone usage and the classification of the measurement data into smartphone usage segments and non-smartphone usage segments based on gyroscope measurements, in a second step a selection of an unprocessed said non-smartphone usage segment, in a third step the determination of the pitch and roll angels based on the gravity vector, in a fourth step the reorientation of sensor data of said non-smartphone usage segment, in a fifth step the detection of curves and the classification into curved and straight road segments based on gyroscope measurements in said non-smartphone usage segment, in a sixth step the determination of the yaw angle in said straight road segments, in a seventh step the reorientation of the said straight road segment based on the determined yaw angle, in an eighth step the determination of the roll angle based on said straight road segment, in a nineth step the detection of standstills, in a tenth step the determination of the pitch angle based on the GPS speed differences, in an eleventh step the reorientation of said segment based on the determined said angles, in a twelfth step the review if each of said non-smartphone segment have been processed and either finish the method in a thirteenth step if all said non-smartphone segments have been processed or continue with the second step.