GB2528877A

GB2528877A - Method and system for detecting a person driving a vehicle while using a mobile computing device

Info

Publication number: GB2528877A
Application number: GB1413666.7A
Authority: GB
Inventors: Matjaz Gams; Hristijan Gjoreski; Mitja Lustrek
Original assignee: Institut Jozef Stefan
Current assignee: Institut Jozef Stefan
Priority date: 2014-08-01
Filing date: 2014-08-01
Publication date: 2016-02-10
Also published as: GB201413666D0; SI24796A

Abstract

In a method and system for detecting a person driving a vehicle while using a mobile computing device, first and second movement patterns are detected by means of the mobile computing device, said first movement pattern being attributable to a movement of a vehicle and said second movement pattern being attributable to a person using the mobile computing device. A relation is established between said first movement pattern and said second movement pattern, and based on said relation it is determined whether said person is driving said vehicle while using said mobile computing device. Detecting said first movement pattern and/or second movement pattern may comprise a step of detecting a linear and/or gravitational acceleration by means of an acceleration sensor unit. The first and second movement patterns can be any type of movement or motion detected by the mobile computing device, such as a single isolated movement or a sequence of movements that may be characteristic of a driving scenario or a person using the mobile phone while driving.

Description

Method and System for Detecting a Person Driving a Vehicle while Using a Mobile Computing Device

Field of the Invention

The invention relates to a method and system for detecting a person driving a vehicle while using a mobile computing device by making use of sensor data provided by said mobile computing device.

Background of the invention and state of the art

Studies show that drivers often use their celiphones or other handheld communication or computing devices while driving, and that traffic safety is heavily affected because of the resulting distraction. It is sometimes estimated that 80 % of car accidents are caused by distractions, and text messaging on mobile computing devices or speaking on a cellphone while driving are an important factor. Many countries have banned drivers from using their mobile phones while driving, or have limited the use of mobile phones while driving to hands-free equipment. The police force usually relies on visual surveillance for enforcing this ban.

Technical solutions have also been proposed in the art to sense the use of mobile phones while driving a vehicle and to lock out the device in case driving is detected. For instance, US patent US 8,706,143 B proposes to use a mobile computing device with a motion sensor and a camera. The motion sensor detects if the motion is above some threshold, which is taken as an indication for driving, and the camera is used for scenery analysis to sense if the person using the device is driving the car. A lock-out mechanism may be employed to disable the mobile computing device, if it is detected that the person using the mobile computing device is driving the vehicle. However, the approach suggested in US' 143 is prone to errors and can be easily sidetracked or made obsolete by, for instance, putting a finger over the camera lens.

In their research article "You're Driving and Texting: Detecting Drivers Using Personal Smart Phones by Leveraging Inertial Sensors", proceedings of MobiCom 13, September 30th to October 4th, 2013, Miami, Florida, USA, C. Bo et cii. teach to use the acceleration data from a smart phone first to detect the person entering the car, then to detect if the person entered the car from the left or from the right side, and finally to detect if the person sits on the back or front seat using magnetometer data.

In their contribution "You driving? Talk to you later", proceedings of MobiSys 11, June28 to July I, 2011, Bethesda, Maryland, USA, H.L, Chu eta!. in addition analyze movements such as reaching for the seatbelt or pressing the gas medal. However, these systems rely on information about events before using the mobile phone, which are susceptible to errors and can be fooled if wanted.

What is needed is a more reliable and more robust method and system for detecting whether a person is driving a vehicle while using a mobile computing device.

Overview of the invention This objective is achieved with a method and system according to independent claims t and 10, respectively. The dependent claims relate to preferred embodiments.

A method for detecting a person driving a vehicle by using a mobile computing device according to the present invention comprises the steps of detecting a first movement pattern by means of a mobile computing device, said first movement pattern being attributable to a movement of a vehicle, and detecting a second movement pattern by means of said mobile computing device, said second movement pattern being attributable to a person using said mobile computing device. The method further comprises a step of establishing a relation between said first movement pattern and said second movement pattern, and a step of determining, based on said relation, whether said person is driving said vehicle while using said mobile computing device.

The invention is based on the realization that simultaneous driving and using a mobile phone can be most reliably detected by dividing the movement detected by the mobile computing device into a first movement pattern that can be attributed to a movement of a vehicle and a second movement pattern that can be attributed to a person using said mobile computing device.

The first and second movement patterns can be any type of movement or motion detected by the mobile computing device, such as a single isolated movement or a sequence of movements that may be characteristic of a driving scenario or a person using the mobile phone while driving.

In particular, said second movement pattern may be attributable to a person using said mobile computing device while driving said vehicle.

The steps of detecting said first movement pattern and detecting said second movement pattern may not necessarily be based on different signals. Rather, the movement patterns can be detected from the same raw data provided from the mobile computing device, such as 3-dimensional accelerometer data.

The method according to the present invention establishes or detects a relation between said first movement pattern and said second movement pattern. The inventors found this an important advantage over the prior art techniques, which rely on one or several parameters in isolation. The relation may comprise a correlation of said first movement pattern and said second movement pattern, in particular a time correlation. For instance, the relation may comprise a first movement pattern attributable to the movement of a vehicle and a second movement pattern attributable to a person using said mobile computing device appearing simultaneously, or with time overlap, or one after the other in close temporal succession.

As an example, driving and using a mobile computing device may be detected only if' the second movement pattern indicates that the user holds the mobile computing device and the user's movements are characteristic movements when driving, and at the same time the first movement pattern indicates that the vehicle reacts according to the movements in the second movement pattern.

As another example, the movements of a user when operating his mobile computing device may be more erratic than usual in case he is driving a vehicle at the same time, since he is unable to fully concentrate on the mobile computing device. If such erratic movement is detected as a second movement pattern in relation with a first movement pattern that indicates a movement of the vehicle, the method according to the present invention may reliably determine that said person is driving said vehicle while using his mobile computing device.

Hence, basing the determining step on an established or tested relation between the first movement pattern and the second movement pattern allows to significantly enhance the reliability of the detection of a person driving a vehicle while using his mobile computing dcvi cc, On the other hand, if the detection does not yield any movement pattern that can be atifibuted to a movement of the vehicle, or any movement pattern attributable to a person using a mobile computing device, no relation would be established, and the method according to the present invention will reliably conclude that the person is not driving a vehicle by using a mobile computing device. Hence, erroneous detections (false positives) can be avoided.

Preferably, said step of establishing said relation comprises a step of integrating said first movement pattern and said second movement pattern into an integrated movement pattern.

Said step of determining whether said person is driving said vehicle while using said mobile computing device may be based on said integrated movement pattern.

The integrated movement pattern may talce into account correlations between said first movement pattern and said second movement pattern, such as the time correlations described above.

In a preferred embodiment, said step of detecting said first movement pattern and/or said step of detecting said second movement pattern and/or said step of establishing said relation employs context-based reasoning.

The inventors found that context-based reasoning provides a very reliable means of establishing said relation between said first movement pattern and said second movement pattern. In particular, multiple contexts may be defined based on the detected first movement pattern and the detected second movement pattern.

In a preferred embodiment, said context-based reasoning comprises a step of determining a first set of context features attributable to said first movement pattern, and/or determining a second set of context features attributable to said second movement pattern.

Said step of establishing said relation may comprise the step of integrating or aggregating said first set of context features with the second set of context features.

Said first set of context features and/or said second set of context features may comprise statistical time-domain features and/or spatial orientation angles and/or frequency-domain features.

Said step of establishing said relation may be based on combining decisions based on said first set of context features with decisions based on said second set of context features.

In a preferred embodiment, said step of combining employs majority voting and/or plurality voting and/or weighted voting and/or meta-learning and/or stacking, In a preferred embodiment said first movement pattern corresponds to a first context group comprising a first set of context features (or attributes), and said second movement pattern corresponds to a second context group comprising a second set of context features (or attributes).

Said step of establishing said relation may preferably comprise a step of selecting a context feature (or attribute) in said first context group and employing, for said selected context feature (or attribute) in said first context group, a first context-based model with context features (or attributes) from said second context group, and/or selecting a context feature (or attribute) in said second context group and employing, for said selected context feature (or attribute) in said second context group, a second context-based model with context features (or attributes) from said first context group.

Said step of establishing said relation may then further comprise the step of aggregating or integrating said first context-based model and said second context-based model.

In this embodiment, the first movement pattern and the second movement pattern are not split or analyzed independently, but are coherently analyzed in a context-based approach.

Said step of establishing said relation may employ machine learning and/or statistical analysis.

The inventors found these reliable techniques that allow to establish with high precision relations between said first movement pattern and said second movement pattern that indicate that a person is driving said vehicle while using his mobile computing device.

In a preferred embodiment, said step of detecting said first movement pattern and/or said second movement pattern comprises a step of comparing an observed movement or movement pattern or sequence of movements against a predefined reference pattern.

For instance, said reference pattern may be a movement pattern that is typical of a movement of a vehicle, or may be a movement pattern of a person texting a message while driving a car, or holding the mobile computing device while turning the steering wheel.

By identifying typical reference patterns against which the detected movements can be compared, in particular in a context-based reasoning, movements attributable to a movement of a vehicle and movements attributable to a person using the mobile computing device can be reliably determined and distinguished.

In a preferred embodiment, said mobile computing device comprises at least an acceleration sensor unit, in particular an acceleration sensor unit that detects accelerations in three orthogonal spatial directions, and/or additional sensor units such as a gyroscope and/or a magnetics sensor.

Said step of detecting said first movement pattern and/or said second movement pattern may comprise a step of detecting a linear and/or a gravitational acceleration by means of said acceleration sensor unit.

The inventors found an acceleration sensor unit very useful for reliably determining both movements that are attributable to a vehicle and movements that are attributable to a person using a mobile computing device. Moreover, many conventional mobile computing devices such as smartphones are routinely equipped with an acceleration sensor unit. Hence, these mobile computing devices may be readily employed in the context of the present invention, without requiring extra equipment.

It is a particular advantage of the present invention that it may be employed with existing mobile computing devices and does not rely on extra sensors external to the mobile computing device.

However, additional sensor data derived from sensors that form part of the mobile computing device or from external sensors may alternatively or additionally be gathered and taken into account to enhance the reliability.

For instance, some mobile computing devices are provided with gyroscopes, and gyroscope data may be taken into account for detecting said first movement pattern and/or for detecting said second movement pattern.

Sensors whose data may additionally or alternatively be taken into account for detecting said first movement pattern and/or for detecting said second movement pattern may comprise a magnetometer and/or an optical camera.

In response to a determination that said person is driving said vehicle while using said mobile computing device, different measures may be taken, For instance, a warning notification may be given to said person, or the mobile computing device may be locked out for further use for a predefined period of time. Alternatively or additionally, a public authority may be notified, The invention further relates to a system for detecting a person driving a vehicle while using a mobile computing device, said system comprising a first detecting means for detecting a first movement pattern, said first movement pattern being attributable to a movement of a vehicle, and a second detecting means for detecting a second movement pattern, said second movement pattern being attributable to a person using a mobile computing device. Said system further comprises a computing means for establishing a relation between said first movement pattern and said second movement pattern, wherein said computing means is adapted to determine whether said person is driving said vehicle while using said mobile computing device based on said relation.

Preferably, said mobile computing device comprises said first detecting means and/or said second detecting means and/or said computing means. In particular, said mobile computing device may comprise the entire system.

In a preferred embodiment, said first detecting means and/or said second detecting means comprises an acceleration sensor unit adapted to detect a linear and/or gravitational acceleration.

Said mobile computing device may be a mobile phone.

Said system, in particular said computing means, may be adapted to implement a method with some or all of the features described above.

The invention further relates to a computer program product comprising computer-readable instructions such that, when said computer readable instructions are executed on a computing device, in particular on a mobile computing device, cause said computing device to implement a method with some or all of the features described above.

Said computing device may be a system with some or all of the features described above.

Detailed Description of Preferred Embodiments

The features and numerous advantages of the present invention will be best apparent from a detailed description of preferred embodiments with reference to the accompanying drawings in which, Figure 1 is a conceptual diagram of a driver in a vehicle holding a handheld computational device according to an embodiment of the present invention; Figure 2a illustrates the movements made by a driver turning the steering wheel and holding a mobile computing device according to an embodiment of the invention; Figure 2b shows the corresponding movement of the car in response to the driver turning the steering wheel; Figure 2c illustrates a reference system as used in the example of Fig. 2a and Fig. 2b; Figure 3 is a conceptual diagram that illustrates the steps of deterniining whether a person is driving a vehicle while using a mobile computing device by means of context-based reasoning according to an embodiment of the present invention; and Figure 4 illustrates the decision-making according to the example of Fig. 3 in a flow diagram in additional detail.

Embodiments of the invention address the need for improved methods of detecting whether a driver of a vehicle at the same time uses a mobile computing device (HCD). The driver may be a driver of any kind of vehicle, including cars, trucks, ships, or aircrafl. The MCD can be a mobile phone, a laptop computer or tablet computer, or a personal digital assistant (PDA).

Figure 1 is a schematic illustration of a driver 10 sitting in the driver's seat (not shown) of a car (not shown) and holding the steering wheel 12 with his left hand, With his right hand, the driver holds a mobile computing device 14, such as a smartphone, to his ear to make a phone call.

The smartphone 14 and some of its components inasmuch they are relevant to the implementation of the present invention are shown in the enlarged sketch on the right-hand side of Figure 1. The smartphone 14 comprises a three-axes accelerometer 16 as well as a processor unit 18 coupled to the accelerometer 16. The processor unit 18 is in turn coupled to or triggers a security application 20, Many smartphones 14 now routinely come equipped with a three-axes accelerometer 16, and it is a particular advantage ofthe present invention that it may exclusively rely on internal sensors and does not require any additional sensor equipment. In the context of the present invention, the three-axes accelerometer 16 can be readily employed as a detecting means for detecting accelerations of the smartphone 14 in three orthogonal spatial axes. The accelerations may be due to movement of the smartphone 14, either as a result of movement of the vehicle or as a result of movement of the driver 10 holding the smartphone 14.

The processor unit 18 may be the standard build-in processor unit of the smartphone 14, but may be programmed or adapted to employ an algorithm for context-based reasoning to process the measurement data provided by the accelerometer 16. As l1 be described in further detail below, the processor unit 18 processes the measurement data provided by the accelerometer 16 to detect a first movement pattern that is attributable to a movement of the vehicle and a second movement pattern that is attributable to a person using the smartphone 14 while driving.

Movement patterns may be single isolated movements, are sequences of movements. In particular, the processor unit 18 may comprise a buffer unit (not shown) and a memory unit (not shown). The memory unit may be adapted to store a plurality of reference sequences, such as reference sequences corresponding to motion signatures of different driving patterns. The buffer unit may be configured to store an observation sequence formed of a train of sensor outputs from the accelerometer 16 for a predetermined period of time. The processor unit 18 compares the sensor outputs in the buffer against the reference sequence stored in the storage unit.

Employing context-based reasoning, the processor unit 18 then establishes a relation between the first movement pattern and the second movement pattern and determines whether the driver uses the smartphone 14 while driving.

In case of a positive detection of using the smartphone 14 while driving, the processor unit 18 may trigger the security application unit 20 to either raise an alarm or to disable the smartphone 14.

The inventors found that the movements sensed by a smartphone 14 that is used by a driver 10 are generally different from the movements that would be sensed by the same smartphone 14 used by a passenger that is not driving. The rationale behind is that the driver 10 makes unique movements which are characteristic only for persons driving a vehicle.

Figure 2a shows an example of the driver 10 holding the smartphone 14 in his right hand while at the same time turning the steering wheel 12 to the right with his left hand, As a result of the turning of the steering wheel 12, the car will turn to the right, as schematically shown in Figure 2b. In particular, the turning will lead to a tangential velocity Vt and to a centripetal acceleration ac directed towards the axis of the turn, as indicated by arrows in Figure 2b, Figure 2c shows the respective coordinate system in which the velocities and accelerations are measured in the example of Figures 2a and 2b. But it is understood that the reference system is completely arbitrary and is shown in Figure 2c for illustration purposes only.

However, the movements of the turning of the car will usually not be the only accelerations that are detected in the accelerometer 16 of the smartphone 14. Even if the driver 10 uses only his left arm to turn the steering wheel 12, his head and right arm holding the smartphone 14 will usually invariably move slightly in the process, resulting in further accelerations detected in the accelerometer 16. For instance, as indicated in Figure 2a, the driver 10 may move the smartphone 14 up and down or left and right, as indicated by arrows in Figure 2a; however, in general each movement is a specific 3D pattern regarding said method. These are movements that can be detected to distinguish a driver from a non-driver, in particular when they appear in conjunction th, or simultaneously with a turn of the car to the right (as indicated in Fig. 2 b).

Methods for detection of driving by evaluating accelerometer data are generally well known in the art, such as in United States patent US 8,706,143 Bi. Such methods can generally be used in the context of the present invention.

However, the inventors found that superior results can be achieved by separating and distinguishing between movement patterns that are due to the movement of the vehicle and movement patterns that are due to additional movements of the driver 10 holding the smartphone 14 while driving. A reliable detection of a user driving while using his smartphone 14 can then be made by establishing a relation between these two movement patterns, If the movements caused by the vehicle are absent, the movements of the HCD 14 are clearly not caused by steering, and hence one would conclude that the user of the HCD 14 is not driving.

lf on the other hand, the movements caused by the user are absent, the user is probably not steering, and hence not driving the vehicle. Further analysis of the movements of the HCD takes into account that, if the driver 10 is using the HCD 14 while driving, both his movements and the movements of the vehicle are more erratic than normal, because the driver 10 cannot fully focus on either task. This information may be used as context for the current situation and can be analyzed by employing context-based reasoning, as will now be described in additional detail with reference to Figures 3 and 4.

In the sense of the present invention, context may be understood to denote any information that characterizes the circumstances in which an event occurs. A general reference for this notion may be found in the research article by A. K. Dey et at, "The Conference Assistant: Combining Context Awareness with Wearable Computing", Proceedings ISWC, 1999.

As illustrated in Figure 3, in a first step the processor unit 18 extracts multiple contexts from the sensor data provided by the accelerometer 16. Context-based reasoning about the motion sensor data allows to detect if the person that is using the smartphone 14 is at the same time driving the car. The minimal context information for the system may comprise (i) the movements of the smartphone 14 that are caused by the user while using the smartphone 14, such as during calling, texting etc., and (ii) the movements of the smartphone 14 caused by the vehicle, such as accelerations, turns, road bumps, driving patterns etc. Several context features may be extracted that quantitatively describe both types of movements.

The flow diagram of Figure 4 shows the corresponding vectors of context features CErn, CFu2,..., CFUN that are attributed to the motion due to the user's movements and the vector of context features CFvi, CFv2 CFwvr that are attributed to motions due to the vehicle's movement.

Motion due to the user's movements while driving are usually unplanned, sudden and fast, because the user is chiefly concentrating on the road and does not pay sufficient attention to the smartphone 14. The context features that can be extracted using the data samples of these movements may include statistical time-domain features (such as mean values, variation, standard deviation, minimum value, maximum value, etc.), sensor orientation angles, frequency-domain features, etc. Additionally, techniques that analyze time-series data can likewise be employed, such as dynamic time warping (DTW).

The movements that are due to the vehicle include the driving pattern of the driver 10: accelerating, breaking, taking turns, shifting hand gears, etc. Moreover driving patterns may be analyzed to determine whether the person holding the smartphone 14 is sitting in the front or in the back. For instance, a larger bump may be first observed at the front wheels and only later at the back wheels, which results in acceleration movements that differ slightly if sitting in the front or in the back, Loss of the necessary concentration needed to operate the vehicle due to distraction by the smartphone 14 may also be detected. The context features that can be extracted for this movement data are again statistical time-domain features (such as mean value, 11.

variation, standard deviation, minimum value, maximum value, etc.) sensor orientation angles, frequency-domain features, etc. Again, dynamic time warping can be used, such as to compare erratic movement of the user to a predetermined reference normal driving of the same user.

The inventors found that reliable results can be achieved by taking into account the interplay between both types of movement patterns. In fact, a key sign of driving while using the smartphone 14 is a relation between the movement of the user, such as the movement corresponding to a turning of the steering wheel 12, and the resulting movement of the vehicle, such as turning of the vehicle to the right or left. A corresponding correlation of movements in time is a reliable indicator of a driver 10 that at the same time uses his smartphone 14.

Once the context features have been extracted, for each of the context features the relation between the remaining context features and whether the user is using the smartphone 14 while driving may be modeled using a machine learning or statistical modeling algorithm. This algorithm may employ neural networks, naïve Bayes, decision trees, SVM, random forest or other classification models. If the output of the reasoning is numeric rather than Boolean, regression models such as model trees, Gaussian processes etc. may be used. The training data set for the model may be a subset of the whole training data set which contains only the data samples which have the corresponding value of the context features. Once the subsets of the training data are defined, the models may be trained using a model-learning method. Such model-learning methods are well-known to the ones skilled in the art, and hence a detailed

description will be omitted.

In the final decision-making step, the decisions of all the context models are aggregated in order to arrive at a final decision. The models that are considered in a given particular situation are the ones that correspond to the particular context of the user. The process of combining the outputs (aggregation) may include majority voting, plurality voting, weighted voting, meta-learning or stacking.

As further indicated in Figures 3 and 4, a positive decision that the driver 10 is using the smartphone 4 may trigger an event at the security application unit 20, For instance, the security application unit 20 may issue a warning message, such as a light signal or an acoustical signal, may lock-out the device, or may notify a central authority.

The method according to the present invention may in addition include pre-possessing of the measurement data from the accelerometer 16 that may take place prior to context definition.

The pre-processing may take place in the processor unit 18 and may employ low-pass, high-pass, band-pass or Kalman filters, as well as various calibration techniques. These techniques may be used to extract different types of acceleration and to help separate the acceleration due to movement of the user from the acceleration due to movement of the vehicle.

In the example described above with reference to Figures 1 to 4, the sensor data on which the decision was made is acceleration data provided by a three-axes accelerometer 16, which may measure linear and gravitational acceleration in the three orthogonal axes x, y, z. However, the methods described above can be adapted to other schemes for measuring accelerations. Moreover, the present invention is not limited to the measurement of acceleration data, but may encompass all measurement techniques that are suitable to detect a first movement pattern being attributable to movement of a vehicle and a second movement pattern being attributable to a person using a mobile computing device.

For instance, the method according to the present invention may emp'oy an additional sensor that detects angular velocity, such as a gyroscope. Additionally, the mobile computing device 14 may comprise a magnetometer to detect a magnetic field, a camera means adapted to capture a visuai image of the environment of the mobile computing device 14, a sound sensor such as a microphone, or a localization sensor, such as a global positioning system (GPS) sensor. These sensor data may be taken into account when detecting said first movement pattern and when detecting said second movement pattern, as well as when establishing a relation between said first movement pattern and said second movement pattern. They may enhance the reliability of the decision making, In particular, these additional sensor data may be incorporated as additional context features to enhance the accuracy of the context-based reasoning described with reference to Figures 3 and 4 above.

The description of the preferred embodiments and the figures merely serve to illustrate the invention and the beneficial effects associated therewith, but should not be understood to imply any limitation, The scope of the invention is to be determined solely based on the appended set of claims, Reference Signs driver 12 steering wheel 14 mobile computing device! smartphone 16 accelerometer 18 processor unit security application unit