EP3590071A1

EP3590071A1 - Device for determining the attentiveness of a driver of a vehicle, on-board system comprising such a device, and associated method

Info

Publication number: EP3590071A1
Application number: EP18708630.1A
Authority: EP
Inventors: Frédéric AUTRAN
Original assignee: Valeo Comfort and Driving Assistance SAS
Current assignee: Valeo Comfort and Driving Assistance SAS
Priority date: 2017-03-03
Filing date: 2018-02-23
Publication date: 2020-01-08
Also published as: CN110383290B; US11170241B2; US20200012872A1; WO2018158163A1; FR3063557B1; CN110383290A; FR3063557A1

Abstract

The invention relates to a device (10) for determining the attentiveness of a driver (4) of a vehicle, comprising: an image-capture unit on-board the vehicle (1), said image-capture unit being adapted to capture at least one image of a detection zone (D) located inside the vehicle (1); and an image-processing unit adapted to receive the captured image and programmed to determine the attentiveness of the driver (4) based on the detection of a distracting object in one of the driver's hands inside the detection zone (D). The invention also relates to an on-board system comprising such a device and to an associated method.

Description

Device for determining the attention state of a vehicle driver, embedded system comprising such a device, and associated method

TECHNICAL FIELD TO WHICH THE INVENTION RELATES The present invention relates to a device for determining the state of attention of a vehicle driver.

It also relates to an embedded system comprising such a device. It also relates to a method associated with such a device.

Finally, it finds a particularly advantageous application in the case of autonomous vehicles, including autonomous motor vehicles.

BACKGROUND

It is known to monitor the driver of a motor vehicle with a monitoring device adapted to determine a state of alertness of the driver and in particular to prevent drowsy driving. Depending on the state of alertness determined, the monitoring device alerts the driver to prevent it from getting into a dangerous situation.

Such a monitoring device deduces the state of vigilance of the driver according to behavioral parameters associated with the driver and / or operating parameters of the vehicle. In practice, the behavioral parameters, for example the closing rate of the eyelids or the gaze direction, are obtained by analyzing images of the driver's head, and the operating parameters of the vehicle, for example the parameters relating to an angle of rotation of the steering wheel, at a vehicle speed or the action of the driver on certain keys, are obtained from physical sensors of the vehicle.

However, in some cases it may be necessary to know not only the state of vigilance of the driver, but also his general attention, which is not always possible from the only behavioral parameters associated with the driver's face and / or operation of the vehicle.

OBJECT OF THE INVENTION

In order to overcome the aforementioned drawback of the state of the art, the present invention proposes an on-board device for determining a state of attention of a vehicle driver.

More particularly, there is provided according to the invention a device for determining a state of attention of a vehicle driver comprising:

an image capture unit embedded in said vehicle, said image capture unit being adapted to capture at least one image of a detection zone situated in said vehicle, and

an image processing unit adapted to receive said captured and programmed image for determining the state of attention of the driver, as a function of the detection of the presence of an object of distraction in one of the hands of the driver located in the detection area.

For the purposes of the invention, an object of distraction is an object other than a driving member of the vehicle. It is an object capable of distracting the driver from his driving, and occupying his hand so that said object of distraction prevents the driver from interacting safely with the driving members of the vehicle.

Thus, the device according to the invention makes it possible to determine the state of attention of the driver as a function of a state of occupation of at least one of the hands of the driver among a "busy" state in which the hand holds an object distraction, and a "free" state in which it is free to interact with the driving organs of the vehicle because it holds no object of distraction. When at least one of the hands is in a busy state, the driver is in a lower state of attention than when both hands are in a free state because the driver may be embarrassed to intervene quickly on the drivers of the vehicle if it holds an object of distraction in his hand.

Other nonlimiting and advantageous features of the device according to the invention are the following:

the image capture unit comprises at least one sensor adapted to capture a three-dimensional image of the detection zone, said three-dimensional image including information relating to the distance, with respect to said sensor, from said distracted object and / or said hand located in the detection zone;

the image capture unit comprises at least one sensor adapted to capture at least one image of a first nature comprising a first type of information relating to said distraction object and / or to said hand located in the detection zone , and a sensor adapted to capture at least one image of a second nature distinct from the first nature, comprising a second type information relating to said distraction object and / or said hand located in the detection zone;

the image of first nature is chosen from: a three-dimensional image comprising information relating to the distance, with respect to said sensor, from at least said distraction object and / or said hand located in the detection zone, a two-dimensional image comprising information relating to the luminance of at least said distraction object and / or said hand located in the detection zone, and a thermal image including information relating to the temperature of at least said amusement object and / or said hand located in the detection zone (the image of second nature can also be chosen from the aforementioned image types, while being of a nature distinct from the first nature as already indicated);

the image capture unit further comprises at least one sensor adapted to capture an image of a third nature distinct from said first and second natures, comprising a third type of information relating to said object and / or to said hand located in the detection zone

the image of the third nature is chosen from: a three-dimensional image comprising information relating to the distance, with respect to said sensor, from at least said distraction object and / or said hand located in the detection zone, a two-dimensional image comprising information relating to the luminance of at least said distraction object and / or said hand located in the detection zone, and a thermal image including information relating to the temperature of at least said amusement object and / or said hand located in the detection zone;

the image processing unit is programmed to implement the following steps:

b1) locating said driver's hand in said image received from the image capturing unit,

b2) detecting the presence of said distraction object in said driver's hand in order to deduce the state of attention of the driver;

the image processing unit is programmed to implement step b1) according to the following substeps:

^* detect at least a portion of an arm of the driver, in the image received the image capture unit, and ^* deducing the position, in said image received from the image capture unit, of said driver's hand from the location of detection of the driver's arm portion in the image received from the capture unit of images;

the image processing unit is programmed to implement step b2) according to the following substeps:

^* detect the presence of any object in the driver's hand, and

^* Determining the nature of the detected object in the driver's hand to infer the driver's attention state;

the image processing unit implements step b1) from the image of first nature, and step b2) from the image of a nature distinct from the first nature;

the image processing unit implements the steps b1) to b2) from each image received from the image capture unit, and implements an additional step according to which it determines a confidence index associated with the detection of the presence of the object of distraction in the hand of the driver according to which it deduces the state of attention of the driver;

the image processing unit comprises processing means using a trained neural network to determine the state of attention of the driver from the image of the detection zone captured by the capture unit; images.

The invention also relates to an on-board vehicle system comprising:

a device according to the invention, adapted to determine the state of attention of the driver,

an autonomous driving unit of said vehicle programmed to control driving members of said vehicle independently of the driver, and

a decision unit programmed to authorize the driver to control at least part of the driving members of the vehicle when determining a free state of the hand (or at least one of the two hands, or both hands) of the driver and / or to alert the driver when determining a busy state of the driver's hand, and / or to switch the autonomous driving in a secure mode. Thus, when the vehicle is an autonomous motor vehicle, that is to say whose drivers are controlled independently of the driver, the driver will not be able to resume control, even partial, of the driving members of the vehicle, if occupation status of his hand is determined as "occupied". In case of determination of a busy state of the driver's hand, the autonomous vehicle can be brought by the decision unit of the onboard system to park on the low side (or on the emergency stop band), according to the aforementioned secure mode.

In the case of a manual or semi-autonomous vehicle, the detection of an object of distraction in at least one of the hands of the driver, may for example be followed by a warning of the driver on the risk taken.

The invention finally proposes a method for determining a state of attention of a vehicle driver, according to which

a) an image capture unit embedded in said vehicle captures at least one image of a detection zone located in said vehicle, and

b) an image processing unit receives said captured image and determines the state of attention of the driver, according to the detection of the presence of a distraction object in at least one of the driver's hands located in the area detection.

Advantageously, when the driving members of said vehicle are controlled independently of the driver, there is further provided a step according to which the driver is authorized to control at least part of the driving members of the vehicle in case of determination of a state. free of one of the hands of the driver and / or alerting the driver when determining a busy state of one of said hands of the driver and / or tilt the autonomous driving in a secure mode.

The method that has just been proposed may also optionally include steps such as those proposed above with respect to the device for determining a state of attention of the driver (in particular steps b1) and b2) and the possible steps for these steps).

DETAILED DESCRIPTION OF AN EXEMPLARY EMBODIMENT The following description with reference to the accompanying drawings, given by way of non-limiting examples, will make it clear what the invention consists of and how it can be implemented. In the accompanying drawings:

- Figure 1 shows schematically, in front view, a motor vehicle comprising a device according to the invention;

FIG. 2 diagrammatically represents the device of FIG. 1 according to two possible embodiments, a first embodiment (in solid lines) in which the image capture unit 11 comprises a single sensor 12, and a second an embodiment in which the image capture unit 11 comprises two sensors 12, 13 (solid line and dotted line, the sensor 13 being shown in dashed lines); and

FIG. 3 represents a flowchart of the main steps of the method according to the invention.

Device

FIG. 1 shows the front of a vehicle 1 embodying a device 10 for determining a state of attention of a driver of the vehicle 1.

More specifically, such a device 10 is adapted to determine the state of attention of the driver 4 according to a state of occupation of at least one of the hands of said driver 4 of the vehicle 1.

Thus, such a device 10 is adapted to detect at least one of the hands of the driver in a detection zone D of the vehicle 1, and to determine the state of occupation of this (or these) hand (s) to deduce the the state of attention of the driver.

The state of occupation is determined from a "busy" state in which the hand holds an object of distraction, and a "free" state in which it does not hold any object of distraction. In this free state, the hand can for example be engaged in driving the vehicle, that is to say act on a driving member of the vehicle, or be empty, for example at rest on an armrest.

As used herein, the object of distraction is an object other than a driving member of the vehicle. This is for example a mobile phone, a book, a road map, a GPS etc. The drivers accessible to the driver's hands are, for example, the steering wheel 3, the gearshift lever, the controls (turn signal or windshield wiper levers), the switches (such as the hazard warning lights) or the parking brake. .

As shown in FIG. 2, the device according to the invention comprises: an image capture unit 1 1 embedded in said vehicle 1, said image capture unit 11 being adapted to capture at least one image of the detection zone D situated in said vehicle 1, and

an image processing unit 15 adapted to receive said captured and programmed image for determining the state of occupation of the hand of the driver 4 located in the detection zone D, as a function of the detection of the presence of a object of distraction in said hand.

Here, the image capture unit 1 1 is embedded in the motor vehicle, that is to say disposed inside the vehicle 1, more precisely inside the passenger compartment of the vehicle 1.

The image capture unit 11 comprises at least one sensor 12, 13 adapted to capture an image of a first nature of the detection zone D.

As shown in Figure 1, the detection zone D is located between the gear lever 2 and the front door of the driver. The detection zone D thus encompasses the steering wheel 3 of the vehicle 1 and contains both hands of the driver. To capture such a detection zone D, the sensor 12, 13 of the image capture unit 1 1 is for example placed in a front ceiling of the vehicle 1 automobile, so that it takes in top view the zone detection D.

Alternatively, the sensor could be placed on the dashboard of the vehicle, in a central area of the latter, so that the detection area would be seen from the front. Depending on the opening angle of the sensor, the detection zone may contain only one of the driver's hands.

In another variant, the sensor could be placed behind the steering wheel 3 of the vehicle, at the dashboard. The detection zone could then easily contain the driver's right hand and left hand.

Any other place to have the hands of the driver in the field of view of the sensor 12, 13 is still possible to dispose the sensor.

According to a first embodiment of the device 10 according to the invention, represented in FIG. 2 in solid line, the image capture unit comprises a single sensor 12 adapted to capture a three-dimensional image of the detection zone D, said three-dimensional image comprising information relating to the distance, with respect to said sensor, of at least a part of the elements of the space contained in the detection zone D.

These elements of the space include the hands of the driver 4 and the object of distraction possibly present in the detection zone D. The elements of the space can also include elements of the environment of the driver, for example elements the passenger compartment of the vehicle and the vehicle control devices such as the gear lever, steering wheel, armrest, etc.

The three-dimensional image includes a scatterplot representing the envelope of the space elements present in the detection zone D, including the hand of the driver, the forearm of the driver, and the distraction object that may be present therein. to be present. The scatterplot thus gives information as to the position in space of the elements of the space present in the detection zone D, in particular information relative to their distance relative to said sensor.

Such a sensor 12 adapted to capture three-dimensional images is known to those skilled in the art and will not be described in detail. It will be specified simply that it could be a flight time sensor, such as a time of flight camera ("Time of Flight" or TOF according to the English acronym) adapted to send the light to the conductor 4 and measure the time it takes for this light to return to said time-of-flight sensor to deduce the three-dimensional image of the detection zone D. Alternatively, it could be a stereoscopic sensor comprising at least two cameras, each capturing an image of the detection zone according to its own point of view, the images of each camera being then combined to deduce the three-dimensional image of the detection zone. It can also be a structured light sensor adapted to project a pattern on the detection zone and to analyze the deformation of this pattern to deduce the three-dimensional image of the conductor 4.

According to this first embodiment, the image processing unit 15 is adapted to receive the three-dimensional image of the sensor, and programmed to determine by means of this image, the state of occupation of at least one of the hands of the conductor, according to the detection of the presence of the object of distraction in said hand.

More specifically, the image processing unit 15 is programmed to implement the following steps: b1) detecting at least one of the hands of the driver in said image received from the image capture unit 1 1, and

b2) detecting the presence of said distraction object in the driver's hand in order to deduce the occupation status of the driver's hand.

The rest of the description is based on the detection of a driver's hand. Of course, the same principle can be applied to the other hand of the driver.

For this purpose, the image processing unit 15 comprises processing means programmed to detect the image of the driver's hand in the image captured by the image capture unit 11, as well as the presence of the object of distraction in this hand.

More specifically, the image processing unit 15 is programmed to implement step b1) according to the following substeps:

detecting at least a portion of an arm of the driver, in the image received by the image capture unit, and

- deduce the position, in said image received from the image capture unit, of the driver's hand from the location of detection of the portion of the driver's arm in the same image.

In practice, the processing unit of the processing unit 15 is programmed to identify, among the points of the cloud of points, those associated with the image of the arm or the forearm of the driver.

The recognition of the shape of the arm or the forearm of the driver (step b1)) is here based on a shape recognition algorithm.

From the point cloud, the processing means are also programmed to recognize characteristic shapes of space elements that may be present in the detection zone D, such as a part of the speed lever 2, a part of the steering wheel 3, part of an armrest, etc. The "shape" of the elements of space corresponds here to its external envelope. The detection of these space elements can facilitate the detection of the driver's arm.

From the identification of the arm or the forearm of the driver, the treatment unit of the processing unit 15 identifies a portion of the cloud of points in which the hand of the driver is likely to be. The recognition of the shape of the hand is here based on an algorithm of form, realized with the determined portion of the cloud of points. The processing means are also programmed to identify at least two different regions in said three-dimensional image captured by the image capture unit 11, for example a first region formed by the images of the points closest to the sensor (foreground ) and a second region formed by the images of the farthest points of the sensor (second plane), and to recognize the shape of the arm or forearm of the driver 4 in these regions.

Thus, the processing means are adapted to determine the location of the driver's arm, and then to deduce the location of the driver's hand in the real space of the detection zone D.

Furthermore, the image processing unit 15 is programmed to implement step b2) according to the following substeps:

detect the presence of any object in the driver's hand, and

determining the nature of said detected object in the driver's hand, in order to deduce the state of attention of the driver.

Thus, the processing means are also programmed to identify the nature of the detected objects, according to the recognized shape at the estimated location of the hand in the image.

In practice, the processing unit 15 is programmed to deduce the state of occupation of the driver's hand, according to the shape recognized at the level of the driver's hand in the three-dimensional image.

This recognized form may be that associated with a distraction object a mobile phone, a glass or a cup, a book, a road map, etc., or that of elements still present in the detection zone D,

As a variant, it would also be possible to deduce the state of occupation of the driver's hand as a function of the distance between the hand and any detected objects in the captured image.

More specifically, the image processing unit 15 is programmed to determine that the driver's hand is in a busy state when it is very likely, given the shape recognized at the driver's hand in the image. three-dimensional, that the driver holds this object of distraction with his hand.

Alternatively, the image processing unit could be programmed to determine that the hand is in a busy state when the distance between the hand and the distraction object, in the captured three-dimensional image, is less than a predetermined threshold value.

On the contrary, it is programmed to determine that the driver's hand is in a free state when it is improbable, given the shape recognized in the hand of the driver, that the driver holds an object of distraction with this hand.

Alternatively, the image processing unit could be programmed to determine that the hand is in a free state when the distance between the hand and the distracted object detected, in the captured three-dimensional image, is greater than a value. predetermined threshold.

When his hand is in this free state, the driver can react faster on the manual driving organs of the vehicle, such as the steering wheel, turn signals, the horn, the hazard warning lights, or the gearshift, in case of need.

The device according to the invention is thus adapted to determine that the overall attention state of the driver is lower when his hand is in a busy state than when it is in a free state.

In a variant, again according to this first embodiment of the device according to the invention, instead of implementing steps b1) and b2), the image processing unit 15 comprises processing means including a data transmission network. trained neurons to directly recognize the state of occupancy of the hand, or even the state of attention of the driver, from an image of the detection zone D.

In practice, the neural network is trained prior to its use in the device 10 according to the invention.

To do this, the neural network is input fed by a plurality of images of the detection zone D in which the hand is in a free state (i.e. the hand is empty of any object of distraction, or holds a driving organ of the vehicle by hand) and it is indicated to the neural network that for its images, the state of the hand is free. According to the same principle, the neural network is also fed by a plurality of images of the detection zone D in which the hand is in an occupied state (that is to say, it holds an object of distraction). , and we indicate to the neural network that for its pictures, the state of the hand is busy.

Once trained, the neural network then receiving as input the image of the detection zone D captured by the image-capturing unit 1 1, is programmed to output, the free or occupied state of the hand of the driver, or even directly the state of high attention (if the hand is in a free state) or low of the driver (if the hand is in a busy state).

According to a second embodiment of the device 10 according to the invention, represented in FIG. 2 in both solid lines and dashed lines, the image capture unit 1 1 comprises at least

a sensor 12 (solid line in FIG. 2) adapted to capture at least one image of a first nature comprising a first type of information relating to the elements of the space contained in the detection zone D, and

a sensor 13 (in dashed line in FIG. 2) adapted to capture at least one image of a second nature distinct from the first nature, comprising a second type of information relating to the elements of the space contained in the zone of detection D.

As in the first embodiment, the elements of the space contained in the detection zone D include in particular the hand of the driver and / or the object of distraction whose presence it is desired to detect. They may also include elements of the driver's environment naturally present in the detection zone D, such as the steering wheel, the gear lever or the armrest of one of the seats.

In practice, the image capture unit 11 comprises either a single sensor (not shown) adapted to capture both the first and second nature images, or at least two distinct sensors 12, 13 adapted to capture respectively the first and second images. first and second natures.

When the image capture unit 1 1 comprises two separate sensors 12, 13, it is conceivable that they be arranged at the same place in the passenger compartment of the vehicle 1, for example in the ceiling lamp, behind the steering wheel 3, or still in a central region of the dashboard as previously described. It is also conceivable that each separate sensor is disposed at a different location of the passenger compartment of the vehicle, especially among the previously described locations.

The first and second nature images are chosen from: a three-dimensional image comprising information relating to the distance, relative to said sensor, of at least a part of the elements of the space contained in the detection zone D, that is to say here at least of the object of distraction and / or the hand of the driver,

a two-dimensional image comprising information relating to the luminance of at least part of the elements contained in the detection zone D, that is to say at least the distraction object and / or the driver's hand and,

a thermal image comprising information relating to the temperature of at least part of the elements contained in the detection zone D, that is to say at least the distraction object and / or the driver's hand .

To capture a three-dimensional image, the sensor 12, 13 may be one of those described previously in the first embodiment of the device 10, namely a time-of-flight sensor, a stereoscopic sensor, or a structured light sensor.

This type of sensor may in some cases also be suitable for capturing two-dimensional images.

A conventional photographic type sensor, or a camera, is also capable of capturing two-dimensional images.

The two-dimensional images are images giving information on the luminance of the elements present in the detection zone D, including the hand and the forearm of the driver. Conventionally, the two-dimensional images comprise pixels representing regions of the detection zone D, that is to say a pixel corresponding to the image of a region of the detection zone D. Each pixel is more or less bright depending on the corresponding luminance of the region of the detection zone D that it represents. Here, the two-dimensional images are in black and white, but one could also have two-dimensional color images, in which case each pixel would represent the chrominance of each corresponding region of the detection zone.

Thanks to the two-dimensional images giving details of the detection zone D, it is thus easier for the image processing unit 15 to recognize the nature of the objects included in the detection zone D than when the only three-dimensional images are processed.

This is for example through conventional form recognition algorithms.

To capture thermal images, the sensor may be a thermal camera, for example a high wavelength infrared camera (or Long Wired InfraRed LWIR camera).

The luminous intensity of the pixels of the thermal images depends on the temperature of the regions of the detection zone D corresponding to each pixel: the higher the temperature, the brighter the pixel, the lower the temperature, the darker the pixel.

Thus, for example, the forearm and the driver's hand will be represented by bright pixels, as well as the battery of the mobile phone, or a cup filled with hot liquid. On the contrary, the gear lever, a book or a road map will be represented by darker pixels.

Thus, thanks to the thermal images, it is also easier for the processing unit to recognize the nature of the elements of the space included in the detection zone D, and in particular to discern the forearm of the driver terminated by the hand of the driver of the rest of these elements of space.

In addition or as a variant of the second embodiment of the device 10 according to the invention, the image capture unit 11 may further comprise at least one sensor adapted to capture an image of a third nature, distinct from said first and second natures, comprising a third type of information relating to the elements of the space contained in the detection zone D.

Preferably, the image capture unit 1 1 will in this case comprise at least two separate sensors, possibly arranged at different locations of the passenger compartment of the vehicle, to capture the images of first, second and third natures.

The third nature image is chosen from the images described above, namely a three-dimensional image, a two-dimensional image or a thermal image. The sensors described above are for example used.

According to this variant, the image processing unit 15 takes into account at least three images of different natures, namely a three-dimensional image, a two-dimensional image and a thermal image, for determining the state of occupation of the driver's hand.

According to the second embodiment of the device according to the invention, whatever the variant envisaged, that is to say that the image capture unit 1 1 captures images of two or three different natures, the image processing unit 15 is programmed to implement steps b1) and b2) explained above, namely:

b1) locating said driver's hand in at least one of said images received from the image capture unit 1 1,

b2) detecting an object of distraction in the hand of the driver 4 in order to deduce the state of occupation of the hand of the driver, then the state of attention of the driver.

For this purpose, the image processing unit 15 comprises processing means programmed to implement the steps b1) and b2) in a so-called sequential implementation or in a so-called concurrent implementation.

It is also possible to envisage that steps b1) and b2) are made from one or two images of the same nature.

However, it is advantageous to use in these steps images of different natures that give access to different information on the detection zone.

It is generally envisaged either to carry out one of the steps b1) and b2) from one or more images of a first nature and to perform the other step b1) or b2) from one or several images of the same nature, different from the first nature, or to realize at least one of the two steps b1) and b2) from several images of different types, or to perform each step b1) and b2) to from several images of different natures. The combination of several images of different natures to carry out the steps b1) and / or b2) makes it possible to improve the accuracy of the detection of the hand and / or the object of distraction.

Example of sequential weaving

For the sake of clarity, the sequential implementation is described in the case where the image capture unit 11 captures images of two different natures. Of course, the same principle can be applied when capturing images of three different natures.

According to this example of the sequential implementation, the image processing unit 15 implements the step b1) from at least a first captured image of first nature, and the step b2) from at least a second captured image of a different nature from the first nature.

In practice, the first image of first nature is for example one of the three-dimensional images captured by the image capture unit 1 1. The processing means are programmed to segment this three-dimensional image, that is to say to identify at least two different regions in said three-dimensional image, and to recognize the shape of the forearm or the arm of the driver 4 in each of these regions, as described in the first embodiment of the device 10, and to deduce the position of the hand of the driver, logically at the end of the arm recognized.

The two aforementioned regions correspond for example, as mentioned above, to a near region and to a region remote from the sensor.

It is advantageous to recognize a part of the arm (for example the forearm) of the driver from the three-dimensional image because it is thus possible to know the region, near or far from the sensor, in which the arm is located. of the driver.

After having located the hand of the driver by means of the first three-dimensional image, the processing means of the image processing unit 15 are programmed to detect the possible presence of an object in the driver's hand by means of the second second nature image captured by the image capture unit 1 1. For this purpose, the processing means of the image processing unit 15 are programmed to recognize the characteristic shape of objects possibly present in the detection zone D, at the position of the hand as evaluated as explained above, from two-dimensional and / or thermal images.

The processing means are then programmed to determine, by combining the information of the first and second images (first and second natures) if an object is present in the hand of the driver or not and, if so, to determine the nature of the this object to estimate whether it is a distraction object (or a driving device). When no object image is detected in the second image (of second nature) captured by the image capture unit 1 1, that is to say that no object is present in the detection zone D, the image processing unit 15 is programmed to determine that the hand is in a free state.

When at least one object is detected in the second (second nature) image captured by the image capture unit 11, the processing unit 15 is programmed to identify the nature of the detected object.

The image processing unit 15 is then programmed to determine that the hand is in a busy state if the detected object is an object of distraction (and to determine that the hand is in a free state, in the sense available for the driving, if the detected object is a driving device).

Of course, as in the first embodiment of the device, the shape recognized at the level of the hand in the first and second nature images confirms the probability that it holds an object of distraction, or greatly reduce this probability. The shape of the contour of the hand can thus be taken into account by the image processing unit 15 for determining the state of occupation of the hand.

Advantageously, the sequential implementation makes it possible to use the most relevant information of each of the first and second nature images, depending on the step to be performed.

Of course, in practice, it would be possible to carry out the sequential implementation using the thermal or two-dimensional images as a first-nature image, and the three-dimensional, two-dimensional or thermal images as an image of second nature, provided that they are chosen from distinct nature of the first nature.

It would still be possible to use the third nature images to facilitate the step of shape recognition of the hand and / or forearm of the driver, and / or to facilitate the step of detecting the object in the detection zone D.

Example of concurrent implementation

According to the concurrent implementation, the image processing unit 15 implements the steps b1) and b2) from each image received from the image capture unit 11, and implements a step that it determines a confidence index associated with the detection of the presence of the object in the driver's hand according to which it deduces the state of occupation of the driver's hand.

More specifically, the image processing unit 15 implements steps b1) and b2) independently for each of the first, second and third images.

In practice, the steps b1) and b2) are implemented with the three-dimensional images as described in the case of the first embodiment of the device 10 according to the invention.

For two-dimensional and thermal images, the processing means of the image processing unit 15 are programmed to identify, from among the pixels of each image, those associated with the image of the hand and / or the image of the image. the forearm of the driver, and those associated with the image of a distraction object possibly present in the detection zone D, from shape recognition algorithms specific to said thermal and two-dimensional images. Depending on the location of the hand and the possible detection of an object of distraction in the hand thus localized, the image processing unit 15 is programmed to deduce the state of occupation of the hand of the driver in these images.

At the end of the processing of each image of first, second and possibly third natures, the image processing unit 15 determines the confidence index of the treatment, that is to say determines whether the treatment has led to a result, and if this result is safe, or if it is rather random.

For example, by modeling a hand by a palm and five fingers each composed of three phalanges, the confidence index is at most if one recognizes the palm and five fingers in the image; the index is lower when fewer elements are recognized, for example when one recognizes the palm and only two fingers.

The image processing unit 15 determines the actual occupation status (occupied or free) of the driver's hand from the image whose processing has led to the highest confidence index, c ' that is to say, whose result is the safest.

Alternatively, still according to the second embodiment of the device according to the invention, instead of implementing steps b1) and b2) concurrently or sequentially, the image processing unit 15 comprises processing means including a trained neuron network for directly recognizing the state of occupancy of the hand from an image of the detection zone D.

In practice, the neural network is trained prior to its use in the device according to the invention according to the same principle as that described in the case of the first embodiment, except that it is trained with images of first, second and possibly third natures.

The neuron network thus driven is thus adapted to receive first, second, and possibly third-nature images as input to determine the state of occupation of the driver's hand.

Advantageously, the fact of using images of different types at the input of the neural network makes it possible to increase the reliability of the result given at the output of said neural network.

The device according to the invention finally finds a particularly useful application in an embedded system 100 for a vehicle comprising

the device according to the invention, adapted to determine the state of occupation of at least one of the hands of the conductor 4 present in the detection zone D,

an autonomous driving unit 50 of said vehicle 1 programmed to control driving members of said vehicle independently of the driver, and

- A decision unit (not shown) programmed to allow the driver 4 to control at least part of the driving members of the vehicle 1 in case of determination of a free state the hand of the driver and / or to alert the driver in case of determination a busy state of the driver's hand, and / or to tilt the autonomous driving in a secure mode.

Here, the driving members include the steering wheel 3, the acceleration and brake pedals, the gear lever, the turn signals, the headlights, the wipers. Thus, the driving members of the vehicle include all the vehicle elements used for driving.

The autonomous driving unit 50 is adapted to control the various driving members so that said vehicle is driven without intervention of the driver.

In this situation, the driver is allowed to be in a state of insufficient vigilance, that is to say, being distracted, and it is possible that he reads a book or looks at his mobile phone without danger for driving.

It sometimes happens that the driver wants to resume at least partially the hand on the driving members (or that the autonomous driving unit 50 requires such recovery by the driver, for example because of traffic conditions). In this case, the decision unit is programmed to allow the driver to at least partially control the driving members only when his state of global attention is high, especially when at least one of his hands is in a state free.

On the contrary, when at least one of his hands is in a busy state

(Or alternatively when both hands are in a busy state), the decision unit does not allow a restart by the driver, but may for example command the display of a warning message to the driver to encourage him to be attentive and / or switch the autonomous driving in a secure mode in which the autonomous driving unit 50 controls for example the parking of the vehicle on the low side (or on the emergency stop band) ).

Process

FIG. 3 shows in flowchart form the main steps of the method implemented by the device according to the invention.

The method according to the invention for determining the state of occupation of at least one of the hands of the driver of the vehicle comprises steps according to which:

a) the image capture unit 1 1 embedded in said vehicle 1 captures at least one image of the detection zone D located in said vehicle 1,

b) the image processing unit 15 receives said captured image and determines the state of occupation of a hand of the conductor 4 located in the detection zone D, as a function of the detection of the presence of an object of distraction in said hand.

More specifically, in step a) (block E1 of FIG. 3), the image capture unit 11 of the device 10 according to the invention captures at least one image of a first nature of the detection zone. D.

Preferably, it captures two or even three images, of the same nature or of different natures, from the detection zone D, preferably of natures different.

Preferably, each image of different nature is captured at a given moment, that is to say that all the images are captured simultaneously by the image capture unit 11 or in a relatively short time interval, in particular well under a minute. This ensures that the analyzed situation has not changed between the captures of images of different natures.

In step b), the image processing unit 15 receives the captured image or images in step a).

According to a first possible implementation of the method according to the invention using the device 10 of the first or second embodiment according to the invention, represented by the channel (1) in FIG. 3, the image processing unit 15 implements the steps described above, namely

detecting at least a portion of the driver's arm in at least one received image (block F1);

locating the hand of the driver in said image or images received from the image capture unit 1 1 (block F2), depending here on the location of detection of the part of the aforementioned arm,

detecting the presence of said distraction object in the driver's hand in order to deduce the state of occupation of the driver's hand (block F3).

When two images of different natures are captured in step a), by the image capture unit 11 according to the second embodiment of the device 10 according to the invention, the implementation of step b) can be sequential or concurrent, as previously described.

According to a second possible implementation of the method according to the invention using the variants of the first and second embodiments of the device 10 in which the image processing unit 15 comprises a trained neuron network represented by the channel (2). ) in FIG. 3, said image processing unit 15 directly recognizes the state of occupation of the hand from the images received from the image capture unit 11 (block G of FIG. 3).

Whatever the implementation of the method envisaged, in step b), the image processing unit 15 sends the state of occupation of the hand of the driver to the devices of the vehicle the applicant (block E2 of the FIG. 3), in particular to the autonomous vehicle decision unit, or to a device monitoring the existing driver of the vehicle.

One could also consider that the device according to the invention informs the driver of the busy state of at least one of his hands, so as to encourage him to refocus on his conduct.

Advantageously, there is further provided an additional step of determining a state of overall attention of the driver taking into account the state of occupation of the hand of the driver and / or a state of alertness of the driver determined by d other known means.

Advantageously, in the case of autonomous vehicles, in a situation where the driving members of said vehicle are controlled independently of the driver, there is further provided a step according to which the driver is authorized to control at least part of the driving members of the vehicle in the event of determination of a free state of the driver's hand and / or in the event of determination of a busy state of the driver's hand, the driver is alerted and / or the autonomous driving is tilted in a secure mode.

The device, the system and the method according to the invention are particularly advantageous in partially or completely autonomous driving situations, during which the driver is allowed to relax his attention, that is to say to present a state of insufficient vigilance . The position of the driver in his situations can be changed to the point that he is no longer in front of a possible surveillance device adapted to capture an image of his head to assess his level of vigilance. It is then very useful to determine the state of occupation of the hands of the driver to apprehend his state of overall attention.

In addition, the invention makes it possible to provide information complementary to that already provided by a possible device for monitoring the driver.

Finally, the invention applies to any type of vehicle, including transport vehicles such as boats, trucks, trains.

Claims

1. Device (10) for determining a state of attention of a vehicle driver (1) comprising:

an image capture unit (1 1) embedded in said vehicle (1), said image capture unit (1 1) being adapted to capture at least one image of a detection zone (D) situated in said vehicle (1), and

an image processing unit (15) adapted to receive said captured and programmed image for determining the state of attention of the driver (4), as a function of the detection of the presence of an object of distraction in one of the hands of the driver (4) located in the detection zone (D).

2. Device (10) according to claim 1, wherein the image capture unit (1 1) comprises at least one sensor (12, 13) adapted to capture a three-dimensional image of the detection zone (D), said three-dimensional image having information relating to the distance, with respect to said sensor (12, 13), from at least said distraction object or said hand located in the detection zone (D).

The device (10) according to claim 1, wherein the image capturing unit (1 1) comprises at least

a sensor (12) adapted to capture at least one image of a first nature comprising a first type of information relating to said distraction object or to said hand located in the detection zone (D), and

a sensor (13) adapted to capture at least one image of a second nature distinct from the first nature, comprising a second type of information relating to said distraction object or to said hand located in the detection zone (D).

4. Device (10) according to claim 3, wherein the first-nature image is chosen from: a three-dimensional image comprising information relating to the distance, relative to said sensor (12, 13), at least said distraction object or said hand located in the detection zone (D), a two-dimensional image comprising information relating to the luminance of at least said distraction object or said hand located in the detection zone (D), and a thermal image comprising information relating to the temperature of at least one of said distraction object or said hand located in the detection zone (D).

5. Device (10) according to one of claims 3 and 4, wherein the image capture unit (1 1) further comprises at least one sensor adapted to capture an image of a third nature distinct from said first and second natures, comprising a third type of information relating to said distraction object or to said hand located in the detection zone (D).

6. Device (10) according to claim 5, wherein the image of third nature is selected from: a three-dimensional image comprising information relating to the distance, with respect to said sensor, at least said object of distraction or said hand located in the detection zone, a two-dimensional image comprising information relating to the luminance of at least said distraction object or said hand located in the detection zone, and a thermal image including information relating to the temperature of said distracted object or said hand located in the detection zone.

7. Device (10) according to one of claims 1 to 6, wherein the image processing unit (15) is programmed to implement the following steps:

b1) locating (F1, F2) said driver's hand (4) in said image received from the image-capturing unit (1 1),

b2) detecting (F3) the presence of said distraction object in said driver's hand (4) in order to deduce the state of attention of the driver (4).

8. Device (10) according to claim 7, wherein the image processing unit (15) is programmed to implement step b1) according to the following substeps:

detecting (F1) at least part of an arm of the driver, in the image received from the image capture unit, and

deducing (F2) the position, in said image received from the image capture unit, of said driver's hand from the detection location of the driver's arm portion in the image received from the driver's unit; image capture.

9. Device (10) according to one of claims 7 and 8, wherein the image processing unit (15) is programmed to implement step b2) according to the following substeps:

detect the presence of any object in the driver's hand, and

determining the nature of said detected object in the driver's hand, to deduce the state of attention of the driver.

10. Device (10) according to one of claims 7 to 9 taken in dependence on one of claims 3 to 6, wherein the image processing unit (15) implements step b1) from the image of first nature, and step b2) from the image of nature distinct from the first nature.

1 1. Device (10) according to one of claims 7 to 9, wherein the image processing unit (15) implements steps b1) and b2) from each image received from the capture unit of images (1 1), and implements an additional step in which it determines a confidence index associated with the detection of the presence of the object of distraction in the driver's hand (4) according to which it deduces the state of attention of the driver (4).

12. Device (10) according to one of claims 1 to 6, wherein the image processing unit (15) comprises processing means using a trained neural network to determine the state of attention of the driver. from the image of the detection zone (D) captured by the image capture unit (1 1).

13. On-board system (100) for a vehicle comprising:

- A device (10) according to one of claims 1 to 12, adapted to determine the state of attention of a conductor (4),

an autonomous driving unit (50) of said vehicle (1) programmed to control driving members of said vehicle (1) independently of the driver, and

- A decision unit programmed to allow the driver to control at least part of the driving members of the vehicle when determining a free state of the driver's hand.

14. Method for determining a state of attention of a driver (4) of a vehicle (1), according to which

a) an image capture unit (1 1) embedded in said vehicle (1) captures at least one image of a detection zone (D) located in said vehicle,

b) an image processing unit (15) receives said captured image and determines the state of attention of the driver, according to the detection of the presence of a distraction object in one of the hands of the driver located in the detection zone (D).