WO2013117787A1

WO2013117787A1 - Process and system for detecting the position of the hands of a driver

Info

Publication number: WO2013117787A1
Application number: PCT/ES2013/070061
Authority: WO
Inventors: Enrique CABELLO PARDOS; Isaac MARTIN DE DIEGO; Cristina CONDE VILDA; Oscar Gerardo SANCHEZ SIORDIA
Original assignee: Universidad Rey Juan Carlos
Priority date: 2012-02-10
Filing date: 2013-02-05
Publication date: 2013-08-15
Also published as: ES2425663B1; ES2425663A1

Abstract

The invention consists of a novel method for detecting the position of the hands of a driver and the physical device which carries out this method. From the images of the hands, as captured by a camera, this method makes it possible to predict a possible lack of attention of the driver and to send a warning message to the latter. The device can also be used to reproduce the causes of an accident or to train drivers.

Description

Title

System for detecting the position of a driver's hands.

Field of the technique to which the invention belongs

Driving assistance systems, especially those dedicated to increasing the safety of the driver and other vehicles.

State of the art

Distraction in drivers is present in 30% of traffic accidents and is currently responsible for a large number of fatalities. The study of driver distractions is a very complex problem due to the large number of factors that can influence it. Within all of them probably one of the most important is related to the driver's behavior. There are three elements involved in a traffic accident: the vehicle, the road and the driver. Of these, the driver's behavior is he has had less attention in the past. The roads have improved a lot in recent times: new types of asphalts, with superior technical characteristics, improvement in road signs or road designs more appropriate for driving are some of the improvements that have been developed in this field. Vehicle manufacturers have also increased safety measures, both active and passive, by creating new electronic or mechanical systems that make vehicle behavior safer.

In recent years, the driver's study has focused mainly on the detection of drowsiness, using a wide range of techniques. Physiological measurements based on electro-encephalograms (EEG) [A. Kircher, M. Uddman, J. Sandin, Vehicle control and drowsiness, Tech. Rep. VTI-922A, Swedish National Road and Transpon Research Institute, 2002.] are the ones that provide the best data for the detection of this situation. The main problem is that they are very intrusive to the subject you are driving.

A less intrusive system is the detection of the driver's eyes. Most systems of this type consider one or two cameras to follow the subject's head and eyes [V. Matsumoto, A. Zelinsky, An algorithm for real-time stereo vision implementation of head pose and gaze direction measurements, Procs. IEEE 4th Int. Conf. Face and Gesture Recognition, 2000, pp. 499-505.] Or [T. D'Orazio, M. Leo, C. Guaragnella, A. Distant, A visual approach for driver inattention detection, Pattern Recognit. 40 (8) (2007) 2341-2355.], To name one. Some companies market systems for installation in vehicles [Smart Eye AB, Image capturing device with reflex reduction, Patent, EP 1349487B1].

REPLACEMENT SHEET (Rule 26) Another alternative is the monitoring of the lateral position of the vehicle, inside its lane, and the crossing time from one lane to another, as elements that are taken into account to assess the vehicle's driving attention [Volvo Car Corp. or AG Daimler]. These techniques are not invasive, and show reliable results.

The present invention proposes an alternative to these systems based on the tracking of the driver's hands of the vehicle. That is why the technical problem that solves this invention is the development of a driver alert system based on the detection and monitoring of the hands of a driver, which is not intrusive, that works in real time and has a Very low computational cost.

Detailed description of the invention

The invention consists of a new method for detecting the position of the hands of a driver and the physical device that performs this procedure.

Figure 1 shows the main stages of this procedure. The first step is to detect the position of the areas of interest to study, this is those where the hands will be searched to know their location: the handbrake, the gearshift and the steering wheel (101), for this there are three possible ways To do so: a) With an initial photo, the areas of interest are drawn and the coordinates of the steering wheel, handbrake and gearbox are obtained. b) Use the first images to track the hands and store the trajectories. The points where the hands are most often are the points of interest. The number of images may vary, depending on driving circumstances, but a range between 15,000 and 40,000 images would be sufficient - this equals a driving time between 15 and 30 minutes. This mechanism has the additional advantage of being able to detect other areas of interest, such as radio, GPS or mobile phone through frequent movement of hands on these devices. The system learns from the driver's movements. c) A combination of the previous two: start with an initial estimate based on a photograph and then confirm these positions from the hand tracking.

Once the areas of interest where the system will search for the hands are defined, the procedure detects the shadow areas in them and overlays them on the image of the steering wheel, change and brake

REPLACEMENT SHEET (Rule 26) hand. We will call this procedure Global Detection Threshold (GTD). As a result, the occlusion areas are obtained, see figure 3.

If the number of occlusions is greater than two, which is the desirable number, for each of them you get the area associated with your digital image, which will be called "candidate" (103). In this step the errors produced by the shine on the steering wheel or hands are eliminated. To calculate the final position of the hands, which we will call "endpoints", those that are less than a certain fixed value (104) are eliminated. The area of the candidates is measured by their number of pixels, being between 15 and 30 the threshold value to discard them, depending on the circumstances. A typical value is 20 pixels.

If the distance between adjacent candidates is less than a certain value, the two candidates join (105). This distance is also measured in pixels and is around 10 pixels. If after this process only one candidate remains, this is considered an "end point" (106).

If there are two points left, it is necessary to determine whether they correspond to one hand or two. Figure 2 shows the procedure followed in this case. The center of a hypothetical ellipse that contains, within itself, the two occlusions (303) is calculated. On the other hand, another ellipse is calculated whose major axis is the distance between the centers of the two previous occlusions (304). If the percentage of the area of the candidates within the new ellipse is greater than 50% then the two candidates represent the same hand (305). For example, this is the case of a hand and an arm "separated" by a wristwatch. If the value is lower than the threshold then each candidate represents a hand (307), this is the case that both hands are on different parts of the steering wheel. A particular case is when both hands are not detected on the steering wheel (106, 306), in that case it is searched in other areas of interest, such as the gearshift or the handbrake (306).

An alternative method of hand detection, which we call Local Detection Threshold (LTD), proposes a new mechanism to digitize the flyer and search for potential candidates.

The flyer is divided into four areas of interest (204) where hand searches will be conducted independently, in each of these new areas. For each one a threshold value of gray level (white on the black of the steering wheel) is defined to be used to decide which occlusions will eventually become "endpoints" (205). This value will be 50%.

Occlusions are sought in each of the quadrants, with their respective thresholds, and the candidates are selected (206). As a last step, the hands (207) are located, in the same way as in the previous procedure. Likewise, on those occasions where there is

REPLACEMENT SHEET (Rule 26) To evaluate whether two occlusions correspond to one hand or two, the procedure described in Figure 2 is used.

To improve the two previous mechanisms (GTD and LTD), once the hands are detected, they are monitored using the procedure called Camshift [Bradski, GR .: Computer vision face tracking for use in a perceptual user interface. Intel Technology Journal, 2nd Quarter, 1998]. This is a well known procedure in this field of art, so it is not described in this patent application. If any of the hands is lost during the follow-up process, one of the previous mechanisms is used again to detect it again.

Together with the hand position data, the system stores a series of vehicle operating parameters. This information comes from the electronic control unit of the vehicle, through its CAN-bus and provides information such as: status of the vehicle lights, steering wheel position, brake pedal actuation, engine revolutions, vehicle speed, position throttle, gear engaged, safety distance, etc.

Simultaneous storage of the position of the hands and this vehicle data will be of great help for the development of applications based on this procedure and device. Additionally, the sound inside the cabin can be recorded through a microphone that can be located inside the cabin itself. It is also possible to record, through a video camera, the exterior images of the car - for example what the driver sees - in integrating it with the rest of the information. The application of this procedure requires an image and sound capture module and an information processing unit, both for the image and sound and for the vehicle's operating data.

The image capture system is composed of a series of lamps / illuminators and diffusers that work in the infrared spectrum, which for this application is in the range between 840 and 860 nm., And a system of cameras and filters sensitive to these wavelengths

The system can have two or three illuminators. One of them is focused on the left hand and the rest are placed on the steering wheel, with the camera located between them. The illuminators emit a continuous infrared light, around 850 nm wavelength, so that it is not visible by the human eye, and is not intrusive to the driver. The CCD camera captures only the infrared part by the action of a filter that eliminates the entire visible spectrum and only lets these wavelengths pass.

REPLACEMENT SHEET (Rule 26) The electronic device that will collect the signals and process them according to the procedure described above can be developed on different types of platforms, from an FPGA or a PIC's to an integrated circuit designed to suit this application (ASIC). Its final installation can be a card connected to a personal computer, an independent module or its total integration into the car electronics.

From this procedure and device different types of applications can be developed. The first of these is the prevention of a possible accident due to a distraction. The pattern of hand movement may indicate a lack of driver concentration. The electronic device, from the data of the positions of the hands, will generate an alert to the driver when these circumstances occur - warning of the potential danger -, being able to make recommendations such as: take a rest, stop definitively or continue with a driving more prudent, for example.

Another possible application may be the development of a system to reproduce the causes of an accident, for this it is necessary to determine the state of the vehicle before the accident, the Bus_CAN of the vehicle provides a large number of physical variables of the operation of the engine / vehicle . Second, the system can acquire information about the internal and external conditions of driving through images taken from "the driver's point of view". It is possible to include the internal sound of the cabin, from a microphone installed for this purpose. Together with this data, the position of the hands will be stored at each moment. All this information is recorded in the memory of the electronic device. When an accident is detected, after the airbag is fired, the system stops taking data and "closes" the access to the memory, so that it contains all the information prior to the incident. This will guarantee the integrity of the information.

Another possible application may be the training or training of drivers based on the behavior or driving patterns of an expert driver. From a physical device such as the one described in this application, the behavior of the expert driver can be monitored for difficult driving situations, either through a simulator that emulates these conditions, or in a real controlled environment, as you can Be a car circuit. These responses to critical situations will serve as the basis for training less experienced drivers.

The physical device that performs this procedure consists of at least: an illuminator, which works in the infrared frequency range (around 850 nm.); a digital camera that records hand movements capable of recording images in this frequency spectrum - for which specific filters are used; optionally a digital camera that records the road; optionally a microphone that records the interior sound of the cabin; and an electronic unit, CENTRALITA as of now, which stores this information and performs the procedure. Everybody

REPLACEMENT SHEET (Rule 26) These devices can be physically connected to the CONTROL UNIT, either with wired connections from each of them, or through a data bus. It is also possible to connect them through a wireless system, type bluetooth or WiFi or any other that provides this functionality. The CONTROL UNIT that carries out the procedure also has an input connection to receive the vehicle data through its Bus-CAN.

This CONTROL UNIT can be built in different ways and on different platforms since current technology allows a wide variety of possibilities. It consists, at least, of one memory or group of memories to store the data and execute the intermediate calculation operations. These can be independent or integrated in another part of the electronic circuitry of the CENTRALITA. It also consists of a unit to process the instructions associated with each step of the procedure that can be implemented in an FPGA, a PIC, a specific processor or any other device that provides this functionality. This electronic circuit will also have a series of inputs and outputs, which can be wired or wireless, to receive information and provide output signals.

Description of the figures

Figures 1 and 2 show, in a simplified way, the data flow of the procedures for the detection of the hands described above.

Figure 3 is a sequence of photos that allows a better understanding of the hand detection procedure on the steering wheel.

Figure 4 shows a particular embodiment of this device.

Detailed description of the particular embodiments

A particular embodiment of this invention is shown in Figure 4. This consists of a CONTROL UNIT (4), for its construction an architecture based on an ARM processor (XScale-PXA270) at 624Mhz with a Linux operating system, kernel 2.6 has been chosen .16 (9). The device has four USB ports that allow the connection of a wireless network adapter and / or a Bluetooth transmitter / receiver (1 1). It also has a serial / USB converter that allows the reception of vehicle data from the Bus-CAN (10). It is worth mentioning that this information includes the airbag trigger signal, which will be used as an indication that there has been an accident. For the storage of the information the device includes a group of solid state memories (SSD) with 50Mb capacity (5, 6, 7, 8).

Using this CONTROL UNIT two examples have been made, varying the lighting system, the driving scenario and the resolution of the digit camera.

REPLACEMENT SHEET (Rule 26) Example 1 .

In the first example three infrared illuminators / lamps have been installed inside the driver's cab of a truck, above the driver's head. These lamps emit a continuous infrared beam with a wavelength of 850 nm so that it is not visible by the human eye. The first one focuses on the left hand of the steering wheel. The other two lamps are focusing on the steering wheel area. None of them have a light diffuser incorporated. Between the last two lamps is the camera that will obtain the images, from which the position of the hands will be calculated. This is a digital camera that has a CCD sensor, an acronym for Coupled Load Device, with a resolution of 5 Mpixeis, capable of capturing 14 images per second (f / s) at maximum resolution. This camera records an average of 25.5 f / s images of size 352x288, 12.4 f / s at 640x480 and 5.7 f / s at 1027x768. In this case the camera records with an image size of 640x480.

The procedure to be used to detect the position of the hands is what we have previously referred to as the Global Detection Threshold. To obtain the results, an interurban scenario will be simulated, where the vehicle will circulate while the system detects the position of the hands.

Under these conditions the system has a success rate of 98%, with average computing times of 23 ms for each image (frame).

Example 2

In this example the arrangement and type of lamps are the same as in the previous example but now a white diffuser has been placed on the two illuminators located on the steering wheel. The video camera will also record under the same image size conditions as in the previous example.

In this case the hand detection procedure will be the Local Detection Threshold (LTD), and an urban route for the vehicle will be simulated.

Under these conditions the degree of success is 88%, slightly lower than in example 1. With this method of detection the process times decrease considerably, being 7 ms per image.

REPLACEMENT SHEET (Rule 26) Although this second procedure should be more precise than the previous one, the decrease in the number of hits is attributable to the use of an urban route, where the driver moves his hands on many more occasions due to the layout of the stage: greater number of curves, traffic signs, increased circulation, etc.

REPLACEMENT SHEET (Rule 26)

Claims

one . Procedure for detecting the position of the driver's hands of a vehicle characterized in that it comprises the following stages:

a) steering wheel position detection, gearshift and handbrake;

c) detection of areas of occlusion on the steering wheel;

e) Integration and processing of the data obtained from the position of the steering wheel, the gearshift, the handbrake, the occlusions on the steering wheel and the operating data of the vehicle;

f) Generation of a warning from the processed data.

2. Method for detecting the position of the driver's hands of a vehicle according to claim 1, characterized in that the position of the steering wheel, the gearshift and the handbrake are determined from an initial photograph.

3. Procedure for detecting the position of the driver's hands of a vehicle according to claim 1 characterized in that the position of the steering wheel, the gearshift and the handbrake are determined from the first n-images, in a range between 15,000 and 40,000.

4. Method for detecting the position of the driver's hands of a vehicle according to claim 1, characterized in that the position of the steering wheel, the gearshift and the handbrake are determined from a combination of the methods described in the claims 2 and 3

5. Procedure for detecting the position of the hands of the driver of a vehicle, according to claims 1 to 4, characterized in that obtaining the position of the hands comprises the following steps:

a) look for areas of occlusion on the steering wheel;

b) eliminates areas of occlusion smaller than a value between 15 and 30 pixels; c) if the occlusion areas are closer to each other than 10 pixels join said occlusion areas;

d) if there is only one occlusion area, this is the position of one of the hands;

e) if there are two areas of occlusion the system calculates the center of a hypothetical ellipse that contains, within itself, the two occlusions and, on the other hand, another ellipse is calculated whose major axis is the distance between the centers of the two occlusions previous;

f) if the percentage of the area of the candidates within the new ellipse is greater than 50% then the two candidates represent the same hand;

g) If no hand is detected on the steering wheel, this search is performed in the gearshift and handbrake.

REPLACEMENT SHEET (Rule 26)

6. Procedure for detecting the position of the hands of the driver of a vehicle, according to claims 1 to 4, characterized in that obtaining the position of the hands on the steering wheel comprises the following steps:

a) divide the flyer four areas of interest;

b) a 50% occlusion threshold is defined for each of the defined areas of interest; c) look for the areas of occlusion in each region of interest;

d) eliminates areas of occlusion smaller than a value between 15 and 30 pixels; e) if the occlusion areas are closer to each other than 10 pixels join said occlusion areas;

f) if there is only one area of occlusion, this is the position of one of the hands;

g) If there are two areas of occlusion, the system calculates the area of a hypothetical ellipse that contains each of the hands;

h) if the percentage of the area within the new ellipse is greater than 50% then there is only one hand;

i) If no hand is detected on the steering wheel, this search is performed in the gearshift and handbrake.

7. Procedure for detecting the position of the hands of the driver of a vehicle, according to any of the preceding claims, characterized in that the tracking of the position of the hands, once detected, is done using a camshif algorithm.

8. Procedure for detecting the position of the hands of the driver of a vehicle, according to the preceding claims, characterized in that in addition to the information on the steering wheel, handbrake and gearshift, that of other electronic devices installed in the vehicle, such as the Global Position System (GPS), traffic information systems or personal communication systems.

9. Device for the detection and monitoring of the driver's hands in a vehicle, according to the procedure described above characterized in that it consists of the following elements: an infrared lighting system, an image capture system in the infrared spectrum and an electronic device for the processing of the data obtained from both the vehicle and the infrared collection device.

10. Device for detecting and monitoring the driver's hands in a vehicle, according to claim 9, characterized in that the lighting system consists of at least one infrared illuminator and a light diffuser, which works in this spectrum.

REPLACEMENT SHEET (Rule 26)

1 1. Device for detecting and monitoring the driver's hands in a vehicle, according to claim 10, characterized in that the infrared illuminator emits in a wavelength between 840 and 860 nm.

12. Device for detecting and tracking the driver's hands in a vehicle, according to claim 9, characterized in that the digital image capture system is capable of recording the infrared spectrum.

13. Device for detecting and monitoring the driver's hands in a vehicle, according to claims 9 to 12, characterized in that the image capture system and the lighting system are located in the upper part of the interior of the cabin vehicle.

14. Device for detecting and monitoring the driver's hands in a vehicle, according to claim 9, characterized in that the electronic device that processes the data consists of a data storage memory, a data processing unit, a bus of communications and at least one input / output interface.

15. Device for detecting and monitoring the driver's hands in a vehicle according to claim 14, characterized in that the data processing unit can communicate wirelessly with some of the data collection devices.

16. Traffic accident prevention system characterized by using the method described in claims 1-8 and the devices described in claims 9-15.

17. System for the reproduction of driving patterns followed by the driver of a vehicle characterized by using the method described in claims 1-8 and the devices described in claims 9-15.

18. System for the evaluation of traffic accidents characterized by using the procedure described in claims 1-8 and the devices described in claims 9-15.

REPLACEMENT SHEET (Rule 26)