JPH0468500A - Vehicle driver monitoring device - Google Patents

Abstract

PURPOSE:To securely detect the direction of the face or the state of eyes at high speed for a lot of undefined vehicle drivers by inputting the vehicle driver picture of a face part position calculated by a position calculating means and monitoring the direction of the face or the state of eyes of the vehicle driver by using a neural network. CONSTITUTION:As an image pickup control means for driving a TV camera 10, a CPU 4 is provided and based on detection signals from respective sensors 1-3, a timer is suitably operated. Then, the presence / absence of the vehicle driver is detected and when the presence / absence of the vehicle driver is detected, the face part position of the vehicle driver is calculated according to difference between a background picture, which is obtained by driving an image pickup means 10, and the picture of the vehicle driver. The face part picture of the vehicle driver is processed by a previously learnt neural network 16, and the direction of the face and the state of the eyes are detected. Thus, for a lot of undefined vehicle drivers, the direction of the face or the state of the eyes can be securely detected at high speed.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a vehicle driver monitoring device, and more particularly to a device for monitoring the facial orientation and eye condition of a vehicle driver using a neural network.

[Background Art] Various support systems have been proposed to ensure safe driving of vehicles such as automobiles.

Such support systems mainly detect when a vehicle driver is falling asleep or looking inattentive, and give a warning to the driver to enable safe driving.

As an example of such a support system for safe driving, there is a vehicle safe driving support system disclosed in Japanese Patent Application Laid-Open No. 63-222940. In this support method,
A monitoring device is provided to monitor the driver's eyes, and information from the monitoring device is used to determine whether the driver's eyes are pointing in the correct forward direction and whether the eye opening degree is equal to or greater than a predetermined value.

This judgment is based on whether the direction of the eyes is more than a certain distance from the center line in the forward direction and whether the visual field of the eyes is within a predetermined range.
This is done by making a determination by the PU. When it is determined that inattentive driving or drowsy driving is occurring, a warning or automatic brakes are gently applied.

Alternatively, as a system specifically for detecting drowsy driving, a drowsy driving prevention device disclosed in Japanese Patent Laid-Open No. 178596/1983 is known.

This device is provided with a detection means for detecting the state position of the vehicle driver, detects drowsy driving from periodic fluctuations in the detected state position, and issues a warning to the driver.

[Problems to be Solved by the Invention] However, in the automobile safe driving support system disclosed in Japanese Patent Application Laid-Open No. 63-222940, inattentive driving or drowsy driving is detected by monitoring only the state of the driver's eyes. Therefore, it is necessary to accurately extract the characteristic part, that is, the eyeball part, and there is a problem in that it is not easy to extract the characteristic part, that is, the eyeball part. Additionally, when driving a vehicle, the driver does not always focus only on the front, but often turns his or her face to either the left or right as appropriate, and in such cases, it is difficult to accurately see the field of view. Another problem was that it was difficult to measure.

Of course, in order to solve this problem, it has been proposed to attach optical markers in advance to the characteristic parts that should be Δ-j (Television Society Technical Report, Vol. 12, No. 2).
4, pp. 13-18), it is not practical to apply such an optical marker during actual vehicle operation. Alternatively, a method has been considered in which the inner corners of the eyes and the end points of the lips are used as reference points to detect the direction of the face, and the direction is detected by focusing on the plane symmetry of the facial structure (Information Processing Society of Japan, 5C -2
, pp162-163), the number of vehicle drivers is unspecified, and there is a problem that the method cannot be applied to all drivers.

In addition, the drowsy driving prevention device disclosed in Japanese Patent Application Laid-open No. 178596/1983 detects a pattern unique to the driver that occurs when drowsiness occurs, that is, periodic fluctuations in position, and determines drowsiness while driving. However, there is a problem in that such a unique pattern does not necessarily appear when drowsy driving occurs, and it cannot be detected accurately.

Therefore, the applicant of the present application previously proposed the following vehicle driver monitoring device in Japanese Patent Application No. 2-51332. This vehicle driver monitoring device includes an imaging means for photographing the face of the vehicle driver, and a neural network that processes facial image information from the imaging means to determine the direction of the vehicle driver's face and the direction of the eyes of the vehicle driver. This is to detect the state of.

However, there is a problem in that it is difficult to cut out an image to extract image information to be input into a neural network from an image obtained by the imaging means, and the processing takes a long time.

The present invention has been made in view of the above-mentioned problems of the prior art, and its purpose is to provide an improved vehicle capable of quickly and reliably detecting the facial orientation and eye conditions of an unspecified number of vehicle drivers. An object of the present invention is to provide a driver monitoring device.

[Means for Solving the Problems] In order to achieve the above object, a vehicle driver monitoring device according to the present invention includes an imaging means provided at a predetermined position within the vehicle;
a detection means for detecting the presence or absence of a vehicle driver; and when the detection means detects that there is no vehicle driver and when the vehicle driver is detected as present, the imaging means is driven to capture a background image and a vehicle, respectively. an imaging control means for obtaining an image of the driver; a position calculation means for calculating the position of the face of the vehicle driver from a difference between the obtained background image and the image of the vehicle driver; and a face calculated by the position calculation means. Detects the direction of the vehicle driver's face and the state of the eyes using a neural network that connects multiple layers of neural units that manually generate an image of the vehicle driver's position and output according to the weighted sum of inputs. It is characterized by comprising a state detection means.

[Operation] The vehicle driver monitoring device of the present invention has such a configuration, and the presence or absence of the vehicle driver is detected by the detection means.

Then, when the presence of the vehicle driver is detected, the imaging control means drives the imaging means provided at a predetermined position within the vehicle to take a photograph, thereby obtaining a background image in which the vehicle driver is not present.

Further, when the detection means detects that the vehicle driver is present, the imaging control means drives the imaging means to take a photograph to obtain an image of the vehicle driver.

By determining the difference between the background image obtained in this way and the vehicle driver image, an image of only the vehicle driver can be obtained, and the position of the face can be calculated from this image of only the vehicle driver. becomes easier.

The facial image of the vehicle driver is then processed by a pre-trained neural network to detect the direction of the face and the state of the eyes.

Since neural networks do not require consideration of predetermined algorithms and can process a large number of parameters at high speed, they are particularly suitable when it is necessary to process a large amount of image information in a short period of time, as in the present invention. In addition, it has the ability to adapt to various image patterns through learning, so it can accommodate a large number of unspecified drivers.

Then, it becomes possible to detect the occurrence of distracted driving or drowsy driving based on the direction of the driver's face and the state of the eyes detected using a neural network, and it is possible to obtain effective information for safe driving. Become.

[Embodiments] Hereinafter, preferred embodiments of the vehicle driver monitoring device according to the present invention will be described with reference to the drawings.

First example FIG. 1 is a block diagram of a first embodiment of the present invention.

The TV left camera o as an imaging means is located at a predetermined position in the vehicle,
For example, it is installed near the front panel. This TV right camera 0 is, for example, a CCD (Charge Coupled
Device) Consists of a solid-state image sensor.

In order to obtain a clear image with this TV right camera 0, a lighting device may be provided at a predetermined position of the vehicle to illuminate the driver's face. Furthermore, a filter may be applied to this TV right camera 0 so that only the irradiated light from this illumination device is incident.

A CP IL4 is provided as an imaging control means for driving this TV right camera 0.

As shown in FIG. 1, this CPU 4 includes a pressure sensor 1 that detects the driver's seat pressure, a door sensor 2 that detects the opening/closing of the door on the driver's seat side, and an ignition switch that turns on the ignition switch.
, each detection signal from the ignition sensor 3 that detects OFF is input, and based on the detection signals from these sensors, a timer is activated as appropriate to detect the presence or absence of a vehicle driver.

The operation of this CPU 4 will be explained in detail below using FIG. 2.

FIG. 2 shows an operation flowchart of the CPU 4 in this embodiment. First, in step 100, it is determined based on the detection signal from the door sensor 2 whether the driver's seat side door is open (SWI?). If the result of step 100 is YES, that is, if it is determined that the door on the driver's seat side is open, the process proceeds to step 102.

In this step 102, a timer is activated based on the detection signal from the pressure sensor 1, and it is determined whether a predetermined period of time, for example, one hour has elapsed since the vehicle driver got off the vehicle (SW2?). If YES in this step 102, that is, it is determined that a predetermined period of time has elapsed since the vehicle driver got off the vehicle, the process moves to the next step 104, and a drive signal is sent to the TV right camera 0 to display the background image P. to photograph.

Note that the situations where YES is determined in both steps 10o1 and 102 described above correspond to the following situations. In other words, if a sufficient amount of time has passed since the vehicle driver exited the vehicle and the driver's side door is open, it means that the vehicle driver is opening the driver's side door to get into the vehicle. Compatible. At this time, the vehicle driver is not yet seated in the seat, so when the TV right camera 0 photographs the vicinity of the driver's seat based on the drive signal from the CPU 4, a background image P is obtained in a state where the vehicle driver is not present. will be photographed.

The background image Po photographed in this way is the first
As shown in the figure, each pixel is binarized into O or 1 by the binarization circuit 12 and stored in the storage device 5.

On the other hand, in step 100 NOl, that is, the door sensor 2
When it is determined that the driver's side door is not open based on the detection signal from The process moves to step 106.

In this step 106, it is determined whether the vehicle driver is sitting on the seat based on the detection signal from the pressure sensor 1 (SW5?). Then, in this step 106, Y
If ES, that is, it is determined that the vehicle driver is sitting in the seat, the process moves to the next step 108, and it is determined whether or not the vehicle driver is actually driving the vehicle. That is, in this step 108, a timer is activated based on the detection signal from the ignition sensor 3, and it is determined whether a predetermined time, for example, 30 minutes has elapsed since the ignition was turned on (SW3?), and YES, that is, the predetermined time after the ignition was turned on. If it is determined that the vehicle driver has elapsed, it is assumed that the vehicle driver is actually driving the vehicle, and the process moves to the next step 11.0.

In step 110, it is determined whether or not a vehicle driver image, which will be described later, can be photographed. That is, in this step 110, the background image P is used in step 104 described above. It is determined whether a predetermined time has elapsed since the photograph was taken (SW47). If it is determined that sufficient time has not yet passed since the background image Po was photographed, that is, the current photographing is the first photographing, the process moves to the next step 112, and step 1
Similarly to 04, the TV left camera 0 photographs the image P1 in response to a drive signal from the CPU 4. Note that this step 112
The background image P photographed in step 104 is photographed in step 104 . In addition to this, an image of the vehicle driver will be taken while the vehicle driver is sitting on the seat.

After the vehicle driver image P1 is photographed in this way, the process moves to the next step 114 and the image differences ΔP-P, -P are obtained.

is calculated.

This difference has the following physical meaning.

That is, as shown in FIG. 4(A), a background image P in which a vehicle driver does not exist. and this background image P as shown in FIG. 4(B). By taking the difference from the vehicle driver image P1 when the vehicle driver is present in addition to the above image, an image ΔP in which only the vehicle driver is present is obtained as shown in FIG. 4(C).

After the image difference ΔP is calculated in step 114 in this way, the process moves to the next step 116, where the position of the vehicle driver's head is specified.

Identification of the head position is achieved by the following process. That is, the image difference Δ obtained in step 114 described above
The absolute value of P is calculated, and a two-dimensional profile of this absolute value ΔP is calculated. For example, if the X and Y directions are taken as shown in Fig. 4(C), in the profile of the absolute value △P in the y direction, △P-0 corresponds to the top of the vehicle driver's head. , this position is y. shall be.

On the other hand, the profile of the absolute value ΔP in the X direction is approximately symmetrical, and the center position of this symmetrical shape is defined as Xo. and,
The coordinates (xOlyO) determined in this way correspond to the central vertex of the vehicle driver's head, and the position of the head is specified using this as the reference coordinate.

In addition, instead of detecting the top of the vehicle driver's head in this way, for example, the part that has an extreme value in the profile of the absolute value ΔP in the y direction usually corresponds to the neck part of the vehicle driver,
By taking the intermediate value between the extreme value corresponding to this neck region and the coordinates of the top of the vehicle driver's head where the absolute value ΔP is 0, it is possible to specify approximately the center of the vehicle driver's face.

In this way, after the position of the vehicle driver's head is specified in step 116, the process moves to the next step 118, and the head position coordinates are stored in the storage device 5.

The information regarding the position of the vehicle driver's head stored in the storage device 5 in this way is read out in step 120, which is proceeded to when the determination is No in step 110.
The signal is sent to the cutout circuit 14 shown in the block diagram of FIG. 1 and used for processing to cut out only the pixels necessary for image processing.

The cutout circuit 14 cuts out a predetermined face area to be processed based on the received position coordinates. An example is shown in FIG. FIG. 5(A) shows the face of the vehicle driver photographed by the TV left camera 0, and from this face image the storage device 5
A window is set in a predetermined range near the eyes based on the position coordinates sent from the camera, and image information in this area is cut out as shown in FIG. 5(B). Furthermore, as mentioned above, CO
When using D solid-state image sensor etc. as TV left camera 0,
The face image information is composed of a plurality of pixel information arranged two-dimensionally as shown in FIG. Become. Now, numbers are assigned to the plurality of pixels arranged two-dimensionally in this way, and the pixel located in the i-th row and j-th column is written as a+j, and its pixel information is written as 11j. The face image information 1. is extracted in this way. j (having either a logic level 1 corresponding to the white level or a logic level O corresponding to the black level) is input to the neural network 16, and is subjected to the processing described below. Note that these binarization processing, cutout processing, and neural network processing described below are performed using DSP (Dlgital Signa).
1 Processor).

Hereinafter, the processing performed on this neural network 16 will be explained in detail using FIG. 6. As is well known, a neural network has a hierarchical structure of an input layer, an intermediate layer, and an output layer, and each layer is composed of a group of neural units. Neural units are given predetermined weights (coupling coefficients)
It is a unit that converts and outputs according to a certain rule according to the sum of inputs to which w+j is added. Various functions are used as this rule, but in this example, when the sum of inputs "net" is set to net-ΣW1jII, f-1/fl+exp [-(net+a)El, where α is a constant s1gIIloid function We decided to use The value range of this function is 0 to 1, and exhibits a characteristic that as the input value increases, it approaches 1, and as the input value decreases, it approaches O.

Now, the binarized face image information from the extraction circuit 14, that is, the image information 11j (0 or 1) of the pixel a1j that constitutes the extracted image, is input to the input layer of the neural network shown in FIG. It is input to each constituent neural unit. Therefore, the extracted image information is 2
If it is composed of 40×40 pixels, there will be 9600 neural units, which is the same number as this number of pixels. If you want to reduce the number of neural units in the input layer, divide the extracted image into several small regions and perform averaging processing to add up the pixel information in the divided small regions to reduce the number of neural units. It is better to reduce For example, if the small area is set to 4x4, the number of neural units is reduced to 60x10-600.

After the pixel information llj of each pixel constituting the image cut out in this way is input to each neural unit of the input layer, a predetermined conversion process is performed in the intermediate layer and output layer, which are made up of a group of similar neural units. That is, the output value of the first neural unit in the input layer that is input to the j-th neural unit in the intermediate layer is 1. Letting the weighting or coupling coefficient at this time be wlj, the total sum net of the human power values input to this j-th neural unit is: net, -ΣW .■.

J ij l , and its output becomes y,−f (net,) J by using the s1gmoid function as described above. Then, the above-mentioned processing is performed in all the neural units existing in the intermediate layer, and the output values are manually inputted to each neural unit in the output layer.

The same conversion process as in the intermediate layer is also performed in the neural unit of the output layer. In other words, the first neural unit in the intermediate layer that is input to the jth neural unit located in the output layer
If the output from the th neural unit is yo, and the weighting at this time is w, 6', then the total sum of input values input to the jth neural unit is net, - is net, -Σ1・°゛yl J IJ, and the output value O0 at this time is O, mf (net, -) JJ. Note that the weighting W i from the input layer to the intermediate layer
The values of j and weighting wIj- from the intermediate layer to the output layer are:
Adjustments are made in advance by learning so that the difference between the actual output value and the desired output value from the output layer is reduced. In addition, when converting the output value from the sum of input values in each neural unit, instead of calculating using the conversion function f each time, the sum value inputted in advance to ROM etc. and the output value at that time are The conversion process may be performed by storing the data as a map and reading it from the ROM without performing arithmetic processing.

By using the neural network in this way, one set of output values corresponding to the - set of input image information can be obtained from each of the neural units 1, 2, . A predetermined state corresponds to l, and the direction of the vehicle driver's face and the state of 1 can be determined based on the output value from this output layer.

That is, as shown in Figure 6, the output from the first neural unit in the output layer means that the driver's face is facing forward, and the output value from the second neural unit means that the driver's face is facing forward. corresponds to the driver's eyes being closed, and the output value from the first neural unit corresponds to the driver facing towards the instrument panel. Therefore, the output value from one neural unit constituting the output layer is (1°0.0
．． ..., 0), it is detected that the vehicle driver's face is facing forward and his eyes are open. Information about the direction of the vehicle driver's face and the condition of his eyes detected in this way is then sent to the ECU, which performs electronic control of the vehicle.
(Electronic Control Unit
) is entered.

This ECU 18 receives input information regarding the direction of the driver's face and the state of the eyes, and an obstacle sensor 20 such as a laser radar.
Based on information from an environment recognition sensor 22 that recognizes the road environment on which the vehicle travels, such as a camera or the like, a warning device, an accelerator actuator, and a brake actuator are controlled in order to drive the vehicle safely. For example, (a) If the vehicle driver's face and eyes are turned to the side, brake operation is performed to increase the distance between the vehicle and the vehicle in front.

(b) If it is detected that the vehicle driver's eyes are open and closed for a long time and he is drowsy while driving, a warning is given to the driver via the warning means.

(c) If the face and eyes of the vehicle driver are turned to the side and the vehicle is traveling around a curve, a warning is given through the warning means and the brakes are operated.

(d) If the vehicle driver's eyes are not directed toward the room mirror or the right side mirror and the vehicle makes a right turn or changes diagonally to the adjacent diagonal line on the right, a warning is given through the warning means.

etc. to ensure safe driving of the vehicle.

In the first embodiment, a so-called forward connection type neural network was used in which output is sequentially performed from the input layer to the intermediate layer and then to the output layer, but there is also a connection from the output layer to the input layer. A so-called mutually coupled neural network may also be used.

Further, in the first embodiment, there is only one intermediate layer, but the same processing can be performed even if a plurality of intermediate layers are provided.

Second Embodiment FIG. 3 is a flowchart showing processing by the CPU 4 in a second embodiment of the present invention. In the first embodiment described above, the vehicle driver captures a background image immediately before getting into the vehicle and an image of the vehicle driver within a predetermined time after capturing this background image, and calculates the difference between the images to determine the position of the vehicle driver's head. However, in the second embodiment, the vehicle driver photographs a background image within a predetermined time period for getting off the vehicle.

For this reason, the second embodiment employs the following processing flow. That is, first, step 200
Based on the detection signal from the ignition sensor 3, it is determined whether the ignition is turned off. Then, in this step 200, YES, that is, the ignition is turned on.
If it is FF, the process moves to step 202.

In this step 202, it is determined whether or not a predetermined time has elapsed since the timer was activated by the detection signal from the door sensor 2 and the driver's seat side door was locked (SW7?). Then, when it is determined YES in this step 202, the CP
A drive signal is sent from U4 to the TV left camera 0, and the TV left camera 0 captures the background image P as shown in step 204.

to photograph.

Here, the situation where YES is determined in steps 200 and 202 corresponds to a situation where the vehicle driver is not present after the vehicle driver has stopped driving, turned off the ignition, got out of the vehicle, and locked the door. In this way, the background image after the vehicle driver gets off the vehicle is photographed by the TV left camera 0.

On the other hand, when it is determined in step 200 that NOl, that is, the ignition is ON, and in step 202, NOl is
That is, when a predetermined period of time has elapsed since the door on the driver's seat side was locked, the process moves to step 108.

In step 108, it is determined whether a predetermined period of time has elapsed after the ignition was turned on, as in the first embodiment described above.
If YES, that is, it is determined that the predetermined time has elapsed, it means that the vehicle driver is actually driving the vehicle, and the process moves to the next step 2.8.

In this step 208, it is determined whether the vehicle's shift lever is set to a range other than P (parking), and if YES, that is, the shift lever is set to a range other than P, the vehicle is not in operation. This means that a vehicle driver image P1 is photographed by going through steps similar to steps 112 to 118 in the first embodiment, and a background image photographed earlier is photographed. The position of the vehicle driver's head is determined by calculating the difference between the two.

In this way, in the second embodiment, the presence or absence of a vehicle driver is determined based on whether the ignition is turned on or not and whether the shift lever is set to a position other than the P range, and a special sensor such as a pressure sensor is used to determine the presence or absence of a vehicle driver. There is an advantage that the detection means can be constructed without using a sensor.

In the first and second embodiments described above, information regarding the position of the vehicle driver's head calculated by the CPU 4 is sent to the cutting circuit 14 to cut out a desired area. In addition to installing the camera near the front panel of the vehicle, a second TV left camera is also installed in the deck room at the rear of the vehicle to take pictures of the rearview mirror that shows the driver's face. It may be configured.

In this case, since the second TV left camera has already captured only the face of the vehicle driver, the image obtained by the second TV left camera and the T in the first and second embodiments described above are
By comparing it with the image taken by the V left camera 0 and cropped by the cropping circuit 14, and adopting the cropped image if the amount of difference between the two images is within a predetermined tolerance range. , it is possible to further improve the system.

[Effects of the Invention] As explained above, according to the vehicle driver monitoring device according to the present invention, it is possible to quickly and reliably detect the face direction and eye condition of an unspecified number of vehicle drivers. Become.

[Brief explanation of drawings]

Fig. 1 is a block diagram of the configuration of the embodiment, Fig. 2 is a processing flowchart of the CPU of the first embodiment, Fig. 3 is a processing flowchart of the CPU of the second embodiment, and Fig. 4 is a processing flowchart of the CPU of the second embodiment. FIG. 5 is an explanatory diagram of image differences, FIG. 5 is an explanatory diagram of window settings in the embodiment, and FIG. 6 is a configuration diagram of a neural network in the embodiment. 1... Pressure sensor 2... Door sensor 3... Ignition sensor 4... CPU 5... Storage device 14... Extraction circuit 16... Neural network 18...
ECU

Claims (1)

[Scope of Claims] An imaging means provided at a predetermined position within a vehicle, a detection means for detecting the presence or absence of a vehicle driver, and when the detection means detects that there is no vehicle driver; an imaging control means that drives the imaging means to obtain a background image and a vehicle driver image, respectively, when the presence of the vehicle driver is detected; A position calculation means for calculating the position of the vehicle driver, and a neural unit group that inputs the image of the vehicle driver at the face position calculated by the position calculation means and outputs it according to the weighted sum of the inputs are connected in multiple layers. A vehicle driver monitoring device comprising: a state detection means for detecting the direction of the face and the state of the eyes of the vehicle driver using a neural network.