JP5935759B2 - Driver status determination system - Google Patents

Description

  The present invention relates to a driver state determination system that can determine the state of a driver (driver) looking aside, sleepiness, anger, and the like.

  In recent years, the driver's face while driving a car is photographed with a camera placed in the passenger compartment such as a dashboard, and the photographed driver's face image (face image) is analyzed to help the driver doze or look aside. Detection technology has been developed.

  For example, in Patent Document 1 below, it is determined whether or not the driver is in a strong sleepiness state from the face image, and if there is a strong sleepiness, not only the driver itself but also a family other than the driver to that effect. Have been disclosed, and techniques for notifying family members by e-mail or the like are disclosed.

JP 2007-304705 A

  However, the above-described conventional technology has a problem in that the vehicle is not equipped with a camera that captures the driver's face in advance, a device that determines the state of the driver from the face image, and the like, and is not sufficiently versatile. It was.

  The present invention has been made to solve the above-mentioned problems, and its purpose is to determine the state of the driver without providing a special device in the automobile and to provide necessary information to the driver, surrounding people, etc. It is an object of the present invention to provide a driver status determination system capable of performing the above.

  The driver state determination system according to the present invention includes an image acquisition unit that acquires an image in which a driver's face and a vehicle number are captured by a plurality of imaging units arranged along a travel route of the vehicle, and the image acquisition unit. Data accumulating means for associating and accumulating the face image data and the number data in each acquired image, and determining the state of the driver based on the face image data accumulated by the data accumulating means Driver state determination means, and information presentation means for presenting information relating to the state of the driver determined by the driver state determination means to an occupant of the vehicle or to occupants of other vehicles around the vehicle; It is provided with.

  In the present invention, the face of the driver and the vehicle number are photographed (according to time series) from the outside of the traveling vehicle by the photographing means arranged along the traveling route (road) of the vehicle. Since the face image data and the number data in these images are stored in association with each other, it is possible to know which vehicle the data relates to. Then, based on the accumulated face image data (in time series), the driver state is determined, and information on the determined driver state (for example, information indicating aside, sleepiness, anger, etc.) Present it to yourself, its passengers and other passengers in the surrounding area. Therefore, the information that has been presented can be appropriately dealt with in accordance with the presented content (for example, taking care of the driver or encouraging a break).

  Therefore, even in the case of a vehicle that does not have a device for determining the state of a camera or a driver in the vehicle itself as in the prior art, the state of the driver during driving can be accurately determined, so that the versatility is high. . Moreover, there exists an advantage that required information can be provided suitably based on the determination result.

  Here, the face image is a face image of the captured image, and the face image data includes a face obtained by analyzing the face image in addition to the image data of the face image itself. Analysis data (for example, analysis data such as face orientation, eye open / close state, facial expression, etc.) indicating the characteristics of the image can be mentioned.

  In the present invention, the direction of the driver's face is detected based on the face image data (image data), the driver's eye closed state (and hence sleepiness) is detected, and the driver's facial expression (and thus emotion). Can be detected.

It is explanatory drawing which shows schematic structure of the driver state determination system of Example 1. FIG. 1 is a block diagram functionally illustrating a driver state determination system according to a first embodiment. 3 is a flowchart illustrating control processing in the entire system of the driver state determination system according to the first exemplary embodiment. It is a flowchart which shows the control processing in a driver state detection apparatus. It is a flowchart which shows the control processing in a driver state determination apparatus. It is explanatory drawing which shows schematic structure of the driver state determination system of Example 3. FIG.

  Next, an embodiment of the driver state determination system of the present invention will be described with reference to the drawings.

a) First, the basic configuration of the driver state determination system of this embodiment will be described with reference to FIG.
In the driver state determination system 1 of the present embodiment, a plurality of poles 3 are set up along a road, and a photographing device 7 is attached to each pole 3 for photographing a car (vehicle) 5. . An electric bulletin board 9 is attached to some poles 3.

  In addition, the photographing device 7 includes an irradiation unit 11 that irradiates the vehicle 5 with near-infrared light, and a camera (for example, a CCD camera) 13 for photographing a face of an occupant (particularly a driver) of the vehicle 5 and a license plate of the vehicle 5. And.

  Further, a driver state detection device 15 capable of extracting facial features from a facial image (face image) photographed by the camera 13 is incorporated in the pole 3. The driver state detection device 15 is an electronic control device (having an image processing function) provided with a known microcomputer.

  As will be described later, the driver state detection device 15 analyzes the face image to detect the open / closed state of the eyes, facial expression, face orientation, and the like, and information on the driver's face detected from the face image is obtained. It is configured to transmit to the driver state determination device 19 which is a central processing unit via the communication line 17. The driver state determination device 19 is an electronic control device including a known microcomputer.

  Further, the driver state detection device 15 reads the number from the license plate of the vehicle 5 by known image processing and associates it with the information related to the driver's face, that is, the driver 5's information can be known. Then, it is configured to transmit to the driver state determination device 19.

In addition, as shown in FIG. 2, a flash 37 for blinking and irradiating light and a speaker 39 for outputting sound and sound are also disposed in the vicinity of the electronic bulletin board 9.
b) Next, the configuration of the driver state determination system 1 will be described in more detail with reference to FIG.

  As shown in FIG. 2, each driver state detection device 15 is functionally (that is, configured to perform processing of a microcomputer), and includes a vehicle traffic detection unit 21, an image acquisition unit 23, a face orientation detection unit 25, an eye A state detection unit 27, a facial expression detection unit 29, and a number detection unit 30 are provided.

  Among these, the vehicle traffic detection part 21 detects the vehicle 5 which is drive | working on the road from the image of the camera 13, for example. Note that the traffic of the vehicle 5 may be detected using information obtained from a known radar or the like.

The image acquisition unit 23 inputs (acquires) data of an image captured by the camera 13 and temporarily stores it in a memory or the like.
The face orientation detection unit 25 detects the face orientation of a driver or a passenger based on the captured face image data (face image data) by a known image processing method to be described later.

The eye state detection unit 27 detects the open / closed state of the driver's eyes based on the face image data by a known image processing method to be described later.
The facial expression detection unit 29 detects the facial expression (for example, anger, irritation, and smile) of the driver based on the facial image data by a known image processing method to be described later.

The number detection unit 30 reads the number of the vehicle 5 from an image obtained by photographing the number plate of the vehicle 5 by a known image processing method.
On the other hand, the driver state determination device 19 functionally includes a data storage unit 31, a driver state determination unit 33, and an information presentation unit 35.

  Among them, the data storage unit 31 includes information on the face orientation of the driver and passengers detected by the face direction detection unit 25, information on the open / closed state of the driver's eyes detected by the eye state detection unit 27, and the face The information of the facial expression of the driver detected by the facial expression detection unit 29 and the information of the number of the vehicle 5 that the driver is driving are stored.

  Since the status of these drivers and the like is sequentially sent from each driver status detection device 15 as the vehicle 5 travels, the data storage unit 31 sequentially stores the information in time series. Go.

  Further, when this information is stored, information on the state of the driver or the like and information on the number of the vehicle 5 are stored in association with each other. In other words, the information of the face of the driver or the like is stored so that the information of the driver or the like of which vehicle 5 is known.

  The driver state determination unit 33 determines the state of the driver or the like (that is, whether the driver is looking aside, sleepy, angry, or the like) based on the facial information of the driver or the like stored in the data storage unit 31. .

  The information presenting unit 35 drives the electronic bulletin board 9, the flash 37, and the speaker 39, which are information presenting devices, based on the state of the driver and the like determined by the driver state determining unit 33, and the driver himself / her passengers and surrounding vehicles 5 shows necessary information.

c) Next, control processing performed in the driver state determination system 1 of the present embodiment will be described with reference to FIGS.
<System-wide processing>
This process shows the entire control process performed by the driver state determination system 1.

As shown in FIG. 3, in step 100 (S), the vehicle 5 and the passenger are photographed by the camera 13 of each pole 3 when the vehicle 5 passes near the pole 3.
Each driver state detection device 15 detects the state of the face of the driver or the like from the photographed image and the vehicle number, and transmits the information to the driver state determination device 19.

  The driver's face and the vehicle number may be detected from the same image, or may be detected from images taken individually (an image obtained by photographing a face and an image obtained by photographing a license plate).

In the following step 110, the state of the driver is determined based on the information transmitted from the driver state detection device 15 and accumulated.
In the following step 120, it is determined whether or not an information presentation flag that is a flag for permitting presentation of information to the outside is set (the information presentation flag is 1). Here, if an affirmative determination is made, the process proceeds to step 130. On the other hand, if a negative determination is made, the present process is temporarily terminated.

Here, as the information presentation flag, there are an information presentation flag indicating aside, an information presentation flag indicating sleepiness, and an information presentation flag indicating negative emotion such as anger, as will be described later.
In step 130, since the presentation of information is permitted, the information presentation unit 35 drives the electronic bulletin board 9, the flash 37, and the speaker 39 as appropriate according to the content of the information presentation flag, as will be described later. This processing is once terminated.

<Processing in Driver State Detection Device 15>
This process indicates a control process performed by the driver state detection device 15.
As shown in FIG. 4, in step 200, processing for detecting the vehicle 5 is performed based on an image photographed by the camera 13 by known image processing. For example, a process of detecting an object of another color that moves at a predetermined speed or more as a vehicle 5 on a gray road surface is performed.

When the vehicle 5 is detected from an image, it is desirable to irradiate the vehicle 5 with near infrared light when taking a picture.
As is well known, a radar wave or the like is emitted from a radar or the like, and the traveling vehicle 5 is detected by the reflected wave. Only when the vehicle 5 is detected, the near infrared light is emitted and photographed. Also good.

  In the following step 210, it is determined whether or not the vehicle 5 has passed. If an affirmative determination is made here, the process proceeds to step 220. If a negative determination is made, the process returns to step 200. That is, if a moving object as described above is detected in step 200, it is determined that the vehicle 5 is passing.

In step 220, since there is a vehicle 5 that actually travels, in order to perform image processing to be described later, processing for capturing an image of the vehicle 5 and storing it in the memory is performed.
In the following step 230, a process for detecting the face direction of the driver is performed. Since the technique for detecting the orientation of the face is known (see, for example, Japanese Patent No. 4725254), it will be briefly described. For example, the position of the nose can be recognized from the face image, and the orientation of the face can be detected based on how much the face is displaced from which side around the nose.

  In the subsequent step 240, processing for detecting the open / closed state of the eyes is performed. Since the technique for detecting the open / closed state of the eye is known (see, for example, Japanese Patent No. 4259585), it will be briefly described. For example, the open / closed state of the eyes can be detected by grasping the distance between the upper eyelid and the lower eyelid from the face image and determining whether the distance is smaller than a predetermined value. Alternatively, the open / closed state of the eyes may be determined based on whether or not black eyes can be detected.

  In the following step 250, processing for detecting facial expressions is performed. Since techniques for detecting facial expressions (facial expressions representing emotions) are known (see, for example, Japanese Patent No. 4101734 and Japanese Patent Application Laid-Open No. 2011-117905), they will be briefly described.

  For example, it is known that the facial expression can be known from the position of the parts such as the eyebrows, eyes, and mouth (mouth corner) (up and down), and the human emotion can be estimated from the facial expression. Therefore, the facial expression (hence human emotion) can be estimated by grasping the state of the part from the face image. That is, anger, irritation, a smile, etc. can be detected from the face image.

  In subsequent step 260, the information obtained in steps 230 to 250, that is, the information on the face orientation, the eye open / close state, and the facial expression as shown in FIG. Then, the data is transmitted to the driver state determination device 19, and the process is temporarily terminated.

  Information to be transmitted includes “landscape” and “front” as the face direction, “open” and “closed” as the open / closed state of the eyes, and “nothing” as the facial expression. “Expression”, “Negative expression (such as anger)”, and “Positive expression (such as smile)”.

At this time, the face direction of the passenger may also be detected. Whether it is a driver or a passenger in the front passenger seat can be determined by, for example, the position of the steering wheel.
<Processing in Driver State Determination Device 19>
This process indicates a control process performed by the driver state determination device 19.

As shown in FIG. 5, in step 300, information transmitted from each driver state detection device 15 is accumulated in the data accumulation unit 31 (such as a hard disk).
In subsequent step 310, it is determined whether or not a predetermined number of pieces of information transmitted from the driver state detection device 15 (that is, information transmitted from a predetermined number of driver state detection devices 15) is stored in the data storage unit 31. If an affirmative determination is made here, the process proceeds to step 320, whereas if a negative determination is made, the process returns to step 300.

  In the subsequent step 320, it is determined whether or not the ratio of the landscape image in the plurality of captured images is equal to or greater than a predetermined threshold (indicating a side look). That is, it is determined whether or not the ratio of “landscape” is equal to or greater than a predetermined threshold among a plurality of pieces of information along the time series indicating the face orientation transmitted from each driver state detection device 15. If an affirmative determination is made here, the process proceeds to step 330, while if a negative determination is made, the process proceeds to step 340.

In step 330, since the ratio of “sideways” is large, it is determined that the driver is looking aside, and an information presentation flag indicating aside is set to 1, and the process proceeds to step 350.
On the other hand, in step 340, since the ratio of “sideways” is small, it is determined that the operation is not looking aside, and an information presentation flag indicating aside is set to 0, and the process proceeds to step 350.

  In step 350, it is determined whether or not the ratio of the closed eye image (image with closed eyes) in the plurality of captured images is equal to or greater than a predetermined threshold value (indicating sleepiness). That is, it is determined whether or not the ratio of “closed” is equal to or greater than a predetermined threshold among a plurality of pieces of information along the time series indicating the open / closed state of the eye transmitted from each driver state detection device 15. If an affirmative determination is made here, the process proceeds to step 360, while if a negative determination is made, the process proceeds to step 370.

In step 360, since the ratio of “closed” eyes is high, it is determined that there is drowsiness, an information presentation flag indicating drowsiness is set to 1, and the process proceeds to step 380.
On the other hand, in step 370, since the ratio of “closed” is small, it is determined that there is no sleepiness, an information presentation flag indicating sleepiness is set to 0, and the process proceeds to step 380.

  In step 380, it is determined whether or not the ratio of images showing negative facial expressions (such as anger and irritation) in the plurality of captured images is greater than or equal to a predetermined threshold value. That is, it is determined whether or not the ratio of “negative facial expression” out of a plurality of pieces of information along the time series indicating facial expressions transmitted from each driver state detection device 15 is equal to or greater than a predetermined threshold. If an affirmative determination is made here, the process proceeds to step 390, whereas if a negative determination is made, the process proceeds to step 400.

  In step 390, since the ratio of “negative facial expression” is large, it is determined that there is a negative emotion (such as anger or irritation), the information presentation flag indicating negative emotion is set to 1, and the flow proceeds to step 410 .

On the other hand, in step 400, since the ratio of “negative facial expression” is small, it is determined that there is no negative emotion, an information presentation flag indicating negative emotion is set to 0, and the processing proceeds to step 410.
In step 410, the electronic bulletin board 9, the flash 37, and the speaker 39 are driven according to each information presentation flag described above to present necessary information.

  For example, when the information presentation flag indicating a side look is 1, in order to notify the driver, passengers, and the surrounding vehicle 5 of this, the warning sign on the electronic bulletin board 9 is displayed. This is notified by voice or by flash 37.

  Similarly, when the information presentation flag indicating drowsiness is 1, in order to notify the driver, passengers, and the surrounding vehicle 5 of the fact, the message board 9 displays “notice of falling asleep”, This is notified by voice or by flash 37. At that time, the number of the vehicle 5 may be presented in order to indicate information of any vehicle 5.

  Similarly, when the information presentation flag indicating negative emotion is 1, in order to notify the passengers and surrounding vehicles of this, the message board 9 displays “Let's take a break” This is notified by voice or by flash 37.

d) Next, the effect of the present embodiment will be described.
In the driver state determination system 1 of the present embodiment, the face of the driver and the number of the vehicle 5 are determined from the outside of the traveling vehicle 5 by the camera 13 of the photographing device 7 arranged along the traveling route (road) of the vehicle 5. Shoot sequentially in time series.

  Then, each image is captured from each camera 13 to each driver state detection device 15, and the face state of the driver is detected from the image (specifically, a face image). Specifically, the direction of the face, the open / closed state of the eyes, and a negative facial expression (such as anger) are detected.

  Information about these faces is transmitted from each driver state detection device 15 to the driver state determination device 19 and accumulated. When a predetermined number of data is accumulated, the driver state is determined based on the data. Specifically, the presence or absence of aside look, sleepiness, or negative emotion is determined.

Then, based on the determination result, necessary information is provided to the driver himself / her passenger and the vehicles around the vehicle using the electronic bulletin board 9 or the like.
Therefore, even in the case of a vehicle that does not have a device for determining the state of the camera or driver in the vehicle itself as in the conventional case, the state of the driver during driving can be accurately determined. Therefore, based on the determination result, there is a remarkable effect that necessary information can be suitably provided to a necessary partner.

Next, the second embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified. In addition, the same number is used for each component as in the first embodiment (the same applies hereinafter).
In the second embodiment, the driver state detection device 15 takes a picture of the face and the vehicle 5 and outputs the image to the driver state determination device 19 (without performing processing for detecting the orientation of the face as in the first embodiment). Send.

Therefore, the driver state determination device 19 detects the face direction, the open / closed state of the eyes, and the facial expression from each of the transmitted images, and determines aside look, sleepiness, and negative emotion.
Also in the second embodiment, the same effects as in the first embodiment are obtained.

Next, the third embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified.
In the third embodiment, the driver state detection device 15 and the driver state determination device 19 are not connected by the communication line 17 and communicate by radio.

For example, as shown in FIG. 6, the driver state detection device 15 may connect to the Internet 41 wirelessly and accumulate data in the server 43.
According to the third embodiment, the same effects as those of the first embodiment can be obtained.

Next, the fourth embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified.
In the fourth embodiment, there is a possibility of a collision not only on the electronic bulletin board 9 but also by the driver when there is a risk of collision when the driver's state is determined to look aside or doze in a situation where the vehicle 5 is congested. The information is presented through the high flash 37 and the speaker 39.

  Whether or not the vehicle 5 is congested can be determined based on well-known congestion information (navigation information) obtained from the outside or the number of vehicles 5 in the image captured by the camera 13.

  The fourth embodiment has an advantage that the safety is further improved.

Next, the fifth embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified.
In the fifth embodiment, there is a risk of collision when the driver's state is determined to be dozing in a situation where the vehicle 5 is congested. The number of the vehicle 5 is displayed on the electronic bulletin board 9 so as to keep a distance.

  The fifth embodiment has an advantage that the safety is further improved.

Next, the sixth embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified.
In the sixth embodiment, when it is determined that the driver is angry or irritated in a situation where the passenger is on the vehicle 5, the passenger can alert the driver. A display such as “Let's take a break” is displayed on the electronic bulletin board 9.

  The sixth embodiment has an advantage that the safety is further improved.

Next, although Example 7 is demonstrated, description of the content similar to the said Example 1 is abbreviate | omitted or simplified.
In the seventh embodiment, in the situation where there is a passenger in the passenger seat of the vehicle 5, when a driver is looking aside with the passenger facing the passenger seat facing each other, the passenger opens the electronic bulletin board 9. Since the possibility of seeing is low, a warning such as “Let's watch out for driving aside” is given through the flash 37 and the speaker 39.

  The direction of the passenger's face in the passenger seat can be detected in the same manner as the detection of the driver's face direction. Also, whether the driver is a passenger in the passenger seat or not can be determined by detecting the steering wheel, for example.

  The seventh embodiment has an advantage that the safety is further improved.

Next, the eighth embodiment will be described, but the description of the same contents as the first embodiment will be omitted or simplified.
In the eighth embodiment, when a positive facial expression such as a smile is detected as the facial expression of the driver, information such as praising smile driving is presented.

The eighth embodiment also has an advantage that the safety is further improved.
In addition, this invention is not limited to the said embodiment and Example at all, and it cannot be overemphasized that it can implement with a various aspect in the range which does not deviate from this invention.

(1) For example, taking an image with a camera may be performed constantly, but may be performed only when traveling of the vehicle is detected.
(2) Further, in each of the above embodiments, for example, the functions of one component may be distributed to a plurality of components, or the functions of a plurality of components may be integrated into one component. Further, at least a part of the configuration of the embodiment may be replaced with a known configuration having a similar function. Furthermore, at least a part of the configuration of the embodiment may be added to or replaced with the configuration of another embodiment.

  (3) In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in the Example as one aspect, and limits the technical scope of the present invention. It is not a thing.

DESCRIPTION OF SYMBOLS 1 ... Driver state determination system 5 ... Vehicle (automobile)
DESCRIPTION OF SYMBOLS 9 ... Electric bulletin board 13 ... Camera 15 ... Driver state detection apparatus 19 ... Driver state determination apparatus

Claims (5)

An image acquisition means (23) for acquiring an image in which the face of the driver and the number of the vehicle (5) are photographed by a plurality of photographing means (13) arranged along the travel route of the vehicle (5);
Data storage means (31) for storing the face image data and the number data in each image acquired by the image acquisition means (23) in association with each other;
A driver state determination unit (33) for determining the state of the driver based on the data of the face image stored by the data storage unit (31);
Information presenting means (35) for presenting information relating to the state of the driver determined by the driver state determining means (33) to an occupant of the vehicle or to occupants of other vehicles around the vehicle;
A driver state determination system comprising: Face orientation detection means (25) for detecting the face orientation of the driver based on the face image data;
Eye state detecting means (27) for detecting the open / closed state of the eyes of the driver based on the data of the face image;
Facial expression detection means (29) for detecting facial expressions of the driver based on the facial image data;
2. The driver state determination system according to claim 1, comprising at least one of the two.   When the ratio of the face image data indicating the landscape orientation of the driver among the plurality of face image data is equal to or greater than a predetermined value, the driver of the other surrounding vehicle (5) may The driver status determination system according to claim 2, wherein information is presented.   When the ratio of the face image data indicating the closed state of the driver among the plurality of face image data is equal to or higher than a predetermined value, the driver's sleepiness is given to the occupants of the other surrounding vehicles (5). The driver state determination system according to claim 2, wherein information on the driver status is presented.   Information that prompts the passenger of the vehicle (5) of the driver to rest when the ratio of the face image data indicating anger or irritation of the driver among the plurality of face image data is equal to or greater than a predetermined value. The driver state determination system according to any one of claims 2 to 4, wherein the driver state determination system is provided.