JP2937030B2 - Image processing device and person state determination device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for extracting features on the face of a person to be detected, and a person state determination for detecting the behavior of the person to be detected such as a dozing state or an inattentive state. It concerns the device.

[0002]

2. Description of the Related Art FIG.
It is a block diagram of an eye position detecting device using a conventional driver photographing device disclosed in Japanese Patent Application Publication No. 03-2003. In the figure, 1 is a driver, 10a and 10b are CCD cameras, and 13 is an infrared LE.
D and 14 are LED driving circuits for supplying a current to the infrared LED 13, and 100 is a driver behavior detecting circuit.

Next, the operation will be described. Infrared LED
13 irradiates the driver 1 in the driver's seat. The image of the driver 1 is input to CCD cameras 10a and 10b installed at positions where a predetermined area including the face of the driver 1 can be photographed. The input image is input to the driver behavior detection circuit 100,
The positions of the eyes and the direction of the face have been extracted by image processing such as edge detection and pattern recognition.

[0004] On the other hand, as shown in a paper "Prototype of eye-gaze detecting device allowing pupil extraction and head movement" (Transactions of the Institute of Electronics, Information and Communication Engineers, D-II Vol.J76-D-II No.3). In addition, when the face is illuminated with a coaxial epi-illumination device, a retinal reflection image can be remarkably photographed, and the position of the eye can be detected by a very simple image processing such as a binarization process.
This is a type of illumination in which the optical axis of the camera and the direction of illumination light illumination are the same.) FIG. 31 shows a configuration of a dozing detection device using this device.
No. 502. In the figure,
1 is a detection target person, 10 is a CCD camera installed at a position where a predetermined area including the face of the detection target person 1 can be photographed, 1
1 is a visible light cut filter for cutting visible light components, 1
2 is a half mirror installed at an angle of 45 degrees with respect to the CCD camera, 13 is an emission wavelength of 860 nm, and a directivity angle is ± 20.
Infrared LED that emits light at an angle and reflects light to the half mirror 12 to illuminate the detection target 1, 14 is an LED drive circuit that drives the infrared LED, and 30 is an imaging device that temporarily stores an image signal output from a CCD camera. A photographed image input memory as an image memory; 31, a timing generation circuit for providing a timing signal to a CCD camera, a photographed image input memory 30, etc .; 40, a feature extraction circuit for extracting a pupil from the contents of the photographed image input memory; A wakefulness determination circuit 51 that determines the wakefulness from the output of the circuit, and 51 is an alarm output circuit that outputs an alarm.

Next, the operation will be described. In FIG. 31, the illumination light of the infrared LED 13 irradiated on the optical path l1 reflects half of the light on the half mirror 12, and irradiates the face of the detection target 1 on the optical path l2. The image of the detection target 1 passes through the half mirror 12 along the optical path l3, and half of the light is
The CCD camera 10 reaches the camera 10 and the detection target person 1
Capture the image of At this time, the optical axes of the optical paths l2 and l3 are coaxial when viewed from the detection target (coaxial epi-illumination). At this time, the captured image of the detection target person is as shown in FIG. In FIG. 32, 2 is an eyebrow, 3 is a nostril, 4 is an iris,
5 is the sclera, 6 is the pupil, and 7 is the face surface. By the coaxial epi-illumination having the above configuration, the pupil 6 is observed as if the pupil is shining by the light reflected by the retina,
Brightness is significantly higher than that of the face surface or other feature points. This is because the retina has the property of returning reflected light in the same direction as incident light. FIG. 33 shows the luminance distribution of the captured image. The waveform shown in the lower part of FIG. 33 shows the luminance distribution along the line AA in the obtained face image (the upper part of FIG. 33). The position of the pupil is clearly different from the other parts in luminance. Is high. The face image captured by the CCD camera 10 is temporarily input to the captured image memory 30. FIG. 34 shows a processing flow of dozing detection by this device. This processing flow corresponds to the feature extraction circuit 40 in FIG.
3 shows the operation of the wakefulness determination circuit 50. The feature extraction circuit 40 inputs a face image in ST10, and performs a binarization process with an appropriate luminance threshold th1 in order to extract only the pupil in ST11. The obtained binarized image is as shown in FIG. In this image, only the pupil becomes two circles as a white area, and the other part becomes a black area. Therefore, projection of white pixels in the Y-axis direction is performed in ST12 (a projected image is shown on the right side of FIG. 35), and in ST13. By determining the area of the projected image, the apparent size of the pupil can be calculated. In ST14, it is determined that the eye is open when the area is equal to or larger than a predetermined value, and that the eye is closed when the area is smaller than a predetermined value. This eye open / closed information is input to the arousal level determination circuit 50, and the arousal level determination circuit 50
It is determined whether or not the eye-close information has been continuously input for 3 seconds or more. If the eye-closed state has been input for 3 seconds or more, it is determined that the user is dozing, and an alarm output command is output to the alarm generation circuit 51 in ST21. . If the closed state is less than 3 seconds, the process proceeds to ST10.

[0006]

A conventional image processing apparatus for extracting features on the face is constructed as described above. In the driver's photographing apparatus shown in FIG. 30, the position of the pupil is detected. Then, the obtained image is subjected to, for example, a filter operation to detect an edge, and from this edge shape, a Hough transform process or the like for obtaining a circle corresponding to a pupil is performed to perform pattern recognition and search for a pupil. Also, in the edge detection, if the edge is not properly detected due to noise or the like, a circle is not obtained, and there is a problem that image processing is complicated and takes time. In addition, the above-mentioned dozing detection device according to the prior art using coaxial epi-illumination (FIG. 31) operates normally when used under conditions of low disturbance light, but does not operate under conditions of high disturbance light such as when used in a car. However, there is a problem that a retinal reflection image cannot be obtained due to extremely strong disturbance light (sunlight) that is 100 to 1000 times or more as much as the illumination light for performing the operation. FIG. 36 is an explanatory diagram for explaining the influence of the above-mentioned disturbance light. The waveform shown in the lower part of FIG. 36 shows the luminance distribution along the line AA in the obtained face image (upper part of FIG. 36), and the curve A is a daytime image in fine weather, which is captured by coaxial epi-illumination. Curve B is a daytime image in rainy weather, taken by coaxial epi-illumination, and curve C is a nighttime image, taken by coaxial epi-illumination.
In the case of coaxial incident light, as the disturbance light increases (curve C; night → curve B; daytime / rainy → curve A; daytime ・
It can be seen that the brightness of the portion other than the pupil 6 becomes high, and it becomes difficult to identify the pupil. An image captured by the curve A is as shown in FIG. In this image, the sclera 5 appears slightly darker than the face surface 7, the iris 4 appears darker than the sclera 5, and the pupil 6 appears even darker. As described above, the image processing method in the dozing detection device has a problem that it is impossible to extract a feature point on the face under the condition that the disturbance light condition changes greatly.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and it has been proposed that the characteristics of the face of the person to be detected and the state of the person to be detected can be obtained even under conditions where disturbance light greatly changes, such as on-vehicle conditions. It is an object of the present invention to obtain an image processing device and a person state determination device capable of detecting the image in a relatively short time.

[0008]

According to a first aspect of the present invention, there is provided an image processing apparatus comprising: a light input unit for inputting an image of a predetermined area including a face of a detection target; illuminating the detection target;
An illuminating means arranged such that an optical axis connecting the detection target person and the light input means and an illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target person, and detects light and darkness around the detection target person Brightness detection means for detecting the brightness of the image of the detection subject output from the light input means.
For dark areas on the face where
Image processing and bright areas on the face whose brightness is higher than the threshold
At least two or more images of dark image processing to extract regions
By image processing, features on the face can be extracted, and one image processing is selected from two or more image processings based on the light / dark signal output by the light / dark detection means, or
It is provided with a feature extracting means for selecting one extraction result from the extraction results obtained by one or more image processing.

According to a second aspect of the present invention, there is provided an image processing apparatus, comprising: a light input unit for inputting an image of a predetermined area including a face of a person to be detected, and outputting a light / dark signal indicating light / dark of the input image; An illuminating unit that illuminates the detection target, and is arranged so that an optical axis connecting the detection target and the light input unit and an illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target; and The brightness is determined from the image of the detection target person output from the light input means.
Bright image processing to extract dark areas on the face that are lower than the value
And extract bright areas on the face whose brightness is higher than the threshold
Image processing for at least two or more of
Thus, the feature on the face can be extracted, and one image processing is selected from two or more image processings based on the light / dark signal output from the light input means, or an extraction result by two or more image processings And a feature extracting means for selecting one of the extraction results.

According to a third aspect of the present invention, there is provided an image processing apparatus, comprising: light input means for inputting an image of a predetermined area including a face of a detection target; illuminating the detection target; An illuminating means arranged such that an optical axis connecting the input means and the illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target, and the detection target or another person switches corresponding to light and dark. A switch for outputting a signal corresponding to light and dark, and a luminance threshold from the image of the detection subject output from the light input means.
Bright image processing to extract dark areas on the face that are lower than the value
And extract bright areas on the face whose brightness is higher than the threshold
Image processing for at least two or more of
The feature on the face can be extracted, and one image processing is selected from the two or more image processings based on the light / dark signal output from the switch, or the extraction result by the two or more image processings is selected. It is provided with feature extraction means for selecting one extraction result from among them.

According to a fourth aspect of the present invention, there is provided an image processing apparatus, comprising: light input means for inputting an image of a predetermined area including a face of a person to be detected; illuminating the person to be detected; An optical axis connecting the input means and an illumination light illuminating direction are arranged at least in the vicinity of the detection target person so as to be substantially coaxial in the vicinity of the detection target person, and the image signal of the detection target person output from the light input means is output. Brightness determining means for determining the brightness of the photographed image by using the light receiving means;
Bright image processing to extract dark areas on lower face
And extract bright areas on the face whose brightness is higher than the threshold
Image processing for at least two or more of
Thus, the feature on the face can be extracted, and one image processing is selected from the two or more image processings based on the brightness signal output by the brightness determination means, or
It is provided with a feature extracting means for selecting one extraction result from the extraction results obtained by one or more image processing.

According to a fifth aspect of the present invention, in the image processing apparatus, each of the feature extracting means extracts a pupil based on a retinal reflection image when a dark signal is input, and extracts a nostril when a bright signal is input. Is performed.

According to a sixth aspect of the present invention, there is provided an image processing apparatus comprising: a light input unit for inputting an image of a predetermined area including a face of a detection target person; illuminating the detection target person; An illumination unit arranged such that an optical axis connecting the input unit and an illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target; and a photographed image of the detection target output from the light input unit Brightness from threshold
Bright image processing to extract dark areas on lower face
And extract bright areas on the face whose brightness is higher than the threshold
Image processing for at least two or more of
Furthermore, it is possible to extract features on the face and to provide a feature extracting means for selecting a final extraction result according to the extraction result by the image processing.

According to a seventh aspect of the present invention, when the pupil is not detected by the dark image processing, the feature extracting means selects an extraction result by the bright image processing. It is.

According to an eighth aspect of the present invention, in the image processing apparatus according to the sixth aspect, the feature extracting means according to the sixth aspect is arranged so that the feature extracting means is either bright or dark.
When the size of the pupil extracted by the image processing is small, the extraction result by the bright image processing is selected.

According to a ninth aspect of the present invention, in each of the above-mentioned image processing apparatuses, there is provided an illumination photographing control means for controlling at least one of a photographing condition of the light input means and a lighting condition of the lighting means. The illumination photographing control unit is configured to extract the light / dark information from any of the light / dark detection unit, the light input unit, the switch, the light / darkness determination unit, or the feature extraction unit, and change the photographing condition or the lighting condition. .

According to a tenth aspect of the present invention, in the image processing apparatus, the illumination photographing control means turns off the illumination light when the brightness information is bright, and the feature extraction means performs the facial image processing on the photographed image by the light image processing. The above features are extracted, and when the light / dark information is dark, the illumination light is turned on, and the feature extracting means extracts features on the face from the captured image by dark image processing.

A person state determining apparatus according to an eleventh aspect of the present invention includes any one of the above image processing apparatuses and a person state determining means for determining a person state from an output of the image processing apparatus.

[0019]

The image processing apparatus according to the first aspect of the present invention measures the illuminance around the object to be detected by the light / dark detection means, and when the image is dark, uses the image processing using the retinal reflection image to characterize the object to be detected. It operates to extract the characteristics of the detection target using image processing that is not particular about the retinal reflection image when it is bright, so that even if the brightness around the detection target changes, the features on the face will be extracted. Can be caught.

According to the image processing apparatus of the present invention, the light input means having the function of outputting a light and dark signal indicating the light and dark of the input image obtains light and dark information around the detection target, and the light and dark information is obtained in the same manner as described above. Since the operation is performed so as to select the corresponding image processing, the feature on the face can be captured even when the brightness around the detection target changes.

According to a third aspect of the present invention, there is provided an image processing apparatus which obtains light / dark information around a detection target by a switch which outputs a signal corresponding to the light / dark when the detection target or another person switches according to the light / dark. As described above, since the operation is performed so as to select the image processing in accordance with the brightness, it is possible to capture the features on the face even if the brightness around the detection target person changes.

According to a fourth aspect of the present invention, there is provided the image processing apparatus, wherein the brightness determining means for determining the brightness of the photographed image using the image signal of the detected person output from the light input means, the light and dark information around the detected person. As described above, the image processing apparatus operates to select image processing according to light and dark, so that a feature on the face can be captured even if the brightness around the detection target person changes.

The image processing apparatus according to claim 5 performs pupil extraction based on a retinal reflection image when a dark signal is input, and performs nostril extraction when a bright signal is input. Feature extraction.

Further, the image processing apparatus according to the present invention operates so as to try the extraction result once and then select the image processing or the extraction result in accordance with the extraction result of the image processing in the feature extraction means. Goes up.

Further, the image processing apparatus of claim 7 operates to select the extraction result by the light image processing when the pupil cannot be detected by the dark image processing, and to select the result by the dark image processing. , The detection accuracy is good.

In the image processing apparatus according to the present invention, the size of the pupil is obtained by either bright or dark image processing, and the brightness information is obtained from the size of the pupil.

According to a ninth aspect of the present invention, there is provided an image processing apparatus, wherein brightness information is extracted from any of the brightness detection means, the light input means, the switch, the brightness determination means, and the feature extraction means, and the light input means is used for photographing. Change the conditions or the lighting conditions of the lighting means.

The image processing apparatus according to the tenth aspect turns off the illumination light when the light / dark information is bright, and turns on the illumination light when the light / dark information is dark.

[0029] In the eleventh aspect, the person state determining apparatus detects the open / closed state of the eyelids, the face direction, and the like based on the extraction result output from the image processing apparatus having the above-described configuration. Various person states are determined.

[0030]

【Example】

Embodiment 1 FIG. Hereinafter, a dozing detection device according to an embodiment of the present invention will be described with reference to the drawings. In FIG. 1, reference numeral 20 denotes an illuminance sensor that is installed around the detection target person 1 and detects illuminance. Reference numeral 41 denotes a feature extraction circuit 41 that can perform two image processes of a feature extraction process for darkness and a feature extraction process for lightness. It is.
Reference numerals 10 to 14, 30, 31, 50, and 51 denote constituent elements that operate in the same manner as the conventional example.

Next, the operation will be described. In FIG. 1, the illumination light of the infrared LED 13 irradiated on the optical path l1 reflects half of the light on the half mirror 12, and irradiates the face of the detection target 1 on the optical path l2. The image of the detection target 1 passes through the half mirror 12 along the optical path l3, and half of the light is
After reaching the camera 10, the CCD camera captures an image of the detection target person 1. At this time, the optical axes of the optical paths l2 and l3 are substantially coaxial or within a predetermined angle (2
Degree) (coaxial epi-illumination). In this embodiment, the angle is 0 degree, and the angle formed by the half mirror 12 and the optical axis l3 of the CCD camera 10 is 45 °. CCD
An image signal output from the camera 10 is once input to the captured image memory 30.

FIG. 2 is a process flow showing the operation of the feature extraction circuit 41 and the arousal level determination circuit 50. The operation up to the input of the face image in ST10 is the same as the conventional example shown in FIG. In ST50, a light / dark signal is input from the illuminance sensor 20. In ST51, for example, when the illuminance is equal to or less than a predetermined illuminance threshold, for example, illuminance of 500Lx or less, it is determined that the image is "dark". The arousal level is determined by the operation. If it is determined in ST51 that the image is "bright", in ST41, a binarization process is performed using a light luminance threshold th2 (<dark threshold th1). The image when it is bright looks like Figure 37,
The obtained binarized image is, as shown in FIG.
The nostrils 3 and the mouth area 9 are black areas, and the other areas are white areas. Next, an eye region is extracted in ST42 and ST43. In the extraction, first, in ST42, the luminance of the black pixel is projected in the Y-axis direction as shown in the right part of FIG. For example, if the obtained binarized image is expressed by G2 (x, y) (x = 0 to 255, y = 0 to 255), the projection on the Y axis is

[0033]

(Equation 1)

## EQU1 ## At this time, eyebrow 2, eye area 8,
The nostrils 3 and the mouth area 9 are the peaks P1, P2, P of the number of black pixels.
3, Appears as P4. In this peak pattern, the peak P1 corresponding to the eyebrows and the peak P2 corresponding to the eye region are large, and the peak P3 corresponding to the next nostril is a small peak. Therefore, in ST43, the second peak P2 is specified as the eye region. I do. Next, ST4
At 4, the eye is opened or closed. FIG. 4 is a binarized image when the eye is closed. As described above, the area of the mountain P2 in the eye region becomes smaller when the eyelid is closed and becomes larger when the eye is opened. Therefore, when the area of the mountain P2 is larger than a predetermined value, it is determined that the eye is open, and when smaller, the eye is determined to be closed. Hereinafter, the operation after ST20 is the same as that of the conventional example.

In the above embodiment, when the image is bright, the black pixels in the binarized image are projected in luminance in the Y-axis direction to extract the eye region, and the open / closed state of the eye is determined based on the mountain area of the eye region. The width of the mountain in the eye region may be used. In addition, even when the image is dark, the pupil is extracted by projecting the luminance of the white pixel in the binarized image in the Y-axis direction, and the opening / closing of the eye is determined based on the area of the pupil. Further, the image processing method when the image is bright is not limited to the above method, and the eye open / close determination may be performed by edge processing of the binarized image.

As described above, in the present embodiment, the illuminance around the detection target person is measured by the illuminance sensor, and when dark, the characteristics of the detection target person are extracted using image processing using a retinal reflection image. When it is bright, it operates to extract the characteristics of the detection target using image processing that is not particular about the retinal reflection image, so even if the brightness around the detection target changes, it always catches the features on the face and falls asleep Will be able to monitor.

Embodiment 2 FIG. FIG. 5 shows a person condition judging device according to another embodiment of the present invention. When the face is turned to the left or right, the apparent pupil interval or nostril interval becomes narrow according to the face angle. This is an embodiment of an inattentiveness detection device that detects inattentiveness and gives an alarm. In FIG. 5, reference numeral 41 denotes a feature extraction circuit, which operates so as to extract features for inattentiveness detection. 52 is an inattentive judgment circuit.

Next, the operation will be described. FIG. 6 is a flowchart showing the operation of the feature extraction circuit 41 and the inattentiveness judgment circuit 52. In ST10, a face image is input, and in ST50
, A light / dark signal is input from the illuminance sensor 20. ST
At 51, as in the first embodiment, for example, when the illuminance is equal to or less than a predetermined illuminance threshold, it is determined to be “dark”, and the process proceeds to ST11
After the binarization process, in ST120, white pixels are projected in the X-axis direction as shown in FIG. In ST130, the positions xq1 and xq2 of the two pupils are extracted. That is, the position of the pupil can be extracted by determining the positions of the peaks of the two peaks appearing in the projected image in the X-axis direction corresponding to the two pupils. In ST140, the apparent distance dpp between the pupils is obtained from the positions of the two pupils obtained in ST130, and the face direction is estimated from the obtained apparent distance. The face angle θf is obtained from cos (θf) = dpp / Dpp. Here, Dpp is the apparent pupil distance when the face is turned to the front, and dpp is the current apparent pupil distance.

On the other hand, in ST51, when the brightness is equal to or more than the predetermined illuminance threshold value, it is determined that the image is "bright". In this case, the angle of the face is estimated from the interval of the nostrils. That is, ST
At 41, as in the first embodiment, the brightness threshold value th for light
2 (<darkness threshold th1), binarization processing is performed, and S
At T420, luminance projection of the black pixel in the Y-axis direction is performed. FIG.
When the luminance is projected on the Y axis as shown in FIG. 7, the nostrils 3 appear as small black peaks P3 below the large peaks P1 and P2. In ST420, the center yp3 of the mountain P3 is further obtained, and a window for nostril extraction is set. The window has a pixel width corresponding to, for example, 2 cm above and below the center of yp3 and a width of 0 to 255. This window looks like the upper part of FIG. Thereafter, luminance projection is performed in the X-axis direction. The luminance projection of the black pixel in the X-axis direction is

[0040]

(Equation 2)

## EQU1 ## Here, y0 is the number of pixels corresponding to 1 cm above and below. When the luminance is projected in the X-axis direction, FIG.
Since the peaks Q1 and Q2 corresponding to the two nostrils appear as shown below, in ST45, the nostril centers xq1,
By obtaining xq2, the position nR (xq
1, yp3) and nl (xq2, yp3). In ST46, the apparent interval between the nostrils is determined from the positions of the two nostrils determined in ST45, and the orientation of the face is estimated from the obtained apparent interval. The face angle θf is cos (θf) = dnn / D
Required from nn. Here, Dnn is the apparent nostril interval when the face is turned to the front, and dnn is the current apparent nostril interval.

The information on the face orientation is input to the inattentiveness determination circuit 52, and the inattentiveness determination circuit 52 determines in step ST22 that the obtained face orientation is 20 degrees or more and this state is 5 degrees.
Judge whether it lasts for more than 2 seconds, and if it lasts for more than 5 seconds,
It is determined that the driver is in the inattentive state, and the alarm generation circuit 51 is determined in ST23.
To output an alarm output command. If it is shorter than 5 seconds, the process proceeds to ST10.

As described above, in the present embodiment, the illuminance around the detection target person is measured by the illuminance sensor, the pupil is extracted using the retinal reflection image when it is dark, and the image which is not particular about the retinal reflection image when it is bright. By extracting the nostrils using processing, features with good detection accuracy are extracted according to light and dark,
The detection accuracy of inattentive judgment is improved.

Embodiment 3 FIG. FIG. 9 is a configuration diagram showing a dozing detection device according to Embodiment 3 of the present invention. In FIG.
15 is CC with AGC (auto gain control)
This is a D camera, and has a function of outputting an AGC signal, which is a light / dark signal indicating light / dark of an input image, to the outside. That is, when the illuminance is high, the CCD camera has a function of reducing the gain of the electric signal amplification after the photoelectric conversion, and when the illuminance is high, the CCD camera has a function of increasing the gain of the electric signal amplification after the photoelectric conversion. This gain information is output to the outside. The operation is the same as that of the first embodiment. Instead of the light / dark signal from the illuminance sensor 20, a light / dark signal corresponding to the gain information is output from the CCD camera 15 to the feature extraction circuit 41. Therefore, in addition to the effects of the first embodiment, the illuminance sensor can be omitted, so that an inexpensive device can be provided.

Embodiment 4 FIG. FIG. 10 is a configuration diagram showing a dozing detection device according to Embodiment 4 of the present invention. In FIG. 10, reference numeral 21 denotes a light / dark switch that is switched by a person according to the light / dark state. That is, the detection target person or another person switches the switch 21 according to the brightness around the detection target person, and outputs a signal corresponding to the brightness. The operation is the same as that of the first embodiment. Instead of the light / dark signal from the illuminance sensor 20, a light / dark signal is output from the switch 21 to the feature extraction circuit 41. Therefore, similarly to the third embodiment, the detection accuracy is improved and the illuminance sensor can be omitted, so that an inexpensive device can be provided.

In the fourth embodiment, the light / dark switch 21 which switches according to the light / dark state is used. However, the above apparatus is applied to an automobile, and the light ON / OFF switch is also used as the light / dark switch 21 to change the light and dark. A corresponding signal may be taken out. FIG. 11 is a partial configuration diagram showing an automobile on which the dozing detection device according to the present embodiment is mounted. In FIG. 11, reference numeral 22 denotes a headlight switch.
A dark signal is output to the feature extraction circuit 41 when it is on, and a bright signal when it is off. With such a configuration, the light / dark signal is interlocked with the on / off switch of the light of the vehicle, and the light / dark signal is generated by an operation naturally performed by the driver, so that the operation is not troublesome.

Embodiment 5 FIG. FIG. 12 is a configuration diagram showing a dozing detection device according to Embodiment 5 of the present invention. In FIG. 12, reference numeral 42 denotes a light / dark judgment circuit which inputs data of the photographed image memory 30 and judges the light / dark level. That is, G1 (x,
If y) is the content of the image memory, the light / dark determination circuit 42
Is the value G1total for determining the brightness of the captured image

[0048]

(Equation 3)

If G1total is equal to or greater than a predetermined value, it is determined to be “bright”.
Is determined. The feature extraction circuit 41 receives the light / dark information of the light / dark determination circuit 42 instead of the light / dark signal from the illuminance sensor 20, and operates in the same manner as in the first embodiment.

As described above, by detecting the brightness using the image memory, a sensor for detecting the brightness can be omitted, and in addition to the effect of the first embodiment, an inexpensive device can be provided.

In the above embodiment, G1total uses the entire image data, but a part of the image may be used.

In the above embodiment, the brightness determination circuit 42 determines the brightness by inputting the data of the photographed image memory 30. However, as shown in FIG. 13, an image signal from the CCD camera 10 is input, and 1/30 of input image signal
The brightness is determined according to the integration amount per second, and the feature extraction circuit 41
Alternatively, a light / dark signal may be output.

Embodiment 6 FIG. Feature extraction circuit 4 of each of the above embodiments
In No. 1, the image processing for light and the image processing for dark are switched by inputting an external signal, but one extraction is performed by using the extraction result by each image processing without depending on the external signal. A result may be selected. FIG. 14 is a configuration diagram showing such a dozing detection device. In FIG. 14, reference numeral 43 denotes a dark feature extracting circuit, 44 denotes a bright feature extracting circuit, and 45 denotes a selecting circuit. The dark feature extraction circuit 43
The data of the captured image memory 30 is input, and the eyelid opening / closing degree and the Y coordinate of the pupil are extracted according to the flow of FIG. That is, in the case of a somewhat dark environment, only the pupil can be extracted as in the first embodiment. However, in a bright environment, the brightness at the tip of the nose and the brightness of the pupil are close to each other. As shown in FIG. 16, when the pupil and the tip of the nose become a white area, and when luminance projection is performed on the Y axis, a peak R1 due to the pupil and a peak R2 due to the tip of the nose appear, and it is impossible to determine which is the pupil. Further, in a bright environment, a large number of white areas appear, so that it is impossible to determine whether the eyelids are opened or closed. The darkness feature extraction circuit 43 outputs a value (Y coordinate yr1, opening / closing degree dr1) corresponding to R1 when the pupil can be extracted as a feature extraction result, and outputs undetectable information when the pupil cannot be specified. In FIG.
The face image is input at 10 and the threshold value th1 is set at 2 at ST11.
Perform value conversion processing. In step ST12, the white pixels are projected in the Y-axis direction. In step ST60, the number of peaks in the Y-axis projected image indicating the white region is detected.
Going to 5, it is determined that the pupil cannot be detected. In the case of one mountain, the pupil can be extracted, and the apparent size of the pupil can be calculated by obtaining the area of the projected image in ST13. In ST61, when the area is equal to or larger than the predetermined value, the process proceeds to ST63, and it is determined that the eye is open, and the Y coordinate yr1 of the pupil and the mountain width dr1 corresponding to the degree of opening and closing of the eyelids are obtained from the projected image. If the area is smaller than a predetermined value, ST6
In 2 it is determined that the eyes are closed. In ST64, ST62, ST6
3, and outputs the result of ST65 to the selection circuit.

On the other hand, the light feature extraction circuit 44 receives the data of the photographed image memory 30 and extracts the eyelid opening / closing degree and the Y coordinate of the pupil according to the flow of FIG. That is, in a bright environment, the peaks P1, P2, P3, and P4 corresponding to the eyebrows, the eye region, the nostrils, and the mouth can be separately extracted as in the first embodiment. However, the contrast becomes slightly dark and the contrast ratio on the face decreases. Then, when binarization is performed using the threshold value th2, a black region appears in addition to the eyebrows, the eye region, the nostrils, and the mouth as shown in FIG. 18, and the mountain P2 of the eye region can be identified even when the Y-axis luminance projection is performed. Disappears.
The light feature extraction circuit 44 outputs, as a feature extraction result, the eyelid opening / closing degree corresponding to P2 when it can be detected, and outputs undetectable information when it cannot be detected. In FIG. 17, ST1
At 0, a face image is input, and at ST41, binarization processing is performed at a threshold th2. In ST42, black pixels are projected in the Y-axis direction, and in ST70, a peak of the Y-axis projected image indicating a black area is detected.
When the mountain is unclear, that is, when the number of black pixels in the projected image is 0 or less, for example, four or less, the eye region is determined to be undetectable in ST75. If the mountains are clear, that is, if there are four areas, the second mountain P2 is specified as the eye area in ST43. In ST71, it is determined whether or not the area of the mountain P2 of the eye region is larger than a predetermined value.
If it is larger, the eye is determined to be open in ST73, and the Y coordinate of the eye area and the width of the mountain corresponding to the degree of opening and closing of the eyelids are obtained from the projected image. If it is smaller, it is determined in ST72 that the eye is closed.
In ST74, the results of ST72, ST73, and ST75 are output to the selection circuit.

The selection circuit 45 selects one of the output results of the dark feature extraction circuit 43 and the light feature extraction circuit 44 in accordance with the flow of FIG. In the present embodiment, light is selected unless the feature extraction result for light is undetectable, and the detection result for dark is selected if the feature extraction result for light is undetectable. That is, the extraction result of the dark feature extraction circuit 43 is input in ST80, and the extraction result of the light feature extraction circuit 44 is input in ST81. If the light feature extraction result can be detected in ST82, the light feature extraction result is selected in ST84,
If the bright feature extraction result cannot be detected, the dark feature extraction result is selected in ST83. In ST85, the selected extraction result is output to the alertness determination circuit 50, and the alertness determination circuit 5
0, and the alarm generation circuit 51 determines a dozing state according to the flow shown in FIG.
Generate an alarm. In FIG. 20, the position of the pupil (or eye region) is input from the feature extraction circuit 41 in ST100,
In ST101, this position is stored. In ST102, data of the above position for 5 minutes is obtained, and in ST103, a standard pupil (eye region) height is calculated by calculating an average value of the data for 5 minutes. In ST104, the position of the pupil (or eye region) is continuously input from the feature extraction circuit 41,
In ST105, the average pupil (eye region) height obtained from the data for 2 minutes is calculated. In ST106, the average pupil (eye region) height is compared with the standard pupil (eye region) height, and if the average pupil (eye region) height is lower than the standard pupil (eye region) height by, for example, 5 cm or more, the state is a dozing state. Judgment is made and a buzzer (alarm) is sounded in ST107.

As described above, in the present embodiment, the feature extraction is performed by the image processing for light and dark and the result of successful feature extraction is used irrespective of the light / dark state. Has the effect of rising.

In the above embodiment, the Y-coordinate of the pupil or the eye area and the degree of eyelid opening and closing are also obtained in the open state, and the awakening degree determination circuit determines the drowsiness based on the position of the pupil or the eye area. As in the case of ~ 5, only the open / closed state may be output as the extraction result, and this may be used to determine dozing.

Further, in the sixth embodiment, bright is preferentially selected, but dark may be preferentially selected.

In the sixth embodiment, the dark feature extracting circuit 4
3. The bright feature extraction circuit 44 outputs three types of results, that is, open, closed, and undetectable based on the determination of detectable and undetectable, but the dark feature extractor (1) determines whether the first candidate pupil is open or closed. Result and position (2) In addition to (1), the second candidate pupil pupil (y in FIG. 16)
Opening / closing judgment result and position of the position of r2) and (3) undetectable.
For each of (1) and (2), open / close determination may be performed. Therefore, there are five types of detection results, and the selection circuit 45
Inputs a plurality of candidates.

FIG. 21 shows the operation of the selection circuit 45. FIG.
In step 1, the extraction result of the dark feature extraction circuit 43 is input in ST80, and the extraction result of the light feature extraction circuit 44 is input in ST81. In ST86, if the dark feature extraction result cannot be detected, the light feature extraction result is selected in ST84. On the other hand, if the result of the feature extraction for darkness can be detected in ST86, it is determined whether or not only the first candidate has been detected in ST110. If only the first candidate has been detected, the process proceeds to ST11.
In step 1, a dark feature extraction result, that is, a first candidate is selected. If other than the first candidate has been detected, it is determined in ST82 whether or not the bright feature extraction result can be detected. If not, the first candidate for the dark feature extraction result and the second candidate in ST112 are determined in ST112. The candidate whose pupil position is closer to the previous extraction result is selected from the candidates. If it is determined in ST82 that detection is possible, a candidate whose pupil position is closer to the bright feature extraction result is selected from the first candidate and the second candidate of the dark feature extraction result in ST113. In ST85, the selected extraction result is output to arousal level determination circuit 50.

As described above, by extracting the second candidate and selecting a more reliable extraction result, the detection accuracy is improved, and the feature can be well captured even in the middle area of light and dark, and the undetectable area is reduced. There is.

Embodiment 7 FIG. In the sixth embodiment, the image processing for darkness and the image processing for lightness are performed, and one extraction result is selected by using the extraction result in each image processing. , May be selected according to the output of the illuminance sensor 20. FIG. 22 is a configuration diagram showing a dozing detection device according to Embodiment 7 of the present invention. In FIG. 22, the operations of the dark feature extracting circuit 43 and the bright feature extracting circuit 44 are the same as in the sixth embodiment. FIG. 23 shows the flow of the selection circuit 45. In FIG. 23, ST8
At 0, the extraction result of the dark feature extraction circuit 43 is input, and ST8 is entered.
In step 1, the extraction result of the bright feature extraction circuit 44 is input. ST
If the bright feature extraction result can be detected in 82, ST86
It is determined whether or not the result of feature extraction for darkness can be detected.
If detection is possible, a light / dark signal is input from the illuminance sensor 20 in ST87, and it is determined in ST88 whether the light / dark signal from the illuminance sensor 20 is a bright signal. If it is a bright signal, the extraction result of the bright feature extraction circuit 44 is selected in ST84. If it is a dark signal, the extraction result of the dark feature extraction circuit 43 is selected in ST83. On the other hand, if the light feature extraction result cannot be detected in ST82, the dark feature extraction circuit 4 in ST83.
3. Select the extraction result. In ST85, the extraction result selected in ST83 or ST84 is output to the arousal level determination circuit 50.

In this way, in addition to the effects of the first embodiment, when features can be extracted by both image processing for light and dark, a more accurate one is selected according to the illuminance. The detection accuracy increases in an intermediate environment.

Embodiment 8 FIG. In the sixth embodiment, the feature extraction circuit 41 performs the image processing for darkness and the image processing for lightness, and selects one extraction result by using the extraction result of each image processing. Depending on the extraction result, the image processing of either light or dark may be selected to obtain the final extraction result. FIG. 24 is a configuration diagram showing a dozing detection device according to Embodiment 8 of the present invention, and FIG. 25 is a flow of the feature extraction circuit 41 and the arousal level determination circuit 50. In FIG. 25, first, a feature for darkness is extracted in ST90 to ST93. In ST94, it is determined whether or not the pupil has been detected by the feature extraction for darkness.
In ST14, a feature for darkness is extracted, and this extraction result is used as a final extraction result, and in ST20, dozing detection is performed as in the above embodiments. That is, it is determined whether or not the eye-close information has been continuously input for 3 seconds or more. If the eye-closed state has been input for 3 seconds or more, it is determined that the user is dozing, and an alarm output command is output to the alarm generation circuit 51 in ST21. I do. Eye closed 3
If it is less than seconds, the process proceeds to ST10. On the other hand, if the pupil cannot be extracted in ST94, light feature extraction is performed in ST40 to ST44, and this extraction result is used as a final extraction result to detect dozing in ST47. That is, it is determined whether or not the eye closing information has been input for three seconds or more continuously.
If the eye is in the closed state for more than a second, it is determined that the user is in a dozing state, and an alarm output command is output to the alarm generating circuit 51 in ST48. If the closed state is less than 3 seconds, the process proceeds to ST40.

As described above, in this embodiment, it is determined whether or not feature extraction is possible by dark image processing, and the subsequent image processing method is selected, so that an inexpensive device can be used without using an illuminance sensor or the like. This has the same effect as the first embodiment. Further, since the image processing for darkness is prioritized, the detection accuracy is good.

In the eighth embodiment, whether or not the pupil can be detected by the dark feature extraction is determined by only one feature extraction process. However, the eye may be accidentally closed only once. Therefore, for example, a feature extraction process for darkness may be performed for about two seconds to determine whether darkness can be used.

In the eighth embodiment, after it is determined whether or not the pupil can be detected by the feature extraction for darkness, the image processing is fixed to either bright or dark. For example, a feature extraction process for darkness is performed once every 10 minutes,
It may be determined again whether or not darkness can be used.

In the eighth embodiment, it is determined whether or not the feature can be extracted by the dark image processing. However, the following image processing method is selected depending on whether or not the feature can be extracted by the bright image processing. Alternatively, bright image processing may be prioritized.

Embodiment 9 FIG. FIG. 26 is a flowchart of the feature extraction circuit 41 and the arousal level determination circuit 50 according to the dozing detection device according to the ninth embodiment of the present invention. The configuration of the dozing detection device is the same as that of FIG. Since the pupil is small when it is bright and large when it is dark, in the present embodiment, when the size of the pupil extracted by image processing corresponding to either light or dark is small, image processing for light and light for dark When an extraction result by image processing is selected and the size of the extracted pupil is large, a dark image processing and an extraction result by dark image processing are selected. In FIG. 26, first, the pupil is extracted by image processing for darkness in ST90 to ST93, and the diameter of the pupil is obtained by detecting the width in the projection image in the Y-axis direction in ST95.
If the pupil diameter is, for example, 4 mm or more, ST10 to ST
In step 14, image processing for light is executed in ST40 to ST44 when dark is determined. The following operations are the same as in the eighth embodiment.

As described above, when the image processing for light or dark is selected, the selection is made according to the size of the pupil, so that the accuracy of feature extraction is improved and an inexpensive apparatus without using an illuminance sensor or the like is used. Thus, there is an effect similar to that of the first embodiment.

In the above embodiment, the image processing method is selected according to the size of the pupil.
Similar effects can be obtained by selecting the extraction results of the bright feature extraction circuit and the dark feature extraction circuit based on the size of the pupil obtained by one of the circuits.

Embodiment 10 FIG. FIG. 27 shows Embodiment 1 of the present invention.
It is a block diagram which shows the dozing detection apparatus by 0. In FIG. 27, reference numeral 16 denotes a shutter speed setting circuit which receives light / dark information from the illuminance sensor 20 and reduces the shutter speed of the CCD camera 10 when the illuminance is below a certain level. If the illuminance is higher than a certain value, the shutter speed of the CCD camera 10 is increased. Other operations are the same as in the first embodiment.

As described above, in the present embodiment, C
Since the shutter speed of the CD camera is changed, in addition to the effects of the first embodiment, the quality of the input image is improved, and the feature extraction accuracy is improved.

Embodiment 11 FIG. FIG. 28 shows Embodiment 1 of the present invention.
1 is a configuration diagram illustrating a dozing detection device according to a first embodiment; In FIG. 28, reference numeral 17 denotes a filter exchange circuit for switching between an optical filter mounted on the CCD camera 10 and the visible light cut filter 11 or the infrared cut filter 18. The filter exchange circuit 17 receives the brightness information from the illuminance sensor 20 and selects the visible light cut filter 11 when the illuminance is below a certain level, and selects the infrared cut filter 18 when the illuminance is above a certain level. Other operations are the same as in the first embodiment.

As described above, in this embodiment, when the image is bright, the image is photographed with visible light, so that an image with good contrast can be obtained.
Light detection accuracy is increased.

In the eleventh embodiment, the optical filters are changed according to the brightness. However, a coaxial epi-illumination imaging system having a visible light cut filter and a non-coaxial imaging system having a half mirror without an infrared cut filter are provided. It is also possible to switch between shooting in a coaxial epi-illumination shooting system when dark and shooting in a non-coaxial shooting system when bright. In this way, the filter replacement circuit is omitted, and the image is taken without passing through the half mirror in the bright state, so that the quality of the image is improved.

In Embodiments 10 and 11, the first embodiment is used.
Although the photographing conditions of the light input means are controlled based on the brightness information of the illuminance sensor 20 shown in FIG. 7, the light input means 15, the switch 21, the brightness determination means 42, or the feature extraction means 41 shown in the second to ninth embodiments. The photographing condition may be changed by extracting light / dark information from any of the above.

Embodiment 12 FIG. FIG. 29 shows Embodiment 1 of the present invention.
FIG. 2 is a configuration diagram illustrating a dozing detection device according to a second embodiment. In FIG. 29, the LED drive circuit 14 receives light / dark information from the illuminance sensor 20 and, when the illuminance is below a certain level, the infrared LED 13
Is turned on, and the infrared LED 13 is turned off when the illuminance is equal to or higher than a certain value. Other operations are the same as in the first embodiment.

As described above, in the present embodiment, since the infrared light is illuminated only when the image is dark, the face image can be reliably obtained without bothering the person to be detected. In addition, when there is no light, no illumination is performed, so that power consumption can be reduced.

In the twelfth embodiment, the on / off control of the optical LED is controlled in accordance with the brightness. However, a mechanism for moving the half mirror is provided. The subject may be directly photographed. In this way, in the bright state, the image is taken without passing through the half mirror, so that the quality of the image is improved.

In the twelfth embodiment, the illumination condition of the illumination means is controlled based on the brightness information of the illuminance sensor 20 shown in the first embodiment.
5, the lighting conditions may be changed by extracting brightness information from any of the switch 21, the brightness determination unit 42, or the feature extraction unit 41.

In each of the above embodiments, the camera is CC
I don't care about D. For example, a CID camera may be used.

Although the photographing system has been described with one camera, a set of a plurality of coaxial epi-illumination and cameras may be used.

Further, the coaxial epi-illumination is shown only in the case of a half mirror, but the coaxial epi-illumination is not limited to such a structure as long as it is co-axial epi-illumination. For example, a structure in which a light emitting element is provided at the center of the lens of the camera and the driver is directly illuminated may be employed.

[0085]

As described above, according to the first aspect of the present invention, the light input means for inputting an image of a predetermined area including the face of the detection target, the illumination of the detection target, and the detection of the detection target Lighting means arranged so that an optical axis connecting the person and the light input means and the illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target, and light / dark detection for detecting light / dark around the detection target Means and a threshold value from a photographed image of the detection target outputted from the light input means .
Image processing for light to extract dark areas on the lower face,
Extract bright areas on the face whose brightness is higher than the threshold
At least two or more image processing of dark image processing
The feature on the face can be extracted, and one image processing is selected from two or more image processings based on the light / dark signal output by the light / dark detection means, or the extraction result by the two or more image processings is selected. Since a feature extraction unit for selecting one extraction result from the inside is provided, the feature on the face of the detection target person can be detected in a relatively short time under both bright and dark conditions.

According to a second aspect of the present invention, there is provided a light input means for inputting an image of a predetermined area including a face of a detection target person and outputting a light / dark signal indicating the brightness of the input image. Illuminating, an optical axis connecting the detection target person and the light input means, and an illuminating means arranged such that an illumination light illumination direction is substantially coaxial at least in the vicinity of the detection target person, and from the light input means. On the face whose brightness is lower than the threshold from the captured image of the detection target person output
Bright image processing to extract dark areas and brightness threshold
Dark image processing to extract bright areas on higher faces
At least two or more image processing
Together it can be extracted, and extracting selects one of the image processing from among the two or more image processing by the dark signal output from said optical input means, or from the extracted result of the two or more image processing of one Since the feature extracting means for selecting the result is provided, the feature on the face of the detection target person can be detected in a relatively short time under both bright and dark conditions as described above.

According to the third aspect, the light input means for inputting an image of a predetermined area including the face of the detection target person, illuminates the detection target person, and connects the detection target person and the light input means. Lighting means arranged so that the optical axis to be connected and the illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target person, and the detection target person or another person corresponds to light and dark by switching corresponding to light and dark. And a switch for outputting a signal having a luminance lower than a threshold from a photographed image of the detection target person output from the light input means .
Bright image processing to extract dark areas and brightness threshold
Dark image processing to extract bright areas on higher faces
At least two or more image processing
Together can be extracted, and selects one of the image processing from among the two or more image processing by brightness signals the switch outputs, or one extraction result from the extracted result of the two or more image processing , The feature on the face of the detection target person can be detected in a relatively short time under both bright and dark conditions as described above.

According to the fourth aspect, the light input means for inputting an image of a predetermined area including the face of the detection target person, illuminating the detection target person, and connecting the detection target person and the light input means An illuminating means arranged so that an optical axis to be connected and an illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target person, and a captured image using an image signal of the detection target person output from the light input means. Darkness determining means for determining the brightness of the subject, and darkness on the face whose luminance is lower than a threshold value from a photographed image of the detection target outputted from the light input means.
Image processing for extracting bright areas and brightness
Image processing for extracting bright areas on
A feature on the face can be extracted by at least two or more image processes, and one image process among the two or more image processes can be performed by a light / dark signal output by the light / darkness determination means. Since there is provided a feature extracting means for selecting or selecting one extracted result from the extracted results obtained by the two or more image processes, the feature on the face of the detection target person can be obtained under both bright and dark conditions as described above. Can be detected in a relatively short time.

According to the fifth aspect, each feature extracting means performs pupil extraction based on a retinal reflection image when a dark signal is input, and performs nostril extraction when a bright signal is input. Therefore, various features can be extracted.

According to the sixth aspect, the light input means for inputting an image of a predetermined area including the face of the detection target person, illuminating the detection target person, and connecting the detection target person and the light input means An illumination unit arranged so that an optical axis to be connected and an illumination light illumination direction are substantially coaxial at least in the vicinity of the detection target person, and a luminance is determined from a captured image of the detection target person output from the light input unit. Dark on face lower than threshold
Image processing for extracting bright areas and brightness
Image processing for extracting bright areas on
Since at least two or more image processes can extract features on the face, and feature extraction means for selecting a final extraction result according to the extraction result by the image processing, Like in the bright and dark conditions, the features on the face of the person to be detected can be detected in a relatively short time, and the image is once subjected to image processing to derive the extraction result. Output, the detection accuracy is improved.

According to the seventh aspect, when the pupil cannot be detected by the dark image processing, an operation is performed to select the extraction result by the bright image processing, so that the detection accuracy is further improved.

According to the eighth aspect, the size of the pupil is reduced.
Since the brightness and darkness information is obtained from the size of the pupil by image processing for either light or dark , similar to the above, the features on the face of the detection target person can be relatively compared under both bright and dark conditions. Detection can be performed accurately in a short time.

According to the ninth aspect, the light / dark detecting means includes:
Since the brightness information is extracted from any of the light input means, the switch, the brightness determination means, and the feature extraction means, and the photographing condition of the light input means or the lighting condition of the lighting means is changed, the detection accuracy is further improved. There is an effect of going up.

According to the tenth aspect, when the light / dark information is bright, the illumination light is turned off, and when the light / dark information is dark, the illumination light is turned on. There is no.

According to the eleventh aspect, the state of the person is determined by using the extraction result output from the image processing apparatus of each of the above configurations, so that the state of the person can be detected regardless of brightness. The effect is obtained.

[Brief description of the drawings]

FIG. 1 is a configuration diagram illustrating a dozing detection device according to a first embodiment of the present invention.

FIG. 2 is a flowchart illustrating operations of a feature extraction circuit and an arousal level determination circuit according to the first embodiment of the present invention.

FIG. 3 is an explanatory diagram illustrating an operation of the dozing detection device according to the first embodiment of the present invention.

FIG. 4 is an explanatory diagram illustrating an operation of the dozing detection device according to the first embodiment of the present invention.

FIG. 5 is a configuration diagram illustrating an inattentiveness detection device according to a second embodiment of the present invention.

FIG. 6 is a flowchart illustrating operations of a feature extraction circuit and an inattentiveness judgment circuit according to the second embodiment of the present invention.

FIG. 7 is an explanatory diagram illustrating the operation of the inattentiveness detection device according to the second embodiment of the present invention.

FIG. 8 is an explanatory diagram for explaining the operation of the inattentiveness detecting device according to Embodiment 2 of the present invention;

FIG. 9 is a configuration diagram illustrating a dozing detection device according to a third embodiment of the present invention.

FIG. 10 is a configuration diagram illustrating a dozing detection device according to a fourth embodiment of the present invention.

FIG. 11 is a partial configuration diagram showing an automobile on which a dozing detection device according to Embodiment 4 of the present invention is mounted.

FIG. 12 is a configuration diagram illustrating a dozing detection device according to a fifth embodiment of the present invention.

FIG. 13 is a configuration diagram showing another dozing detection device according to Embodiment 5 of the present invention.

FIG. 14 is a configuration diagram illustrating a dozing detection device according to a sixth embodiment of the present invention.

FIG. 15 is a flowchart showing the operation of a dark feature extraction circuit according to Embodiment 6 of the present invention;

FIG. 16 is an explanatory diagram illustrating an operation of a dozing detection device according to a sixth embodiment of the present invention.

FIG. 17 is a flowchart showing the operation of the lighting feature extraction circuit according to Embodiment 6 of the present invention.

FIG. 18 is an explanatory diagram illustrating an operation of a dozing detection device according to a sixth embodiment of the present invention.

FIG. 19 is a flowchart showing the operation of the selection circuit according to Embodiment 6 of the present invention.

FIG. 20 is a flowchart showing an operation of a waking degree determination circuit according to Embodiment 6 of the present invention.

FIG. 21 is a flowchart showing an operation of another selection circuit according to Embodiment 6 of the present invention.

FIG. 22 is a configuration diagram illustrating a dozing detection device according to a seventh embodiment of the present invention.

FIG. 23 is a flowchart showing the operation of the selection circuit according to Embodiment 7 of the present invention.

FIG. 24 is a configuration diagram illustrating a dozing detection device according to an eighth embodiment of the present invention.

FIG. 25 is a flowchart showing the operations of a feature extraction circuit and an arousal level determination circuit according to Embodiment 8 of the present invention.

FIG. 26 is a flowchart showing the operation of a feature extraction circuit and an arousal level determination circuit according to Embodiment 9 of the present invention.

FIG. 27 is a configuration diagram illustrating a dozing detection device according to a tenth embodiment of the present invention.

FIG. 28 is a configuration diagram showing a dozing detection device according to Embodiment 11 of the present invention.

FIG. 29 is a configuration diagram showing a dozing detection device according to Embodiment 12 of the present invention.

FIG. 30 is a configuration diagram showing a conventional eye position detection device.

FIG. 31 is a configuration diagram showing a conventional dozing detection device.

FIG. 32 is an explanatory diagram illustrating an operation of a conventional dozing detection device.

FIG. 33 is an explanatory diagram illustrating an operation of a conventional dozing detection device.

FIG. 34 is a flowchart showing the operation of the conventional dozing detection device.

FIG. 35 is an explanatory diagram illustrating an operation of a conventional dozing detection device.

FIG. 36 is an explanatory diagram illustrating an operation of a conventional dozing detection device.

FIG. 37 is an explanatory diagram illustrating an operation of a conventional dozing detection device.

[Explanation of symbols]

1 detection target, 3 nostrils, 6 pupils, 8 eye areas, 1
0 CCD camera, 12 half mirror, 13 infrared L
ED, 15 CCD camera with AGC, 16 shutter speed setting circuit, 17 filter exchange circuit, 18 infrared cut filter, 20 illuminance sensor, 21 light / dark switch, 22 headlight switch, 30 photographed image input memory, 31 timing generation circuit, 41 Extraction circuit, 42 light / dark judgment circuit, 43 dark feature extraction circuit,
44 Lighting feature extraction circuit, 45 selection circuit, 50 alertness determination circuit, 51 alarm generation circuit, 52 inattentiveness determination circuit.

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Toshihide Satake 8-1-1 Tsukaguchi Honcho, Amagasaki City Mitsubishi Electric Corporation, Industrial Systems Research Laboratory (72) Inventor Kenji Ogawa 840 Chiyoda-cho, Himeji-shi Mitsubishi Electric Corporation Inside Himeji Works (72) Inventor Hiroyoshi Suzuki 840 Chiyoda-cho, Himeji City Mitsubishi Electric Corporation Himeji Works (56) References JP-A-3-42337 (JP, A) JP-A-3-194661 (JP, A JP-A-4-68500 (JP, A) JP-A-4-174309 (JP, A) JP-A-6-32154 (JP, A) JP-A-6-262958 (JP, A) 32907 (JP, A) JP-A-7-296299 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G06T 1/00 B60K 28/06 G08B 21/00

Claims (11)

(57) [Claims] 1. An optical input means for inputting an image of a predetermined area including a face of a detection target person, an optical axis for illuminating the detection target person and connecting the detection target person and the light input means, and illumination light. An illuminating means arranged so that an illuminating direction is substantially coaxial at least in the vicinity of the detection target; a light / dark detection means for detecting light and dark around the detection target; and the detection target output from the light input means Brightness from the photographed image
For dark areas on the face where
Image processing and bright areas on the face whose brightness is higher than the threshold
At least two or more images of dark image processing to extract regions
By the image processing, the feature on the face can be extracted, and the above-mentioned 2
An image processing apparatus comprising a feature extraction unit for selecting one image processing from one or more image processing or selecting one extraction result from the extraction results of the two or more image processing. 2. An optical input means for inputting an image of a predetermined area including a face of a detection target person and outputting a light / dark signal indicating the brightness of the input image, illuminating the detection target person, and detecting the detection target person An illumination axis arranged so that an optical axis connecting the light input means and the illumination light illumination direction is substantially coaxial at least in the vicinity of the detection target; and the detection target output from the light input means. bright from the photographed image
Light for extracting dark areas on the face whose degree is lower than the threshold
Image processing and brightness on the face whose brightness is higher than the threshold
At least two or more of dark image processing for extracting a region
By image processing, features on the face can be extracted, and one image processing is selected from the two or more image processings by the light / dark signal output by the light input means, or the two or more images are selected. 1 out of the extraction result by processing
An image processing apparatus comprising a feature extraction unit for selecting one of the extraction results. 3. An optical input means for inputting an image of a predetermined area including a face of a detection target person, an optical axis for illuminating the detection target person and connecting the detection target person and the light input means, and illumination light. A lighting unit arranged so that the illumination direction is substantially coaxial at least in the vicinity of the detection target person, a switch that outputs a signal corresponding to light and dark by switching the detection target person or another person according to light and dark, And the brightness of the captured image of the detection target person output from the light input means.
Light for extracting dark areas on the face whose degree is lower than the threshold
Image processing and brightness on the face whose brightness is higher than the threshold
At least two or more of dark image processing for extracting a region
By image processing, features on the face can be extracted, and one image processing is selected from the two or more image processings by the light / dark signal output from the switch, or the extraction is performed by the two or more image processings An image processing apparatus comprising a feature extraction unit for selecting one extraction result from the results. 4. An optical input means for inputting an image of a predetermined area including a face of a detection target person, an optical axis for illuminating the detection target person and connecting the detection target person and the light input means, and illumination light. An illuminating means arranged so that an illumination direction is substantially coaxial at least in the vicinity of the detection target person, and a brightness level for determining the brightness of a captured image using an image signal of the detection target person output from the light input means. Judging means, and brightness from a photographed image of the detection target outputted from the light input means.
For dark areas on the face where
Image processing and bright areas on the face whose brightness is higher than the threshold
At least two or more images of dark image processing to extract regions
By image processing, features on the face can be extracted, and one image processing is selected from the two or more image processings based on the light / dark signal output from the light / darkness determination means, or the two or more images are processed. 1 out of the extraction result by processing
An image processing apparatus comprising a feature extraction unit for selecting one of the extraction results. 5. The feature extracting means according to claim 1, wherein a pupil is extracted by a retinal reflection image when a dark signal is input, and a nostril is extracted when a bright signal is input. Image processing apparatus. 6. An optical input means for inputting an image of a predetermined area including a face of a detection target person, an optical axis for illuminating the detection target person and connecting the detection target person and the light input means, and illumination light. An illuminating means arranged so that an illumination direction is substantially coaxial at least in the vicinity of the detection target; and a luminance from a photographed image of the detection target output from the light input means.
For dark areas on the face where
Image processing and bright areas on the face whose brightness is higher than the threshold
At least two or more images of dark image processing to extract regions
An image processing apparatus capable of extracting a feature on a face by image processing, and including a feature extracting unit for selecting a final extraction result according to an extraction result by the image processing. 7. An image processing apparatus according to claim 6, wherein the feature extraction means selects an extraction result by light image processing when a pupil cannot be detected by dark image processing. 8. The feature extraction means according to claim 6, wherein
An image processing apparatus which selects an extraction result by light image processing when the size of a pupil extracted by image processing using either dark or dark is small. 9. The image processing apparatus according to claim 1, further comprising an illumination imaging control unit configured to control at least one of an imaging condition of the light input unit and an illumination condition of the illumination unit. An image processing apparatus wherein the control means extracts brightness information from any of the brightness detection means, the light input means, the switch, the brightness determination means, and the feature extraction means, and changes the photographing condition or the lighting condition. 10. The illumination photographing control means according to claim 9,
If the light / dark information is bright, the illumination light is turned off. The feature extraction means extracts the features on the face from the captured image by light image processing. If the light / dark information is dark, the illumination light is turned on, and the feature extraction is performed. The image processing apparatus includes means for extracting features on the face from the captured image by image processing for darkness. 11. A person state determination device comprising: the image processing device according to claim 1; and a person state determination unit that determines a person state from an output of the image processing device.