JP4151341B2 - Face condition detection device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a face state detection device that detects the face state of a driver of a vehicle.
[0002]
[Prior art]
As a conventional face state detection device, for example, a “side-view driving prevention device” (hereinafter referred to as a conventional example) described in Japanese Patent Application Laid-Open No. 7-69139 is known.
[0003]
In the conventional example, a face image captured by a camera is taken as an inspection image, the coordinates of the left and right eye images of the driver are obtained based on the inspection image, the distance between the left and right eyes is calculated based on the coordinates of the eye image, The reference eye distance is calculated by fetching the same eye distance data a predetermined number of times, and it is detected from the reference eye distance and the current eye distance that there is no face in front.
[0004]
At this time, when the subject who detects the orientation of the face is wearing spectacles, reflection may occur in the lens portion of the spectacles due to the influence of the light environment, and the eyes may not be detected from the face image. Further, when the subject wears sunglasses, the eyes cannot be detected at all. When the subject's eyes can no longer be detected in this way, instead of the eyes, for example, a face part such as the nose or mouth is detected, and the face orientation is detected based on the change in the position of the face part. It was.
[0005]
[Problems to be solved by the invention]
However, in the above-described conventional example, when the driver wears spectacles such as sunglasses, the surrounding image may be reflected on the lens portion of the spectacles, and the eye state may not be determined. In addition, when the driver wears glasses and detects the face orientation using the nose or mouth image, the contour of the nose or mouth cannot be detected accurately, and the position of the nose or mouth changes. In addition, since the change in the size of the nose or mouth in the face image cannot be detected accurately, there is a drawback that the face orientation cannot be detected with high accuracy.
[0006]
[Means for Solving the Problems]
  The present invention is based on the face image in a face state detection device that captures the face of the subject with a face image capturing unit and detects the orientation of the face of the subject based on the captured face image. The eyeglass detection means for detecting whether or not the subject is wearing glasses, and when the eyeglass detection means detects that the subject is wearing glasses, there is a reflection in the glasses. Reflection detection means for detecting whether or not,When the reflection is detected by the reflection detection means, the vehicle body structure detection means detects an image of the vehicle body structure included in the reflection image.When,Vehicle body structure position detection means for detecting the position of the image of the vehicle body structure detected by the vehicle body structure detection means on the lens of the glasses.When,Position on the lens of the image of the vehicle body structure detected by the vehicle body structure position detection meansAnd a first face orientation detecting means for detecting the orientation of the face of the subject.
[0007]
【The invention's effect】
According to the present invention, even when the person to be monitored wears spectacles that cause reflection, and even when the eye state cannot be directly detected, the state of the reflection image generated on the spectacles or the state of the spectacle frame is detected. Therefore, the orientation of the face of the person to be monitored can be detected, so that the face direction of the person to be monitored can be detected with high accuracy regardless of whether or not the person to be monitored is wearing glasses.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0009]
<First Embodiment>
[System block diagram]
FIG. 1 is a block diagram showing the configuration of the first embodiment according to the face state detection apparatus of the present invention. As shown in the figure, this face state detection apparatus includes a face image imaging means CL1, a glasses detection means CL2, a reflection detection means CL3, a vehicle body structure detection means CL4 (feature amount detection means), and a vehicle body structure. An object position detecting means CL5, a first face orientation detecting means CL6, and a third face orientation detecting means CL7 are provided.
[0010]
The face image imaging means CL1 images the face of the person to be monitored and outputs face image data obtained by the imaging.
[0011]
The glasses detection means CL2 detects whether or not the person to be monitored is wearing glasses based on the image captured by the face image imaging means CL1.
[0012]
The reflection detection means CL3 processes the face image data output from the face image imaging means CL1, and determines whether or not a scene inside and outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored. To detect.
[0013]
The vehicle body structure detection means CL4 processes the face image data output from the face image imaging means CL1, and determines the vehicle body structure from the inside and outside scenes reflected on the lens surface of the glasses worn by the person to be monitored. Detect objects.
[0014]
The vehicle body structure position detection means CL5 processes the face image data output from the face image imaging means CL1, detects the lens frame of the glasses worn by the person to be monitored, and is detected by the vehicle body structure detection means CL4. The position of the body structure reflected on the glasses on the glasses based on the relative positional relationship between the body frame reflected on the lens surface of the glasses worn by the person being monitored and the lens frame Is detected.
[0015]
The first face orientation detection means CL6 is based on the position of the body structure reflected on the glasses detected by the body structure position detection means CL5 on the glasses and the direction of the face of the person to be monitored. Is detected.
[0016]
The third face orientation detection means CL7 is a case where the glasses cannot be detected by the glasses detection means CL2, or on the lens surface of the glasses worn by the driver in the reflection detection means CL3. When the reflection of the image cannot be detected, the face image data output from the face image capturing means CL1 is processed to detect the position of the eyes, and the positions of the left and right eyes or the detected positions of the eyes are used as a reference. From the determined range, the face orientation of the driver (monitor target person) is detected based on the shape of the left or right eye obtained by image processing.
[0017]
This device can be used for monitoring automobiles, railway vehicles, ship drivers, plant operators, etc., but in all the embodiments shown below, examples are applied to the eyes of automobile drivers. I will explain.
[0018]
[Device layout]
FIG. 2 is a device layout diagram of the face state detection apparatus according to the present embodiment. The TV camera 1 as the face image capturing means CL1 is installed at a position where the driver can be imaged substantially in front on the vehicle instrument, and images the driver's face. In this embodiment, the input image of the TV camera 1 is composed of, for example, 640 pixels in the horizontal direction (X-axis direction) and 480 pixels in the vertical direction (Y-axis direction). And the input image imaged with the TV camera 1 is input as image data to the microcomputer 2 installed inside the vehicle body such as the back side of the instrument.
[0019]
The microcomputer 2 includes glasses detection means CL2, reflection detection means CL3, body structure detection means CL4, body structure position detection means CL5, first face orientation detection means CL6, and third A program relating to each processing of the face direction detection means CL7 is programmed. Next, the processing status of the system will be described.
[0020]
[System-wide processing]
FIG. 3 shows the overall processing flow of the system. First, when processing is started, in step S1 (hereinafter, “step S” is simply referred to as “S”), a face image of the person to be monitored is captured by the TV camera 1 and input to the microcomputer 2 as image data. To do.
[0021]
In S2, it is detected whether or not the person to be monitored is wearing glasses. In S3, based on the detection result of S2, if the person to be monitored is wearing glasses (YES in S3), the process proceeds to S4, and if the person to be monitored is not wearing glasses (NO in S3), The process proceeds to S9.
[0022]
In S4, the image data input to the microcomputer 2 is subjected to image processing to detect whether a scene inside or outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored.
[0023]
In S5, based on the detection result in S4, if a scene inside or outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored (YES in S5), the process proceeds to S6. If a scene inside or outside the vehicle is not reflected on the lens surface of the glasses worn by (No in S5), the process proceeds to S9.
[0024]
In S6, the image data input to the microcomputer 2 is subjected to image processing, thereby detecting the vehicle body structure from the inside and outside scenes reflected on the lens surface of the glasses worn by the person to be monitored.
[0025]
In S7, the image data input to the microcomputer 2 is subjected to image processing to detect the lens frame of the glasses worn by the monitor subject, and the monitor subject detected by the vehicle body structure detection means CL4 The position of the vehicle body structure reflected on the spectacles on the spectacles is detected based on the relative positional relationship between the body structure reflected on the lens surface of the spectacles and the lens frame.
[0026]
In S8, the direction of the face of the person to be monitored is detected based on the position of the body structure reflected on the glasses detected in S7 on the glasses, and the process ends.
[0027]
In S9, if the spectacles cannot be detected in S3, or if a scene inside or outside the vehicle is not detected on the lens surface of the spectacles worn by the person to be monitored in S5, input to the microcomputer 2 By detecting the position of the eyes by performing image processing on the captured image data, the face is determined from the left or right eye position or the left or right eye shape obtained by image processing from the range based on the detected eye position. Detect the direction of. That is, the third face orientation detection process is performed. Thereafter, the process ends.
[0028]
[Glasses detection processing]
As a method for detecting glasses, it is possible to operate the device that the person to be monitored is wearing glasses with a switch installed on the instrument panel, or whether the device automatically wears glasses. May be determined. As a method for automatic discrimination, for example, a method proposed in Japanese Patent No. 2546188 and Japanese Patent Laid-Open No. 9-216611 can be used.
[0029]
[Reflection detection processing]
The reflection of spectacles increases as the light transmittance of the spectacle lens is lower and the reflectivity of light on the spectacle lens surface is higher. As spectacles having low light transmittance of spectacle lenses, spectacles using colored lenses generally called sunglasses correspond to this. Generally speaking, the types and darkness of sunglasses vary widely depending on the glasses, but the darker the color, the more the reflection tends to become clearer.
[0030]
In addition, if the lens surface has the same material and color, the reflection tends to decrease as the coating for reducing reflection, which is generally referred to as an anti-reflection coating, is applied. Even with the same glasses, the intensity of the reflection changes depending on the light environment of the outside world, so even if the glasses use lenses that are colorless and transparent and have a non-reflective coating, the reflection should not occur. Is not limited.
[0031]
As shown in FIG. 5, if the person to be monitored is wearing glasses, the eyes on the image of the person to be monitored are located inside the frame of the glasses. However, as shown in FIG. 7, it is known that if the spectacles inside and outside the vehicle are clearly reflected in the glasses, the reflection hides the eyes of the person to be monitored who should be located inside the frame of the glasses. .
[0032]
Therefore, if it is known that the person to be monitored is wearing glasses, the eyes are detected from the entire face image. As a result, if the eyes cannot be detected, it can be determined that the image is reflected on the glasses. .
[0033]
Hereinafter, the reflection detection process S4 will be described with reference to the flowchart shown in FIG. First, in S41 of FIG. 4, a process for specifying the position of the eye candidate is executed. In S42, an eye determination process is performed.
[0034]
In S43, it is determined whether all the eye candidate points detected in the specific processing of the position of the eye candidate have been determined. If all eye candidate points have been determined (YES in S43), the process proceeds to S44. If all the eye candidate points have not been determined (NO in S43), the process returns to the eye determination process in S42.
[0035]
In S44, it is determined whether or not there is a candidate point that can be determined as an eye as a result of performing the eye determination process in S42. If there is a candidate point that can be determined to be an eye (YES in S44), it is determined that there is no reflection in S45, and if there is no candidate point that can be determined to be an eye (NO in S44), it is reflected in S45. If it is determined that there is, the process proceeds to S5.
[0036]
For details of the eye candidate position specifying process in S41 and the eye determining process in S42, the third face orientation detecting means described later, the eye candidate position specifying process in S91, Since this is the same as the eye determination process in S92, a description thereof is omitted here.
[0037]
[Vehicle structure detection processing]
Next, the vehicle structure detection process S6 (see FIG. 3) will be described with reference to FIGS. FIG. 6 shows a face image in which a scene inside and outside the vehicle is reflected on the spectacle lens. In FIGS. 6 to 8, the monitoring target person is looking at the front.
[0038]
From this face image, a rectangular area including glasses surrounded by a frame line in FIG. 6 is extracted. The rectangular area may be determined using the market value of the area where the eye is present on the image, or the rectangular area may be determined from the position of facial components other than the eyes such as the nose, mouth and ears.
[0039]
An image of this rectangular area is extracted as shown in FIG. 7. As a result of extracting the contour of the image of FIG. 7, contour data as shown in FIG. 8 can be generated.
[0040]
From the contour data of FIG. 8, a vehicle body structure such as an A-pillar is extracted from the area surrounded by the contour of the right side window, the contour of the front window, the room mirror, the contour of the right side window and the contour of the right edge of the front window. Can do.
[0041]
[Detection of position on spectacles on vehicle structure image]
Next, the position detection processing S7 on the glasses on the image of the vehicle structure will be described with reference to FIG.
[0042]
A spectacle frame is extracted from the contour data obtained in the vehicle structure detection process S6. It is detected in which position in the extracted spectacle frame the vehicle structure extracted in the vehicle structure detection process S6 exists.
[0043]
In FIG. 8, there is a right A pillar surrounded by a right side window outline, a front window right edge outline, a right side window outline and a front window right edge outline in the right frame of the glasses. In the left frame, there is an outline at the top center of the front window and a rearview mirror.
[0044]
[First face orientation detection processing]
The first face orientation detection process S8 will be described with reference to FIGS. FIG. 9 shows the situation when the person to be monitored turns to the right. FIG. 9A shows the original image, and FIG. 9B shows the contour data obtained by extracting the contour of the original image of FIG. ing.
[0045]
In FIG. 9 (b), there is an upper contour of the right side window in the right frame of the glasses, and the front frame of the right side window, the right edge of the front window, the upper right contour of the left frame of the glasses, There is a right A pillar surrounded by the contour of the front portion of the right side window and the contour of the right end portion of the front window.
[0046]
Comparing the positions of the vehicle components reflected when the person to be monitored shown in FIG. 8 is looking at the front, the outline of the right side window shows the glasses when the person to be monitored is looking at the front. When the person to be monitored turns to the right, the right frame of the right frame of the eyeglass moves to the upper left part of the right frame of the glasses, and the contour of the front part of the right side window is the left frame of the glasses. Has moved to.
[0047]
In addition, the contour line at the right end of the front window is reflected in the right frame of the glasses when the person to be monitored is looking in front, but moves to the left frame of the glasses when the person to be monitored turns to the right.
[0048]
As a result, the right A pillar moves to the left frame. In addition, the outline of the center upper part of the room mirror and the front window was shown on the left frame of the glasses when the person being monitored was looking in front, but when the person to be monitored turned to the right, Moves out of the left side of, and is no longer reflected on the spectacle lens.
[0049]
FIG. 10 shows the situation when the person to be monitored turns to the left. FIG. 10 (a) shows the original image, and FIG. 10 (b) shows the contour extracted from the original image of FIG. 10 (a). Data are shown.
[0050]
In FIG. 10 (b), the upper contour of the front window and the rearview mirror are present in the right frame of the glasses, and the left edge of the front window, the contour of the upper left corner, the contour of the left side window, and the left side of the left frame of the glasses. There is a left A pillar surrounded by the contour of the front portion of the window and the contour of the left end portion of the front window.
[0051]
Comparing the positions of the vehicle components reflected when the monitoring subject shown in FIG. 8 is looking at the front, the contour of the right side window shows the right side when the monitoring subject is looking at the front. Although it is at a relatively low position in the frame, when the person to be monitored turns to the left, it moves out of the lower right side of the right frame of the glasses and does not appear on the lens of the glasses.
[0052]
In addition, the contour of the right edge of the front window was also reflected in the right frame of the glasses when the person being monitored was looking in front. , It is no longer reflected on the lens of the glasses. In addition, the outline of the center upper part of the rearview mirror and the front window was shown on the left frame of the glasses when the person being monitored was looking in front, but when the person to be monitored turned to the left, Has moved to.
[0053]
Changed to the front window left upper edge, left edge outline, left side window, left side window front outline and front window left edge outline that was not reflected when the person being monitored was looking in front The enclosed left A-pillar is reflected in the left frame of the glasses.
[0054]
FIGS. 11 to 13 show original images of a situation where the person to be monitored faces rightward from the front. This image shows 24 frames ((1) to (24)) sampled at 33 milliseconds (msec) per frame.
[0055]
Looking at this continuous image, the contour of the right side window moves from the relatively low position on the right side of the right frame of the glasses to the upper left side as the person to be monitored faces from the front to the right side. It can be seen that after the 18th frame, the contour line of No. 1 starts to appear in the left frame of the glasses and then gradually moves to the left.
[0056]
In addition, the outline at the right end of the front window, which was reflected in the right frame of the glasses, gradually moves to the left and disappears from the right frame of the glasses at the seventh frame. On the other hand, the right edge of the front window is shown from the sixth frame in the left frame of the glasses.
[0057]
The right A-pillar gradually moves to the left on the right frame of the glasses until the 4th frame, and appears on the left and right frames from the 5th frame, and then gradually disappears from the right frame of the glasses. The left frame gradually expands the area to the left.
[0058]
After the 18th frame, the right A-pillar is not reflected in the right frame of the glasses, the right A-pillar is reflected only in the left frame of the glasses, and then gradually moves to the left in the left frame of the glasses.
[0059]
Further, it can be seen that the contour line at the center upper part of the rearview mirror and the front window gradually moves to the left side as the monitor turns to the right, and after 17th frame, the rearview mirror disappears outside the left side of the left frame of the glasses.
[0060]
As described above, it is possible to correlate the position of the vehicle structure in the spectacle frame reflected in the spectacle frame and the direction of the face, thereby detecting which direction the face of the person to be monitored is facing. be able to.
[0061]
[Third face orientation detection processing]
Next, the third face orientation detection process S9 (FIG. 3) will be described with reference to the flowchart of FIG.
[0062]
First, in S91, a process for specifying the position of the eye candidate is executed. In S92, an eye determination process is performed. In S93, it is determined whether all the eye candidate points detected in the specific processing of the position of the eye candidate in S91 have been determined. If all eye candidate points have been determined (YES in S93), the process moves to S94. If all the eye candidate points have not been determined (NO in S93), the process returns to the eye determination process in S92. In S94, face orientation determination processing is performed to determine the face orientation of the person to be monitored.
[0063]
[Specific processing of eye candidate position]
The flow of the eye candidate position specifying process S91 will be described with reference to the flowchart of FIG. 15 and FIGS.
[0064]
First, in S911 in FIG. 15, the entire image data captured in the face image capturing process S1 (see FIG. 3) and input to the microcomputer 2 is stored in the image memory as the entire image G. Next, in step S912, after one line ends in the vertical direction, the process proceeds to the process of the next adjacent line, and it is determined whether or not point extraction has been completed for all lines in the vertical direction.
[0065]
If it is determined in S912 that point extraction has not been performed on all lines (NO in S912), the process proceeds to S913.
[0066]
In S913, an arithmetic mean calculation of the density values of one line in the vertical direction (Y-axis direction) is performed. The purpose of this process is to eliminate small variations in density value changes during image data capture, and to capture global changes in density values.
[0067]
In S914, a differentiation operation is performed on the arithmetic mean value that is the calculation result of S913. In S915, point extraction is performed based on the differential value that is the calculation result of S914. After this process is completed for one line, the process is switched to the process for the next line in S916.
[0068]
If it is determined in the above-described processing of S912 that the point extraction of all lines has been completed, the process proceeds to S917, where the Y coordinate values of the extraction points of adjacent lines are compared, and the Y coordinate value is within a predetermined value. As continuous data, (1) group number of continuous data, (2) continuous start line number, (3) number of continuous data, (4) average value of vertical position of each extraction point constituting continuous data (5) The average value of the horizontal position of the continuous start line and the end line (representative horizontal position of the continuous data) is stored. Since the detection target here is the eye, it can be said that the feature amount is data that lasts for a relatively long time. Therefore, continuous data can be selected on the condition that it continues for a predetermined value or more in the horizontal direction.
[0069]
FIG. 16 shows the facial feature quantity selected in this way as continuous data G. FIG. Although the method for extracting the continuous data G has been described simply with the flow of the flowchart, the details of the processing state are also described in “JP 10-40361 A”, “JP 10-143669 A”, and the like. ing.
[0070]
The continuous data G is a so-called eye candidate, and the representative coordinate value C of the continuous data G is the position of the eye candidate point.
[0071]
Next, in S918 of FIG. 15, an existence area EA including each continuous data G is set based on the representative coordinate value C of each continuous data G as shown in FIG. This existence area EA is determined as follows.
[0072]
(How to determine the size of the existence area EA)
The size of the existence area EA is determined as shown in FIGS. FIG. 17 shows the size of the existence area EA, and FIGS. 18 and 19 show statistical data on the lengths of the horizontal Xa and the vertical Ya, in which the sizes of several eyes were examined. Here, the size of the existence area EA is preferably as small as possible in order to reduce noise (extract facial wrinkles, brightness and darkness) and not reduce the processing speed.
[0073]
The size used in the current process such as detection of snoozing is determined by examining the size of several people's eyes and adding a margin (for example, x1.5). As a method for statistically obtaining the eye sizes of several people, as shown in FIGS. 18 and 19, data on the vertical and horizontal dimensions of the eyes are collected, and a margin that covers 95% of the distribution is sufficient. A method of determining by looking at the above can be considered.
[0074]
Then, the dimensions covering 95%, that is, the horizontal dimension xa and the vertical dimension ya are determined by taking a margin (× 1.5) as shown in FIG. A method of estimating the eye width and height by image processing and adding a margin to the vertical and horizontal sizes is also conceivable.
[0075]
(How to determine the position of the existence area EA)
FIG. 20 shows a method for positioning the presence area EA of the right eye, for example. Based on the eye coordinate values (x1, y1), a reference point P for drawing the existence area EA is determined at the distance x2, y2, and the dimensions x3, y3 of the existence area EA determined in advance from the P point are drawn. Determine the position. x2 and y2 are 1/2 of x3 and y3, and have a length such that the existing area EA is in the center of the eye in advance. The existence area EA is set for all continuous data G found in the entire image.
[0076]
[Eye determination processing]
The eye determination process will be described with reference to the flowcharts shown in FIGS. 21 and 22 and FIGS. First, in S9201 of FIG. 21, the image data of the eye candidate point existence area EA is stored in the image memory as a minute image IG. The state of the whole image G and the stored minute image IG is shown in FIG.
[0077]
The entire image data captured by the face image capturing process S1 (see FIG. 3) and input to the microcomputer 2 is stored in the image memory as the entire image G.
[0078]
Next, in S9202, a binarization threshold is set based on the density information of the range AR with reference to the representative coordinate value IC of the minute image IG corresponding to the representative coordinate value C of the entire image G.
[0079]
This range AR is smaller than the existence area EA so that the binarization threshold can be set accurately.
[0080]
An example of a binarization threshold calculation method in each range AR will be described with reference to FIG. In the range AR, density values of several lines are read in the vertical direction. FIG. 24 shows that there are four lines in the vertical direction. In each line, the highest (bright) density value and the lowest (dark) density value are stored in memory, and when all lines are stored, the highest (bright) density value in each line is stored. Then, find the lowest density value (skin part) and the lowest density value (eye part) among the lowest (dark) density values of each line, and calculate the median value as the binarization threshold. And
[0081]
The range AR for the binarization threshold is set so that the black part of the eye and the white part of the skin around the eye enter, and is the minimum necessary to reduce the influence of the brightness variation of the image. The size is In addition, the binarization threshold is set to the median value of the lowest (dark) density value of the eye in the region and the lowest (dark) density value of the skin portion, so that the eye can This value is suitable for cutting out a portion.
[0082]
Furthermore, the reason why the lowest (dark) density value of the density value of the skin portion is used to determine the binarization threshold is as follows. As described above, even if the range AR from which the density value is read is made as small as possible in order to reduce the influence due to the brightness variation around the eyes, a portion where direct light is hitting a part of the range AR This is to prevent this portion from being used for determining the binarization threshold.
[0083]
In step S9203, the minute image IG is binarized using the binarization threshold value determined in this manner, and stored in the image memory as a binary image bG.
[0084]
By detecting a binarized candidate object using such a binarization threshold, the eye can be accurately captured and a determination using the candidate object's geometric shape can be made more accurately. The accuracy can be further improved.
[0085]
Next, the process proceeds to S9204, where the position bC of the binary image bG corresponding to the representative coordinate value C of the entire image G is set as the initial position.
[0086]
In S9205, it is determined whether or not the setting position is a black pixel. If the setting position is determined to be a black pixel (YES in S9205), the process proceeds to S9206 in FIG. 22 and the setting position must be determined to be a black pixel. If (NO in S9205), in S9213, the set position is shifted one pixel up, down, left, and right, and it is determined again whether or not the set position is a black pixel, and processing is performed until the set position becomes a black pixel. .
[0087]
In step S9206, a connected component that includes the black pixel is set as a candidate object. In S9207, the geometric shape of the candidate object is calculated, and the geometric shape of the eye template to be specified in S9208 is compared with the geometric shape of the candidate object.
[0088]
An example of a method for comparing the candidate object and the geometric shape of the eye template in S9208 will be described with reference to FIG.
[0089]
The binarized shape of the eye is as shown in FIG. 25 (a) if the light environment is good and stable, but the light environment deteriorates due to direct sunlight hitting the vehicle interior from one side. In such a case, the shape shown in FIGS. 25B and 25C may be obtained.
[0090]
The eye template has a condition (1) that the lateral width is 2/3 or more of the market value of the eye and has an upwardly convex curvature, and a concave shape condition on the left side of the black eye (2) ▼ and the concave shape condition (3) on the right side of the black eye are set in combination, and in order to allow the example of FIGS. 25 (b) and (c), (1) and (2) or ▲ It may be that satisfying the conditions of 1 ▼ and (3).
[0091]
In S9209, as a result of S9208, it is determined whether the candidate object and the geometric shape of the eye template match. If the geometric shape of the candidate object matches the eye template (YES in S9209), the candidate object is determined to be an eye in S9210. If the geometric shape of the candidate object and the eye template do not match (NO in S9209), it is determined in S9214 that the candidate object is not an eye.
[0092]
In S9211, the representative coordinate value C in the entire image G of the candidate object determined to be the eye is stored as the eye coordinate in this image frame.
[0093]
In S9212, the micro image IG of the representative candidate point determined to be the eye is stored in the image memory as the eye image MGi.
[0094]
[Face orientation determination processing]
Next, face orientation determination processing S94 (see FIG. 14) will be described with reference to the flowchart of FIG. First, in S941, a process for detecting the outer peripheral contour of the eye is executed. In step S942, a process for detecting the positions of the top and bottom of the eyes is performed.
[0095]
In S943, processing for detecting the peak position of the arcuate line is performed. In S944, face orientation calculation processing is performed to determine the face orientation based on the positions of the corners and the eyes detected in S942 and the peak positions of the arcuate lines detected in S943.
[0096]
[Detection process of outer periphery of eye]
The eye outer peripheral contour detection process S941 will be described with reference to FIGS. In this description, the right eye on the image (the actual left eye) is assumed.
[0097]
FIG. 27 shows a minute image centered on the eye position specified by the eye position specifying process. As the micro image here, it is normal to use the micro image MGi stored in the image memory in S9212 (FIG. 22) of the eye determination process, but the entire image G stored in the image memory is used depending on the situation. A minute image with a redefined size and position may be extracted from the image and used.
[0098]
The binary image in FIG. 28 is binarized so that pixels smaller than the binarization threshold are black (density value 0) and areas larger than the binarization threshold are white (density value 255). A binarized image can be obtained. The binarization threshold of the binarization process performed here may be the same as the binarization threshold used in the binarization process performed in the eye determination process.
[0099]
In FIG. 29, a black pixel having a pixel value of 0 is searched from the upper left of the obtained binarized image downward. When the search is completed up to the bottom pixel, the pixel row on the right is searched. When the black pixel first found in the pixel row and the black pixel found last are obtained for each pixel row, the outer peripheral contour of the eye can be obtained as shown in FIG.
[0100]
[Detection of eye head and eye corners]
Eye head / eye corner detection processing S942 will be described with reference to FIG. In the case of the right eye on the image, as shown in FIG. 31, the left end of the outer peripheral contour line is the eye head and the right end is the eye corner position.
[0101]
When specifying the position of the eye and the corner of the eye as the plane coordinates, the left and right ends of the lower line are used as the head position and the position of the eye corner as shown in FIG. There are a method of using the coordinates as the position and the position of the corner of the eye, and a method of using the coordinates as the center position of the left and right ends of the upper line and the lower line as the eye position and the position of the eye corner.
[0102]
[Detection of peak position of arc-shaped line]
The arc-shaped line peak position detection processing S943 will be described with reference to FIGS.
[0103]
When the upper eyelid is used as the arc-shaped line, as shown in FIG. 31, the highest position of the upper line in the outer peripheral contour line of the eye can be set as the peak position.
[0104]
Also, as shown in FIG. 32, when the face is not facing the front with respect to the camera, the upper eye of the image is tilted as shown in FIG. 33, and the highest position is not necessarily the peak in the height direction of the upper line. It is not a position. Assuming such a case, it is also possible to set the peak position where the distance between the upper line is the largest among the line segment connecting the eye position and the eye corner position and the outer peripheral outline of the eye.
[0105]
[Face orientation calculation processing]
The face orientation calculation process S944 will be described with reference to FIGS. As shown in FIG. 31, when the face is faced to the front, the peak position is almost at the center of the eye, and the distance L1 from the peak position to the eye position and the distance L2 from the peak position to the eye corner position are Almost equal. Next, when the face is on the right side on the image as shown in FIG. 32, the distance L1 from the peak position to the eye position increases as shown in FIG. The distance L2 becomes smaller.
[0106]
When the relationship between L1 and L2 is expressed by the parameter L2 / L1, L2 / L1 is 1 when there is a face in the front, and L2 / L1 gradually decreases as the face is directed to the right side of the image. When the face is facing straight to the camera (when the face is turned 90 degrees to the right), the peak of the arc is closer to the corner of the eye and the length of L2 is 0, so L2 / L1 is 0. .
[0107]
However, after the face direction actually exceeds about 45 degrees to the right, the right eye is hidden behind the face and cannot be seen by the camera, so both eyes are monitored in the original processing. If there are no abnormalities such as strabismus, the eyeballs move in the same way for both the left and right eyes. If one eye is not visible, the other eye is used for complementation.
[0108]
On the other hand, as the face turns to the left, L2 / L1 gradually increases, and when the face turns straight to the camera (when the face is turned 90 degrees to the left), the peak of the arc is on the eye side. Since the length of L1 is 0, L2 / L1 is infinite.
[0109]
A graph showing this relationship is shown in FIG. From this graph relationship, it is possible to obtain an angle θf in which the amount of face orientation from L2 / L1 is 0 degrees when the face is directed to the front and the right direction is positive.
[0110]
The face orientation detection processing illustrated here is an example, and besides this method, a method of detecting the face orientation from the distance between both eyes is known.
[0111]
  In this way, in the face state detection device according to the present embodiment, even if the driver (subject) of the vehicle is wearing glasses, the reflection on the spectacle lens is reflected.OnlyAccordingly, the orientation of the face of the subject can be detected, so that the orientation of the face can be detected with high accuracy regardless of whether the subject is wearing glasses.The
[0113]
  Furthermore, the detection accuracy can be further improved by adopting a configuration that detects the contour of the front window, the contour of the side window, the room mirror, or the A-pillar that exists in front of the subject as the body structure. CanThe
[0119]
<Second Embodiment>
[System block diagram]
FIG. 35 is a block diagram showing the configuration of the second embodiment of the face state detection apparatus to which the present invention is applied. The face state detection device includes a face image capturing unit CL1, a spectacle detection unit CL2, a spectacle lens diameter detection unit CL8, a second face direction detection unit CL9, and a third face direction detection unit CL7. It has.
[0120]
The face image imaging means CL1 images the face of the person to be monitored and outputs face image data. The eyeglass detection means CL2 detects whether or not the person to be monitored is wearing eyeglasses.
[0121]
The spectacle lens diameter detection means CL8 processes the face image data output from the face image imaging means CL1 and detects at least one of the horizontal diameter and the vertical diameter of the spectacle lens worn by the person to be monitored.
[0122]
The second face orientation detection means CL9 detects the face direction from at least one of the horizontal diameter and the vertical diameter of the spectacle lens worn by the monitoring subject detected by the spectacle lens diameter detection means CL8.
[0123]
The third face orientation detection means CL7 detects the eye position by processing the face image data output from the face image imaging means CL1 when the eyeglass detection means CL2 cannot detect the eyeglasses. The face orientation is detected from the left or right eye shape obtained by image processing within the range based on the eye position or the detected eye position.
[0124]
[Device layout]
Since the arrangement of the apparatus of the present invention and the TV camera 1 as the face image capturing means CL1 are the same as those in the first embodiment described above, description thereof is omitted. The microcomputer 2 is programmed with programs relating to the processing of the eyeglass detection means CL2, the eyeglass lens diameter detection means CL8, the second face orientation detection means CL9, and the third face orientation detection means CL7.
[0125]
[System-wide processing]
FIG. 36 shows the overall processing flow of the system. First, when the process is started, a face image of the person to be monitored is picked up by the TV camera 1 and input to the microcomputer 2 as image data in S1.
[0126]
In S2, it is detected whether or not the person to be monitored is wearing glasses. In S3, based on the detection result of S2, if the person to be monitored is wearing glasses (YES in S3), the process proceeds to S10, and if the person to be monitored is not wearing glasses (NO in S3) ), The process proceeds to S9.
[0127]
In S10, the image data input to the microcomputer 2 is subjected to image processing to detect at least one of the horizontal diameter and the vertical diameter of the eyeglass lens worn by the person to be monitored.
[0128]
In S11, the orientation of the face of the monitor target is detected based on the data of at least one of the horizontal diameter and the vertical diameter of the eyeglass lens worn by the monitor target detected in S10 (the second face of the second face). (Direction detection process), and the process ends.
[0129]
In S9, if the spectacles cannot be detected in S3, or if a scene inside or outside the vehicle is not detected on the lens surface of the spectacles worn by the person to be monitored in S5, it is input to the microcomputer 2. From the left and right eye positions, or from the left or right eye shape obtained by image processing from the range based on the detected eye position Then, the face orientation is detected (third face orientation detection process), and the process ends.
[0130]
[Glasses detection processing]
Since it is the same as that of 1st Embodiment, the detailed description is abbreviate | omitted.
[0131]
[Detection processing of eyeglass lens diameter]
The lens diameter detection process S10 will be described with reference to FIG. By extracting the contour, the spectacle frame contour data is extracted from the original image. The circumscribed quadrangle of the right frame and the left frame of the contour line of the extracted spectacle frame is detected. The lateral sides of the circumscribed rectangle of the detected right frame and left frame are the horizontal diameter LHR of the right frame, the horizontal diameter LHL of the left frame, the vertical sides are the vertical diameter LVR of the right frame, and the vertical diameter LVL of the left frame.
[0132]
[Second face orientation detection process]
The second face orientation detection process S11 will be described with reference to FIGS. 37 is an outline of the spectacle frame when the person to be monitored is facing the front, and FIG. 38 is an outline of the spectacle frame when the person to be monitored is facing the right. Is the outline of the eyeglass frame when the person to be monitored is facing left.
[0133]
As can be seen from this figure, when the person to be monitored faces left and right, the lateral dimension of the eyeglass frame on the side facing the person is smaller than that on the front. Further, the lateral diameter of the spectacle frame on the facing side is smaller than the lateral diameter of the spectacle frame on the opposite side. A graph shown in FIG. 40 shows the relationship.
[0134]
Here, the method of detecting using the horizontal diameter of the spectacle frame only when the person to be monitored faces the left and right is illustrated, but when the face is directed in the vertical direction, detection is performed using the vertical diameter of the spectacle frame. be able to. Further, since the face of the person to be monitored moves vertically and horizontally, the face direction can be detected more accurately by using the horizontal and vertical diameters of the spectacle frame at the same time.
[0135]
[Third face orientation detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0136]
  Thus, in the face state detection device according to the second embodiment, the second face orientation detection means for detecting the orientation of the face of the subject based on the longitudinal or lateral diameter of the eyeglass lens diameter is provided. And when the vehicle body structure is not detected from the image reflected on the glasses, the second face direction detecting means is used to detect the face direction of the subject. In addition to the effects described in the embodiment, the detection accuracy can be further improved.The
[0137]
<Third Embodiment>
[System block diagram]
FIG. 41 is a block diagram showing the configuration of the third embodiment of the face state detection apparatus to which the present invention is applied. The face state detection device includes a face image capturing unit CL1, a glasses detection unit CL2, a reflection detection unit CL3, a vehicle body structure detection unit CL4, a vehicle body structure position detection unit CL5, and a first face orientation. Detecting means CL6, third face orientation detecting means CL7, spectacle lens diameter detecting means CL8, and second face orientation detecting means CL9 are provided.
[0138]
The face image imaging means CL1 images the face of the person to be monitored and outputs face image data. The eyeglass detection means CL2 detects whether or not the person to be monitored is wearing eyeglasses.
[0139]
The reflection detection means CL3 processes the face image data output from the face image imaging means CL1 to determine whether a scene inside or outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored. To detect.
[0140]
The vehicle body structure detection means CL4 processes the face image data output from the face image imaging means CL1, and detects the vehicle body structure from the inside and outside scenes reflected on the lens surface of the glasses worn by the person to be monitored. Is detected.
[0141]
The vehicle body structure position detection means CL5 processes the face image data output from the face image imaging means CL1, detects the lens frame of the glasses worn by the person to be monitored, and detects it by the vehicle body structure detection means CL4. The position of the body structure reflected on the glasses is detected by the relative positional relationship between the body frame reflected on the lens surface of the glasses worn by the monitored person and the lens frame.
[0142]
The first face orientation detection means CL6 detects the face direction from the position of the body structure reflected on the glasses detected by the body structure position detection means CL5.
[0143]
The third face orientation detection means CL7 reflects the scene inside or outside the vehicle when the glasses cannot be detected by the glasses detection means CL2 or on the lens surface of the glasses worn by the driver by the reflection detection means CL3. If it cannot be detected, the face image data output from the face image capturing means CL1 is processed to detect the eye position, and image processing is performed from the range based on the left and right eye positions or the detected eye position. The direction of the face is detected from the shape of the left or right eye obtained by.
[0144]
The spectacle lens diameter detection means CL8 processes the face image data output from the face image imaging means CL1 and detects at least one of the horizontal diameter and the vertical diameter of the spectacle lens worn by the person to be monitored.
[0145]
The second face orientation detection means CL9 detects the face direction from at least one of the horizontal diameter and the vertical diameter of the spectacle lens worn by the monitoring subject detected by the spectacle lens diameter detection means CL8.
[0146]
[Device layout]
Since the arrangement of the apparatus of the present invention and the TV camera 1 as the face image capturing means CL1 are the same as those in the first embodiment described above, description thereof is omitted.
[0147]
The microcomputer 2 includes glasses detection means CL2, reflection detection means CL3, body structure detection means CL4, body structure position detection means CL5, first face orientation detection means CL6, and third face. Programs relating to the processing of the direction detection means CL7, the spectacle lens diameter detection means CL8, and the second face direction detection means CL9 are programmed.
[0148]
[System-wide processing]
FIG. 42 shows the overall processing flow of the system. First, when the process is started, a face image of the person to be monitored is picked up by the TV camera 1 and input to the microcomputer 2 as image data in S1. In S2, it is detected whether or not the person to be monitored is wearing glasses.
[0149]
In S3, based on the detection result of S2, if the person to be monitored is wearing glasses (YES in S3), the process proceeds to S4, and if the person to be monitored is not wearing glasses (NO in S3). , The process proceeds to S9.
[0150]
In S4, the image data input to the microcomputer 2 is subjected to image processing to detect whether a scene inside or outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored.
[0151]
In S5, based on the detection result in S4, when a scene inside or outside the vehicle is reflected on the lens surface of the glasses worn by the person to be monitored (YES in S5), the process proceeds to S6. If a scene inside or outside the vehicle is not reflected on the lens surface of the glasses being worn (NO in S5), the process proceeds to S9.
[0152]
In S6, the vehicle body structure is detected from the inside and outside scenes reflected on the lens surface of the glasses worn by the person to be monitored by performing image processing on the image data input to the microcomputer 2.
[0153]
In S12, when a vehicle body structure can be detected from the inside and outside scenes reflected on the lens surface of the glasses worn by the person to be monitored (YES in S12), the process proceeds to S7, as shown in FIG. In such a situation, when the vehicle body structure cannot be detected from the inside / outside scene reflected on the lens surface of the glasses worn by the person to be monitored (NO in S12), the process proceeds to S10.
[0154]
In S7, the image data input to the microcomputer 2 is subjected to image processing to detect the lens frame of the glasses worn by the monitor subject, and the monitor subject detected by the vehicle body structure detection means CL4 puts it on. The position on the spectacles of the car body structure reflected on the spectacles is detected by the relative positional relationship between the car body structure reflected on the lens surface of the spectacles and the lens frame.
[0155]
In S8, the orientation of the face of the person to be monitored is detected from the position of the body structure reflected on the glasses detected in S7 on the glasses (first face orientation detection process), and the process ends. To do.
[0156]
In S9, if the spectacles cannot be detected in S3, or if a scene inside or outside the vehicle is not detected on the lens surface of the spectacles worn by the person to be monitored in S5, it is input to the microcomputer 2. The eye position is detected by performing image processing on the captured image data, and the direction of the face is determined from the shape of the left or right eye or the shape of the left or right eye obtained from the range based on the detected eye position. Is detected (third face orientation detection process), and the process ends.
[0157]
In S10, image data input to the microcomputer 2 is subjected to image processing to detect at least one of the horizontal diameter and the vertical diameter of the lens of the spectacle worn by the person to be monitored.
[0158]
In S11, the orientation of the face of the monitoring subject is detected from at least one of the horizontal diameter or the longitudinal diameter of the glasses of the glasses worn by the monitoring subject detected in S10 (second face orientation detection process), The process ends.
[0159]
[Glasses detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0160]
[Reflection detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0161]
[Vehicle structure detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0162]
[Detection of position on spectacles on vehicle structure image]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0163]
[First face orientation detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0164]
[Third face orientation detection processing]
Since it is the same as that of 1st Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0165]
[Detection processing of eyeglass lens diameter]
Since it is the same as that of 2nd Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0166]
[Second face orientation detection process]
Since it is the same as that of 2nd Embodiment mentioned above, the detailed description is abbreviate | omitted.
[0167]
  In this manner, the face state detection device according to the present embodiment detects the position of the eye from the face image of the subject, and detects the orientation of the face of the subject based on the eye position or the eye shape. A third face position detection means is provided, and when the vehicle body structure cannot be detected from the image reflected on the glasses lens, the third face direction detection means detects the face direction of the subject. Therefore, in addition to the effects described in the face state detection devices described in the first and second embodiments, the detection accuracy can be further improved.The
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a face state detection apparatus according to a first embodiment of the present invention.
FIG. 2 is a layout diagram of each component of the face state detection device according to the present invention.
FIG. 3 is a flowchart showing an overall processing procedure of the face state detection apparatus according to the first embodiment.
FIG. 4 is a flowchart showing specific contents of a reflection determination process.
FIG. 5 is an explanatory diagram showing a state when no reflection occurs in the glasses.
FIG. 6 is a face image of a person to be monitored when reflection is generated in the glasses.
7 is an explanatory diagram showing an image obtained by extracting a spectacle portion from the image shown in FIG. 6. FIG.
FIG. 8 is an explanatory view showing contour data of glasses and a reflected image.
FIG. 9 is an explanatory diagram showing an original image of a spectacle portion and contour data when the monitoring target person looks at the right side.
FIG. 10 is an explanatory diagram showing an original image of a spectacle portion and contour data when the monitoring subject looks at the left side.
FIG. 11 is a first partial view of an explanatory view showing a change state when the monitoring subject turns to the right from the front.
FIG. 12 is a second partial view of an explanatory view showing a change state when the monitoring subject turns to the right from the front.
FIG. 13 is a third partial view of an explanatory view showing a change state when the monitoring subject turns to the right from the front.
FIG. 14 is a flowchart showing specific contents of face orientation determination processing;
FIG. 15 is a flowchart showing specific contents of eye candidate position specifying processing;
FIG. 16 is an explanatory diagram showing a state of detecting the eye position of a subject using an eye template.
FIG. 17 is an explanatory diagram showing the size of an eye presence region.
FIG. 18 is an explanatory diagram showing statistics of the length of the eye in the lateral direction.
FIG. 19 is an explanatory diagram showing statistics of the length of the eye in the vertical direction.
FIG. 20 is an explanatory diagram showing processing for determining the position of an eye presence region.
FIG. 21 is a first part of a flowchart showing the specific contents of the eye determination processing.
FIG. 22 is a second part of the flowchart showing the specific contents of the eye determination process.
FIG. 23 is an explanatory diagram of a process of extracting a minute image.
FIG. 24 is an explanatory diagram showing processing for obtaining a binarization threshold value.
FIG. 25 is an explanatory diagram illustrating processing for detecting the position of the eye using an eye template.
FIG. 26 is a flowchart showing specific contents of face orientation determination processing;
FIG. 27 is an explanatory diagram of processing for extracting an eye outline by image processing;
FIG. 28 is an explanatory diagram of a process of extracting an eye contour by image processing.
FIG. 29 is an explanatory diagram of a process of extracting an eye contour by image processing.
FIG. 30 is an explanatory diagram of a process of extracting an eye contour by image processing.
FIG. 31 is an explanatory diagram showing the positions of the top of the eyes, the corner of the eyes, and the arc-shaped line peak.
FIG. 32 is an explanatory diagram showing an image when the subject moves the eyeball.
FIG. 33 is an explanatory diagram showing an eye / eye corner and an arcuate line peak position when a subject moves his / her eyeball.
FIG. 34 is a characteristic diagram showing the relationship between the face direction, the peak peak peak distance, and the peak peak distance.
FIG. 35 is a block diagram showing a configuration of a face state detection apparatus according to a second embodiment of the present invention.
FIG. 36 is a flowchart showing the entire processing of the face state detection apparatus according to the second embodiment.
FIG. 37 is an explanatory diagram showing frame contour data of glasses when the person to be monitored is looking in front.
FIG. 38 is an explanatory diagram showing frame contour data of glasses when the monitoring subject is looking at the right side.
FIG. 39 is an explanatory diagram showing frame contour data of glasses when the monitoring subject is looking at the left side.
FIG. 40 is a characteristic diagram showing the relationship between the orientation of the face and the lateral diameter of the glasses.
FIG. 41 is a flowchart showing the overall processing of the face state detection apparatus according to the third embodiment.
FIG. 42 is a flowchart showing the entire processing of the face state detection apparatus according to the third embodiment.
FIG. 43 is an explanatory diagram showing an image of a spectacle portion when a vehicle body structure cannot be detected from a scene inside and outside the vehicle reflected on the lens surface of the spectacle worn by the person to be monitored.
[Explanation of symbols]
1 Camera
2 Microcomputer
CL1 face image capturing means
CL2 glasses detection means
CL3 Reflection detection means
CL4 body structure detection means
CL5 body structure position detection means
CL6 first face orientation detection means
CL7 Third face orientation detection means
CL8 Eyeglass lens diameter detection means
CL9 Second face orientation detection means

Claims (6)

In a face state detection device that captures an image of a face of a subject with a face image capturing unit and detects the orientation of the face of the subject based on the captured face image.
Glasses detection means for detecting whether the subject is wearing glasses based on the face image;
When it is detected by the glasses detection means that the subject is wearing glasses, reflection detection means for detecting whether reflection is present in the glasses;
A vehicle body structure detection means for detecting an image of the vehicle body structure included in the reflection image when reflection is detected by the reflection detection means ;
Vehicle body structure position detection means for detecting the position of the image of the vehicle body structure detected by the vehicle body structure detection means on the lens of the glasses ;
First face direction detecting means for detecting the face direction of the subject based on the position of the image of the car body structure on the lens detected by the car body structure position detecting means ;
A face state detection apparatus comprising: When the eyeglass detection means detects that the subject is wearing eyeglasses, the eyeglass lens frame image is detected from the face image, and based on the lens frame image, the face of the subject person is detected. A second face orientation detecting means for detecting the orientation;
When the image of the vehicle structure is not detected based on the reflection image detected by the reflection detection means, the orientation of the face of the subject is detected by the second face direction detection means. The face state detection apparatus according to claim 1. The second face orientation detection means detects a horizontal diameter and a vertical diameter of the lens part of the glasses included in the face image, and based on the size of the horizontal diameter and the vertical diameter of the lens part, The face state detection apparatus according to claim 2, wherein the face direction of the subject is detected . The face state detection device according to any one of claims 1 to 3, wherein the vehicle body structure is present in front of the subject . 5. The vehicle body structure according to claim 1, wherein the vehicle body structure is at least one of a contour of a front window, a contour of a side window, a rearview mirror, or an A pillar. Face condition detection device. Eye detection means for detecting an image of the eye of the subject from the face image;
Third face orientation detection means for detecting the orientation of the subject's face from the eye image detected by the eye detection means based on the positions of the left and right eyes of the subject and the shape of the eyes; Further comprising
The direction of the face of the subject is detected by the third face orientation detection means when the eyeglass detection means does not detect that the subject is wearing glasses. The face state detection apparatus of any one of Claims 1-5.

Images