JP5359266B2 - Face recognition device, face recognition method, and face recognition program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face recognition device for improving the precision of face recognition by specifying a shielded face site by detecting the shielded object of the face site. <P>SOLUTION: In the face recognition device 10, a nose region detection means 2 detects a nose region from a pickup image by an image pickup means 100, and a mask wearing determination means 3 determines mask wearing from the luminance value of the pickup image when the nose region is not detected, and a mask region detection means 31 detects a mask region, and an in-mask nose region detection means 32 detects the nose region from the mask region, and a face recognition means 5 detects an eye region from the detected nose region to perform face recognition. Thus, the precision of face recognition is improved by performing face recognition based on the nose region by detecting and using the mask region. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a face recognition device that recognizes a person's face based on a photographed image, and more particularly to a face recognition device that improves face recognition accuracy by detecting a facial part concealment.

The face recognition device recognizes the person's face from the captured image and identifies it, so that the person can be authenticated and the situation can be ascertained. It is used in a wide range of fields including safety confirmation devices. In general, a face recognition device detects each face part including a face region, an eye position, a nose position, and a mouth position, and determines the face orientation and eye opening / closing judgment from the situation of the face part. Face recognition is performed. However, when the face part is concealed by the concealment object, the face recognition device may erroneously recognize the face part by the concealment object.
Some conventional face recognition apparatuses perform face recognition with emphasis on the characteristic part of the face part selected by the user (see, for example, Patent Document 1).
JP 2007-257221 A

  However, the conventional face recognition device has a problem that it takes time and effort for the user to select the characteristic part of the facial part each time. In addition, even if a conventional face recognition device detects a concealed object, it uses other face parts without using the concealed face part. There is a problem that accuracy is lowered.

  The present invention has been made to solve the above-described problems, and it is an object of the present invention to provide a face recognition device that identifies a concealed face part by detecting a concealment object of the face part and improves the accuracy of face recognition. To do.

  According to one aspect, the face recognition device disclosed in the present application detects a nose region from a captured image, and detects a mask region by wearing a mask from a luminance value of the captured image when the nose region is not detected, thereby detecting the nose region from the mask region. , And an eye region is detected from the detected nose region to perform face recognition.

  Thus, the face recognition apparatus disclosed in the present application performs face recognition based on the nose region using the mask region by detecting the mask region, and can improve the accuracy of face recognition.

(First embodiment of the present invention)
Hereinafter, a face recognition device according to a first embodiment of the present invention will be described with reference to FIGS. 1 is a block diagram showing a configuration of a face recognition apparatus according to a first embodiment of the present invention, FIG. 2 is an overall configuration diagram of the face recognition apparatus described in FIG. 1, and FIG. 3 is described in FIG. 2 shows an overall flowchart of the face recognition apparatus. FIG. 4 shows a flowchart of face area and face part detection of the face recognition apparatus shown in FIG. 1, and FIG. 5 shows a flowchart of nose area detection of the face recognition apparatus shown in FIG. FIG. 6 is a flowchart of mask detection for transmission data reception of the face recognition device shown in FIG. 1, and FIG. 7 is a flowchart of eyeglass bridge region detection of the face recognition device shown in FIG. 8 is a flowchart of eyeglass frame region detection of the face recognition device described in FIG. 1, FIG. 9 is a flowchart of sunglasses region detection of the face recognition device described in FIG. 1, and FIG. 10 is described in FIG. 6 shows a flowchart of the sunglasses region comprehensive determination of the face recognition device. 11 is a flowchart for detecting the position of the eye region of the face recognition apparatus shown in FIG. 1, FIG. 12 is a flowchart for detecting the eye patch region of the face recognition device shown in FIG. 1, and FIG. 13 is shown in FIG. Explanatory drawing of the face direction detection of the made face recognition apparatus is shown.

  In FIG. 1, the face recognition device 10 according to the present embodiment includes a face region detection unit 1, a nose region detection unit 2, a mask wearing determination unit 3, a mask region detection unit 31, and an in-mask nose region detection unit 32. And an eye area detecting means 4, a face recognizing means 5, and a face information accumulating means 5a. In addition, the face recognition apparatus includes a spectacle frame region detection unit 41, a sunglasses determination unit 42, an eyepatch determination unit 43, an eyepatch region detection unit 44, a face recognition prediction unit 6, a feature determination unit 7, and an alarm unit. 8.

  The face area detecting means 1 detects a face area as a human face existing area from the captured image taken by the photographing means 100. The nose region detecting means 2 detects a nose region as a human nose existing region, and stores and stores it in the face information accumulating unit 5a. The mask wearing determination unit 3 determines whether a person is wearing a mask from the horizontal and vertical luminance values of the captured image 100 when the nose region detection unit 2 does not detect the nose region.

  Based on the mask wearing determination means 3, the mask area detecting means 31 detects a mask area as a mask existing area when a person is wearing a mask. The in-mask nose area detecting means 32 specifies the nose area included in the mask area based on the mask area detected by the mask area detecting means, and stores and stores it in the face information accumulating means 5a.

  The eye area detecting means 4 detects an eye area as a human eye existing area based on the nose area specified by the nose area detecting means 2 or the nose area detecting means 32 in the mask. The face recognizing means 5 recognizes the eye opening and the line-of-sight direction from the image information of the eye area detected by the eye area detecting means 4, recognizes the face direction from the eye area and the nose area, The recognized contents are accumulated and stored in the face information accumulating means 5a.

  Further, the spectacle frame region detecting means 41 detects a spectacle frame region as an existing region of spectacle frames worn by a person from the image information of the detected eye region. The sunglasses determination means 42 determines whether the glasses are sunglasses from the luminance distribution of the eyeglass frame area detected by the eyeglass frame detection means 41.

  The eyepatch determination means 43 determines whether a person wears the eyepatch based on the horizontal and vertical luminance values of the image information in the detected eye area. In addition, when the eye patch determination unit 43 determines that the eye patch is worn, the eye patch region detection unit 44 detects the eye patch region as the eye patch existing region.

  The face recognition predicting means 6 predicts the state of the face recognized at the next time point from the stored and stored face information storing means 5a. The feature judging means 7 judges the feature of the person's face from the stored and stored face information accumulating means 5a. The alarm means 8 estimates the current person's situation according to the face status of the person recognized by the face recognition means 5, the face recognition prediction means 6 and the feature determination means 7, and issues an alarm when necessary. generate.

  As shown in FIG. 2, the overall configuration of the present embodiment includes the face recognition device 10, an automobile 200, and a driver 300 that drives the automobile 200. The face recognition device 10 includes an in-vehicle camera 100A as the photographing unit 100 that photographs the face 300A of the driver 300, and an alarm device 8A as the alarm unit 8 that causes the driver 300 to generate an alarm when necessary. With.

  Hereinafter, the operation of the face recognition device of the present embodiment based on the above configuration will be described. Hereinafter, the X direction refers to the horizontal direction of the face, and the Y direction refers to the vertical direction of the face. Further, as shown in FIG. 3, the face recognition apparatus sequentially calculates image frames in time series using the image information generated by the processing from S1 to S9 as one image frame, and the face information storage means Accumulated and stored in 5a.

First, the face recognition device 10 detects an area of the face 300A of the driver 300 as a face area from an image frame taken by the in-vehicle camera 100A (S2).
Regarding the detection of the face area, the face area detecting means 1 first sequentially adds the luminance values of the respective pixel data arranged adjacent to each other in the vertical direction of the image frame, as shown in FIG. (S21). Next, the face area detecting means 1 uses a preset threshold value of the brightness value, and when the sum of the sequentially added brightness values is larger than the threshold value, the range in which the brightness values are sequentially added. Is determined to be included in the face area, and the face area is specified (S22). Here, the identification of the face area utilizes the fact that the face area tends to be imaged relatively brightly compared to the hair area and the background area.

  The face area detecting means 1 calculates the amount of change in the horizontal direction with respect to the luminance values sequentially added in S21, and identifies the position where the calculated amount of change greatly changes as the face contour portion in the horizontal direction. Then, the face area is specified (S23). Here, the identification of the face region is such that the amount of change in the luminance value in the horizontal direction in the image frame tends to change relatively greatly in the boundary region between the background region and the face region as compared with other parts. Take advantage of something.

  The face area detecting means 1 stores the face area related to the specified face 300A, and takes over the information of the face area to subsequent processing (S24). Next, as shown in FIG. 3, the face recognition device 10 detects a face part included in the face 300A from the specified face area (S3).

  Regarding the detection of the face part, the nose region detection means 2 first detects the nose region included in the face 300A as shown in FIG. 4B (S31). As shown in FIG. 5, the nose region detecting means 2 sequentially adds luminance values of pixel data arranged adjacent to each other in the vertical direction (S311). The nose region detecting means 2 specifies a minimum value in the sequentially added luminance values (S312).

  The nose region detection means 2 determines whether or not there are a plurality of local minimum values (S313). When there are a plurality of local minimum values, the nose region detecting means 2 calculates a horizontal distance between the two local minimum values (S314).

  The area detection unit 2 determines whether the calculated distance is smaller than a threshold (S315). When the calculated distance is smaller than the threshold value, the area detecting unit 2 calculates the center position between the two minimum values (S316). The area detecting means 2 identifies the calculated center position as a nose area and stores it in the face information accumulating means 5a (S317).

  In addition, when there are not a plurality of the minimum values in S313, the nose region detection unit 2 specifies the minimum value as a nose region and stores it in the face information storage unit 5a (S318). Here, the identification of the nose region utilizes the fact that the nose region tends to be imaged relatively darkly in the face region. Further, if the calculated distance is larger than the threshold value in S315, the area detecting unit 2 determines that the nose area cannot be detected (S319).

  Next, the face area detection means 1 sets a mouth presence area below the detection position of the nose area in the previous period S31, and detects the mouth area from the continuity in the horizontal direction of the vertical edge in the area ( S32). Next, the face area detecting means 1 uses a technique for specifying the eye Y position based on, for example, the sum of horizontal pixel values (luminance values) in the image data. This technique is described in, for example, Japanese Patent Application Laid-Open No. 2004-234367 filed by the applicant of the present application, and the eye Y position can be accurately detected by using these techniques (S33). Here, the eye Y position indicates the position of the eye of the driver 300 in the vertical direction.

  Next, the face area detection means 1 determines whether the nose area or mouth area has been detected (S34). The mask wearing determination means 3 performs mask detection when the nose region or mouth region cannot be detected by this determination (S35).

  Regarding this mask detection, the mask wearing determination means 3 sets a mask detection region as shown in FIG. 6 (S351). Regarding the setting of this mask detection area, the mask wearing determination means 3 sets the eye position (Y) in this eye area when the upper end of the mask detection area has already been acquired in the previous image frame. . Here, the eye position (Y) is the Y position in the vertical direction of one eye when only one eye can be acquired, and this is the right eye and left eye when both eyes are acquired. The center of the Y position is shown.

 The mask wearing determination means 3 indicates the eye Y position when the eye area cannot be acquired in the immediately preceding image frame, and the captured image when the eye Y position cannot be acquired. Indicates the top edge of the frame.

  Next, the mask wearing determination means 3 sets the lower end of the mask detection area to upper end + face width × N (N is a threshold value). Moreover, the said mask wearing judgment means 3 makes the right end of a mask detection area the right end of the said face area. Further, the mask wearing determination means 3 sets the left end of the mask detection area as the left end of the face area.

  Next, the mask wearing determination means 3 projects the horizontal direction in the detected mask detection area, and detects a position where the gradient of the projected luminance value is maximum (S352). Here, in the horizontal projection, the mask wearing determination means 3 sets a portion where the gradient of the projected luminance value rapidly increases as the upper end of the mask area, and a portion where the gradient of the projected luminance value rapidly decreases Is the lower end of the mask area.

  Next, the mask wearing determination means 3 projects the vertical direction in the detected mask detection area, detects the position where the gradient of the projected luminance value is maximum, and masks, similarly to S352. The right end and the left end of the area are detected (S353).

  Next, the mask wearing determination means 3 determines from the detected mask area whether the aspect ratio of the mask area satisfies the condition within the threshold (S354). When the aspect ratio of the mask area is within the threshold value, the mask wearing determination means 3 determines whether the ratio of the mask width and the face width satisfies the condition within the threshold value from the detected mask area (S355). .

  When the ratio between the mask width and the face width is within the threshold, the mask wearing determination means 3 determines whether the ratio between the mask height and the face width satisfies the condition within the threshold from the detected mask region ( S356). When the ratio between the mask height and the face width is within the threshold value, the mask wearing determining means 3 determines whether the average luminance value of the mask region satisfies the condition equal to or greater than the threshold value from the detected mask region (S357). ).

  When the average brightness value of the mask area is equal to or greater than the threshold value, the mask wearing determination means 3 determines whether or not a condition that the brightness variance value of the mask area is less than the threshold value is satisfied from the detected mask area (S358). . When the luminance dispersion value of the mask area is less than the threshold value, the mask wearing determination means 3 determines that the detected mask area includes a mask (S359).

  The mask wearing determination means 3 determines that the detected mask area does not include a mask if any one of the conditions is not satisfied in the determination from S354 to S358. As shown in FIG. 4B, the mask area detecting means 31 detects a mask from the detected mask area based on this determination (S35). In addition, the mask nose region detecting means 32 can detect, for example, the center position of the detected mask region as a nose region, detect the nose region, and store it in the face information accumulating unit 5a.

  Next, the spectacle frame region detecting means 41 first detects a spectacle bridge bridged between the left and right spectacle frames (S36). As shown in FIG. 7, the eyeglass frame area detecting means 41 checks whether or not an eye area has been stored in the past immediately before based on the face information accumulating means 5a (S361). If the eye area has been stored in the previous past, the eyeglass frame area detecting means 41 calculates the eyeglass bridge detection processing range from this eye area (S362).

  Regarding the calculation of this detection processing range, the horizontal direction is the midpoint of the left and right eye positions (X) of the previous frame, and the vertical direction is the eye Y position or the midpoint of the left and right eye positions (Y) of the previous frame. An area having N1 pixels on the upper side in the vertical direction, N2 pixels on the lower side, and N3 pixels on both sides in the horizontal direction can be calculated as the reference point. The N1, N2, and N3 may be determined by a fixed ratio with respect to the driver's 300 face width.

  As described above, the spectacle frame region detection unit 41 calculates an appropriate detection processing region regardless of the distance between the in-vehicle camera 100 and the driver 300 by changing the detection processing range according to the face width. can do. Specifically, the spectacle frame region detection means 41 widens this detection processing range when the driver 300 is close to the in-vehicle camera 100, and when the driver 300 is far from the in-vehicle camera 100. This detection processing range can be narrowed.

  The spectacle frame region detection means 41 calculates the luminance integrated value by integrating the luminance values of the pixels arranged in the horizontal direction in this detection processing range (S363). The spectacle frame region detection means 41 detects the spectacle bridge vertical position candidate based on the vertical difference value of the luminance integrated value (S364).

  Here, the vertical difference value can be calculated by, for example, vertical difference value = M1 (y + constant pixel) −M1 (y). However, y represents one of the Y coordinates in the horizontal line, and M1 (y) represents the luminance integrated value in the horizontal line.

  Further, with respect to the upper / lower position candidate, the spectacle frame region detecting means 41 detects the Y coordinate having the maximum vertical difference value as the upper position candidate of the spectacle bridge, and the Y coordinate having the minimum vertical difference value is determined by the spectacle bridge. It can be detected as a lower position candidate. Here, regarding this detection, when a spectacles bridge is present, the fact that the luminance integrated value of the spectacles bridge region in the horizontal direction is relatively lower than other parts is used.

  The spectacle frame region detection means 41 calculates a luminance integrated value by integrating the luminance values in the vertical direction in this detection processing range (S365). The spectacle frame region detection means 41 detects the right and left position candidates of the spectacle bridge based on the left-right difference value of the luminance integrated value (S366).

  Here, the left-right difference value can be calculated by, for example, left-right difference value = M2 (x + constant pixel) −M2 (x). However, x indicates one of the Y coordinates in the vertical line, and M2 (x) indicates the luminance integrated value in the vertical line.

  Further, the eyeglass frame region detecting means 41 detects the X coordinate having the maximum left-right difference value as the left position candidate of the eyeglass bridge with respect to the left-right position candidate, and determines the X coordinate having the minimum left-right difference value of the eyeglass bridge. It can be detected as a right position candidate. Here, regarding this detection, when the eyeglass bridge is present, the fact that the integrated luminance value of the eyeglass bridge region in the vertical direction is relatively low as compared with other portions is used.

  The spectacle frame region detection means 41 determines the presence of the spectacle bridge based on the detected spectacle bridge left-right position candidate and vertical position candidate processing range (S367). The eyeglass frame region detecting means 41 has a first condition that the sum of the absolute value of the upper / lower difference value of the upper position candidate of the spectacle bridge and the absolute value of the upper / lower difference value of the lower position candidate is greater than or equal to a threshold value. Judge whether to satisfy.

  If the spectacle frame region detection means 41 determines that the first condition is satisfied, the absolute value of the left / right difference value of the right position candidate and the absolute value of the left / right difference value of the left position candidate are determined. It is determined whether the second condition that the sum is equal to or greater than the threshold value is satisfied. Further, when the spectacle frame region detecting means 41 determines that the second condition is satisfied, the distance between the left position candidate and the right position candidate of the spectacle bridge is within a threshold range with respect to the face width. When the condition 3 is satisfied, it is determined that there is an eyeglass bridge. When it is determined that the eyeglass bridge exists, the eyeglass frame area detecting means 41 detects and stores the eyeglass bridge area (S368).

  As described above, the spectacle frame region detection unit 41 can detect the spectacle bridge upper and lower position candidates and the left and right position candidates as spectacle bridges only when the natural positional relationship is satisfied under these three conditions. The spectacle frame region detecting means 41 can change the first threshold and the second threshold by learning the detection result of the spectacle bridge. The spectacle frame region detection means 41 can also increase or decrease the first and second threshold values based on the spectacle bridge acquisition rate. In this case, an optimum threshold value corresponding to the type of spectacles is obtained. It is possible to accurately detect the eyeglass bridge.

  The eyeglass frame area detecting means 41 calculates the eyeglass bridge detection processing range from the nose area when the eye area is not stored in the previous past in S361 (S369). In this case, the spectacle frame region detection means 41 uses the position of the N4 pixel from the midpoint of the left and right nostrils in the horizontal direction to the upper side in the vertical direction as a reference position, and performs the spectacle bridge detection process in the same manner as in S362. Calculate the range.

  The N4 can also be determined by a fixed ratio with respect to the driver's 300 face width. The eyeglass frame region detection means 41 performs the same processing from S363 to S368 after S369. Further, when the spectacle frame area detecting means 41 determines in S367 that no spectacle bridge exists, the spectacle bridge area is not stored in S368.

  Next, as shown in FIG. 4B, the spectacle frame region detecting means 41 detects a spectacle frame (S37). As shown in FIG. 8A, the spectacle frame region detection means 41 first calculates a detection processing range for detecting the spectacle lower frame (S371).

  The horizontal range of this detection processing range is a range of N9% of the face width on the left and right with the eye position detected in the past as the center. The lower end of the detection processing range is a position N10% below the face width from the midpoint of the nostril. The upper end of this detection processing range is the midpoint of the eye Y position or the eye position (Y) of the previous frame.

  After calculating the detection processing range, the spectacle frame region detection means 41 executes a differential filter extraction process using a differential filter in the detection processing range (S372). This differential filter is a filter that extracts an image region having a high luminance from the top to the bottom in the vertical direction, an image region having a low luminance, and a lower frame portion that changes to an image region having a high luminance.

  The spectacle frame region detection means 41 calculates differential filter values in order from the lower side to the upper side of the detection processing range in the Y direction line, and detects pixels that first become equal to or greater than the threshold value. The value of this differential filter is a numerical value obtained by multiplying the 8 pixels near 8 centered on the target pixel by the coefficient of the differential filter and adding the values obtained by multiplication.

  The spectacle frame region detection means 41 has a luminance of a pixel that is one pixel above the target pixel during the differential filter processing, for example, a pixel whose Y coordinate is y1-1 with the Y coordinate of the target pixel as y1. In the above, it determines with the light from an external light source being the part which the spectacles lens has reflected. The spectacle frame region detection means 41 detects this pixel as a reflection partial pixel. The first luminance is 240, for example, with 255 gradations. As this external light source, an LED light source installed in the vehicle for illuminating the driver 300 for various purposes such as face recognition can be used.

  Further, the spectacle frame region detection means 41 has a luminance of a pixel one pixel above or below the target pixel at the second side in the lower part of the detection processing range that is the target pixel at the start of the differential filter processing. If it is higher than the brightness, it is determined that the entire face tends to be bright. The spectacle frame region detection means 41 increases the first luminance based on this determination.

  For example, when the second luminance is 210 at 256 gradations, the spectacle frame region detection unit 41 increases the first luminance from 240 to 250. As described above, the spectacle frame region detection unit 41 can detect the reflection partial pixel more accurately by controlling the first luminance and the second luminance in a correlated manner.

  When the eyeglass frame region detecting means 41 finishes this detection for a certain Y direction line, the same processing is sequentially performed for other Y direction lines from the left side to the right side in the detection processing region. Note that the spectacle frame region detection means 41 can calculate a maximum of n results by thinning out Y-direction lines to be subjected to differential filter processing in accordance with the horizontal width of the detection processing range. This n can be set according to the correlation between the processing speed and the determination accuracy.

  The spectacle frame region detection means 41 detects the lower position of the spectacle lower frame when the differential filter extraction processing is completed (S373). When a plurality of pixels are detected in S372, the spectacle frame region detection means 41 sets the lower position of the spectacle lower frame to the middle of the Y coordinate of each pixel included in the detected n pixels. The value is the Y coordinate of the lower position. Further, the spectacle frame region detection means 41 stores the X coordinate of the horizontal central portion of the detection processing range as the X coordinate of the lower position.

  The spectacle frame region detection means 41 determines whether or not the reflection partial pixel is detected when no pixel is detected in S372. When it is determined that the reflection partial pixel is detected, the Y coordinate of the reflection partial pixel is set as the Y coordinate of the lower position. When there are a plurality of the reflection partial pixels, the spectacle frame region detection means 41 sets the maximum Y coordinate of the reflection partial pixel as the Y coordinate of the lower position.

  The spectacle frame region detecting means 41 does not detect the lower frame when the spectacle has no lower frame, but considers that the spectacle exists in the reflective portion when the reflective portion is detected. Therefore, the lowermost end of the reflection portion is set as the lower position. The spectacle frame region detecting means 41 determines that there is no spectacle lower frame when no pixel is detected in S372 and the reflection partial pixel is not detected.

  The spectacle frame region detecting means 41 can calculate the average value of the differential filter values in the n detected pixels and use this as the spectacle lower frame score. The spectacle frame region detection means 41 can determine whether the lower position detected by the spectacle lower frame score is spectacles or eyes. The spectacle frame region detection means 41 determines that the spectacles lower frame score is larger than the first threshold value, and determines that the spectacles frame region score is smaller than the first threshold value. Can be determined to be eyes.

  Next, as shown in FIG. 8A, the spectacle frame region detection means 41 calculates a detection processing range in the same manner as the processing of the spectacle lower frame with respect to the spectacle upper frame detection (S374), and performs the differentiation. The filter is extracted (S375), and the frame on the glasses is detected (S374).

  As shown in FIG. 4B, the spectacle frame region detecting means 41 determines whether or not the spectacle region is detected in S37 (S38). When the glasses area is detected, the sunglasses determination means 42 detects the sunglasses area (S39).

  As shown in FIG. 9, the sunglasses determination means 42 initializes the sunglasses flag indicating the result of determination as to whether or not the sunglasses are off (S391). Next, the sunglasses determination means 42 calculates a luminance average and luminance dispersion value within the sunglasses processing range (S392). The upper and lower sides of the sunglasses processing range are the upper position of the glasses upper frame and the lower position of the glasses lower frame, and the left and right of the sunglasses processing range are the same as the processing range of the glasses lower frame.

  Next, the sunglasses determination means 42 determines whether or not the average luminance value is equal to or less than a threshold value (S393). If it is determined that the average luminance value is equal to or smaller than the threshold value, the sunglasses determining means 42 determines whether or not the variance value of the luminance value is equal to or smaller than the threshold value (S394).

  If the determination is below the threshold value in this determination, the sunglasses determination means 42 turns on the sunglasses flag and detects sunglasses (S395). The eye area detection means 4 sets the eye position as the center part of the sunglasses when the sunglasses are detected.

  Thus, even when sunglasses are worn, the eye position is specified from the sunglasses, and the accuracy of face recognition can be improved. Next, the sunglasses determination means 42 determines whether or not the sunglasses determination on both the left and right sides of the glasses has been executed (S396).

  When it is determined that the sunglasses determination means 42 has not performed the left and right side sunglasses determination or any one of the sunglasses determination, the processing after S391 is similarly performed. Further, when it is determined that the sunglasses determination on both the left and right sides has been performed, the sunglasses determination unit 42 ends the process related to the sunglasses determination.

  Further, when it is determined in S393 that the average luminance value is greater than the threshold value, the sunglasses determination means 42 performs the sunglasses determination in S396 and performs the same processing as in S396 and subsequent steps. Further, when it is determined in S394 that the dispersion value is greater than the threshold value, the sunglasses determination means 42 performs the sunglasses determination in S396 and performs the same processing as in S396 and subsequent steps.

  Next, as shown in FIG. 10, the sunglasses determination means 42 reads the sunglasses flag of the frame stored in the past (S397). The sunglasses determination means 42 determines whether or not the sunglasses flag is on for the number of frames equal to or greater than the threshold (S398).

  When it is determined that the sunglasses are turned on in a frame equal to or greater than the threshold, the sunglasses determination means 42 turns on a comprehensive sunglasses flag indicating the result of comprehensively determining the presence or absence of sunglasses (S399-1). If it is determined in S398 that the number of frames is less than the threshold value, the sunglasses determination means 42 turns off the general sunglasses flag (S399-2).

  In this way, the sunglasses determination means 42 makes a comprehensive determination using the sunglasses determination results in a plurality of frames, thereby preventing misrecognition of sunglasses that tend to occur only with one frame result indicated by the sunglasses flag. Can do. Factors that cause this misrecognition of sunglasses include, for example, an instantaneous luminance change and hand approach.

  Next, the eye area detecting means 4 detects the position of the eye area as shown in FIG. 3 (S4). As shown in FIG. 11, the eye area detecting means 4 first determines whether or not the condition of the position of the eye area satisfies the condition that the sunglasses area exists and the lens area of the sunglasses area is dark. (S41). The eye area detecting means 4 can make a judgment on this condition depending on whether the average luminance value of the lens area is less than a threshold value.

When the condition of S41 is not satisfied, the eye area detection unit 4 starts a loop process for applying the subsequent process to each of the right eye and the left eye (S42). In this loop process, first, the eye patch region detecting means 44 detects the eye patch region (S43).
Regarding the detection of the eye patch area, the eye patch determination unit 43 sets an eye patch detection area which is a detection area of the eye patch area as shown in FIG. 12A (S431). As shown in FIG. 2B, the eye patch determining means 43 first sets the eye patch detection area parallel to the nose position (X) which is the nose position in the X direction and the face edge which is one end of the face area. The center line to be calculated is calculated.

  The eye patch determining means 43 determines a range that falls within N3% of the face width on the left side in the X direction from the center line and a range that fits within N4% of the face width on the right side in the X direction from the center line in the X direction. The width of Further, the eye patch judging means 43 sets the position that becomes N1% of the face width along the Y direction upward from the eye Y position as the upper position in the Y direction of the eye patch detection region, and faces downward from the eye Y position. A position that is N2% of the face width along the Y direction is set as a lower position in the Y direction of the eyeband detection region. Further, when the eye Y position does not exist, the eye patch determination unit 43 can substitute the eye position (Y) in the previous frame as the eye Y position.

  Next, the eye patch determining means 43 determines the presence of the eye patch based on the detected eye patch detection region as shown in FIG. The eye patch determination means 43 determines that an eye patch is present when the brightness variance value of the eye patch detection area is less than the threshold value N and the brightness average value of the eye patch detection area occupying the brightness average value of the face area is greater than or equal to the threshold value M. to decide. The eye patch determining means 43 can make the position and size of the eye patch the same as the eye patch detection region.

  Next, as shown in FIG. 11, the eye patch determining means 43 determines whether an eye patch area has been detected (S44). If the eye patch area is not detected, the eye patch determining means 43 detects the eye position (S45).

  Regarding the detection of the eye position, the eye region detection means 4 can detect the eye position based on the detected nose position as shown in FIG. Further, as shown in FIG. 4B, the eye patch judging means 43 regards the center position of the detected mask area as the alternative position of the nose and similarly detects the eye position even when a mask is present. Can do. As described above, the eye patch determining unit 43 calculates the eye position based on the nose position, so that the eye position can be detected with high accuracy.

  Further, the eye patch judging means 43 can detect the eye position by using a detection method using binarization processing, for example. Specifically, the eye patch judging means 43 binarizes the acquired image frame with numerical values of 1 and 0, detects an area where pixels with numerical value 1 of the pixel data gather in the obtained image frame, The region can be identified as a region that is highly likely to be a pupil.

  In addition, as shown in FIG. 11, when the eye patch area is detected in S44, the eye area detecting means 4 detects the eye position as the central part of the eye patch area. Thus, even when the eye patch is worn, the eye position is specified from the eye patch, and the accuracy of face recognition can be improved.

  When the eye area detection unit 4 performs the process relating to the eye area from S43 to S45 for the left and right eyes, the loop process is terminated (S46). Next, as shown in FIG. 3, the eye region detection means 4 extracts the eye shape for the detected eye (S5).

  Next, as shown in FIG. 3, the face recognizing means 5 calculates the face orientation by calculating the face orientation using the positional relationship of each part of the detected face (S6). Specifically, as shown in FIG. 13C, the face recognizing means 5 can calculate the lateral angle of the face from the relative position B of the nose with respect to the face width A of the face area. In addition, as shown in FIG. 4D, the face recognition means 5 determines the angle of the face in the horizontal direction from the relative position B of the nose at the center of the mask with respect to the face width A of the face area, even when a mask is present. Can be calculated.

  Next, as shown in FIG. 3, the eye region detection means 4 calculates the eye opening (S7). Specifically, the eye region detection means 4 calculates the eye region in the image frame as feature data based on the luminance distribution in the eye region, the luminance value average of the pixels, and the standard deviation of the luminance values. The eye opening can be calculated by comparing the amount of change in the previous and next image frames.

  The face recognition prediction means 6 predicts a face in the next frame (S8). In addition, the feature determination means 7 can determine the features of the driver 300, for example, features such as blinking, shaking his / her head, little eye openness, etc., and can provide information on this prediction. .

  The alarm means 8 generates a specific action for the driver 300 according to the face orientation and the open / closed state of the eyes (S9). As described above, the face recognition prediction unit 6 can determine whether the driver 300 is included in the prediction range because the face motion is predicted using the correlation between the previous and next frames. The detection accuracy of the abnormal operation of the driver 300 can be improved.

  For example, when it is possible to determine that the driver 300 is dozing or overworking based on the degree of opening of the eyes of the driver 300, the warning unit 8 can generate a rest instruction alarm by voice. Further, for example, when the warning means 8 can determine that the driver is looking aside from the direction of the face of the driver 300, the alarm means 8 can generate an alarm for prohibiting the driver to look aside.

When the processing of one frame from S1 to S9 is completed, the face recognition predicting means 6 determines whether to perform the next frame processing (S10). When the next frame process is performed, the face recognition predicting unit 6 returns to S1, and thereafter, the process of one frame from S1 to S9 is performed as the next frame process. The face recognition prediction unit 6 ends the process when the next frame process is not performed, for example, when the operation of the face recognition apparatus is stopped from the outside.
Further, the hardware configuration of the face recognition device 10 according to the present embodiment includes a CPU, a memory, a storage device, an infrared light emitting device, and a bus connecting the above-described units.

  In this embodiment, the mask has been described as an example of a face shield. However, the shield is not limited to a mask, and can be widely applied as long as it covers a face. Even in the case of a handkerchief, this embodiment can be applied. In the present embodiment, the face recognition device 10 includes a warning device for a driver with an in-vehicle camera and performs a warning according to the result of face recognition. The present invention can be applied to a wide range of fields that use face recognition, such as monitoring based on accumulation of recognition results and identity authentication based on face recognition results.

[Appendix] The following appendices are further disclosed with respect to the embodiment including the above examples.
(Supplementary Note 1) In a face recognition device that recognizes a person's face based on a captured image of a captured person, a nose that detects a nose region as a region where the captured person's nose exists based on the captured image When the nose region is not detected by the region detection unit and the nose region detection unit, the captured person shields at least a part of the face based on the luminance values in the horizontal and vertical directions of the captured image. A shielding object wearing judging means for judging whether or not the shielding object is worn, and the presence area of the shielding object when the imaged person is judged to be wearing the shielding object by the shielding object wearing judging means. A shielding object region detecting means for detecting the shielding object region, and, based on the shielding object region detected by the shielding object region detection means, the nose region in the shielding object for specifying the nose region included in the shielding object region Inspection And an eye area detecting means for detecting an eye area as a presence area of the captured human eye based on the nose area specified by the nose area detecting means or the nose area detecting means in the shield. Face recognition device.

(Appendix 2) Recognizing the eye opening and / or line-of-sight direction based on the image information of the eye area detected by the eye area detecting means, and recognizing the face direction based on the eye area and / or nose area The face recognition device according to appendix 1, further comprising face recognition means for

(Additional remark 3) Based on the image information of the detected eye area | region, the spectacles frame area | region detection means which detects the spectacles frame area | region as an existing area | region of the spectacles frame | wear worn by the person, and the said spectacles frame detection means detect Sunglasses determining means for determining whether the glasses are sunglasses based on the luminance distribution of the spectacle frame area, and when the eye area detecting means determines that the glasses are sunglasses by the sunglasses determining means The face recognition device according to Supplementary Note 1 or Supplementary Note 2, wherein the eye region in the sunglasses is determined and detected.

(Supplementary Note 4) Based on the horizontal and vertical luminance values of the image information in the detected eye area, eye patch determination means for determining whether a person is wearing the eye patch, and the eye patch is worn by the eye patch determination means If it is determined that the eye patch area is detected, the eye area detection unit detects an eye patch area as an existing area of the eye patch, and the eye area detection unit determines that the eye patch is worn by the eye patch determination unit. If it is, the face recognition device according to appendix 1 to appendix 3, wherein the eye area in the eye patch is determined and detected.

(Supplementary note 5) The face recognition device according to any one of supplementary notes 1 to 4, further comprising face recognition prediction means for predicting a result recognized by the face recognition means after a minute time based on a result recognized by the face recognition means. .

(Appendix 6) In a face recognition method for recognizing a person's face based on a captured image of a captured person, a nose for detecting a nose region as an existing region of the captured person's nose based on the captured image When the nose region is not detected by the region detection step and the nose region detection step, the imaged person shields at least a part of the face based on the luminance values in the horizontal and vertical directions of the captured image. An area where the shielding object exists when it is determined by the shielding object wearing determination step that determines whether or not the imaged person is wearing the shielding object by the shielding object wearing determination step. A shielding object region detecting step for detecting a shielding material region as a shielding object region, and a shielding object inner nose region for identifying the nose region included in the shielding material region based on the shielding material region detected by the shielding material region detection step Inspection And an eye region detection step of detecting an eye region as a region where the imaged person's eye is present based on the nose region specified by the nose region detection step or the shield nose region detection step. Face recognition method.

(Appendix 7) Recognizing the eye opening and / or line-of-sight direction based on the image information of the eye area detected by the eye area detection step, and recognizing the face direction based on the eye area and / or nose area The face recognition method according to appendix 6, further comprising a face recognition step.

(Additional remark 8) Based on the image information of the detected eye area, a spectacle frame area detecting step for detecting a spectacle frame area as an existing area of a spectacle frame worn by a person, and detection by the spectacle frame detecting step A sunglasses determination step of determining whether the glasses are sunglasses based on the luminance distribution of the spectacle frame region, and when the eye region detection step determines that the glasses are sunglasses by the sunglasses determination step The face recognition method according to appendix 6 or appendix 7, wherein the eye area in the sunglasses is determined and detected.

(Supplementary note 9) An eyepatch determination step for determining whether a person is wearing an eyepatch based on horizontal and vertical luminance values of image information in the detected eye region, and the eyepatch is worn by the eyepatch determination step The eye patch region detecting step of detecting the eye patch region as the region where the eye patch is present, and the eye region detecting step determines that the eye patch is worn by the eye patch determining step. The face recognition method according to any one of appendix 6 to appendix 8, wherein the eye region in the eye patch is determined and detected.

(Additional remark 10) The face recognition method of Additional remark 6 thru | or Additional remark 9 provided with the face recognition prediction process which estimates the result recognized by the said face recognition process of micro time after based on the result recognized by the said face recognition process .

(Additional remark 11) In the face recognition program which makes a computer function so that a person's face may be recognized based on the captured image of the imaged person, based on the captured image, as a nose existence area of the imaged person A nose region detecting unit for detecting a nose region; and when the nose region detecting unit detects no nose region, the captured person is at least a face based on luminance values in the horizontal and vertical directions of the captured image. Shield wear judging means for judging whether or not a shield covering a part is worn, and when the imaged person is judged to be wearing the shield by the shield wear judging means, the shielding Shielding object area detecting means for detecting a shielding object area as an object existence area, and based on the shielding object area detected by the shielding object area detecting means, the shielding object area includes the shielding object area. Based on the nose area detection means for identifying the area, the nose area detection means or the nose area detection means for the occlusion object, an eye area as the presence area of the captured human eye is determined. A face recognition program that causes a computer to function as eye area detection means for detection.

(Appendix 12) Recognizing the eye opening and / or line-of-sight direction based on the image information of the eye area detected by the eye area detecting means, and recognizing the face direction based on the eye area and / or nose area 12. A face recognition program according to appendix 11, which causes a computer to function as face recognition means.

(Supplementary note 13) Based on the image information of the detected eye region, a spectacle frame region detecting unit for detecting a spectacle frame region as a region of spectacle frames worn by a person, detected by the spectacle frame detecting unit. When the computer functions as a sunglasses determination unit that determines whether the glasses are sunglasses based on the luminance distribution of the glasses frame region, and the eye region detection unit determines that the glasses are determined to be sunglasses by the sunglasses determination unit Is a face recognition program according to supplementary note 11 or supplementary note 12 for judging and detecting the eye area in the sunglasses.

(Supplementary note 14) Based on the luminance values in the horizontal and vertical directions of the image information in the detected eye region, eye patch determining means for determining whether a person is wearing the eye patch, and the eye patch is being worn by the eye patch determining means If it is determined that the computer is functioning as eye patch area detecting means for detecting an eye patch area as an area where the eye patch is present, the eye area detecting means is wearing the eye patch by the eye patch determining means. 14. The face recognition program according to appendix 11 to appendix 13, wherein if judged, the eye area in the eye patch is judged and detected.

(Supplementary note 15) The supplementary notes 11 to 14, which cause the computer to function as a face recognition prediction unit that predicts a result recognized by the face recognition unit after a minute time based on the result recognized by the face recognition unit. Face recognition program.

The block diagram which shows the structure of the face recognition apparatus which concerns on the 1st Embodiment of this invention. 1 is an overall configuration diagram of a face recognition device according to a first embodiment of the present invention. Entire flowchart of face recognition apparatus according to first embodiment of the present invention Flowchart of face area and face part detection of face recognition device according to first embodiment of the present invention Flowchart of nose region detection of face recognition device according to first embodiment of the present invention Flowchart of mask detection of face recognition apparatus according to first embodiment of the present invention Flowchart of eyeglass bridge area detection of face recognition device according to first embodiment of the present invention Flowchart of eyeglass frame region detection of face recognition device according to first embodiment of the present invention Flowchart of sunglasses area detection of face recognition device according to first embodiment of the present invention Flowchart of sunglasses region comprehensive determination of face recognition device according to first embodiment of the present invention. Flowchart for detecting the position of the eye region of the face recognition device according to the first embodiment of the present invention. Flow chart of eye patch region detection of the face recognition device according to the first embodiment of the present invention. Explanatory drawing of face direction detection of the face recognition apparatus which concerns on the 1st Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Face area detection means 2 Nose area detection means 3 Mask wear determination means 31 Mask area detection means 32 In-mask nose area detection means 4 Eye area detection means 5 Face recognition means 5a Face information storage means 41 Eyeglass frame area detection means 42 Sunglasses determination Means 43 Eyepatch judgment means 44 Eyepatch area detection means 6 Face recognition prediction means 7 Feature judgment means 8 Alarm means 8A Alarm device 10 Face recognition apparatus 100 Imaging means 100A Car-mounted camera 200 Car 300 Driver 300A Face

Claims (7)

In a face recognition device for recognizing a person's face based on a captured image of the person,
Nose region detecting means for detecting a nose region as a nose present region of the imaged person based on the captured image;
When no nose area is detected by the nose area detecting means, it is determined whether or not a shield that shields the nose of the imaged person is worn based on the luminance values in the horizontal and vertical directions of the captured image. Means for determining whether to wear a shield,
When it is determined by the shielding object wearing determination means that the imaged person is wearing a shielding object, a shielding object region detecting unit that detects a shielding object region as the existence region of the shielding object;
On the basis of the shielding object area detected by the shielding object area detection means, the nose area detection means in the shielding object for identifying the nose area included in the shielding object area;
An eye area detecting means for detecting an eye area as an existing area of the person's eye imaged based on the nose area specified by the nose area detecting means or the nose area detecting means in the shield. . The face recognition device according to claim 1,
Face recognition means for recognizing the opening degree and / or line-of-sight direction of the eye based on image information of the eye area detected by the eye area detection means, and recognizing the face direction based on the eye area and / or nose area A face recognition device. The face recognition device according to claim 1 or 2,
A spectacle frame region detecting means for detecting a spectacle frame region as an existing region of a spectacle frame worn by a person based on the image information of the detected eye region;
Sunglasses determining means for determining whether the glasses are sunglasses based on the luminance distribution of the glasses frame region detected by the glasses frame detecting means;
A face recognition device that determines and detects the eye area in the sunglasses when the eye area detection means determines that the glasses are sunglasses by the sunglasses determination means. The face recognition device according to claim 1, wherein
Eye patch determination means for determining whether a person wears an eye patch based on the horizontal and vertical luminance values of the image information in the detected eye region;
When it is determined by the eye patch determining means that the eye patch is worn, the eye patch region detecting means for detecting an eye patch area as an existing area of the eye patch,
A face recognition device that determines and detects the eye region in the eye patch when the eye region detection unit determines that the eye patch is worn by the eye patch determination unit. The face recognition apparatus according to claim 1, wherein:
A face recognition apparatus comprising: a face recognition prediction unit that predicts a result recognized by the face recognition unit after a minute time based on a result recognized by the face recognition unit. In a face recognition method for recognizing a person's face based on a captured image of a captured person,
A nose region detecting step of detecting a nose region as a nose present region of the imaged person based on the captured image;
When the nose region is not detected by the nose region detection step, based on the luminance values in the horizontal and vertical directions of the captured image, the captured person wears a shield that shields at least a part of the face. A shield wearing judgment process for judging whether or not
When it is determined that the imaged person is wearing a shield by the shield wear determination step, a shield region detection step of detecting a shield region as the presence region of the shield,
On the basis of the shielding object area detected by the shielding object area detection process, the shielding nose area detection process for identifying the nose area included in the shielding object area;
An eye area detecting step of detecting an eye area as an existing area of the imaged person's eye based on the nose area specified by the nose area detecting step or the nose area detecting step in the shield.
Face recognition method. In a face recognition program for causing a computer to function to recognize a person's face based on a captured image of the person who is imaged,
Nose region detecting means for detecting a nose region as a nose present region of the imaged person based on the captured image;
When the nose region is not detected by the nose region detection means, the captured person wears a shield that shields at least a part of the face based on the horizontal and vertical luminance values of the captured image. Means for judging whether or not the shield is worn,
A shield area detection means for detecting a shield area as an existence area of the shield when it is determined by the shield wear determination means that the imaged person is wearing a shield;
Based on the shielding object area detected by the shielding object area detection means, the shielding object nose area detection means for specifying the nose area included in the shielding object area,
Based on the nose region specified by the nose region detecting unit or the nose region detecting unit in the shield, the computer functions as an eye region detecting unit that detects an eye region as a region where the captured human eye exists.
Face recognition program.

Images