JP5594015B2 - Image processing program and image processing apparatus - Google Patents

Description

  The technology disclosed in the present application relates to an image processing program and an image processing apparatus.

  2. Description of the Related Art Conventionally, for example, there is a technique for detecting so-called facial parts, which are parts constituting a face such as eyes, nose, and mouth, from image images of faces as image feature points.

  For example, a plurality of candidate points that may be facial parts are extracted from an image obtained by photographing a face, and feature points of the facial parts are detected from the extracted candidate points based on the positional relationship of the facial parts. There is Prior Art 1. In addition, for example, after extracting a plurality of candidate points that may be facial parts from an image of a face, the extracted candidate points are narrowed down to some under a predetermined condition, and then each candidate point that has been narrowed down Among them, there is a conventional technique 2 for detecting a feature point of a facial part based on the positional relationship of the facial part.

JP 2008-3749 A

Kazuhiro Fukui et al., “Face feature point extraction by combination of shape extraction and pattern matching”, IEICE Transactions, Vol. J-80-D-II, no. 8, pp. 2170-2177, August 1997

  However, the above-described prior art 1 and prior art 2 have a problem that a feature point of a facial part may be erroneously detected from an image obtained by photographing a face. Hereinafter, the case of detecting the feature points of the pupil of the eye and the nostril as face parts from the image obtained by photographing the face will be described as an example, and the problems of the prior art 1 and the prior art 2 described above will be described with reference to FIGS. Will be described. 20 and 21 are diagrams used for explaining the problems of the prior art. P1 described in FIGS. 20 and 21 indicates a feature point candidate point corresponding to the pupil of the eye, and P2 described in FIGS. 20 and 21 indicates a feature point candidate point corresponding to the nostril.

  In the prior art 1 described above, first, as shown by A1 in FIG. 20, a plurality of candidate points P1 that may be a pupil and a plurality of candidate points P2 that may be a nostril from an image obtained by photographing a face. To extract. Subsequently, as shown in A2 of FIG. 20, the related art 1 evaluates the positional relationship for the plurality of candidate points P1 and the plurality of candidate points P2 based on the positional relationship as the face part. And as shown in A3 of FIG. 20, the prior art 1 finally detects the feature point of a pupil and a nostril from the image which image | photographed the face. As described above, in the related art 1, if a candidate point is likely as the positional relationship of the facial part, even a candidate point corresponding to a part other than the facial part is detected as a feature point of the facial part. For this reason, the feature point of the face part may be detected erroneously.

  In the related art 2 described above, first, as shown in B1 of FIG. 21, a plurality of candidate points P1 that may be pupils and a plurality of possibilities that are nostrils from the image obtained by photographing the face. Candidate point P2 is extracted. Subsequently, as shown in B2 of FIG. 21, the prior art 2 narrows down the plurality of candidate points P1 and the plurality of candidate points P2 extracted from the face image based on a predetermined condition, and then uses them as face parts. Based on the positional relationship, the positional relationship between the plurality of candidate points P1 and the plurality of candidate points P2 is evaluated. And as shown in B3 of FIG. 21, the prior art 2 finally detects the feature point of a pupil and a nostril from the image which image | photographed the face. As described above, in the prior art 2, when candidate points for facial parts are narrowed down, candidate points corresponding to facial parts may be lost as shown in B2 of FIG. 21 described above. For this reason, the feature point of the face part may be detected erroneously.

  The disclosed technology has been made in view of the above, and an image processing program and an image that can reduce the possibility of erroneously detecting a feature point of a facial part from an image obtained by photographing a face An object is to provide a processing apparatus.

  In one aspect, an image processing program disclosed in the present application extracts, to a computer, a plurality of facial feature point candidates included in an image obtained by photographing a face, and the plurality of extracted facial feature point candidates are stored in the computer program. A process of acquiring a likelihood value indicating the likelihood that is a feature point is executed. Further, the computer is caused to execute a process of evaluating the distance between the plurality of extracted facial feature point candidates based on the reference value and the likelihood value for evaluating the distance between the feature points extracted as candidates. . Further, the computer is caused to execute a process of detecting the feature points according to the result of the evaluation relating to the distance between the facial feature point candidates.

  According to one aspect of the technology disclosed in the present application, it is possible to reduce a possibility that a feature point of a face part is erroneously detected from an image obtained by photographing a face.

FIG. 1 is a functional block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a conceptual diagram used for explaining the information stored in the distance information storage unit according to the first embodiment. FIG. 3 is a diagram illustrating an example of information stored in the distance information storage unit according to the first embodiment. FIG. 4 is a diagram used for explaining the acquisition unit according to the first embodiment. FIG. 5 is a diagram used for explaining the acquisition unit according to the first embodiment. FIG. 6 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 7 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 8 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 9 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 10 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 11 is a diagram used for explaining the evaluation unit according to the first embodiment. FIG. 12 is a diagram illustrating a process flow of the image processing apparatus according to the first embodiment. FIG. 13 is a functional block diagram illustrating the configuration of the image processing apparatus according to the second embodiment. FIG. 14 is a diagram illustrating an example of information stored in the importance storage unit according to the second embodiment. FIG. 15 is a diagram used for explaining the likelihood adjusting unit according to the second embodiment. FIG. 16 is a diagram illustrating the flow of processing by the image processing apparatus according to the second embodiment. FIG. 17 is a flowchart illustrating a likelihood adjustment process according to the second embodiment. FIG. 18 is a flowchart illustrating the likelihood adjustment process according to the third embodiment. FIG. 19 is a diagram illustrating an example of an electronic device that executes an image processing program. FIG. 20 is a diagram used for explaining the problems of the prior art. FIG. 21 is a diagram used for explaining the problems of the prior art.

  Hereinafter, embodiments of an image processing program and an image processing apparatus disclosed in the present application will be described in detail with reference to the drawings. The technology disclosed in the present application is not limited by the examples described later as an embodiment of the image processing program and the image processing apparatus disclosed in the present application. In the following embodiments, an image in which a human face is reflected in an image is called a face image, and eyes and nose included in the face are called facial parts. In the following examples, an embodiment will be described in which, for example, the positions of the pupils and nostrils of the eyes shown in the face image are detected as feature points from the face image. In addition, the techniques disclosed in the following embodiments can be appropriately combined within a range that does not cause a contradiction in processing contents.

[Configuration of Image Processing Apparatus (Example 1)]
FIG. 1 is a functional block diagram illustrating the configuration of the image processing apparatus according to the first embodiment. FIG. 1 illustrates functional blocks necessary for explaining the functions of the image processing apparatus according to the first embodiment. As illustrated in FIG. 1, the image processing apparatus 200 according to the first embodiment is connected to, for example, a photographing apparatus 100.

  The imaging device 100 corresponds to, for example, a monocular camera or a stereo camera. The imaging device 100 captures and acquires a face image, for example.

  As illustrated in FIG. 2, the image processing apparatus 200 includes an input unit 210, a storage unit 220, and a control unit 230.

  The input unit 210 is a camera interface, for example, and inputs a face image acquired by the photographing apparatus 100. The input unit 210 stores the face image input from the photographing apparatus 100 in the image storage unit 221.

  As illustrated in FIG. 1, the storage unit 220 includes an image storage unit 221 and a distance information storage unit 222. The storage unit 220 is a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory.

  For example, the image storage unit 221 stores the face image input by the input unit 210.

  The distance information storage unit 222 is a distance information for evaluating the distance between candidate feature points displayed as face part candidates in the face image when a face image is shot under a predetermined shooting condition, in other words, the face Information on the correct positional relationship of the facial parts projected in the image is stored. Hereinafter, distance information stored in the distance information storage unit 222 will be described with reference to FIGS. 2 and 3. FIG. 2 is a conceptual diagram used for explaining the information stored in the distance information storage unit according to the first embodiment. FIG. 3 is a diagram illustrating an example of information stored in the distance information storage unit according to the first embodiment.

  Α in FIG. 2 is a model showing the positional relationship between the pupil of the eye and the nostril in the face image. As shown in FIG. 2, the model α is composed of pupil models E1 and E2, nostril models N1 and N2, pupil links E1 and nostril model N1, and a pupil. A link connecting the model E2 and the nostril model N2. If the distance information described above is conceptually described, the length of the link connecting the pupil and the nostril, that is, the distance between E1 and N1 and the distance between E2 and N2 shown in FIG. 2, in other words, This corresponds to a distance corresponding to the number of pixels on the image. In the example shown in FIG. 2, the distance between E1 and N1 and the distance between E2 and N2 are “1.2”, respectively.

  Therefore, the distance information storage unit 222, as shown in FIG. 3, for the above-mentioned model α, the distance between the pupil model E1 and the nostril model N1, and the pupil model E2 and the nostril model N2 Is stored as distance information. It should be noted that the distance “1.2” between the pupil and the nostril for the model α stored in the distance information storage unit 222 is the distance between the pupils, that is, the distance between E1 and E2 is “1”. This is a value obtained by normalizing the distance between N2 and N1 and the distance between E2 and N2. The distance information storage unit 222 may not have the normalized distance between the pupil and the nostril. For example, the distance information storage unit 222 may store the number of pixels as the distance between the pupil and the nostril, for example.

  In addition, since the first embodiment describes an embodiment in which the positions of the pupils and nostrils of the eyes displayed in the face image are detected as feature points, the distance information storage unit 222 includes the pupils. The distance information between the nostril and the nostril is stored. However, when face parts other than the pupils and nostrils of eyes, for example, mouth (mouth corners) and ears (ear holes) are also detected, distance information between the face parts to be detected is stored. Will be.

  The distance information stored in the distance information storage unit 222 calculates, for example, the distance between the pupil and the nostril for each of a plurality of learning face images, and calculates an average distance based on each calculated distance. It can be obtained by deriving. The distance information described above is an example of a reference value.

  Returning to FIG. 1, the control unit 230 includes a face detection unit 231, an acquisition unit 232, an evaluation unit 233, and a detection unit 234. Note that the control unit 230 corresponds to, for example, an electronic circuit or an integrated circuit. Examples of the electronic circuit include a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). Examples of integrated circuits include ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array).

  When an image is input from the input unit 210, the face detection unit 231 acquires the input image from the image storage unit 221, and detects a face from the acquired image. For example, the face detection unit 231 attempts to detect a face from an image using existing techniques such as edge extraction, pattern matching, inter-frame difference, and a neural network.

  When the face detection unit 231 detects a face from the image, the acquisition unit 232 extracts a plurality of pupil and nostril candidates included in the image, and extracts a plurality of extracted pupil and nostril candidates. A likelihood value indicating the likelihood that is a pupil or nostril is acquired. Hereinafter, the acquisition unit will be described with reference to FIGS. 4 and 5. 4 and 5 are diagrams used for explaining the acquisition unit according to the first embodiment. E3 and E4 described in FIGS. 4 and 5 indicate candidate feature points extracted from the images as pupil candidates. N3 and N4 described in FIGS. 4 and 5 indicate candidate feature points extracted as nostril candidates from the image.

  For example, the acquisition unit 232 uses a separability filter to extract a plurality of candidate feature points that are pupil candidates from the image, such as E3 and E4 illustrated in FIG. Also, the acquisition unit 232 extracts a plurality of candidate feature points that are nostril candidates from the image, for example, N3 and N4 illustrated in FIG. Note that the acquisition unit 232 may extract candidate feature points using a method different from the method using the separability filter, such as pattern matching. Then, the acquisition unit 232 acquires the response value of the separability filter obtained when extracting pupil and nose hole candidates from the image as a likelihood value.

  And the acquisition part 232 stores the data regarding the candidate feature point acquired from the image in memory | storage parts, such as RAM which has internally. For example, as illustrated in FIG. 5, the acquisition unit 232 stores, in the storage unit, the identification mark of the candidate feature point, the coordinates on the image of the candidate feature point, and the likelihood value for each type of face part. For example, as illustrated in FIG. 5, the acquisition unit 232 determines the candidate feature point identification symbol “E3”, the candidate feature point coordinates “(X1, Y1)” and the likelihood for the face part type “pupil”. The degree value “V1” is associated and stored in the storage unit. Further, as illustrated in FIG. 5, the acquisition unit 232 identifies the candidate feature point identification symbol “E4”, the candidate feature point coordinates “(X2, Y2)” and the likelihood for the face part type “pupil”. The degree value “V2” is associated and stored in the storage unit. In addition, as illustrated in FIG. 5, the acquisition unit 232 identifies the candidate feature point identification symbol “N3” and the candidate feature point coordinates “(X3, Y3)” for the face part type “nose hole”. And likelihood value "V3" is matched and stored in a memory | storage part. In addition, as illustrated in FIG. 5, the acquisition unit 232 identifies the candidate feature point identification symbol “N4” and the candidate feature point coordinates “(X4, Y4)” for the face part type “nose hole”. And likelihood value "V4" is matched and stored in a memory | storage part.

  Returning to FIG. 1, the evaluation unit 233 determines whether the plurality of candidate feature points extracted based on the distance information stored in the distance information storage unit 222 and the likelihood value of the candidate feature points acquired by the acquisition unit 232. Evaluate the distance. Hereinafter, the evaluation unit will be described with reference to FIGS. 6 to 11 are diagrams used for explaining the evaluation unit according to the first embodiment.

  As illustrated in FIG. 6, the evaluation unit 233 creates a combination of candidate feature points extracted by the acquisition unit 232. For example, the evaluation unit 233 creates all combinations of two candidate feature points extracted as pupil candidates and two candidate feature points extracted as nostril candidates. When the distance information stored in the distance information storage unit 222 is normalized with respect to the distance between the pupils, the evaluation unit 233 determines that the distance between the pupils is the distance at the stage of creating the combination. Combinations that are not equivalent to information can be avoided. In this way, it is possible to reduce the subsequent processing load.

  Subsequently, the evaluation unit 233 acquires one combination from the created combinations of candidate feature points, and calculates a distance between the candidate feature points of the acquired combination. For example, the evaluation unit 233 acquires a combination from the combinations of candidate feature points, and calculates the distance between the acquired candidate feature point of the pupil and the candidate feature point of the nostril. For example, in the example illustrated in FIG. 7, the evaluation unit 233 obtains the combination β of candidate feature points, and determines the distance “1.5” between the candidate feature point E3 of the pupil and the candidate feature point N3 of the nostril, The distance “1.0” between the candidate feature point E4 and the nose hole candidate feature point N4 is calculated.

  Subsequently, the evaluation unit 233 determines between the candidate feature points of the combination β described above based on the distance information stored in the distance information storage unit 222 and the likelihood value of the candidate feature points acquired by the acquisition unit 232. Evaluate the distance. First, the evaluation unit 233 acquires the distance information from the distance information storage unit 222 and acquires the likelihood value of each candidate feature point of the combination β from the acquisition unit 232. Subsequently, the evaluation unit 233 collates the distance between the candidate feature points of the combination β and the distance information, and calculates the difference between the distance between the candidate feature points of the combination β and the distance information. For example, as illustrated in FIG. 8, the evaluation unit 233 compares the distance between the candidate feature points of the combination β and the distance information, and determines the distance between “E3 and N3” and the distance between “E1 and N1”. The difference “0.3” is calculated. Further, as illustrated in FIG. 8, the evaluation unit 233 calculates a difference “0.2” between the distances “between E4 and N4” and “between E1 and N1”.

  Subsequently, the evaluation unit 233 adjusts the difference between the distance between the candidate feature points of the combination β and the distance information according to the likelihood value of each candidate feature point of the combination β. For example, the evaluation unit 233 calculates the sum of the likelihood values of the candidate feature point E3 and the candidate feature point N3 of the combination β, and the sum of the likelihood values of the candidate feature point E4 and the candidate feature point N4 of the combination β. Then, the evaluation unit 233 multiplies the difference between the distance between each candidate feature point of the combination β and the distance information by a coefficient C corresponding to the sum of the likelihood values, so that the distance between the candidate feature points and the distance information. Adjust the difference with.

  For example, as shown in FIG. 9, the value of the coefficient C becomes a maximum value (MAX) when the sum of likelihood values is “0”, and decreases as the sum of likelihood values increases. The sum of the values is a certain minimum value (MIN) above a certain value. This coefficient C adjusts the distance between candidate feature points having a high likelihood value so that the difference from the distance information stored in the distance information storage unit 222 is small. That is, by using the coefficient C, the evaluation unit 233 evaluates a candidate feature point that is likely to be a facial part as having a high possibility of having a correct positional relationship between the facial part, that is, the pupil and the nostril.

  For example, in the example illustrated in FIG. 10, the evaluation unit 233 adjusts the difference between the distance between the candidate feature points of the combination β and the distance information, and as a result, the distance between “E3 and N3” and “between E1 and N1”. The distance difference of “0.1” is the distance difference “0.3” between “between E4 and N4” and “between E1 and N1”. Subsequently, the evaluation unit 233 calculates the sum of the distance differences between the candidate feature points. As the sum of the distance differences is smaller, for example, the positional relationship between the candidate point of the pupil extracted from the image and the candidate point of the nostril, for example, the combination β shown in FIG. For example, it shows that it is closer to the model α shown in FIG.

  Then, the evaluation unit 233 stores the processing result for the combination β in a storage unit such as an internal RAM. For example, as illustrated in FIG. 11, for each combination of candidate feature points, the evaluation unit 233 determines the combination identification symbol, the identification symbol of the candidate feature point included in the combination, the coordinates of the candidate feature point on the image, and the candidate feature point. The sum of the difference in distance between them is associated and stored in the storage unit. For example, as illustrated in FIG. 11, the evaluation unit 233 associates the identification symbol “E3” of the candidate feature point included in the combination with the coordinate “β” on the image of the candidate feature point in association with the identification symbol “β” of the combination. (X1, Y1) "is stored in the storage unit. Similarly, as illustrated in FIG. 11, the evaluation unit 233 associates the identification symbol “E4” of the candidate feature point included in the combination with the identification symbol “β” of the combination and the coordinates on the image of the candidate feature point. “(X2, Y2)” is stored in the storage unit. Similarly, as shown in FIG. 11, the evaluation unit 233 associates the identification symbol “N3” of the candidate feature point included in the combination with the identification symbol “β” of the combination, and the coordinates of the candidate feature point on the image. “(X3, Y3)” is stored in the storage unit. Similarly, as illustrated in FIG. 11, the evaluation unit 233 associates the identification symbol “N4” of the candidate feature point included in the combination with the identification symbol “β” of the combination, and the coordinates of the candidate feature point on the image. “(X4, Y4)” is stored in the storage unit. Further, as illustrated in FIG. 11, the evaluation unit 233 stores the sum “0.4” of the distance difference between the candidate feature points in the storage unit in association with the combination identification symbol “β”.

  The evaluation unit 233 determines whether or not the processing described above has been completed for all of the combinations of candidate feature points that have been created. If the processing has not been completed for all of the combinations, a combination is selected from the remaining combinations. Is acquired, and the above-described processing is executed. In this way, the evaluation unit 233 performs the above-described processing for all combinations of candidate feature points, and creates, for example, data as shown in FIG. 11, so that candidate feature points for each combination of candidate feature points. The sum of the distance differences between them and the coordinates of the candidate feature points are known.

  Returning to FIG. 1, the detection unit 234 acquires the position of the feature point corresponding to the pupil or nostril according to the evaluation result of the combination of the candidate feature points by the evaluation unit 233. For example, the detection unit 234 acquires an evaluation result by the evaluation unit 233, and specifies a record having a minimum sum of distance differences between candidate feature points. When a record having the smallest sum of distance differences between candidate feature points is specified, the coordinates of each candidate feature point of the record are acquired.

  When the image processing apparatus 200 shown in FIG. 1 is mounted on a mobile terminal such as a mobile phone, for example, the coordinate value of the feature point acquired by the detection unit 234 is, for example, a corneal reflection method. It is used for processing to detect the user's line of sight. Further, when the image processing apparatus 200 shown in FIG. 1 is mounted on, for example, an ECU (Electronic Control Unit) mounted on a vehicle, the coordinate value of the feature point acquired by the detection unit 234 is, for example, a corneal reflection method or the like Is also used for processing for detecting the line of sight of the driver of the vehicle. For example, a method for detecting the line of sight using the coordinate value of the feature point acquired by the detection unit 234 will be described. First, the pupil center position is determined using the X and Y coordinates of the pupil acquired by the detection unit 234. Ask. Subsequently, the obtained center position of the pupil and the corneal reflection point are tracked, and the line of sight is detected based on the difference between the center of curvature of the cornea and the center of rotation of the eyeball. Further, the coordinate values of the feature points of the image acquired by the detection unit 234 are used when a processing target region is cut out from the image during authentication processing for authenticating an individual based on a face image. For example, it can be used when a processing target region including a pupil and a nostril is cut out.

[Processing by Image Processing Apparatus (Example 1)]
Subsequently, the flow of processing performed by the image processing apparatus according to the first embodiment will be described with reference to FIG. FIG. 12 is a diagram illustrating a process flow of the image processing apparatus according to the first embodiment. The process shown in FIG. 12 is executed every time an image frame is input.

  As shown in FIG. 12, for example, when the controller 230 detects an input of an image by the input unit 210 (step S <b> 101, YES), the face detection unit 231 acquires the input image from the image storage unit 221. Then, a face is detected from the acquired image (step 102). In addition, when the input of the image by the input part 210 is not detected (step S101, NO), the control part 230 determines repeatedly the input of the image by the input part 210. FIG.

  If no face is detected from the image by the face detection unit 231 (step S102, NO), the control unit 230 ends the process and waits for the input of the next image. When the face is detected from the image by the face detection unit 231 (step S102, YES), the acquisition unit 232 extracts a plurality of candidate feature points for pupils and nostrils included in the image (step S103). In step S103, for example, the acquisition unit 232 acquires, as the likelihood value, the response value of the classification filter obtained when extracting candidates for pupils and nostrils from the image.

  Subsequently, the evaluation unit 233 creates a combination of candidate feature points extracted by the obtaining unit 232 (step S104), and acquires one combination from the created candidate feature point combinations (step S105). Subsequently, the evaluation unit 233 calculates the distance between the candidate feature points of the combination acquired in step S105 (step S106). Subsequently, the evaluation unit 233, based on the distance information stored in the distance information storage unit 222 and the likelihood value of the candidate feature point acquired by the acquisition unit 232, the candidate feature of the combination acquired in step S105. The distance between the points is evaluated (step S107). That is, in step S107, adjustment is performed so that the difference between the distance between candidate feature points having a high likelihood value and the distance information stored in the distance information storage unit 222 is small. And the evaluation part 233 stores an evaluation result (step S108), and determines whether the process was completed about all the combinations created in step S104 (step S109).

  If the result of determination is that processing has not been completed (step S109, NO), the evaluation unit 233 returns to step S105 described above and executes processing from step S105 to step S109. On the other hand, when the evaluation unit 233 determines that the processing has been completed (step S109, YES), the detection unit 234 detects a feature point of the image based on the evaluation result by the evaluation unit 233 ( Step S110). In other words, the detection unit 234 acquires the coordinates of the candidate feature points with the smallest sum of the distance differences between the candidate feature points.

  In the example illustrated in FIG. 12, the process of detecting a face from an image is performed as illustrated in step S102. However, when only an image obtained by capturing a face is input, the process of step S102 is omitted. You can also.

[Effects of Example 1]
As described above, the image processing apparatus according to the first embodiment extracts a plurality of facial feature candidate feature points included in a facial image, and acquires a likelihood value indicating the likelihood of being a facial component. Then, the image processing apparatus according to the first embodiment evaluates the distance between the candidate feature points based on the distance information for evaluating the distance between the candidate feature points and the likelihood value of the candidate feature points. Then, the image processing apparatus according to the first embodiment detects a feature point included as a face part in the face image according to the evaluation result regarding the distance between the candidate feature points. In other words, the distance between the candidate feature points is adjusted according to the likelihood value of the candidate feature points, and the candidate feature points where the difference between the distance between the candidate feature points and the distance information stored in the distance information storage unit 222 is as small as possible The feature point is detected as having a high possibility of a face part. For this reason, according to the first embodiment, it is possible to reduce the possibility that a feature point of a face part is erroneously detected from an image obtained by photographing a face.

  In the first embodiment described above, for example, the importance may be set in advance for each type of facial part corresponding to the characteristics of the input image, and the likelihood value of the candidate feature point may be adjusted using this importance. .

[Configuration of Image Processing Apparatus (Example 2)]
FIG. 13 is a functional block diagram illustrating the configuration of the image processing apparatus according to the second embodiment. The image processing apparatus according to the second embodiment is different from the first embodiment in that it includes an importance degree storage unit 223 and a likelihood adjustment unit 235.

  The importance level storage unit 223 stores the importance level set in advance for each type of face part corresponding to the characteristics of the input image. FIG. 14 is a diagram illustrating an example of information stored in the importance storage unit according to the second embodiment. As illustrated in FIG. 14, the importance storage unit 223 stores the importance set in advance for each type of facial part, and for example, corresponds to the importance “1.5” corresponding to the pupil and the nostril. The importance “0.8” is stored. Here, the characteristics of the input image include, for example, shooting from the direction facing the face, shooting from the lower side to the upper side of the face, and direction from the upper side to the lower side of the face. It means what kind of shooting direction, such as shooting, is taken. For example, in an image photographed from the direction facing the face, there is a high possibility that the pupil is photographed more clearly than the nostril. Therefore, the importance of the pupil is set higher than that of the nostril. Alternatively, in an image taken in the direction from the lower side to the upper side of the face, there is a high possibility that the nostril will be taken more clearly than the pupil, so the importance of the nostril is set higher than the pupil .

  The likelihood adjustment unit 235 adjusts the likelihood value of the candidate feature point acquired by the acquisition unit 232 using the importance stored in the importance storage unit 223. FIG. 15 is a diagram used for explaining the likelihood adjusting unit according to the second embodiment. For example, the likelihood adjustment unit 235 multiplies the likelihood value of the candidate feature point of the pupil by the importance “1.5” stored in the importance storage unit 223, as shown in FIG. Thus, the likelihood value of the candidate feature point of the pupil is adjusted. Further, the likelihood adjustment unit 235 multiplies the likelihood value of the candidate feature point of the nostril by the importance “0.8” stored in the importance storage unit 223, as shown in FIG. By doing so, the likelihood value of the candidate feature point of the nostril is adjusted.

[Processing by Image Processing Device (Example 2)]
Subsequently, the flow of processing by the image processing apparatus according to the second embodiment will be described with reference to FIGS. 16 and 17. FIG. 16 is a diagram illustrating the flow of processing by the image processing apparatus according to the second embodiment. FIG. 17 is a flowchart illustrating a likelihood adjustment process according to the second embodiment.

  First, the flow of processing by the image processing apparatus according to the second embodiment will be described with reference to FIG. The process of step S204 shown in FIG. 16 is different from the process by the image processing apparatus according to the first embodiment, that is, the process shown in FIG.

  When a plurality of candidate feature points for pupils and nostrils are extracted by the acquisition unit 232 (step S203), the likelihood adjustment unit 235 uses the importance stored in the importance storage unit 223 to select candidate feature points. A likelihood adjustment process for adjusting the likelihood value is executed (step S204). Hereinafter, the processing from step S205 to step S211 is the same as the processing from step S104 to step S110 shown in FIG. 12 described above.

  That is, the evaluation unit 233 creates a combination of candidate feature points (step S205) and acquires one combination (step S206). Subsequently, the evaluation unit 233 calculates the distance between the candidate feature points of the combination acquired in step S206 (step S207). Subsequently, the evaluation unit 233 determines the distance between the candidate feature points of the combination acquired in step S206 based on the distance information stored in the distance information storage unit 222 and the likelihood value adjusted in step S204 described above. Is evaluated (step S208). And the evaluation part 233 stores an evaluation result (step S209), and determines whether the process was completed about all the combinations created in step S205 (step S210). If the result of determination is that processing has not been completed (step S210, NO), processing returns from step S206 to processing from step S206 to step S210. On the other hand, when it is determined that the processing has been completed (step S210, YES), the detection unit 234 detects a feature point of the image based on the evaluation result by the evaluation unit 233 (step S211).

  Next, the likelihood adjustment process according to the second embodiment will be described with reference to FIG. As shown in FIG. 17, the likelihood adjustment unit 235 reads the importance for each face part from the importance storage unit 223 (step S301). Then, the likelihood adjustment unit 235 adjusts the likelihood value of each candidate feature point acquired by the acquisition unit 232 using the importance read from the importance storage unit 223 (step S302).

[Effects of Example 2]
As described above, the image processing apparatus according to the second embodiment presets the importance for each type of face part, and adjusts the likelihood value of the candidate feature point using this importance. Then, the image processing apparatus according to the second embodiment sets the importance level according to, for example, in what shooting direction the image that has been input is input. For this reason, according to the second embodiment, the likelihood value of the candidate feature point can be adjusted while taking into consideration that the facial parts that are more clearly projected in the image differ depending on the shooting direction.

  In the above-described second embodiment, for example, a case has been described in which the importance for adjusting the likelihood value of the candidate feature point is set in advance for each type of face part corresponding to the characteristics of the input image. For example, the importance may be dynamically adjusted according to the state of the facial part included in the input image.

[Configuration of Image Processing Apparatus (Example 3)]
The image processing apparatus according to the third embodiment has basically the same configuration as the image processing apparatus according to the second embodiment, but the processing function of the likelihood adjusting unit 235 described below is different.

  The likelihood adjustment unit 235 reads the importance for each face part stored in the importance storage unit 223 as an initial value. Subsequently, the likelihood adjustment unit 235 cuts out a region of a predetermined size including the candidate feature point of the pupil from the input image, and acquires the luminance value of each pixel included in the cut out region.

  Subsequently, the likelihood adjustment unit 235 determines whether or not the input image is a photograph of a state where the eyes are closed based on the acquired luminance value of each pixel. For example, the likelihood adjustment unit 235 creates a histogram of luminance values, and determines that the eyes are closed when the luminance value corresponding to the skin color has a peak frequency.

  As a result of the determination, the likelihood adjustment unit 235 changes the initial value of the importance of the pupil to a low value when the input image is a photographed state in which the eyes are closed. The likelihood value of the candidate feature point is adjusted. On the other hand, the likelihood adjustment unit 235 determines the importance of the pupil when the input image is not obtained by photographing the closed state of the eyes, that is, by photographing the opened state. The initial value is changed to a high value, and the likelihood value of the candidate feature point of the pupil is adjusted using this importance. When the input image is an image of an opened image, the likelihood adjustment unit 235 uses the importance read as the initial value from the importance storage unit 223 as it is, and uses the candidate feature points of the pupil as they are. The likelihood value may be adjusted.

  The likelihood adjustment unit 235 may dynamically adjust the importance of the pupil by changing the interval between the upper eyelid and the lower eyelid in the input image. For example, the likelihood adjusting unit 235 detects the edges of each of several frames of images input in time series, and extracts the eye contour. Then, the likelihood adjustment unit 235 adjusts the importance level to be lower if the current input frame is narrower than the previous frame. Note that the value for determining whether or not the eyes are open and the value indicating the interval between the upper eyelid and the lower eyelid are examples of feature amounts.

[Processing by Image Processing Device (Example 3)]
Subsequently, the flow of the likelihood adjustment process according to the third embodiment will be described with reference to FIG. FIG. 18 is a flowchart illustrating the likelihood adjustment process according to the third embodiment. As shown in FIG. 18, the likelihood adjusting unit 235 reads the importance for each face part stored in the importance storage unit 223 as an initial value (step S401). Subsequently, the likelihood adjustment unit 235 cuts out a region of a predetermined size including the candidate feature point of the pupil from the input image, and acquires the luminance value of each pixel included in the cut-out region (step S402). Subsequently, the likelihood adjusting unit 235 determines whether or not the input image is a photograph of a state where the eyes are closed based on the acquired luminance value of each pixel (step S403).

  If the likelihood adjustment unit 235 determines that the input image is taken with the eyes closed (step S403, YES), the likelihood adjustment unit 235 performs the following process. That is, the likelihood adjustment unit 235 adjusts the initial value of the importance of the pupil to a low value, adjusts the likelihood value of the candidate feature point of the pupil using this importance (Step S404), and ends the process. On the other hand, when the likelihood adjustment unit 235 determines that the input image is not a photograph of a state where the eyes are closed (step S403, NO), the likelihood adjustment unit 235 performs the following process. That is, the likelihood adjustment unit 235 adjusts the initial value of the importance of the pupil to a high value, adjusts the likelihood value of the candidate feature point of the pupil using this importance (step S405), and ends the process.

[Effects of Example 3]
As described above, the image processing apparatus according to the third embodiment dynamically adjusts the importance according to the state of the facial part included in the input image. For this reason, according to the third embodiment, for example, even if an attempt is made to detect a feature point corresponding to a pupil from an image, a candidate cannot be detected depending on a situation in which a facial part cannot be detected well, such as when the eyes are closed and cannot be detected well. The likelihood value of the feature point can be adjusted.

  Hereinafter, other embodiments of the image processing program and the image processing apparatus disclosed in the present application will be described.

(1) Device Configuration, etc. For example, the functional block configuration of the image processing apparatus 200 shown in FIG. 1 is conceptual, and does not necessarily need to be physically configured as illustrated. For example, the face detection unit 231 and the acquisition unit 232 illustrated in FIG. 1 may be integrated functionally or physically. Similarly, the evaluation unit 233 and the detection unit 234 illustrated in FIG. 1 may be functionally or physically integrated. As described above, all or a part of the functional blocks of the image processing apparatus 200 can be configured to be functionally or physically distributed / integrated in arbitrary units according to various loads or usage conditions.

(2) Image processing program The various processes executed by the image processing apparatus 200 described in the first to third embodiments are, for example, a microcomputer mounted in an ECU (Electronic Control Unit) mounted on the vehicle. It can also be realized by executing a predetermined program on an electronic device such as the above. In the following, an example of an electronic device that executes an image processing program that realizes the same function as the processing executed by the image processing apparatus 200 described in the first to third embodiments will be described with reference to FIG. FIG. 19 is a diagram illustrating an example of an electronic device that executes an image processing program.

  As illustrated in FIG. 19, an electronic device 300 that implements various processes executed by the image processing apparatus 200 includes a CPU (Central Processing Unit) 310 that executes various arithmetic processes. As shown in FIG. 19, the electronic device 300 includes a camera interface 320 for acquiring a camera image and a display interface 330 for exchanging various data with the display. Further, as illustrated in FIG. 19, the electronic device 300 includes a graphic engine 340 that functions as a hardware accelerator.

  As shown in FIG. 19, the electronic device 300 includes a hard disk device 350 that stores programs and data for realizing various processes by the CPU 310, and a memory such as a RAM (Random Access Memory) that temporarily stores various information. 360. Each device 310 to 360 is connected to a bus 370.

  Instead of the CPU 310, for example, an electronic circuit such as an MPU (Micro Processing Unit) or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) can be used. Further, instead of the memory 360, a semiconductor memory device such as a flash memory can be used.

  The hard disk device 350 stores an image processing program 351 and image processing data 352 that exhibit functions similar to those of the image processing device 200. Note that the image processing program 351 can be appropriately distributed and stored in a storage unit of another computer that is communicably connected via a network.

  Then, the CPU 310 reads the image processing program 351 from the hard disk device 350 and develops it in the memory 360, whereby the image processing program 351 functions as an image processing process 361 as shown in FIG. The image processing process 361 expands various data such as the image processing data 352 read from the hard disk device 350 in an area allocated to itself on the memory 360 as appropriate, and executes various processes based on the expanded data. .

  Note that the image processing process 361 includes, for example, processing executed by the control unit 230 of the image processing apparatus 200 described in the first to third embodiments, for example, processing illustrated in FIGS. 12 and 16 to 18.

  Note that the image processing program 351 is not necessarily stored in the hard disk device 350 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card that can connect a corresponding drive to the ECU on which the electronic device 300 is mounted. Let me. The electronic device 300 may read out and execute each program from these.

  Further, each program is stored in “another computer (or server)” connected to the ECU on which the electronic device 300 is mounted via a public line, the Internet, a LAN, a WAN, or the like. The electronic device 300 may read out and execute each program from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1)
Extracting a plurality of facial feature point candidates included in the image of the face, and obtaining a likelihood value indicating the likelihood of the feature point for each of the extracted facial feature point candidates;
Based on the reference value for evaluating the distance between feature points extracted as a plurality of candidates and the likelihood value corresponding to each of the feature point candidates, between the feature point candidates of the plurality of extracted face points Evaluate the distance of
An image processing program for executing a process of detecting the feature points from the plurality of extracted facial feature point candidates according to a result of evaluating the distance.

(Supplementary note 2)
Further executing a process of adjusting the likelihood value using the importance according to the feature point to be detected;
The image processing program according to appendix 1, wherein a distance between the plurality of extracted facial feature point candidates is evaluated based on the reference value and the adjusted likelihood value.

(Additional remark 3) The image processing program of Additional remark 2 which further performs the process which changes the said importance according to the feature-value obtained from the said input image.

(Additional remark 4) The process which detects the said feature point detects the feature point with the highest evaluation result regarding the distance between the said feature point candidates,
In the computer,
The processing according to claim 1, further comprising: executing a process of acquiring a position of the detected feature point on the image and detecting a line of sight of the face photographed in the image using the acquired position. Image processing program.

(Supplementary Note 5) A plurality of facial feature point candidates included in an image obtained by photographing a face, and a likelihood value indicating the likelihood of the feature point for each of the extracted facial feature point candidates An acquisition unit for acquiring
Based on the reference value for evaluating the distance between feature points extracted as a plurality of candidates and the likelihood value corresponding to each of the feature point candidates, between the feature point candidates of the plurality of extracted face points An evaluation unit for evaluating the distance of
An image processing apparatus comprising: a detection unit configured to detect the feature points from the plurality of extracted facial feature point candidates according to the evaluation result of the distance.

(Additional remark 6) It further has a likelihood adjustment part which adjusts the said likelihood value using the importance according to the said feature point made into a detection target,
6. The image processing apparatus according to appendix 5, wherein a distance between the plurality of extracted facial feature point candidates is evaluated based on the reference value and the adjusted likelihood value.

(Supplementary note 7) The image processing apparatus according to supplementary note 6, further comprising a changing unit that changes the degree of importance according to a feature amount obtained from the input image.

(Additional remark 8) The said detection part detects the feature point with the highest evaluation result regarding the distance between the candidate of the said feature point,
The supplementary note 5 further comprising: a gaze detection unit that acquires a position of the detected feature point on the image and detects a gaze of the face photographed in the image using the acquired position. Image processing apparatus.

DESCRIPTION OF SYMBOLS 100 Image capturing apparatus 200 Image processing apparatus 210 Input unit 220 Storage unit 221 Image storage unit 222 Distance information storage unit 223 Importance storage unit 230 Control unit 231 Face detection unit 232 Acquisition unit 233 Evaluation unit 234 Detection unit 235 Likelihood adjustment unit 300 Electronic Device 310 CPU
320 Camera Interface 330 Display Interface 340 Graphic Engine 350 Hard Disk Device 351 Image Processing Program 352 Image Processing Data 360 Memory 361 Image Processing Process

Claims (5)

On the computer,
A plurality of facial feature point candidates included in an image obtained by photographing a face are extracted for each face part group , and the likelihood indicating the likelihood of the feature point for each of the extracted facial feature point candidates. Get the value
For all of the candidate combinations obtained by extracting one facial feature point candidate from each of the facial part groups, the distance between the facial parts in the facial part placement model, and the feature point in the candidate combination. A difference between corresponding distances with a distance between each face part that is a candidate is calculated, and an adjustment value is calculated by adjusting each difference based on the likelihood value corresponding to each candidate for the feature point. , Summing the adjustment values for each candidate combination,
An image processing program for executing a process of detecting , as the feature point, a feature point candidate included in a candidate combination having the smallest adjustment value among all the candidate combinations . In the computer,
And further executing a process of adjusting the likelihood value using importance corresponding to the feature point to be detected,
An adjustment value obtained by adjusting each difference is calculated based on the distance between each face component and the adjusted likelihood value in the face component arrangement model , and the adjustment values are summed for each of the candidate combinations. The image processing program according to claim 1, wherein: The image processing program according to claim 2, further comprising executing a process of changing the degree of importance according to a feature amount obtained from the image. A plurality of facial feature point candidates included in an image obtained by photographing a face are extracted for each face part group , and the likelihood indicating the likelihood of the feature point for each of the extracted facial feature point candidates. An acquisition unit for acquiring a value;
For all of the candidate combinations obtained by extracting one facial feature point candidate from each of the facial part groups, the distance between the facial parts in the facial part placement model, and the feature point in the candidate combination. A difference between corresponding distances with a distance between each face part that is a candidate is calculated, and an adjustment value is calculated by adjusting each difference based on the likelihood value corresponding to each candidate for the feature point. , An evaluation unit that sums the adjustment values for each candidate combination ;
An image processing apparatus comprising: a detection unit that detects, as the feature points, feature point candidates included in a candidate combination having a minimum adjustment value among all the candidate combinations . A plurality of facial feature point candidates that are facial parts included in an image of a face taken, and a likelihood value indicating the likelihood of the feature point for each of the extracted facial feature point candidates An acquisition unit for acquiring
Import the importance of each facial part as an initial value, cut out a region of a predetermined size including pupil candidates from the image, acquire the luminance value of each pixel included in the region, and obtain the luminance value of each acquired pixel The likelihood corresponding to the pupil candidate by creating a histogram and reducing the importance of the pupil from the initial value when the peak of the luminance value of the pixel included in the region is a predetermined luminance value corresponding to the skin color An adjustment unit for adjusting the value;
Based on the reference value for evaluating the distance between feature points extracted as a plurality of candidates and the likelihood value adjusted by the adjustment unit corresponding to each of the feature point candidates, An evaluation unit that evaluates the distance between the extracted facial feature point candidates;
A detection unit for detecting the feature points from the plurality of extracted facial feature point candidates according to the result of evaluating the distance;
An image processing apparatus comprising:

Images