JP5153434B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to an information processing apparatus and an information processing method.

  Personal recognition using facial images is attracting attention as a recognition technique that can be performed in a non-contact manner and does not give a negative image such as criminal investigation to a user as compared with fingerprint recognition or vein recognition. In the face recognition technique, a feature amount is extracted from a face image of a person to be recognized in advance and is performed based on the generated dictionary data. Therefore, the identification ability depends on the illumination state of the face image in the dictionary data or the posture variation.

  Further, since the face recognition technology can recognize when there is a sufficient distance from the person to be recognized, it is expected to realize a simultaneous recognition system for a plurality of persons in an open environment. Here, the open environment refers to a general environment that does not impose restrictions such as specifying the standing position, specifying the facing to the imaging device, optimizing the lighting conditions, or specifying the facial expression. is there. In other words, the face recognition technology needs to be robust against changes such as size change, posture change, illumination change or expression change.

  For these reasons, the face recognition technology introduces a mechanism for acquiring images with various fluctuations when creating dictionary data and allowing the fluctuations (see Non-Patent Document 1). More specifically, for example, when it is desired to construct dictionary data that allows posture variation, the registrant obtains a plurality of images in various posture states, thereby achieving the above-described object. The same idea can be applied to cases where facial expression variation and illumination variation are allowed in addition to posture variation.

  In the technique of Patent Document 1, knowledge for removing illumination fluctuations from a plurality of arbitrary images is acquired by learning in advance. And the technique of patent document 1 is based on the knowledge regarding the illumination fluctuation | variation obtained by learning, The technique which extracts the feature-value which does not have the influence of illumination fluctuation | variation from an input image and a dictionary image, and is compared and collated is disclosed. .

  Furthermore, Patent Document 2 discloses a technique for preventing instability of accuracy due to a difference in face resolution between an input image and a dictionary image. More specifically, in the technique of Patent Document 2, for example, the image is standardized with reference to the center position of the eyes, and stabilization is achieved by making the resolution the same.

  Further, in Patent Document 3, a feature amount related to posture variation and facial expression variation is extracted from a series of images, it is determined whether the feature amount is appropriate for recognition processing, and a frame of an image determined to be appropriate. A technique for recognizing using the above is disclosed. Note that the frame of the image determined to be appropriate is an image in which the subject is facing the imaging apparatus or the eyes are not opened in an unnatural format.

JP 2004-145576 A Japanese Patent Laid-Open No. 2005-084979 JP-A-6-259534 “Face Recognition by Computer”, IEICE Transactions D-2, 1997, Vol. J80, no. 8, pp. 2031-2046

  Japanese Patent Application Laid-Open No. 2004-228688 addresses this problem by using a means for acquiring knowledge of a fluctuation pattern by learning and a means for normalizing fluctuation for the purpose of stabilizing recognition accuracy against fluctuation. However, it is practically difficult to comprehensively acquire knowledge of fluctuation patterns by learning because a large amount of data is required.

  Japanese Patent Laid-Open No. 2004-228561 addresses this problem by using a means for normalizing fluctuations for the purpose of stabilizing recognition accuracy against fluctuations. However, it is difficult to standardize a fluctuation pattern, for example, a posture (in a two-dimensional image) or an expression, and degradation of recognition accuracy becomes a problem due to the accuracy of the normalization process.

  Further, Patent Document 3 extracts a plurality of effective frames with small fluctuations from a series of images, and recognizes the extracted effective frames. Patent Document 3 is based on the premise that a plurality of effective frames with small fluctuations can be acquired from a series of images. However, a face recognition system in an open environment does not always satisfy such a precondition. Therefore, Patent Document 3 is insufficient for realizing a face recognition system in a general environment.

  Therefore, the above-described conventional technology has insufficient measures against the above-described variation pattern such as posture variation or facial expression variation in an open shooting environment.

  The present invention has been made in view of such problems, and an object of the present invention is to realize accurate and stable object recognition even when an image with fluctuations is input.

Therefore, the present invention is extracted by a receiving unit that receives a time-series image including an object, a feature point extracting unit that extracts a plurality of feature points related to the object from each image of the time-series image , and the feature point extracting unit. A facial expression determination unit that determines the facial expression of the object based on a plurality of feature points, and the facial expression determination unit determines whether the object has a facial expression when the facial expression determination unit determines that the facial expression has a facial expression. A plurality of areas are set based on the coordinate values of the extracted plurality of feature points , a feature vector including arrangement information or shape information of the set area is generated, and the feature vector and dictionary data relating to an object are obtained. collating the collation means for calculating the reliability of the verification result, the accumulated to a reliability calculated by said matching means for each image over a plurality of images of the time-series image irradiation relating to the object A cumulative value calculating means for calculating a cumulative value according to the result of the reliability, output determination determines whether to output a matching result in the matching unit based on the cumulative value calculated by the cumulative value calculation means And means.

  According to the present invention, accurate and stable object recognition can be realized even when an image with fluctuations is input.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a diagram illustrating an example of a hardware configuration of an object recognition apparatus that is an example of an information processing apparatus (computer). As illustrated in FIG. 1, the object recognition apparatus includes an imaging unit 100, a control unit 101, a face detection unit 102, a feature extraction unit 103, a face matching unit 104, and a dictionary data DB unit 105. . The object recognition apparatus further includes an accumulated reliability calculation unit 106, an output determination unit 107, a storage unit 108, and a display unit 109.

  The imaging unit 100 captures an image based on a control signal from the control unit 101, and acquires input image data (a time-series image including an object) (or receives a time-series image). The control unit 101 performs overall control and is connected to the imaging unit 100, the face detection unit 102, the feature extraction unit 103, the face collation unit 104, the cumulative reliability calculation unit 106, the output determination unit 107, the storage unit 108, and the display unit 109. Has been. Note that the image capturing unit 100 may be configured to receive input image data from the image capturing unit 100 as if it were externally connected to the object recognizing device.

  The face detection unit 102 extracts a face area from the image data acquired by the imaging unit 100. The face detection unit 102 detects the face position, the number of faces, the face size, and the orientation, cuts out an image around the face region (face region image), and outputs the cut out face region image to the feature extraction unit 103. The processing content of the face detection unit 102 will be described later. The feature extraction unit 103 extracts coordinate values of facial parts such as eyes, nose, or mouth from the face area image cut out by the face detection unit 102. The face collation unit 104 performs collation based on the information extracted by the feature extraction unit 103 and the dictionary data of the dictionary data DB unit 105, and determines who the detected face belongs to. . Based on the collation result of the face collation unit 104, the cumulative reliability calculation unit 106 calculates a cumulative value of values such as collation similarity for a series of images (cumulative value calculation). The output determination unit 107 determines the output of the collation result by comparing the accumulated value with a predetermined threshold value. The storage unit 108 records the acquired input image data, intermediate output values of each component, and the like. The display unit 109 is a CRT, LCD, or the like, and displays an input image captured by the imaging unit 100 or various calculation results.

FIG. 2 is a flowchart illustrating an example of overall processing of the object recognition apparatus according to the first embodiment.
In step S <b> 200, the cumulative reliability calculation unit 106 performs cumulative reliability initialization processing. In step S <b> 201, the face detection unit 102 receives image data including a face from the imaging unit 100. Next, in step S202, the face detection unit 102 performs processing for detecting a face from the input image data (face detection processing). Details regarding the face detection processing will be described later.

  As a method for detecting a face, the face detection unit 102 may receive information on a face area designated by a user using a pointing device or the like. The face detection unit 102 may detect a face using a filter such as a face detection template. However, in this embodiment, the face detection unit 102 performs face detection using a neural network-based face detection technique. Here, an example of face detection processing using a neural network in the face detection unit 102 in the present embodiment will be described with reference to FIG.

  FIG. 3 is a diagram for explaining an example of face detection processing in the face detection unit 102 using a neural network. As shown in FIG. 3, the neural network related to face detection used in the present embodiment has a hierarchical structure and sequentially takes processing steps from low-order features to high-order features. That is, in the first hierarchical level 301, primitive features (for example, edges) are extracted from the input image data 300 in the detection layer, and higher-order features (for example, in the second hierarchical level 302 are used using the result integrated in the integrated layer. Detect edges that make up eyes and mouth). Similarly, at the third hierarchy level 303, higher-order features are detected using the integration result of the second hierarchy level 302. Finally, the face is detected using the integration result of the third hierarchy level 303 at the fourth hierarchy level 304.

Next, features detected by the face detection process will be described with reference to FIG. FIG. 4 is a diagram for explaining the features detected by the face detection process.
Here, the human face necessary for explaining the face detection process is the model face shown in FIG. Hereinafter, description regarding the face will be described using the model face of FIG.
In the present embodiment, accompanying the face detection process, the face detection unit 102 acquires the firing distribution of neurons near the eyes of both eyes and in the vicinity of the corners of the eyes, both ends of the mouth, the eyes, and the mouth. These are called intermediate output distribution or detection output distribution. The first hierarchical level 301 detects the lowest-order feature that preserves facial features (the face detection neural network used in this embodiment includes eight detection units from the first to the eighth in the first hierarchical level. Have). The feature detection at the first hierarchical level 301 may be, for example, a detection level that is about the degree of luminance change or line segment direction extraction.

  Next, the second hierarchical level 302 includes a right-open V-shaped edge detection output distribution shown in FIG. 4B, a left-open V-shaped edge detection output distribution shown in FIG. 4C, and a line segment edge 1 detection shown in FIG. Output distributions such as the output distribution, the line segment edge 2 detection output distribution of FIG. Here, the right-open V-shaped edge detection output distribution (the black region indicates the V-shaped edge detection output distribution and the gray line segment indicates the face part) in FIG. 4B is the left eye corner and the right eye head. FIG. 5 shows the result of detecting the left end of the eyebrows and the left end of the mouth. Similarly, for 402 to 407, the detection output distribution is indicated by a black region, and the face structure is indicated by a gray line segment. As described above, the V-shaped edge feature is effective for detecting the left and right end features 408 and 409 of the mouth, the eye corner features 410 and 411 of the eyes, or the eye features 412 and 413 of the eyes and the ends of the eyebrows 418. Further, the line segment edge 1 and the line segment edge 2 are effective for detecting the upper and lower eyelids 414 and 415 or the upper and lower lips 416 and 417 of the eyes.

Next, the third hierarchy level 303 receives the feature detection of the second hierarchy level 302, and outputs the eye detection output distribution of FIG. 4 (f) and the mouth detection output distribution 406 of FIG. 4 (g). The last fourth hierarchy level 304 outputs the face detection output distribution of FIG. 4H from the eye and mouth detection results of the third hierarchy level 303. At this time, as the receptive field structure used for face detection in the face detection layer at the fourth hierarchical level 304, a structure suitable for each size and each rotation amount is prepared. In the face detection process, when a result indicating that a face is present is obtained, the face detection unit 102 can obtain face data such as the size and orientation of the face depending on which receptive field structure is used for detection. .
The above is a series of detection output distributions generated by the process of step S202 using the face detection neural network.

Note that the face detection method in step S201 is not limited to the method described above, and a method such as Eigen Face may be used, for example.
In step S202, the face detection unit 102 cuts out a face area image around the face area using the face size acquired in the face detection process. If the face detection unit 102 determines that the determination in step S <b> 203 is OK, the face detection unit 102 outputs the cut face area image to the feature extraction unit 103.

  In step S204, the feature extraction unit 103 extracts feature points of parts constituting the face such as eyes, mouth, and nose, and calculates coordinate values. Note that the eye, mouth, and nose feature points may be extracted by applying a technique such as template matching obtained by scanning a template of a facial part. In this embodiment, FIG. The eye detection output distribution and the mouth detection output distribution of FIG. More specifically, the feature extraction unit 103 calculates the centroid for each detection output distribution, and uses the coordinate value of each centroid as the coordinate value of the feature point. At this time, the feature extraction unit 103 may binarize each detection output distribution with a predetermined threshold value and calculate the center of gravity for the binarized distribution.

  FIG. 5 is a diagram illustrating an example of feature points extracted by the feature extraction unit 103. In FIG. 5, black spots existing on parts such as eyes, mouth, or nose are characteristic points of each part. The extracted feature points are referred to as a right eye feature point 500, a left eye feature point 501, a nose feature point 502, and a mouth feature point 503.

  Next, in step S205, the feature extraction unit 103 checks the arrangement relationship of the feature points (the coordinate values of the feature points) extracted in step S204. That is, when the coordinate value of the extracted feature point is inappropriate for describing the target object, the feature extraction unit 103 returns the process to step S201. More specifically, in the present embodiment, the feature extraction unit 103 determines that the coordinate value of the feature point is the target object when the position of each eye is below the center position of the face area extracted in the face detection step S202. Judged as inappropriate for description. However, this condition does not hold when input of a face image upside down is permitted. This depends on the usage scene of the system. However, it is necessary to check the consistency of the features based on the feature placement rules corresponding to the types of objects to be recognized. That is, the feature extraction unit 103 determines in step S205 whether or not the arrangement of the coordinate values of the feature points satisfies the placement rule based on the placement rule (placement rule file) stored in the storage unit 108, for example. To do.

Next, in step S206, the face matching unit 104 normalizes the size and rotation of the input image data. For example, as shown in FIG. 6, the face matching unit 104 makes the distance 600 between both eyes calculated from the coordinate values 602 and 603 of the feature points of both eyes extracted in step S204 be the same for all images. Apply affine transformation. Here, FIG. 6 is a diagram for explaining size normalization and rotation variation. In addition, the face matching unit 104 detects a slope 601 of a straight line connecting both eyes, and adds affine transformation correction considering rotation. Thus, the face matching unit 104 realizes size variation and in-plane rotation variation.
Next, in step S207, the face collation unit 104 collates the dictionary data in the dictionary data DB unit 105 with the input image data (or a human face that is an example of an object included in the input image data).

Details of the processing in step S207 will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of the collation process.
First, in step S701, the face matching unit 104 acquires the detection output distribution as illustrated in FIG. 4 from the feature extraction unit 103, and sets a region for the input image data according to the detection output distribution.
Here, the areas set by the face matching unit 104 include a local area and a global area. Hereinafter, the local region and the global region will be described.

  Information indicating individual differences is the shape of features such as eyes, nose, or mouth, and the arrangement relationship of features. Here, the local region is set in order to extract shape information of features indicating individual differences such as eyes, nose or mouth. FIG. 8 is a diagram (part 1) illustrating an example of a local region set by the face matching unit 104.

  In FIG. 8, the local region 800 of the eye region is set in the vicinity of the left and right end points of both eyes and the upper and lower eyelid vertices. Further, the local region 801 of the mouth region is set near both end points of the mouth and the upper and lower lip vertices. Further, a local region 802 around the nose is set near the apex of the nose and the left and right nose feature points. Information indicated by these local areas, for example, luminance information, reflects the shape of features such as eyes, nose or mouth. That is, the line of the upper eyelid and the line of the lower eyelid of a person with large eyes tend to cross at a more obtuse angle than a person with thin eyes. Therefore, the information in the local region around the eye end point reflects the shape of the eye. Further, the local region in the vicinity of the corner of the eye reflects a shape such as a cut length of the eye.

  Therefore, the face matching unit 104 can extract eye shape information from a plurality of local regions arranged at both end points of the eyes and the upper and lower eyelid vertices. Further, the face matching unit 104 can also extract each shape information from a plurality of local regions set for the mouth and nose, respectively. In addition, installation of a local area | region is not limited to what is shown by FIG. 8, You may set to eyebrows etc. Further, as shown in FIG. 9, the face matching unit 104 may set a right eye local region 900, a left eye local region 901, a nose local region 902, and a mouth local region 903. However, when set in this way, the object recognition device becomes sensitive to posture fluctuations or shape fluctuations such as eyes or mouth. FIG. 9 is a diagram (part 2) illustrating an example of a local region set by the face matching unit 104.

Next, the global area will be described.
The global region includes a plurality of features, and is set in order to acquire arrangement information between features such as eyes, nose, or mouth. FIG. 10 is a diagram (part 1) illustrating an example of a global area set by the face matching unit 104. Since the face matching unit 104 includes both eyes from the global region 1000 between both eyes, the face matching unit 104 can extract the degree-of-separation information of both eyes using an image standardized with a predetermined distance between feature points. . Further, since the global region 1001 between the eyes and the mouth is set based on the eye feature points and the mouth feature points, the face matching unit 104 similarly uses the standardized image to Can be extracted. The installation of the global area is not limited to FIG. As shown in FIG. 11, the face matching unit 104 may set a global area 1101 that includes the entire face, and a global area 1103 that includes the outline of the jaw in addition to features such as eyes and mouth. FIG. 11 is a diagram (part 2) illustrating an example of a global area set by the face matching unit 104.

The face collation unit 104 acquires shape information and / or arrangement information from the local region and / or the global region in the detection output distribution acquisition process of step S701.
The face matching unit 104 determines a local region from which shape information is acquired based on one edge extraction-like detection output distribution that preserves a feature shape among the eight detection output distributions at the first hierarchical level shown in FIG. Set.

  FIG. 12 is a diagram illustrating an example of an edge extraction-like detection output distribution. The edge extraction-like output distribution is, for example, the black solid line in FIG. The black solid line reflects the eye contour and mouth contour information. FIG. 12B is a diagram in which a plurality of local regions are applied to the detection output distribution of FIG. Note that the local region in FIG. 12B is set based on the right eye feature point 1201, the left eye feature point 1202, the mouth feature point 1203, and the nose feature point 1204.

  On the other hand, the global area 1000 between the two eyes in FIG. 10 for acquiring the arrangement information is a right-open V-shaped edge detection output distribution of FIG. 4B or a left-open V-shaped edge detection output distribution of FIG. Is set based on the output distribution of either one or both.

  FIG. 13 is a diagram illustrating an example of setting a global area between both eyes. FIGS. 4B and 4C show the results of detecting the V-shaped edge, which corresponds to the detection of both end points of the eyes or both end points of the mouth. Therefore, the face matching unit 104 sets the global area 1303 between both eyes as shown in FIG. 13 by setting the middle point 1301 between both eyes, which is the middle point between the right eye feature point and the left eye feature point, as a starting point. Arrangement information can be acquired. The detection output distribution in FIG. 13A corresponds to a distribution obtained by superimposing the left and right V-shaped edge detection output distributions. By doing in this way, the face collation part 104 can acquire the width information of each eye.

  Based on the same idea, the global region 1001 between the eyes for obtaining the arrangement information is the line edge 1 detection output distribution of FIG. 4D or the line edge 2 detection output distribution of FIG. Either one of 404 or both are set based on the output distribution.

FIG. 14 is a diagram illustrating an example of setting a global area between the eyes and the mouth. FIG. 4D or FIG. 4E shows the result of detecting a line segment, in other words, corresponding to the detection of upper and lower eyelids or upper and lower lips. Therefore, the face matching unit 104 sets the eye-mouth global area 1404 as shown in FIG. Can be obtained.
As described above, the face matching unit 104 sets a region according to the detection output distribution.

Next, the high-dimensional feature vector acquisition process in step S702 will be described. Data for obtaining a high-dimensional feature vector is generated from a region applied to each detected output distribution by the face matching unit 104 as described above. FIG. 15 is a diagram illustrating the definition of a high-dimensional feature vector. FIG. 15 shows, as an example, a method for generating a feature vector of the local region 1500 and a definition of a higher-order feature vector. As shown in FIG. 15, the high-order feature vector F is generated by combining one-dimensional vector data f _k generated from a local region and / or a global region. Here, the face collation unit 104 generates one-dimensional vector data f _k by scanning a two-dimensional array of detected output values f _i in the local region 1500 in a predetermined direction.

Next, the collation process in step S703 will be described. In step S703, the face matching unit 104 inputs the high-order feature vector F to a support vector machine (hereinafter referred to as SVM), and performs matching using the dictionary data in the dictionary data DB unit 105.
Here, SVM is one of learning algorithms. In the present embodiment, lisvm is used as an example of SVM.

  Next, a process for classifying input data (higher order feature vectors) using dictionary data will be described. The dictionary data generation process will be described later. lisvm has a format having a plurality of classifiers (or classification functions) for classifying into two classes. For example, it is assumed that there are four registrants (dictionary creator or learner), each of which is class A, class B, class C, class D, and class other than the registrant is class E. At this time, the classifier that classifies into two classes is a classifier that performs threshold determination (threshold is generated every two classes when generating dictionary data) which is most likely AorB. Therefore, the input feature vector is classified into two classes among all classes like AorB, AorC, AorD, AorE, BorC..., And the final class is determined by majority processing of the classification results. .

Here, an example of generation of dictionary data will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of dictionary data generation processing.
In step S1600, for example, the control unit 101 determines the presence or absence of a learner. For example, if there is a learner, the control unit 101 advances the process to step S1601, and if there is no learner, advances the process to step S1611.

  In step S <b> 1611, for example, the face matching unit 104 or the like executes a learning process using lisvm. On the other hand, in step S1601, for example, the control unit 101 or the like determines whether or not the learner's image exists in a predetermined region or the like. For example, the control unit 101 or the like advances the process to step S1602 when the learner image exists, and returns the process to step S1600 when the learner image does not exist.

In step S1602, the face detection unit 102 secures image data including a face on the memory. Hereinafter, the processing from step S1603 to step S1607 is the same as the processing from step S202 to step S206 in FIG.
Further, the processing in step S1608 and step S1609 is the same as the processing in step S701 and step S702 in FIG.

In step S1610, the higher-order feature vector generated by the processing up to step S1609 is recorded in a storage unit such as a memory by the control unit 101 or the like.
After recording the higher-order feature vector, the object recognition apparatus acquires another image (image file) and executes step S1601. At this time, in step S1601, for example, if there is no image of a person being processed, the control unit 101 or the like performs processing to search for an image of another person.

  Next, the cumulative reliability calculation process in step S208 of FIG. 2 will be described. In the cumulative reliability calculation process, the cumulative reliability calculation unit 106 cumulatively calculates the reliability described later with respect to the time series image. In the present embodiment, the cumulative reliability calculation unit 106 calculates the reliability based on the output value of the SVM used in the collation process in step S703. Details of the reliability calculation are shown below.

  The cumulative reliability calculation unit 106 determines that the determination result has a high reliability when the number of votes in the final determination result determined by the majority process of the SVM exceeds a predetermined threshold value, and obtains the reliability by incrementing. This predetermined threshold is set to a majority or more of the number of discriminating classes (if the input is classified into five classes, the discriminating class number is 5). In addition, the number of votes in favor refers to the number of classifiers in which the results of a plurality of classifiers are the same as the final determination result. More specifically, the process for calculating the reliability will be described with reference to the flowchart of FIG.

FIG. 17 is a flowchart illustrating an example of a process for calculating the reliability. Note that the flowchart shown in FIG. 17 is executed as many times as the number of faces subjected to subject collation processing in step S207.
In step S <b> 1701, the cumulative reliability calculation unit 106 initializes by assigning zero to a variable face indicating the element number assigned to each face.

  In step S1702, the cumulative reliability calculation unit 106 checks whether or not the processing for the number of faces has been executed. Here, the constant faceNum indicates the total number of faces to be processed. FIG. 18 is a diagram illustrating a state in which an element number is designated for a face. Image data 1800 in FIG. 18A shows a state in which element numbers are assigned to the face of the acquired input image. The cumulative reliability calculation unit 106 assigns the same number (element number) to the same face while calculating the cumulative reliability. That is, when the image data 1800 changes to the image data 1801 in time series, it is assumed that in the image data 1801, each face has moved from a dotted face to a solid face. At this time, the solid face is assigned the same element number as the dotted face, that is, the past face. Therefore, in order to perform the above-described operation, a technique for tracking the face in the screen is necessary. Tracking is performed using a known technique. In this embodiment, it is assumed that the cumulative reliability calculation unit 106 obtains a movement vector calculated from the optical flow and performs face tracking based on the movement vector.

If the cumulative reliability calculation unit 106 determines that the processing for the number of faces has been executed, the cumulative reliability calculation unit 106 ends the processing illustrated in FIG. 17.
In step S1703, the cumulative reliability calculation unit 106 assigns zero to the variable voteCount and initializes it.
Next, step S1704 will be described. For example, the number of discrimination classes is 5 (5 patterns of A, B, C, D, and E are used, and each is labeled as label = 1, label = 2, label = 3, label = 4, label = 5) And At this time, there are ten 2-class classifiers. This is shown in FIG. FIG. 19 is a diagram illustrating a determination result of the 2-class classifier. In FIG. 19, an AorB classifier that classifies whether input data is A or B is a portion where A and B cross from the column of comparison target class. The result of the classifier is indicated by italic characters in FIG. That is, in FIG. 19, the classification result of the AorB classifier is A. The final determination result is determined as A by majority processing. From the classification result of each classifier in FIG. 19, the number of votes in favor of the final determination result is 4. Finally, in step S1704, the cumulative reliability calculation unit 106 substitutes the acquired number of votes for the variable voteCount that stores the number of votes for the final determination result.

Next, step S1705 will be described. In step S1705, the cumulative reliability calculation unit 106 compares the acquired variable voteCount with a predetermined threshold value. In the present embodiment, the predetermined threshold is set to a value larger than the majority of the number of determination classes as described above. However, a value obtained by a statistical method or a heuristic may be used as the predetermined threshold. Here, the predetermined threshold is represented by Th _conf in step S1705. When the variable voteCount is larger than Th _conf , the cumulative reliability calculation unit 106 increments the reliability variable of the label of the final determination result by 1 in step S1706, assuming that the final determination result is reliable. Here, the reliability variable is a variable confidence in the flowchart. The variable confidence is expressed by a two-dimensional array of face numbers and labels.

When the variable voteCount is not larger than Th _conf , the cumulative reliability calculation unit 106 increments the value of the variable face by 1 in step S1707, and returns to the process of step S1702.

Next, the output determination process in step S209 in FIG. 2 will be described. In step S209, the output determination unit 107 compares the cumulative reliability described above with a predetermined threshold value. More specifically, the output determination unit 107 performs output determination by evaluating the following expression for each label of each detected face.

は、あるラベルのある時刻ｔの入力画像までの累積信頼度を表し、Ｔｈｏｕｔｐｕｔは、所定の出力判定閾値を表し、ｎは判別クラス数を表す。数式（１）の比較処理は、ステップＳ２０７で処理対象、かつ、トラッキング可能である顔、全てに対して行われる。所定閾値は、統計的手法やヒューリスティックに求めることで獲得される。出力判定部１０７は、数式（１）が真であった場合、ステップＳ２１０へと進む。また、出力判定部１０７は、数式（１）が偽であった場合、ステップＳ２０１へと進む。このとき、複数の顔が写っている１つの画像データにおいて、ステップＳ２１０の評価が、ＯＫとＮＧとを共に含んでいた場合、以後、ＮＧ顔のみが図２のフローチャートの処理対象となる。ＯＫ顔は、照合処理等を行わずトラッキングで画面内追尾される。 here,

Represents the cumulative reliability up to an input image at a certain time t with a certain label, Thoutput represents a predetermined output determination threshold, and n represents the number of discrimination classes. The comparison processing of Expression (1) is performed on all the faces that can be processed and tracked in step S207. The predetermined threshold is obtained by a statistical method or heuristic determination. If the mathematical expression (1) is true, the output determination unit 107 proceeds to step S210. Further, when the mathematical formula (1) is false, the output determination unit 107 proceeds to step S201. At this time, if the evaluation in step S210 includes both OK and NG in one image data showing a plurality of faces, only the NG face will be processed in the flowchart of FIG. The OK face is tracked in the screen by tracking without performing a matching process or the like.

  Next, step S210 in FIG. 2 will be described. In step S210, the display unit 109 (or the control unit 101) outputs a collation result for the target face whose cumulative reliability exceeds the predetermined threshold. More specifically, in step S210, the display unit 109 executes a process of outputting a registered name corresponding to a label that exceeds a predetermined threshold in the vicinity of the target face.

  As described above, in the first embodiment, face recognition with high certainty can be realized by calculating the cumulative reliability based on the number of approval votes obtained from the results of a plurality of classifiers. Therefore, it is possible to provide a face recognition system in which erroneous recognition is reduced even in face recognition processing in situations where there is no restriction such as posture.

<Second Embodiment>
Hereinafter, the second embodiment will be described. The basic configuration follows the first embodiment. Differences from the first embodiment will be described below. In the second embodiment, the face state, for example, a variation such as a facial expression, posture, or illumination is estimated, and subject verification is performed based on the estimation result. Furthermore, in the second embodiment, the point is to calculate the cumulative reliability using the estimation result. In the second embodiment, the state of the object to be estimated is limited to a facial expression, and a more specific description will be given below with reference to FIG.

FIG. 20 is a flowchart illustrating an example of overall processing of the object recognition apparatus according to the second embodiment.
Steps S2000, S2001, S2002, S2003, S2004, S2005, and S2009 are the same as the corresponding steps in FIG. 2 described in the first embodiment. Therefore, description of these step groups is omitted.

  First, the process of step S2006 will be described. A representative example of facial expressions is shown in FIG. FIG. 21 is a diagram illustrating an example of a representative example of facial expressions. The face for which the expression is determined is a face having an expression that does not exist in the dictionary data. For example, when the facial expression of the registrant who generates the dictionary image is a neutral face in FIG. 21A, the face with the mouth open or the eyes closed is the face for facial expression determination. . In the present embodiment, the case where the face of the registrant of the dictionary data is only the neutral face in FIG. 21A with no expression or the majority of the neutral faces in FIG. 21A will be described more specifically.

  The neutral face in FIG. 21 (a) is a facial expression that a person normally takes, and is a face in which the eyes are open and the mouth is closed. On the other hand, in the faces other than the neutral face in FIG. 21A (FIGS. 21B to 21J), the features of the eyes or mouth of the neutral face show a change in size. More specifically, the mouth open face shown in FIG. 21B and the mouth half open face shown in FIG. 21C correspond to the case where the mouth shape is changed with respect to the neutral face shown in FIG. . This change occurs in a natural state where a person in the image is uttering a word or a state where the mouth is intentionally wide open.

  21 (d), both eyes half-closed face in FIG. 21 (e), one eye closed face in FIG. 21 (f), or one eye half-closed face in FIG. This is equivalent to The faces shown in FIGS. 21 (d) and 21 (e) appear as facial expressions as a result of a human eye-catching operation. The faces shown in FIGS. 21 (f) and 21 (g) appear as facial expressions as a result of intentional movement of a person such as wink. Further, FIG. 21 (h) both eyes closed + mouth open face, FIG. 21 (i) one eye closed + mouth open face, or FIG. 21 (j) one eye half closed + one eye closed face, each shape change between eyes and mouth. This is a representative example of a combined face. Next, a method for determining these facial expressions will be described.

  The facial expression determination performed in the present embodiment is realized by using a known technique. More specifically, as shown in FIG. 22, the object recognition device (for example, the feature extraction unit 103) detects feature points around the outline of the eyes, nose, mouth, eyebrows, or jaws, and arranges the feature points. A facial expression is detected by a dynamic change of a state or a feature point. FIG. 22 is a diagram illustrating an example of feature point extraction. Here, the outline feature point is an example of a black circle 2201 in FIG. 22, and is extracted not only from the part in contact with the face surface but also from the part in contact with the oral cavity with respect to the periphery of the lips. As a method for extracting contour feature points of each feature, for example, a technique of detecting by extracting the contour of each feature by edge extraction is used. In this embodiment, the feature extraction unit 103 uses an AAM (Active Appearance Model). The AAM can extract a feature point as shown in FIG. 22 by fitting a model having a relative relationship between feature points and a luminance value of each feature point as an evaluation value to the face.

  Any of the above-described known techniques relating to contour feature point extraction is performed simultaneously or stepwise with the eye, mouth and nose feature point extraction process of step S2004 described above. Next, the expression determination method will be described more specifically.

  The facial expression is determined by a discriminator (hereinafter referred to as a facial expression discriminator) that has learned the arrangement relationship of contour feature points for each facial expression. A parameter used when the facial expression classifier learns is a mesh distance or angle formed by connecting feature points. FIG. 23 is a diagram for explaining meshes and parameters. The mesh indicates a mesh set for the face. As shown in FIG. 23A, it is set around the eyes where the facial expression appears or around the mouth. The parameters are distances a, b, c and angles θ, φ, ψ of each side of the unit mesh, as shown in FIG. However, the distance parameter needs to be normalized by the distance between both eyes in order to remove individual differences. The facial expression discriminator 2400 includes a plurality of classifiers that identify each facial expression as shown in FIG. 24, and an output integrator 2401 that integrates the outputs of the facial expression classifiers. FIG. 24 is a diagram illustrating an example of a facial expression determiner. Each expression classifier 2402, 2403, and 2404 is constructed by calculating and learning the above parameters for all meshes for each expression. The output integrator 2401 compares the outputs of the expression classifiers to determine the final expression. In this embodiment, learning of each facial expression classifier is performed using a neural network that is a known technique.

  When the facial expression determination is completed, the process proceeds to step S2007. In step S2007, the object recognition apparatus (or the feature extraction unit 103) receives the result of step S2006 and selects a local area and a global area.

  The region selection step S2007 will be described. If the facial expression determination in the facial expression determination step S2006 is, for example, an open mouth face 2101 shown in FIG. 21, the shape of the mouth feature is changed. Therefore, the information in the local region 801 set in the mouth shown in FIG. 8 may not be similar to the neutral mouth shape. Therefore, the object recognition device excludes the local region set in the mouth from the feature extraction region. If it is determined in step S2006 that facial expression is closed with both eyes closed + open mouth face 2107, the local area set for the eyes (both eyes) and the local area set for the mouth are set as the object for the same reason as described above. The recognition device excludes it from the feature extraction region. At this time, regarding the global area, the object recognition apparatus does not change the arrangement of the features due to the expression, and therefore does not select the area. As described above, in step S2007, the object recognition apparatus performs adaptive selection of the feature extraction region in response to the result of step S2006.

  Next, in step S2008, the object recognition apparatus (or face collation unit 104) selects dictionary data suitable for the selected area from the dictionary data DB unit 105. More specifically, in step S2008, the object recognition apparatus performs a process of searching for dictionary data constructed by the area selected in step S2007. Therefore, the dictionary data needs to be constructed in advance by combining a plurality of local areas and global areas. The construction of the dictionary data group will be described later.

  Next, in step S2010, the face collation unit 104 collates the extracted dictionary data with the input data. The collation method is the same as in the first embodiment, and is processed according to the procedure shown in FIG. However, the difference from the first embodiment is that the face matching unit 104 generates a vector from the region selected in the region selection processing in step S2007 and performs matching.

  Next, the flow of dictionary data generation in the second embodiment will be described using the flowchart of FIG. FIG. 25 is a flowchart illustrating an example of overall processing of the object recognition apparatus according to the second embodiment. In executing the flowchart of FIG. 25, the object recognition apparatus allocates the still images or moving images of each registrant and non-registrant in advance in folder units. The registrant's facial expression is assumed to be a face with almost no facial expression.

In step S2500, for example, the control unit 101 determines the presence or absence of a learner. For example, when there is a learner, the control unit 101 advances the process to step S2501, and when no learner exists, the control unit 101 advances the process to step S2511.
In step S2511, for example, the face matching unit 104 or the like executes a learning process using lisvm. On the other hand, in step S2501, for example, the control unit 101 or the like determines whether or not the learner's image exists in a predetermined region or the like. For example, when there is a learner image, the control unit 101 advances the process to step S2502, and when there is no learner image, the control unit 101 returns the process to step S2500.

In step S2502, the face detection unit 102 secures image data including a face on the memory. Hereinafter, the processing from step S2503 to step S2507 is the same as the processing from step S202 to step S206 in FIG.
First, the process of step S2508 will be described.
Here, for example, the face matching unit 104 sets a plurality of selected region patterns as the local regions to be set in the eye, mouth, and nose features shown in FIG. FIG. 26 is a diagram illustrating an example of a region pattern. The area pattern in FIG. 26A corresponds to a state in which facial expressions are detected in the left eye and mouth, and local areas set in the left eye and mouth are excluded. A pattern other than the area pattern of FIG. 26A will be described with reference to FIG.

The area pattern 2601 is a pattern when there is a shape change only in the mouth. The area pattern 2602 is a pattern when there is a shape change only in the right eye (or left eye). The area pattern 2603 is a pattern when there is a shape change in both eyes. 2604 is a pattern when there is a shape change in both eyes and mouth. The area pattern 2605 is a pattern when there is a shape change in the nose. The area pattern 2606 is a pattern when the mouth is spread on both sides. Standard pattern in the absence of the global area, both patterns use the global area between both eyes and the eye-mouth. The reason is that the shape of the feature such as the eye changes depending on the facial expression, but the positional information on the eyes, mouth, and nose is shifted by the facial expression as long as the global area set based on the feature points as shown in FIG. 4 is used. Absent. Therefore, the positional information extracted by the global area shows individual differences even when facial expressions exist, and is an effective feature when there is facial expression fluctuation.

In step S2508, the face matching unit 104 sets the plurality of area patterns described above for the input face.
Next, step S2509 for generating a dictionary vector for each area pattern will be described. For example, the face matching unit 104 sets the region pattern group set in step S2508 for the detection output distribution shown in FIGS. 11, 12, and 13, and performs processing for generating higher-order feature vectors. The generated higher-order feature vector is recorded in a storage unit such as a memory by the control unit 101 or the like in step S2510. After recording, for example, the control unit 101 or the like searches for image data in order to read another image file. At this time, for example, when there is no person image data being processed, the control unit 101 or the like performs a process of searching for an image of another person.

Next, the process of step S2011 in FIG. 20 will be described. In the second embodiment, the cumulative reliability calculation unit 106 calculates the reliability based on the number of regions selected in step S2007. A more specific reliability calculation method is shown below.
The object recognition apparatus of the present embodiment prepares the certainty factor as a lookup table shown in FIG. FIG. 27 is a diagram illustrating an example of a certainty lookup table. In the lookup table 2700 of FIG. 27, the first and third columns indicate the number of effective areas N, and the second and fourth columns indicate the reliability. The number of effective areas is the number of areas for extracting features determined as a result of selection in step S2007 in response to the result of step S2006. In addition, the certainty factor is set in a format that simply increases with respect to the number of regions based on the idea that the amount of information increases and the collation accuracy improves as the number of regions increases. Further, the certainty factor obtained here is learned by a known learning algorithm, for example, a subspace method, for example, Aor! It is integrated into the output value of the discriminator for classifying A. The cumulative reliability calculation unit 106 sets a value obtained by calculating the above-described integrated value for a predetermined time as the cumulative reliability.

  As described above, according to the second embodiment, it is possible to calculate a reliable parameter even when there is a change in facial expression, and misrecognition of the face recognition system when there is a change in facial expression (or posture change or the like). Can be reduced.

<Third Embodiment>
A third embodiment will be described. The third embodiment is an embodiment including processing in the case where there is no change in facial expression or the like in the input face. When there is no change, the similarity is high because it is considered that the face state of the image group of the dictionary data and the state of the input face are substantially equal. Therefore, when it is determined that the input face is in a state where there is no fluctuation, the selection process for the predetermined area is not executed. In order to explain more specifically, a flowchart of the third embodiment is shown in FIG.

FIG. 28 is a flowchart illustrating an example of overall processing of the object recognition apparatus according to the third embodiment.
The flowchart of FIG. 28 is different from the flowchart of FIG. 20 in that step S2800 for initializing high-ranking candidate information is added, and that the process of step S2006 for performing facial expression determination branches to step S2807 for branching facial expression determination. The difference is that it has been changed. Also, in the flowchart of FIG. 28, a series of processes when it is determined that there is a facial expression in step S2807 is added. The explanation about each processing step added or changed is described below.

  In step S2800, for example, the control unit 101 initializes higher-ranking candidate information described later. Next, in step S2807, the facial expression discriminator performs processing in which the branch processing function is added to the processing in step S2006 in the processing flow of the second embodiment.

  Next, in step S2815, for example, the cumulative reliability calculation unit 106 extracts the top candidate information in step S2813 while processing the time-series image. More specifically, the high-ranking candidate information is the names of a predetermined number of high-ranking candidates in a candidate list ordered by the cumulative reliability of a certain input face matching result. The candidate list is created as a candidate list 2901 shown in FIG. Here, FIG. 29 is a diagram illustrating an example of a candidate list and the like. A candidate list 2901 shows a list of face number 0 of the image data 2900 at a certain time. The candidate list 2901 is a table that is ranked by the reliability in the third column of the list and associates each ID label. Here, LABEL_A and LABEL_C indicate registrant A and registrant C of dictionary data of registrants A, B, C,. REJECT indicates that the person is a non-registered person other than the registered person.

  In step S <b> 2816, for example, the cumulative reliability calculation unit 106 acquires, for example, three ID labels that are higher in the candidate list 2901. Alternatively, in step S2816, for example, the cumulative reliability calculation unit 106 may take the reliability differences in order from the top and acquire an ID label having the above difference equal to or greater than a predetermined threshold. More specifically, when the reliability difference between the fourth place and the fifth place exceeds the threshold value, for example, the cumulative reliability calculation unit 106 acquires the ID labels of the top four names.

  Next, in step S2817, for example, the cumulative reliability calculation unit 106 acquires, from the dictionary data DB unit 105, dictionary data of a plurality of candidates extracted in step S2816. In the present embodiment, the dictionary data is constructed by using SVM, except for the registrant A and the registrant A, that is, A or! A classifier. In this embodiment, the “reject or!” Like the reject classifier, a classifier indicating whether the user is a registered person or a non-registered person is also prepared as dictionary data.

  Next, in step S2819, for example, the cumulative reliability calculation unit 106 performs a matching process using the dictionary data of only the candidate acquired in step S2817. More specifically, for example, the cumulative reliability calculation unit 106 determines that the input data is A or! A, C or! C, reject or! Input to the reject classifier. A classifier built by SVM provides a binary result output. Thus, for example, if the input face is A, A or! The output appears in the A classifier.

  As described above, according to the third embodiment, it is possible to provide a collation process that is faster than large-scale dictionary data in an open face recognition system by using the top candidate information determined by the cumulative reliability. Can do.

<Other embodiments>
As described above, in the above-described embodiment, an example in which each function of the information processing apparatus (object recognition apparatus) is implemented as hardware has been described as an example. However, a part of the above-described function may be implemented as software. Good. More specifically, for example, the control unit 101, the face detection unit 102, the feature extraction unit 103, the face matching unit 104, the cumulative reliability calculation unit 106, the output determination unit 107, and the like described above are included in the information processing apparatus as software. May be implemented. That is, the functions may be realized by the CPU of the information processing apparatus reading out and executing the programs related to these functions from the memory or the like.

  As described above, according to the above-described embodiments, accurate and stable object recognition can be realized even when an image having fluctuations is input.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

It is a figure which shows an example of the hardware constitutions of the object recognition apparatus which is an example of information processing apparatus (computer). It is a flowchart which shows an example of the whole process of the object recognition apparatus in 1st Embodiment. It is a figure for demonstrating an example of the face detection process in the face detection part 102 using a neural network. It is a figure for demonstrating the characteristic detected by a face detection process. It is a figure which shows an example of the feature point which the feature extraction part 103 extracted. It is a figure for demonstrating size normalization and rotation fluctuation. It is a flowchart which shows an example of a collation process. FIG. 6 is a diagram (part 1) illustrating an example of a local region set by a face matching unit 104; FIG. 10B is a diagram illustrating an example of a local region set by the face collation unit 104 (part 2); FIG. 5 is a diagram (part 1) illustrating an example of a global area set by a face matching unit 104; FIG. 10B is a diagram illustrating an example of a global area set by the face matching unit 104 (part 2); It is a figure which shows an example of detection output distribution like edge extraction. It is a figure which shows an example of the setting of the global area | region between both eyes. It is a figure which shows an example of the setting of the global area | region between eyes-mouths. It is a figure which shows the definition of a high-dimensional feature vector. It is a flowchart which shows an example of a dictionary data generation process. It is a flowchart which shows an example of the process which calculates reliability. It is a figure which shows a mode that an element number is designated to a face. It is a figure which shows the determination result of a 2 class classifier. It is a flowchart which shows an example of the whole process of the object recognition apparatus in 2nd Embodiment. It is a figure which shows an example of the representative example of a facial expression. It is a figure which shows an example of extraction of a feature point. It is a figure for demonstrating a mesh and a parameter. It is a figure which shows an example of a facial expression determination device. It is a flowchart which shows an example of the whole process of the object recognition apparatus in 2nd Embodiment. It is a figure which shows an example of an area | region pattern. It is a figure which shows an example of the look-up table of reliability. It is a flowchart which shows an example of the whole process of the object recognition apparatus in 3rd Embodiment. It is a figure which shows an example of a candidate list etc.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image pick-up part 101 Control part 102 Face detection part 103 Feature extraction part 104 Face collation part 105 Dictionary data DB
106 Cumulative reliability calculation unit 107 Output determination unit 108 Storage unit 109 Display unit

Claims (6)

Receiving means for receiving a time-series image including an object;
Feature point extracting means for extracting a plurality of feature points related to the object from each image of the time-series image;
Facial expression determination means for determining facial expression of the object based on a plurality of feature points extracted by the feature point extraction means;
When it is determined by the facial expression determination means that the object has a facial expression, a plurality of regions are set based on the coordinate values of the plurality of feature points extracted by the feature point extraction means according to the facial expression, Generating a feature vector including arrangement information or shape information of the set region , collating the feature vector with dictionary data related to an object, and collation means for calculating the reliability of the collation result ;
Cumulative value calculation means for accumulating the reliability calculated by the matching means for each image over a plurality of images of the time-series image and calculating a cumulative value related to the reliability of the matching result for the object;
Output determination means for determining whether or not to output a collation result in the collation means based on the cumulative value calculated by the cumulative value calculation means;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, further comprising dictionary data generation means for generating the dictionary data based on an image including an object. The collation means collates the dictionary data of the object whose cumulative value calculated by the cumulative value calculation means is higher with the feature vector when the facial expression determination means determines that the object has no facial expression. The information processing apparatus according to claim 1 , wherein: An information processing method in an information processing apparatus,
Receiving a time-series image including an object;
A feature point extracting step of extracting a plurality of feature points related to the object from each image of the time series image;
A facial expression determination step for determining a facial expression of the object based on a plurality of feature points extracted in the feature point extraction step;
When it is determined that the object has an expression in the expression determination step, a plurality of regions are set based on the coordinate values of the plurality of feature points extracted in the feature point extraction step according to the expression, Generating a feature vector including arrangement information or shape information of the set region, collating the feature vector with dictionary data relating to an object, and calculating a reliability of the matching result;
A cumulative value calculating step of accumulating the reliability calculated in the collating step for each image over a plurality of images of the time-series image to calculate a cumulative value related to the object;
An output determination step for determining whether to output a collation result in the collation step based on the cumulative value calculated in the cumulative value calculation step;
An information processing method comprising: The information processing method according to claim 4, further comprising a dictionary data generation step of generating the dictionary data based on an image including an object. In the collation step, when it is determined in the facial expression determination step that the object has no facial expression, the cumulative value calculated in the cumulative value calculation step is collated with the dictionary data of the upper object. The information processing method according to claim 4, wherein:

Images