JP2007272398A - Classification device, program, and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To classify classification object data as a classification problem whose number of categories is unknown. <P>SOLUTION: The learning of a self-organized map is operated by using face image data 62 for learning so that the coupled loads of map layer units U0 to U4 of a map layer 64 are determined as a visualization image 66. Then, correlation coefficients r0 to r3 of the coupled loads between the adjacent map layer units are calculated, and a gap between the map layer units U3 and U4 showing the minimum r3 is defined as a division boundary 70 in the first layer. The map layer unit U4 is classified into only one, and the classification of the second layer is not operated. On the other hand, the classification of the map layer units U0 to U3 is insufficient, and the classification of the second layer is operated. This processing is continued until the classification is sufficiently operated based on the classification characteristics. Thus, the expression feature space of a binary tree corresponding to the face image data for learning is finally formed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a data classification technique using a self-organizing map, and more particularly to a technique for further classifying classified data.

  A technique for classifying face image data according to facial expressions is known.

  Patent Document 1 listed below discloses a technique for estimating a facial expression by quantifying changes in the contour shape of a facial organ (such as eyes and mouth). In this technique, facial expressions are estimated for four facial expressions of surprise, joy, anger, and neutrality.

  Patent Document 2 listed below is a technology for determining the category of basic six facial expressions (surprise, joy, sadness, fear, disgust, anger) from the beginning, and performing facial expression recognition by matching based on a code book based on those feature vectors. Is disclosed. Then, the recognition process using the HMM (Hidden Markov Model) is performed for the transition from the expressionless to the basic six expressions.

  The technique of the following Patent Document 3 performs determination only for a smile based on measurement of the degree of opening of the mouth. Specifically, the lip region (mouth region) and the tooth region are extracted using the color information, and the opening degree of the mouth is recognized from the outer shape, inner diameter, and area of the mouth region, the area of the tooth region, and the like.

  In the techniques of these documents, if the extraction of the facial organ contour shape or the feature points on the facial organ fails, the recognition processing may fail or the processing load will increase. High recognition becomes difficult. In addition, these techniques classify face image data into several predetermined facial expressions. However, because the characteristics of facial expressions may differ even for the same emotion due to factors such as individual differences, it may be better to use a facial expression different from the basic six facial expressions as the basis of classification.

  On the other hand, in the techniques of Non-Patent Documents 1 to 3 below, individual facial expressions are classified using a self-organizing map. In this technique, the classification category of facial image data is not set in advance, and facial expressions are classified into two parts one after another from a large feature to a small feature. As a result, each facial expression is classified into a category suitable for the characteristics of the individual facial expression.

JP 2005-293539 A JP-A-8-249453 JP 2005-234686 A Masaki Ishii, Kazuto Sato, Hirokazu Mazokoro, Sakura Kadowaki, Satoshi Nishida, “Examination on Optimization of Facial Space Model Focusing on Dynamic Changes in Face”, IEICE Technical Report, IEICE, March 11, 2005, Vol. 104, No. 742, PRMU 2004-250, pp. 109-114 Masaki Ishii, Kazuhito Sato, Hirokazu Masho, Satoshi Nishida, “Formation of personalized facial expression feature space based on phase characteristics of facial images”, Proceedings of Image Recognition and Understanding Symposium 2005, Information Processing Society of Japan, 2005 July 18, 2005, pp. 763-770 Masaki Ishii, Satoshi Nishida, Kazuto Sato, Hirokazu Mazokoro, “Examination of the formation of personal facial expression feature space based on the phase characteristics of facial images”, Proceedings of the 2005 Annual Conference of the Tohoku Branch of the Electrical Engineering Society, 2005 Electrical Engineering Society Tohoku Branch Union Conference Executive Committee, August 25, 2005, p. 297

  The techniques of Non-Patent Documents 1 to 3 do not perform classification into fixed classification categories as in Patent Documents 1 to 3. However, in the technique of Non-Patent Document 1, the hierarchical classification depth is determined in advance. For this reason, it may be impossible to capture various facial expressions. In particular, considering differences such as individual differences, gender differences, ethnic differences, face orientations, lighting differences, etc., it may be preferable to perform more classifications than planned. In addition, even though there are actually not so many different expressions, it may be forcibly classified until the number of categories reaches a predetermined number.

  Although facial expression classification is technically difficult, it is expected to be applied in various fields including Human Computer Interaction. However, facial expression classification technology is not necessarily limited to facial data classification, and it can be applied to various types of data such as graphics and documents as well as various sensor data such as general images, movies, and sounds. There are many things you can do.

  An object of the present invention is to establish a technique for classifying data to be classified as a classification problem with an unknown number of categories.

  Another object of the present invention is to realize a technique for changing the number of classification categories of face image data in accordance with the variety of facial expressions.

  Still another object of the present invention is to develop a new technique for forming a facial expression feature space unique to an individual based on the individual face image data.

  The classification apparatus according to the present invention performs a learning operation for sequentially mapping a plurality of learning data to a map layer, evaluates classification characteristics of the obtained self-organizing map, and divides the map layer into two or more categories. A classification processing unit that classifies the classification characteristics, a determination unit that evaluates the classification characteristics and determines the necessity of reclassification of a certain category, and when the determination unit determines that reclassification is necessary, the category And hierarchizing means for causing the classification processing means to classify again the learning data belonging to.

  The classification device can be constructed using computer hardware having an arithmetic function and a storage function and software for controlling the operation. The classification device may be constructed as a distributed system using a plurality of communicable hardware or may be constructed as a centralized system using a single piece of hardware.

  The classification processing means performs learning calculation of the self-organizing map and classifies the map layer into two or more categories. The self-organizing map is a mathematical model that includes an input layer to which data to be classified is input and a map layer to which the input is mapped (mapped) (and can further include an intermediate layer). It is built on a storage device (memory). In the learning of the self-organizing map, the teacher signal is not used, and instead, the competition and update of the map layer units constituting the map layer are performed with respect to the input of the learning data as the classification target data. As a result, in general, the self-organizing map has classification characteristics that map similar classification target data to the same or nearby map layer units. The classification processing means evaluates the classification characteristics by applying to a preset classification condition and classifies the map layer into two or more categories.

  The determining means is means for determining the necessity of reclassification of a certain category. The determination is made by evaluating the classification characteristics. This evaluation mode may be the same as the classification processing unit, but may be different. For example, the determination unit evaluates the classification characteristics in a certain category, and determines the necessity of reclassification based on preset reclassification conditions.

  The hierarchizing unit is a unit that, when it is determined that reclassification is necessary, performs classification by the classification processing unit again on the learning data belonging to the category. Thereby, the category is reclassified into a plurality of lower categories.

  According to this configuration, the self-organizing map corresponding to the learning data is formed hierarchically. Whether or not the lower classification is performed is determined by classification characteristic determination, and thus a category having an appropriate number and classification characteristic according to the learning data and the determination criterion is formed. That is, a feature space corresponding to the feature of the learning data is formed in a self-organizing manner.

  Note that new classification target data as well as learning data can be classified using the formed classification space. That is, it is possible to classify the classification target data by performing classification using the self-organizing map in order from the upper hierarchy. Since this classification process can be performed by calculation for the number of layers, it is also suitable for real-time processing.

  When the hierarchizing means causes the classification processing means to perform classification, the same number of map layer units is usually used regardless of the depth of the hierarchy. This is effective from the viewpoint of performing classification with the same degree of accuracy regardless of the depth. However, it is possible to change the number of map layer units to be used. For example, when the number of map layer units is increased in accordance with the depth of the hierarchy, it is possible to accurately analyze a relatively small difference.

  In one aspect of the classification apparatus according to the present invention, the map layer includes a plurality of map layer units in which joint weights are set, and in the learning calculation, the similar learning data are set to the same or nearby map layer units. The connection weight is changed so that the mapping is performed, the classification characteristic is a characteristic of the connection weight obtained by the learning operation, and the determination unit evaluates a correlation of the connection weight between the map layer units. Then, the necessity for reclassification is determined.

  The combined load is data corresponding to the phase of the input layer, and is set in each map layer unit. In the learning calculation, by adjusting the coupling weight, the competitive characteristic for the input data is changed, and consequently, the classification characteristic is changed. And a determination means determines the necessity for a reclassification by the correlation of the joint load between map layer units. The evaluation of the correlation may be an absolute evaluation or a relative evaluation of the correlations. By evaluating the correlation, for example, it is possible to evaluate the degree of similarity of classification characteristics within the same category and the degree of difference of classification characteristics between different categories. The determination means determines whether or not the lower classification should be performed by comparing such an evaluation result with an appropriate reference value. In this aspect, the classification processing means evaluates the combined load in the obtained self-organizing map and classifies the map layer. As a specific example, a mode in which a map layer is divided using a portion having a small correlation of coupling loads between adjacent map layer units as a classification boundary can be exemplified.

  In one aspect of the classification apparatus of the present invention, the determination means determines the necessity of reclassification of a category in the hierarchy based on the number of categories determined by classification up to a certain hierarchy. In other words, the classification characteristics of the self-organizing map are evaluated using the number of categories that have not been reclassified, and it is determined whether or not to perform further reclassification.

  In one aspect of the classification apparatus of the present invention, the classification processing unit classifies the map layer into two categories. By classifying into two categories, it can be roughly divided based on clear features. In this case, it is considered that a relatively small number of map layer units is sufficient. Since the specific number depends on parameters such as the quantity of learning data, it is desirable to determine the number by performing experiments as appropriate. For example, it may be about 20 or less, or even 10 or less. In addition, in the case of classification into two categories, it is considered sufficient that the arrangement of the map layer units is one-dimensional. This is expected to increase the calculation speed. The binary tree structure of the classification formed in this way represents the result of using the characteristics included in the learning data group as a classification criterion in descending order of contribution.

  In one aspect of the classification apparatus of the present invention, the means for classifying the classification target data into the hierarchical classification structure of the formed binary tree and the index representing the hierarchical classification structure of the binary tree are classified. Means for assigning to the classification target data; and search means for searching the classified target data using the index. The classification target data may be learning data used for learning, or may be new data. There are various known search algorithms using binary trees. Here, by using the binary tree structure obtained as a result of the classification for the search as it is, such a search algorithm can be used immediately.

  In one aspect of the classification device of the present invention, the learning data is sensor data generated based on a detection result of the sensor. In one aspect of the classification device of the present invention, the sensor data is human face image data captured by a camera. That is, a classification device that can be called a facial expression recognition device or a facial expression classification device is realized. In this case, the learning data is typically a plurality of face image data of the same individual. Thereby, the expression feature space unique to the individual can be formed. However, of course, data of a plurality of individuals may be classified together. In this case, there is a high possibility that an individual difference appears first, but there is a possibility that characteristic facial expressions are classified first.

  In one aspect of the classification apparatus of the present invention, facial expression analysis is performed on a visualized image generated from a combination load of a certain category, and semantic information characterizing this category is calculated with reference to a preset correspondence table. The calculating means to perform is provided. Semantic information may be calculated for all categories, or semantic information may be calculated only for the category of the last (lowest) hierarchy. Examples of facial expression analysis include rule-based analysis based on the movement of feature points and template-based analysis compared with template data. The semantic information is information that gives the meaning of a category, such as information about emotions such as joy and sadness, and copying information such as smiles and crying faces. By setting and storing semantic information corresponding to the facial expression analysis result in advance as a correspondence table, it is possible to calculate semantic information about the category.

  In one aspect of the classification apparatus of the present invention, an inspection unit is provided for inspecting whether or not conflicting semantic information has been calculated by the calculation unit from a category in an upper hierarchy and a category in a lower hierarchy. For example, when a category classified as joy is classified as anger and sadness in a lower hierarchy, it can be said that the categories are contradictory (usually, combinations that are contradictory in advance are regularly set and set) ). Therefore, we decided to provide inspection means to verify these contradictions. In addition, when the conflicting classification is made, for example, the process of increasing the number of learning data may be performed, or the classification of the lower category generated based on a small difference compared to the upper category is not adopted. Processing may be performed.

  In one aspect of the classification apparatus of the present invention, for the partial data forming part of the learning data, means for performing each processing by the classification processing means, the determination means, the hierarchy means, and the calculation means; Semantic information that characterizes this face image data by classifying new face image data into a classification structure based on the learning data and a classification structure based on the partial data, and comparing the semantic information obtained from each. Means for outputting. The determination processing of semantic information based on the comparison may be performed by, for example, majority (this is particularly effective when similar processing is performed on a plurality of partial data and semantic information is obtained from each of the partial data) You may carry out by the selection process of a common part. It is also effective to place more weight on the semantic information based on the learning data (wide range of data) than the semantic information based on the partial data (narrow range data).

  According to the present invention, it is possible to determine an appropriate number of categories in accordance with the characteristics included in the processing target data and classify into the categories. For example, when personal face image data is used as the processing target data, a facial expression feature space suitable for the individual can be formed.

  FIG. 1 is a block diagram illustrating a schematic configuration of a facial expression recognition system 10 according to the present exemplary embodiment. The facial expression recognition system 10 includes a facial expression recognition device 20, an imaging device 40 that inputs facial image data to the facial expression recognition device 20, and a display terminal 50 that displays a classification result by the facial expression recognition device 20.

  The facial expression recognition device 20 is usually formed using a computer such as a PC (personal computer). The facial expression recognition device 20 includes a signal input processing unit 22, a facial expression learning processing unit 24, a facial expression recognition processing unit 30, and a recognition result output unit 34. The signal input processing unit 22 inputs face image data from the imaging device 40 and outputs the facial image data to the facial expression learning processing unit 24 and the facial expression determination processing unit 32. At that time, the signal input processing unit 22 cuts out a still image from the moving image data as necessary, and performs area demarcation and size normalization.

  The facial expression learning processing unit 24 forms facial expression feature space using input facial image data as learning data. The facial expression learning processing unit 24 includes an facial expression feature space formation processing unit 26 and an facial expression feature space database 28. The expression feature space formation processing unit 26 has a built-in SOM (self-organizing map), and sequentially divides the classification result into two to form an expression feature space having a binary tree structure. The formed facial expression feature space is stored in the facial expression feature space database 28.

  In addition, the facial expression learning processing unit 24 can analyze what facial expression the classified facial expression feature space reflects. That is, by analyzing the visualized image representing the classified feature space, semantic information about basic facial expressions such as surprise, joy, sadness, fear, disgust, anger, and expressionlessness, and basic facial expressions are expressed in a composite manner. Semantic information indicating the meaning and semantic information expressing a finer facial expression than the basic facial expression are obtained. In addition, when the feature space reflects a difference in face orientation or a difference in shooting conditions, semantic information based on such difference can be given. Of course, it is not always necessary to provide semantic information to the feature space, but providing semantic information has the advantage of facilitating understanding of the classification results and facilitating search and use. The analysis for calculating the semantic information can be automatically performed using a conventionally known method. Examples of techniques include a template-based technique for preparing a plurality of templates corresponding to various facial expressions and determining similarity, and a rule-based technique for measuring the distance and area of facial organs. . And it becomes possible to obtain | require semantic information from an analysis result by rule-regulating matching with these analysis results and semantic information beforehand.

  The facial expression recognition processing unit 30 includes the facial expression determination processing unit 32 and also shares the facial expression feature space database 28 with the facial expression learning processing unit 24. The facial expression determination processing unit 32 classifies newly input facial image data according to the facial expression feature space formed in the facial expression feature space database 28. Specifically, classification is performed by inputting face image data to the formed SOM and repeating the process of which category is classified to the lowest SOM.

  The recognition result output unit 34 inputs the result of facial expression recognition by the facial expression recognition processing unit 30 and outputs the result to the display terminal 50. At that time, the recognition result output unit 34 organizes the data into a form that is easy to understand by the user, such as displaying the recognition results in a hierarchical manner or displaying the obtained semantic information.

  FIG. 2 is a flowchart showing a procedure for the expression learning processing unit 24 shown in FIG. 1 to form an expression feature space. When the processing is started, the facial expression feature space formation processing unit 26 sequentially inputs face image data from the signal input processing unit 22 (S10), and performs SOM learning processing (S12). SOM and its learning method will be described in detail later. Subsequently, the process of visualizing the combined load set in the map layer unit of the SOM is performed (S14), and the correlation coefficient of the combined load of the adjacent map layer unit is calculated (S16). Then, the map layer is divided into two parts between the map layer units having the smallest correlation coefficient (S18), and the division ratio of the further divided map layer is examined (S20). Here, it is assumed that there are five map layer units, and when the two division is performed at 4: 1 or 1: 4, the classification process is terminated without reclassifying one unit. (S22). On the other hand, for the 4 units when the 2 division is performed at 4: 1 or 1: 4, or for each unit when the 2 division is performed at 3: 2 or 2: 3, the classification process is performed. It becomes a target. The reason why the re-division is not performed for only one unit is that it can be considered that a sufficiently fine division has already been performed for one unit. However, depending on the general division depth and the number of map layer units, it may be better to redivide one unit, particularly in the upper hierarchy.

  Subsequently, the correlation coefficient of the coupling load between the map layer units that are the target of reclassification is calculated, and the value is evaluated (S24). This is because if the map layer units classified into the same category have almost the same combined load, the necessity for reclassification is small. Specifically, when the variation (relative magnitude difference) in the correlation coefficient between the map layer units is smaller than the reference value, it is determined that the classification is not necessary again, and the process ends (S22). ). On the other hand, when the variation in the correlation coefficient between the map layer units is large, it is determined that reclassification is necessary, and the processing is continued. Note that the determination can also be made based on the absolute size of a plurality of correlation coefficients. Specifically, an example in which reclassification is repeated until the correlation coefficient becomes a certain reference value or more can be given. Also, in order to judge the convergence / divergence of classification, it will be effective to evaluate how the absolute or relative magnitude of the correlation coefficient changes as it goes down, and perform reclassification judgment. . Specific judgment criteria in the judgment depend on the characteristics of the face image data, and may be determined experimentally and empirically.

  Finally, face image data for reclassification is extracted (S26). This is performed by inputting face image data to the SOM fixed with the final combined load and confirming which map layer unit has been mapped. And the process after step S12 is repeated with respect to the face image data used as the reclassification object. The classification result information is stored in the facial expression feature space database 28.

Next, the classification process will be described in more detail with reference to FIG. FIG. 3 is a schematic diagram showing an example of forming a facial expression feature space. The SOM is composed of two layers, an input layer 60 and a map layer 64. The input layer 60 includes the same number of input layer units (here, n) as the number of pixels of the face image data, and the i-th (1 ≦ i ≦ n) input layer unit includes the face image data. The pixel value x _i (0 ≦ x _i ≦ 255) of the 62nd i-th pixel is input.

The map layer 64 includes five map layer units U0, U1,. . , U4. Each map layer unit is associated with each input layer unit and has a coupling load corresponding to each input layer unit. That is, the j-th (1 ≦ j ≦ 5) map layer unit has a combined load represented by w _ij (0 ≦ w _ij ≦ 1). The magnitude of the value of the combined load w _ij is related to the pixel value.

At the start of SOM learning, the connection weight w _ij of each map layer unit is initialized using a random number. In the t-th learning, the pixel value x _i of the i-th pixel in the face image data is input to the i-th input layer unit. At this time, the Euclidean distance d _j between x _i and w _ij is calculated by the following equation.

Based on this calculation, a map layer unit that minimizes d _j , that is, a winner unit is searched. Next, the combined load of the map layer unit included in the winner unit and its neighboring area is updated by the following equation.

Here, α (t) is a learning rate coefficient (0 <α (t) <1). This series of processes is repeated the maximum number of learning times.

  In the present embodiment, the number of learnings t is empirically set to 200,000 times, and the neighborhood region is fixed in time as only the map layer unit adjacent to the winner unit. In addition, the initial value of α (t) was set to 0.5, and was linearly decreased to 0.02 when the number of learning was 100,000 and to 0 when the maximum number of learning was reached. Furthermore, the update rate of the combined load was set to 1 for the winner unit and 0.5 for the adjacent map layer unit.

The finally obtained value of the combined load w _ij of each unit of the map layer is converted to a value from 0 to 255, and a visualized image 66 is generated. Further, correlation coefficients r0, r1, r2, and r3 of the coupling load w _ij are calculated between the adjacent units in the map layer. In the figure, it is assumed that the correlation coefficient r3 is the smallest, and the map layer is divided into two parts between the map layer units U3 and U4. This is because the map layer units having the smallest correlation coefficient are considered to have the largest phase characteristic difference.

  Input groups classified into both of the two divided map layers can be regarded as constituting independent subproblems. Therefore, the classification process by SOM is recursively repeated for the input groups classified into both. However, in the illustrated example, since the map layer is divided into two parts between the map layer units U3 and U4, it can be said that the map layer unit U4 has a specific phase characteristic of the facial expression as compared with others. Therefore, the hierarchization process is terminated and handled as an independent category. On the other hand, for the face image data 72 classified into the four map layer units U0 to U3, if the correlation coefficients r0, r1 and r2 between them are smaller than the reference value, the classification is not yet sufficient, Classification processing is performed.

  In the illustrated example, as a result of the reclassification process, the division boundary 74 is set between the map layer units U2 and U3, and the face image data 72 is classified into two face image data 76 and 78. Similarly, the necessity of reclassification is determined for these face image data 76 and 78, and further classification is continued.

  Next, processing in the facial expression recognition processing unit 30 shown in FIG. 1 will be described with reference to FIG. FIG. 4 is a diagram showing the analysis result of the semantic information of the facial expression feature space stored in the facial expression feature space database 28. The figure shows the visualized images obtained by the first layer classification and the second layer classification. The visualized image is an image created from the combined load of each map layer unit U0 to U4. That is, when the illustrated visualized image or a similar image is input to the SOM input layer unit, it is classified into the map layer unit.

  In the first layer, the map layer unit U4 is classified from one unit and determined as an independent category. Therefore, the visualized image of the map layer unit U4 is compared with the visualized image of the adjacent map layer unit U3. Here, the comparison is performed according to AU of FACS (Facial Action Coding System). FACS is known as a representative facial expression analysis model, and qualitatively defines the characteristics of the movement of facial parts such as eyebrows, eyes, and mouth corresponding to facial expressions as AU (Action Unit).

  As a result, in the first layer, the visualized image of the map layer unit U4 has the AU1 “raise the inside of eyebrows” feature, the AU2 “raise the outside of eyebrows” feature, the AU5 “raise the upper eyelid” feature, AU26 It has been found that it has the features of “opening the lips by lowering the chin” and the “opening of mouth” feature of AU27. Then, by applying these features to the FACS classification, the map layer unit U4 was determined to have a surprising expression.

  Similarly, the visualized image of the map layer unit U4 classified into an independent category in the second layer includes the AU6 “raise the cheek” feature, the AU25 “open lips” feature, the AU11 It has been shown that it has the features of “deep nasal lip deep”, AU12 “pulls up the lip sharply”, and AU13 “picks up the lip sharply to swell the cheeks”. Based on the FACS classification, it was determined that the expression was a joyful expression.

  In the illustrated example, the operator visually analyzes the AU. However, for example, if a conventional technique for calculating a facial organ difference by setting feature points is used, a face classification result based on FACS can be automatically performed in exactly the same manner. Further, the automatic analysis is not limited to the technique using feature points, and can be performed using various general techniques such as a technique using a template. As a result of automatic analysis, for example, there is a possibility that it is judged as a complex facial expression that is mixed at a ratio of surprise and joy, in which case it means that it is a complex facial expression Information only needs to be given. Furthermore, here, the comparison was made between the map layer units at the boundary of the classification. However, it would be effective to include the entire category in the judgment target, for example, by comparing the average values of the two divided classifications. .

  FIG. 5 is a diagram illustrating the formation result of the facial expression feature space. The facial image data used here is for seven facial expressions (no expression, anger, disgust, fear, joy, sadness, surprise) subjectively expressed by the subject, and the normal indoor environment (fluorescence that is considered to be common in everyday life) It was acquired as a moving image (24 bit Color) under lighting with a lamp. The acquired video was converted to a grayscale still image. For each subject, about 50 to 60 images for each facial expression, a total of about 350 to 420 images were used as learning data. In this example, the classification process is terminated by the classification up to the fourth layer.

  First, in the first layer, an independent category representing surprise is constructed in the map layer unit U4. This is thought to be because the surprised expression has specificity compared to other expressions. In the second layer, an independent category representing pleasure is constructed in the map layer unit U4.

  The third layer is divided into categories of map layer units U0 and U1 and categories of map layer units U2 to U4. For these, the classification of the fourth layer (referred to as 4.1 layer and 4.2 layer, respectively) is performed. In the 4.1 layer, the correlation coefficient between the map layer units U2 and U3 is the lowest, the map layer units U0 and U1 are analyzed to be a fearful expression, and the map layer units U2 to U4 are analyzed to have no expression. Yes. On the other hand, in the 4.2 layer, the correlation coefficient between the map layer units U2 and U3 is the lowest, the map layer unit U0 is disgusted, the map layer units U1 and U2 are sad, and the map layer units U3 and U4 are angry. It is analyzed to represent.

  In this example, the classification process is finished in the fourth layer. However, when the classification to the lower layer is permitted, it is desirable to evaluate the classification characteristics of the fourth layer. For example, in the case of the 4.1 layer, U3 and U4 have similar classification characteristics, as can be seen from the fact that both are classified as expressionless. Therefore, the correlation load between the two combined loads will be relatively large. Therefore, it can be said that there is a high possibility that this category does not need to be reclassified (depending on the reference value). On the other hand, as shown by the fact that U0 to U3 are classified as feared or expressionless, the correlation coefficient of the combined load between them will be relatively small. Therefore, it can be said that there is a high possibility that this category needs to be reclassified.

  Subsequently, a classification result for another subject will be described with reference to FIG. 6A is a diagram showing the facial expression feature space analyzed for the subject of FIG. 5 and FIGS. 6B to 6D are three subjects that are different from FIG. Here, for the sake of simplicity, the classification is suspended up to the third layer.

  In the case of the subject in FIG. 6A, all the face image data 100 is classified into the category 102 and the category 104 in the first layer. The category 104 is composed of independent map layer units and has been analyzed as a surprising expression. On the other hand, the category 102 is classified into a category 106 and a category 108 in the second layer, and the category 108 including independent map layer units is analyzed to be an expression of joy. The category 106 is classified into a category 110 representing fear or no expression in the third layer and a category 112 representing disgust, sadness, and anger. In this way, in the case of FIG. 6A, a facial expression feature space consisting of four categories is formed by classification up to the third layer.

  In the case of the subject in FIG. 6B, a category 122 representing a surprised expression is obtained in the first layer, and a disgusting category 124, a joy category 126, a sadness category 128, and anger and fear are obtained in the third layer. The category 130 including no expression is obtained. That is, in the case of FIG. 6B, a facial expression feature space composed of five categories is formed by classification up to the third layer.

  In the case of the subject in FIG. 6C, a category 142 representing a surprised expression in the second layer and a category 144 of no expression are obtained, and an anger category 146, a fear and aversion category 148 in the third layer, A joy category 150 and a sadness category 152 are obtained. That is, in the case of FIG. 6C, a facial expression feature space composed of six categories is formed by classification up to the third layer.

  In the case of the subject of FIG. 6 (d), the category of pleasure 162 is obtained in the second layer, the category of fear 164, the category of disgust 166, the category of sadness 168, the category of anger 170, surprise and The expressionless category 172 and the expressionless category 174 are obtained. That is, in the case of FIG. 6D, a facial expression feature space composed of seven categories is formed by classification up to the third layer.

  Thus, even if the number of layers is the same, the number of feature spaces and the semantic information thereof are different for each subject. That is, there are individual differences in the facial expressions of subjects, and the similarities between facial expressions are also different. Therefore, it can be understood that the number of categories to be classified and the type of category to be classified cannot be determined in a general manner. In spite of this property, according to the method of the present embodiment, classification is performed in descending order of feature difference, and a facial expression feature space suitable for the individual can be formed. It also functions effectively in the classification of facial expressions in which multiple emotions are mixed.

  In the example shown in FIG. 6, the depth of classification is limited to the third layer, but it is also possible to classify deeper, and the depth is dynamically determined based on the classification result. It is also possible to do. As an example in which the depth is dynamically determined, a mode in which the next layer is classified until the number of categories exceeds a certain value (for example, about 6 to 8) can be given. In the case of FIG. 6 (a), only four categories are obtained by the third layer, and it would be effective to classify one layer deeper. As another example of dynamically determining the depth, as mentioned in the description of FIG. 5, the category is reclassified based on the correlation coefficient value of the combined load between the map layer units in the category. An aspect of determining whether or not it is necessary can be given. In the case of FIG. 6A, dislike, sadness, and anger are mixed in the category 112, and the correlation coefficient between them is generally large. Therefore, in such a case, it would be effective to classify further (at least the category).

  FIG. 7 is a flowchart showing the flow of processing when new facial image data is classified using the determined facial expression feature space. This process is performed using the facial expression recognition processing unit 30 shown in FIG.

  In the process, first, the facial expression determination processing unit 32 loads data of the facial expression feature space of the individual to be classified from the facial expression feature space database 28 (S40). Then, face image data cut out from the moving image at an appropriate sampling interval is input from the signal input processing unit 22 (S42). The face image data is first mapped to the first map layer (S44), and it is determined to which category it is mapped (S46). If the category is not the last layer (S48), it is mapped again to the map layer one level below (S50). On the other hand, when the classified category is the lowest layer, information indicating that the category is classified is output together with the semantic information assigned to the category (S52).

  Finally, a modification of the classification process for face image data will be described. FIG. 8 is a diagram for explaining processing performed by dividing face image data into a plurality of parts.

  In general, facial expressions appear as changes in the entire face, but the degree of change is not necessarily uniform across the entire face. Moreover, even if the facial expression is the same, a portion where a prominent feature appears may be different due to individual differences. Therefore, here, the analysis is performed using the entire face image and the partial image.

  In the illustrated example, an entire face area image 202, an upper face area image 204, and a lower face area image 206 are cut out from the captured head area image 200. The entire face area image 202 is an image including the entire face including the eyebrows, eyes, nose, and mouth. The upper face region image 204 is an image including eyebrows and eyes, and the lower face region image 206 is an image including a nose and mouth. These images can be cut out automatically using, for example, a pattern matching technique.

  The expression feature space is formed for each region image using the self-organizing map. That is, first, learning using a plurality of whole face area image 202 is performed, a face whole area expression feature space 208 is obtained, and semantic information is given. Similarly, the upper face area facial expression feature space 210 and its semantic information are given by learning using the upper face area image 204, and the lower face area facial expression feature space 212 and its semantic information are given by learning using the lower face area image 206. Is given.

  When new facial image data is classified using these facial expression feature spaces, an entire face region image, upper face region image, and lower face region image are created from the face image data, and each feature space is created. Make a determination using. As a result, an entire face area facial expression determination result 214, an upper face area facial expression determination result 216, and a lower face area facial expression determination result 218 are obtained.

  Finally, a facial expression recognition result 220 is obtained by comprehensively judging these. Various judgments can be made. For example, a mode in which a majority decision is made and the semantic information of the dominant facial expression among the three determination results is employed. In addition, for example, the results for the whole face area are set as the basic classification. For facial expressions that cannot be subdivided well in the whole face area, if there is a result that can clearly classify these in the upper face area or lower face area, The aspect to utilize can also be mentioned. In this way, it is possible to perform facial expression recognition corresponding to individual characteristics in detail by setting rules as to which result is used at which stage among the three.

  The facial expression recognition technique described above can be applied to various fields. For example, by storing the facial expression feature space, it is possible to perform personal authentication (representative vector extraction method) that is not affected by facial expressions. Also, when performing computer input by face pattern recognition (for example, pointing device operation), it is possible to use a face pattern suitable for an individual, so that it is particularly effective in the medical field and the welfare field. I can expect that. The technology will also be useful in fields that collect emotional information such as product monitoring and psychological tests.

  Furthermore, the technique of the present embodiment can be diverted to various data classifications by targeting data other than face image data. Examples of data include general image data other than face image data, audio data, and the like.

It is a schematic functional block diagram which shows the system configuration | structure of this Embodiment. It is a flowchart which shows the process at the time of formation of an expression feature space. It is a figure which shows the process outline | summary at the time of formation of an expression feature space. It is a figure explaining the process which determines semantic information by analysis of a facial expression. It is a figure which shows the example of the formation result of expression feature space. It is a figure which shows the facial expression feature space of four different test subjects. It is a flowchart which shows the process of facial expression recognition with respect to new face image data. It is a figure which shows the process which integrates the facial expression recognition of the whole face, the upper face, and the lower face.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 facial expression recognition system, 20 facial expression recognition apparatus, 22 signal input processing part, 24 facial expression learning processing part, 26 facial expression feature space formation processing part, 28 facial expression feature space database, 30 facial expression recognition processing part, 32 facial expression judgment processing part, 34 recognition Result output unit, 40 imaging device, 50 display terminal, 60 input layer, 62 face image data, 64 map layer, 66 visualized image, 72 face image data, 74 division boundary.

Claims (12)

Classification processing means for performing a learning operation for sequentially mapping a plurality of learning data to a map layer, evaluating classification characteristics of the obtained self-organizing map, and classifying the map layer into two or more categories; ,
Determining means for evaluating the classification characteristics to determine the necessity of reclassification of a certain category;
A hierarchizing unit that causes the classification processing unit to perform classification again for the learning data belonging to the category when the determination unit determines that reclassification is necessary;
A classification apparatus comprising: The classification device according to claim 1,
The map layer includes a plurality of map layer units in which a combined load is set,
In the learning operation, the joint weight is changed so that similar learning data are mapped to the same or neighboring map layer units,
The classification characteristic is a characteristic about the connection weight obtained by the learning operation,
The determination unit is a classification device that evaluates the correlation of the coupling load between the map layer units and determines the necessity of reclassification. The classification device according to claim 1,
The determination unit is a classification device that determines the necessity of reclassification of a category in a hierarchy based on the number of categories determined by classification up to a certain hierarchy. The classification device according to claim 1,
The classification processing means classifies the map layer into two categories. The classification device according to claim 4,
Means for classifying the data to be classified into a hierarchical classification structure of the formed binary tree;
Means for giving an index representing a hierarchical classification structure of a binary tree to the classified data to be classified;
Search means for searching for the classified data classified using the index;
A classification apparatus comprising: The classification device according to claim 1,
The classification device, wherein the learning data is sensor data generated based on a detection result of the sensor. The classification device according to claim 6,
The sensor data is a classification device that is human face image data captured by a camera. The classification device according to claim 7,
A classification apparatus including a calculation unit that performs facial expression analysis on a visualized image generated from a combined load of a certain category and calculates semantic information that characterizes the category with reference to a preset correspondence table. The classification device according to claim 8, wherein
A classification apparatus comprising inspection means for inspecting whether or not conflicting semantic information is calculated by the calculation means from an upper hierarchy category and a lower hierarchy category. The classification device according to claim 8, wherein
Means for performing each processing by the classification processing means, the determination means, the hierarchy means, and the calculation means for partial data forming a part of the learning data;
Semantic information that characterizes this face image data by classifying new face image data into a classification structure based on the learning data and a classification structure based on the partial data, and comparing the semantic information obtained from each. Means for outputting
A classification apparatus comprising: Computer
Classification processing means for performing a learning operation for sequentially mapping a plurality of learning data to a map layer, evaluating classification characteristics of the obtained self-organizing map, and classifying the map layer into two or more categories; ,
Determining means for evaluating the classification characteristics to determine the necessity of reclassification of a certain category;
A hierarchizing unit that causes the classification processing unit to perform classification again for the learning data belonging to the category when the determination unit determines that reclassification is necessary;
Classification program to make it function. A method performed by a computer,
A classification processing procedure for performing a learning operation for sequentially mapping a plurality of learning data to a map layer, evaluating classification characteristics of the obtained self-organizing map, and classifying the map layer into two or more categories; ,
A determination procedure that evaluates the classification characteristics to determine the need for reclassification of a category;
A hierarchizing procedure for reclassifying the classification processing procedure for the learning data belonging to the category when it is determined by the determination means that reclassification is necessary;
Classification method including