JP2014048722A - Image classification device and image classification method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To classify a plurality of pieces of face image data in image data of a classification object to an appropriate classification destination.SOLUTION: An image classification device 1 is provided that comprises: a similarity input part 14 that inputs information showing similarity between a plurality of pieces of face image data and a face image data group; a classification part 16 that classifies the face image data on the information showing the similarity input by the similarity input part 14 to the face image data group; a classification determination data exclusion part 17 that excludes the classified face image data from a classification destination as classification determination face image data, and excludes the face image data group the classification destination of the classification determination face image data from the classification destination as the classification determination face image data group; and a repeat processing control part 18 that controls so as to repeat processing by the classification part 16 and the classification determination data exclusion part 17.

Description

  The present invention relates to an image classification device and an image classification method.

  Conventionally, a method of classifying human face image data included in image data into one of face image data groups related to pre-registered person face image data for the purpose of appropriately organizing image data, etc. It has been known. The classification method calculates a probability (similarity) that a face image data of a person included in the image data to be classified is similar to a face image data group registered in advance, and the similarity is The face image data to be classified is classified into face image data groups of a certain level or more.

  Specifically, as a method of classifying face image data to be classified into face image data groups, after specifying a range of face image data from image data, face image data and registered face image data groups Is known, and when the similarity exceeds a certain threshold, a method of classifying face image data into the face image data group is known (see, for example, Patent Document 1). Here, the similarity is calculated by determining the feature quantity indicating the shape of the eyes, mouth, etc., and calculating the reciprocal of the difference value of each feature quantity between the face image data and the registered face image data group. Is called.

JP 2010-27034 A

  However, the conventional method simply classifies the face image data (classification target) having a certain degree of similarity with the face image data group which is a candidate for classification into a corresponding face image data group, and classifies the image by the APL logic. It is. Thus, for example, the image data to be classified is a photograph, and there are a plurality of face image data (a plurality of persons) in the photograph, and the plurality of face image data have a certain degree of similarity with the same face image data group. In such a case, the plurality of face image data are classified into the same face image data group. Usually, a plurality of face image data in the same photo are related to different persons and should not be classified into the same face image data group, but the degree of similarity is simply constant as in the conventional method. When the classification destination is determined based on whether or not the above is true, such a situation may occur that a plurality of face image data of the same photograph is classified into the same face image data group.

  The present invention has been made in view of the above circumstances, and provides an image classification apparatus and an image classification method capable of classifying a plurality of face image data in image data to be classified into appropriate classification destinations. Objective.

  An image classification device according to the present invention is an image classification device that classifies a plurality of face image data in image data based on similarity, and includes a plurality of face image data in image data to be classified and a classification destination. The similarity input means for inputting information indicating the similarity to the classified face image data group, and the face image data to be classified based on the information indicating the similarity input by the similarity input means. Classifying means for classifying the face image data group that is the classification destination, and the face image data to be classified classified by the classification means are excluded from the classification target as classification determined face image data, and the classification target of the classification determined face image data is classified A classification finalized data excluding unit that excludes the face image data group from the classification destination as a classification finalized face image data group, a classification unit, and a classification finalized data excluding unit Comprising a repetition processing control means for controlling so as to repeat the process, the.

  In this image classification device, face image data classified as a classification destination based on similarity among a plurality of face image data in the image data is excluded from classification targets as classification-confirmed face image data, and the classification-confirmed face The face image data group of the classification destination related to the image data is excluded from the classification destination as a classification determined face image data group. By excluding the classification-confirmed face image data group from the classification destination, the classification target face image data that has not yet been classified is classified into the classification-confirmed face image data group that is the classification destination related to the classification-confirmed face image data. Can be prevented. That is, it is possible to prevent a plurality of face image data included in the image data from being classified into the same face image data group. Since the plurality of face image data included in the same image data is usually considered to be related to different persons, control is performed so that the plurality of face image data included in the image data is not classified into the same face image data group. This can contribute to classifying the face image data into an appropriate classification destination.

  As described above, according to the present invention, it is possible to classify a plurality of face image data in image data to be classified into appropriate classification destinations.

  In the image classification device according to the present invention, it is preferable that the classification unit excludes information whose similarity score is less than a predetermined threshold from information indicating similarity.

  By excluding information whose similarity does not satisfy a predetermined condition, inappropriate classification can be prevented, and classification into an appropriate classification destination of a classification target can be made more reliable.

  In the image classification device according to the present invention, the iterative processing control unit classifies all of the plurality of face image data to be classified into a face image data group, or classifies into any of the face image data groups. It is preferable to perform control so that the processing by the classifying unit and the classification finalized data excluding unit is repeated until it is determined that the data cannot be classified.

  By performing the process relating to the determination of the classification destination for all the face image data, the classification of the classification target to the appropriate classification destination can be made more reliable.

  The image classification apparatus according to the present invention preferably further includes a face image data group setting unit that newly sets the classification target face image data that is unclassifiable data as a face image data group that is a classification destination. .

  The non-classifiable data for which the classification destination does not exist can be set as a new face image data group so that it can be further classified when classifying the face image data of different image data. In this way, by classifying face image data of a classification target for which no suitable classification destination exists into a new face image data group, it is possible to further reliably classify the classification target to an appropriate classification destination. Can do.

  In the image classification device according to the present invention, date information is set in the face image data group that is the classification destination, and the creation date of the image data that is the classification target and the face image data group that is the classification destination are set. It is preferable to further include an inappropriate classification destination exclusion unit that compares the date information with the date and time information and excludes the face image data group related to the date information from the classification destination when the creation date is earlier than the date information.

  For image data created before specific date and time information of a face image data group, set the date and time information that can be determined that the image data cannot be classified into the face image data group in the face image data group. Thus, inappropriate classification can be prevented, and classification to an appropriate classification destination of a classification target can be made more reliable.

  The image classification apparatus according to the present invention derives image data input means for inputting image data to be classified, and facial feature amounts of a plurality of face image data in the image data input by the image data input means. It is preferable that the image processing apparatus further includes similarity calculation means for calculating the similarity between the plurality of face image data and the face image data group that is the classification destination based on the face feature amount.

  By inputting the image data to be classified and calculating the similarity between the plurality of face image data and the face image data group in the image data, the calculation of the similarity of the face image data to the classification destination Can be performed centrally.

  An image classification method according to the present invention is an image classification method for classifying a plurality of face image data in image data based on similarity, and includes a plurality of face image data in image data to be classified Based on the similarity input step for inputting information indicating the similarity to the classified face image data group that is the classification destination, and the information indicating the similarity input in the similarity input step, the face image to be classified The classification step of classifying the data into the face image data group that is the classification destination, and the face image data of the classification target classified in the classification step are excluded from the classification target as the classification fixed face image data, and the classification fixed face image data A classification decision data exclusion step for excluding the classification target face image data group from the classification destination as a classification decision face image data group, and a classification step Including a repetitive processing control step of controlling so as to repeat the processing in the fine classification determined data eliminating step. This image classification method can achieve the same effect as the above-described image classification device.

  According to the present invention, it is possible to provide an image classification device and an image classification method capable of classifying a plurality of face image data in image data to be classified into appropriate classification destinations.

It is a figure which shows the function structure of the image classification device which concerns on embodiment of this invention. It is a figure which shows the outline | summary of the classification | category process etc. of the image classification apparatus which concerns on embodiment of this invention. It is a figure which shows the new face image data group setting image of the image classification device which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the image classification device which concerns on embodiment of this invention. It is a figure which shows the processing flow of the image classification device which concerns on embodiment of this invention. It is a figure which shows the classification processing content of the image classification device which concerns on embodiment of this invention. It is a figure which shows the classification processing content of the image classification device which concerns on embodiment of this invention. It is a figure which shows the classification processing content of the image classification device which concerns on embodiment of this invention.

  Hereinafter, preferred embodiments of an image classification device and an image classification method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  A user who obtains image data by taking a picture with a digital camera or the like usually classifies and organizes the image data. For example, when a plurality of people are photographed as subjects, it is conceivable to collect photographs of the persons for each person who is the subject. It is very time consuming if the user confirms the person in each photo and classifies and organizes the person.

  An image classification apparatus according to an embodiment of the present invention reduces the classification work by a user by automatically classifying and organizing image data, and enables easy browsing of a target image when browsing image data. It is. The image data classification device is, for example, a server operated by a business operator. The image classification device is connected to a terminal used by a user via a network such as a mobile communication network, and can transmit / receive information to / from each other. The image classification device automatically classifies image data (for example, photos) uploaded by the user to the image classification device. The image data automatic classification method by the image classification apparatus according to the present invention classifies face image data included in image data into a registered face image data group of the same person as the person related to the face image data. .

  First, the function of the image classification device 1 according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a diagram showing a functional configuration of an image classification device 1 according to an embodiment of the present invention. FIG. 2 is a diagram showing an outline of classification processing and the like of the image classification device according to the embodiment of the present invention. As shown in FIG. 1, the image classification apparatus 1 according to the present embodiment includes an image data input unit 11, a face image data group storage unit 12, a similarity calculation unit 13, a similarity input unit 14, and an inappropriate classification. A destination exclusion unit 15, a classification unit 16, a classification determination data exclusion unit 17, a repetition processing control unit 18, a face image data group setting unit 19, a classification result output unit 20, and a classification state storage unit 21. It is prepared for. The image classification device 1 is, for example, a server operated by a business operator. The image classification device 1 classifies a plurality of face image data included in the image data based on the similarity.

  The image data input unit 11 is an image data input unit that inputs image data to be classified. Here, the image data is, for example, photo data taken by a user with a digital camera. In the image data, the creation date is automatically set. When the image data is a photograph taken with a digital camera, additional information (exif information, etc.) such as the shooting date and time is set at the time of shooting. Is done. The image data input unit 11 is created by a user (if the image data is a photograph taken with a digital camera or the like, the user of the digital camera) and then uploaded to the server (image classification apparatus 1) via the Internet, for example. The obtained image data is input as a classification target. In the present embodiment, as shown in FIG. 2, the image data input unit 11 inputs image data 30 including a plurality of face image data.

  The face image data group storage unit 12 stores a classified face image data group that is a classification destination of face image data (classification target) included in the image data input by the image data input unit 11. Here, the face image data is face image data relating to a person included in the image data. For example, the face image data included in the image data is identified by searching for the face image data using a template prepared for detecting the face image data with respect to the image data, and having high correlation with the template in the image data. This is done by specifying the indicated portion as face image data. The face image data group is a group composed of one or more face image data, and a plurality of face image data included in the same face image data group is assumed to be related to the same person. Therefore, the face image data in the same face image data group have a certain degree of similarity with each other. It should be noted that persons related to different face image data groups are not necessarily different persons, and may be the same person depending on the face orientation (details will be described later) of the face image data included in these face image data groups. is there. The face image data groups included in the face image data group table can be distinguished by group id. The group id can be automatically set from the youngest number when the face image data group table is set, for example.

  The similarity with respect to the face image data is calculated based on the face feature amount. Here, the face feature amount refers to a feature amount indicating a facial feature such as the shape and contour of each part of the face such as eyes and mouth, and is digitized and digitized for each facial feature. It is represented by a vector whose element is the value. Regarding the degree of similarity regarding the face image data, it is determined that the face image data of the similarity determination target is more similar as the face feature amount vector between the face image data of the similarity determination target is closer. A conventional method can be used to calculate the similarity of face image data based on such face feature amounts.

  The face image data group storage unit 12 stores face image data groups by a face image data group table. In the example illustrated in FIG. 2, the face image data group table 201 includes face image data groups 101 to 105 having group ids 101 to 105, and each face image data group has an average feature. Amount (front), average feature amount (45 degrees left), average feature amount (45 degrees right), and birthday data are managed.

  The average feature amount indicates an average value of values related to the face feature amount of each face image data included in the face image data group. Note that the average feature amount of the face image data group composed of only one face image data is the face feature amount of the one face image data. Front, 45 degrees to the left, and 45 degrees to the right are the face orientations (face orientation) of the face image data, and the similarity calculation unit 13 described below is used only for face image data having the same face orientation. Calculate similarity of image data. The face image data group 101 to the face image data group 105 of the face image data group table 201 shown in FIG. 2 have an average feature value (front) value of “Yes”. The similarity is calculated only for certain face image data. In FIG. 2, only the fact that the average feature value (front) data is “Yes” is managed, but in practice, a specific value of the average feature value needs to be managed. . Further, the face feature amount of each face image data and the face image data itself of each face image data included in the face image data group may be managed. Further, the data relating to the birthday is information indicating the date of birth of the person related to the face image data group. Note that the date of birth may be managed including not only the date but also information related to the time.

  The face image data group storage unit 12 adds the face image data belonging to the existing face image data group when the classification result output unit 20 described later outputs the classification result, and sets a new face image data group. Is done.

  The similarity calculation unit 13 derives face feature amounts of a plurality of face image data in the image data input by the image data input unit 11, and is based on the face feature amounts and a plurality of face image data and a classification destination. It is a similarity calculation means for calculating the similarity with the face image data group. The face feature amount of the face image data group is the above-described average feature amount, and is an average value of the face feature amounts of the face image data included in the face image data group.

  The similarity calculation unit 13 first specifies face image data included in the input image data (face extraction). Specifically, the face image data is searched for the image data using a template prepared for detecting the face image data, and the portion of the image data that shows a high correlation with the template is specified as the face image data. To do. The similarity calculation unit 13 specifies the face orientation (outplane) when specifying the face image data. For example, any one of front, 45 degrees to the right, and 45 degrees to the left is specified as the face orientation. When calculating the similarity, the similarity is calculated only between images having the same face orientation.

  In the example of FIG. 2, image data 30 (photographed on April 19, 2010 by a digital camera) is input from the image data input unit 11, and the similarity calculation unit 13 includes a plurality of faces included in the input image data 30. Face image data A, face image data B, face image data C, face image data D, and face image data E, which are image data, are specified. Then, the similarity calculation unit 13 specifies that the face direction of each of the face image data A to the face image data E is the front.

  The similarity calculation unit 13 derives facial feature amounts of the detected face image data A to face image data E. Here, the conventional face recognition method can be used for the derivation of the face feature amount. After deriving the facial feature amounts of the facial image data A to facial image data E, the respective facial feature amounts and the facial feature amounts of the facial image data groups 101 to 105 stored in the facial image data group storage unit 12 are calculated. Similarity is calculated (similarity determination). Note that the degree of similarity can be represented by a numerical value as a score. Here, the face feature amount of the face image data group is an average feature amount (the average of the face feature amounts of the face image data included in the face image data group), and the similarity to each face image data is the average feature amount. It shall be calculated from In the present embodiment, it is assumed that the similarity between the face image data A to the face image data E and the face image data group 101 to the face image data group 105 is as shown in the similarity list 40 of FIG.

  The similarity input unit 14 is a similarity input unit that inputs information indicating the similarity between a plurality of face image data in the image data to be classified and the classified face image data group to be classified. That is, the similarity input unit 14 inputs information indicating the similarity between the plurality of face image data and the face image data group calculated by the similarity calculation unit 13. Note that the plurality of image data includes processing included in one image data. That is, classification of face image data, which will be described later, is performed for each image data.

  In the example illustrated in FIG. 2, the similarity calculation unit 13 calculates the similarity between the face image data A to the face image data E and the face image data group 101 to the face image data group 105 as illustrated in the similarity list 40. Calculated. The similarity input unit 14 inputs each similarity shown in the similarity list 40.

  The inappropriate classification destination exclusion unit 15 sets date / time information for the face image data group that is the classification destination, the creation date of the image data that is the classification target, and the date / time information set for the face image data group that is the classification destination. If the creation date is earlier than the date / time information, it is an inappropriate classification destination exclusion means for excluding the face image data group related to the date / time information from the classification destination.

  The date / time information set for the face image data group is information set by accepting a user input, specifically, a birthday of a person related to the face image data group. Note that the timing at which the user input is received may be, for example, the timing at which the face image data group is created. The creation date of the image data to be classified is automatically set when the image data is created. When the image data is a photograph taken with a digital camera, the creation date of the image data is determined from information (exif information) such as the shooting date and time set at the shooting timing.

  The inappropriate classification destination exclusion unit 15 obtains the date and time information set for each face image data group and the creation date of the image data from the similarity calculation unit 13 to obtain the creation date of the image data to be classified. Comparison with date / time information set in the face image data group that is the classification destination is performed.

  In the example shown in FIG. 2, image data 30 including face image data A to face image data E is a photograph taken on April 19, 2010. That is, the image data 30 is created on April 19, 2010. On the other hand, among the face image data groups that are the classification destinations, the face image data group 102 has a birthday of a person related to the face image data group 102 on April 20, 2010. Therefore, since the creation date of the image data 30 is before the birthday of the person related to the face image data group 102, the face image data group 102 is excluded from the classification destination by the inappropriate classification destination exclusion unit 15.

  The exclusion of the face image data group that is the classification destination by the inappropriate classification destination exclusion unit 15 is performed before the classification process by the classification unit 16 described later. The inappropriate classification destination exclusion unit 15 notifies the excluded face image data group to the classification unit 16 so that it is not used for the classification process by the classification unit 16. Note that date information may not be set in the face image data group. When the date and time information is not set for the face image data group, the classification destination (face image data group) is not excluded by the inappropriate classification destination exclusion unit 15 and the processing by the classification unit 16 is performed. In addition, date information may be set only for a part of the face image data groups. In this case, the process by the inappropriate classification destination exclusion unit 15 is performed only for the face image data group for which date / time information is set.

  The classification unit 16 classifies face image data so that a plurality of face image data included in the same image data is not classified into the same classification destination (classified as the same person).

  The classification unit 16 is a classification unit that classifies the face image data that is the classification target into the face image data group that is the classification destination based on the information indicating the similarity input by the similarity input unit 14. Here, based on the information indicating the similarity, for example, is based on the height of the similarity score. When classifying the face image data that is the classification target based on the high score of similarity, the face image having the highest similarity score with the face image data group among all the face image data that is the classification target The data is classified into the face image data group.

  In the example illustrated in FIG. 2, when the similarities shown in the similarity list 40 are input by the similarity input unit 14, the classification unit 16 includes each face image data to be classified, each face image data group, and If the information indicating each degree of similarity related to the face image data group 102 excluded from the classification destination by the inappropriate classification destination exclusion unit 15 is excluded, the similarity between the face image data B and the face image data group 104 0.97 (97%) is determined to have the highest similarity score, and the face image data B is classified into the face image data group 104.

  Each time the classification unit 16 classifies the face image data into the classification destination (classification of one face image data into one face image data group), the classification confirmed face by the classification confirmed data exclusion unit 17 described later. Exclusion of the image data and the classification determined face image data group is performed. Here, the exclusion of the classification-confirmed face image data and the classification-confirmed face image data group indicates each similarity degree related to the classification-confirmed face image data (similarity between the classification-confirmed face image data and all face image data groups). This means that information and information indicating each similarity degree related to the classification determined face image data group (similarity between the classification determination face image data group and all the face image data) are excluded. When the classification unit 16 classifies certain face image data, the next time the classification unit 16 classifies the face image data included in the same image data, Classification is performed in a state in which the information indicating the similarity and the information indicating the total similarity related to the classified face image data group are excluded.

  Further, when the face image data to be classified cannot be classified into any of the face image data groups to be classified, the classification unit 16 sets the face image data as unclassifiable data. Here, the face image data that cannot be classified into any of the face image data groups (non-classifiable data) is a classification-determined face image data group whose similarity to the face image data is not less than a predetermined threshold value. This refers to face image data in which no face image data group exists.

  Further, the classification unit 16 excludes information whose similarity score is less than a predetermined threshold from information indicating similarity. For example, the predetermined threshold value may be 0.90 (similarity).

  In the example shown in FIG. 2, when the similarity score threshold is 0.90 and each similarity shown in the similarity list 40 is input by the similarity input unit 14, the classification unit 16 The face image data group 105 (similarity 0.5) as the classification destination of the face image data A, the face image data group 103 (similarity 0.7) as the classification destination of the face image data B, and the face image data C Face image data group 101 (similarity 0.7) as a classification destination, face image data group 101 (similarity 0.7) as a classification destination of face image data E, and a face image as a classification destination of face image data E The data group 104 (similarity 0.7) and the face image data group 105 (similarity 0.7) as the classification destination of the face image data E are excluded from the classification destinations.

  The classification unit 16 also inputs data indicating similarity between each face image data that is a classification target and each face image data group that is a classification destination, which is input by the similarity input unit 14 (hereinafter, data indicating similarity). Is input to the classification state storage unit 21. An image of data indicating the similarity is shown in FIG. Then, when the classification destination of the face image data to be classified is determined, the classification unit 16 classifies the face image data determined by the classification destination of the data indicating the similarity stored in the classification state storage unit 21. Set completion flag. In the data indicating the similarity stored in the classification state storage unit 21, the classification unit 16 associates the face image data for which the classification completion flag is set and the classification target face image data group corresponding to the face image data. Manage. Further, the classification unit 16 also applies to the face image data that has become unclassifiable data of the data indicating the similarity stored in the classification state storage unit 21 even when there is face image data that has become unclassifiable data. And set the classification impossible flag. Each face image data may be managed by an index of the face image data. In this case, a classification impossible flag may be set for the index corresponding to the face image data. Note that the index of the face image data may be automatically set when the similarity calculation unit 13 specifies the face image data (face extraction).

  The classification unit 16 also includes information on the face image data determined by the classification destination (information relating the face image data and the face image data group of the classification destination, an index of the face image data determined by the classification destination), Information relating to unclassifiable data (such as an index of face image data that has become unclassifiable data) is input to a classification finalized data exclusion unit 17 and a repetitive processing control unit 18, which will be described later. It is conceivable that the input is performed at the timing when the classification destination of the face image data is determined (or the timing at which unclassifiable data exists).

  The classification unit 16 refers to the classification completion face image set by the classification confirmation data exclusion unit 17 described later, which is set in the data indicating the similarity stored in the classification state storage unit 21. Classification is performed by excluding information indicating each degree of similarity relating to the data and information indicating each degree of similarity relating to the classified face image data group. In this case, the classification unit 16 refers to the exclusion completion flag set in the data indicating the similarity stored in the classification state storage unit 21 at the timing of classification.

  The classification confirmation data exclusion unit 17 excludes the classification target face image data classified by the classification unit 16 from the classification target as classification confirmation face image data, and classifies the face image data group that is the classification destination of the classification confirmation face image data. Classification final data excluding means for excluding from a classification destination as a definite face image data group.

  Here, the face image data classified by the classification unit 16 is excluded from the classification target as the classification confirmed face image data. The face image data for which the classification destination is confirmed is classified into the confirmed classification destination. And after confirmation, it means that it will not be classified into different classification destinations. Similarly, excluding a classification-destination face image data classification target face image data group as a classification-confirmed face image data group from the classification destination means that the classification-confirmed face image determined as the classification-destination face image data classification destination This means that the data group should not be a classification destination of other face image data (classification target that has not been classified).

  The classification confirmation data exclusion unit 17 refers to the classification completion flag of the data indicating the similarity stored in the classification state storage unit 21, thereby classifying face image data (classification confirmation face image) classified by the classification unit 16. Data) exists. Alternatively, the classification determination data excluding unit 17 determines the information of the face image data determined by the classification destination from the classification unit 16 (information that associates the face image data with the face image data group of the classification destination, and determines the classification destination. It is also possible to determine whether classification-confirmed face image data exists based on the input of the face image data index or the like. Then, when the classification confirmed face image data exists, based on the data indicating the similarity stored in the classification state storage unit 21 or the input of the face image data information determined by the classification unit 16 from the classification destination. Thus, the classification determined face image data group that is the classification destination of the classification determined face image data is specified.

  Then, the classification confirmation data excluding unit 17 includes, in the data indicating the similarity stored in the classification state storage unit 21, information indicating each similarity related to the classification determination face image data, and the classification determination face image data group. An exclusion completion flag is set for the information indicating each similarity. The classification unit 16 refers to the exclusion completion flag of the data indicating the similarity stored in the classification state storage unit 21, so that information indicating each similarity related to the classification determined face image data and the classification determined face image data group are displayed. Classification can be performed by excluding the information indicating each degree of similarity.

  The classification determination data excluding unit 17 excludes the classification determination face image data from the classification target, and information indicating that the classification determination face image data group is excluded from the classification destination, and a repetitive processing control unit described later. 18 may be input. The input may be performed at the excluded timing.

  In the example shown in FIG. 2, when the face image data B is classified into the face image data group 104 by the classification unit 16, the classification confirmation data exclusion unit 17 classifies the face image data B classified by the classification unit 16 into the classification confirmation face. The image data is excluded from the classification target, and the face image data group 104 that is the classification destination of the face image data B is excluded from the classification destination as the classification determined face image data group. After the processing by the classification determination data exclusion unit 17, in the classification of the face image data related to the same image data 30, the face image data B that is the classification determination face image data is not a classification target, and the classification determination is performed. The face image data group 104 that is a face image data group is never a classification destination.

  The repetitive process control unit 18 is a repetitive process control unit that performs control so that the processes by the classification unit 16 and the classification determination data exclusion unit 17 are repeated. In addition, the iterative processing control unit 18 determines that all of the plurality of face image data to be classified are classified into face image data groups or cannot be classified into any of the face image data groups. Control until the processing by the classification unit 16 and the classification determination data exclusion unit 17 is repeated.

  The iterative processing control unit 18 refers to the classification completion flag and the classification impossibility flag of the data indicating the similarity stored in the classification state storage unit 21 to perform processing by the classification unit 16 and the classification determination data exclusion unit 17. Determine whether to repeat. That is, when all the face image data of the data indicating the similarity stored in the classification state storage unit 21 has either the classification completion flag or the classification impossible flag, control is performed so as not to repeat the process. When there is face image data that does not have either the classification completion flag or the classification impossibility flag, control is performed so as to perform repetitive processing. Note that the iterative processing control unit 18 receives information on the face image data determined by the classification destination from the classification unit 16 (information linking the face image data and the face image data group of the classification destination, and the classification destination is determined). Whether the processing by the classification unit 16 and the classification finalized data exclusion unit 17 is repeated based on input of information on the face image data) and information on the non-classifiable data (index of the face image data that has become non-classifiable data) It may be determined whether or not.

  In the example illustrated in FIG. 2, the classified image data excluding unit 17 excludes the divided face image data B from the classification target as the classified image data, and sets the face image data group 104 that is the classification destination of the face image data B. When the classification-confirmed face image data group is excluded from the classification destination, the iterative processing control unit 18 refers to the classification completion flag and the classification impossible flag. At this time, the classification destination is determined only for the face image data B, and the classification destination is not determined for the other face image data. Therefore, the face image data and the face stored in the classification state storage unit 21 are determined. As for the data indicating the degree of similarity with the image data group, it can be confirmed that the classification completion flag is set only for the face image data B among the face image data. Therefore, the iterative processing control unit 18 determines from the classification completion flag that there is face image data for which the classification destination has not been determined, and performs control so that the processing by the classification unit 16 and the classification determination data exclusion unit 17 is repeated.

  In addition, the iterative processing control unit 18 does not perform the iterative processing because all of the face image data of the data indicating the similarity stored in the classification state storage unit 21 has either a classification completion flag or a classification impossible flag. If the determination is made as described above, information relating to the face image data determined by the classification destination and the face image data group of the classification destination input from the classification unit 16 without performing the control of the repetitive processing will be described later. The result is input to the classification result output unit 20. In addition, the repetitive processing control unit 18 also acquires the facial feature amount of the facial image data, the facial image data itself, the index of the facial image data, and the like, and inputs them to the classification result output unit 20.

  The repetitive processing control unit 18 also indicates that there is non-classifiable data when face image data with a non-classifiable flag exists in the face image data of the data indicating the similarity stored in the classification state storage unit 21. And the information of the face image data which is the unclassifiable data (such as the index of the face image data that has become unclassifiable data) is input to the face image data group setting unit 19 described later.

  The face image data group setting unit 19 is a face image data group setting unit that newly sets classification target face image data that is non-classifiable data as a face image data group that is a classification destination. The face image data group setting unit 19 notifies the repetitive processing control unit 18 that there is non-classifiable data and information on the face image data that is the non-classifiable data (the face image data that has become non-classifiable data). When an index or the like) is input, a face image data group related to the face image data that is the unclassifiable data is newly set.

  The face image data group setting unit 19 inputs information on face image data that is unclassifiable data (such as an index of face image data that has become unclassifiable data) to the similarity calculation unit 13. When the input is made, the similarity calculation unit 13 inputs the face feature amount of the face image data, which is non-classifiable data, information on the face direction, and the face image data itself to the face image data group setting unit 19. Then, the face image data group setting unit 19 uses the face feature amount input from the similarity calculation unit 13 as the average feature amount of the face image data group, and the face direction input from the similarity calculation unit 13 as the face image data. A new face image data group is set as the face orientation of the group.

  Further, the face image data group setting unit 19 receives a user operation for setting the name and birthday of a person related to the face image data of the face image data group at the timing of setting a new face image data group. Note that the timing for accepting the user operation may be output by the classification result output unit 20 described later. Here, when outputting by the classification result output unit 20, the classification result output unit 20 indicates that a new face image data group (new classification destination of image data) has been set on a display such as a user terminal. Refers to the display timing. The face image data group setting unit 19 inputs information related to the new face image data group in which the average feature amount (face orientation) and the birthday are set to the classification result output unit 20.

  The setting image of the face image data group by the face image data group setting unit 19 will be described with reference to FIG. FIG. 3 is a diagram showing a new face image data group setting image of the image classification device 1 according to the embodiment of the present invention. Assume that the image data 50 input to the image data input unit 11 includes face image data F and face image data G. The face image data F has a face orientation of front, and the face image data G has a face orientation of 45 degrees to the right. Now, there are face image data groups 106 to 110 shown in FIG. 3 as face image data groups included in the face image data group table 202 stored in the face image data group storage unit 12. As for the face orientation, the face image data group 106, the face image data group 108, and the face image data group 110 are front, and the face image data group 107 and the face image data group 109 are 45 degrees to the right.

  Since the similarity calculation by the similarity calculation unit 13 is performed only between face image data having the same face orientation, the face image data G is obtained only from the face image data group 107 and the face image data group 109. Is calculated. As shown in FIG. 3, the similarity between the face image data G and the face image data group 107 is 0.50, and the similarity between the face image data G and the face image data group 109 is 0.79. When the predetermined threshold value for determining the similarity is 0.90, the classification unit 16 sets the face image data G as unclassifiable data. In this case, the face image data group setting unit 19 sets a new face image data group 111 shown in FIG. 3 based on the information of the face image data G.

  The classification result output unit 20 receives information of face image data determined by the classification destination input from the iterative processing control unit 18 (information relating the face image data and the face image data group of the classification destination, the face image Data face feature amount, the face image data itself, the index of the face image data, and the like, and information about the new face image data group input from the face image data group setting unit 19 (average feature amount, face orientation, Birthday, face image data belonging to the face image data group). As a result of the classification, new face image data is added to the existing face image data group, so that the average feature amount of each existing face image data group is changed. Therefore, the classification result output unit 20 inputs the classification result to the similarity calculation unit 13, and receives from the similarity calculation unit 13 the input of the facial feature amount of the classified face image data and the classified face image data itself. Then, the average feature amount of the existing face image data group is updated in consideration of the face feature amount of the classified face image data input from the similarity calculation unit 13. That is, an average feature amount that is an average of face feature amount values of all face image data of the face image data group including the face image data classified this time is calculated, and the average feature amount is updated. The output by the classification result output unit 20 is performed on the face image data group storage unit 12. That is, the classification result output unit 20 stores in the face image data group storage unit 12 information on the face image data determined by the classification destination (information relating the face image data and the face image data group of the classification destination, the face Face feature amount of the image data, the face image data itself, the index of the face image data, and the like, and information on the new face image data group input from the face image data group setting unit 19 (average feature amount, face orientation) , Birthday, face image data belonging to the face image data group). Then, the face image data group storage unit 12 updates the information of the existing face image data group based on the information output from the classification result output unit 20 (addition of the face image data classified this time, the average feature amount Update) and addition of a new face image data group (setting of average feature amount, face orientation, birthday).

  The classification result output unit 20 provides the user who has acquired the image data (for example, if the image data is a photograph taken with a digital camera, a user who has captured and uploaded the photograph) via the Internet or the like, The classification result may be displayed on a display such as a personal computer terminal. The classification result is displayed by displaying the face image data group related to the classified face image data in a folder format, and each folder has face image data belonging to the folder (face image data group). Is stored.

  Note that the folder for each face image data group is managed in units of users. Normally, other users' folders cannot be browsed, but for example, by performing authentication between users, It is good also as being able to browse a folder (sharing between users is possible).

  Next, the hardware configuration of the image classification device 1 will be described with reference to FIG. FIG. 4 is a diagram showing a hardware configuration of the image classification device 1 according to the embodiment of the present invention. As shown in FIG. 4, the image classification device 1 includes a computer including a CPU 131, a RAM 132 and a ROM 133 which are main storage devices, a communication module 134 for performing communication, and an auxiliary storage device 135 such as a hard disk. Configured as a thing. Each of the above-described functions is exhibited when these components are operated by a program or the like.

  Subsequently, a processing flow of the image classification apparatus 1 will be described with reference to FIGS. 2, 5, and 6 to 8. FIG. 5 is a diagram showing a processing flow of the image classification device 1 according to the embodiment of the present invention. 6-8 is a figure which shows the classification processing content of the image classification apparatus 1 which concerns on embodiment of this invention.

  The image input from the image data input unit 11 is the image data 30 shown in FIG. 2, and the face image data groups are the face image data groups 101 to 105 shown in FIG.

  First, as shown in S101 of FIG. 5, the image data 30 is input by the image data input unit 11 (S101, image data input step). Then, after the face image data A to face image data E included in the image data 30 are specified by the similarity calculation unit 13 and the face feature amounts and face orientations of the face image data A to face image data E are specified, Similarities between the image data A to the face image data E and the face image data group 101 to the face image data group 105 stored in the face image data group storage unit 12 are calculated (S102, similarity calculation step).

  Then, the similarity input unit 14 inputs the similarity between the face image data A to the face image data E and the face image data group 101 to the face image data group 105 (S103, similarity input step). A table in which the similarities are arranged for determining the classification destination is as shown in FIG. Each similarity shown in FIG. 6A and the similarity list 40 shown in FIG. 2 show the same similarity.

  Next, the unsuitable classification destination exclusion unit 15 excludes the face image data group 102 whose birthday related to the face image data group is later than the creation date of the image data 30 from the classification destination (S104, unsuitable). Classification destination exclusion step). Therefore, the table for determining the classification destination is the table shown in FIG. 6B in which the face image data group 102 is excluded from the table shown in FIG. If the date / time information is not set for the face image data group, the classification destination (face image data group) is not excluded (the process of S104) by the inappropriate classification destination exclusion unit 15. Further, when the threshold value of the similarity score that can be determined as the classification destination is set to 0.90, data whose similarity score is less than 0.90 is excluded by the classification unit 16. Therefore, the table shown in FIG. 6C obtained by excluding data whose similarity is less than 0.90 from the table shown in FIG. 6B is a table for determining the classification destination.

  Next, the classification unit 16 classifies the face image data having a high similarity score with the face image data group into the face image data group related to the similarity score (S105, classification step). In this case, as shown in FIG. 7, the table shown in FIG. 6C is a list sorted in the order of similarity (first sort: score, second sort: face image data). If the degree of similarity is the same, the face image data index (face_index) is small (for example, when there is face image data A to face image data E, face image data A has the largest face_index. You may sort preferentially in a similar manner. Then, the classification destination of the face image data B having the highest similarity score shown in FIG. 7 is determined as the face image data group 104.

  Next, the classification confirmation data exclusion unit 17 excludes the face image data B, which is the classification confirmation face image data, from the classification target, and excludes the face image data group 104, which is the classification confirmation face image data group, from the classification destination ( S106, classification determination data exclusion step). FIG. 8A shows a list in which the face image data B is excluded from the classification target and the face image data group 104 is excluded from the classification destination.

  Next, the iterative processing control unit 18 confirms whether all classification targets are classified or non-classifiable data by using the classification completion flag and the classification impossible flag, and determines whether or not to perform the repetition processing. (S107, repetitive process control step). At this time, only the face image data B has been determined as the classification destination, and no non-classifiable data has been generated. Therefore, the repetitive process control unit 18 controls to repeat the processes of S105 and S106.

  Then, returning to S105, the classification unit 16 determines that the face image data A having the highest similarity score among the data not excluded is classified into the face image data group 103. In step S106, the classification determination data exclusion unit 17 excludes the face image data A that is the classification determination face image data from the classification target, and the face image data group 103 that is the classification determination face image data group from the classification destination. Excluded. FIG. 8B shows a list in which the face image data A is excluded from the classification target and the face image data group 103 is excluded from the classification destination.

  By repeating the processes of S105 to S107, the face image data D is classified into the face image data group 101 (the list after exclusion is shown in FIG. 8C), and the face image data C is the face image data group. It is classified into 105 (the list after exclusion is shown in FIG. 8D). In this case, as shown in FIG. 8D, the face image data E is non-classifiable data because there is no face image data group having a similarity score exceeding 0.90.

  In the process of S107 after the face image data C is classified into the face image data group 105, referring to the classification completion flag and the classification impossible flag of the data indicating similarity stored in the classification state storage unit 21, the classification unit 16, it is determined that face image data A to face image data D have been classified, and face image data E is unclassifiable data. Therefore, in the process of S107, all the face image data matches the condition of either classified or unclassifiable data, and the process proceeds to S108.

  Next, it is determined by the repetitive processing control unit 18 whether there is unclassifiable data (S108). Now, since the face image data E is non-classifiable data, the process proceeds to S109.

  Then, the face image data group setting unit 19 newly sets a face image data group related to the face image data E (S109, face image data group setting step).

  Finally, by the classification result output unit 20, information of the face image data determined by the classification destination (information that associates the face image data with the face image data group of the classification destination, the facial feature amount of the face image data, the Information on the face image data itself, the index of the face image data, and the new face image data group input from the face image data group setting unit 19 (average feature, face orientation, date of birth, face image data group) Face image data) is output to the face image data group storage unit 12 (S110, classification result output step). The above is the description of the processing flow of the image classification device 1 according to the embodiment of the present invention.

  Next, operations and effects of the image classification device 1 according to the present embodiment will be described.

  In the image classification device 1 according to the present embodiment, among the plurality of face image data in the image data, the face image data classified as the classification destination by the classification unit 16 based on the similarity is classified by the classification determination data exclusion unit 17. The classification-confirmed face image data is excluded from the classification target, and the face image data group of the classification destination related to the classification-confirmed face image data is excluded from the classification destination as the classification-confirmed face image data group. By excluding the classification-confirmed face image data group from the classification destination, the classification target face image data that has not yet been classified is classified into the classification-confirmed face image data group that is the classification destination related to the classification-confirmed face image data. Can be prevented. That is, it is possible to prevent a plurality of face image data included in the image data from being classified into the same face image data group. Since the plurality of face image data included in the same image data is usually considered to be related to different persons, control is performed so that the plurality of face image data included in the image data is not classified into the same face image data group. This can contribute to classifying the face image data into an appropriate classification destination. As described above, according to the image classification device 1 according to the present embodiment, a plurality of face image data in the image data to be classified can be classified into appropriate classification destinations.

  Further, in the image classification device 1 according to the present embodiment, the classification unit 16 excludes information whose similarity score is less than a predetermined threshold from information indicating similarity, thereby dissimilar face image data groups Therefore, it is possible to prevent inappropriate classification such that the classification target is classified into the classification target, and to further reliably classify the classification target to an appropriate classification destination.

  Further, in the image classification device 1 according to the present embodiment, the iterative processing control unit 18 classifies all of the plurality of face image data to be classified into a face image data group or any of the face image data groups. The processing related to the determination of the classification destination for all the face image data is performed by controlling so that the processing by the classification unit 16 and the classification determination data exclusion unit 17 is repeated until it is determined that the data cannot be classified. This can be performed, and the classification of the classification target into an appropriate classification destination can be made more reliable.

  The image classification apparatus 1 according to the present embodiment further includes a face image data group setting unit 19 that newly sets the classification target face image data that is non-classifiable data as a face image data group that is a classification destination. As a result, the non-classifiable data for which no classification destination exists can be set as a new face image data group, which is then used as a further classification destination when classifying the face image data of different image data. be able to. In this way, by classifying face image data of a classification target for which no suitable classification destination exists into a new face image data group, it is possible to further reliably classify the classification target to an appropriate classification destination. Can do.

  Also, in the image classification device 1 according to the present embodiment, date information is set in the face image data group that is the classification destination, and the creation date of the image data that is the classification target and the face image data group that is the classification destination are set. If the creation date is earlier than the date and time information, an inappropriate classification destination exclusion unit 15 that excludes the face image data group related to the date and time information from the classification destination is provided. Image data created before a specific date and time information (for example, birthday) of a face image data group cannot be classified into the face image data group (not yet born but cannot be reflected in the image data). By setting the date and time information that can be determined to be in the face image data group, inappropriate classification can be prevented, and classification to an appropriate classification destination for classification can be made more reliable. That.

  In general, siblings and parents / children have similar genes, so if the image data to be classified and the classification target are related to siblings / parents / children, they are determined to be the same person even though they are different persons. It tends to be. By specifying face image data in which a person related to the face image data group cannot be captured by date and time information such as a birthday, even if it is determined that the person is the same person as a result of the similarity determination, It is possible to prevent the face image data from being classified into face image data groups. That is, for data that is difficult to determine similarity, the classification destination is determined even for elements other than the similarity determination, so that the classification of the classification target to an appropriate classification destination is made more reliable.

  The image classification device 1 according to the present embodiment includes an image data input unit 11 that inputs image data to be classified, and facial features of a plurality of face image data in the image data input by the image data input unit 11. And a similarity calculation unit 13 that calculates a similarity between a plurality of face image data and a face image data group that is a classification destination based on the face feature value. From the calculation of the similarity to the classification to the classification destination can be performed centrally.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments. The present invention is modified without departing from the gist described in each claim or applied to other embodiments. It may be a thing.

  For example, in the present embodiment, a functional configuration (image data input unit 11, face image data group storage unit 12, similarity calculation unit) that calculates the similarity between face image data and a face image data group by inputting image data. 13) is included in the same server (image classification apparatus 1) as the functional configuration related to classification (similarity input unit 14, classification unit 16, etc.), but is not limited to this. The functional configuration for calculating the function may be a function configuration included in another server, and may be classified by the classification unit 16 by receiving the similarity calculated by the other server. In this case, the similarity may be calculated by an existing image recognition engine.

  In this embodiment, the image classification device 1 classifies image data by a method of classifying face image data included in the image data into a registered face image data group of the same person as the person related to the face image data. However, in addition to the function, event information (information about time, information about place, information about people, information about action contents, etc.) is set (tag setting) in the image data, and based on the contents set by the tag The image data may be classified.

  In the present embodiment, the average feature amount of the face image data included in the face image data group is used in calculating the similarity between the face image data to be classified and the face image data group. However, the present invention is not limited to this. For example, for all face image data included in the face image data group, the degree of similarity with the face image data to be classified is calculated using each face feature amount, and the highest similarity is obtained. The degree may be the similarity between the face image data to be classified and the face image data group. Further, a method using the similarity using the average feature amount and a method using the highest similarity may be used in combination.

DESCRIPTION OF SYMBOLS 1 ... Image classification device, 2 ... Image data input part, 12 ... Face image data group memory | storage part, 13 ... Similarity calculation part, 14 ... Similarity input part, 15 ... Unsuitable classification destination exclusion part, 16 ... Classification part, 17 DESCRIPTION OF CLASSIFICATION DATA EXCLUSION UNIT 18 REPEAT PROCESSING CONTROL UNIT 19 FACE IMAGE DATA GROUP SETTING UNIT 20 CLASSIFICATION RESULT OUTPUT UNIT 21 CLASSIFICATION STATE STORAGE UNIT 30, 50. 201, 202 ... face image data group table, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111 ... face image data group, A, B, C, D, E, F, G ... Face image data.

Claims (7)

An image classification device that classifies a plurality of face image data in image data based on similarity,
Similarity input means for inputting information indicating the similarity between the plurality of face image data in the image data to be classified and the classified face image data group to be classified;
Classification means for classifying the face image data that is the classification target into the face image data group that is the classification destination based on the information indicating the similarity input by the similarity input means;
The classification target face image data classified by the classification means is excluded from the classification target as classification fixed face image data, and the face image data group into which the classification fixed face image data is classified is classified as a classification fixed face image data group. An image classification apparatus comprising: classification determination data exclusion means to be excluded from the classification destination; and repetitive processing control means for controlling to repeat processing by the classification means and the classification confirmation data exclusion means.   The image classification device according to claim 1, wherein the classification unit excludes information indicating that the similarity score is less than a predetermined threshold from information indicating the similarity.   The repetitive processing control means is configured such that all of the plurality of face image data to be classified are classified into the face image data group or cannot be classified into any of the face image data groups. The image classification device according to claim 1 or 2, wherein control is performed so that the processing by the classification unit and the classification determination data exclusion unit is repeated until it is determined.   The image classification device according to claim 3, further comprising a face image data group setting unit that newly sets the face image data to be classified as the classification impossible data as a face image data group to be classified.   Date and time information is set in the face image data group that is the classification destination, and the creation date of the image data that is the classification target is compared with the date and time information set in the face image data group that is the classification destination, 5. The apparatus according to claim 1, further comprising: an inappropriate classification destination exclusion unit that excludes the face image data group related to the date information from the classification destination when the creation date is earlier than the date information. The image classification device according to claim 1. Image data input means for inputting the image data to be classified;
Deriving face feature amounts of the plurality of face image data in the image data input by the image data input means, and based on the face feature amounts, the plurality of face image data and the face image data that is the classification destination The image classification apparatus according to claim 1, further comprising a similarity calculation unit that calculates a similarity with a group. An image classification method for classifying a plurality of face image data in image data based on similarity,
A similarity input step of inputting information indicating the similarity between the plurality of face image data in the image data to be classified and the classified face image data group to be classified;
A classification step of classifying the face image data that is the classification target into the face image data group that is the classification destination based on the information indicating the similarity input in the similarity input step;
The classification target face image data classified in the classification step is excluded from the classification target as classification fixed face image data, and the classification target face image data group of the classification fixed face image data is defined as a classification fixed face image data group. An image classification method comprising: a classification determination data exclusion step that is excluded from the classification destination; and a repetition processing control step that controls to repeat the processing in the classification step and the classification determination data exclusion step.

Images