JP2014067301A - Image processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of increasing a processing speed and improving accuracy thereof, while performing person recognition with relatively simple operation, and a program.SOLUTION: An image processing apparatus detects a face image portion included in a collection of image data to be a processing object from each of the collection of image data, estimates a direction of the face in the detected face image portion, selects data related to a face image portion in which the estimated direction of a face falls within a predetermined range as a clustering object, executes predetermined clustering processing to the data related to the face image portion included in the selected clustering object, and receives input of identification information from each of obtained clusters.

Description

  The present invention relates to an image processing apparatus and a program for executing image processing relating to a human face.

  In recent years, an image processing apparatus that recognizes a face image and estimates a person name related to the face image has been developed. At present, such facial image recognition is used for, for example, crime prevention and device usability improvement by personal recognition.

  Note that Patent Document 1 discloses an example in which faces are classified based on the face orientation, weight coefficients are calculated for each face orientation, and used for recognition processing (paragraph 0080).

JP 2011-221791 A

  However, in the conventional face recognition process, the face image facing in a direction unsuitable for the recognition process is used as a key for the recognition process in spite of the importance of the face orientation. At present, high-load processing is required for improvement.

  The present invention has been made in view of the above circumstances, and provides an image processing apparatus and program capable of improving the accuracy while increasing the processing speed while performing human recognition by a relatively simple operation. Is one of its purposes.

  The present invention for solving the problems of the above conventional example is an image processing apparatus, which detects a face image portion from each image data included in a set of image data to be processed, and detects the detected face Means for outputting an estimation result of the face orientation of the image portion, means for selecting data relating to the face image portion in which the estimated face orientation is in a predetermined range as a clustering target, and the selected Means for executing a predetermined clustering process on the data relating to the face image portion included in the clustering target, and means for receiving input of identification information for each of the clusters obtained as a result of the clustering process; Of the data related to the face image portion included in each of the clusters, the data related to the face image portion to be a key is determined based on a predetermined condition. Means for associating and storing the key data and the identification information about the corresponding cluster, data relating to the face image included in the image data to be subjected to recognition processing, and the data serving as the key Is used for recognition processing for setting identification information in the image data to be subjected to the recognition processing.

  The means for comparing the data related to the face image subjected to the recognition processing and the data used as the key for each cluster, the data relating to the face image subjected to the recognition processing, and the cluster The degree of coincidence with the data used as the key for each data, the similarity calculated based on a predetermined comparison condition between the data, and the weight determined in relation to the orientation of the face image portion in the data used as the key It is good also as calculating using.

  According to another aspect of the present invention, a program detects a face image portion from each image data included in a set of image data to be processed, and estimates a face orientation of the detected face image portion. Means for outputting a result; means for selecting data relating to a face image portion in which the estimated face orientation is in a predetermined range; and a face image included in the selected clustering target. Means for executing a predetermined clustering process on the data relating to the part; means for accepting input of identification information for each of the clusters obtained as a result of the clustering process; and faces included in each of the clusters From the data relating to the image portion, data relating to the face image portion to be a key is determined based on a predetermined condition, and the key and Means for associating and storing identification data about the corresponding cluster, and using the data relating to the face image included in the image data to be recognized and the key data Thus, the image data to be subjected to the recognition process is subjected to a recognition process for setting identification information.

  In the present invention, a face image of a person facing a predetermined range is selectively used for clustering, and a face image to be compared (key) is determined, so that the person can be operated by a relatively simple operation. While performing recognition, the processing speed can be increased and the accuracy can be improved.

It is a block diagram showing the example of a structure of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the face database which the image processing apparatus which concerns on embodiment of this invention produces | generates. It is a functional block diagram showing the example of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing showing the example of the content of the data which the image processing apparatus which concerns on embodiment of this invention records during the production | generation of a face database. It is a flowchart figure showing the example of operation of the image processing device concerning an embodiment of the invention.

  Embodiments of the present invention will be described with reference to the drawings. As illustrated in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, a communication unit 15, and an input / output interface 16. Yes. Here, the control unit 11 is a program control device such as a CPU, and operates according to a program stored in the storage unit 12.

  Specifically, in the present embodiment, the control unit 11 accepts image data to be processed via the communication unit 15 or the input / output interface 16 and accumulates and stores the image data in the storage unit 12. Further, the control unit 11 detects a face image portion from each image data included in the set of image data to be processed, and outputs a face direction estimation result of the detected face image portion. Execute the process.

  The control unit 11 further extracts a face image portion whose face direction estimated here is within a predetermined range, and generates a face image data set. And the control part 11 performs a predetermined | prescribed clustering process with respect to the data which concern on the face image part contained in the face image data set produced | generated here.

  As a result of the clustering process, the control unit 11 accepts input of identification information for each of the obtained clusters, and based on a predetermined condition from the data related to the face image portion included in each cluster. To determine the key data. The control unit 11 associates the data serving as the key with the identification information input for the corresponding cluster and stores it in the storage unit 12 as a face database. Using the face database generated in this way, the control unit 11 compares the data related to the face image part that is the key with the data related to the face image included in the image data that is the target of the recognition process, and performs the recognition process. It is subjected to a recognition process for setting identification information to the image data that is the target of the above. The processing of the control unit 11 will be described in detail later.

  The storage unit 12 stores a program executed by the control unit 11. The program may be provided by being stored in a computer-readable recording medium such as a DVD-ROM (Digital Versatile Disc Read Only Memory) and stored in the storage unit 12. The program may be distributed via a network or the like and stored in the storage unit 12.

  In the present embodiment, image data is accumulated and stored in the storage unit 12. Further, as illustrated in FIG. 2, the storage unit 12 associates data (for example, a feature amount calculated from the face image part) Fk and identification information (ID) related to the face image part as a key. The face database containing the entries is stored.

  The operation unit 13 may be, for example, a mouse or a keyboard, or may be an infrared input interface. In an example of the present embodiment, the operation unit 13 is an infrared input interface, and receives information representing the content of the user's instruction operation transmitted from the remote controller that has received the user's instruction operation. The operation unit 13 outputs information representing the contents of the received instruction operation to the control unit 11.

  The display unit 14 is an interface that outputs an image to an external display or a home television device in accordance with an instruction input from the control unit 11. The communication unit 15 is a network interface, for example, and is connected to the network by wire or wirelessly and outputs information received via the network to the control unit 11. Further, the communication unit 15 receives an input of information to be transmitted via the network from the control unit 11 and transmits the information via the network.

  The input / output interface 16 is, for example, an SD card slot or a USB (Universal Serial Bus) interface. The input / output interface 16 reads out image data from an SD card, a USB memory, a USB hard disk drive, or the like connected thereto according to an instruction input from the control unit 11, for example, and outputs the image data to the control unit 11. The control unit 11 may accept image data received by the communication unit 15 via a wired or wireless network as a processing target.

  Next, the operation of the control unit 11 will be described. The control unit 11 of the present embodiment is functionally configured to include a preliminary processing unit 20 and a recognition processing unit 30 as illustrated in FIG. Here, the preliminary processing unit 20 includes a face image processing unit 21, a data extraction unit 22, a clustering processing unit 23, an identification information acquisition unit 24, and a key extraction unit 25.

  The face image processing unit 21 performs a process for specifying a face image part included in the given image data and a process for estimating the orientation of the face included in the specified face image part. Since the processing executed by the face image processing unit 21 is widely known, a detailed description thereof will be omitted here, but here the face image processing unit 21 is a face extracted from the image data to be processed. Assume that partial image data of an image portion and a facial feature amount (corresponding to data relating to the face image portion) included in the partial image data are output. When a plurality of face image portions are extracted from the image data, the face image processing unit 21 according to the present embodiment outputs the partial image data and feature amount data for each face image portion. Note that the feature amount data is vector information that enumerates numerical values representing predetermined facial recognition features such as the distance between eyes.

  The face image processing unit 21 performs processing for estimating the orientation of the face included in the extracted partial image data of the face image portion, and outputs the estimation result. It is assumed that the face orientation estimated by the face image processing unit 21 is an angle in a cross section having a normal line as a midline (intersection line between a coronal plane and a sagittal plane). Specifically, the face image processing unit 21 sets θ as the face orientation when the front direction of the image data is set to a direction of 0 °, centering on the point where the median line passes in the cross section. The direction in which the person imaged in the image data faces to the left (as viewed from the person) is defined as the angle in the negative direction of θ, and the direction in the right direction (as viewed from the person) is defined as the angle in the positive direction of θ. Keep it. As an example, the face orientation angle can be determined based on the position of the nose recognized in the face image for the entire specified face image portion.

  For each face image portion specified from the given image data, the face image processing unit 21 obtains partial image data of the specified face image portion, feature amount data, and angle data representing the face direction. Will be output.

  The data extraction unit 22 inputs the partial image data of the face image portion output by the face image processing unit 21 for each image data, the feature amount data related to the face image portion, and the angle data representing the face direction. In response, the entries associated with each other are accumulated and stored in the storage unit 12 (FIG. 4). The data extraction unit 22 selects an entry having a face orientation estimated by the face image processing unit 21 within a predetermined range (θlth <θ <θhth) as the clustering target entry from the accumulated entries. Information specifying each clustering target entry is output to the clustering processing unit 23.

  The clustering processing unit 23 clusters the feature amounts included in the clustering target entry specified by the information output from the data extraction unit 22 among the entries stored in the storage unit 12. This clustering method may be any method, but as an example, a widely known method such as a k-mean method can be used. Further, the order of the feature quantity (the dimension number of the feature quantity vector) may be reduced by a method such as principal component analysis and then clustering processing such as k-mean method may be performed.

  The clustering processing unit 23 includes, for each feature quantity classified into clusters by the clustering process, information for identifying a cluster as a classification result (which may be a unique number for each cluster assigned to the cluster) including the corresponding feature quantity. Record in association with the clustering target entry.

  When the clustering processing by the clustering processing unit 23 ends, the identification information acquisition unit 24 outputs the partial image data included in the clustering target entry associated with the cluster specifying information for specifying each cluster to the display unit 14 for display. The input of identification information related to the displayed partial image data is requested. Here, the identification information is, for example, the name of the person imaged in the displayed partial image data. At this time, the identification information acquisition unit 24 may accept an operation such as “do not name” (ignore) in addition to accepting an input of a name.

  The key extraction unit 25 uses the feature amount data associated with the same cluster identification information as the partial image data whose name (identification information) is input (not ignored) by the identification information acquisition unit 24 as key data. Extract as The key extraction unit 25 is configured such that, for example, among the clustering target entries associated with the same cluster identification information as the partial image data whose name is input by the identification information acquisition unit 24, the feature amount data is close to the center of gravity of the cluster. Using the feature quantity data of a predetermined number in order as key data, these feature quantity data (Fk) and an angle representing the orientation of the face stored in the storage unit 12 in relation to each feature quantity And the input identification information (name: ID) are stored in the storage unit 12 as a face database.

  The key data stored in the face database may include image data of face image portions associated with each of a predetermined number of feature amounts in order from the feature amount data close to the center of gravity of the cluster. . What kind of data is selected as key data in this way may be determined by the recognition processing unit 30 described below. The recognition processing unit 30 includes a face image processing unit 31, a data comparison unit 32, and a weight calculation unit 33.

  The face image processing unit 31 of the recognition processing unit 30 accepts input of image data to be processed. Then, a face image portion is extracted from the image data. When the face image portion is extracted, the face image processing unit 31 generates a predetermined feature amount for the face image portion (if there are a plurality of face image portions). Here, the feature amount data is vector information that enumerates numerical values representing predetermined facial recognition features such as the distance between eyes, like the face image processing unit 21 of the preliminary processing unit 20. . The face image processing unit 31 may be substantially realized by the same program module as the face image processing unit 21 of the preliminary processing unit 20.

  The data comparison unit 32 sequentially selects identification information stored in the face database stored in the storage unit 12. The data comparison unit 32 reads the feature amount related to the selected identification information, and outputs the angle data representing the face orientation stored in the face database in association with the read feature amount to the weight calculation unit 33. When the data comparison unit 32 accepts the weight data from the weight calculation unit 33, the data comparison unit 32 generates information indicating the similarity between the read feature amount and the feature amount output by the face image processing unit 31. The processing for generating the information indicating the similarity can be a well-known processing for comparing the feature amount of the face image, and a detailed description thereof is omitted here.

  When there are m pieces of feature quantity data associated with one piece of identification information, the data comparison unit 32 calculates similarity R1, R2,... Rm for each piece of feature quantity data. Further, the data comparison unit 32 acquires m weight data from the weight calculation unit 33 using the angle data representing the orientation of the face associated with each of the m feature quantities. The data comparison unit 32 is a product of the similarity with each of the generated m feature values and the weight data W1, W2,... Wm output by the weight calculation unit 33 for each of the m feature values ( Using R i × W i, the degree of coincidence M (%) is calculated as M = (Σ (R i × W i)) / ΣW i (Σ means that i is the sum of 1 to m). To do.

  For example, when m = 2 and there are two feature amounts, the similarity to one feature amount is “95”, and the calculation is based on angle data representing the face direction associated with the feature amount. It is assumed that the weight data is “0.9”. Also, it is assumed that the degree of similarity with the other feature amount is “70”, and the weight data calculated based on the angle data representing the orientation of the face associated with the feature amount is “0.3”. .

  Then, the degree of coincidence M calculated from these is (85.5 + 21) /1.2=88.75. In other words, the degree to which each degree of similarity contributes to the finally calculated degree of coincidence is controlled depending on the weight associated with the feature quantity that is the basis of the calculation of each degree of similarity.

  The data comparison unit 32 calculates the degree of coincidence M by performing the above processing for each piece of identification information stored in the face database. Of the calculated degrees of coincidence, the maximum degree of coincidence is found and identification information about the maximum degree of coincidence is output. The data comparison unit 32 may display information indicating that there is no matching identification information on the display unit 14 when the found maximum matching value is smaller than a predetermined threshold value.

  In this way, the data comparison unit 32 determines the feature amount obtained from the image data of the face image that is the object of recognition processing, and the data previously extracted as a key for each cluster in the preliminary processing unit 20 (here, the feature amount). The degree of coincidence with Fk) is calculated.

  The weight calculator 33 receives input of angle data representing the face direction, and outputs data representing a predetermined weight according to the angle data representing the face direction. As an example, the weight calculation unit 33 assumes that the angle when the face is in front is θ = 0, “1” when θ = 0, “0.9” when 0 <| θ | ≦ 30, 30 <| “0.5” when θ | ≦ 45, “0.3” when 45 <| θ | ≦ 60, “0.1” when 60 <| θ | <90, “0” when | θ | ≧ 90 Is output. The weight calculator 33 may calculate a weight using a predetermined function value related to θ and output the weight as data.

  The image processing apparatus 1 according to the present embodiment has the above-described configuration and operates as follows. The image processing apparatus 1 accepts an instruction to import image data of various photographs such as a photograph in which a person is captured or a landscape photograph. Then, in accordance with the instruction, the image processing apparatus 1 captures the image data of the photograph, accumulates it in the storage unit 12, and stores it.

  A program module for identifying a face portion is installed in the image processing apparatus 1 and can be executed. The image processing apparatus 1 uses the program module to store past image data among the accumulated image data. Then, processing that identifies a face portion is performed by selecting an unprocessed item.

  As shown in FIG. 5, the image processing apparatus 1 uses the selected image data as given image data, and executes a process of specifying a face image portion from the image data (S1). The image processing apparatus 1 further estimates face orientation data θ included in the specified face image portion (S2). As described above, θ represents θ = 0 when the face is facing the front, and represents the orientation of the face in a plane parallel to the cross section of the human body. That is, the angle when facing sideways is θ = 90 degrees. Furthermore, the image processing apparatus 1 generates facial feature data included in the identified facial image portion (S3).

  Note that if the face image portion cannot be identified from the image data, the image processing apparatus 1 does not perform any further processing on the image data.

  For each piece of image data for which a face image portion has been identified, the image processing apparatus 1 performs a process of combining the partial image data of the face image portion, the feature amount data related to the face image portion, and the angle data representing the face direction. The associated entries are accumulated and stored in the storage unit 12 (S4). The image processing apparatus 1 further selects, from the accumulated entries, an entry whose angle data θ representing the face orientation is in a predetermined range (θlth <θ <θhth) as a clustering target entry (S5). ). The range determined here is a range including “0” indicating the front direction. For example, θlth = −30 degrees, θhth = 30 degrees, and the like.

  The image processing apparatus 1 clusters the feature amounts included in the clustering target entry (S6). The image processing apparatus 1 sequentially selects clusters obtained as a result of clustering, and outputs and displays the partial image data (face image portion) associated with each feature quantity classified into the selected cluster on the display unit 14. Then, input of identification information of the image classified into the cluster is requested (S7). For example, the input of the name of a person included in the displayed face image portion is requested.

  When identification information such as a name is input here, the image processing apparatus 1 uses a program module that executes face recognition processing to compare the face image from the entry including the feature quantity classified into the selected cluster. The data used for this purpose is extracted as key data (S8). Specifically, the key data is feature amount data Fk and angle data representing the face orientation. In another example, all of the data included in the entry may be extracted as key data.

  Identification data such as the input name is associated with the key data (Fk) and stored in the storage unit 12 as a face database (S9). This face database is later used for processing (recognition processing) for identifying the name of the person being imaged. As an example, a face image of a person being imaged is extracted, and the feature amount of the extracted face image is compared with key data stored in the face database, and calculated by a predetermined calculation method. The person who is calculating the degree of coincidence, finds the key data corresponding to the most matching feature quantity, and refers to the identification information associated with the found data, and the person being imaged is identified by the identification information Set to be.

  Furthermore, the data extraction unit 22 of the preliminary processing unit 20 may perform the following processing. That is, as already described, the partial image data of the face image portion output by the face image processing unit 21 for each image data, the feature amount data related to the face image portion, and the angle data representing the face direction Are stored in the storage unit 12 in association with each other. The data extraction unit 22 selects, from the accumulated entries, an entry whose face direction estimated by the face image processing unit 21 is in a predetermined range (θlth <θ <θhth) as a clustering target entry.

  Then, the data extraction unit 22 outputs the feature amount included in the selected clustering target entry to the data comparison unit 32 of the recognition processing unit 30. In this case, the data comparison unit 32 calculates the degree of coincidence between the feature amount input from the data extraction unit 22 and the feature amount that has been previously extracted as a key for each cluster in the preliminary processing unit 20 and stored in the face database. To do. Specifically, the data comparison unit 32 sequentially selects identification information stored in the face database. Then, the data comparison unit 32 calculates the similarities R1, R2,... Rm for the data of each feature quantity (m pieces) associated with the selected identification information. In addition, the data comparison unit 32 outputs angle data representing the orientation of the face stored in the face database in association with the m feature value data to the weight calculation unit 33.

  The data comparison unit 32 calculates the similarity R1, R2,... Rm between each calculated feature value and the input feature value, and the weight data W1 output by the weight calculation unit 33 for each of the m feature values. , W2,... Wm, and using the product (R i × Wi), M = (Σ (R i × Wi)) / ΣWi (Σ is the sum of i from 1 to m). (Means to calculate) and output.

  The data extraction unit 22 obtains the degree of coincidence M output for each identification information stored in the face database from the data comparison unit 32, and the degree of coincidence calculated using the feature amount related to any identification information It is determined whether M exceeds a predetermined threshold value. When a matching degree M exceeding a predetermined threshold is found, the selected entry is removed from the clustering target entry.

  In other words, this processing corresponds to controlling the image data of the face partial image having the feature quantity for which the identification information is already set not to perform the clustering and the identification information input process.

  As described above, in the present embodiment disclosed herein, control is performed so that a face image portion whose face orientation is not within a predetermined range does not contribute to generation of a key for recognition processing. Therefore, the face image portion facing in a direction unsuitable for the recognition process is not used as a key for the recognition process, and the recognition accuracy can be improved while performing the person recognition by a relatively simple operation. Further, since the process is relatively simple, it contributes to increasing the processing speed.

  Furthermore, in the above description, the range of face orientation (θlth <θ <θhth), which is a condition for selecting as a clustering target entry, is divided into a plurality of sections, and for each section, the estimated face Entries with orientations may be selected as clustering target entries.

  Also in this case, the image processing apparatus 1 clusters the feature amounts included in the selected clustering target entry. Then, the clusters obtained as a result of clustering are sequentially selected, and the partial image data (face image portion) associated with each feature amount classified into the selected cluster is output and displayed on the display unit 14, and the cluster is displayed on the cluster. Prompts for input of identification information of the recorded image.

  When the identification information (name, etc.) is input here, the image processing apparatus 1 compares the face image with a program module that executes face recognition processing from the entry including the feature quantity classified into the selected cluster. Data used for the purpose is extracted as key data.

  In this example, a plurality of clusters to be associated with the common identification information exist for each inclination. Then, feature value data used for recognition processing is extracted from each cluster. In this case, when comparing the feature value used as the key with the feature value obtained from the image subjected to recognition processing, May weight the comparison operation in accordance with the slope of the cluster from which the key feature quantity is extracted. For example, the weight is adjusted so that the comparison result becomes negative (so that the coincidence value becomes smaller) in a cluster corresponding to a section with a larger inclination from the front.

  Further, the weight used in the calculation of the recognition process is a predetermined representative value (in the interval of the angle of the face included in the image subjected to the recognition process and the inclination of the cluster from which the key feature quantity is extracted ( For example, the weight may be adjusted so that the comparison result becomes negative as the difference from the median value of the section)) becomes larger (so that the value of coincidence becomes smaller). For example, when the angle of the face included in the image subjected to recognition processing is 45 degrees, the cluster whose median value of the slope of the cluster that is the extraction source of the key feature amount is 45 degrees, A weight is set so that the comparison result is affirmative, and a cluster having a central value of 0 degree (front direction) of a cluster inclination section from which a key feature quantity is extracted is 90 degrees (horizontal direction). With respect to a certain cluster, the same weight may be set so as to obtain a comparable comparison result.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Control part, 12 Storage part, 13 Operation part, 14 Display part, 15 Communication part, 16 Input / output interface, 20 Preliminary processing part, 21, 31 Face image processing part, 22 Data extraction part, 23 Clustering Processing unit, 24 identification information acquisition unit, 25 key extraction unit, 30 recognition processing unit, 32 data comparison unit, 33 weight calculation unit.

Claims (3)

Means for detecting a face image portion from each image data included in a set of image data to be processed and outputting an estimation result of the face orientation of the detected face image portion;
Means for selecting, as a clustering target, data relating to a face image portion in which the estimated face orientation is in a predetermined range;
Means for performing a predetermined clustering process on data relating to a face image portion included in the selected clustering target;
Means for accepting input of identification information for each of the clusters obtained as a result of the clustering process;
From the data relating to the face image portion included in each of the clusters, data relating to the face image portion serving as a key is determined based on a predetermined condition, and the cluster corresponding to the data serving as the key is identified. Means for associating and storing information;
Including
Image processing to be used for recognition processing for setting identification information in image data to be subjected to recognition processing using data relating to a face image included in image data to be subjected to recognition processing and the data used as the key apparatus. The image processing apparatus according to claim 1,
A means for comparing the data related to the face image subjected to the recognition processing and the data used as a key for each cluster, the data relating to the face image subjected to the recognition processing, and the data for each cluster The degree of coincidence with the data used as the key is determined using a similarity calculated based on a predetermined comparison condition between the data and a weight determined in relation to the orientation of the face image portion in the data used as the key. An image processing device that performs calculations. Computer
Means for detecting a face image portion from each image data included in a set of image data to be processed and outputting an estimation result of the face orientation of the detected face image portion;
Means for selecting, as a clustering target, data relating to a face image portion in which the estimated face orientation is in a predetermined range;
Means for performing a predetermined clustering process on data relating to a face image portion included in the selected clustering target;
Means for accepting input of identification information for each of the clusters obtained as a result of the clustering process;
From the data relating to the face image portion included in each of the clusters, data relating to the face image portion serving as a key is determined based on a predetermined condition, and the cluster corresponding to the data serving as the key is identified. Means for associating and storing information;
Function as
A program for use in recognition processing for setting identification information in image data to be subjected to recognition processing using data relating to a face image included in image data to be subjected to recognition processing and data used as the key.

Images