JP2016200969A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to easily reflect his/her own preference to a database when an image is automatically selected by an image processing apparatus.SOLUTION: The present invention determines the quality of an input image on the basis of the feature amount extracted from the input image, when the determination result is different from a determination result of a user, extracts candidate images from a database and displays the images on a display part, and reflects the user's personal preference to the database on the basis of a candidate image selected by the user from the displayed candidate images.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for presenting an image asking a user about personal preference and learning personal preference.

  Imaging devices such as digital cameras in recent years have improved continuous shooting performance, and with the increase in capacity of storage media such as memories, the number of images collected even by individuals has increased dramatically. As a result, the task of selecting a favorite image from a large number of images has become a burden on the user.

  For such a point, a method for simplifying the operation of selecting a desired image from a large number of images by a user has been proposed as disclosed in Patent Document 1. In the method of Patent Document 1, the user will register the priority order of the characters and the favorite facial expression of the characters in the database in advance, calculate the recommendation degree of the image from those information, and the user will desire. Images are selected and displayed.

JP 2013-196417 A

P. Viola and M.M. Jones (2001). “Rapid Object Detection using a Boosted Cascade of Simple Features”, IEEE Conference on Computer Vision and Pattern Recognition. M.M. Turk and A.M. Pentland, “Eigenfaces for recognition”, Journal of Cognitive Neuroscience 3, pp. 71-86 M.M. Eckhardt et al. (2009), “Towards Practical Facial Feature Detection”, International Journal of Pattern Recognition and Artificial Intelligence Vol. 23, no. 3 (2009) 379-400. Gwen Littlewater et al. , “The Computer Expression Recognition Toolbox (CERT)”, IEEE Conference on Automatic Face and Gesture Recognition, 2011. P. Ekman and W.H. Friesen, “Facial Action Coding System: A Technology for the Measurement of Facial Movement”, Consulting Psychologists Press, Palo Alto, 1978. Hiroyuki Takano, Koichiro Deguchi, “About the use of face contours to cope with posture changes in face alignment by contours”, Information Processing Society of Japan. CVIM, 2012-CVIM-183 (11), 1-8, 2012-08-26 J. et al. Saragih, S .; Lucey, and J.L. Cohn, “Face Alignment through Subspace Constrained Mean-Shifts”, IEEE International Conference on Computer Vision (ICCV), 2009. Wei Luo et al., “Content-based photo quality assessment”, IEEE Conference on Computer Vision, (Poster, ICCV 2011). Takumi Tsuji, Shunta Yamamoto, Yoshinori Inaba, Shigeru Akamatsu, “Facial expression generation using a three-dimensional morphing model: Guidelines for facial expression generation using principal component parameters”, IEICE technical report. MVE, Multimedia and Virtual Environment Foundation 110 (457), 95-100, 2011-02-28

  However, in the method of Patent Document 1, it is necessary for the user to register a favorite facial expression of a character in a database using a score or the like. However, there is a problem that it is difficult to accurately reflect the user's preferences and preferences for subtle changes in facial expressions.

  The present invention refers to a database that associates and stores a quality relationship between a feature quantity extracted from an image and the quality of the image, and determines whether the input image is acceptable based on the feature quantity extracted from the input image Means for extracting a candidate image from the database when the determination result of the quality of the input image by the determination unit is different from the result of determination of the quality of the input image by the user, and the extracted Display control means for displaying a candidate image on a display unit; and setting means for setting a pass / fail relationship based on the determination of the user in the database based on a candidate image selected by the user from the displayed candidate images. It is characterized by having.

  According to the above configuration, in the present invention, the user can easily set his / her preference and palatability.

1 is a block diagram showing a functional configuration of an image processing apparatus according to a first embodiment. The conceptual diagram which shows a mode that the image was classified into the feature space in 1st Embodiment. The figure explaining the choice display screen which the display control part in connection with 1st Embodiment displays. The flowchart which shows the process sequence of the quality registration part in connection with 1st Embodiment. The flowchart which shows the process sequence of the separate preference learning part in connection with 1st Embodiment. The conceptual diagram of the process which extracts a choice candidate image in 1st Embodiment. The block diagram which shows the function structure of the image processing apparatus in connection with 2nd Embodiment. The figure explaining the option display screen which the display control part in connection with 2nd Embodiment displays.

[First Embodiment]
As a first embodiment, a method for learning the preference of an individual photographer or subject will be described based on an image that the photographer or subject individual has determined to be “preferable” based on a different standard from everyone. Hereinafter, this embodiment will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a software configuration (functional configuration) of an image processing apparatus 10 according to the present embodiment. Each functional unit of the image processing apparatus 10 illustrated in FIG. 1 is realized by a CPU included in the image processing apparatus executing a control program stored in a storage unit such as a ROM, a RAM, or an HDD. In the present embodiment, a case where the image processing apparatus is an imaging apparatus such as a digital camera will be described as an example.

  In FIG. 1, the pass / fail registration unit 100 is for registering in advance, for each image feature, a pass / fail (“preferred” or “preferred”) determination criterion that everyone feels when viewing an image. The pass / fail registration unit 100 includes an image acquisition unit 101, an image feature extraction unit 102, and an image feature classification unit 103.

  The image acquisition unit 101 acquires a learning image for learning a criterion for pass / fail determination. The learning image is taken by an imaging device such as a camera, and is input to the learning device in advance from a PC, a scanner, a storage device (hard disk drive, etc.), or a storage medium (nonvolatile flash memory, CD-ROM, DVD-ROM, etc.). Are stored in an HDD (not shown). It is assumed that the learning image is tagged as “preferred”, “not preferable”, or divided into three states of “no tag”. The tagging of the learning image is performed, for example, by a professional photographer looking at each image and determining “preferable” or “not preferable”.

  The image feature extraction unit 102 extracts image features from the image data of the learning image acquired by the image acquisition unit 101. The image feature classifying unit 103 classifies images based on the image features extracted by the image feature extracting unit 102 and tag information of the learning image. As a method for classifying the images, a k-means method, a Nearest Neighbor method, or the like, which is a typical method of clustering for machine learning, can be used. Each cluster classified by the image feature classifying unit 103 corresponds to a factor for determining the quality of an image (good / bad determination factor).

  FIG. 2 is a conceptual diagram schematically illustrating how the learning image is classified into the feature space and mapped by the image feature classifying unit 103. The feature space 200 is a space for mapping the image feature of each learning image based on the number of dimensions of the feature amount extracted by the image feature extraction unit 102. In the figure, image features of “unfavorable” learning images are clustered for each necessity determination factor such as “eye line” and “eye meditation”.

  In the present embodiment, the learning environment is set such that the shooting environment (background, lighting, etc.) and shooting conditions (same person, face orientation, occurrence of cut-out, etc.) are set to a desired state and it is determined that everyone is not preferable. I have prepared it. In the pass / fail registration unit 100 according to the present embodiment, these “unfavorable images” are mapped to the feature space 200 as shown in FIG. 2 in a meaningful manner. For example, the cluster 201 is a cluster in which only the face direction changes as in the image 202. In addition, images tagged as “unfavorable” are classified into clusters according to pass / fail judgment factors such as “facial expression”, “eye meditation”, “line of sight”, “out of eye”, and registered in the database 104. ing. As described above, the database 104 stores the feature quantity extracted by the image feature extraction unit 102 and the quality relationship between the quality of the image in association with each other.

  The learning image registered in the database 104 may use a group of images obtained by photographing the same person in advance under various shooting conditions. Depending on individual differences after shooting a plurality of persons under various conditions. Only feature quantities that are not present may be extracted and classified into clusters by the image feature classification unit 103. Moreover, even if it is not an actual person, computer graphics are used to generate an image of a fictitious person under various shooting conditions, analyze pass / fail judgment factors from those image features, and register them in the database 104. May be.

  The individual preference learning unit 110 uses a plurality of images registered in the database 104 and identifies user-specific preference when the system and the user have different pass / fail results. Further, the database 104 is updated based on the specified user preference. The individual preference learning unit 110 includes a quality determination unit 111, a preference option extraction unit 112, a display control unit 113, and an individual preference setting unit 114.

  The image processing apparatus 10 of the present embodiment is an imaging apparatus such as a digital camera, and the quality determination unit 111 receives an image captured by the imaging unit as an input signal. Then, the quality of the acquired captured image is determined with reference to the database 104. The quality determination unit 111 performs image feature extraction and classification in the same manner as the image feature extraction unit 102 and the image feature classification unit 103 in determining quality of the acquired image.

  Here, for example, even though the pass / fail determination unit 111 determines that the image is a “preferable image”, the user may determine that the image is an “unfavorable image”. In this case, the image is clearly determined as “not preferable” due to the user-specific preference. Therefore, if the reason why the user determines that the image is “not preferable” can be specified, it becomes possible to incorporate user-specific preferences into the pass / fail determination factor image database. Therefore, in the present embodiment, an image for specifying the reason (good / bad determination factor) for determining “unfavorable image” is displayed on the display unit of the imaging apparatus, and the user is allowed to select.

  The preference option extraction unit 112 extracts image candidates to be selected by the user from the database 104 in order to specify user-specific preference. In the present embodiment, an image registered in the database 104 as a candidate image to be used as an option is used as an option image for confirming the personal preference of the user. In general, when the user's preference is expressed in language, for example, it is often an ambiguous expression such as “It is preferable to shoot at a slightly more right angle” or “A slightly softer expression is preferable”. It is also difficult to express this as a score. In addition, since the user himself / herself does not always judge the quality of the image while being conscious of the personal preference, there are many cases where the user cannot accurately answer the personal preference with the language or the score. In the present embodiment, by using a group of images prepared in advance as a visual hint, the user indicates the preference factor in the language for the factors that the user determines to be “not preferable”. Even if it is not, it becomes possible to register palatability intuitively.

  The display control unit 113 displays the image group extracted from the database 104 by the preference option extraction unit 112 on the display unit of the imaging apparatus. FIG. 3 shows an option display screen 300 that is displayed on the display unit of the imaging apparatus by the display control unit 113 when the result of the quality determination by the user is different from the result of the quality determination by the quality determination unit 111. The option display screen 300 includes an area (upper part) where the user selects an “unfavorable image” and an area (lower part) that displays an image group serving as an option for specifying a factor that is unfavorable. It is a user interface.

  In the present embodiment, first, images captured by the imaging unit are displayed in the upper region as a preview image list 301 arranged in the order of shooting. Note that the display order can be displayed according to other rules such as descending order or ascending order for each subject, event unit, and time. The user can select “unfavorable image” from the image group of the preview image list 301 and give an instruction using a button or a touch user interface provided in the apparatus. FIG. 3 shows a state in which the user judges and selects the image 302 as “unfavorable”, and the image 302 is displayed as a thick frame because it is designated as a “desirable image” for deletion. .

  Next, the pass / fail determination unit 111 refers to the database 104 in response to the selection of “unfavorable image” from the preview image list 301, and the pass / fail of the image 302 that the user has determined as “not preferable” as a deletion target. Determine. If the pass / fail judgment results are different, an area that can be a basis for pass / fail judgment of the image 302 is estimated and displayed as pass / fail judgment candidate areas 303 as shown in FIG. For the pass / fail judgment candidate area 303, for example, when a face area calculated by face detection processing or continuous images are targeted, the fluctuation amount between images is calculated using an optical flow or the like. A large area may be set as a pass / fail judgment candidate area. In addition, the user may directly specify the area without performing complicated calculations.

  Subsequently, the preference option extraction unit 112 extracts image candidates to be selected by the user from the database 104 based on the set pass / fail judgment candidate region 303. Then, the display control unit 113 displays the image group extracted by the preference option extraction unit 112 as an image group including an image 304 that specifies a pass / fail determination factor in a lower area of the option display screen 300. Details of the image group extraction method by the preference option extraction unit 112 will be described later.

  The option display screen 300 functions as a user interface that allows the user to select one or a plurality of images. The user selects an image close to the reason for determining that the image 302 is “unfavorable image” from the option candidate image group, and presses the button 305. In FIG. 3, an image close to the image 302 with respect to the three pass / fail judgment factors of the line of sight, the expression, and the face orientation is displayed as an option image. Further, in order to make it easy to understand which pass / fail judgment factor each image corresponds to, each pass / fail judgment factor is illustrated and added to each of the displayed image groups. For example, in the rightmost image of the three images, the pass / fail determination factor is the face-oriented image, and therefore, a three-dimensional coordinate axis is displayed as annotation information on the face. If an image close to the reason that the user has determined “not preferable” is not displayed as an option candidate, the button 306 is pressed to notify that there is no option candidate.

  Returning to FIG. 1, the individual preference setting unit 114 accepts operation instruction information for an image group that identifies a user's failure determination factor. Then, the update contents of the database 104 are determined, and user-specific preferences are set.

  Next, details of the process of registering the pass / fail criterion in the pass / fail registration unit 100 will be described. FIG. 4 is a flowchart showing a processing procedure of the pass / fail registration unit 100 according to the present embodiment.

  First, in step S <b> 401, the image acquisition unit 101 performs a process of reading a learning image for registration in the database 104. Here, in the pass / fail registration unit 100, it is easier to separate an image group into clusters for each pass / fail determination factor when it is registered in the database 104 as a set of images grouped to some extent for each factor that is the basis for pass / fail determination. Note that the image read in step S401 is assumed to be an image group that has been determined in advance by a professional photographer or the like based on a certain standard, or an image group that has been intentionally changed in facial expression or face orientation.

  The processing from step S402 to S406 is processing for extracting image features by the image feature extraction unit 102. Since the present embodiment assumes that personal preference appears and the person performs the pass / fail determination of the image of the subject, the features extracted in steps S402 to S406 can be extracted mainly from the person's face. Use quantity. Examples of the image features calculated in steps S402 to S406 include personally identified information, facial expression information, face orientation information, and image quality (blur and color) information. Although many features that can be discriminated from the face of the subject are listed here, other feature amounts such as composition information that affects the aesthetics of the image may be introduced. In addition, attribute information such as age and sex of the subject may be added.

  Hereinafter, each process of steps S402 to S406 will be described in detail. In step S402, in order to extract various information related to the face of the subject in the image read in step S401, first, the face of the subject is detected from the image. As a method for detecting a face from an image, the method described in Non-Patent Document 1 is widely used. For example, by using the method disclosed in Non-Patent Document 1, the position and size of each person's face in the image can be calculated.

The principle of Non-Patent Document 1 is to collect feature quantities called Haar-Like from the face of the learning image, and to make it possible to identify features that are statistically facial by AdaBoost. If a face is included in the image in the process of step S402, information such as the number of detected faces, the upper left and lower right coordinate values of the face detection frame (including the face), and the barycentric position are output.
In step S403, personal identification processing is performed in response to the detection of one or more faces in step S402. Various methods have been proposed for personal identification processing since the 1980s, and Non-Patent Document 2 is an example. The method described in Non-Patent Document 2 calculates an eigenface from a face image, performs a principal component analysis of the eigenface, and identifies an individual by comparing it with an image database in which the eigenface is registered in advance. In the present embodiment, the individual identification of the subject is performed in step S403, and the personal preference is set in the database 104 for each subject.

  In step S404, for example, facial expression identification and expression intensity estimation are performed by using the technique disclosed in Non-Patent Document 3.

  In step S404, first, facial feature points (for example, the position of the corner of the eye, the pupil, the head of the eye, the tip of the nose, the mouth corner, and the center of the mouth) within the face detection frame detected in step S402 are calculated. The calculation method of these face feature points can be detected by the method disclosed in Non-Patent Document 2. In the method described in Non-Patent Document 3, the face feature points in the face detection frame detected in step S402 are calculated using the GentleBoost method. The facial feature point calculation method using the GentleBoost method is also described in Non-Patent Document 2, but in the same way as Non-Patent Document 1, the characteristic facial parts such as the corners of the eyes and the corners of the mouth are preliminarily described. Can be detected by learning with a weak classifier. In addition, it is also possible to calculate facial feature points using a technique called AAM (Active Appearance Model) or CLM (Constrained Local Model).

  Then, a feature amount is calculated from the luminance information around the face feature point calculated in step S402. The feature amount calculation method can be obtained using, for example, the method disclosed in Non-Patent Document 4. In the method disclosed in Non-Patent Document 4, a 96 × 96 pixel surrounding the face feature point calculated in step S402 is cut out as a face patch, the cut face patch is fast Fourier transformed, and then subjected to a Gabor filter. A feature vector is calculated. In Non-Patent Document 4, calculation is performed using a Gabor filter for calculation of the feature amount. Instead, there is a method using LBP (Local Binary Pattern), LPQ (Local Phase Quantization), or the like.

  Here, the feature quantity related to the facial expression registered in the database 104 can be calculated, but it is also possible to apply this feature quantity to a classifier and classify it according to a predetermined facial expression classification. In the method of Non-Patent Document 4, a change corresponding to an AU (Action Unit) of FACS (Facial Action Coding System), which is known as a method for objectively describing facial expression behavior, is detected. The expression is identified by the combination of all 46 AUs and their strengths. The following Non-Patent Document 5 can be referred to for details of FACS and AU.

  In step S404, a binary SVM (support vector machine) is used to determine which AU has changed, using the method of Non-Patent Document 4, using the calculated feature vector as an input, and changes according to facial expression. AU can be specified. It is possible to identify which facial expression category (joy, sadness, fear, anxiety, anger, surprise, no expression) belongs to the facial expression of the subject by the AU combination specified here. For example, if AU4 (Brow Lower) and AU10 (Upper Lip Raiser) have changed, it is said that they belong to the facial expression category of “anger”.

  Furthermore, the expression intensity of the subject's expression can be calculated as a feature amount from the specified change intensity of AU. Specifically, the distance between the hyperplane of the linear SVM that identifies the specified AU and the feature vector can be calculated and used as the change intensity of the AU. In other words, the greater the facial expression change, the longer the distance between the feature vector and the hyperplane of the linear SVM that identifies the AU, and the smaller the facial expression change, the closer the distance is to zero.

  In the technique of Non-Patent Document 4, the current AU combination and the change intensity of the AU are compared with the expression table extracted from the expression database prepared in advance and the reference table of the expression intensity. Estimates the facial expression intensity of the photographer. Alternatively, the expression intensity may be estimated by adding the combination of AUs and the change intensity of AUs by linear weighted addition. However, there are individual differences in the expression of facial expressions, and depending on the subject, there are some people who have difficulty expressing facial expressions. Therefore, it is desirable to change the weight used for the calculation of the expression intensity for each subject. In addition, since facial feature points have already been calculated in the calculation process of step S404, in addition to facial expression identification, it is also possible to calculate a feature such as the degree of opening and closing of eyes as a feature amount.

  In step S405, the face orientation (posture) is calculated for the face area detected in step S402. Similar to personal identification, a number of methods have been proposed for calculating the face orientation, and a suitable means may be selected as appropriate from among them. Examples thereof include the methods described in Non-Patent Document 6 and Non-Patent Document 7. In both methods, a feature detector is created in advance using the appearance information created from the facial model represented by facial feature points and the shade pattern of the facial image, and the matching with the input image is calculated. .

  In step S405, the three-dimensional face position and orientation are estimated using these methods. In the database 104 according to the present embodiment, information on the position and orientation of the face of the subject obtained from the input image is also recorded as a feature amount. For example, in the case of a person who cares about taking a picture of the face, the preference is often largely divided between a photograph taken from the right side and a photograph taken from the left side, and such personal preferences are registered in the system. Is needed to do.

  Next, in step S406, a feature amount relating to image quality (the state of the subject image) is extracted. This is a feature amount for evaluating aesthetic factors, for example, whether the subject is in focus, blur is not generated, and whether the position and size of the subject in the image are appropriate. Examples of a method for calculating a feature amount for evaluating an aesthetic factor include a method described in Non-Patent Document 8. Non-Patent Document 8 divides the image into several regions for each object in addition to the existing face detection technology, and extracts the clearness of pixels in the region (whether blurring occurs) as a feature quantity And pass / fail judgment is performed. In Non-Patent Document 8, in addition to pixel clearness, various other feature amounts are extracted, and a score for comprehensively evaluating the aesthetics of the entire image is calculated.

  As described above, in the present embodiment, in steps S403 to S406, the image feature extraction unit 102 includes four feature amounts of personal identification information, information on facial expressions, information on face orientation (posture), and information on image quality as various feature amounts. Calculate In addition to these feature amounts, attribute information such as the age and sex of the subject and information on the posture of the body other than the face (including gestures) may be extracted at the same time and used as a pass / fail judgment factor.

  Next, in step S407, the image feature classification unit 103 registers the feature amount and the image extracted from the image in each process of steps S402 to S406 in the database 104. In step S408, it is determined whether the feature amount extraction and image registration operations for all images registered in the database 104 have been completed. If all the images have not been registered, the process returns to step S401. When all the images have been registered, the process proceeds to the next step in order to collect the image features for each factor that is the basis for the pass / fail judgment.

  In step S409, the image feature classification unit 103 performs clustering on the extracted feature space for all images registered in the database 104 in step S407. For clustering, for example, a cluster clustered for each image feature is calculated using the k-means method described above. As described above, the clusters generated in this step are collected for each meaningful feature such as a line of sight, a face orientation, and an expression as shown in FIG. It is also possible to perform principal component analysis on each image feature distributed in the feature space and to dimensionally compress the image feature to determine the principal component of each image feature. Since the calculated principal component is a parameter axis that remarkably indicates a change in the factor that is a basis for the pass / fail judgment, image features may be classified using the calculated principal component axis instead of the above-described cluster. .

  In the present embodiment, a photographed photographed image is used as a learning image and registered in the database 104. For example, as in Non-Patent Document 9, a face image generated by computer graphics in addition to a photographed image. May be used. Changing the line of sight, face orientation, and facial expression by computer graphics can be easily performed only by changing parameters as long as a three-dimensional face model can be prepared.

  In addition, when clustering can be performed for each factor that is a basis for pass / fail judgment in the database 104, feature selection is performed, and only the feature amount that contributes most to classify the cluster is used, and classification into each cluster is performed. May be. This completes the process of registering the pass / fail criterion in the pass / fail registration unit 100.

  Next, details of the process of learning the user-specific preference in the individual preference learning unit 110 will be described. FIG. 5 is a flowchart showing a processing procedure of the individual preference learning unit 110 according to the present embodiment. First, in step S501, the quality determination unit 111 receives an image captured by the imaging unit as an input signal. Here, an example in which processing is performed on an image captured by an imaging device such as a digital camera will be described, but an image stored in a storage device or the like of the imaging device may be read.

  In step S502, the quality determination unit 111 refers to the database 104 and determines quality of the image acquired in step S501. Specifically, the quality determination unit 111 first performs image processing similar to that of the image feature extraction unit 102 to extract image features. Then, using the extracted feature amount as a feature vector, it is determined whether or not it is classified into a “desirable image” cluster registered in the pass / fail determination factor image database. Whether or not the feature vector of the captured image is classified into a certain cluster can be determined by the same method as the clustering method used by the pass / fail registration unit 100. Specifically, classification can be performed by calculating the distance between the feature vector and the center or centroid of each cluster, and finding a cluster whose distance is equal to or less than a threshold. Alternatively, the cluster to which the feature vector with the shortest distance belongs may be set as a candidate cluster to which the feature vector of the captured image belongs. Note that the processing in step S502 is preferably completed immediately after imaging or before showing the captured image to the user.

  In step S503, the display control unit 113 displays the captured image acquired in step S501 on the image display unit (liquid crystal display or the like) of the imaging device. In step S504, the pass / fail determination unit 111 receives the pass / fail determination result instructed by the user. Specifically, in the state in which the preview is displayed on the liquid crystal display of the imaging device, for example, a process of accepting designation as to whether or not to delete the captured image is performed.

  In step S505, the pass / fail determination unit 111 compares the pass / fail result determined in step S502 with the pass / fail determination result received by the user in step S504. As a result of the comparison, if both pass / fail results match, there is no need to learn the user's personal preference, so the process returns to step S501 to wait for the input of a new image. On the other hand, if the pass / fail result does not match, for example, if the user selects deletion even though the image is determined to be preferable by the pass / fail determination unit 111, it is determined that it is an opportunity to learn the user's personal preference, The processing from S506 to S509 is executed.

  In step S506, the preference option extraction unit 112 inquires of the user which one of those images is closest to the “unfavorable image group” registered in the database 104 in order to grasp the personal preference of the user. Extract image candidates. FIG. 6 is a conceptual diagram illustrating an example in which option candidate images are extracted from the clusters registered in the database 104 in the process of step S506.

  Hereinafter, an example of extraction of option candidate images will be specifically described with reference to FIGS. 3 and 6. As a premise, it is assumed that the image processing apparatus of the present embodiment has determined the image 302 in FIG. 3 as a “preferred image” before learning the user's personal preference. Then, it is assumed that the user designates deletion for the reason that “the line of sight was not facing forward (goal)” with respect to the image 302 in FIG. Note that the image feature amount of this image 302 has already been extracted in the course of the pass / fail judgment process in step S502. A point obtained by mapping the extracted image feature amount as a feature vector into a feature space registered in the pass / fail determination factor image database is shown as a feature point 601 in FIG.

  In step S506, the preference option extraction unit 112 extracts an image whose distance (similarity) is close to the feature point 601 as an option candidate. The distance in the feature space of FIG. 6 can be obtained by calculating the Euclidean distance between feature vectors. In the example of FIG. 6, the top three feature points are extracted as option candidates in order of the calculated distance. Since each registered image and feature vector are registered in the database 104 as a set, an original image corresponding to the feature vector can be searched. In FIG. 6, three option candidate images are displayed as an example. However, the number of displayed images is not limited, and a necessary number of images may be extracted and displayed from images registered in the database 104.

  In step S507, the display control unit 113 displays the option candidate image extracted in step S506 on the display unit of the apparatus. An image 304 in FIG. 3 is an example in which option candidate images are displayed. In FIG. 3, the feature vectors calculated in step S <b> 506 are displayed in order from the shortest distance between the feature vectors. However, the order of the distances may be increased or the order of the distances may be changed. In addition, depending on the user, it may be possible to appropriately reflect individual preference by assigning priorities to each feature amount. Therefore, a distance may be calculated by assigning priority weights to feature vectors. .

  In step S508, the individual preference setting unit 114 receives information on option candidate images selected by the user from the images displayed in step S507. Here, since the user feels that the direction of the line of sight is unfavorable and the user selects the image 302, the user is directed to the upward direction of the line of sight of the image 304 displayed in the option candidate display area of FIG. Select the image facing. At that time, as shown in the image 304 in FIG. 3, the features may be visualized and displayed on the display screen so that the image features can be seen at a glance. For example, the amount of variation from the facial image when there is no expression may be extracted, and the direction of the region where the amount of variation is equal to or greater than the threshold may be indicated by an arrow, or may be displayed as a rectangular region. If there is a feature in the face orientation, a three-dimensional coordinate axis may be displayed as annotation information on the face.

  In step S509, the individual preference setting unit 114 adds, as a parameter, information related to the individual preference registered in the database 104 based on the image selected by the user. Here, the information related to individual preferences refers to various numerical information (parameters) that are changed when a new cluster is generated for each pass / fail judgment cluster registered in the database 104. Alternatively, it indicates various numerical information (parameters) that are changed when an image having individual preference information is newly added to an existing cluster and the range of the cluster is relearned.

  When the user selects one or more option candidate images on the option display screen 300, in step S509, the individual preference setting unit 114 adds individual preference information to the cluster registered in the database 104, The cluster range is calculated again. Specifically, the range of the cluster 201 indicating the face-oriented features in FIGS. 2 to 6 is expanded to the feature point 601. When an image similar to the feature point 601 is input again by this cluster recalculation process, the pass / fail judgment unit 111 can be classified into clusters that reflect the user's personal preference. By performing the cluster recalculation process in this way, the database 104 can re-set the cluster that is the basis for the pass / fail judgment according to the personal preference while sequentially adding information with personal preference. become.

  On the other hand, when the user presses the “no option” button 306 on the option display screen 300, this means that there is no cluster as an option in the database 104. For this reason, the individual preference setting unit 114 newly generates a cluster centered on the feature point 601 of the image 302 designated as a deletion target by the user. At this time, it is desirable to set the range of the cluster centering on the feature point 601 so as not to overlap the adjacent cluster. In addition, the setting range may be set in advance as an initial value, or may be directly set by the user when a new cluster is generated.

  In step S510, it is determined whether or not to end this process. When the process is continued, the process returns to the image acquisition process in step S501.

  In the description of the present embodiment, the form of learning the user-specific preference for the facial expression of the person who is the subject has been described, but the application range of the present embodiment is not limited to the facial expression of the person. For example, if the criteria for pass / fail judgment varies depending on the subject due to the position and size of the face (composition), face orientation (posture), gaze direction, eyes, blink, gender, age, etc. Applicable.

  As described above, according to the present embodiment, it is possible to learn personal preference by extracting an image close to the image feature of an image that the user has determined to be “not preferable” from the database and allowing the user to select it. In particular, in the present embodiment, it is possible for the user to intuitively register the palatability even if the user has not indicated the factor that is the basis for the image determined as “not preferable” in the score or language. Become.

[Modification of First Embodiment]
In the first embodiment, a case has been described in which the user determines “not preferable” with respect to an image determined as “preferable image” by the apparatus. On the other hand, the present invention can also be applied to a case where the user determines “preferred” to an image determined by the apparatus as “unfavorable image”.

  In this modification, “preferred images” are clustered for each necessity determination factor as learning images and registered in the database 104. When the quality determination unit 111 determines “not preferable” for a certain image and the user determines “preferred”, the preference option extraction unit 112 selects the “preferred” image group from the database 104. Extract candidate images. The setting method for reflecting the user's personal preference in the database 104 is the same as in the first embodiment.

  In this modification, the image registered in the database 104 may be, for example, an image of a model shown in a magazine or the like in addition to an image group prepared in advance or an image taken by a user. This is because desirable image features can be extracted from the way the image can be used as a model and registered in the database 104. On the other hand, an image determined by the user as “not preferable” needs to reflect a complex implicitly held by the user. For this reason, it is considered that it is better to set parameters for an image in which the user himself or a person who is mainly the subject is shown, as in the first embodiment described above.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, a configuration will be described in which the user is allowed to select a plurality of candidate images at the same time, thereby learning individual preference related to a plurality of pass / fail judgment factors. In the following description, when a personal computer (PC) functions as the image processing apparatus of the present invention and creates or prints an album, the apparatus selects images that the photographer and subject do not like from a large number of image groups. This will be described as an example. In addition, about the structure already demonstrated in 1st Embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted.

  FIG. 7 is a block diagram illustrating a software configuration (functional configuration) of the image processing apparatus 10 according to the present embodiment. In the figure, what is different from the configuration of the first embodiment is that an individual preference parameter setting unit 701 is provided instead of the individual preference setting unit 114. In the first embodiment, an image captured by the imaging unit is acquired. In the present embodiment, the image acquisition unit 101 acquires an image to be processed in addition to the learning image. The image acquisition unit 101 reads an image stored in a storage device in the image processing apparatus, a storage medium connected to the apparatus, a storage device, or the like.

  In addition, the structure by which the function part which acquires the image for learning and the function part which acquires the image of a process target became separate may be sufficient. In the present embodiment, it is assumed that a large number of image groups are processed, and it is desirable that the input unit (interface) for reading the learning image and the processing target image is configured separately. For example, a configuration in which a learning image is read from a storage medium or a storage device connected to the apparatus and a large amount of image data is downloaded and read from a disk storage service called a cloud on the Internet is assumed.

  Returning to FIG. 7, the individual preference parameter setting unit 701 learns the user's individual preference based on the candidate image selected by the user, as in the first embodiment. Then, individual preference information is added to the clusters already registered in the database 104, and the range of the clusters on the feature space is calculated again. In addition, parameters to be registered in the database 104 are set based on the candidate image selected by the user on the option display screen 300.

  Next, a process for learning user-specific preferences in the present embodiment will be described. FIG. 8 shows an example of the option display screen 300 displayed by the display control unit in the present embodiment. In the figure, in the preview image list 301, for example, images determined as “preferred” by the quality determination unit 111 of the apparatus are selected and displayed from various scenes taken at an athletic meet.

  Here, for example, it is assumed that the user determines that the face orientation of the child, which is the main subject shown in the image 302, is not preferable, and selects deletion or non-display. The preference option extraction unit 112 of the image processing apparatus 10 according to the present embodiment receives a user's designation operation and extracts an image for resetting a parameter indicating personal preference from the database 104. The extracted image is displayed by the display control unit 113 in the option image display area 801 of FIG.

  In the option image display area 801, the image 802 is an image that has already been determined as “unfavorable” by the user and is registered as an “unfavorable image” in the database 104. As shown in FIG. 8, an image registered as an “unfavorable image” is displayed so that the user can easily recognize it with a thick or slanted frame.

  On the other hand, the image 803 has been determined as “preferred” by the user and is registered as a “preferred image” in the database 104. The image 804 is an image designated as an image close to the reason why the user newly determines that it is not “preferable” (in this case, the face direction). The image 804 displays a frame line and a hatched line with a broken line or the like so that it can be easily distinguished from the image 802 and the image 803.

  The category display field 805 presents to the user which pass / fail judgment factor is based on the image group shown in each row. In the case of FIG. 8, “category 1” and “category 2” are simply displayed. However, since “category 1” displays a group of images based on “eyes”, instead of displaying “category 1”. In addition, a tag “line of sight” may be displayed. However, the pass / fail judgment factors are not only those expressed by easy-to-understand words such as “line of sight” as described above, but there are many facial expressions and ways of being photographed that are difficult to verbalize. For this reason, the category display field 805 does not necessarily display a word indicating the meaning of the image group as the category name. Further, in the case of an image group that is difficult to verbalize, for example, the cluster identification ID shown in the first embodiment may be displayed in the category display field.

  The parameter 806 and the parameter 807 illustrate parameters indicating the user's personal preference. The parameter 806 and the parameter 807 are set by the individual preference parameter setting unit 701 based on the candidate image selected by the user, and are displayed by the display control unit 113. For example, category 1 is an image group related to the direction of the line of sight, and it is specified that a face image with the line of sight from the lower left to the lower right and a face image facing upward are not preferable. It is displayed in the area specified in gray. Here, the parameter 806 is displayed as a circle in order to display the angle regarding the line of sight as a parameter in an easy-to-understand manner for the user. In general, it is difficult to illustrate the contents of the category in an easy-to-understand manner as described above. In this case, the maximum value and the minimum value of the parameter are displayed.

  Category 2 lists images related to face orientation. The parameter 807 indicates 0 degrees (front) with the face facing the center, and indicates from -180 degrees rightward to 180 degrees leftward. Here, since the user newly designates an image 804 (a slightly rightward face image) as an unfavorable image, the display of the parameter 807 is updated, and the angle of the face orientation determined to be unfavorable is increasing.

  As described above, on the option display screen 300 according to the present embodiment, an image group for changing parameters of personal preference is displayed line by line for each pass / fail judgment factor, and the parameters are also displayed at the same time. The user can register the user's personal preference while confirming parameters that change while selecting an image in the option image display area 801.

  On the other hand, since the parameter registered in the database 104 is changed by specifying the image 804, in the present embodiment, the image 808 indicated by the dotted frame in the preview image list is also newly determined as an “unfavorable” image. Become so. This is because the orientation of the face of the subject of the image 808 is also facing right like the image 302. Therefore, in this embodiment, the priority of each category displayed in the category display field 805 can be designated. For example, when the user determines that the image 808 is “preferred”, the user lowers the priority related to the face direction of the subject among the categories displayed in the category display field 805 and prioritizes other categories. Can be set to increase the degree. As a result, even if the image has an unfavorable face orientation, it is determined as a “preferred image” if other factors (such as facial expressions) are good.

  As described above, according to the present embodiment, it is possible to learn individual preference related to a plurality of pass / fail determination factors by causing the user to select candidate images for each pass / fail determination factor (category). In the present embodiment, the individual preference parameter setting unit 701 displays numerical information (parameters) to be reflected in the database 104 on the option display screen 300, so that the individual preference set by the user is intuitive to the user. Can be shown.

[Other Embodiments]
In addition, the present invention supplies software (program) for realizing the functions of the above-described embodiments to a system or apparatus via a network or various storage media, and the computer of the system or apparatus (or CPU, MPU, etc.) programs Is read and executed. Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not. That is, the present invention includes all the combinations of the above-described embodiments and modifications thereof.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 101 Image acquisition part 102 Image feature extraction part 103 Image feature classification part 104 Database 111 Pass / fail judgment part 112 Preference choice extraction part 113 Display control part 114 Individual preference setting part

Claims (10)

A determination unit that determines whether or not the input image is good based on a feature value extracted from the input image with reference to a database that stores the quality value extracted from the image and the quality relationship between the image and the image in association with each other;
An extraction unit that extracts a candidate image from the database when a determination result on the input image by the determination unit is different from a determination result on the input image by the user;
Display control means for displaying the extracted candidate images on a display unit;
Based on a candidate image selected by the user from the displayed candidate images, setting means for setting a pass / fail relationship based on the determination of the user in the database;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the database stores the quality relationship classified by factor related to image quality.   The image processing apparatus according to claim 2, wherein the setting unit sets a parameter related to the factor. The extraction means extracts candidate images for each of the plurality of factors,
The image processing apparatus according to claim 3, wherein the setting unit sets a parameter relating to each of the plurality of factors.   The image processing apparatus according to claim 3, wherein the display control unit displays information on parameters set by the setting unit on a display unit.   The image processing apparatus according to claim 2, wherein the control unit displays the candidate image with information on a factor related to the quality.   The image processing apparatus according to claim 1, wherein the extraction unit extracts a plurality of images that are close to the input image from the database.   2. The feature amount according to claim 1, wherein the feature amount is at least one of a face direction, an eye angle, a facial expression intensity, an eye opening degree, a mouth opening degree, and an image quality. 8. The image processing apparatus according to any one of items 1 to 7. A determination step of determining the quality of the input image based on the feature value extracted from the input image with reference to a database that stores the quality value extracted from the image and the quality relationship with the image in association with each other;
An extraction step of extracting candidate images from the database when the determination result of the quality of the input image in the determination step is different from the result of determination of the quality of the input image by the user;
A display control step of displaying the extracted candidate image on a display unit;
A setting step for setting a pass / fail relationship based on the determination of the user in the database based on a candidate image selected by the user from the displayed candidate images;
An image processing method comprising:   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 8.

Images