JP2008234377A - Image sorting device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To retrieve an image including a desired person without performing any complicated operation. <P>SOLUTION: A camera 10 acquires the face images of a predetermined face with a plurality of different configurations. First and second similarity calculation parts 34 and 36 calculate the similarity of the predetermined face and each of the plurality of images based on the face images of the face with the plurality of configurations by an algorithm corresponding to one or more different facial configurations. A sorting part 38 sorts the plurality of images in the order of the calculated high similarity. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image sorting apparatus and method for sorting a plurality of images, and a program for causing a computer to execute the image sorting method.

In recent years, photography has been easily performed with the spread of digital cameras. Images obtained by shooting can be printed out using a minilab or a network print service. However, it is very troublesome to search for an image desired to be printed from a large number of images acquired by photographing. For this reason, a system has been proposed in which a face image of a person to be searched is acquired and an image including the face image is searched from a plurality of images (see Patent Document 1). The method described in Patent Document 1 is based on the similarity between a face of a person who is a search target registered in advance by acquiring images of various situations using a camera installed in a facility such as an amusement park. Thus, an image including a person to be searched is searched. The technique described in Patent Document 1 can also capture the face of a target person after image capturing and register the face of the person.
JP 2002-24229 A

  Here, in the method described in Patent Document 1, since a face image is registered in advance by photographing a person's face or the like, when searching for an image, Therefore, it is necessary to find out the face image of the person who desires the search. However, when searching for an image including a large number of persons as in the technique described in Patent Document 1, it is necessary to search for a face image of a person to be searched from a large number of face images at the time of the search. Will be very complicated. Also, Patent Document 1 describes a method of reading a user's face recorded on an ID card or transmitting a user's ID number to specify a face image. However, such work is very complicated. It is.

  The present invention has been made in view of the above circumstances, and an object thereof is to search for an image including a desired person without performing complicated work.

An image sorting apparatus according to the present invention is an image sorting apparatus that sorts a plurality of images.
Facial image acquisition means for acquiring facial images of different forms of faces for a predetermined face;
One or a plurality of similarities in which the similarity between the predetermined face and each of the plurality of images is calculated by an algorithm according to one or a plurality of different face modes based on the face images of the faces of the plurality of modes. Degree calculation means;
Sorting means for sorting the plurality of images in descending order of the calculated similarity is provided.

  In the image sorting apparatus according to the present invention, the face image acquisition unit may be a unit that acquires face images of a plurality of different aspects of the predetermined face by photographing.

An image sorting method according to the present invention is an image sorting method for sorting a plurality of images.
Obtaining face images of different aspects of a given face,
Based on the face images of the faces of the plurality of aspects, the similarity between the predetermined face and each of the plurality of images is calculated by an algorithm according to one or more different face aspects;
The plurality of images are sorted in descending order of the calculated similarity.

  The image sorting method according to the present invention may be provided as a program for causing a computer to execute the image sorting method.

  According to the present invention, face images of a plurality of different aspects of a predetermined face are acquired, and based on the face images of the plurality of aspects, the degree of similarity between the predetermined face and each of the plurality of images is obtained. It is calculated by an algorithm according to one or a plurality of different face modes, and a plurality of images are sorted in descending order of the calculated similarity. Therefore, it is not necessary to search for a face image of a predetermined face to be searched from a plurality of registered face images, and as a result, an image including a desired face image can be efficiently searched from the plurality of images. . In particular, since the plurality of images are sorted based on the degree of similarity of the predetermined face with the faces of the plurality of aspects, the plurality of images are arranged so that the image including the predetermined face is ranked high regardless of the face form. Are sorted. Therefore, it becomes easy to find a predetermined face.

  Also, by acquiring a plurality of modes of face images by photographing, it is possible to more easily acquire face images of different modes for a predetermined face.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an external perspective view of an image print order receiving apparatus to which an image sorting apparatus according to an embodiment of the present invention is applied. As shown in FIG. 1, a print order receiving apparatus 1 according to the present embodiment is installed at a photo shop in order to receive an image print order by a user, and an image for ordering a print is recorded. Various memory cards 2 are loaded, a plurality of types of card slots 4 for reading out images from the memory card 2 and recording images on the memory card 2, and liquid crystal for performing various displays for ordering prints, etc. It has a display unit 6 and is connected to a printer 8 installed in a photo shop. The display unit 6 includes a touch panel type input unit, and the user can make an input necessary for the print order by touching the display unit 6 in accordance with the display on the display unit 6. In addition, the print order receiving apparatus 1 according to the present embodiment includes a camera 10 for photographing the face of a user who makes a print order.

  FIG. 2 is a schematic block diagram showing the configuration of the print order receiving apparatus according to the embodiment of the present invention. As shown in FIG. 2, the print order receiving device 1 performs various controls such as recording control and display control of image data representing an image, and also controls each part of the device 1, and an image print order. A system order memory 14 composed of a print ordering program for performing the processing, a basic program for operating the apparatus 1 and a ROM in which various constants are recorded, and a RAM serving as a work area when the CPU 12 executes processing. The touch panel type input unit 16 for giving various instructions to the apparatus 1 and the display unit 6 described above are provided.

  The camera 10 includes an imaging lens, an imaging device such as a CCD, an A / D conversion unit that converts a signal acquired by the imaging device into digital, an image processing unit that performs image processing on the digital signal to improve image quality, and the like. Prepare. Here, the photographing operation on the camera 10 is performed from the input unit 16 under the control of the CPU 12. The user who places the print order causes the camera 10 to photograph the front face and the oblique face of the user according to the display on the display unit 6.

  FIG. 3 is a diagram showing a shooting screen when shooting the face of the user. As shown in FIG. 3, a message display area 51 for displaying an instruction message, an image display unit 52 for displaying an image to be photographed, and a photographing button 53 are displayed on the photographing screen 50. According to the message displayed in the message display area 51, the user turns front and obliquely toward the camera 10, confirms the image captured by the image display unit 52, and then touches the capture button 53, thereby The front face and the oblique face can be photographed by the camera 10. In FIG. 3, “Please turn to the front” is displayed in the message display area 51, but the message “Please turn diagonally” is displayed after the front face is photographed, and the oblique face is photographed. Will be done.

  The print order receiving apparatus 1 also includes a card slot 4 described above, a compression / decompression unit 22 that compresses image data by a technique typified by JPEG, and decompresses the compressed image data, and a hard disk that stores a plurality of images. 24, a memory control unit 26 for controlling the system memory 14, the card slot 4 and the hard disk 24, a display control unit 28 for controlling the display of the display unit 6, and a printer interface 30 for connecting the printer 8 to the apparatus 1. Is provided.

  The print order receiving apparatus 1 also includes a face detection unit 32 that detects a face area from each of a plurality of images stored in the hard disk 24, and first and second similarity calculation units 34 and 36, as will be described later. And a sorting unit 38 for sorting a plurality of images.

  The face detection unit 32 uses a template matching method, a method using a face discriminator obtained by machine learning learning using a large number of sample images of a face, and the like on the user's front face image, oblique face image, and hard disk 24. Face positions on a plurality of stored images are detected as face regions. The face detection unit 32 also detects the face orientation. Specifically, the positions of both eyes and nose are detected from the detected face area, and the face orientation is calculated as a value between 0 and 90 degrees on the left and right based on the positional relationship between both eyes and nose. When the face orientation is smaller than the predetermined threshold value Th0, it is detected that the face direction of the detected face area is the front face, and when the face orientation is greater than or equal to the predetermined threshold value Th0, it is detected. It is detected that the face direction of the face area is oblique.

  The first and second similarity calculation units 34 and 36 are included in the front face Tf0 and the oblique face image included in the user's front face image acquired by the camera 10 by shooting according to algorithms according to different face modes. The degree of similarity with the oblique face Ts0 is calculated. Specifically, the first similarity calculation unit 34 calculates the similarity by an algorithm that is particularly effective for solid discrimination of a face facing the front (front face), and the second similarity calculation unit 36 particularly The similarity is calculated by an algorithm effective for solid discrimination of a face facing diagonally (oblique face).

  The first and second similarity calculators 34 and 36 first configure the front face Tf0 and the oblique face Ts0 for the front face Tf0 and the oblique face Ts0 detected from the user's front face image and oblique face image. Facial parts are detected as facial feature points. Specifically, for example, by using a template matching method using a template of each facial part or a method using a discriminator for each facial part obtained by machine learning learning using a large number of sample images of facial parts. The positions of a total of nine feature points of the eyes, left and right eyes, left and right noses, left and right mouth corners, and upper lip are detected.

  Then, the first similarity calculation unit 34 sets affine transformation parameters to be applied to the front face Tf0 so that the detected nine feature points approach the reference position of the reference face that faces in front that is determined in advance. The normalized face T1 is obtained by performing affine transformation on the front face Tf0.

  The first similarity calculation unit 34 projects the image data of the normalized face T1 onto the lower-dimensional first feature amount space using the first projection matrix, and extracts the feature amount FT1. Here, the first feature amount space is determined by the following first learning.

  First, the principal learning analysis or linear discriminant analysis (LDA) is used for the first learning face image group normalized as described above with respect to a large number of learning face images facing the front. And the eigenspace projected by this projection matrix or a space equivalent thereto is defined as a first feature amount space. The first feature amount space determined in this way is a space where it is easy to distinguish a face facing the front.

  Further, the first similarity calculation unit 34 normalizes the target face Pf0 of the front face that is the target of similarity calculation among the faces included in each of the plurality of images in the same manner as described above. The face P1 is acquired, and the image data of the normalized face P1 is projected onto the lower-dimensional first feature amount space using the projection matrix to extract the feature amount FP1 in the normalized face P1.

  Then, the feature amount FT1 of the front face Tf0 and the feature amount FP1 of the target face Pf0 are compared, and the similarity R1 between the front face Tf0 and the target face Pf0 is calculated. Here, the similarity R1 is calculated using an AGM model (additional Gaussian model) described in, for example, JP-A-2005-149506. Here, the similarity R1 is calculated as a value between 0 and 1, and the closer to 1, the higher the similarity.

  The AGM model is a probabilistic model that assumes that face data can be expressed by the sum of variables representing individual differences and variables representing changes in the appearance of individuals (lighting changes, face orientation changes, secular changes, etc.). Shall follow a normal distribution. By estimating the parameters of each normal distribution in advance, it is possible to perform a robust similarity calculation process in consideration of changes in appearance even when the number of faces to be subjected to similarity calculation is small. For parameter estimation of each normal distribution, a learning face image group used when determining the feature amount space is used.

  In the feature amount space, the Euclidean distance between the coordinates of the front face Tf0 defined by the feature amount FT1 and the coordinates of the target face Pf0 for similarity calculation defined by the feature amount FP1 is used as it is as the similarity R1. Other similarity calculation methods such as a method may be used.

  On the other hand, the second similarity calculation unit 36 sets affine transformation parameters to be applied to the oblique face Ts0 so that the detected nine feature points approach the reference position of the reference face that faces in advance. The normalized face T2 is obtained by performing affine transformation on the oblique face Ts0. In this case, if the orientation of the oblique face Ts0 is different from the orientation of the reference face, the image data of the oblique face Ts0 is reversed left-right symmetrically before the affine transformation, and the orientation of the oblique face Ts0 and the orientation of the reference face are determined. Match. In addition, it is preferable to make it image | photograph so that it may become the direction of a reference | standard face at the time of imaging | photography of a user's diagonal face.

  The second similarity calculation unit 36 projects the image data of the normalized face T2 onto the lower-dimensional second feature amount space using the second projection matrix, and extracts the feature amount FT2. Here, the second feature amount space is determined by the second learning as described below.

  Using the normalized learning face image group normalized as described above with respect to a large number of learning face images facing diagonally, the second is performed by analysis such as principal component analysis or linear discriminant analysis (LDA). And the eigenspace projected by this projection matrix or a space equivalent thereto is defined as the second feature amount space. The second feature amount space determined in this way is a space where it is easy to determine the solid of a face facing diagonally. Note that the learning face image is the same as the direction of the reference face.

  In addition, the second similarity calculation unit 36 performs normalization by performing normalization in the same manner as described above on the target face Ps0 of the oblique face that is the target of similarity calculation among the faces included in each of the plurality of images. The face P2 is acquired, and the image data of the normalized face P2 is projected onto the lower-dimensional second feature amount space using the projection matrix to extract the feature amount FP2 in the normalized face P2. Also in this case, before the affine transformation for normalization, if the orientation of the target face Ps0 is different from the orientation of the reference face facing diagonally, the image data of the target face Ps0 is inverted symmetrically. The direction of the target face P0 is matched with the direction of the reference face.

  Then, the feature amount FT2 of the oblique face Ts0 and the feature amount FP2 of the target face Ps0 are compared, and the similarity between the oblique face Ts0 and the target face Ps0 is obtained using the same method as the first similarity calculation unit 34. R2 is calculated.

  In the feature amount space, the Euclidean distance between the coordinates of the oblique face Ts0 defined by the feature amount FT2 and the coordinates of the target face Ps0 for similarity calculation defined by the feature amount FP2 is used as it is as the similarity R2. Other similarity calculation methods such as a method may be used.

  Here, when the target face Pf0 is the face of the same person as the front face Tf0, the similarity R1 is high. Further, when the target face Ps0 is the face of the same person as the oblique face Ts0, the similarity R2 is high.

When a plurality of faces are included in the image, as shown in FIG. 4, the similarities R1 and R2 of the front face Tf0 and the oblique face Ts0 are calculated for each face according to the face orientation. Here, in FIG. 4, the calculation of the degree of similarity of only the front face Tf0 is described as an example. The highest similarities R1 and R2 among the similarities R1 and R2 calculated for one image may be simply referred to as the face T0 (hereinafter collectively referred to as the front face Tf0 and the diagonal face Ts0). Therefore, as shown in FIG. 5, the image G0 includes four front faces, and the front face Tf0 and the target face Pf0 facing each front are as shown in FIG. 5 is 0.9, 0.6, and 0.2 as shown in FIG. 5, and when the similarity R2 with the target face Ps0 facing diagonally is 0.7, the user's face T0 And the degree of similarity R3 of the image G0 is determined to be 0.9.

  Hereinafter, processing performed in the present embodiment will be described. FIG. 6 is a flowchart showing processing performed in the present embodiment. When the user touches the display unit 6 to instruct image sorting, the CPU 12 starts processing, and by causing the user to photograph the front face and the oblique face of the user, the user's face T0 (front face Tf0 and front face Tf0). The input of the oblique face Ts0) is accepted (step ST1).

  Next, the CPU 12 reads the first image to be processed for similarity calculation from the plurality of images stored in the hard disk 24 (step ST2), and the face detection unit 32 determines the face area and the face orientation from the target image. It detects (step ST3). Then, the CPU 12 sets the target face P0 for similarity calculation as the first face (for example, the face at the left end of the target image) (step ST4), determines the orientation of the face of the target face P0 (step ST5), and determines the front face. In some cases, the first similarity calculation unit 34 calculates the similarity R1 between the user's front face Tf0 and the target face Pf0 (step ST6). On the other hand, when the target face P0 is an oblique face, the second similarity calculation unit 36 calculates the similarity R2 between the user's oblique face Ts0 and the target face Ps0 (step ST7).

  Then, the CPU 12 determines whether or not the similarities R1 and R2 of all the faces included in the target image have been calculated (step ST8). If step ST8 is negative, the target face P0 is set as the next face. (Step ST9), the process returns to step ST5, and the processes after step ST5 are repeated. If step ST8 is affirmed, the first or second similarity calculation units 34 and 36 determine the similarity R3 between the user's face T0 and the target image (step ST10).

  Next, the CPU 12 determines whether or not the similarity R3 of all images stored in the hard disk 24 has been determined (step ST11). If step ST11 is negative, the target image is set to the next image (step ST11). ST12) Returning to step ST3, the processing after step ST3 is repeated.

  If step ST11 is affirmed, the sorting unit 38 sorts the plurality of images in descending order of similarity R3 with the user's face T0 (step ST13), and displays the plurality of images on the display unit 6 in the sorted order ( Step ST14), the process ends. When the similarity is the same, the images are sorted in ascending order of shooting date / time. Further, it is not necessary to display all of the plurality of images in the order of similarity. For example, only images whose similarity R3 exceeds a predetermined threshold Th1 may be sorted and displayed.

  FIG. 7 is a diagram showing a display screen for sorting results. As shown in FIG. 7, in the display area 55 of the display screen 54, a plurality of images are displayed in the order of similarity R3. The display screen 54 displays arrow buttons 56A and 56B for switching images displayed in the display area 55 when a plurality of images cannot fit on one screen.

  As shown in FIG. 7, in each image displayed in the display area 55, a face with the highest similarity to the user's front face Tf0 or the oblique face Ts0 is given a thick frame. The user can select a plurality of displayed images by touching the display unit 6 with an image to be printed, for example. For example, an image can be selected by touching an image including his / her face surrounded by a thick frame. Then, after the image is selected, a print instruction is issued, and the selected image is printed out from the printer 8.

  As described above, in the present embodiment, since the front face and the oblique face of the user are photographed and used for image sorting, a plurality of registered faces are used as in the method described in Patent Document 1 above. There is no need to find a desired face image to be searched from the image. Therefore, an image including the user's face can be efficiently searched from a plurality of images. In particular, since the images are sorted based on the faces of a plurality of aspects such as a front face and an oblique face, a plurality of images including the user's face are placed at the top regardless of whether the face is a front face or an oblique face. The images will be sorted, which makes it easier for the user to find the image in which he is shown.

  In the above-described embodiment, the first similarity calculation unit 34 that calculates the similarity by an algorithm effective for front face solid discrimination and the second similarity calculation by an algorithm effective for solid face discrimination. Although the similarity calculation unit 36 is used, for example, the second similarity calculation unit 36 is omitted, and the similarity R3 between the user's face T0 and the image is calculated using only the first similarity calculation unit 34. You may make it do. In this case, the front face Tf0 and the diagonal face Ts0 are acquired as the user's face T0. The average face Tm0 of the user is generated from the front face Tf0 and the diagonal face Ts0, and the similarity R1 between this and the target face P0. May be calculated.

  In the above embodiment, before sorting in descending order of similarity, a plurality of images may be classified in advance according to shooting locations and shooting times, and sorting may be performed in units of sorted images. .

  In the above embodiment, the display screen 54 has thick frames added to the target faces Pf0 and Ps0 corresponding to the user's face T0. In particular, the sorted images are displayed without adding a thick frame. You may do it. However, since it can be easily confirmed whether or not it is similar to the user's face by giving a thick frame, it is preferable to give a thick frame.

  In the above embodiment, a front face and an oblique face are used as face modes. However, the present invention is not limited to this. If a face with an expression or a face without an expression is used as the face mode, Good. In this case, a face with an expression and a face without an expression display a message such as “Please make a serious face” or “Please laugh” in the message display area 51 on the shooting screen 50 shown in FIG. Then, it may be acquired by causing the user to perform shooting.

  In the above embodiment, the image sorting apparatus according to the present invention is applied to a print order receiving apparatus installed at a photo shop. However, the present invention is not limited to this, and a print order is placed via a network. An image sorting device may be applied to the order server to sort images stored in the order server via the network.

  Although the apparatus according to the embodiment of the present invention has been described above, the computer corresponds to the face detection unit 32, the first similarity calculation unit 34, the second similarity calculation unit 36, and the sorting unit 38 described above. A program that functions as shown in FIG. 6 and performs processing as shown in FIG. 6 is also one embodiment of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention. Further, such a program may be incorporated in viewer software for browsing images.

1 is an external perspective view of a print order receiving apparatus to which an image sorting apparatus according to an embodiment of the present invention is applied. 1 is a schematic block diagram showing a configuration of a print order receiving apparatus according to an embodiment of the present invention. Figure showing the shooting screen The figure for demonstrating calculation of the similarity degree of a user's face and the face contained in an image (the 1) The figure for demonstrating calculation of the similarity degree of a user's face and the face contained in an image (the 2) A flowchart showing processing performed in the present embodiment Figure showing the sort result display screen

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Print order reception apparatus 2 Memory card 4 Card slot 6 Display part 8 Printer 10 Camera 12 CPU
DESCRIPTION OF SYMBOLS 14 System memory 16 Input part 22 Compression / decompression part 24 Hard disk 26 Memory control part 28 Display control part 30 Printer interface 32 Face detection part 34 1st similarity calculation part 36 2nd similarity calculation part 38 Sort part

Claims (4)

In an image sorting device for sorting a plurality of images,
Facial image acquisition means for acquiring facial images of different forms of faces for a predetermined face;
One or a plurality of similarities in which the similarity between the predetermined face and each of the plurality of images is calculated by an algorithm according to one or a plurality of different face modes based on the face images of the faces of the plurality of modes. Degree calculation means;
An image sorting apparatus comprising: sorting means for sorting the plurality of images in descending order of the calculated similarity.   2. The image sorting apparatus according to claim 1, wherein the face image acquisition means is means for acquiring face images of a plurality of different aspects of the predetermined face by photographing. In the image sorting method for sorting multiple images,
Obtaining face images of different aspects of a given face,
Based on the face images of the faces of the plurality of aspects, the similarity between the predetermined face and each of the plurality of images is calculated by an algorithm according to one or more different face aspects;
An image sorting method, wherein the plurality of images are sorted in descending order of the calculated similarity. In a program for causing a computer to execute an image sorting method for sorting a plurality of images,
A procedure for acquiring facial images of different aspects of a predetermined face;
A step of calculating a similarity between the predetermined face and each of the plurality of images based on face images of the plurality of aspects by an algorithm according to one or a plurality of different face aspects;
A program for sorting the plurality of images in descending order of the calculated similarity.

Images