JP2008217385A - Image sorting device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to easily find out an image including the primary face without acquiring a face image being the object of retrieval. <P>SOLUTION: A plurality of images are displayed at a display part 6, and the selection of an image by a user is accepted. A similarity calculation part 34 calculates the similarity of the faces included in the plurality of images in a prescribed timing, and a labeling part 36 labels the face included in a plurality of images on the basis of similarity. A primary face specification part 38 specifies at least one primary face included in the plurality of images on the basis of the labeling result of the face included in the selected image. A similarity calculation part 34 calculates the primary face similarity being similarity between the primary face and any image other than the selected image, and a sorting part 40 re-sorts any image other than the image selected in the order of the high primary face similarity, and displays the images in the order of re-sort at a display part 6. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image sorting apparatus and method for sorting a plurality of images, and a program for causing a computer to execute the image sorting method.

In recent years, photography has been easily performed with the spread of digital cameras. Images obtained by shooting can be printed out using a minilab or a network print service. However, it is very troublesome to search for an image desired to be printed from a large number of images acquired by photographing. For this reason, a system has been proposed in which a face image of a person to be searched is acquired and an image including the face image is searched from a plurality of images (see Patent Document 1).
JP 2002-24229 A

  However, in the method described in Patent Document 1, since it is necessary to acquire a face image to be searched in advance, an image including a subject from which a face image has not been acquired cannot be searched. As a result, it is difficult to search for an image for printing or the like.

  The present invention has been made in view of the above circumstances, and an object thereof is to make it possible to easily search for an image including a main face without acquiring a face image to be searched.

An image sorting apparatus according to the present invention is an image sorting apparatus that sorts a plurality of images.
Sorting means for sorting the plurality of images in a predetermined order;
Display means for displaying the plurality of images;
Selection means for accepting selection of a plurality of desired images from the displayed plurality of images;
Similarity calculation means for calculating the similarity between faces included in the plurality of images at a predetermined timing;
Labeling means for labeling faces included in the plurality of images based on the similarity;
Main face specifying means for specifying at least one main face based on a labeling result of the face included in the selected image,
The similarity calculation means is a means for calculating a main face similarity which is a similarity between the main face and another image other than the selected image.
The sorting means is means for re-sorting the other images in descending order of the main face similarity and displaying the other images on the display means in the re-sorted order. .

An image sorting method according to the present invention is an image sorting method for sorting a plurality of images.
Accepting selection of a plurality of desired images from a plurality of images displayed on the display means,
At a predetermined timing, the similarity between faces included in the plurality of images is calculated,
Label faces included in the plurality of images based on the similarity,
Identifying at least one primary face based on a labeling result of faces included in the selected image;
Calculating a main face similarity that is a similarity between the main face and another image other than the selected image;
The other images are re-sorted in descending order of the main face similarity, and the other images are displayed on the display means in the re-sorted order.

  The image sorting method according to the present invention may be provided as a program for causing a computer to execute the image sorting method.

  According to the present invention, selection of a plurality of desired images from a plurality of displayed images is accepted, and the degree of similarity between faces included in the plurality of images is calculated at a predetermined timing. Then, the faces included in the plurality of images are labeled based on the similarity. This labeling is performed, for example, by assigning the same label to faces whose similarity is equal to or higher than a predetermined value. Next, at least one main face included in the plurality of images is identified based on a labeling result of the faces included in the selected image. Then, the main face similarity, which is the similarity between the main face and other images other than the selected image, is calculated, the other images are re-sorted in descending order of the main face similarity, An image is displayed.

  For this reason, since it is possible to identify the main faces included in the image selected by the user and sort the images without acquiring a face image to be searched, the user can more easily find an image including the main face. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an external perspective view of an image print order receiving apparatus to which an image sorting apparatus according to an embodiment of the present invention is applied. As shown in FIG. 1, a print order receiving apparatus 1 according to the present embodiment is installed at a photo shop in order to receive an image print order by a user, and an image for ordering a print is recorded. Various memory cards 2 are loaded, a plurality of types of card slots 4 for reading out images from the memory card 2 and recording images on the memory card 2, and liquid crystal for performing various displays for ordering prints, etc. It has a display unit 6 and is connected to a printer 8 installed in a photo shop. The display unit 6 includes a touch panel type input unit, and the user can make an input necessary for the print order by touching the display unit 6 in accordance with the display on the display unit 6.

  FIG. 2 is a schematic block diagram showing the configuration of the print order receiving apparatus according to the embodiment of the present invention. As shown in FIG. 2, the print order receiving device 1 performs various controls such as recording control and display control of image data representing an image, and also controls each part of the device 1, and an image print order. A system order memory 14 composed of a print ordering program for performing the processing, a basic program for operating the apparatus 1 and a ROM in which various constants are recorded, and a RAM serving as a work area when the CPU 12 executes processing. The touch panel type input unit 16 for giving various instructions to the apparatus 1 and the display unit 6 described above are provided.

  The print order receiving apparatus 1 also includes a card slot 4 described above, a compression / decompression unit 22 that compresses image data by a technique typified by JPEG, and decompresses the compressed image data, and a hard disk that stores a plurality of images. 24, a memory control unit 26 for controlling the system memory 14, the card slot 4 and the hard disk 24, a display control unit 28 for controlling the display of the display unit 6, and a printer interface 30 for connecting the printer 8 to the apparatus 1. Is provided.

  The print order receiving apparatus 1 also includes a face detection unit 32 that detects a face area from each of a plurality of images stored in the hard disk 24, a similarity calculation unit 34, a labeling unit 36, a main face, as will be described later. The identification unit 38 and a sorting unit 40 that sorts a plurality of images are provided.

  The face detection unit 32 detects a face position on each image as a face region by a template matching method or a method using a face discriminator obtained by machine learning learning using a large number of sample images of a face. .

  First, the similarity calculation unit 34 calculates the similarity between faces included in each of a plurality of images stored in the hard disk 24 at a timing described later. Then, after the main face is specified, the main face similarity, which is the similarity between the main face and other images other than a plurality of images described later, is calculated.

  The similarity calculation unit 34 first detects main facial parts constituting the face T0 as face feature points for a predetermined face T0 included in any of the images used as a criterion for similarity determination. Specifically, for example, by using a template matching method using a template of each facial part or a method using a discriminator for each facial part obtained by machine learning learning using a large number of sample images of facial parts. The positions of a total of nine feature points of the eyes, left and right eyes, left and right noses, left and right mouth corners, and upper lip are detected.

  Then, an affine transformation parameter to be applied to the predetermined face T0 is obtained so that the nine detected feature points approach a predetermined reference position, and the affine transformation is performed on the predetermined face T0 to obtain a normal The converted face T1 is acquired.

  Then, the image data of the normalized face T1 is projected onto a lower-dimensional feature amount space using a projection matrix, and the feature amount FT1 in the normalized face T1 is extracted. Here, the feature amount space is determined by learning as described below.

  A projection matrix is analyzed by principal component analysis or linear discriminant analysis (LDA) analysis using the normalized learning face image group normalized as described above for a large number of learning face images having various facial expressions and orientations. And the eigenspace projected by this projection matrix or a space equivalent thereto is defined as the feature amount space.

  On the other hand, the similarity calculation unit 34 performs normalization in the same manner as described above on the target face P0 that is the target of similarity calculation among the faces included in each of the plurality of images, and acquires the normalized face P1. Further, the image data of the normalized face P1 is projected onto a lower dimensional feature amount space using a projection matrix, and the feature amount FP1 in the normalized face P1 is extracted.

  Then, the feature amount FT1 of the predetermined face T0 and the feature amount FP1 of the target face P0 are compared, and the similarity R1 between the predetermined face T0 and the target face P0 is calculated. Here, the similarity R1 is calculated using an AGM model (additional Gaussian model) described in, for example, JP-A-2005-149506. Here, the similarity R1 is calculated as a value between 0 and 1, and the closer to 1, the higher the similarity.

  The AGM model is a probabilistic model that assumes that face data can be expressed by the sum of variables representing individual differences and variables representing changes in the appearance of individuals (lighting changes, face orientation changes, secular changes, etc.). Shall follow a normal distribution. By estimating the parameters of each normal distribution in advance, it is possible to perform a robust similarity calculation process in consideration of changes in appearance even when the number of faces to be subjected to similarity calculation is small. For parameter estimation of each normal distribution, a learning face image group used when determining the feature amount space is used.

  In the feature amount space, the Euclidean distance between the coordinates of the face T0 specified by the feature amount FT1 and the coordinates of the target face P0 for similarity calculation specified by the feature amount FP1 is used as it is as the similarity R1. For example, other similarity calculation methods may be used.

  Then, the similarity calculation unit 34 sequentially changes the predetermined face T0 and calculates all the similarities R1 of the faces included in the plurality of images.

  On the other hand, the similarity calculation unit 34 calculates the main face similarity R2 that is the similarity between the main face and the target face P0 in the same manner as described above after the main face is specified as described later.

  The labeling unit 36 labels the faces included in the plurality of images based on the similarity R1 between the faces included in the plurality of images calculated by the similarity calculation unit 34. Specifically, faces having a similarity R1 of a predetermined threshold Th0 (for example, 0.9) or more are labeled with the same label.

  The main face specifying unit 38 specifies the main face included in the plurality of images based on the labeling result of the face included in the selected image as will be described later.

  The sorting unit 40 sorts the images in descending order of the main face similarity R2.

  Hereinafter, processing performed in the present embodiment will be described. FIG. 3 is a flowchart showing processing performed in the present embodiment. When the user touches the display unit 6 to instruct image sorting, the CPU 12 starts processing, and the sorting unit 40 sorts and displays the images stored in the hard disk 24 in a predetermined order. Displayed on the unit 6 (step ST1). The order of sorting is in order of shooting date and time in the present embodiment, but is not limited to this, and the images are sorted in descending order of the number of faces included in a plurality of images stored in the hard disk 24. May be. Then, the CPU 12 accepts selection of an image by the user from the displayed plurality of images (step ST2).

  FIG. 4 is a diagram showing a display screen for sorting results. As shown in FIG. 4, the sorting result display screen 50 is displayed in the display area 51 for displaying thumbnail images of a plurality of images in the sort order, and when the plurality of images cannot fit on one screen. Arrow buttons 52A and 52B for switching the images to be displayed are displayed.

  The user can select a plurality of images displayed in the display area 51 by touching the display unit 6 with an image to be printed, for example. In FIG. 4, when images G1 to G4 and images G6 and G7 are selected from the eight images G1 to G8 displayed in the display area 51, the selected images G1 to G4, images G6 and G7 are thick. A frame is given.

  Next, the CPU 13 starts monitoring whether or not an instruction to switch an image to be displayed in the display area 51 is given by the user touching the page buttons 52A and 52B (step ST3), and step ST3 is affirmed. The face detection unit 32 detects a face from a plurality of images (step ST4), and the similarity calculation unit 34 calculates the similarity R1 between the faces included in the plurality of images as described above (step ST5). The labeling unit 36 labels faces whose similarity R1 is equal to or greater than a predetermined threshold Th0 with the same label (step ST6).

  FIG. 5 shows the labeling result. FIG. 5 shows the labeling results of the images G1 to G8 displayed in the display area 51 of the display screen 50 shown in FIG. As shown in FIG. 5, images G1 to G8 include nine human faces, and labels A to I are assigned to the faces according to the similarity R1 between the faces.

  Then, the main face specifying unit 38 specifies the main face included in the plurality of images stored in the hard disk 24 based on the labeling result of the face included in the selected image (step ST7). Here, among the images G1 to G8, when a face included in the selected images G1 to G4, images G6 and G7 is viewed, a plurality of images including the faces A, B, C, D, and E are selected. . One image including faces F, G, H, and I is selected. There are images that are not selected for faces C, D, and F. The main face specifying unit 38 specifies the main face based on the selection status of the images including the faces A to I. Specifically, a face other than a face having an unselected image among the faces included in a plurality of selected images is set as a main face. Thereby, the faces A, B, E other than the faces C, D among the faces A, B, C, D, E are specified as the main faces.

  When there is no face with an unselected image, a face other than the face with only one selected image among the faces included in the plurality of selected images may be used as the main face. When there are no faces included in the selected images, a face other than the face with the unselected image among the faces included in the selected image may be used as the main face. A face included in a plurality of selected images may be used as the main face.

  Next, the similarity calculation unit 34 calculates a main face similarity R2 that is a similarity between the main faces A, B, and E and other images other than the selected image among the plurality of images stored in the hard disk 24. (Step ST8). Note that the main face similarity is calculated by calculating the similarity R1 with the predetermined face T0 as the main face A, B, E and the similarities with all the faces included in other images. This is done by calculating as Here, when the image includes a plurality of faces, as shown in FIG. 6, for example, the similarity R1 with the main face A is calculated for each face, and the highest similarity among the plurality of similarities R1 is calculated. Let R1 be the similarity R2 'between the main face A and its image. For example, as shown in FIG. 7, the image G0 includes four faces, and the similarity between each face and the main face A is 0.9, 0.7, 0.6, In the case of 0.2, the similarity R2 ′ between the main face A and the image G0 is calculated as 0.9.

  Here, in this embodiment, since there are three main faces A, B, and E, the similarity calculation unit 34 calculates the similarity R2 ′ for each of the three main faces A, B, and E. Then, the added value is calculated as the main face similarity R2. For this reason, an image including three faces A, B, and E has a high value of the main face similarity R2, and hereinafter, the value of the main face similarity R2 decreases as the number of the faces A, B, and E decreases. Becomes smaller. For example, in the images G11 to G13 shown in FIG. 8, when the image G11 includes the face A, the image G12 includes the faces A and B, and the image G13 includes the faces A, B, and E, the main face similarity R2 Is G13> G12> G11.

  Then, the sorting unit 40 re-sorts images other than the images selected in descending order of the main face similarity R2 (step ST8), and displays a plurality of images on the display unit 6 in the sorted order (step ST9). The process ends. When the main face similarity R2 is the same, the images are sorted in ascending order of shooting date / time. Further, it is not necessary to display all in order of similarity, and for example, only images whose main face similarity R2 exceeds a predetermined threshold value Th1 may be sorted and displayed.

  FIG. 9 is a diagram showing a display screen of the sorting result when sorting is performed in descending order of the main face similarity R2. As shown in FIG. 9, in the display area 54 of the display screen 53, the images G1 to G3, images G5 and G7 selected in FIG. 4 are displayed in this order. Note that the images G1 to G4, the images G6, and G7 are provided with a thick frame indicating that they have been selected. Further, in addition to the images G1 to G4, the images G6, and G7, among the three images shown in FIG. 8, for example, the images G12 and G13 including more main faces are displayed in the display area 54 in the order of G13 and G12. Is done.

  The user selects an image to be printed from the images displayed in the display area 54. Then, by performing a print instruction, the selected image is printed out from the printer 8.

  Thus, in the present embodiment, the main face similarity R2 that is the similarity between the main face and other images other than the selected image is calculated, and the images are re-sorted in descending order of the main face similarity R2. Therefore, as in the technique described in Patent Document 1, the main face included in the image selected by the user is identified and the image is acquired without acquiring the face image to be searched. Since sorting is possible, the user can more easily find an image including the main face from a plurality of images.

  In the above embodiment, the image is resorted at the timing of switching the image displayed in the display area 51. However, every time the user selects an image, another image other than the image selected is resorted. May be. In addition, after the user selects an image, re-sorting may be performed by instructing re-sorting.

  In the above embodiment, before sorting, a plurality of images may be classified in advance according to shooting locations and shooting times, and sorting may be performed in units of sorted images.

  In the above embodiment, the image sorting apparatus according to the present invention is applied to a print ordering apparatus installed at a photo shop. However, the present invention is not limited to this, and an order for placing a print order via a network is used. An image sort device may be applied to the server to sort images stored in the order server via the network.

  Although the apparatus according to the embodiment of the present invention has been described above, the computer functions as a unit corresponding to the face detection unit 32, the similarity calculation unit 34, the labeling unit 36, the main face specifying unit 38, and the sorting unit 40. A program that causes the processing as shown in FIG. 3 to be performed is also one embodiment of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention. Further, such a program may be incorporated in viewer software for browsing images.

1 is an external perspective view of a print order receiving apparatus to which an image sorting apparatus according to an embodiment of the present invention is applied. 1 is a schematic block diagram showing a configuration of a print order receiving apparatus according to an embodiment of the present invention. A flowchart showing processing performed in the present embodiment Figure showing the sort result display screen Diagram showing labeling results FIG. (1) for explaining calculation of similarity between a predetermined face and a face included in the image FIG. (2) for explaining the calculation of the similarity between a predetermined face and a face included in the image Diagram for explaining calculation of main face similarity The figure which shows the display screen of the sorting result when sorting in descending order of main face similarity

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Print order reception apparatus 2 Memory card 4 Card slot 6 Display part 8 Printer 12 CPU
DESCRIPTION OF SYMBOLS 14 System memory 16 Input part 22 Compression / decompression part 24 Hard disk 26 Memory control part 28 Display control part 30 Printer interface 32 Face detection part 34 Similarity calculation part 36 Labeling part 38 Main face specific part 40 Sort part

Claims (3)

In an image sorting device for sorting a plurality of images,
Sorting means for sorting the plurality of images in a predetermined order;
Display means for displaying the plurality of images;
Selection means for accepting selection of a plurality of desired images from the displayed plurality of images;
Similarity calculation means for calculating the similarity between faces included in the plurality of images at a predetermined timing;
Labeling means for labeling faces included in the plurality of images based on the similarity;
Main face specifying means for specifying at least one main face based on a labeling result of the face included in the selected image,
The similarity calculation means is a means for calculating a main face similarity which is a similarity between the main face and another image other than the selected image.
The sorting means is means for re-sorting the other images in descending order of the main face similarity and displaying the other images on the display means in the re-sorted order. . In the image sorting method for sorting multiple images,
Accepting selection of a plurality of desired images from a plurality of images displayed on the display means,
At a predetermined timing, the similarity between faces included in the plurality of images is calculated,
Label faces included in the plurality of images based on the similarity,
Identifying at least one primary face based on a labeling result of faces included in the selected image;
Calculating a main face similarity that is a similarity between the main face and another image other than the selected image;
The image sorting method characterized by re-sorting the other images in descending order of the main face similarity and displaying the other images on the display means in the re-sorted order. In a program for causing a computer to execute an image sorting method for sorting a plurality of images,
Receiving a selection of a plurality of desired images from a plurality of images displayed on the display means;
A procedure for calculating the similarity between faces included in the plurality of images at a predetermined timing;
Labeling faces included in the plurality of images based on the similarity;
Identifying at least one primary face based on a labeling result of faces included in the selected image;
Calculating a main face similarity that is a similarity between the main face and another image other than the selected image;
A program comprising: re-sorting the other images in descending order of the main face similarity, and displaying the other images on the display unit in the re-sorted order.

Images