JP2008276668A - Image-managing device, image-displaying device, image pickup device, processing method in these devices, and program which makes computer execute the processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make it possible to select an image, which indicates the outline of all images, from among the images. <P>SOLUTION: A digest generation part 160 groups similar images from among a plurality of images on the basis of similarity characteristics of respective images recorded in a similarity characteristics recording part 141 and a similarity threshold value of a parameter A of a parameter storage part 150, adds weighting to image scores, which are recorded in an image score-recording part 142 in correspondence to the respective images included in each group, with a weighting coefficient of the parameter A to find weighting-added values, and selects representative images from the respective groups on the basis of the values. Further, the digest generation part 160 carries out image-selecting processing in a similar manner by using a similarity threshold value and a weighting coefficient of a parameter B to select at least one representative image from the representative images selected by using the parameter A. Thus, the representative image is selected, through the two-stage selection processing, from among the plurality of images stored in a picked-up image storage part 110. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image management apparatus, and more particularly to an image management apparatus, an image display apparatus, an imaging apparatus, a processing method in these, and a program for causing a computer to execute the method.

  In recent years, digital still cameras equipped with a large-capacity storage medium have become widespread. For this reason, it is possible to take a large number of photographs in one event. It is also possible to take a number of photos continuously with the same composition in a short time and store these photos in a storage medium.

  Furthermore, recently, digital still cameras capable of continuous shooting and bracket photography have become widespread. For this reason, it has been widely practiced to continuously take similar photographs by such continuous shooting or bracket photography, and store these photographs in a storage medium.

  As described above, the photograph taken with the digital still camera and stored in the storage medium can be browsed on the display using a personal computer (PC), for example. In the case of browsing using a personal computer in this way, for example, when browsing a relatively large number of photos taken continuously, it is possible to browse these photos in a list.

  Thus, when relatively many photographs are displayed in a list and viewed, similar photographs are often displayed continuously. In this case, it takes a long time to overlook the entire photograph. In addition, the slide show may end indefinitely, and time may elapse. In such a case, it is often impossible to enjoy a photograph taken.

  Therefore, when browsing a relatively large number of these photos, instead of listing all of these photos, it may be possible to select a relatively small number of photos and output these selected photos. It is done.

For example, based on the degree of similarity representing the degree of similarity between a plurality of image data, adjacent images of the plurality of image data are grouped and the evaluation value for each item of the image data is calculated. An image output device has been proposed in which corrected evaluation values obtained by correcting each evaluation value are calculated and image data higher than the corrected evaluation value in the same group is selected as print encouragement image data (for example, Patent Document 1). reference.).
JP 2006-259788 A (FIG. 6)

  According to the above-described conventional technology, it is possible to select image data higher in the correction evaluation value as the print encouragement image data in the same similar group. For this reason, it is possible to print an image photographed relatively well, not an image in which camera shake occurs or an image out of focus.

  Here, in the case where there are a large number of images taken in a relatively large number of events, consider the case of browsing these images. In such a case, it is convenient to quickly grasp the outline of each event and browse only the photos of the event of interest. However, although the above-described conventional technique can print a relatively beautifully shot image, a relatively small number of images cannot be appropriately selected when there are a plurality of dissimilar images. . For this reason, there exists a problem that the user cannot grasp | ascertain the outline | summary of each event rapidly.

  Therefore, an object of the present invention is to select an image showing an outline of the entire image from a plurality of images.

  The present invention has been made to solve the above problems, and a first aspect of the present invention is an image holding unit that holds a plurality of images, and a parameter in which a plurality of values are set. A parameter holding means for holding a first parameter and a second parameter, each of which is a parameter in which at least one of the values is set to a different value; and a feature amount of each image held in the image holding means. A plurality of features held in the image holding unit based on a feature amount recording unit to be recorded, a feature amount of an image recorded in the feature amount recording unit, and a first parameter held in the parameter holding unit First image selection means for selecting at least one representative image from the images, and feature quantities recorded in the feature quantity recording means corresponding to the selected representative images; Second image selection means for selecting at least one representative image from the representative images selected by the first image selection means based on the second parameter held in the parameter holding means. And a processing method thereof, and a program for causing a computer to execute the method. Accordingly, at least one representative image is selected from the plurality of images based on the feature amount of each image and the first parameter, and the feature amount corresponding to the selected representative image and the second parameter are selected. And at least one representative image is selected from the representative images.

  In the first aspect, the first parameter includes at least an image feature amount threshold, and the second parameter includes at least the image feature amount threshold included in the first parameter. The image feature value threshold value included in the first parameter includes a threshold value of the image feature value that is a different value. The condition of the image feature value threshold value included in the second parameter is relaxed. And the first image selection means sets the image based on the feature quantity of the image recorded in the feature quantity recording means and the threshold value of the first parameter held in the parameter holding means. A plurality of images held in the holding unit are classified and a representative image is selected from the classified plurality of images, and the second image selecting unit selects the representative image selected by the first image selecting unit. The representative image selected by the first image selection unit based on the feature amount recorded in the feature amount recording unit and the threshold value of the second parameter held in the parameter holding unit. At least one representative image can be selected from the plurality of classified representative images. Thereby, a plurality of images are classified based on the feature amount of the image and the threshold value of the first parameter, a representative image is selected from the plurality of classified images, and the selected representative image is further selected. The representative image is classified based on the corresponding feature amount and the threshold value of the second parameter, and at least one representative image is selected from the plurality of classified representative images. In this case, the threshold value of the feature amount of the image included in the first parameter and the second parameter is a similarity threshold value used for similarity determination between images, and the first image selection unit includes the feature amount. Similar images among a plurality of images held in the image holding unit based on the feature amount of the image recorded in the recording unit and the similarity threshold of the first parameter held in the parameter holding unit The second image selection unit stores the feature amount recorded in the feature amount recording unit and the parameter corresponding to the representative image selected by the first image selection unit. Based on the similarity threshold of the second parameter held in the means, the similar images of the representative images selected by the first image selection means are grouped and classified. Rukoto can. Thereby, based on the feature amount of the image and the similarity threshold of the first parameter, the similar images of the plurality of images are grouped and classified, and a representative image is selected from the plurality of classified images. In addition, based on the feature amount corresponding to the selected representative image and the similarity threshold value of the second parameter, the similar images of the selected representative images are grouped and classified. Bring.

  In the first aspect, the first parameter includes at least a weighting coefficient related to an image feature amount, and the second parameter includes at least a weighting coefficient related to an image feature amount included in the first parameter. The first image selection means includes a weighting coefficient relating to a feature quantity of the image having a value different from the first feature, and the first image selection means holds the feature quantity of the image recorded in the feature quantity recording means and the parameter holding means. A plurality of images held in the image holding means are classified based on one parameter, and the image feature quantity recorded in the feature quantity recording means and the first parameter held in the parameter holding means. A representative image is selected from the plurality of classified images based on the weighting coefficient of the parameter, and the second image selection means is configured to select the first image selection. Selected by the first image selection means based on the feature quantity recorded in the feature quantity recording means corresponding to the representative image selected by the means and the second parameter held in the parameter holding means The representative image is classified and the feature amount recorded in the feature amount recording unit corresponding to the representative image selected by the first image selection unit and the second stored in the parameter holding unit At least one representative image can be selected from the classified representative images based on the parameter weighting coefficient. As a result, a plurality of images are classified based on the feature amount of the image and the first parameter, and the representative is selected from the plurality of classified images based on the feature amount of the image and the weighting coefficient of the first parameter. An image is selected, the selected representative image is classified based on the feature amount corresponding to the selected representative image and the second parameter, and each representative image is selected from the plurality of classified representative images. Further, the representative image is further selected based on the feature amount corresponding to, and the weighting coefficient of the second parameter. In this case, the first image selection unit weights and adds the feature amount of the image recorded in the feature amount recording unit by the weighting coefficient of the first parameter held in the parameter holding unit. A representative image is selected from the plurality of classified images based on the weighted value, and the second image selection unit corresponds to the representative image selected by the first image selection unit. The feature quantity recorded in the quantity recording means is weighted and added by the weighting coefficient of the second parameter held in the parameter holding means, and from the plurality of representative images classified based on the weighted value. At least one representative image can be selected. As a result, the feature amount of the image is weighted and added by the weighting coefficient of the first parameter, and based on the weighted value, a representative image is selected from a plurality of images classified first, and the selected image is selected. The feature amount corresponding to the representative image is weighted and added by the weighting coefficient of the second parameter, and at least one representative image is selected from the plurality of classified representative images based on the weighted value. Bring. In this case, the weighting coefficient related to the image feature amount includes at least a weighting coefficient related to the degree of image blur, and the weighting coefficient related to the degree of image blur included in the second parameter corresponds to the first parameter. It can be set to a value that is lighter in weight than the weighting coefficient related to the degree of blur of the included image. As a result, with respect to the weighting related to the degree of blurring, the second weighted addition is performed using a lighter value than the first weighted addition.

  In the first aspect, the image processing apparatus further includes an operation receiving unit that receives an operation input for designating an upper limit number of representative images to be selected, and the second image selection unit includes the first image selection unit. It is determined whether or not the representative image selected by the number is less than or equal to the specified upper limit number. If the representative image is not less than or equal to the upper limit number, the representative image is held in the parameter holding unit based on the representative image. The second parameter value is rewritten and selected by the first image selection means based on the feature quantity recorded in the feature quantity recording means corresponding to the representative image and the rewritten second parameter value. At least one representative image can be selected from the displayed representative images. Thereby, it is determined whether or not the representative image selected first is less than or equal to the designated upper limit number. If the representative image is not less than or equal to the upper limit number, the second parameter is set based on the representative image. Is rewritten, and at least one representative image is selected from the first selected representative image based on the feature amount corresponding to the representative image and the rewritten second parameter value.

  In this first aspect, the parameter holding means holds a plurality of combinations of the first parameter and the second parameter that differ in at least one of a plurality of set values, and holds the parameter holding The apparatus may further include an operation receiving unit that receives an operation input for designating a desired combination from a plurality of parameter combinations held in the unit. This brings about the effect that a representative image is selected by designating a desired combination from a plurality of parameter combinations.

  Further, in the first aspect, the image processing apparatus further comprises event cluster recording means for classifying and recording each image held in the image holding means for each event, and the first image selection means includes the event cluster The representative image can be selected for each event recorded in the recording means. This brings about the effect that a representative image is selected for each event in which a plurality of images are classified.

  In the first aspect, the image processing apparatus may further include output means for outputting the representative image selected by the second image selection means. As a result, the representative image selected for the second time is output. In this case, the output means can be a display means for displaying the representative image selected by the second image selection means. As a result, the representative image selected for the second time is displayed.

  According to a third aspect of the present invention, there is provided an image holding unit that holds a plurality of images, and a parameter in which a plurality of values are set, and at least one of the plurality of values is set to a different value. Parameter holding means for holding the first parameter and the second parameter, the feature quantity recording means for recording the feature quantity of each image held in the image holding means, and the feature quantity recording means A first selection unit selects at least one representative image from a plurality of images held in the image holding unit based on a feature amount of an image recorded in the image and a first parameter held in the parameter holding unit. Image selection means, a feature quantity recorded in the feature quantity recording means corresponding to the selected representative image, and a second value held in the parameter holding means. A second image selecting means for selecting at least one representative image from the representative images selected by the first image selecting means based on the parameters, and the representative image selected by the second image selecting means. An image display apparatus comprising the display means, a processing method therefor, and a program for causing a computer to execute the method. Accordingly, at least one representative image is selected from the plurality of images based on the feature amount of each image and the first parameter, and the feature amount corresponding to the selected representative image and the second parameter are selected. Based on the above, at least one representative image is selected from the representative images, and the selected representative image is displayed.

  According to a third aspect of the present invention, there is provided an image holding unit for holding a plurality of captured images, and a parameter for which a plurality of values are set, wherein at least one of the plurality of values is different from each other. Parameter holding means for holding the first parameter and the second parameter that are parameters set in the above, a feature quantity recording means for recording the feature quantity of each image held in the image holding means, and the feature At least one representative image is selected from a plurality of images held in the image holding means based on the feature amount of the image recorded in the quantity recording means and the first parameter held in the parameter holding means A first image selection means, a feature quantity recorded in the feature quantity recording means corresponding to the selected representative image, and held in the parameter holding means. Second image selecting means for selecting at least one representative image from representative images selected by the first image selecting means based on the second parameter, and selected by the second image selecting means An image pickup apparatus, a processing method thereof, and a program for causing a computer to execute the method include display means for displaying a representative image. Accordingly, at least one representative image is selected from the plurality of images based on the feature amount of each image and the first parameter, and the feature amount corresponding to the selected representative image and the second parameter are selected. Based on the above, at least one representative image is selected from the representative images, and the selected representative image is displayed.

  According to the present invention, it is possible to achieve an excellent effect of being able to select an image showing an outline of the entire image from a plurality of images.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a functional configuration example of an image management apparatus 100 according to an embodiment of the present invention. The image management apparatus 100 includes a captured image holding unit 110, a time information acquisition unit 121, a clustering unit 122, an event cluster recording unit 123, a feature extraction unit 130, a similar feature recording unit 141, and an image score recording unit 142. A parameter storage unit 150, a digest generation unit 160, a display control unit 170, an operation reception unit 180, and a display unit 190.

  The photographed image holding unit 110 is a holding unit that holds a photographed image (photographic image) photographed by a digital still camera or the like. The captured image recorded in the captured image holding unit 110 stores an Exif (Exchangeable Image File Format) tag, and various metadata such as shooting position information and shooting time (Date Time Original) are recorded in the Exif tag. Is done. Each captured image is given a captured image ID that is an identifier of the captured image, and the captured image ID is recorded in the metadata.

  The time information acquisition unit 121 acquires time information from the Exif tag of each captured image stored in the captured image storage unit 110, and outputs the acquired time information to the clustering unit 122. The acquisition of the time information is performed as necessary when a captured image is recorded in the captured image holding unit 110.

  The clustering unit 122 classifies the captured images held in the captured image holding unit 110 for each event using a method called event clustering. In this event clustering, binary tree structure data with each photographed image as a leaf is used. Nodes (sections) in this binary tree structure data correspond to clusters. Specifically, the clustering unit 122 calculates a shooting interval between shot images based on the time information received from the time information acquisition unit 121, and generates a binary tree structure hierarchical cluster according to the calculated shooting interval. create. Then, the photographed images are classified for each event by determining the optimum cluster hierarchy based on the event cluster grouping condition from the created binary cluster structured hierarchical cluster. Thereby, a some picked-up image can be classify | categorized into the group arranged in order of the some time series of the unit called an event. One or more photographed images belong to this group. The creation of the hierarchical cluster will be described in detail with reference to FIG.

  The event cluster recording unit 123 records event cluster information including information on captured images classified for each event by the clustering unit 122 and information on event clusters corresponding to the event. In the event cluster recording unit 123, for example, a cluster ID, a photographed image list, and the like are recorded as event cluster information. The cluster ID is an identifier of each cluster in the binary tree structure hierarchical cluster, and an integer value is recorded, for example. The photographed image list is a list of photographed image IDs included in each cluster in the binary tree structure hierarchical cluster, and the photographed image ID of each photographed image is recorded.

  The feature extraction unit 130 performs various image analyzes on each image held in the captured image holding unit 110, extracts similar features, and calculates an image score. The similar features extracted by the feature extraction unit 130 are recorded in the similar feature recording unit 141, and the image score calculated by the feature extraction unit 130 is recorded in the image score recording unit 142. Note that these various types of image analysis may be performed on the photographed image at any time when the photographed image is recorded in the photographed image holding unit 110, and each photographed image is analyzed based on an analysis start request from the user. You may make it carry out collectively about an image.

  Here, the similar features and image scores obtained by image analysis by the feature extraction unit 130 will be described in detail. Note that, when the feature extraction unit 130 performs image analysis, resolution conversion is performed on a captured image that is an object of image analysis, and the size of each captured image is made constant.

  The similar feature indicates a feature amount of an image used when determining the degree of similarity between a reference image and another image, and can be obtained using a color distribution, a frequency distribution, or the like of a captured image. . Then, the obtained feature amount is recorded in the similar feature recording unit 141.

  The image score is a value used when a representative image is selected from a plurality of images, and is an evaluation value representing the appearance of the image. As an image score, for example, a score based on the degree of blur of the image (blur_score), a score based on the color spread of the image (color_variation), a score based on the number of faces included in the image (face_count), and a score of the face included in the image There are a score based on the position (face_position), a score based on the area of the face included in the image (face_area), a score based on the degree of smile related to the face included in the image (smile), and the like. In the embodiment of the present invention, description is made using the six image scores shown here, but image scores other than these image scores may be used. Further, at least one of these image scores may be used. Further, a combination of at least one of these image scores and another image score may be used.

The score based on the degree of blur of the image (blurring degree score: BS) is a score indicating the height of the frequency included in the image, and can be obtained by the following equation for the FFT response of the image. That is, when there are few high frequency components, it is estimated that the image is blurred. Note that, in order to apply FFT to an image whose one side is a power of 2, calculation is performed using a portion obtained by cutting out 256 × 128 or 128 × 256 from the center of the image.

  Here, x and y represent positions in the FFT response, r represents the distance from the center, ν represents the FFT response value, w represents the width of the image, and h represents the height of the image. Indicates.

  The score based on the color spread of the image is a score indicating the diversity of colors included in the image. For example, a dispersion value in a color space (here, La * b * color space) of pixels included in the image can be used.

  Each of the score based on the number of faces included in the image, the score based on the position of the face included in the image, the score based on the area of the face included in the image, and the score based on the degree of smile regarding the face included in the image The image score is a score obtained based on face information detected from an image.

  The score based on the number of faces included in the image is a score indicating the number of faces included in the image, and the numerical value of the number of faces is used as it is.

The score FS based on the position of the face included in the image is a score indicating the degree to which the face position is reflected at an appropriate position near the center of the image. For example, the average position of the center of gravity of all the faces included in the image can be obtained using the following equation (two-dimensional normal distribution equation).

Here, x and y indicate the average positions of the centroids of all the faces included in the image. In this case, a value normalized to [−0.5, 0.5] with the center of the image as the origin is used. Further, (μ _x , μ _y ) = (0.500, 0.431) and (σ _x , σ _y ) = (0.159, 0.118).

  The score based on the area of the face included in the image is a score indicating the size of the face included in the image. When a plurality of faces are included in the image, calculation is performed using an average value of the areas of the plurality of faces.

  The score based on the degree of smile related to the face included in the image is a score indicating the degree of smile related to the face detected from the image. In addition, when a plurality of faces are included in the image, an average value of values indicating the degree of smile regarding the plurality of faces is used. When determining the degree of smile, a smile related to the detected face is determined using a known face recognition technique, and the degree of smile is quantified.

  The similar feature recording unit 141 is a recording unit that records the similar feature of the photographed image extracted by the feature extraction unit 130 in association with the photographed image ID.

  The image score recording unit 142 is a recording unit that records the image score of the captured image calculated by the feature extraction unit 130 in association with the captured image ID.

  The parameter holding unit 150 holds parameters used when the digest generation unit 160 executes a captured image selection process. This parameter will be described in detail with reference to FIG.

  Based on the similar feature of the captured image recorded in the similar feature recording unit 141, the digest generation unit 160 determines between the captured images for a plurality of captured images belonging to each event cluster recorded in the event cluster recording unit 123. The similarity distance is calculated, and one or a plurality of groups are created based on the calculation result and the parameters held in the parameter holding unit 150. In addition, the digest generation unit 160 is created based on the similar characteristics of the captured image based on the image score of the captured image recorded in the image score recording unit 142 and the parameters stored in the parameter storage unit 150. One captured image is selected as a representative image from one or a plurality of captured images belonging to each group. Furthermore, the digest generation unit 160 calculates the similarity distance between the captured images for a plurality of captured images selected as the representative image based on the similar features of the captured images recorded in the similar feature recording unit 141. One or more groups are created based on the calculation results and the parameters held in the parameter holding unit 150. The digest generation unit 160 also belongs to each group created from the representative image based on the image score of the captured image recorded in the image score recording unit 142 and the parameter stored in the parameter storage unit 150. Alternatively, one captured image is further selected as a representative image from a plurality of captured images. The grouping and selection of these captured images will be described in detail with reference to FIGS. 4 and 5.

  The display control unit 170 acquires, from the digest generation unit 160, representative image information related to the representative image selected for each event when an operation input for instructing the digest viewing of the captured image is received from the operation reception unit 180. In addition, the photographed image is read from the photographed image holding unit 110, the photographed image is classified for each event based on the read representative image information, and the browse view is configured to be displayed on the display unit 190. It is. A display example of the browsing view is shown in FIG.

  The operation reception unit 180 includes various operation buttons. When an operation input is received from these buttons, the operation reception unit 180 outputs the contents of the received operation input to the parameter holding unit 150, the digest generation unit 160, and the display control unit 170. is there. For example, the operation reception unit 180 is provided with a digest generation button for giving an instruction for selecting a representative image, a digest browsing button for giving an instruction for displaying the selected representative image, and the like. Note that a digest generation button, a digest browsing button, and the like may be displayed on the display unit 190, and these buttons may be pressed according to an operation input of the operation reception unit 180. Further, at least a part of the operation receiving unit 180 and the display unit 190 may be integrated as a touch panel.

  The display unit 190 is a display unit that displays a captured image or the like held in the captured image holding unit 110 based on control from the display control unit 170. As the display unit 190, for example, a liquid crystal display (LCD) can be used.

  FIG. 2A is a graph showing a relationship between the shooting frequency of a plurality of images held in the shot image holding unit 110 and the shooting time.

  In the graph shown in FIG. 2A, the horizontal axis indicates the time axis, and the vertical axis indicates the imaging frequency. For example, a portion where the value of the photographing frequency is large indicates that many photographings are performed in a short time. In the graph shown in FIG. 2A, for example, a section 201 indicates a period in which “Family picnic” is an event, a section 203 indicates a period in which “Children's athletic meet” is an event, and a section 205 indicates a period in which “fishing with friends” is present as an event, and sections 202 and 204 indicate periods in which there is no particular event. For example, in “family picnics”, there are many opportunities to shoot at lunch time, so the frequency of shooting increases in the middle of the section 201. In addition, in the “children's athletic meet”, there are many opportunities to photograph children's activities before and after lunch, and there are fewer opportunities to photograph at lunch time, so the frequency of photographing increases at the front and rear portions of the section 203, and the middle portion. Then, the shooting frequency is low. In addition, in “fishing with friends”, there are many opportunities to shoot at the start, and there are fewer opportunities to shoot after starting fishing, so the shooting frequency increases in the front part of the section 203 and the shooting frequency in the rear part. Lower.

  As shown in FIG. 2A, in the sections 201, 203, and 205 where each event occurred, the shooting frequency value was relatively large, and in the sections 202 and 204 where there was no special event, the shooting frequency value was Get smaller. In this way, it is possible to determine the presence or absence of an event according to the magnitude of the photographing frequency value. Thereby, a section in which the value of the imaging frequency is continuously large is classified as one event. For example, photographed images photographed in the sections 201, 203, and 205 are classified into respective events.

  FIG. 2B is a diagram illustrating an outline of event clustering performed when the clustering unit 122 classifies captured images held in the captured image holding unit 110 for each event. As shown in FIG. 2B, for example, on the time axis 210, the captured images 211 to 216 are arranged in time series at the positions of the capturing times. In this case, the shooting intervals of the shot images 211 to 216 are calculated based on the shooting times of the shot images 211 to 216. Then, each cluster is created based on the calculation result. Note that the leaves of the binary tree shown in FIG. 2B correspond to the captured images 211 to 216. Each created cluster corresponds to a node (node) of the binary tree shown in FIG.

  For example, a cluster 224 is created with the captured images 215 and 216 with the shortest shooting interval, and a cluster 221 is created with the captured images 211 and 212 with the second shortest shooting interval. Subsequently, the respective shooting intervals between the created clusters 221 and 224 and the shot images 213 and 214 are calculated, and further clusters are created based on the calculation results. For example, a cluster 222 is created from the cluster 221 and the captured image 213, and a cluster 225 is created from the cluster 224 and the captured image 214. Further, a cluster 223 to which all of the captured images 211 to 216 belong is created by the clusters 222 and 225.

  Subsequently, an optimal cluster hierarchy is determined from the clusters 221 to 225 in the created binary tree based on the event cluster grouping condition. The event cluster grouping condition is set based on, for example, the shooting frequency of the shot images belonging to each cluster. For example, in the example shown in FIG. 2B, the interval between the captured images 211 to 213 and the captured images 214 to 216 is short, and the interval between the captured images 213 and 214 is relatively long. Yes. For this reason, the value of the shooting frequency of the shot images belonging to the clusters 222 and 225 is relatively large, whereas the value of the shooting frequency of the shot images belonging to the cluster 223 is relatively small. Therefore, the clusters 222 and 225 are determined from the clusters 221 to 225. Thus, the captured images 211 to 213 belonging to the cluster 222 are determined as one event, the captured images 214 to 216 belonging to the cluster 225 are determined to be one event, and the captured images 211 to 216 are classified into two events. Is done. In this way, the captured images 211 to 216 are classified into groups composed of captured images arranged in a plurality of time series in units called event clusters.

  FIG. 3 is a diagram illustrating an example of parameters held in the parameter holding unit 150. Two parameters A and B are used as the parameter 230 used when the digest generation unit 160 executes the picked-up image selection process. The two parameters A and B have different values for at least one item.

  The parameters 230 include “similarity threshold”, “weighting coefficient of blur degree”, “weighting coefficient of color spread”, “weighting coefficient of the number of faces”, “weighting coefficient of face position”, “face size” "Weighting coefficient" and "smiling degree weighting coefficient" are recorded in parameter A and parameter B, respectively. In this example, in parameter A and parameter B, different values are recorded for “similar threshold” and “weighting coefficient of blur degree”, and the same values are recorded for other items.

  The “similar threshold” is a threshold used when it is determined whether or not two captured images to be compared are similar. For example, if the difference (similar distance) between the similar feature values of two captured images recorded in the similar feature recording unit 141 is equal to or smaller than the value of the “similar threshold”, it is determined that the two captured images are similar. . Thus, using the “similarity threshold”, similar captured images are grouped, and a plurality of captured images are classified into one or a plurality of groups.

  Each weighting coefficient such as “weighting coefficient of blur degree” is a weighting coefficient used when one photographed image is selected from the photographed images grouped using the “similar threshold”. For example, in each group, each image score of each photographed image recorded in the image score recording unit 142 is normalized, and this normalized image score is weighted and added using each weighting coefficient. The added value is calculated as an evaluation value for each captured image. In each group, the calculated evaluation values are compared, and as a result of the comparison, a photographed image having the largest evaluation value is selected as a representative image. Note that selection of a captured image using these two parameters A and B will be described in detail with reference to FIG.

  FIG. 4 is a transition diagram schematically showing a transition when a representative image is selected from captured images belonging to one event. Here, of the captured images held in the captured image holding unit 110, two captured images 313 and 317 are selected from the captured images 311 to 318 belonging to the event 310 recorded in the event cluster recording unit 123. The case will be described.

  FIG. 4A is a diagram illustrating captured images 311 to 318 belonging to the event 310 among the captured images held in the captured image holding unit 110. Note that the captured images 311 to 318 are arranged from the captured image 311 in the order of the capturing time. The captured images 311 to 318 are images captured when camping with four families (father, mother, daughter, and son). The photographed image 311 is an image of a scene where four families are preparing a tent in Kawahara, and the photographed images 312 to 315 are photographed when a father and son are fishing in Kawahara. The captured images 316 to 318 are images in which a scene where four families are resting in the grassland is captured. Here, it is assumed that the captured images 313 and 314 have similar persons, backgrounds, positions of persons, and the like included in the images. Also, the captured images 317 and 318 are similar in the person included in the image, the background, the arrangement of people in the entire image, and the like. Further, the photographed image 316 and the photographed images 317 and 318 are similar in the arrangement of people in the entire image and the like, the person included in the image, the atmosphere and color of the background, and the like. Further, the photographed images 313 and 314 and the photographed images 311, 312, and 315 are similar in background atmosphere, color, etc., although the people included in the images and the arrangement of people in the whole are slightly different. .

  First, regarding the captured images 311 to 318 shown in FIG. 4A, similar captured images are grouped using the “similar threshold” value “10000” recorded in the parameter A of the parameter 230. . This grouping is sequentially performed for a plurality of captured images in time series. In this case, for the first captured image in time series in the event, there is no previous image in time series and there is no comparison target, so a new group is assigned, and the second image in time series Sequentially taken from subsequent captured images. For example, in the case of the captured images 311 to 318, a new group 321 is assigned. Then, the second captured image 312 is extracted in time series, and the value of the similar feature recorded in the similar feature recording unit 141 corresponding to the captured images 311 and 312 is extracted. Then, a difference (similar distance) between similar feature values of the captured images 311 and 312 is calculated, and it is determined whether the difference value is equal to or smaller than the similarity threshold “10000” of the parameter A. When the difference value is equal to or smaller than the similarity threshold “10000” of the parameter A, it is determined that the two captured images are similar, and the two captured images are included in one group. On the other hand, when the difference value is not equal to or smaller than the similarity threshold “10000” of the parameter A, it is determined that the two captured images are not similar, and the two captured images are included in another group. In this way, for the captured images 311 to 318, the difference between the values of similar features between adjacent captured images is sequentially calculated and grouped.

  For example, as illustrated in FIG. 4A, the captured images 311, 312, 315, and 316 are determined not to be similar to adjacent captured images, and are assigned to one group 321, 322, 324, and 325, respectively. It is done. On the other hand, the captured images 313 and 314 and the captured images 317 and 318 are determined to be similar to the adjacent captured images and are assigned to groups 323 and 326 including two captured images.

  Subsequently, for each of the groups 321 to 326 shown in FIG. 4A, one representative image is selected from each group using the value of each weighting coefficient recorded in the parameter A of the parameter 230. This selection is performed sequentially for each group. In this case, for a group including only one captured image, the captured image is selected as a representative image. For example, for the groups 321, 322, 324, and 325 including only one captured image, the captured images 311, 312, 315, and 316 are selected as representative images, respectively.

  For a group including a plurality of captured images, one captured image is selected from the plurality of captured images. For example, in the case of the group 323, each value of the image score recorded in the image score recording unit 142 corresponding to the captured images 313 and 314 included in the group 323 is extracted. Then, each value of the image score of the captured images 313 and 314 is normalized. For example, each value of the image score is normalized so that the highest score in the group is “1.0” and the lowest score is “0.0”.

  Subsequently, the normalized image score of each captured image is weighted and added using each weighting coefficient of parameter A. A value obtained by weighted addition is used as an evaluation value of each captured image. Then, the photographed image having the largest evaluation value among the evaluation values of the photographed images in the group is selected as the representative image.

  For example, in the case of the group 323, it is determined that the evaluation value of the captured image 313 is large among the evaluation values of the captured images 313 and 314 included in the group 323, and the captured image 313 is selected as the representative image. In the case of the group 326, it is determined that the evaluation value of the captured image 317 is large among the evaluation values of the captured images 317 and 318 included in the group 326, and the captured image 317 is selected as the representative image. Thus, one representative image is selected from each group.

  FIG. 4B is a diagram showing a representative image group 330 selected from each of the groups 321 to 326. For example, photographed images 311 to 313 and 315 to 317 are selected as representative images from the groups 321 to 326, respectively.

  For the photographed images 311 to 313 and 315 to 317 selected as the representative images, similar photographed images are grouped using the “similar threshold” value “35000” recorded in the parameter B of the parameter 230. . Since this grouping is the same as that shown in FIG. 4A except that the parameter “similarity threshold” is different, detailed description thereof is omitted here.

  Here, the setting of “similar threshold” will be described. As shown in FIG. 4A, in the first grouping, a “similarity threshold” is set so that very similar captured images are grouped. By setting in this way, it is possible to group, for example, continuously shot photographs and images that have been retaken twice and three times with the same composition. Further, as shown in FIG. 4B, a relatively loose “similarity threshold” is set in the second grouping. By setting in this way, it is possible to group images having different subjects but similar in appearance. As described above, a relatively large number of captured images can be grouped by increasing the “similarity threshold value” and grouping the captured images selected as the representative images.

  For example, as shown in FIG. 4B, the captured images 315 and 316 are not determined to be similar captured images, but the captured images 311 to 313 and 315 are determined to be similar to the adjacent captured images. Each captured image is assigned to the group 331. The captured images 316 and 317 are determined to be similar to the adjacent captured images, and each captured image is assigned to the group 332.

  Subsequently, for the groups 331 and 332 shown in FIG. 4B, one representative image is selected from each group using the value of each weighting coefficient recorded in the parameter B of the parameter 230. Note that the selection of the representative image is the same as that shown in FIG. 4A except that the parameter “weighting coefficient for blur degree” is different, and thus detailed description thereof is omitted here.

  Here, the setting of each weighting coefficient will be described. First, the “weighting coefficient for the degree of blur” will be described.

  The score based on the degree of blur of the image is a score indicating the height of the frequency included in the image. For example, when comparing two images that are very similar, if the frequency of one image is high and the frequency of the other image is low, the low frequency image is relatively blurred or out of focus, etc. Can be judged. That is, since very similar shot images are grouped in the first grouping, the first representative image is selected in consideration of a score based on the degree of blurring of the image and the image is relatively blurred. It is possible to select an image other than as a representative image.

  On the other hand, when the subjects are greatly different, it is often meaningless to simply compare the heights of the frequencies. For example, if the subject is empty, the frequency is low, and if the subject is forest, the frequency is high. For this reason, even if both frequencies are compared, it cannot be determined whether or not the image itself is blurred. That is, in the second stage grouping, captured images with different subjects are also grouped, and therefore, the selection of the representative image in the second stage does not consider a score based on the degree of image blur. Thus, in selecting the first representative image, “1.0” is set as the value of the “weighting coefficient of the degree of blur” in order to consider the score based on the degree of blur of the image. On the other hand, in the selection of the first representative image, the score based on the degree of blur of the image is not taken into consideration, and therefore “0.0” is set as the value of the “weighting coefficient of the degree of blur”.

  Next, the “weighting coefficient for the number of faces”, “weighting coefficient for the face position”, “weighting coefficient for the face size”, and “weighting coefficient for the degree of smile” will be described. In the parameters A and B of the parameter 230 shown in FIG. 3, these four weighting coefficients use the same value. However, as shown below, these values may be appropriately changed and used in each of A and B.

  For example, very similar captured images are grouped in the first grouping. For this reason, in the selection of the first representative image, among the very similar captured images, an image with a face, an image with a face in the middle, an image with a large face, or an image with a smiling face As described above, the values of these four weighting factors can be set relatively large or small so that an image according to the user's preference is selected as a representative image.

  On the other hand, in the second grouping, captured images with different subjects are also grouped. For this reason, in the selection of the representative image at the second stage, for example, when it is desired to preferentially select an image showing a face among photographed images with different subjects, the value of the “weighting coefficient for the number of faces” is selected. Can be set larger. In addition, when it is desired to preferentially select an image with a large face, only the value of “weighting coefficient of face size” can be set relatively large. Thus, it can adjust according to a user's liking.

  As described above, the setting of what images are grouped and what image is selected as the representative image is used in the parameters used in the first image selection processing and the image selection processing in the second stage. Since the parameter can be set differently, it is possible to select an appropriate representative image according to the user's preference.

  For example, in the case of the group 331, it is determined that the evaluation value of the captured image 313 is large among the evaluation values of the captured images 311 to 313 and 315 included in the group 331, and the captured image 313 is selected as the representative image. . In the case of the group 332, it is determined that the evaluation value of the photographed image 317 is large among the evaluation values of the photographed images 316 and 317 included in the group 332, and the photographed image 317 is selected as the representative image.

  FIG. 4C shows a representative image group 340 selected from each of the groups 331 and 332. For example, the captured images 313 and 317 are selected as representative images from the groups 331 and 332, respectively. As described above, the photographed images 313 and 317 are selected from the photographed images 311 to 318 belonging to the event 310 among the photographed images held in the photographed image holding unit 110.

  FIG. 5 is a diagram illustrating a display example when the captured image held in the captured image holding unit 110 is displayed on the display unit 190.

  FIG. 5A is a diagram illustrating a display example when the captured images 311 to 318 belonging to the event 310 are displayed on the display unit 190 among the captured images retained in the captured image retaining unit 110. . As shown in FIG. 5A, the plurality of captured images held in the captured image holding unit 110 can be displayed on the display unit 190 for each event recorded in the event cluster recording unit 123. Note that the display example shown in FIG. 5A shows a display example of the captured images 311 to 318 after the digest generation button is pressed. When the digest generation button is pressed, representative images 313 and 317 are selected for the captured images 311 to 318 as shown in FIG. Then, representative image marks 351 and 361 indicating that the selected representative images 313 and 317 are selected as the representative images are attached to the lower right portion of the image, for example. Further, group frames 350 and 360 for indicating from which captured image the representative image is selected are displayed so as to surround each captured image, for example.

  FIG. 5B is a diagram illustrating a display example in the case where representative images 313 and 317 selected from the captured images 311 to 318 belonging to the event 310 are displayed on the display unit 190. As shown in FIG. 5A, when the digest browsing button is pressed in a state after the digest generation button is pressed, as shown in FIG. Only the representative image selected from the images can be displayed. In addition, an image for indicating how many photographed images the representative image is selected is displayed on the upper side and the right side of each representative image. For example, since the representative image 313 is a representative image selected from five photographed images, the representative image 313 is displayed so that five images are overlaid as shown in FIG. Further, since the representative image 317 is a representative image selected from three photographed images, the representative image 317 is displayed so that three images are superimposed as shown in FIG. 5B. For example, when there are many photographed images belonging to each event, by displaying only the representative images selected in this way, an overview of each event such as the person who participated in each event and the status of each event can be quickly obtained. I can grasp it. Although FIG. 5 shows an example in which only the captured image belonging to one event is displayed on the display unit 190, a plurality of events may be displayed on the display unit 190 simultaneously. As shown in FIG. 5B, when only the representative images are displayed, the display area of each event may be reduced according to the number of representative images, and a large number of events may be displayed.

  Next, the operation of the image management apparatus 100 according to the embodiment of the present invention will be described with reference to the drawings.

  FIG. 6 is a flowchart showing a processing procedure of a feature extraction process of a captured image by the image management apparatus 100. Here, an operation of extracting a similar feature from a captured image held in the captured image holding unit 110, calculating an image score, and recording the image score in the similar feature recording unit 141 and the image score recording unit 142 will be described.

  First, the captured image held in the captured image holding unit 110 is input to the feature extraction unit 130 (step S901). Subsequently, resolution conversion is performed on each input captured image so that the size of each captured image becomes a constant size (step S902). Note that the size of the captured image subjected to resolution conversion is set to be 640 × 480 pixels in the case of a horizontally long image, for example, and 480 × 640 pixels in the case of a vertically long image. Set to

  Subsequently, similar features are extracted for each captured image whose resolution has been converted (step S903), and the extracted similar features are recorded in the similar feature recording unit 141 in association with the captured images held in the captured image holding unit 110. Is done.

  Subsequently, an image score is calculated for each captured image whose resolution has been converted (step S904), and the calculated image score is recorded in the image score recording unit 142 in association with the captured image stored in the captured image storage unit 110. Is done. As described above, after the similar features related to the respective captured images held in the captured image holding unit 110 are recorded in the similar feature recording unit 141 and the image score is recorded in the image score recording unit 142, the feature extraction process of the captured image is performed. Exit.

  FIG. 7 is a flowchart illustrating a processing procedure of event cluster creation processing by the image management apparatus 100.

  First, the captured image stored in the captured image storage unit 110 is input to the time information acquisition unit 121 (step S911). Subsequently, the time information acquisition unit 121 acquires time information from the Exif tag of the input captured image (step S912). When the acquired time information is input to the clustering unit 122, a shooting interval between captured images is calculated based on the time information (step S913), and a binary tree structure is formed according to the calculated shooting interval. A cluster is created (step S914).

  Subsequently, it is determined whether or not there is a non-clustered captured image (step S915). If there is a non-clustered captured image, the process returns to step S912 to repeat the cluster creation processing procedure (step S915). S912 to step S914).

  On the other hand, if there is no captured image that has not been clustered (step S915), an optimal cluster hierarchy is determined based on the event cluster grouping conditions from among the created binary clusters of the binary tree structure. Then, each captured image is classified for each event (step S916). For example, as shown in FIG. 2A, each captured image is classified into each event corresponding to each of the sections 201, 203, and 205. Event cluster information corresponding to each classified event is recorded in the event cluster recording unit 123, and the event cluster creation process is terminated.

  FIG. 8 is a flowchart illustrating a processing procedure of representative image selection processing by the image management apparatus 100. Here, an operation when the digest generation button is pressed in the operation reception unit 180 will be described.

  First, event cluster information recorded in the event cluster recording unit 123 is input to the digest generation unit 160 (step S921). Subsequently, event cluster information corresponding to one event is extracted from the input event cluster information (step S922). For example, as shown in FIG. 2A, when the event is classified into three events corresponding to the sections 201, 203, and 205, event cluster information corresponding to the section 201 that is the first event is extracted.

  Subsequently, the parameters held in the parameter holding unit 150 are acquired (step S923). Here, since this is the first selection process, the first parameter is acquired. For example, parameter A of parameter 230 shown in FIG. 3 is acquired.

  Subsequently, image selection processing is executed for each captured image belonging to the event extracted in step S922 (step S930). This image selection process will be described in detail with reference to FIG.

  Subsequently, for each captured image belonging to the event extracted in step S922, it is determined whether or not the image selection process executed in step S930 is the first image selection process (step S924). If the image selection process executed in step S930 is the first image selection process (step S924), the parameters held in the parameter holding unit 150 are different from the parameters used first. Is acquired (step S925). For example, parameter B of parameter 230 shown in FIG. 3 is acquired. Then, it returns to step S930 and repeats an image selection process.

  On the other hand, if the image selection process executed in step S930 is not the first image selection process but the second image selection process (step S924), whether or not the image selection process has been completed for all events. (Step S926), and if the image selection process has not been completed for all events, the process returns to step S922, event cluster information corresponding to another event is extracted (step S922), and the image selection process is executed. (Step S922 to Step S925, Step S930). In this way, image selection processing is executed using two different parameters for each captured image belonging to one event.

  Subsequently, if the image selection processing has been completed for all events (step S926), each captured image selected as the representative image is classified for each event and displayed on the display unit 190 (step S927). For example, as shown in FIG. 5B, the representative image is displayed on the display unit 190. In step S927, as shown in FIG. 5A, each captured image belonging to the event is displayed, a representative image mark and a group frame are displayed, and after the digest view button is pressed, FIG. ), Each captured image selected as the representative image may be displayed on the display unit 190 after being classified for each event.

  As described above, an optimal captured image can be selected by sequentially narrowing down a plurality of captured images for each event in two stages. It should be noted that in the initial image narrowing down (step S930), it is necessary to eliminate the redundancy of extremely similar images such as continuously shot images or images in which a plurality of images are shot with the same composition. As an objective, narrow down the image. In the second image narrowing (step S930), the representative images selected by the first image narrowing are grouped based on similar distances on a relatively loose basis, and the number of images is further narrowed down for viewing production. Narrow the image for the purpose of improving.

  In the embodiment of the present invention, the method of processing the same algorithm twice using different parameters in the image narrowing process (image selection process) (step S930) has been described as an example. However, the first image selection process and the second image selection process may be performed using appropriate different algorithms.

  FIG. 9 is a flowchart showing an image selection processing procedure (processing procedure of step S930 shown in FIG. 8) in the processing procedure of representative image selection processing by the image management apparatus 100.

  First, a similar image grouping process is executed for each captured image belonging to the event extracted in step S922 (shown in FIG. 8) (step S940). This grouping process of similar images will be described in detail with reference to FIG.

  Subsequently, one group is extracted from one or a plurality of groups created by the grouping process of similar images (step S931). Subsequently, a representative image selection process is executed for the extracted group (step S950). This representative image selection process will be described in detail with reference to FIG.

  Subsequently, it is determined whether or not the representative image selection process has been completed for all the groups created in step S940 (step S932). If the representative image selection process has not been completed for all the groups, step S940 is performed. Returning to S931, representative image selection processing is executed (steps S931 and S950). On the other hand, if the representative image selection process has been completed for all the groups created in step S940 (step S932), the representative image selection process ends.

  FIG. 10 is a flowchart showing a similar image grouping processing procedure (processing procedure of step S940 shown in FIG. 9) in the processing procedure of image selection processing by the image management apparatus 100.

  First, one captured image is extracted from a plurality of captured images belonging to the event extracted in step S922 (shown in FIG. 8) (step S941). In this case, each captured image belonging to the event is extracted in order of the shooting time. In the event extracted in step S922, a new group is assigned to the first captured image in time series because there is no previous image in time series and there is no comparison target. For this reason, it is sequentially extracted from the second and subsequent captured images in time series. For example, the captured image 312 is extracted at the event 310 illustrated in FIG.

  Subsequently, the value of the similar feature between the taken-out photographed image and the previous photographed image in time series is taken out from the similar feature recording unit 141, and the similarity distance (similarity) between two adjacent photographed images is obtained. Calculated (step S942). Specifically, a difference value between two similar feature values extracted from the similar feature recording unit 141 is obtained by calculation.

  Subsequently, it is determined whether or not the value of the similarity distance obtained by calculation is equal to or less than the value of the parameter similarity threshold (step S943). Here, the parameters used are, for example, the parameter A shown in FIG. 3 in the case of the first image selection process, and the parameter B in the case of the second image selection process.

  Subsequently, when the calculated similarity distance value is not less than or equal to the parameter similarity threshold value (step S943), the taken-out photographed image and the previous photographed image in time series are not. A new group is created for the taken image that is determined to be similar (step S944). For example, as illustrated in FIG. 4A, when the similarity value between the captured image 311 and the captured image 312 is not less than or equal to the parameter similarity threshold, a new group 322 for the captured image 312 is created. .

  On the other hand, if the calculated similarity distance value is equal to or smaller than the parameter similarity threshold value (step S943), the taken image and the previous image taken in time series are obtained. Since they are similar, the extracted captured image is assigned to the same group as the previous captured image in time series (step S945). For example, as illustrated in FIG. 4A, when the value of the similarity distance between the captured image 313 and the captured image 314 is equal to or smaller than the parameter similarity threshold, the captured image 314 is the same as the immediately preceding captured image 313. Assigned to group 323.

  Subsequently, it is determined whether or not the similar image grouping process has been completed for all of the plurality of photographed images belonging to the event extracted in step S922 (shown in FIG. 8) (step S946). If the similar image grouping process has not been completed for the image, the process returns to step S941, and the similar image grouping process is executed (steps S941 to S945). On the other hand, if the similar image grouping process has been completed for all the captured images (step S946), the similar image grouping process is terminated. In this manner, similar captured images are grouped based on the similar distance between the captured images.

  FIG. 11 is a flowchart showing a representative image selection processing procedure (processing procedure of step S950 shown in FIG. 9) in the processing procedure of image selection processing by the image management apparatus 100.

  First, an image score relating to each captured image belonging to the group extracted in step S931 (shown in FIG. 9) is acquired from the image score recording unit 142. Then, each image score of each acquired image is normalized in one group (step S951). For example, each image score of each captured image is normalized so that the highest score in one group is “1.0” and the lowest score is “0.0”.

  Subsequently, the normalized image score of each captured image is weighted and added using each parameter weighting coefficient (step S952). The value obtained by weighted addition is used as the evaluation value of each captured image. The parameters used here are, for example, the parameter A shown in FIG. 3 in the case of the first image selection process, and the parameter B in the case of the second image selection process.

  Subsequently, among the evaluation values of the respective captured images in the group, the captured image having the maximum evaluation value is selected as a representative image (step S953). For example, in the case of the first image selection process, it is determined that the evaluation value of the photographed image 313 is large among the evaluation values of the photographed images 313 and 314 included in the group 323 as illustrated in FIG. The photographed image 313 is selected as a representative image.

  FIG. 12 is a flowchart illustrating a processing procedure of representative image selection processing by the image management apparatus 100. This representative image selection process is a modification of part of the representative image selection process shown in FIG. 8, and is different in that steps S961 to S963 are executed instead of step S925 shown in FIG. For this reason, Steps S961 to S963 will be described here, and the others are the same as the representative image selection process shown in FIG.

  In this example, the operation accepting unit 180 allows the user to specify the upper limit number of representative images selected in each event. As a method for specifying the upper limit number, for example, a method of directly inputting a number using a numeric keypad provided in the operation reception unit 180 and specifying a selection button such as 10 sheets or 20 sheets is selected. In preparation for this, a method of designating by pressing these selection buttons can be considered. Further, an upper limit number may be set in advance as a system default value. Instead of providing these various keys and selection buttons in the operation receiving unit 180, these various keys and selection buttons are displayed on the display unit 190 and designated by operating these various keys and selection buttons on a touch panel or the like. You may make it do. In this example, a case where N (N: natural number) is designated as the upper limit number will be described.

  In this example, the first image selection process (step S930) for each captured image belonging to the event extracted in step S922 is the same as the representative image selection process shown in FIG. Then, after the first image selection process (step S930) (step S924), the number of photographed images selected as representative images for the event extracted in step S922 is equal to or less than the designated upper limit number (N). Is determined (step S961).

  If the number of selected representative images is equal to or less than the designated upper limit number (step S961), the process proceeds to step S926, and it is determined whether or not the image selection process has been completed for all events (step S926). ). As described above, when the number of representative images is equal to or less than the designated upper limit number, the representative image selected by the first image selection process is set as the final representative image, and the second image selection process (step S930). Do not execute.

  On the other hand, if the number of selected representative images is not less than or equal to the designated upper limit number (step S961), among the parameters held in the parameter holding unit 150, there is a parameter different from the parameter used first. Obtained (step S962). For example, parameter B of parameter 230 shown in FIG. 3 is acquired.

  Subsequently, a similarity threshold used for the second image selection process (step S930) is calculated (step S963). Specifically, with respect to the event extracted in step S922, the similarity distance between each adjacent image of each captured image selected as the representative image is calculated. Subsequently, the similar distances obtained by calculation are sorted in descending order of the similar distances. Subsequently, for each sorted similar distance, the value of the Nth similar distance from the head is taken out, and the value of the Nth similar distance is obtained as a similarity threshold (step S963).

  Subsequently, the value of the similarity distance obtained in step S963 is set instead of the similarity threshold value of parameter B of the parameter 230 acquired in step S962 (step S964). That is, only the similarity threshold value in the parameter B of the parameter 230 is replaced with the value of the similarity distance obtained in step S963. Then, it returns to step S930 and repeats an image selection process.

  Thus, by using the similarity threshold obtained in step S963 for the second image selection process (step S930), the number of representative images selected in the second image selection process is set to the designated upper limit number. The following can be achieved.

  In addition, by designating the upper limit number for each event in this way, it is possible to display a predetermined number of representative images for each event even when the number of photographed images belonging to each event is greatly different. This makes it easier for the user to view the representative image for the captured image belonging to each event.

  In addition, for example, one event can be expressed by one thumbnail image, and a large number of events can be easily viewed in a short time.

  In the embodiment of the present invention, parameter A of parameter 230 is used in the first image selection process (step S930), and parameter B of parameter 230 is used in the second image selection process (step S930). Although an example using parameters of one pattern has been described, parameters of a plurality of patterns having different values for each item are held in the parameter holding unit 150, and a desired pattern is designated by a user operation from these patterns. You may do it. Further, in the operation reception unit 180, the user may input and set the values of the items of the two parameters.

  For example, as the patterns to be prepared, “face-oriented” and “landscape-oriented” patterns can be considered. In the case of “Focus on face”, the “Face weighting coefficient”, “Face position weighting coefficient”, “Face size weighting coefficient”, and “Smiling degree weighting coefficient” parameters Increase the weighting factor. Thereby, a representative image with an emphasis on the face can be selected. When an image with an emphasis on the face is selected, for example, an image showing a face is easier to select than an image showing no face. It is easy to select an image including

  Further, in the case of “landscape-oriented”, the value of the weighting coefficient for the item “weighting coefficient of color spread” is increased. In this case, it can be set so that an image having a good color arrangement is selected as a representative image without being greatly affected by whether or not a face is reflected in the image.

  Further, in the embodiment of the present invention, grouping is performed based on the similar distance between the captured images, but grouping may be performed based on the feature amount of another image. For example, the shooting time of the shot image can be used.

  In the embodiment of the present invention, when grouping processing of similar images is performed, the previous captured image in time series is the captured image to be compared, but the captured image to be compared. May be taken as the next captured image in time series. Further, the captured image to be compared may be set to every other captured image that is adjacent in time series, every other captured image, or the like.

  In the embodiment of the present invention, the example in which the representative image selected in the second image selection process is displayed on the display unit 190 has been described. However, the embodiment of the present invention also applies to the case where the representative image is printed. Can be applied. Further, an output unit that outputs the representative image may be provided in the image management apparatus 100, and the representative image may be output by another image output apparatus. Thereby, even when there are a large number of images, only a suitable small number of images can be output.

  In the embodiment of the present invention, an example in which image selection processing including grouping processing of similar images and selection processing of representative images is performed in two stages using different parameter values to select a representative image will be described. However, the embodiment of the present invention can also be applied to a case where a representative image is selected by executing image selection processing in three or more stages using different parameter values.

  In the embodiment of the present invention, the image management apparatus has been described as an example. However, the present invention is applied to an image display apparatus such as a portable terminal apparatus that can hold a plurality of images and an imaging apparatus such as a digital still camera. The embodiment can be applied.

  As described above, according to the embodiment of the present invention, it is possible to narrow down an appropriate captured image from among these captured images by the digest generating unit 160 for a large number of captured images captured by a digital still camera or the like. For this reason, when displaying a some image on the display part 190, the browsing property can be improved. In addition, when a list of photographed images such as photographs is displayed as a thumbnail image on the display unit 190, it is not necessary to perform a scroll operation from the operation receiving unit 180 over many pages, so that the viewability can be improved. . Further, also in the slide show reproduction on the display unit 190, it is possible to eliminate the redundancy that similar images are continuously reproduced on the display unit 190. As described above, according to the embodiment of the present invention, it is possible to select an image showing an outline of the entire image from a plurality of images.

  The embodiment of the present invention is an example for embodying the present invention and has a corresponding relationship with the invention-specific matters in the claims as shown below, but is not limited thereto. However, various modifications can be made without departing from the scope of the present invention.

  That is, in claims 1 to 11, the image management apparatus corresponds to, for example, the image management apparatus 100. The image display device corresponds to the image management device 100, for example. In claim 13, an imaging apparatus corresponds to the image management apparatus 100, for example.

  Further, in claims 1, 12, and 13, the image holding unit corresponds to the captured image holding unit 110, for example. The feature amount recording unit corresponds to, for example, the similar feature recording unit 141 or the image score recording unit 142.

  Further, in claims 1, 8, 12, and 13, the parameter holding means corresponds to the parameter holding unit 150, for example.

  Further, in claims 1 to 5, 9, 12, and 13, the first image selection unit corresponds to, for example, the digest generation unit 160.

  Further, in claims 1 to 5, 7, 12, and 13, the second image selection unit corresponds to, for example, the digest generation unit 160.

  Further, in claim 7 or claim 8, the operation reception means corresponds to, for example, the operation reception unit 180.

  Further, in claim 9, the event cluster recording means corresponds to, for example, the event cluster recording unit 123.

  Further, in claim 10, the output means corresponds to the display unit 190, for example.

  Further, in claims 11 to 13, display means corresponds to the display unit 190, for example.

  In claim 14 or claim 15, the first image selection procedure corresponds to, for example, step S <b> 930. The second image selection procedure corresponds to, for example, step S930.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program May be taken as

2 is a block diagram illustrating a functional configuration example of an image management apparatus 100. FIG. It is a figure showing the relation between the photography frequency of a plurality of pictures, and photography time, and the figure showing the outline of the event clustering performed when classifying the photography picture for every event. 6 is a diagram illustrating an example of parameters held in a parameter holding unit 150. FIG. It is a transition diagram which shows roughly the transition in the case of selecting a representative image from the picked-up image which belongs to one event. 6 is a diagram illustrating a display example when a captured image held in a captured image holding unit 110 is displayed on a display unit 190. FIG. 5 is a flowchart illustrating a processing procedure of a feature extraction process of a captured image by the image management apparatus 100. 4 is a flowchart illustrating a processing procedure for creating an event cluster by the image management apparatus 100. 4 is a flowchart illustrating a processing procedure of representative image selection processing by the image management apparatus 100. 5 is a flowchart illustrating an image selection processing procedure in a processing procedure of representative image selection processing by the image management apparatus 100. 6 is a flowchart illustrating a similar image grouping process procedure in an image selection process performed by the image management apparatus 100. 4 is a flowchart illustrating a representative image selection processing procedure in an image selection processing procedure performed by the image management apparatus 100. 4 is a flowchart illustrating a processing procedure of representative image selection processing by the image management apparatus 100.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image management apparatus 110 Captured image holding part 121 Time information acquisition part 122 Clustering part 123 Event cluster recording part 130 Feature extraction part 141 Similar feature recording part 142 Image score recording part 150 Parameter holding part 160 Digest generation part 170 Display control part 180 Operation Reception part 190 Display part

Claims (15)

Image holding means for holding a plurality of images;
Parameter holding means for holding a first parameter and a second parameter that are parameters in which a plurality of values are set and at least one of the plurality of values is set to a different value;
Feature quantity recording means for recording the feature quantity of each image held in the image holding means;
At least one representative image from the plurality of images held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. First image selecting means for selecting
Selected by the first image selection unit based on the feature amount recorded in the feature amount recording unit corresponding to the selected representative image and the second parameter held in the parameter holding unit. And a second image selection means for selecting at least one representative image from the representative images. The first parameter includes at least a threshold value of an image feature amount,
The second parameter includes an image feature amount threshold value that is different from at least the image feature amount threshold value included in the first parameter;
The threshold value of the feature amount of the image included in the first parameter is set to a value whose condition is relaxed than the threshold value of the feature amount of the image included in the second parameter,
The first image selection unit is held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the threshold value of the first parameter held in the parameter holding unit. Classify a plurality of images and select a representative image from the classified images,
The second image selection means corresponds to the representative quantity selected by the first image selection means, and the feature quantity recorded in the feature quantity recording means and the second held in the parameter holding means. 2. The representative image selected by the first image selection unit is classified based on a threshold value of the parameter, and at least one representative image is selected from the plurality of classified representative images. The image management apparatus described. The threshold value of the feature amount of the image included in the first parameter and the second parameter is a similarity threshold value used for similarity determination between images,
The first image selection unit holds the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the similarity threshold value of the first parameter held in the parameter holding unit. Group images that are similar to each other,
The second image selection means corresponds to the representative quantity selected by the first image selection means, and the feature quantity recorded in the feature quantity recording means and the second held in the parameter holding means. 3. The image management apparatus according to claim 2, wherein similar images of the representative images selected by the first image selection unit are grouped and classified based on a similarity threshold of the parameters. The first parameter includes at least a weighting coefficient related to the feature amount of the image,
The second parameter includes at least a weighting coefficient related to an image feature value that is different from a weighting coefficient related to an image feature value included in the first parameter;
The first image selection unit is held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. Based on the feature quantity of the image recorded in the feature quantity recording means by classifying a plurality of images and the weighting coefficient of the first parameter held in the parameter holding means, from the classified images Select a representative image,
The second image selection means corresponds to the representative quantity selected by the first image selection means, and the feature quantity recorded in the feature quantity recording means and the second held in the parameter holding means. And the representative image selected by the first image selection means is classified and recorded in the feature amount recording means corresponding to the representative image selected by the first image selection means. The at least one representative image is selected from the plurality of classified representative images based on a feature amount and a weighting coefficient of a second parameter held in the parameter holding unit. Image management device. The first image selection unit weights and adds the feature amount of the image recorded in the feature amount recording unit by the weighting coefficient of the first parameter held in the parameter holding unit, and the weighted addition is performed. Select a representative image from a plurality of classified images based on the value,
The second image selection means stores the feature quantity recorded in the feature quantity recording means corresponding to the representative image selected by the first image selection means in the parameter holding means. 5. The image management apparatus according to claim 4, wherein at least one representative image is selected from the plurality of classified representative images based on the weighted addition value by the weighting coefficient of the parameter. The weighting coefficient related to the feature amount of the image includes at least a weighting coefficient related to the degree of blur of the image,
5. The weighting coefficient related to the degree of blur of the image included in the second parameter is set to a value that is lighter than the weighting coefficient related to the degree of blur of the image included in the first parameter. The image management apparatus described. An operation receiving means for receiving an operation input for designating an upper limit number of representative images to be selected;
The second image selection means determines whether the representative image selected by the first image selection means is less than or equal to the designated upper limit number, and when the representative image is not less than or equal to the upper limit number. Rewrites the value of the second parameter held in the parameter holding means based on the representative image and the feature quantity recorded in the feature quantity recording means corresponding to the representative image and the rewritten second 2. The image management apparatus according to claim 1, wherein at least one representative image is selected from the representative images selected by the first image selection unit based on a parameter value. The parameter holding means holds a plurality of combinations of the first parameter and the second parameter that are different in at least one of a plurality of set values,
The image management apparatus according to claim 1, further comprising an operation receiving unit that receives an operation input for designating a desired combination from a plurality of parameter combinations held in the parameter holding unit. Event cluster recording means for classifying and recording each image held in the image holding means for each event,
The image management apparatus according to claim 1, wherein the first image selection unit selects the representative image for each event recorded in the event cluster recording unit. 2. The image management apparatus according to claim 1, further comprising output means for outputting the representative image selected by the second image selection means. The image management apparatus according to claim 10, wherein the output unit is a display unit that displays a representative image selected by the second image selection unit. Image holding means for holding a plurality of images;
Parameter holding means for holding a first parameter and a second parameter that are parameters in which a plurality of values are set and at least one of the plurality of values is set to a different value;
Feature quantity recording means for recording the feature quantity of each image held in the image holding means;
At least one representative image from the plurality of images held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. First image selecting means for selecting
Selected by the first image selection unit based on the feature amount recorded in the feature amount recording unit corresponding to the selected representative image and the second parameter held in the parameter holding unit. Second image selecting means for selecting at least one representative image from the representative images;
An image display apparatus comprising: display means for displaying a representative image selected by the second image selection means. Image holding means for holding a plurality of captured images;
Parameter holding means for holding a first parameter and a second parameter that are parameters in which a plurality of values are set and at least one of the plurality of values is set to a different value;
Feature quantity recording means for recording the feature quantity of each image held in the image holding means;
At least one representative image from the plurality of images held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. First image selecting means for selecting
Selected by the first image selection unit based on the feature amount recorded in the feature amount recording unit corresponding to the selected representative image and the second parameter held in the parameter holding unit. Second image selecting means for selecting at least one representative image from the representative images;
An image pickup apparatus comprising: display means for displaying a representative image selected by the second image selection means. Image holding means for holding a plurality of images, a first parameter and a first parameter that are parameters for which a plurality of values are set and at least one of the plurality of values is set to a different value. In an image management method in an image management apparatus comprising: a parameter holding unit that holds two parameters; and a feature amount recording unit that records a feature amount of each image held in the image holding unit,
At least one representative image from the plurality of images held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. A first image selection procedure for selecting
At least one selected from the selected representative image based on the feature quantity recorded in the feature quantity recording means corresponding to the selected representative image and the second parameter held in the parameter holding means. An image management method comprising: a second image selection procedure for selecting a representative image. Image holding means for holding a plurality of images, a first parameter and a first parameter that are parameters for which a plurality of values are set and at least one of the plurality of values is set to a different value. An image management apparatus comprising: a parameter holding unit that holds two parameters; and a feature amount recording unit that records a feature amount of each image held in the image holding unit;
At least one representative image from the plurality of images held in the image holding unit based on the feature amount of the image recorded in the feature amount recording unit and the first parameter held in the parameter holding unit. A first image selection procedure for selecting
At least one selected from the selected representative image based on the feature quantity recorded in the feature quantity recording means corresponding to the selected representative image and the second parameter held in the parameter holding means. A program causing a computer to execute a second image selection procedure for selecting a representative image.

Images