JP4731303B2 - Image display device, image display method, program, and information storage medium - Google Patents

Description

  The present invention relates to an image display device, an image display method, a program, and an information storage medium.

  With the widespread use of digital cameras and hard disk recorders, many still images and moving images can be recorded in various media in electronic data format. 2. Description of the Related Art Conventionally, there has been known a method for displaying thumbnail images of each image as a list so that a desired image can be quickly accessed from a large number of images thus recorded.

  However, if an excessively large number of thumbnail images are displayed in the order of recording, for example, the user cannot quickly access a desired image.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image display device, an image display method, a program, and an information storage medium that allow a user to quickly find a desired image from among a large number of images. Is to provide.

  In order to solve the above-described problems, an image display device according to the present invention includes an image acquisition unit that acquires a plurality of images, and a plurality of types of image feature amounts for each of the images acquired by the image acquisition unit. Based on the value calculated by the image feature quantity calculating means for calculating the value and the value calculated by the image feature quantity calculating means, the position coordinate of each image in the first space having the coordinate axis corresponding to each image feature quantity is calculated. First image position coordinate calculating means, feature direction calculating means for calculating one or more feature directions in the first space based on the position coordinates calculated by the first image position coordinate calculating means, and the features Based on the feature direction calculated by the direction calculation unit and the position coordinate calculated by the first image position coordinate calculation unit, the feature direction calculation unit calculates Second image position coordinate calculation means for calculating the position coordinates of each image acquired by the image acquisition means in a second space having coordinate axes corresponding to the respective feature directions, and the second image position coordinate calculation means. Spatial image display means for displaying an image showing all or a part of the second space displayed according to the respective images at the respective position coordinates calculated by the above.

  The image display method according to the present invention includes an image acquisition step for acquiring a plurality of images, and an image feature for calculating each value of a plurality of types of image feature amounts for each of the images acquired in the image acquisition step. A first image position coordinate for calculating a position coordinate of each image in a first space having a coordinate axis corresponding to each image feature amount based on a value calculated in the amount calculation step and the image feature amount calculation step Calculated in the calculation step, the feature direction calculation step for calculating one or a plurality of feature directions in the first space based on the position coordinates calculated in the first image position coordinate calculation step, and the feature direction calculation step. Calculated in the feature direction calculation step based on the feature direction and the position coordinates calculated in the first image position coordinate calculation step. In a second space having a coordinate axis corresponding to each feature direction, a second image position coordinate calculating step for calculating a position coordinate of each image acquired in the image acquiring step, and a second image position coordinate calculating step A spatial image display step of displaying an image showing all or a part of the second space displayed according to each image at each calculated position coordinate.

  Further, the program according to the present invention includes an image acquisition unit that acquires a plurality of images, and an image feature amount calculation unit that calculates each value of a plurality of types of image feature amounts for each of the images acquired by the image acquisition unit. First image position coordinate calculation means for calculating position coordinates of each image in a first space having a coordinate axis corresponding to each image feature quantity based on a value calculated by the image feature quantity calculation means; Based on the position coordinates calculated by the first image position coordinate calculating means, feature direction calculating means for calculating one or a plurality of feature directions in the first space, the feature direction calculated by the feature direction calculating means, Based on the position coordinates calculated by the first image position coordinate calculating means, corresponding to the feature directions calculated by the feature direction calculating means. Second image position coordinate calculation means for calculating the position coordinates of each image acquired by the image acquisition means, and each position calculated by the second image position coordinate calculation means, in a second space having a target axis It is a program for causing a computer to function as a spatial image display means for displaying an image showing all or part of the second space displayed in coordinates according to each image. Furthermore, the information storage medium according to the present invention is a computer-readable information storage medium storing this program.

  In the present invention, for each image, each value of a plurality of types of image feature values is calculated, and the position coordinates of each image in the first space are calculated according to these values. The first space is a space having coordinate axes corresponding to each image feature amount. It can be evaluated that the images having the close position coordinates of the first space calculated in this way have similar characteristics. In the present invention, the feature direction in the first space is calculated. For example, the direction of the first space extending in a certain direction with the position coordinates of many images may be calculated as the feature direction. Thereafter, based on the feature direction thus calculated and the position coordinates of each image in the first space, the position coordinates of each image in the second space having the coordinate axis corresponding to each feature direction are calculated.

  The coordinate axis of the first space corresponds to each feature amount, and it can be evaluated that the feature direction indicates typical features of many acquired images. For example, an image that uses a lot of green and has a specific high spatial frequency component will be an image of a macro shot of a plant, and an image that uses a lot of blue and has a lot of low spatial frequency components will be summer. An image of the sea. In the present invention, each image is arranged in a second space having coordinate axes corresponding to each feature direction, and a part or all of the images are displayed. Therefore, each image and a large number of acquired images are displayed. The user can intuitively understand the relationship between typical characteristics.

  In one aspect of the present invention, the first image position coordinate calculation means calculates position coordinates having the value of each image feature amount as a coordinate component corresponding to the image feature amount for each image.

  In this case, the image feature amount calculating means may calculate at least a part of the value of the image feature amount based on color information in part or all of the images. In this way, the position of the image in the first and second spaces is determined according to part or all of the color of each image, and the user can search for a desired image based on the color.

  Further, the image feature amount calculating means may calculate at least a part of the value of the image feature amount based on a value of a predetermined spatial frequency component in a part or all of each image. In this way, the position of each image in the first and second spaces is determined according to the spatial frequency of the image, and the user can search for a desired image depending on the fineness of the image.

  Further, the image feature amount calculating means may calculate at least a part of the value of the image feature amount based on character information or numerical information associated with each image. In this way, the position of each image in the first and second spaces is determined in accordance with the textual information and numerical information such as the explanatory text of each image, the shooting date, etc., and the user can define the meaning content and shooting year of each image. A desired image can be searched depending on the date.

  Further, the image feature quantity calculating means may perform segmentation on each image and calculate at least a part of the value of the image feature quantity based on a value related to each segment. In this way, the position of each image in the first and second spaces is determined according to the characteristics of the segment of each image, and the user can search for a desired image according to the contents of the segment.

  At this time, the image feature amount calculating means may calculate at least a part of the value of the image feature amount based on at least one of the size or position of the segment represented in each image. . In this way, a desired image can be searched according to the size and position of the segment.

  In addition, each image may be divided into a plurality of partial areas, and the image feature amount calculating unit may calculate the value of the image feature amount for each partial area of each image. In this way, images having feature values similar to the same partial area can be arranged close to each other in the second space, and images of the same composition can be arranged close to each other in the second space.

  Further, the image feature amount calculating means may calculate at least a part of the value of the image feature amount based on a face recognition processing result in each image. In this way, images having similar face recognition processing results (for example, the presence / absence, position, size, etc. of the face) can be arranged close to each other in the second space.

  The feature direction calculation unit calculates one or a plurality of eigenvectors of a similarity matrix having the similarity between the position coordinates calculated by the first image position coordinate calculation unit as an element. The eigenvector may be the one or more feature directions.

  In this case, the second image position coordinate calculation unit is based on an inner product of the feature direction calculated by the feature direction calculation unit and the position coordinate calculated by the first image position coordinate calculation unit. You may make it calculate the position coordinate of each said image acquired by the said image acquisition means in the said 2nd space.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  In the image display apparatus according to the present embodiment, an image recorded by an imaging device such as a digital camera or an image collected on a network such as the Internet is displayed on a monitor so that the relevance between them can be easily understood. It is displayed and realized by executing a program according to the present embodiment on a computer.

  FIG. 1 is a conceptual diagram illustrating processing in the image display apparatus. In the present embodiment, first, a large number of images to be displayed are arranged on the first space 10. The first space 10 has a large number of coordinate axes 12, and each coordinate axis 12 corresponds to an image feature amount.

  For example, one of the coordinate axes 12 corresponds to the average value of the red component used in a predetermined area of the image, and the other one of the coordinate axes 12 represents the green component used in the predetermined area of the image. The other one of the coordinate axes 12 corresponds to the average value of the blue component used in a predetermined area of the image.

  The other coordinate axis 12 includes a specific word (for example, “sea” or “mountain” in character string data (for example, image description data or image file name data) associated with the image. ”, Etc.) (or a value indicating whether or not the same word is included), and the other one of the coordinate axes 12 is whether or not the image shooting date / time is within a predetermined period. Corresponds to the value indicating. Further, another one of the coordinate axes 12 corresponds to the number of segments having a predetermined size in a predetermined area when the image is subjected to segmentation processing. Corresponds to the number of faces (or a value indicating whether or not a face is represented). One of the coordinate axes 12 corresponds to the total amount of spatial frequency components in a predetermined range of the image.

  In the present embodiment, a large number of images to be displayed are acquired, and the value of each feature amount of each image is calculated. Then, the position coordinate of each image in the first space is calculated by using the value of each feature amount as a component of the coordinate axis corresponding to the feature amount. The following equation (1) is for calculating the position coordinate gi of the i-th image in the first space.

  The image to be displayed is divided into 12 parts vertically and horizontally as shown in FIG. 2, and a feature value is calculated for each partial area as necessary. In the equation (1), gki (k = 1 to m) is the k-th component of the position coordinate gi, and g1i to gpi are red used in the partial areas A1 to A12 of the image, respectively. , Green, and blue, the average values g′1i to g′pi are respectively multiplied by the weighting coefficient ωc.

  In the above equation (1), gp + 1i to gqi are obtained by multiplying the number of times g′p + 1i to g′qi each of which a specific word is included in the character string data associated with the image by the weighting coefficient ωl. It is done. In the above equation (1), gq + 1i to gri are obtained by multiplying values g′q + 1i to g′ri indicating whether the image shooting date / time is within a predetermined period by a weighting coefficient ωt.

  In the present embodiment, segmentation is performed on each image. Segmentation is a process of dividing into partial areas with uniform features, for example, a process of selecting, as each segment, a partial area composed of a pixel group to which color information within a predetermined color difference is assigned. In this process, as shown in FIG. 3, the gravity center position P, height H, and width W of each segment S are calculated. In the above equation (1), gr + 1i to gsi are the number g ′ of segments S in which the center of gravity P is arranged in each of the partial areas A1 to A12 of each image and the height H and width W satisfy the predetermined condition g ′. It is obtained by multiplying r + 1i to g′si by a weighting coefficient ωs.

  In the present embodiment, face recognition processing is applied to each image. In the above equation (1), gs + 1i multiplies the number of faces g ′s + 1i in the image by the weighting coefficient ωr. Is obtained. Further, in the above equation (1), gs + 2i to gmi are obtained by multiplying the total amount g ′s + 2i to g′mi of a predetermined range of spatial frequency components in each image by a weighting coefficient ωf. FIG. 1 shows a state in which each image is arranged at the position coordinates of the first space 10 calculated in this way.

  In the present embodiment, next, one or more feature directions of the first space 10 are calculated based on the position coordinates of the images arranged in the first space 10 in this way. The feature direction can be, for example, a direction indicating a location where images are concentrated and arranged in the first space 10. Here, the feature direction of the first space 10 is calculated as follows. That is, the following expression (2) calculates the similarity Hij between the position coordinate gi of the i-th image and the position coordinate gj of the j-th image in the first space 10, and as shown in the following expression (3): An N × N similarity matrix having the similarity Hij as an ij element is obtained. Here, N is the total number of words. Then, all or part of the eigenvectors of the similarity matrix H are used as feature directions.

  The similarity matrix is a real symmetric matrix, and its eigenvector can be calculated by the power method. The power method has an effect of strengthening the feature elements of the matrix elements and averaging the non-feature elements, and the eigenvector calculated thereby reveals the feature direction inherent in the first space 10. It can be evaluated as a thing. Note that the feature direction calculation method is not limited to the method for calculating the eigenvector of the similarity matrix H, and any method may be used as long as it calculates the characteristic direction in the first space. May be. For example, the direction in which images are concentrated in the first space 10 may be set as the characteristic direction.

  Next, in the present embodiment, a second space spanned by the coordinate axes corresponding to the characteristic directions thus calculated is prepared, and the position coordinates of each image in the second space are calculated. FIG. 4 is an example of an image showing the second space displayed by the image display device according to the present embodiment. Each coordinate axis 16 of the second space 14 corresponds to the feature direction of the first space 10 and shows typical features of a certain type of image. When obtaining the position coordinates of each image in the second space 14, specifically, a vector indicating the position coordinates of each image in the first space 10 and a unit vector indicating each feature direction in the first space 10 (the above similarity) The inner product of the eigenvector of the matrix H having a length of 1) is calculated, and the value is used as the component of the coordinate axis 16 corresponding to the feature direction among the position coordinates of the image in the second space 14. That is, the inner product of the unit vector ei (that is, the i-th eigenvector of the similarity matrix H having the magnitude 1) indicating the feature direction in the first space 10 and the position coordinate gj of the j-th image is The i-th component of the position coordinates of the j-th image in the second space 14 is used. However, if the total number of feature directions is M, 1 ≦ i ≦ M.

  Here, the configuration of the image display apparatus according to the present embodiment will be described. FIG. 5 is a functional block diagram of the image display apparatus 20 according to the present embodiment. The image display device 20 is realized by using various computer systems such as a personal computer and a home game machine, for example, and by installing and executing a program to which the present invention is applied, the image shown in FIG. Storage unit 22, image acquisition unit 24, image feature amount calculation unit 26, first image position coordinate calculation unit 28, similarity matrix calculation unit 30, eigenvalue / eigenvector calculation unit 32, second image position coordinate calculation unit 34, spatial image The display unit 36 is realized.

  The image storage unit 22 is composed of, for example, a hard disk storage device or the like, and stores images taken with a digital camera and images obtained via a communication network such as the Internet. Also, character string data indicating the description of each image and numerical data indicating the shooting date and time of each image are stored in association with those images.

  The image acquisition unit 24 sequentially acquires data of each image stored in the image storage unit 22, character string data and numerical data associated with each image, and passes them to the image feature amount calculation unit 26. The image feature amount calculation unit 26 calculates each value of a plurality of types of feature amounts for each image passed from the image acquisition unit 24. Here, as shown in FIG. 6, the image feature amount calculation unit 26 includes an average color calculation unit 26 a, a character information acquisition unit 26 b, a date and time information acquisition unit 26 c, a segment evaluation unit 26 d, a face recognition unit 26 e, and a DCT conversion unit 26 f. Is included. The average color calculation unit 26a calculates values g′1i to g′pi of red, green, and blue color components of the average colors of the partial areas A1 to A12 of each image as the feature value of each image. . The character information acquisition unit 26b calculates values g′p + 1i to g′qi of the number of times a predetermined word is used in the character string data associated with each image as the feature value of each image. The date and time information acquisition unit 26c calculates values g′q + 1i to g′ri indicating whether or not the shooting date and time indicated by the date and time data associated with each image is within a predetermined period as the feature value of each image. . The segment evaluation unit 26d uses the number of segments S satisfying a predetermined condition regarding the height H or the width W in which the gravity center position P is arranged in each of the partial areas A1 to A12 of each image as the feature value of each image. g′r + 1i to g′si are calculated. The face recognition unit 26e applies face recognition processing to each image, and calculates the number of faces g ′s + 1i represented in the image as the feature value of the image. For the face recognition processing, various known methods can be used. Further, the DCT transform unit 26f performs orthogonal transform processing on each image, and the total amount g ′s + 2i to g ′ of the spatial frequency components in a predetermined range of each of the partial regions A1 to A12 of the image as the feature amount of each image. Calculate mi.

  The first image position coordinate calculation unit 28 multiplies the feature value of each image calculated in this way by a weighting factor corresponding to the type of the feature value, and the position coordinates of the image in the first space 10. Gi (i = 1, 2, 3,...) is calculated. The position coordinates calculated in this way are supplied to the similarity matrix calculation unit 30 and the second image position coordinate calculation unit 34.

  The similarity matrix calculation unit 30 calculates the similarity Hij between the images (position coordinates in the first space 10) according to the above equation (2), and calculates the similarity matrix H using the similarity as a matrix element. The eigenvalue / eigenvector calculation unit 32 calculates the eigenvalue / eigenvector of the similarity matrix H thus calculated. Since the similarity matrix H is a real symmetric matrix, its eigenvalue / eigenvector can be easily obtained by a method suitable for computer processing such as a power method. The second image position coordinate calculation unit 34 selects a predetermined number from all the eigenvectors calculated in this way or in descending order of eigenvalues, converts them into unit vectors, and sets them as vectors indicating the characteristic direction of the first space 10. Then, the position coordinates of each image in the second space 14 are calculated by calculating the inner product of those vectors and the position coordinates gi supplied from the first image position coordinate calculation unit 28. These position coordinates are supplied to the spatial image display unit 36, which arranges each image at the position coordinate of the image in the second space 14 and sets the state in the second space 14. An image viewed from a given viewpoint (not shown) is generated and displayed on a monitor.

  According to the image display device 20 described above, by displaying an image showing the second space 14 on the monitor, it is possible to convey the relevance of a large number of images to the user in an easy-to-understand manner. You will be able to find it quickly.

It is a figure which shows the 1st space notionally. It is a figure explaining the method of calculating the value of the feature-value from an image. It is a figure which shows the value of the segment contained in an image. It is a figure which shows the image of 2nd space. 1 is a functional block diagram of an image display device according to an embodiment of the present invention. It is a figure which shows the detailed structure of an image feature-value calculation part.

Explanation of symbols

  10 first space, 12, 16 coordinate axes, 14 second space, 20 image display device, 22 image storage unit, 24 image acquisition unit, 26 image feature amount calculation unit, 26a average color calculation unit, 26b character information acquisition unit, 26c Date and time information acquisition unit, 26d segment evaluation unit, 26e face recognition unit, 26f DCT conversion unit, 28 first image position coordinate calculation unit, 30 similarity matrix calculation unit, 32 eigenvalue / eigenvector calculation unit, 34 second image position coordinate calculation Part, 36 spatial image display part.

Claims (12)

Image acquisition means for acquiring a plurality of images;
For each image acquired by the image acquisition means, an image feature value calculation means for calculating each value of a plurality of types of image feature values;
First image position coordinate calculation means for calculating the position coordinates of each image in a first space having a coordinate axis corresponding to each image feature value based on the value calculated by the image feature value calculation means;
Feature direction calculation means for calculating one or more feature directions in the first space based on the position coordinates calculated by the first image position coordinate calculation means;
Corresponding to each feature direction calculated by the feature direction calculation means based on the feature direction calculated by the feature direction calculation means and the position coordinates calculated by the first image position coordinate calculation means Second image position coordinate calculating means for calculating the position coordinates of each of the images acquired by the image acquiring means in a second space having coordinate axes;
Spatial image display means for displaying an image showing all or part of the second space displayed according to each image at each position coordinate calculated by the second image position coordinate calculation means;
Including
The image feature amount calculating means calculates at least a part of the value of the image feature amount based on the number of times a specific word is included in the character string data associated with each image, and each image is divided into a plurality of partial regions, it performs segmentation against respective regions, on the basis of the center of gravity of each segment definitive to each partial region among the plurality of segments, contained in the respective partial region Calculate the number of segments in which the height and width of each segment satisfy a predetermined condition, and calculate at least another part of the image feature value based on the number of segments included in each partial region To
An image display device characterized by that. The image display device according to claim 1,
The first image position coordinate calculation means calculates, for each image, position coordinates having the value of each image feature amount as a coordinate component corresponding to the image feature amount.
An image display device characterized by that. The image display device according to claim 2,
The image feature amount calculating means calculates at least a part of the value of the image feature amount based on color information in a part or all of each image.
An image display device characterized by that. The image display device according to claim 2 or 3,
The image feature amount calculating means calculates at least a part of the image feature amount value based on a value of a spatial frequency component of a predetermined range in a part or all of each image;
An image display device characterized by that. The image display device according to any one of claims 2 to 4,
The image feature amount calculating means calculates at least a part of the value of the image feature amount based on numerical information associated with each image.
An image display device characterized by that. The image display device according to any one of claims 1 to 5,
Each image is divided into a plurality of partial regions,
The image feature amount calculating means calculates a value of the image feature amount for each partial region of each image;
An image display device characterized by that. The image display device according to any one of claims 1 to 6,
The image feature amount calculating means calculates at least a part of the value of the image feature amount based on a face recognition processing result in each image;
An image display device characterized by that. The image display device according to claim 1,
The feature direction calculation means calculates one or a plurality of eigenvectors of a similarity matrix whose elements are similarities between the position coordinates calculated by the first image position coordinate calculation means, and the one or more eigenvectors are calculated. The one or more characteristic directions;
An image display device characterized by that. The image display device according to claim 8,
The second image position coordinate calculation unit is configured to calculate the second image position coordinate based on the inner product of the feature direction calculated by the feature direction calculation unit and the position coordinate calculated by the first image position coordinate calculation unit. Calculating the position coordinates of each image acquired by the image acquisition means in space;
An image display device characterized by that. An image acquisition step of acquiring a plurality of images;
For each image acquired in the image acquisition step, an image feature amount calculating step for calculating each value of a plurality of types of image feature amounts;
A first image position coordinate calculating step for calculating a position coordinate of each image in a first space having a coordinate axis corresponding to each image feature amount based on the value calculated in the image feature amount calculating step;
A feature direction calculating step of calculating one or more feature directions in the first space based on the position coordinates calculated in the first image position coordinate calculating step;
Corresponding to each feature direction calculated in the feature direction calculation step based on the feature direction calculated in the feature direction calculation step and the position coordinates calculated in the first image position coordinate calculation step A second image position coordinate calculating step for calculating a position coordinate of each of the images acquired in the image acquiring step in a second space having a coordinate axis.
A spatial image display step for displaying an image showing all or a part of the second space displayed according to each image at each position coordinate calculated in the second image position coordinate calculation step;
Including
The image feature amount calculating step calculates at least a part of the value of the image feature amount based on the number of times a specific word is included in the character string data associated with each image, and each image is divided into a plurality of partial regions, it performs segmentation against respective regions, on the basis of the center of gravity of each segment definitive to each partial region among the plurality of segments, contained in the respective partial region Calculate the number of segments in which the height and width of each segment satisfy a predetermined condition, and calculate at least another part of the image feature value based on the number of segments included in each partial region To
An image display method characterized by the above. Image acquisition means for acquiring a plurality of images;
For each image acquired by the image acquisition unit, an image feature amount calculation unit that calculates each value of a plurality of types of image feature amounts;
First image position coordinate calculation means for calculating position coordinates of each image in a first space having coordinate axes corresponding to each image feature quantity based on values calculated by the image feature quantity calculation means;
Feature direction calculation means for calculating one or more feature directions in the first space based on the position coordinates calculated by the first image position coordinate calculation means;
Corresponding to each feature direction calculated by the feature direction calculation means based on the feature direction calculated by the feature direction calculation means and the position coordinates calculated by the first image position coordinate calculation means A second image position coordinate calculation unit that calculates a position coordinate of each image acquired by the image acquisition unit in a second space having coordinate axes to perform, and each position calculated by the second image position coordinate calculation unit A program that causes a computer to function as a spatial image display unit that displays an image showing all or part of the second space in which coordinates are displayed according to each image,
The image feature amount calculating means calculates at least a part of the value of the image feature amount based on the number of times a specific word is included in the character string data associated with each image, and Each image is divided into a plurality of partial areas, segmented with respect to each of the areas, and based on the position of the center of gravity of each segment in each of the partial areas , the plurality of segments included in the partial areas Calculate the number of segments in which the height and width of each segment satisfy a predetermined condition, and calculate at least another part of the image feature value based on the number of segments included in each partial region ,
A program characterized by that. A computer-readable information storage medium storing the program according to claim 11.

Images