JP3457617B2 - Image search system and image search method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval system and an image retrieval method for retrieving and displaying an image stored in a database according to a retrieval key, and in particular, it has been improved to improve the browseability of retrieved images. The present invention relates to an image search system and an image search method.

[0002]

2. Description of the Related Art In recent years, with the development of image processing technology by computers and the widespread use of digital images, the development of image search systems for searching and displaying digital images stored in a database has been advanced. The image search system searches the database for an image similar to the input key image and presents it to the user. In this case, the retrieved images are usually displayed in a list in order from the image having the highest similarity to the key image.

There is also known an image search system using a narrowing-down search technique of gradually narrowing down target images from a list of searched images (Japanese Patent Laid-Open No. 10-162020).

[0004]

By the way, the purpose of image retrieval is mainly to search the database for an image that is the same as or has a very high degree of similarity with the key image, but recently, the degree of similarity with the key image has been found. By including images that are not very expensive in the search target, it is also beginning to be used for the purpose of extracting as many similar images as possible from the database and comparing and matching them with key images.

For example, when a new product is developed, it is investigated whether or not a product similar to the design of the product is already on sale, or it is similar to a character or mark to be newly used for the product or the like. For example, when investigating whether something already exists. In such an investigation use, the search itself for the same image is not the purpose, and therefore it is also required to search for an image or the like having a partially similar pattern.

Therefore, it is necessary to display an image whose degree of similarity to the key image is not so high to judge the degree of similarity to the key image, a partial similar portion, and the like. The presentation of candidates will be required.

In such image retrieval for the purpose of investigation, sufficient browsing performance cannot be expected by the conventional method of sequentially displaying similar image candidates in descending order of similarity to the key image. This is because an image having a very high degree of similarity to the key image is preferentially displayed, but a partially similar image that is similar only to a certain specific portion is determined to have a low degree of similarity. . In this case, with regard to such partially similar images, since the similarities between the images displayed adjacently are low, the images having similar points different from each other are displayed side by side, rather than the key images. It becomes difficult to compare.

The present invention has been made in view of the above circumstances, and makes it possible to present a search result in consideration of not only the similarity with the key image but also the similarity between the searched images. It is an object of the present invention to provide an image search system and an image search method that are optimal for image search such as for research purposes in which presentation of many similar image candidates is required.

[0009]

In order to solve the above problems, the present invention is an image retrieval system for retrieving and displaying an image stored in a database based on a key image, and a plurality of images are retrieved according to the key image. When the image is searched for, the cluster classification for classifying the searched plurality of images into clusters so that similar images are included in the same cluster based on the image feature extracted from each of the searched plurality of images. Means and for each of the retrieved images
Determines whether the similarity to the key image is greater than a predetermined value
Similarity determination means and the cluster classification means
The result of rasterization and the determination result of the similarity determination means
Based on this, the similarity with the key image is larger than a predetermined value.
For images that are not displayed, search results are displayed in the order of similarity.
Is displayed on the screen and the similarity to the key image is below a specified value
Image is classified into each cluster and the search result table
As shown on the display screen.
Display means for displaying an image on the search result display screen .

[0010]

[0011]

[0012]

For images having a very high similarity to the key image, it can be expected that the similarities between the images are also high. On the other hand, in images that are not very similar to the key image, the similarities between the retrieved images are not necessarily high, so rather than ranking based on the difference in similarity with the key image,
It is more effective to pay attention to the similarity between the retrieved images to improve the viewability of the images. Therefore, by combining the display in the order of similarity and the classification display for each cluster, it is possible to improve the browseability of both the image having a very high degree of similarity to the key image and the image having a not so high degree of similarity with the key image.

[0014]

[0015]

[0016]

[0017]

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) FIG. 1 shows the functional arrangement of an image browsing system according to the first embodiment of the present invention. This image browsing system is for browsing while searching for digital images from a database, and an image similar to the key image input as a search key is searched from the database and displayed on the screen.

In this image browsing system, for example, an image which is wholly similar to the key image or an image having a part which is partially similar to the key image is searched from among the images relating to the registered trademark, and the image is compared with the key image. Used for collation. In order to provide the optimum browsing performance for such an application, the image browsing system of the present embodiment does not only have the similarity with the key image at the time of displaying the search image, but also the similarity between the searched images. In consideration of
A function for collectively presenting similar images is provided. The specific functional configuration of the image browsing system will be described below.

As shown in the figure, the image browsing system of the present embodiment has an image input unit 101, an image feature extraction unit 102, an image feature storage unit 103, an image feature database 104, a similar image search unit 105, and a cluster generation unit 10.
6. The image display unit 107 and other functional modules.

The image input unit 101 is a module used for inputting an image to be registered in the image feature database 104 and a key image used as a search key.
For example, it has a function of taking in a picture on paper as image data using a scanner or taking in an image file created by a computer.

The image feature extraction unit 102 is an image input unit 101.
The image feature is extracted from each image data input via. Where image features (feature extracted from i
mage) indicates a feature amount such as a shape feature and a color feature of each image. As the image feature amount, various parameters such as the amount or ratio of white / black pixels in the entire image or a predetermined divided region in the image, the number of runs, the vertical / horizontal ratio of the outer rectangle, and the number of connected regions can be arbitrarily combined. Can be used. For images to be registered in the image feature database 104, the image features extracted by the image feature extraction unit 102 are recorded in the image feature database 104 together with the input original image data. On the other hand, for the key image used as the search key, the image features extracted by the image feature extraction unit 102 are sent to the similar image search unit 105.

The similar image retrieving unit 105 is for retrieving an image similar to the key image from the image feature database 104. This retrieval is performed by storing the image feature extracted from the key image and the image feature database 104. The comparison is performed by comparing and collating the image characteristics of each of the images being displayed. Each image retrieved from the image feature database 104 by the similar image retrieval unit 105 is sent to the cluster generation unit 106.

The cluster generation unit 106 is for dividing the retrieved images into clusters on the basis of the image features of the respective images, so that images having similar image features are included in the same cluster. Each searched image is classified into a plurality of clusters. A cluster is a group in which similar images are grouped together, and images having similar feature distances in the image feature space are classified into the same cluster.

The image display unit 107 includes a cluster generation unit 10
Based on the search image cluster classification result obtained in step 6, each search image is classified into each cluster and displayed on the monitor. In this case, among the searched images, images belonging to the same cluster, that is, similar images having a high degree of similarity are collectively displayed in order to improve the viewability of the image by the user.

<Construction of Image Feature Database> Next, the procedure of the construction process of the image feature database 104 will be described with reference to the flowchart of FIG. First, the image input unit 101 inputs an image to be registered (step S
(T101), the input image is sent to the image feature extraction unit 102. The image feature extraction unit 102 extracts image features from the input image (step ST102), and the image feature storage unit 103 stores the image features in the image feature database 1
It is additionally recorded in 04 (step ST103). By repeating this operation for all input images (step ST104), the image feature database 104
Is built.

<Similar Image Retrieval / Display Processing> Next, referring to the flowchart of FIG. 3, a processing procedure for similar image retrieval / display will be described. First, the image designated by the user as the key image k is input to the image input unit 101 (step ST201), and the key image k is sent to the image feature extraction unit 102. The image feature extraction unit 102 extracts image features from the key image k (step ST
202). The image feature extracted from this key image k is sent to the similar image search unit 105, and the image feature database 1
The process of searching for a similar image is started from 04 (step ST203).

Here, if the similarity between the key image k and the registered image i in the image feature database 104 is S (k, i), where (S (k, i) ≧ 0), the similar image retrieval unit By 105, an image with S (k, i)> T is searched as a similar image candidate from the image feature database. Here, T is a similarity threshold. When browsing all images, T = 0 may be set.

Then, the cluster generation unit 106 performs clustering so that images having similar image features are included in the same cluster based on the distribution of the image feature amount for each of the retrieved similar image candidates (step ST204). As a method of cluster generation, for example, a method of setting clusters so that the dispersion between clusters is large and the dispersion within clusters is small, and a method of sequentially grouping images with close feature distances are possible. FIG. 4 shows an example of image clusters that are clustered by the cluster generation unit 106.

FIG. 4 shows an example in which an X, Y two-dimensional image feature space is used. The number of dimensions of the image feature space corresponds to the number of parameters used as an index of the feature amount of the image, and in reality, an image feature space of three dimensions or more may be used. Clusters A, B, and C are image clusters divided into clusters by the cluster generation unit 106. Also, a, b, c, d, e, f, g, h are S (k,
i)> Similar image candidates satisfying T, and S (a, k)>
S (b, k)> S (c, k)> S (d, k)> S (e,
k)> S (f, k)> S (g, k)> S (h, k). In this way, the retrieved similar image candidates are classified according to their image features, that is, categories.

Subsequently, the image display unit 107 collectively displays similar image candidates for each cluster using the result of clustering (step ST205). In this category display, the cluster display order is such that the barycentric position of the image feature of the images forming the cluster is the key image k.
For example, a method of displaying clusters in order from the one closest to the image feature may be used. As shown in the flowchart of FIG. 6, first, the images belonging to each cluster are sequentially read from the image feature database 104 (step ST301), and subsequently, the barycentric position of the images forming the cluster is the key. This can be realized by performing a process of displaying the images side by side in order from the cluster close to the image k (step ST302).

Clusters A, B, C in the example shown in FIG.
Assuming that the center of gravity of the cluster is closer to the image feature of the key image k in this order, as shown in FIG.
The search images are displayed in the order of c, g), (b, d, h), (e, f). For cluster A, the images a, c, g contained therein have similar features, and for cluster B, the images b, d, contained therein.
h has similar features to each other in different points from the images a, c, and g. The same applies to cluster C.

As described above, considering not only the similarity with the key image k but also the similarity between the retrieved images,
By presenting similar images together, FIG.
Compared to the conventional method in which images are simply displayed in order of similarity to the key image k as shown in (B), images having similar image characteristics are displayed adjacent to each other in the present embodiment,
It has an advantage that the image can be easily browsed.

That is, the user can collectively observe a plurality of images having a partial similarity with the key image k, and can easily compare and verify with the key image k by visual observation. Therefore, it is possible to perform the image browsing that is most suitable for the investigation of the registered trademark. Of course, not only the investigation of registered trademarks but also the accumulation of images such as micrographs in the database enables effective use for the purpose of investigation of, for example, microorganisms and viruses.

(Second Embodiment) Next, a second embodiment of the present invention will be described with reference to FIGS. In the image browsing system of the second embodiment, the key image k
When presenting an image similar to the key image k, the priority of the similarity order display or the category display is switched according to the value of the similarity with the key image k. The image search and cluster generation processing are the same as in the first embodiment.

That is, in the second embodiment, when displaying an image on the image display unit 107, for images having a very high degree of similarity to the key image k, “similarity order display” is given priority, and the images are similar to some extent. For images with low frequency, "category display" is given priority. This will be specifically described with reference to FIG.

As shown in FIG. 7, even when the cluster generation unit 106 separates the clusters into A, B, and C, the similarity to the key image k is very high like a and b (S).
For the (k, i)> T2) images, the display in the order of similarity is given priority, and the remaining images (c, d, e,
For f, g, h), the category display has priority. Here, it is assumed that T2 satisfies the relationship of T2> T.

A display example according to this embodiment is shown in FIG.
As shown in the figure, the images a and b having a very high similarity to the key image k are excluded from the display group for each cluster and are independently displayed, and the other images c, d, e, and
f, g, and h are displayed in a state of being classified for each cluster.

Since the images a and b have a very high degree of similarity with the key image k, it can be expected that the degree of similarity between the images a and b is also high. Therefore, even in the display form of FIG.
Images similar to each other are displayed side by side. Further, the images a and b are images that are totally similar to the key image k, and the images c, d, e, f, g, and h are images that are partially similar to the key image k. Can be considered. Therefore, by combining the display in the order of the degree of similarity and the category display for each cluster, it is possible to improve the viewability of both the image that is totally similar to the key image k and the image that is partially similar to the key image k. Is possible.

Next, referring to the flowchart of FIG.
The procedure of the image display process in the second embodiment will be described.

First, in the image display unit 107, for each image retrieved by the similar image retrieval unit 105, S
It is determined whether or not the image satisfies (k, i)> T2 (step ST311). This determination can be performed based on the similarity calculated for each image by the similar image search unit 105 at the time of search, for example. Images that satisfy S (k, i)> T2 are classified as "similarity order display" (step ST312), and images that do not satisfy S (k, i)> T2 are classified as "category display". This is the case (step ST313).

Subsequently, in the image display unit 107, the retrieved images are displayed according to the above-described display classifications ("similarity order display" and "category display") determined for each image (step ST314). ).

As described above, in the second embodiment, images having a high similarity to the key image k can be preferentially displayed, and images having a relatively low similarity are adjacent to each other. , Has an advantage that the image can be easily browsed.

(Third Embodiment) Next, FIGS.
A third embodiment of the present invention will be described with reference to FIG. The image browsing system of the third embodiment is characterized in that, when a plurality of very similar images are searched, only the representative images of them are displayed and the display of the remaining images is omitted. The construction of the image feature database, similar image search, and cluster generation processing are the same as in the first embodiment.

That is, among the images registered in the image feature database 104, there may be a case in which a plurality of identical images or images having almost the same pattern are registered.
Therefore, when there are images with almost the same image characteristics, only the representative image can be displayed to reduce the number of images to be browsed, and more types of images can be displayed on the screen. Becomes This will be specifically described with reference to FIG.

In FIG. 10, the similar image retrieval unit 105
Of the images searched by, the image feature amounts of the image a and the image b are very similar, and the image feature amounts of the image d, the image e, and the image f are also very similar. It is shown. In such a case, one image is selected as a representative image for each of the similar image groups, only the representative image is displayed, and the display of the remaining other images is omitted. A display example in this case is shown in FIG.
Shown in 1.

In FIG. 11, a is displayed as a representative image for a similar image group consisting of a and b, and d is displayed as a representative image for a similar image group consisting of d, e, and f. Has been done. Also,
The image c that does not have a very similar image is displayed independently. A border may be provided around the representative image to indicate that there are a plurality of very similar images.

As a method of selecting the representative image, there can be used a method of selecting, as a representative image, an image having the highest similarity to the key image k from the image groups belonging to the same similar image group. Of course, between images belonging to the same similar image group, the degree of similarity with the key image k often hardly changes, so one image may be randomly selected as a representative image.

FIG. 12 shows a second display example. here,
As shown in FIG. 10, images a, b and g are cluster A, images d, e and h are cluster B, and further image c is cluster C.
Cluster A, image a,
Although b is very similar, g is not very similar to a and b, and similarly for cluster B, images d,
Although e and f are very similar, it is assumed that h is not so similar to d, e and f.

In this case, category display for each of the clusters A, B, and C is performed in the same manner as in the first embodiment, but for the cluster A, not all the images a, b, and g are displayed, Only the image g and the representative image a selected from the images a and b forming the similar image group are displayed, and the b is not displayed. Also for cluster B,
Instead of all the images d, e, f, h, only the image h and the representative image d selected from the images d, e, f forming the similar image group are displayed.

When the cluster generation unit 106 sets a cluster with a very small intra-cluster variance, all the images belonging to the same cluster are very similar to each other. Therefore, in such a case, only one image may be selected and displayed as the representative image for each cluster. FIG. 13 shows an example of the processing procedure for selecting and displaying the representative image for each cluster in this way.

First, the image display unit 107 determines, for each cluster generated by the cluster generation unit 106, an image having a highest similarity to the key image k among the images belonging to the cluster, and sets it as a representative image. It is selected (step ST321). For example, when the clusters are divided as shown in FIG.
However, the image d for the cluster B and the image c for the cluster C are selected as the representative images.

Next, the image display unit 107 displays the representative images selected for each cluster side by side (step ST).
322). As a result, the search image is displayed in the same display form as in FIG.

Next, with reference to the flow chart of FIG. 14, a processing procedure for browsing an image whose display is omitted from the representative image will be described.

When a specific representative image is selected by the user's mouse click operation or the like on the display screen of FIG. 11 or FIG. 12 (YES in step ST331), the image display unit 107 displays the selected representative image. Another image in the cluster to which it belongs is read from the image feature database 104 (step ST332), and the read image is
The screen is displayed so as to be adjacent to the selected representative image (step ST333). A display example in this case is shown in FIG.

FIG. 15 shows an example in which the representative image d of the cluster B is selected. In this case, among the other images belonging to the cluster B, the images very similar to the representative image d, that is, the remaining images e and f not selected as the representative image
(Or images e, f, h) are image feature database 1
It is read from 04 and displayed.

As described above, in the third embodiment, when there are a plurality of images that are very similar to each other, only the representative image is displayed and the display of the remaining similar images is omitted. It has the advantage that the number can be reduced and more types of images can be displayed on the screen.

The features of the image browsing system of the present invention have been described above by dividing them into the first to third embodiments, but a system having all the display functions of each of the first to third embodiments has been realized and is required. The display form may be selected accordingly. For example, normally when the display form of the first embodiment is used and the number of retrieved images is very large, that is, 1
When the number of images per cluster is very large, the viewability can be further improved by switching to the display form of the third embodiment.

Further, any of the image browsing systems of the first to third embodiments can be realized as software executable on a computer. In this case, it is possible to easily obtain the same effect as that of the present embodiment only by introducing a computer program including the procedure into a normal computer through a computer-readable recording medium.

[0060]

As described above, the present invention has the following effects.

1) Compared with the conventional method in which the images are simply displayed in the order of similarity to the key image, the similar images are adjacent to each other, so that the images can be easily browsed.

2) Images having a high similarity to the key image can be preferentially displayed, and images having a relatively low similarity are adjacent to each other, so that the images can be easily browsed. .

3) When there are a plurality of very similar images, by displaying only the representative image and omitting the display of the remaining similar images, the number of browsed images can be reduced,
More types of images can be displayed on the screen.

[Brief description of drawings]

FIG. 1 is a block diagram showing a functional configuration of an image browsing system according to an embodiment of the present invention.

FIG. 2 is an exemplary flowchart showing a procedure of image feature database construction processing in the image browsing system of the embodiment.

FIG. 3 is an exemplary flowchart showing a procedure of similar image search / display processing in the image browsing system of the embodiment.

FIG. 4 is an exemplary view showing an example of a result of clustering performed by the image browsing system of the embodiment.

FIG. 5 is an exemplary view showing an example of a search result display screen in the image browsing system of the embodiment.

FIG. 6 is an exemplary flowchart showing an example of a search result display processing procedure in the image browsing system of the embodiment.

FIG. 7 is an exemplary view showing another example of a result of clustering performed by the image browsing system of the embodiment.

FIG. 8 is a diagram showing another example of a search result display screen in the image browsing system of the embodiment.

FIG. 9 is a flowchart showing another example of a search result display processing procedure in the image browsing system of the embodiment.

FIG. 10 is a diagram showing still another example of the result of clustering performed by the image browsing system of the embodiment.

FIG. 11 is a diagram showing still another example of the search result display screen in the image browsing system of the embodiment.

FIG. 12 is a diagram showing yet another example of the search result display screen in the image browsing system of the embodiment.

FIG. 13 is a flowchart showing still another example of the search result display processing procedure in the image browsing system of the embodiment.

FIG. 14 is an exemplary flowchart showing a procedure of a browsing process from a representative image in the embodiment.

FIG. 15 is a view showing still another example of the search result display screen in the image browsing system of the embodiment.

[Explanation of symbols]

101 ... Image input section 102 ... Image feature extraction unit 103 ... Image feature storage unit 104 ... Image feature database 105 ... Similar image search unit 106 ... Cluster generation unit 107 ... Image display unit

─────────────────────────────────────────────────── --- Continuation of front page (56) Reference JP-A-11-175535 (JP, A) JP-A-9-231238 (JP, A) JP-A-9-101990 (JP, A) JP-A-63- 78228 (JP, A) JP 3-214257 (JP, A) JP 6-76004 (JP, A) JP 2001-155026 (JP, A) JP 7-244668 (JP, A) Sumita Kazuo, Seike Miike, Information Filtering Technology, Toshiba Review, Japan, January 1, 1996, Vol. 51, No. 1, pp. 42-44 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 17/30-17/30 419 G06T 1/00 200 JISST file (JOIS)

Claims (3)

(57) [Claims] 1. An image retrieval system for retrieving and displaying an image stored in a database based on a key image, wherein when a plurality of images are retrieved according to the key image, each of the retrieved plurality of images is retrieved. Based on the extracted image features, a cluster classification unit that divides the retrieved plurality of images into clusters so that similar images are included in the same cluster; and the key image for each of the retrieved images Where the similarity is
Similarity determination means for determining whether the value is greater than a fixed value, the result of clustering by the cluster classification means, and
The key image based on the determination result of the similarity determination means.
For images whose similarity to
The results are displayed in order of similarity and displayed on the search result display screen.
-For images whose similarity to the image is less than a specified value, the class
It will be displayed on the search result display screen classified according to data
As described above, the plurality of searched images are displayed on the search result display screen.
An image retrieval system comprising: a display unit for displaying on a surface . 2. An image stored in a database is a key image.
An image search method of searching and displaying based on images, wherein when a plurality of images are searched according to the key image,
Image features extracted from each of the retrieved images
Based on this, similar images should be included in the same cluster.
A cluster that divides the retrieved images into clusters.
The star classification step and the degree of similarity with the key image are determined for each of the searched images.
Similarity determination step to determine whether it is greater than a fixed value
And the result of clustering by the cluster classification step
Based on the determination result of the similarity determination step,
For images whose similarity to the key image is greater than a predetermined value
Are displayed on the search result display screen in the order of their similarity.
For images whose similarity to the key image is less than a specified value,
Are sorted by cluster and displayed on the search result display screen.
As shown, the searched multiple images are searched
And a display step for displaying on the result display screen.
An image retrieval method characterized by and. 3. An image stored in the database is a key image.
Computer pro for searching and displaying based on images
Computer-readable recording medium in which gram is recorded
A computer program , the computer program detects when a plurality of images are retrieved according to the key image.
Image features extracted from each of the retrieved images
Based on this, similar images should be included in the same cluster.
A cluster that divides the retrieved images into clusters.
The star classification step and the degree of similarity with the key image are determined for each of the searched images.
Similarity determination step to determine whether it is greater than a fixed value
And the result of clustering by the cluster classification step
Based on the determination result of the similarity determination step,
For images whose similarity to the key image is greater than a predetermined value
Are displayed on the search result display screen in the order of their similarity.
For images whose similarity to the key image is less than a specified value,
Are sorted by cluster and displayed on the search result display screen.
As shown, the searched multiple images are searched
And a display step for displaying on the result display screen.
And a recording medium characterized by: