JP2003256427A - Image retrieval device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To dispense with the input of the data excluding the image data, such as a key code used in retrieval when an image is accumulated and retrieved, and to reduce the load in retrieval processing. <P>SOLUTION: When a plurality of images 10 are accumulated, a specific characteristic vector for classification is extracted from the image data of the images 10 by a characteristic vector extracting part for classification 12. The images 10 are classified into a plurality of classification items by a classifying part 14 on the basis of the vector. The relativity of the classification items is determined on the basis of the reference different from that of the classification of the classification items, and the information for linking the classification items of high relativity is applied by a linking part 18. When the image similar to a key image 28 is retrieved, the classification item to which the key image belongs and the classification item linked with the same are regarded as a range of a retrieval object. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to accumulation of image data and retrieval of accumulated image data.

[0002]

2. Description of the Related Art Conventionally, in order to store, manage, and retrieve image data, it has been a mainstream to add code information such as a keyword to the image data when registering the image. This is because the code information is used as a key to search for an image to which the code information is added. However, with this method, when registering the image data to be searched,
There was a problem that a lot of work was required because code information had to be input together from a keyboard.

Therefore, a method has been proposed in which various feature quantities are extracted from image data to form a feature vector, which is used for retrieval as information in place of code information. At this time, in order to capture images from various aspects, it is necessary to extract many types of feature quantities from one image data. The feature vector is a vector having each of the feature quantities as a spatial axis, and the fact that the vector has many feature quantities (spatial axes) means that the processing time increases. . That is, when the image data is searched, the search time becomes long. In order to reduce the processing time, in many cases, the feature vector of the accumulated image data is subjected to principal component analysis to ensure the orthogonality of the feature vector and at the same time eliminate the spatial axis with a small variance. Therefore, the number of spatial axes of the feature vector is suppressed.

Further, in Japanese Unexamined Patent Publication No. 2000-112943, the number of feature vectors at the time of collating image data with each other is narrowed down by narrowing down the feature vectors to be searched by the distance along one predetermined dimension axis. Hold down. Further, in Japanese Unexamined Patent Application Publication No. 2001-134573, the number of feature vectors at the time of matching is suppressed by dividing the feature vector space by a predetermined distance and describing each region hierarchically.

[0005]

However, when attempting to reduce the number of spatial axes of the feature vector by the principal component analysis, if the number of spatial axes is very large or the accumulated feature vector has a large variance. , In some cases, the number of spatial axes could not be reduced sufficiently.

Further, in the method proposed by the former publication, since only the distance of a predetermined one-dimensional axis is focused when narrowing down the feature vector of the object, the similarity by other feature vectors is different. There is a possibility that the case where it is determined to be high is excluded. That is, there is a problem that image data that should be determined to be originally similar is overlooked.

Similarly, in the latter publication, the case where it is determined that the similarity is high due to the feature vector other than the feature vector used for the above-mentioned delimiter is excluded, and the image data that is originally similar is missed. There was a problem.

The present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide an image search apparatus which can reduce the load of search processing and reduce omission of search.

[0009]

In order to solve the above-mentioned problems, an image searching apparatus according to the present invention is an image searching apparatus for searching a plurality of accumulated images for an image similar to a key image. Then, the accumulated image is classified into a plurality of classification items based on at least one feature vector extracted from the image data of the image, and the classification item is different from the criterion for classifying the classification items. It has a link means for associating those similar in reference. Then, a feature vector is extracted from the image data of the key image, the key image feature vector is collated with a feature vector representing the category item, and a category item similar to the key image is selected. means,
The extracted feature vector of the key image is compared with the feature vector of the accumulated image belonging to the selected classification item or the classification item associated with the selected classification item, and the accumulated image similar to the key image is accumulated. A similar image selecting means for selecting a selected image.

The accumulated images to be searched are classified into a plurality of items based on at least one feature vector, and similar items are associated with each other by a criterion different from the criterion of this classification. Therefore, not only the classification items selected based on the similarity with the feature vector of the key image but also other linked classification items as search targets, narrowing down the search target range and searching too narrowly. Prevents leakage.

Further, the similar image selection means further extracts a plurality of candidates having a relatively high similarity based on the feature vector within the range of the classification item, and further narrows down the extracted candidates. It is possible to have a narrowing down selection means for selecting images with high similarity.

Further, the key image includes an original key image and a converted key image obtained by a predetermined conversion of the original key image, and a search is executed based on these plural key images. it can. This transformation can be an affine transformation in which translation is added to a linear transformation such as enlargement, reduction, rotation, and inversion.

According to another aspect of the present invention, an image search database is provided. This database is
The accumulated image is classified into a plurality of classification items based on at least one feature vector extracted from the image data of the image, and a classification similar to a criterion different from the criterion for classifying the classification item. Items are associated with each other.

[0014]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention (hereinafter referred to as embodiments) will be described below with reference to the drawings. Figure 1
It is a figure which shows schematic structure of the network to which the image search apparatus of this embodiment is applied. The network includes a client 1, which is a computer that functions as an image search apparatus according to the present embodiment, a scanner 2 that captures an image on a print medium such as paper, a database server 3 that provides a database, and a network. A web server 4 for management is connected. The database server 3 stores image data to be searched by the client 1 as an image search device and data associated with the image data. A router 6 for connecting a network to an external terminal 5 such as a notebook computer or a personal digital assistant is also provided. In addition to the above-mentioned scanner, the image data can be sent from the external terminal 5
From the external storage medium drive of the client 1.

FIG. 2 is a functional configuration diagram of the present embodiment, and particularly shows a configuration related to formation of a database of images to be searched. From the image 10 accumulated in the database or accumulated, the feature vector extracting unit 12 for classification extracts a feature vector having a value that can be quantitatively extracted from the image data of the image 10. This feature vector indicates a feature for classifying accumulated images, which will be described later. The feature vector for classification is, for example, limited colorization of image data based on a predetermined standard,
It can be a vector indicating the proportion of each color. Here, the limited colorization is a process in which the color of each pixel is blue, for example, all the colors similar to blue, such as navy blue to light blue, and the colors are aggregated into a limited number of colors such as red and yellow. is there. In the classification unit 14, based on the distance between the classification feature vectors,
The image 10 is classified into a plurality of classification items. Therefore,
The classification feature vector extraction unit 12 and the classification unit 14 function as a classification unit 16 that classifies a plurality of classification items based on the feature vector extracted from the accumulated image data.

Further, the linking unit 18 judges the relevance between the classified items by a predetermined standard,
It is a means for linking items that are judged to be highly related. This relevance can be evaluated, for example, by the impression of the limited color of the image data subjected to the limited color. There are groups of colors that give a particular impression to a person, such as warm impression, cold impression, soft impression, and hard impression. The limited colors are divided into several groups for each impression, and the classification items are associated with each other based on what impression the limited colorized image data gives a person.

On the other hand, from the image data of the image 10, the search feature vector extraction unit 20 extracts the feature vector used in the search. This search feature vector is a vector composed of values that can be quantitatively extracted from the image data. For example, the image data is divided into n × n meshes, and the average of pixel values and the amount of edges in each mesh are calculated. Is the feature vector data.

The search feature vector extraction unit 20 further includes two feature vector extraction units having different resolutions. The low-resolution feature vector extraction unit 22 divides the image data into coarse (for example, 4 × 4) meshes, and uses the average of pixel values and the amount of edges in each mesh as feature vector data. The other high-resolution feature vector extraction unit 24 divides the mesh into smaller (for example, 16 × 16) meshes, and uses the average of pixel values and the amount of edges in each mesh as vector data.

To each image 10, a classification item to which the image belongs, information relating to association, and two types of feature vectors extracted for retrieval are added, and the image data is stored in the database 26.

FIG. 3 is a functional block diagram of the present embodiment, and particularly shows a configuration for searching for an image similar to one image (key image). In this specification, when “similar” is mentioned, “match” unless otherwise specified.
Shall be included.

An image (hereinafter referred to as a key image) 28 which is a key for retrieving a predetermined image from the database 26 is input. The classification feature vector extraction unit 30 extracts a feature vector from the image data of the key image 28 for matching with the above-described classification item. The classification feature vector extraction unit 30 has the same function as the classification feature vector extraction unit 12 described above, and extracts the feature vector based on the same criterion. In the classification item collation unit 32, based on the extracted feature vector, the classification item having the most similar key image 28 is collated by searching the image data accumulated in the database 26. Further, the linking classification item calling unit 34 calls the classification item linked by the linking unit 18 with respect to the classification item most similar to the key image 28. Then, the search candidate extraction unit 36 extracts the most similar classification item and the feature vector group of image data belonging to the classification item linked to the classification item as candidates to be used in the subsequent searches.

A predetermined feature vector is extracted from the key image 28 by the search feature vector extraction unit 38. Here, the search feature vector extraction unit 38 has the same function as the search feature vector extraction unit 20 described above, and extracts the feature vector based on the same criterion. The search feature vector extraction unit 38 further includes two feature vector extraction units having different resolutions. The low-resolution feature vector extraction unit 40 divides the image data into coarse (for example, 4 × 4) meshes, and uses the average of pixel values and the amount of edges in each mesh as feature vector data. The other high-resolution feature vector extraction unit 4
2 is divided into finer mesh portions (for example, 16 × 16), and the average of pixel values in each mesh and the amount of edges are used as vector data.

Next, in the low resolution feature vector matching unit 44, each vector of the feature vector group extracted by the search candidate extraction unit 36 and the low resolution feature vector extraction unit 40.
Compare with the feature vector of the key image extracted in
Find the similarity of each. Then, the candidate narrowing unit 46 narrows the search range based on this similarity. That is, a feature vector having a high degree of similarity is extracted and used as the range of the next search. In the high resolution feature vector matching unit 48, each vector of the above-mentioned narrowed down feature vector group and the high resolution feature vector extraction unit 42.
Compare with the feature vector of the key image extracted in
Find the similarity of each. The search result output unit 50 outputs the image data corresponding to the feature vector having a high similarity based on the similarity.

FIG. 4 is a flowchart relating to the storage of image data. First, the image data to be registered is input (S100). From the input image data, three types of feature vectors, a classification feature vector, a low resolution feature vector, and a high resolution feature vector are extracted (S102, S10).
4, S106). When all the image data to be accumulated are input and the extraction of their feature vectors is completed, the process proceeds to the next step (S108).

Next, using the extracted classification feature vector, the distance between the feature vectors is obtained (S110), and the feature vector is classified based on this distance (S11).
2). Here, the classification is based on the distance between the feature vectors,
The things that are close to each other are put together. A method generally called cluster analysis is used. The cluster analysis includes a hierarchical method and a non-hierarchical method, but either method can be used. Each classified group is a classification item. If the classification of all feature vectors is completed (S11
4) Next, the classification items are linked (S11).
6). This is to associate similar items with each other by a criterion different from the criterion used for classification. When linking is completed for all the classification items (S118), each image data and the accompanying three types of feature vectors for classification, low resolution and high resolution, classification items, and linking information are stored in the database. To register (S12
0).

FIG. 5 is a flow chart relating to the retrieval of image data. First, image data to be a search key is input (S122), and a classification feature vector used for collation of classifications, a low resolution feature vector used for narrowing down search candidates, and a high resolution feature vector used for obtaining a search result are input. , Is extracted from the image data for the search key (S124, S126, S128). At this time, the search key image data may be subjected to affine transformation, which is a transformation including a primary transformation and a parallel movement, and the image data after this transformation may be searched as a key image. For example, 216 key images can be obtained from the original image data and the image data by the affine transformation of 3 patterns for enlargement / reduction, 4 patterns for 90 ° rotation, 2 patterns for inversion, and 9 patterns for parallel movement.

Next, based on the classification feature vector of the key image, the classification item already registered in the database is collated to obtain the classification item to which the key image belongs (S13).
0). Furthermore, the classification item linked to this classification item is called (S132). The image data belonging to these classification items becomes the subsequent search range, and the low resolution feature vectors of the images belonging to these classification items are extracted from the database 26 (S134). The extracted low resolution feature vector is collated with the low resolution feature vector of the key image to determine the degree of similarity (S136). The search target is narrowed down to the image of the feature vector having a high degree of similarity, and the high resolution feature vector of the narrowed down image is extracted from the database 26 (S138). The extracted high resolution feature vector is collated with the feature vector of the key image to select a feature vector having a high degree of similarity (S14).
0). The image corresponding to this feature vector is called from the database 26 and used as the search result (S14).
2).

As described above, in the present embodiment, the database is linked so that the items having high relevance among the above-mentioned classification items are associated with each other by the criterion different from the criterion when the classification items are divided. Then, when performing a search for a key image, the classification item to which the key image belongs and the images belonging to the classification item linked to this classification item are set as the search target range. Specifically, the classification item A to which the key image belongs and all the classification items B, C ... Linked to this classification item A are set as the search range. When only the classification item A is set as the search target, even if an image similar to the key image exists in the classification item B or the like, it cannot be searched, but in the present embodiment, this is possible. In addition, if the classification items are made more roughly so that the number of omissions in the search is reduced, or if the classification items are not provided and the entire data is targeted for detailed search from the beginning, it is dragged by minute differences and " Although similar images may be overlooked or the processing load may increase, in the present embodiment, it is possible to perform a search close to a person's impression and suppress an excessive increase in the processing load.

Further, since the search is performed using the feature vector extracted from the image data of the registered image and the key image, it is not necessary to input code information such as a keyword when registering and storing the image. .

[Brief description of drawings]

FIG. 1 is a diagram illustrating an example of a network environment.

FIG. 2 is a block diagram showing functions of the present embodiment,
It is a figure which shows the structure especially regarding registration of image data.

FIG. 3 is a block diagram showing functions of the present embodiment,
It is a figure which shows the structure especially regarding the search of image data.

FIG. 4 is a flowchart showing a process of the present embodiment, and particularly a chart showing a process relating to registration of image data.

FIG. 5 is a flowchart showing a process of the present embodiment, and is a chart showing a process particularly related to image data search.

[Explanation of symbols]

10 images, 12, 30 classification feature vector extraction unit,
14 classification unit, 18 linking unit, 20, 38 search feature vector extraction unit, 22, 40 low resolution feature vector extraction unit, 24, 42 high resolution feature vector extraction unit,
26 database, 28 key image, 32 collation part with classification item, 34 linking classification item calling part, 36
Search candidate extraction unit, 44 low resolution feature vector matching unit, 46 candidate narrowing unit, 48 high resolution feature vector matching unit, 50 search result output unit.

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 5B050 AA02 AA06 AA09 CA07 CA08                       DA02 EA04 EA18 FA02 FA13                 5B075 ND06 NK06 NK31 NK39 NK44                       NR12 PR06 QM08                 5L096 AA06 FA06 FA32 GA19 HA07                       JA11

Claims (6)

[Claims] 1. An image retrieval device for retrieving an image similar to a key image from among a plurality of accumulated images, wherein at least one characteristic of the accumulated image extracted from image data of the image. Classification means for classifying into a plurality of classification items based on a vector, link means for associating the classification items with each other that are similar to each other in a criterion different from the criterion for classifying them, and a feature vector Extract
This key image feature vector is compared with a feature vector representative of the classification item to select a classification item similar to the key image, classification item selection means, the extracted feature vector of the key image, Similar image selecting means for comparing the selected classification item or the feature vector of the accumulated image belonging to the classification item associated therewith to select the accumulated image similar to the key image, An image retrieval device having. 2. The image search device according to claim 1, wherein the similar image selecting unit selects the feature vector extracted from the key image, the selected classification item or the classification item associated with the selected classification item. Used by the candidate extraction means, which compares the feature vector of the stored image to which it belongs and extracts the plurality of stored images similar to the key image as candidates for similar images; , Comparing the key image and the extracted candidate image by a feature vector different from the feature vector of the key image and the stored image,
An image search device comprising: a narrowing-down selection unit that selects the stored image similar to the key image. 3. The image search device according to claim 1, wherein the key image includes an original key image and a conversion key image obtained by performing a predetermined conversion from the original key image. Search device. 4. The image search device according to claim 3, wherein the predetermined conversion performed on the original key image is an affine transformation. 5. An image search database in which a plurality of images are accumulated, wherein the accumulated images are classified into a plurality of classification items based on at least one feature vector extracted from the image data of the image. Furthermore, the image search database in which the classification items are associated with each other similar to each other in a criterion different from the criterion by which the classification items are classified. 6. An image search apparatus for searching an image similar to a key image among the images stored in the image search database according to claim 5, wherein a feature vector is extracted from the image data of the key image. Then
Classification item selection means for matching the key image feature vector with a feature vector representative of the classification item to select a category item similar to the key image, the extracted feature vector of the key image, and the selection A similar image selection means for comparing the stored classification image or the feature vector of the stored image belonging to the classification item associated with the selected classification item, and selecting the stored image similar to the key image. An image retrieval device having.