JP2000285141A - Image retrieval device, image classifying device, and computer readable recording medium recorded with program for functioning computer as these devices - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval device capable of exactly expressing the intention of a user as a retrieval condition. SOLUTION: A U/I part 151 designates a plurality of inquiry images and designates the kind of a feature amount to be used for finding similarity to the image of a retrieval object for each of respective designated inquiry images, a matching engine 153 finds the degree of similarity to each inquiry image while using the feature amount of the kind designated by the U/I part 151 for each image of the retrieval object and finds the total degree of similarity to the image of the retrieval object while totaling the found degrees of similarity to each inquiry image, and the U/I part 151 outputs the retrieved result on the basis of the total degree of similarity found for each image of the retrieval object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieving apparatus for retrieving an image desired by a user from a large number of images and an image classifying apparatus for classifying a large number of images into a predetermined classification destination. The present invention relates to an image retrieval apparatus and an image classification apparatus capable of performing highly accurate image retrieval processing and image classification processing by the method described above.

[0002]

2. Description of the Related Art In recent years, with the development of computer technology and image processing technology, many attempts have been made to build up an image database by storing a large amount of electronic images. What is important when constructing an image database is an image search device that enables an image to be used to be easily obtained from a large amount of stored images.

[0003] As one of the image retrieval devices, a search key using words is given to each image, and in the same manner as in document retrieval, the entered keyword is collated with the search key attached to the image. In some cases, an image to which a search key matching a keyword is attached is returned as a search result.

However, since the impression that a person receives from an image varies from person to person, the search key attached to the image cannot be said to express all the impressions that a person receives from the image. Therefore, a search key using words is used. In many cases, a satisfactory image cannot be obtained by image search. Because
This is because, in many cases, the shape features and positional relationships of objects (hereinafter, referred to as “objects”) such as humans, animals, and buildings appearing in images cannot be expressed in appropriate words.

Therefore, a specific image (hereinafter, referred to as a “query image”) is given as a search condition instead of searching for an image by collating a keyword with a search key of words, and an image similar to the query image is searched. Is desirably searchable from an image database. In order to realize such a search device, a feature amount representing a feature of each image is extracted from the query image and the search target image, and the similarity between the feature amount of the query image and the feature amount of the search target image is determined. A method has been proposed in which an image having a feature amount similar to the feature amount of a query image is returned as a search result. In other words, this image search device searches for similar images by considering the similarity between features as the similarity of the images themselves.

Further, instead of using the image itself as the query image, the image color (histogram of color), texture, the shape of the object, etc. are extracted and stored in a database as the feature amount of each image, and the image is searched. At this time, a search device that searches for an image having a specified feature amount by designating the feature amount of an image in a database as a search condition has also been proposed (Myron Flickner et al.
al, “Query by Imageand Video Content: The QBIC Sy
stem, ”Computer, September 1995, pp23-32).

[0007]

However, in an image search apparatus that searches for an image similar to the query image by specifying the query image, for example, a query image similar to the image desired by the user as a search result exists. If not, a process of searching for an image to be designated as a query image is required, and there is a problem that the search operation is not convenient.

Further, even in an image search apparatus for searching for an image by designating the feature amount of the image, for example, the user selects a color scheme of the image desired as a search result or exists in the image desired as a search result. Since it is necessary to specify the feature amount of an image as a search condition by drawing the shape of an object, there is a problem that a user's burden in performing a search is large. In addition, since the feature amount of an image is specified as a search condition, it is difficult for a user to accurately express a desired image as a search result in the search condition, and the user's intention for the search cannot be properly transmitted to the device. There was a point. As a result, it has been difficult to obtain a search result desired by the user.

[0009] That is, the above-described conventional image retrieval apparatus is inconvenient in that there is no degree of freedom in specifying the retrieval conditions, and the user's intention cannot be accurately expressed as the retrieval conditions. Therefore, the user's intention cannot be accurately communicated to the device side, and it has been difficult to obtain an image desired by the user with high accuracy. Furthermore, in order to accurately convey the user's intention to the device side and perform the search processing as intended by the user, various search methods are prepared so that the search processing can be performed in a manner desired by the user. There must be.

The present invention has been made in view of the above, and an object of the present invention is to provide an image search device capable of executing an image search process by a new method.

Further, the present invention has been made in view of the above, and provides an image search apparatus capable of accurately expressing a user's intention as a search condition and capable of performing a highly accurate image search process. The purpose is to do.

Further, the present invention has been made in view of the above, and is capable of executing an image search process of a new method capable of reflecting a user's search intention in search results, thereby achieving high-accuracy image search. It is an object to provide an image search device capable of performing a search process.

It is another object of the present invention to provide an image classification apparatus which can apply the image retrieval processing in the image retrieval apparatus as described above to the image classification processing and can execute the image classification processing by a new method. .

Further, the present invention applies the image search processing in the image search apparatus as described above to the image classification processing, so that the intention of the user can be accurately represented as a classification condition serving as a reference for the image classification. It is an object of the present invention to provide an image classification device capable of performing high image classification processing.

Further, the present invention applies the image search processing in the image search apparatus as described above to the image classification processing, and can execute the image classification processing of a new method that can reflect the user's search intention in the classification result. Accordingly, an object of the present invention is to provide an image classification device capable of performing highly accurate image classification processing.

[0016]

In order to achieve the above object, an image retrieval apparatus according to a first aspect of the present invention extracts and extracts at least one type of feature quantity representing a feature of an image from an inquiry image and a plurality of images to be searched. An image search apparatus that searches for the same or similar image as the query image from the plurality of search target images by calculating the similarity between the query image and the search target image using the at least one type of feature amount obtained above. A designating unit that designates one query image for each type of feature amount used when calculating the similarity between the query image and the search target image, and a designating unit that specifies one query image for each of the search target images. The degree of similarity with each of the specified query images is obtained using the feature amount of the corresponding type, and the obtained degree of similarity with each of the query images is summed to obtain the total of the images to be searched. Search means for obtaining the similarity, based on the overall similarity obtained for each image of the search target in the search unit, in which and an output means for outputting a search result.

Further, the image retrieval apparatus according to claim 2 extracts at least one kind of feature quantity representing a feature of the image from the query image and the plurality of search target images, and uses the extracted at least one kind of feature quantity. In an image search apparatus that searches for the same or similar image as the query image from the plurality of search target images by calculating the similarity between the query image and the search target image, a plurality of the query images are specified and specified. Specifying means for specifying a type of a feature amount to be used when obtaining a similarity with the image to be searched for each query image, and a feature of the type specified by the specifying means for each image to be searched. Search means for calculating the similarity with each query image using the amount, and summing the calculated similarities with each query image to obtain the overall similarity of the search target image, Based on the overall similarity obtained for each image of the search target in the serial search means, in which and an output means for outputting a search result.

The image retrieval apparatus according to a third aspect of the present invention extracts at least one kind of feature quantity representing the feature of the image from the query image and the plurality of search target images, and uses the extracted at least one kind of feature quantity. An image search device that searches for the same or similar image as the query image from the plurality of search target images by calculating the similarity between the query image and the search target image; Calculating, for each of the images to be searched, the degree of similarity with each of the query images specified by the specifying means, and summing up the degrees of similarity with each of the obtained query images to obtain the overall similarity of the images to be searched. A search unit to be obtained; and an output unit to output a search result based on the overall similarity obtained for each image to be searched by the search unit.

The image retrieval apparatus according to a fourth aspect of the present invention extracts at least one type of feature quantity representing the feature of the image from the query image and the plurality of images to be searched, and uses the extracted at least one type of feature quantity. An image search device that searches for the same or similar image as the query image from the plurality of search target images by calculating the similarity between the query image and the search target image; For each of the images to be searched, determine the similarity with each of the query images specified by the specifying unit, and select the similarity indicating the highest similarity from the obtained similarities with the query images. Means for obtaining the overall similarity of the image to be searched by the search means, and an output means for outputting a search result based on the overall similarity obtained for each image to be searched by the search means. When, those having a.

According to a fifth aspect of the present invention, in the image search apparatus according to the second, third, or fourth aspect, the designation means determines the type of the query image with respect to the similarity determined by the search means. It is possible to specify an importance level for each of the query images so that any weight can be given according to the query image, and the search unit determines, for each of the search target images, a similarity to each of the query images. The degree of similarity is obtained, and the obtained degree of similarity with each query image is weighted according to the degree of importance specified by the specifying means, and then the overall degree of similarity is obtained.

According to a sixth aspect of the present invention, in the image search apparatus according to the third or fourth aspect, the specifying means uses a plurality of types of feature amounts to determine the similarity between the query image and the image to be searched. When determining the degree, the importance of each type of feature is determined by each query so that the similarity determined by the search means can be arbitrarily weighted in accordance with the type of the query image and the feature. It is possible to specify for each image, the search means obtains the similarity with each of the query images for each type of feature amount for each of the search target images, After weighting the similarity with each query image obtained in (1) according to the degree of importance specified by the specifying means, the overall similarity is obtained.

According to a seventh aspect of the present invention, in the image search apparatus according to the third or fourth aspect, further, the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts. In such a case, the similarity between the query images specified by the specifying unit is determined for each type so that the similarity obtained by the search unit can be arbitrarily weighted in accordance with the type of the feature amount. Calculated for each feature,
An importance determining unit that determines the importance of each type of feature amount between the query images based on the obtained similarity degree of each type of feature amount; For each of the types of feature amounts, a similarity with each query image is obtained, and the similarity with each query image obtained for each type of feature amount is determined by the importance determining unit. After performing weighting according to the importance for each feature amount, the overall similarity is obtained.

Further, the image retrieval apparatus according to claim 8 extracts at least one kind of feature quantity representing the feature of the image from the query image and the plurality of images to be searched, and uses the extracted at least one kind of feature quantity. In an image search apparatus that searches for the same or similar image as the query image from the plurality of search images by determining the similarity between the query image and the search target image, at least one of the query images is specified. Specifying means for specifying at least one roughness level of a feature to be used for search, based on an extraction criterion in which the level of roughness of a feature to be extracted from an image is defined stepwise from coarse to fine. A feature corresponding to at least a roughness level specified by the specifying unit from the query image and the plurality of images to be searched based on the extraction criterion. Extracting means at a predetermined timing; and searching means for obtaining, for each image to be searched, a similarity with the query image using a feature amount corresponding to a roughness level specified by the specifying means. And output means for outputting a search result based on the similarity obtained for each image to be searched by the search means.

According to a ninth aspect of the present invention, in the image retrieval apparatus according to the eighth aspect, the extraction criterion includes a step-by-step process in which the number of divisions indicating at least how many images are divided is set as the roughness level. When the feature extraction unit extracts the feature amount according to any one of the stepwise defined roughness levels, the feature extraction unit divides the image based on the corresponding division number. A plurality of divided images, and extracting the feature amount from each of the generated divided images, thereby extracting a feature amount group including feature amounts in divided image units from the query image and the search target image, When the search unit obtains the similarity between the query image and the search target image using the feature amount group, the similarity of each of the divided images corresponding to the query image and the search target image is a feature amount. By finding respectively, using,
The similarity between the query image and the image to be searched is obtained.

According to a tenth aspect of the present invention, in the image search apparatus of the ninth aspect, the extraction criterion is
As the level indicating the coarsest feature amount among the stepwise defined roughness levels, without dividing the image, includes a level at which the entire image is to be extracted as a feature amount, and the feature extraction means includes: When extracting a feature amount in accordance with a level at which the entire image is to be extracted, the feature amount is extracted from the query image and the image to be searched for the entire image.

[0027] According to an eleventh aspect of the present invention, in the image search apparatus according to any one of the eighth, ninth and tenth aspects, the extraction criterion is such that an object existing in the image is extracted as a feature amount. In the case of a target, the definition of the roughness level of the feature quantity extracted from the object is included.

According to a twelfth aspect of the present invention, in the image search apparatus according to any one of the eighth to eleventh aspects, the feature extracting means determines that the plurality of images to be searched are determined by the extraction criterion. The feature amounts corresponding to all the roughness levels defined in are extracted in advance, and among the query images designated by the designation unit, for the query image from which the feature amounts have not been extracted, the search unit Before executing the processing for obtaining the degree of similarity, a feature amount corresponding to the roughness level specified by the specifying means is extracted.

[0028] The image retrieval apparatus according to a thirteenth aspect extracts at least one type of feature quantity representing an image feature from the query image and the plurality of search target images, and uses the extracted at least one type of feature quantity. In an image search apparatus that searches for the same or similar image as the query image from the plurality of search target images by calculating the similarity between the query image and the search target image, a designation that specifies at least one of the query images Means for obtaining, for each image to be searched, a degree of similarity between a region of a predetermined size in the image to be searched and the query image specified by the specifying means using a predetermined feature amount extracted in advance; Based on the similarity obtained for each of the search target images by the search unit. And output means for outputting, in which with a.

According to a fourteenth aspect of the present invention, in the image search device according to any one of the first to thirteenth aspects, the designating means can designate a search result image as the query image. Things.

According to a fifteenth aspect of the present invention, there is provided a computer-readable recording medium on which a program for causing a computer to function as each means of the image retrieval device according to any one of the first to fourteenth aspects is recorded. is there.

The image classification apparatus according to claim 16 sets a reference image as a classification reference of the image to be classified for each classification item to which the image to be classified is classified, and sets the reference image for each classification item. By using at least one type of feature amount representing the characteristics of the extracted image from the reference image and the image of the classification target, the similarity of the reference image and the image of the classification target for each of the classification items, An image classification device that classifies the image to be classified into any of the classification items based on the obtained similarity, wherein the classification item is set, and the reference image and the classification target are set for each of the set classification items. Setting means for setting one reference image for each type of feature amount used when calculating the similarity between images, and setting each reference image set by the setting means for each classification item. Calculating means for calculating the similarity between the image and the image to be classified using the corresponding type of feature quantity, and calculating the total similarity of the images to be classified by summing up the obtained similarities; and Determining means for determining a classification item to which the classification target image is to be classified, based on the overall similarity of the classification target image obtained for each classification item and a preset classification destination determination criterion. Things.

The image classification apparatus according to claim 17 sets a reference image as a classification standard of the image to be classified for each classification item to which the image to be classified is classified, and sets the reference image for each of the classification items. By using at least one type of feature amount representing the characteristics of the extracted image from the reference image and the image of the classification target, the similarity of the reference image and the image of the classification target for each of the classification items, An image classification device that classifies the image to be classified into one of the classification items based on the obtained similarity, wherein the classification item is set, and a plurality of the reference images are set for each of the set classification items. A setting unit for setting a type of a feature amount used when obtaining a similarity with the image to be classified for each of the set reference images; and a type set by the setting unit for each of the classification items. Calculating means for calculating the similarity between each reference image and the image to be classified by using the feature amounts of the above, and calculating the total similarity of the images to be classified by summing up the obtained similarities; Determining means for determining a classification item to which the classification target image is to be classified, based on the overall similarity of the classification target image obtained for each item and a preset classification destination determination criterion. It is.

An image classification apparatus according to claim 18 sets a reference image as a classification standard of the image to be classified for each classification item to which the image to be classified is classified, and sets the reference image for each of the classification items. By using at least one type of feature amount representing the characteristics of the extracted image from the reference image and the image of the classification target, the similarity of the reference image and the image of the classification target for each of the classification items, An image classification device that classifies the image to be classified into one of the classification items based on the obtained similarity, wherein the classification item is set, and a plurality of the reference images are set for each of the set classification items. Setting means for obtaining, for each of the classification items, a similarity between each of the reference images set by the setting means and the image to be classified, and summing up the obtained similarities to obtain a total of the images to be classified. Calculating means for calculating the degree of similarity, based on the overall similarity of the images to be classified obtained for each of the classification items by the calculating means and a predetermined classification destination determination criterion, Determining means for determining a classification item to be classified.

The image classification apparatus according to the nineteenth aspect sets a reference image as a classification criterion of the classification target image for each classification item to which the classification target image is classified, and sets the reference image for each of the classification items. By using at least one type of feature amount representing the characteristics of the extracted image from the reference image and the image of the classification target, the similarity of the reference image and the image of the classification target for each of the classification items, An image classification device that classifies the image to be classified into one of the classification items based on the obtained similarity, wherein the classification item is set, and a plurality of the reference images are set for each of the set classification items. Setting means for obtaining, for each of the classification items, a similarity between each of the reference images set by the setting means and the image to be classified, and a similarity indicating the highest similarity among the obtained similarities. Calculating means for calculating the overall similarity of the images to be classified, and calculating the total similarity of the images to be classified obtained for each of the classification items by the arithmetic means and a preset classification destination determination criterion. Determining means for determining a classification item to which the image to be classified is to be classified.

According to a twentieth aspect of the present invention, in the image categorizing apparatus according to the seventeenth, eighteenth, or nineteenth aspect, the setting means determines the type of the reference image with respect to the similarity obtained by the arithmetic means. It is possible to set the importance for each of the reference images so that any weighting can be performed according to the reference image. Is obtained, and the obtained similarity is weighted according to the importance set by the setting means, and then the overall similarity is obtained.

According to a twenty-first aspect of the present invention, in the image classification apparatus according to the eighteenth or nineteenth aspect, the setting means uses a plurality of types of feature amounts to obtain a similarity between the reference image and the image to be classified. When determining the degree, the importance for each type of feature is determined by each reference level so that the similarity determined by the calculation means can be arbitrarily weighted in accordance with the type of the reference image and the feature. It is possible to set the similarity between the reference image and the image to be classified for each type of feature amount for each of the classification items, After weighting the similarity obtained for each quantity according to the importance set by the setting means, the overall similarity is obtained.

According to a twenty-second aspect of the present invention, in the image classification apparatus of the eighteenth or nineteenth aspect, the similarity between the reference image and the image to be classified is further determined using a plurality of types of feature amounts. In this case, for each of the classification items, a similarity calculated by the calculation unit can be arbitrarily weighted in accordance with the type of the feature amount.
The similarity between the reference images set by the setting means is obtained for each type of feature amount, and the similarity between each of the reference images is obtained based on the similarity obtained for each type of feature amount. An importance determining unit for determining an importance, wherein the calculating unit obtains a similarity between the reference image and the image to be classified for each of the types of feature amounts for each of the classification items, and After weighting the similarity obtained for each feature amount according to the importance determined by the importance determining means, the overall similarity is obtained.

According to a twenty-third aspect of the present invention, in the image classification apparatus according to any one of the sixteenth to twenty-second aspects, the setting means further classifies the images classified for each of the classification items. Can be set.

Further, the image classification apparatus according to claim 24 extracts at least one kind of feature amount representing a feature of an image from a plurality of images to be classified, and uses the extracted feature amount to convert each of the images to be classified. An image classification device for classifying similar images into groups, wherein a designating means for designating the type of the feature quantity, and a feature quantity of the designated type when the kind of the feature quantity is designated by the designating means Classifying means for calculating the degree of similarity between the images of the classification target using, and classifying each image of the classification target into a group of similar images based on the obtained degree of similarity between the images of the classification target; Display means for selecting a representative image representing each group from each of the groups classified by the classifying means and displaying the selected image on a screen, wherein the specifying means further comprises: And the type of the characteristic amount can be specified, and when the classifying unit specifies the type of the representative image and the characteristic amount by the specifying unit, the classifying unit specifies the characteristic amount of the specified type. Classifying each image classified into the group to which the representative image belongs into a group of similar images, and the display unit selects a representative image representing each group from each group classified by the classification unit. On the screen.

According to a twenty-fifth aspect of the present invention, there is provided the image classification device according to the twenty-fourth aspect, wherein the display means displays on the screen the types of characteristic amounts that can be designated by the designation means. A representative image of each group classified by the classifying means is displayed on a screen in association with a type of a specified feature amount, and a result of classification of the plurality of images to be classified is displayed on a screen in a tree structure.

According to a twenty-sixth aspect of the present invention, in the image classification device according to any one of the sixteenth to twenty-fifth aspects, an arbitrary image classified for each of the classification items is further used as a query image. Thereby, a search means for searching for the same or similar image as the inquiry image is provided.

According to a twenty-seventh aspect of the present invention, in the image classification device according to the twenty-sixth aspect, the search means is the image search device according to any one of the first to fourteenth aspects. It is.

According to a twenty-eighth aspect of the present invention, there is provided the image classification device according to any one of the sixteenth to twenty-fifth aspects, wherein the image to be classified is searched for by the retrieval means according to the twenty-sixth aspect, An image searched by the image search device according to any one of claims 1 to 14.

A computer-readable recording medium according to a twenty-ninth aspect stores a program for causing a computer to function as each means of the image classification device according to any one of the sixteenth to twenty-eighth aspects. is there.

[0045]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of an image retrieval apparatus and an image classification apparatus according to the present invention will be described below in detail with reference to the accompanying drawings.

[First Embodiment] First, an image retrieval apparatus according to the present invention will be described. FIG. 1 is a block diagram of the image search device according to the first embodiment. The image search apparatus shown in FIG. 1 is composed of a computer 100 such as a personal computer or a workstation, and has a CPU 101 for controlling the whole, a ROM 102 storing a boot program and the like, and a RAM 103 used as a work area of the CPU 101. And a mouse 105 for inputting a query image, a search execution command, etc., and a CR for displaying a search result image, for example.
A monitor 106 such as a T or a liquid crystal display; image search software 107 for causing the CPU 101 (computer 100) to execute the function as the image search device of the present invention; an image DB 108 in which a plurality of images are registered;
An image feature DB in which feature amounts extracted from the image inside are registered
A computer 100 (image search device) is connected to a hard disk 110 storing an image and the like, an interface (I / F) 111 for connecting a scanner or digital camera for inputting an image, and a network 112 such as the Internet or a LAN. A network adapter 113 for connection, and a bus 114 for connecting the above-described units.
And

FIG. 2 is a block diagram showing the processing executed in the image search device shown in FIG. Although the details of the processing will be described later, in FIG.
An interface (U / I) unit 151 (corresponding to a designation unit and an output unit of the present invention) displays a search condition input screen, a search result display screen, and the like for performing an image search for a user, and displays the search condition. Input processing, output processing of search results, and the like. The feature extraction engine 152 (corresponding to the importance determination means and the feature extraction means of the present invention) includes an image DB
A process of extracting the feature amount of the image from the image registered in 108 (the image to be searched) and the query image specified at the time of the search is executed. Further, the matching engine 153
(Corresponding to the search means of the present invention) compares the feature amount extracted from the query image with the feature value of the image in the image DB 108 registered in the image feature DB 109, and determines the feature corresponding to the feature of the query image. A process for searching for an image is executed.

In FIG. 2, the U / I section 151, the feature extraction engine 152, and the matching engine 15
Reference numeral 3 denotes a module constituting the image search software 107 shown in FIG. The processing by these units is realized by the CPU 101 shown in FIG.

Next, the processing executed by the image search apparatus having the above-described configuration will be described in the order of (1) feature amount extraction processing and (2) image search processing.

(1) Feature Amount Extraction Process First, a process for extracting a feature amount representing the feature of an image from each image will be described with reference to an example of a process of registering a search target image in the image DB 108. Here, as an example, color characteristics (color histogram) of an image that have been well known from the past,
It is assumed that an edge feature of an image and a texture feature in the image are extracted as feature amounts.

FIG. 3 is a flowchart showing an outline of an image registration process executed in the image search device according to the first embodiment. Feature extraction engine 152 shown in FIG.
Inputs an image to be registered via the U / I unit 151 (S11). The input image may be any image such as an image read by a scanner, an image captured by a digital camera, an image downloaded via the network 112, an image stored in the hard disk 110 in advance, and the like.

Then, the feature extraction engine 152 executes a process of extracting a feature amount from the input image (S1).
2). In this step S12, it is assumed that the three types of feature amounts are extracted from each image. These three types of feature amounts can be extracted from each image using a conventionally known method as described below.

Color histogram of an image (hereinafter simply referred to as "color")
) Is an appropriate color space (for example, Lab, Lu)
v, HSV, etc.), and divides the image into a plurality of regions, checks which pixel in the image corresponds to which region in the color space, and normalizes the number of pixels in each region by the total number of pixels Can be obtained. An edge of an image (hereinafter referred to as “structure”) is, for example,
It can be obtained by using an edge extraction filter such as l. Further, the texture can be obtained by texture extraction based on a co-occurrence matrix (Reference: Mikio Takagi, supervised by Hirohisa Shimoda, "Image Analysis Handbook", University of Tokyo Press, 1991).

After extracting the features from the image in step S12, the feature extraction engine 152 registers the extracted features in the image feature DB 109 in association with the original image to be registered in the image DB 108 (S13). The original image is also registered in the image DB 108.

From the viewpoint of speeding up image retrieval,
For the search target images registered in the image DB 108,
It is preferable that a feature amount be extracted in advance and registered in the image feature DB 109 at the time of registration. However, when extracting feature amounts from each image to be searched at the time of image search, it is not necessary to extract feature amounts in advance.

By executing the processing shown in FIG. 3,
The image registered in the image DB 108 is mapped to a feature space as shown in FIG. 4 for each type of the extracted feature amount. At the time of the image search, the feature amount of the image is extracted from the query image as described later, and is mapped to the feature space shown in FIG. The points (black dots) shown in FIG. 4 indicate images mapped in the feature space, and the distance between the points of the query image and the points of each image is the similarity of the images.
Note that most of the feature amounts of an image are vector data, and a distance definition formula of a vector such as a Euclidean distance is generally used for calculating a distance between points which are similarities. Further, a unique distance can be defined according to the type of the feature amount of the image.

(2) Image Retrieval Processing Next, the image retrieval processing executed in the image retrieval apparatus according to the first embodiment will be described. The image search device according to the first embodiment takes into consideration that simply specifying an inquiry image as a search condition cannot convey a similar viewpoint between an inquiry image and an image desired by a user as a search result to the device. In addition, by designating a query image for each type of feature amount used in the search, it is possible to represent a single virtual query image that the user considers appropriate. Note that, as described above, the types of feature amounts that can be used here are three types of colors, structures, and textures.
It is assumed that the image is registered in the image 108.

FIG. 5 is an explanatory diagram showing a search window used at the time of a search, and FIG. 6 is a flowchart showing an image search process according to the first embodiment. U / I unit 1 in FIG.
When the image search processing is executed, a search window 200a shown in FIG. 5 is displayed on the monitor 106 shown in FIG. The search window 200a includes image specification columns A, B, and C for specifying one query image for each color, structure, and texture feature amount used for image search, and for specifying execution of image search processing. Search button 201 and image display field 2 for displaying the image of the search result
02.

The user specifies an inquiry image for each type of feature amount in each image specification column shown in FIG. 5 (S101). The image specified as the inquiry image may be any image such as an image read by a scanner, an image captured by a digital camera, an image downloaded via the network 112, an image stored in the hard disk 110 in advance, and the like. .

Here, a method of specifying an inquiry image will be specifically described. As a premise, there is no single image that appropriately represents an image desired by a user as a search result, but for example, an image A having an image similar in color to the image desired as a search result, an image structure (object It is assumed that an image B with a similar atmosphere and an image C with a similar texture atmosphere in the image have been found.

As shown in FIG. 5, the user designates these three images A, B and C in the image designation columns A, B and C, respectively, as query images. More specifically, an inquiry image A having an image color similar to that of a desired image as a search result is specified in an image specification column A corresponding to the feature amount “color”. Similarly, a query image B in which the atmosphere of the image structure is similar to the desired image as a search result is added to the query image C in which the texture atmosphere in the image is similar to the image specification column B corresponding to the feature amount “structure”. The feature amount "texture"
Is designated in the image designation field C corresponding to.

In the following description, the image designation column A
Are referred to as a query image A, an image designated in the image designation column B is referred to as a query image B, and an image designated in the image designation column C is referred to as a query image C.

As described above, not only by specifying a plurality of query images, but also by specifying a query image for each type of feature amount used in the image search, a virtual query The image can be expressed, and the user's intention for the search can be accurately transmitted to the device side. Therefore, by the search processing described later,
It is possible to obtain an image of the search result reflecting the intention of the user.

Returning to the description of the flowchart of FIG. When the user designates the search button 201 in FIG.
51 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting corresponding types of feature amounts from the query images A, B, and C (S102). . That is, the feature extraction engine 152 extracts a color feature from the query image A, a structural feature from the query image B, and a texture feature from the query image C. Then, feature extraction engine 1
52 matches and extracts the feature amount of each extracted query image.
It is passed to the engine 153 (mapped to the feature space in FIG. 4).

When the feature amounts extracted from the inquiry images A, B, and C are registered in the image feature DB 109, the processing in step S102 in FIG. 6 can be skipped.

Subsequently, the matching engine 153
For each image to be searched registered in the image DB 108,
The similarity to each of the query images A, B, and C is obtained using the corresponding feature amount, and the obtained similarities to the query images A, B, and C are summed to obtain the total similarity of the search target image. Is performed (S103).

The processing in step S103 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount. That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using
The distance between the images corresponds to the degree of similarity between the two images, and the shorter the distance, the more similar the two images in color characteristics. Similarly, the matching engine 153 obtains the distance between the search target image and the query image B based on the structural feature amount, and calculates the distance between the search target image and the query image C based on the texture feature amount. Ask.

Thereafter, the matching engine 153
The distance between the image to be searched and each of the inquiry images A, B, and C is totaled, and the total value of the distances is defined as the overall similarity. here,
The number of types of feature amounts is N, the query image of feature amount j is Q _j ,
The image to be searched is I _i , and the distance formula between the images of the feature amount j is D _j
Then, the total similarity S _i for each image to be searched is expressed as follows.

[0069]

(Equation 1)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

The U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of the overall similarity in the image display column 202 of FIG. (S104).
Here, since the sum of the distances between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, actually, the search targets are searched in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

It is not necessary to display all the images to be searched in the image display column 202, and only a predetermined number of images may be displayed, or a predetermined number of images may be displayed. (If it is represented), only the image that has obtained the overall similarity may be displayed. That is, how the search result is displayed can be arbitrarily set and changed (the same applies to the following embodiments).

When the user wants to perform a narrowing-down search as a result of referring to the image of the search result displayed in the image display column 202, the user designates the image displayed in the image display column 202 as a query image and performs a similar search. Processing can also be performed. Further, the image displayed in the image display field 202 is
By clicking on, processing for enlarging and displaying an image can also be performed.

As described above, according to the image search apparatus of the first embodiment, one query image is used for each type of feature used when obtaining the similarity between the query image and the search target image. For each image to be searched, the similarity with each specified query image is obtained using the corresponding type of feature quantity, and the obtained similarity with each query image is summed to obtain the image to be searched. By calculating the total similarity of the search results and outputting the search results based on the total similarity obtained for each image to be searched, the user's intention can be accurately expressed as a search condition. In addition to being able to communicate, the user's intention for the search can be reliably transmitted to the device, so that a search result that accurately reflects the user's intention for the search can be obtained, and the user can obtain the desired result. Degrees of high image retrieval processing can be performed.

For example, even if an appropriate image does not exist as an inquiry image, an image having a similar color atmosphere to an image desired as a search result or an image having an atmosphere similar to the shape of an object is designated as an inquiry image. A combination of these images can be used as one query.

In the description of the first embodiment, an inquiry image is specified corresponding to all types of feature amounts. However, an inquiry image is always specified corresponding to all types of feature amounts. No need. In addition, three types of feature amounts of color, structure, and texture are used, but these are merely examples, and are not intended to limit the type and number of feature amounts.

[Embodiment 2] In the image search apparatus according to Embodiment 2, just by designating the query image as a search condition, the query image and the user can obtain desired search results as in the case of Embodiment 1. In consideration of the fact that a similar viewpoint to the image to be performed cannot be transmitted to the device, by specifying a plurality of query images and the type of the feature amount used for the search, the virtual This makes it possible to express one inquiry image. In addition,
In the image retrieval device according to the second embodiment, the same reference numerals are used for the configuration already described in the first embodiment, and a detailed description is omitted, and a description of a common point in the image retrieval process is appropriately omitted. I do.

FIG. 7 is an explanatory diagram showing a search window used at the time of a search, and FIG. 8 is a flowchart showing an image search process according to the second embodiment. U / I unit 1 in FIG.
Reference numeral 51 displays a search window 200b shown in FIG. 7 on the screen of the monitor 106 shown in FIG. 1 when executing the image search process. The search window 200b is used to determine the similarity between the image specification fields A, B, and C for specifying a plurality of query images and the search target image for each query image specified in each image specification field. The image processing apparatus includes a feature amount designation button 203 for designating a type of feature amount, a search button 201 for designating execution of image search processing, and an image display field 202 for displaying an image of a search result.

As described in the first embodiment, it is assumed that there is no single image that appropriately represents the image desired by the user as a search result. It is assumed that an image A having a similar atmosphere, an image B having a similar atmosphere of the image structure, and an image C having a similar atmosphere of the texture in the image have been found.

The user designates a plurality of query images in the respective image designation fields shown in FIG. 7, and specifies the type of feature used when obtaining the similarity with the search target image for each designated query image. It is specified (S201).

Specifically, as shown in FIG.
The images A, B and C are designated as inquiry images in the image designation fields A, B and C. Then, the user designates “color” in the feature amount designation button 203 for the inquiry image A in which the color atmosphere of the image is similar to the desired image as a search result. Similarly, “structure” is designated for the query image B having a similar atmosphere of the image structure to the desired image as a search result, and “texture” is designated for the query image C having a similar texture atmosphere in the image. .

As described above, in addition to simply designating a plurality of query images, by designating a feature amount corresponding to a feature of an image emphasized by the user for each of the designated query images, a virtual image that the user considers appropriate is provided. In addition to being able to express a single query image, it is possible to accurately convey the user's intention for the search to the device side. Therefore, it is possible to obtain an image of a search result in which a user's intention is reflected by a search process described later.

In FIG. 7, however, different types of feature amounts are specified for each query image, but the same type of feature amounts may be specified. That is,
The same type of feature amount may be specified for all of a plurality of query images.

Then, the user presses the search button 201 in FIG.
Is specified, the U / I unit 151 transmits the query images A, B, and C, and the type of the feature amount to the feature extraction engine 15.
2, the feature extraction engine 152 executes a process of extracting a specified type of feature amount from each of the query images A, B, and C (S202). That is, the feature extraction engine 152 extracts a color feature from the query image A, a structural feature from the query image B, and a texture feature from the query image C. Then, the feature extraction engine 15
2 passes the extracted feature amount of each query image to the matching engine 153 (maps them to the feature space in FIG. 4).

When the feature amounts extracted from the inquiry images A, B, and C are registered in the image feature DB 109, the processing in step S202 in FIG. 8 can be skipped.

Subsequently, the matching engine 153
For each image to be searched registered in the image DB 108,
The similarity with each of the query images A, B, and C is obtained using the type of feature amount designated by the user, and the obtained similarities with the query images A, B, and C are summed up to obtain the search target image. A process for obtaining the overall similarity is executed (S20)
3).

The processing in step S203 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount. That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using
The distance between the images corresponds to the degree of similarity between the two images, and the shorter the distance, the more similar the two images in color characteristics. Similarly, the matching engine 153 obtains the distance between the search target image and the query image B based on the structural feature amount, and calculates the distance between the search target image and the query image C based on the texture feature amount. Ask.

Thereafter, the matching engine 153
The distance between the image to be searched and each of the inquiry images A, B, and C is totaled, and the total value of the distances is defined as the overall similarity. here,
The number of feature values is N, the number of query images is M, the k-th query image is Q _k , the image to be searched is I _i , the distance formula between images of the feature value j is D _j (), and the k-th image _Assuming that the presence / absence of the feature amount j is t _jk (1 if specified, 0 if not specified), the overall similarity S _i for each image to be searched is expressed as follows.

[0089]

(Equation 2)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

The U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of the overall similarity in the image display column 202 in FIG. (S204).
Here, since the sum of the distances between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, actually, the search targets are searched in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

It is not necessary to display all the images to be searched in the image display column 202. Only a predetermined number of images may be displayed, or the predetermined number of images may be displayed below a predetermined threshold (the total similarity is determined by the distance between the images). May be displayed only when the overall similarity of (indicated by) is obtained.

When a narrow search is desired as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed. Further, the image displayed in the image display field 202 is
By clicking on, processing for enlarging and displaying an image can also be performed.

As described above, according to the image search apparatus of the second embodiment, a plurality of query images are designated, and the feature used when obtaining the degree of similarity with the search target image for each designated query image. By specifying the type of quantity, for each image to be searched, the degree of similarity with each query image is obtained using the specified type of feature quantity, and the similarity with each obtained query image is summed to obtain a total similarity. The user's intention can be accurately expressed as a search condition by outputting the search result based on the total similarity obtained for each image to be searched, so that the user's intention can be accurately conveyed to the device. In addition to the above, since the user's intention for the search can be reliably transmitted to the apparatus, a search result that accurately reflects the user's intention for the search can be obtained, and a highly accurate image desired by the user can be obtained. It is possible to perform the search process.

For example, even when an appropriate image does not exist as an inquiry image, an image having a similar color atmosphere to an image desired as a search result or an image having an atmosphere similar to an object shape is designated as an inquiry image. A combination of these images can be used as one query.
In addition, since the user can specify a desired feature amount for each specified query image, search conditions can be set freely compared to the image search device according to the first embodiment.

In the description of the second embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. That is, even if only one inquiry image is specified, the search process can be performed in the same manner, but in the second embodiment, it is meaningful to specify a plurality of images. In addition, although three types of feature amounts of color, structure, and texture are used, these are merely examples, and are not intended to limit the types of feature amounts.

[Third Embodiment] An image retrieval apparatus according to a third embodiment is different from the image retrieval apparatus according to the second embodiment in that a user's intention for retrieval can be expressed more accurately.

FIG. 9 is an explanatory diagram showing a search window 200c used in the image search device according to the third embodiment. The search window 200c shown in FIG.
In addition to the configuration of the search window 200b shown in FIG. 7, an importance level designation bar 204 for designating the relative importance for each of the query images designated in the image designation fields A, B and C is provided.

As described in Embodiments 1 and 2,
As a premise, there is no single image that appropriately represents an image desired by a user as a search result, but for example, an image A in which the color of the image is similar to the image desired as the search result, an atmosphere of an image structure It is assumed that an image B similar to the image B and an image C similar to the texture in the image are found.

As shown in FIG. 9, the user designates these three images as inquiry images A, B and C in image designation columns A, B and C, respectively. Next, the user designates “color” in the feature amount designation button 203 for the query image A in which the color atmosphere of the image is similar to the desired image as a search result. Similarly, the query image B in which the atmosphere of the image structure is similar to the desired image as the search result
Is designated as "structure", and as for the query image C having a similar texture atmosphere in the image, "texture" is designated.

Subsequently, the user uses the importance designation bar 204 to designate the importance for each inquiry image. As an example, the importance designation bar 204 is “0” for each query image.
The importance of "+" and the importance of "-" can be designated with "." Note that the importance is not in both the "+" direction and the "-" direction,
Only the "+" direction or only the "-" direction may be designated.

For example, of the query images A, B, and C, the query image A having a color scheme similar to the desired image as the search result is the most important. When the user feels that he / she particularly wants an image, the user moves the importance designation bar 204 to a desired position in the “+” direction, and designates that the inquiry image A is important. When the importance is designated in this manner, the similarity between the image to be searched and the query image A is set to “+” corresponding to the significance specified for the query image A during the search described later. Is weighted.

If the user does not want a texture image such as the query image C as a search result, the user moves the importance designation bar 204 to a desired position in the “-” direction. That is, a NOT condition can be given to the inquiry image by specifying the importance in the “-” direction. Therefore, it is also possible to designate, as a query image, an image that the user thinks that an image having such characteristics is not desired as a search result. In this case, for the similarity between the image to be searched and the query image C,
"-" According to the degree of importance specified for query image C
Will be weighted.

As described above, by enabling the user to specify the importance for each query image, it is possible to accurately express the user's intention for the search as a search condition. That is, the importance is reflected as a weight to the similarity by the search processing described later,
It is possible to obtain an image of the search result reflecting the intention of the user. Note that it is not always necessary to specify the degree of importance, and if the user desires, the importance level may be designated for a desired query image.

Next, an image search process by the image search device according to the third embodiment will be described. FIG. 10 is a flowchart showing a search process according to the third embodiment.
The user designates a plurality of query images in each image designation field shown in FIG. 9 and designates a type of a feature amount used when obtaining a similarity with the image to be searched for each designated query image. The importance is designated as necessary (S301).
As illustrated in FIG. 9, it is assumed that the user has specified the inquiry images A, B, and C, and has specified the type and importance of the feature amount for each of the inquiry images A, B, and C.

Then, the user presses the search button 201 in FIG.
Is specified, the U / I unit 151 transmits the query images A, B, and C, and the type of the feature amount to the feature extraction engine 15.
2, the feature extraction engine 152 executes a process of extracting a specified type of feature amount from each of the query images A, B, and C (S302). That is, the feature extraction engine 152 extracts a color feature from the query image A, a structural feature from the query image B, and a texture feature from the query image C. Then, the feature extraction engine 15
2 passes the extracted feature amount of each query image to the matching engine 153 (maps them to the feature space in FIG. 4).

Subsequently, the matching engine 153
For each image to be searched registered in the image DB 108,
The similarity to each of the query images A, B, and C is obtained using the feature amount of the type designated by the user, and the obtained similarities are weighted according to the importance, and each of the query images A, B, and C is weighted.
A process is performed to obtain the total similarity of the search target image by summing the similarities with B and C (S303).

The process in step S303 will be specifically described with reference to one image to be searched. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount. That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using
Subsequently, the matching engine 153 compares the obtained distance with the query image A designated by the importance designation bar 204.
Is weighted according to the importance of. Similarly, the matching engine 153 calculates the distance between the image to be searched and the query image B based on the feature amount of the structure, weights the obtained distance according to the importance of the query image B, The distance between the image to be searched and the inquiry image C is obtained based on the feature amount of the texture, and the obtained distance is weighted according to the importance of the inquiry image C.

Since the similarity to be weighted is the distance between the feature values of the images, the weighting of “+” based on the importance of “+” works to shorten the distance between the images. The weighting of "-" based on the importance of "-" works to increase the distance between images.

Thereafter, the matching engine 153
The distance between the image to be searched and each of the inquiry images A, B, and C is totaled, and the total value of the distances is defined as the overall similarity. here,
The number of feature values is N, the number of query images is M, the k-th query image is Q _k , the image to be searched is I _i , the distance formula between images of the feature value j is D _j (), and the k-th image _Assuming that the importance of the feature amount j is w _jk (0 if the corresponding feature is not specified), the total similarity S _i for each image to be searched is expressed as follows.

[0111]

(Equation 3)

The matching engine 153 obtains the total similarity for each image to be searched in this way, and outputs the total similarity for each image to be searched to the U / I unit 151.

The U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of the overall similarity in the image display column 202 of FIG. (S304).
Here, since the sum of the distances between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, actually, the search targets are searched in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

If a narrow search is to be performed as a result of referring to the image of the search result displayed in the image display column 202, the image displayed in the image display column 202 is designated as a query image, and a similar search is performed. Processing can also be performed. Further, the image displayed in the image display field 202 is
By clicking on, processing for enlarging and displaying an image can also be performed.

As described above, according to the image retrieval apparatus of the third embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the query image, so that the similarity is obtained for each query image. It is possible to specify the degree of importance. For each image to be searched, the degree of similarity to each query image is obtained, and the similarity to each obtained query image is weighted according to the specified degree of importance. By doing so, since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the user's intention is reflected. It is possible to obtain an image of the reflected search result.

In the description of the third embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. However, in the third embodiment, since the importance is specified for each query image,
It is meaningful to specify a plurality of query images.

Although detailed description is omitted,
Also in the image search device described in the first embodiment, the importance may be designated for each query image as described in the third embodiment.

[Fourth Embodiment] The image search apparatus according to the fourth embodiment enables an AND search to be executed as one method for accurately expressing a user's intention as a search condition. In the image search device according to the fourth embodiment, the same components as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. Is omitted.

FIG. 11 is an explanatory diagram showing a search window 200d used in the image search device according to the fourth embodiment. The search window 200d includes an image designation field A, B, and C, a search button 201, an image display field 202, and an A for designating execution of an AND search.
An ND designation button 205 and an OR designation button 206 for designating execution of an OR search are provided. The OR search will be described in detail in an eighth embodiment, which will be described later. Here, only the AND search will be described. Although one type of feature amount is used in the search, a plurality of types of feature amounts may naturally be used.

Next, an image search process by the image search apparatus according to the fourth embodiment will be described. FIG. 12 is a flowchart illustrating an image search process according to the fourth embodiment. The user specifies a plurality of inquiry images in each image specification column shown in FIG.
The image search processing based on the ND condition is designated (S401).
As shown in FIG. 11, it is assumed that the user has designated three inquiry images A, B, and C.

Then, the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting a predetermined feature amount from each of the query images A, B, and C (S402). Thereafter, the feature extraction engine 152 passes the extracted feature amount of each query image to the matching engine 153 (FIG. 4).
Map to each feature space).

The matching engine 153 has an image DB
For each image to be searched registered in 108, the similarity with each of the query images A, B, and C is obtained using a predetermined feature amount, and the similarity with each of the obtained query images A, B, and C is obtained. Are executed to calculate the total similarity of the images to be searched (S403).

The processing in step S403 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. 4 based on the predetermined feature amount by using a distance definition expression of a vector such as a Euclidean distance. Is determined using Similarly, the matching engine 153 obtains the distance between the search target image and the query image B based on the predetermined feature amount, and calculates the distance between the search target image and the query image C.

Thereafter, the matching engine 153
The distance between the image to be searched and each of the inquiry images A, B, and C is totaled, and the total value of the distances is defined as the overall similarity. here,
When the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between the images is D (),
The overall similarity S _i for each image to be searched is represented as follows.

[0125]

(Equation 4)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

When a search is performed using a plurality of types of feature amounts, the similarity (distance) between the image to be searched and each query image is obtained for each type of feature amount, and for each image to be searched, The total similarity may be obtained by summing all the similarities with each query image obtained for each feature amount.

Subsequently, U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display column 202 of FIG. 11 in descending order of overall similarity (S
404). Since the total similarity here means the similarity of each of the search target images to all of the query images A, B, and C, the AND search of the images has been performed. However, since the sum of the distances between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, actually, the search targets are sorted in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the overall similarity has a value of 0 or more,
When the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

If the user wants to perform a narrowing-down search as a result of referring to the image of the search result displayed in the image display section 202, the user designates the image displayed in the image display section 202 as a query image and performs a similar search. Processing can also be performed.

As described above, according to the image search apparatus of the fourth embodiment, a plurality of query images are designated, and for each of the images to be searched, the degree of similarity to each of the designated query images is determined, and the similarity is determined. Since the total similarity with each query image is summed to obtain a total similarity, and a search result is output based on the total similarity obtained for each image to be searched, an AND search can be performed. It is possible to provide one method for accurately expressing a user's intention as a search condition.

In the description of the fourth embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. In addition, although the types of the feature amounts are not specifically shown, the colors, structures, textures, etc.
Any type of feature can be used alone or in combination.

[Fifth Embodiment] An image retrieval apparatus according to the fifth embodiment is different from the image retrieval apparatus according to the fourth embodiment in that the user's intention for retrieval can be expressed more accurately.

FIG. 13 is an explanatory diagram showing a search window 200e used in the image search device according to the fifth embodiment. The search window 200e shown in FIG.
In addition to the search window 200d shown in FIG. 11, an importance designation bar 207 for designating relative importance for each of the inquiry images designated in the image designation columns A, B, and C is provided.

As described in the third embodiment, the importance designation bar 207 designates the importance of “+” and the importance of “−” centering on “0” for each query image. Is what makes it possible. Therefore, if the user considers the query image to be important, specify the importance of "+"
If an image similar to such an image is not desired as a search result, a "-"
Can be specified. The specified severity is
It is reflected in the search results as a weight for the obtained similarity. In addition, only the "+" direction or only the "-" direction may be designated as the importance, not both the "+" direction and the "-" direction.

Next, an image search process by the image search apparatus according to the fifth embodiment will be described. FIG. 14 is a flowchart illustrating a search process according to the fifth embodiment.
The user designates a plurality of inquiry images in each image designation column shown in FIG. 13, designates the importance as needed for each designated inquiry image, and further designates an AND designation button 205.
Is selected to specify an image search process based on an AND condition (S501). As illustrated in FIG. 13, it is assumed that the user has specified the inquiry images A, B, and C, and has specified any degree of importance for each of the specified inquiry images A, B, and C.

Then, the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting a predetermined feature amount from each of the query images A, B, and C (S502). Thereafter, the feature extraction engine 152 passes the extracted feature amount of each query image to the matching engine 153 (FIG. 4).
Map to each feature space).

The matching engine 153 has an image DB
For each of the images to be searched registered in 108, the similarity with each of the inquiry images A, B, and C is obtained using a predetermined feature amount, and the obtained similarities are weighted according to the importance. Then, a process of calculating the total similarity of the images to be searched by summing the similarities with the query images A, B, and C is executed (S503).

[0138] The processing in step S503 will be specifically described using one image to be searched as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. 4 based on the predetermined feature amount by using a distance definition expression of a vector such as a Euclidean distance. Is determined using Next, the matching engine 153 weights the obtained distance in accordance with the importance of the query image A specified in the search window 200e. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B, weights the obtained distance according to the importance of the query image B, and also calculates the distance between the image to be searched and the query image B. The distance to C is obtained, and the obtained distance is weighted according to the importance of the inquiry image C.

After that, the matching engine 153
The distance between the image to be searched and each of the inquiry images A, B, and C is totaled, and the total value of the distances is defined as the overall similarity. here,
Assuming that the number of query images is M, the k-th query image is Q _k , the search target image is I _i , the distance formula between the images is D (), and the importance of the k-th image is w _k , The overall similarity S _{i for} each image is represented as follows.

[0140]

(Equation 5)

The matching engine 153 obtains the total similarity for each image to be searched in this way, and outputs the total similarity for each image to be searched to the U / I unit 151.

Subsequently, U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display field 202 of FIG. 13 in descending order of the overall similarity (S
504). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, the images to be searched are actually sorted in ascending order of the value of the overall similarity. It will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

If the user wants to perform a narrowing-down search as a result of referring to the image of the search result displayed in the image display section 202, the user designates the image displayed in the image display section 202 as a query image and performs a similar search. Processing can also be performed.

As described above, according to the image retrieval apparatus of the fifth embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the query image, so that each query image can be weighted. It is possible to specify the degree of importance. For each image to be searched, the degree of similarity to each query image is obtained, and the similarity to each obtained query image is weighted according to the specified degree of importance. By doing so, since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the user's intention is reflected. It is possible to obtain an image of the reflected search result.

In the description of the fifth embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. In addition, although the types of the feature amounts are not specifically shown, the colors, structures, textures, etc.
Any type of feature can be used alone or in combination.

[Embodiment 6] An image retrieval apparatus according to Embodiment 6 is different from the image retrieval apparatus of Embodiment 4 in that a user's intention for retrieval can be expressed more accurately.

FIG. 15 is an explanatory diagram showing a search window 200f used in the image search device according to the sixth embodiment. The search window 200f shown in FIG.
In addition to the search window 200d shown in FIG. 11, each query image A, B specified in the image specification columns A, B, and C
And C, an importance level designation bar 208 for designating an importance level for each type of feature amount used in the search. In addition, as an example of the feature amount used for the search,
Three types of colors, structures and textures are used.

The importance designation bar 208 enables the importance of each type of feature to be designated for each query image. The importance designation bar 204 of the third embodiment is used.
As described with reference to FIG. 9, it is possible to designate the importance of "+" and the importance of "-" centering on "0". In addition, only the "+" direction or only the "-" direction may be designated as the importance, not both the "+" direction and the "-" direction.

More specifically, the method of designating the importance is shown in FIG.
This will be described with reference to the inquiry image A of FIG. For example, if it is considered that the color scheme in the inquiry image A is close to the desired image as a search result, the user operates the color importance level designation bar 208 to designate the importance level of “+”. On the other hand, when it is considered that the structure in the query image A is hardly similar to the image desired as a search result, the user operates the structure importance designation bar 208 to designate the importance of “−”.

Then, in a search process described later,
The similarity between the query image A and the search target image is the color feature amount,
It is obtained for each feature amount of the structure and each feature amount of the texture. In the above example, since the importance of “+” is designated as the color feature and the feature of “−” is designated as the structure feature, the similarity calculated using the color feature is determined. Weighting of “+” is performed in accordance with the importance of “+”, and “−” is weighted in accordance with the importance of “−” for the similarity obtained using the feature amount of the structure. This processing is executed for the similarity with each image to be searched.

For the inquiry images B and C as well, the processing like the inquiry image A is executed by designating the importance for each feature amount described above.

As described above, by specifying the importance for each type of feature amount for each query image, it is possible to reliably convey the user's intention for the search to the apparatus side.
Then, by the search processing, the importance is reflected as a weight to the similarity obtained for each type of feature amount, so that it is possible to obtain a search result image in which the user's intention is reflected. Note that it is not always necessary to designate the importance, but if the user desires, the importance may be designated for any feature amount of the desired query image.

Next, an image search process performed by the image search device according to the sixth embodiment will be described. FIG. 16 is a flowchart illustrating a search process according to the sixth embodiment.
The user designates a plurality of query images in each image designation field shown in FIG. 15, designates the importance of each type of feature amount for each designated query image, and furthermore, an AND designation button 2
05 is selected and the image search processing based on the AND condition is designated (S601). As shown in FIG. 15, it is assumed that the user has designated the inquiry images A, B, and C, and has designated the importance for each type of feature amount for each of the inquiry images A, B, and C.

Then, the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting three types of feature amounts of color, structure, and texture from the query images A, B, and C (S602). Thereafter, the feature extraction engine 152 passes the feature amounts of each query image extracted for each of the three types of feature amounts to the matching engine 153 (maps them to the feature space in FIG. 4).

Subsequently, the matching engine 153
For each image to be searched registered in the image DB 108,
The similarity to each of the query images A, B, and C is obtained using three types of feature amounts, and the obtained similarities are weighted in accordance with the importance, and the similarity to each of the query images A, B, and C is calculated. A process of calculating the total similarity of the search target images by summing the similarities is executed (S603).

The processing in step S603 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount. That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using
Subsequently, the matching engine 153 weights the obtained distance in accordance with the importance of the feature amount of the color specified for the query image A in the search window 200f. Further, the matching engine 153 obtains distances using the structural feature amount and the texture feature amount with respect to the search target image and the query image A, respectively, and applies the obtained distances to the query image A in the search window 200f. On the other hand, weighting is performed according to the importance of the feature of the designated structure and the importance of the feature of the texture.

The matching engine 153 also performs
The distance between the image to be searched and the query image B is obtained for each of the three types of feature values, and the obtained distances are weighted according to the importance of each of the feature values specified for the query image B. . In addition, the distance between the search target image and the query image C is set to 3
For each type of feature amount, weighting is performed on each of the obtained distances in accordance with the importance of each feature amount specified for the inquiry image C.

After that, the matching engine 153
Images to be searched and inquiry images obtained for each of the three types of feature values
The distances to the images A, B, and C are summed, and the sum of the distances is summed up.
The similarity is assumed. Here, the number of feature amounts is N, the query image
Is M, the k-th query image is Q_k, Images to search for
To I_i, The distance formula between images of the feature amount j is D_j(), K-th
Let w be the importance of image feature j_jkWill give you
Total similarity S for each image _iIs expressed as follows.

[0159]

(Equation 6)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

The U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of the overall similarity in the image display column 202 of FIG. (S60)
4). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, the images to be searched are actually sorted in ascending order of the value of the overall similarity. It will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

If the user wants to perform a refined search by referring to the image of the search result displayed in the image display field 202, the user designates the image displayed in the image display field 202 as a query image and performs a similar search. Processing can also be performed.

As described above, according to the image search device of the sixth embodiment, when the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts, It is possible to specify the importance of each type of feature amount for each query image so that any weight can be given according to the type of the query image and feature amount. By calculating the degree of similarity to each query image for each type of feature amount, and by weighting the similarity to each query image obtained for each type of feature amount in accordance with the specified importance, Since the user's intention can be accurately expressed as a search condition, the user's intention can be accurately conveyed to the device, and the importance is reflected as a weight to the similarity, so that the user's intention is reflected. It becomes possible to the obtain an image of the search results.

In the description of the sixth embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. Further, the types of the feature amounts are not limited to the three types of color, structure, and texture, and may be any combination of the feature amounts.

[Embodiment 7] An image retrieval apparatus according to Embodiment 7 is different from the image retrieval apparatus of Embodiment 4 in that the apparatus can extract the user's intention for the retrieval.

In the above-described fifth and sixth embodiments, the user has to specify the importance of the inquiry image or the importance of each feature amount for each inquiry image. In, the similarity between query images specified by the user is obtained using the feature amounts used for image search, and the importance for each feature amount between query images is calculated based on the obtained similarity for each feature amount. The feature is that it is determined on the device side and reflected as a weight for the similarity. Here, as an example, it is assumed that search processing is performed using three types of feature amounts of color, structure, and texture, and a search window for specifying search conditions is shown in FIG.
It is assumed that this is the same as the search window 200d shown in FIG.

Next, an image search process performed by the image search apparatus according to the seventh embodiment will be described. FIG. 17 is a flowchart illustrating a search process according to the seventh embodiment.
The user displays the search window 200d shown in FIG. 11, specifies a plurality of query images in each image specification column, selects the AND specification button 205, and specifies image search processing based on AND conditions (S701). It is assumed that the user has specified the inquiry images A, B, and C.

Then, when the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting three types of feature amounts of color, structure, and texture from each of the query images A, B, and C (S702).

Subsequently, the feature extraction engine 152 sets each of the query images A, B, and C for each of the three extracted feature amounts.
The distance (similarity) between them is determined, and the importance of each of the query images A, B, and C for each of the feature amounts is determined based on the obtained distance of each of the feature amounts (S703).

More specifically, the feature extraction engine 152
The feature-to-feature distance between query images is obtained for each type of feature amount. If the variance of the distance obtained for each feature amount is large, it means that the user is not aware of the feature amount, and if the variance is small, it can be determined that the user values the feature amount. . Therefore, the importance is set to be small when the feature amount has a large variance, and the importance is set to be large when the variance is small.

For example, if σ ² is the variance of the distance between all query images in feature i, the importance w _k of the i-th feature can be determined by the following equation. k is an arbitrary coefficient for keeping w _k within an appropriate value.

[0172]

(Equation 7)

As described in the third, fifth and sixth embodiments, "+"
And the importance of “−” can be set.

Thereafter, the feature extraction engine 152 passes the feature amount of each query image extracted for each of the three types of feature amounts and the corresponding importance to the matching engine 153.

The matching engine 153 has an image DB
For each of the images to be searched registered in 108, the similarity to each of the query images A, B, and C is obtained using three types of feature amounts, and the obtained similarities are weighted according to the importance. Is performed, and a process of calculating the total similarity of the search target image by summing the similarities with the query images A, B, and C is executed (S704).

The process in step S704 will be specifically described with reference to one image to be searched. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount. That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using Subsequently, the matching engine 153 weights the obtained distance according to the importance determined for the color feature by the feature extraction engine 152. Further, the matching engine 153 searches for the image to be searched and the query image A
For each of the distances, a distance using the feature amount of the structure and the feature amount of the texture is obtained, and for each of the obtained distances, according to the importance of the feature amount of the structure and the importance amount of the feature amount of the texture determined by the feature extraction engine 152. Weighting.

The matching engine 153 also performs
The distance between the image to be searched and the query image B is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the degrees of importance determined for each of the types of feature amounts. Further, the distance between the image to be searched and the query image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the degrees of importance determined for each of the types of feature amounts.

Then, the matching engine 153
The distances between the search target image obtained for each of the three types of feature amounts and the query images A, B, and C are totaled, and the total value of the distances is used as the overall similarity. The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

Subsequently, the U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display column 202 of FIG. 11 in descending order of overall similarity (S
705). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, the images to be searched are actually sorted in ascending order of the value of the overall similarity. It will be displayed in the image display column 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

If a narrow search is to be performed as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed.

As described above, according to the image search apparatus of the seventh embodiment, when the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts, The similarity between the query images is obtained for each type of feature amount so that arbitrary weighting can be performed according to the type of the feature amount, and each similarity is determined based on the obtained similarity degree for each type of feature amount. The importance of each type of feature amount between query images was determined, and for each image to be searched, the similarity with each query image was obtained for each type of feature amount, and the similarity was obtained for each type of feature amount. For the similarity with each query image,
By performing weighting according to the determined importance of each type of feature amount, the apparatus determines the importance of each type of feature amount between query images specified by the user, and determines the determined importance level between the images. , It is possible to return a search result that meets the user's intention. Therefore, search processing desired by the user can be realized, and search results desired by the user can be obtained.

In the description of the seventh embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. Further, the types of the feature amounts are not limited to the three types of color, structure, and texture, and may be any combination of the feature amounts.

[Eighth Embodiment] An image search apparatus according to an eighth embodiment enables an OR search to be executed as one method for accurately expressing a user's intention as a search condition. The image search processing by the image search device according to the eighth embodiment is performed by using the AND designation button 2 in the search window 200d of FIG. 11 described in the fourth embodiment.
This is executed when the OR designation button 206 is designated instead of 05.

FIG. 18 is a flowchart showing an image search process according to the eighth embodiment. The user designates a plurality of query images in each image designation column shown in FIG. 11, selects the OR designation button 206, and designates an image search process based on the OR condition (S801). It is assumed that the user has specified three inquiry images A, B, and C as shown in FIG.

Then, when the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting a predetermined feature amount from each of the query images A, B, and C (S802). Thereafter, the feature extraction engine 152 passes the extracted feature amount of each query image to the matching engine 153 (FIG. 4).
Map to each feature space).

The matching engine 153 has an image DB
For each image to be searched registered in 108, the similarity with each of the query images A, B, and C is obtained using a predetermined feature amount, and the similarity with each of the obtained query images A, B, and C is obtained. Then, a process is performed to select a similarity having the highest similarity from among the similarities and obtain the overall similarity of the image to be searched (S803).

The processing in step S803 will be specifically described with reference to one image to be searched. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. 4 based on the predetermined feature amount by using a distance definition expression of a vector such as a Euclidean distance. Is determined using Similarly, the matching engine 153 obtains the distance between the search target image and the query image B based on the predetermined feature amount, and calculates the distance between the search target image and the query image C.

Then, the matching engine 153
The shortest distance is selected from the distance between the image to be searched and each of the inquiry images A, B, and C, and is set as the overall similarity. In other words, the matching engine 153 selects the similarity having the highest similarity from the similarity between the image to be searched and each of the query images A, B, and C. Here, assuming that the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between the images is D (), the overall similarity S for each search target image is _i is represented as follows.

[0189]

(Equation 8)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

When a search is performed using a plurality of types of feature amounts, the similarity (distance) between the image to be searched and each query image is obtained for each type of feature amount, and for each image to be searched, Similarity (shortest distance) indicating the highest similarity among the similarities to each query image obtained for each type of feature amount
May be selected as the overall similarity.

Subsequently, the U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display column 202 of FIG. 11 in descending order of overall similarity (S
804). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the overall similarity, actually, the search target is sorted in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the inquiry images and the feature amount of the image to be searched are equal.

In the eighth embodiment, if the image to be searched is similar to any one of query images A, B, and C, an overall similarity indicating high similarity can be obtained. Therefore, an OR search is realized.

If a narrow search is to be performed as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed.

As described above, according to the image search apparatus of the eighth embodiment, a plurality of query images are specified, and for each of the images to be searched, the degree of similarity to each of the specified query images is obtained and the search is performed. By selecting a similarity indicating the highest similarity from the similarities with each query image and outputting a search result based on the selected similarity, an OR search can be performed. It is possible to provide one method for accurately expressing an intention as a search condition.

In the description of the eighth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. In addition, although the types of the feature amounts are not specifically shown, the colors, structures, textures, etc.
Any type of feature can be used alone or in combination.

[Embodiment 9] The image search apparatus according to the ninth embodiment differs from the image search apparatus of the embodiment 8 in that the user's intention for the search can be expressed more accurately. The image search process performed by the image search device according to the ninth embodiment is similar to the image search process shown in FIG.
The third search window 200e is executed when the OR designation button 206 is designated instead of the AND designation button 205.

FIG. 19 is a flowchart showing an image search process according to the ninth embodiment. The user specifies a plurality of query images in each image specification column shown in FIG.
The importance is designated for each designated query image as required, and the OR designation button 206 is selected to designate an image search process based on the OR condition (S901). The user
As shown in FIG. 13, it is assumed that the inquiry images A, B, and C are designated, and arbitrary importance is designated for each of the designated inquiry images A, B, and C.

Then, the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting a predetermined feature amount from each of the query images A, B, and C (S902). Thereafter, the feature extraction engine 152 passes the extracted feature amount of each query image to the matching engine 153 (FIG. 4).
Map to each feature space).

The matching engine 153 has an image DB
For each of the images to be searched registered in 108, the similarity with each of the inquiry images A, B, and C is obtained using a predetermined feature amount, and the obtained similarities are weighted according to the importance. Then, a process of selecting the similarity having the highest similarity from among the similarities with the query images A, B, and C and calculating the overall similarity of the search target image is executed (S903).

The processing in step S903 will be specifically described with reference to one image to be searched. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. 4 based on the predetermined feature amount by using a distance definition expression of a vector such as a Euclidean distance. Is determined using Subsequently, the matching engine 153 weights the obtained distance in accordance with the importance of the query image A specified in the search window 200e. matching·
The engine 153 similarly calculates the distance between the image to be searched and the query image B, and calculates the query image B
, The distance between the image to be searched and the inquiry image C is obtained, and the obtained distance is weighted according to the importance of the inquiry image C.

Thereafter, the matching engine 153
The shortest distance is selected from the distance between the image to be searched and each of the inquiry images A, B, and C, and is set as the overall similarity. Here, the number of query images is M, the k-th query image is Q _k , the search target image is I _i , the distance formula between images is D (),
Assuming that the importance of the k-th image is w _k , the overall similarity S _i for each image to be searched is expressed as follows.

[0203]

(Equation 9)

The matching engine 153 obtains the overall similarity for each image to be searched in this way, and outputs the overall similarity for each image to be searched to the U / I unit 151.

Subsequently, U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display field 202 of FIG. 13 in descending order of the overall similarity (S
904). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the overall similarity, actually, the search target is sorted in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the inquiry images and the feature amount of the image to be searched are equal.

When a narrowing search is desired as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed.

As described above, according to the image search apparatus of the ninth embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the query image, so that It is possible to specify the degree of importance. For each image to be searched, the degree of similarity to each query image is obtained, and the similarity to each obtained query image is weighted according to the specified degree of importance. By doing so, since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the user's intention is reflected. It is possible to obtain an image of the reflected search result.

In the description of the ninth embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. In addition, although the types of the feature amounts are not specifically shown, the colors, structures, textures, etc.
Any type of feature can be used alone or in combination.

[Embodiment 10] An image retrieval apparatus according to Embodiment 10 is different from the image retrieval apparatus of Embodiment 8 in that a user's intention for retrieval can be expressed more accurately. The image search processing by the image search device according to the tenth embodiment is performed in the search window 200f of FIG.
This is executed when the OR designation button 206 is designated instead of the designation button 205.

FIG. 20 is a flowchart showing an image retrieval process according to the tenth embodiment. The user specifies a plurality of query images in each image specification column shown in FIG. 15, specifies the importance of each feature type in each specified query image, and further selects the OR specification button 206 to select the OR condition. Image search processing is designated (S1001). The user selects the inquiry images A, B and C as shown in FIG.
, And the importance for each type of feature is specified for each of the query images A, B, and C.

Then, when the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting three types of feature amounts of color, structure, and texture from the query images A, B, and C (S1002). Thereafter, the feature extraction engine 152 passes the feature amount of each query image extracted for each of the three types of feature amounts to the matching engine 153 (FIG. 4).
Map to each feature space).

The matching engine 153 has an image DB
For each of the images to be searched registered in 108, the similarity to each of the query images A, B, and C is obtained using three types of feature amounts, and the obtained similarities are weighted according to the importance. Is performed to select the similarity having the highest similarity from among the similarities with the query images A, B, and C, and execute the process of obtaining the overall similarity of the image to be searched (S100).
3).

[0213] The processing in step S1003 will be specifically described using one image to be searched as an example. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using Subsequently, the matching engine 153 weights the obtained distance in accordance with the importance of the feature amount of the color specified for the query image A in the search window 200f. Further, the matching engine 153
For the image to be searched and the query image A, the distances using the feature amount of the structure and the feature amount of the texture are obtained, and for each obtained distance, the feature of the structure specified for the query image A in the search window 200f. Weighting is performed according to the importance of the quantity and the importance of the feature quantity of the texture.

[0214] Similarly, the matching engine 153
The distance between the image to be searched and the query image B is obtained for each of the three types of feature values, and the obtained distances are weighted according to the importance of each of the feature values specified for the query image B. . In addition, the distance between the search target image and the query image C is set to 3
For each type of feature amount, weighting is performed on each of the obtained distances in accordance with the importance of each feature amount specified for the inquiry image C.

Thereafter, the matching engine 153
The shortest distance is selected from the distances between the search target image obtained for each of the three types of feature amounts and each of the query images A, B, and C, and the total similarity is determined. Here, the number of feature amounts is N, the number of query images is M, the k-th query image is Q _k , the image to be searched is I _i , and the distance formula between images of the feature value j is D _j (), k
_Assuming that the importance of the feature amount j of the second image is w _jk , the overall similarity S _i for each image to be searched is represented as follows.

[0216]

(Equation 10)

The matching engine 153 obtains the total similarity for each image to be searched in this way, and outputs the total similarity for each image to be searched to the U / I unit 151.

Subsequently, U / I section 151 performs matching
The engine 153 refers to the overall similarity obtained for each image to be searched, and displays, for example, images to be searched in the image display column 202 of FIG. 15 in descending order of the overall similarity (S
1004). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the overall similarity, actually, the search target is sorted in ascending order of the value of the overall similarity. The image will be displayed in the image display column 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the inquiry images and the feature amount of the image to be searched are equal.

[0219] When the user wants to perform a narrowing-down search as a result of referring to the image of the search result displayed in the image display field 202, the user specifies the image displayed in the image display field 202 as a query image and performs a similar search. Processing can also be performed.

As described above, according to the image search apparatus of the tenth embodiment, when the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts, It is possible to specify the importance of each type of feature amount for each query image so that any weight can be given according to the type of the query image and feature amount. By calculating the degree of similarity to each query image for each type of feature amount, and by weighting the similarity to each query image obtained for each type of feature amount in accordance with the specified importance, Since the user's intention can be accurately expressed as a search condition, the user's intention can be accurately conveyed to the device, and the importance is reflected as a weight for the similarity.
It is possible to obtain an image of the search result reflecting the intention of the user.

In the description of the tenth embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. Further, the types of the feature amounts are not limited to the three types of color, structure, and texture, and may be any combination of the feature amounts.

[Embodiment 11] The image retrieval apparatus according to the eleventh embodiment is the same as the image retrieval apparatus according to the embodiment 8, except that the user can sample the intention of the user for the retrieval. The image search processing performed by the image search device according to the eleventh embodiment corresponds to the search window 2 shown in FIG. 11 used in the description of the eighth embodiment (the fourth embodiment).
00d, not the AND designation button 205
This is executed when the R designation button 206 is designated.

FIG. 21 is a flowchart showing an image retrieval process according to the eleventh embodiment. The user designates a plurality of query images in each image designation column shown in FIG. 11, and designates an image search process based on an OR condition by selecting an OR designation button 206 (S1101). The user sees FIG.
It is assumed that the inquiry images A, B, and C are designated as shown in FIG.

Thereafter, the user operates the search button 20 shown in FIG.
When 1 is specified, the U / I unit 151 makes the inquiry images A and B
And C are input to the feature extraction engine 152, and the feature extraction engine 152 executes a process of extracting three types of feature amounts of color, structure, and texture from each of the query images A, B, and C (S1102).

Subsequently, the feature extraction engine 152 sets each of the query images A, B and C for each of the three extracted feature amounts.
The distance (similarity) between them is determined, and the importance of each of the query images A, B, and C for each of the feature amounts is determined based on the obtained distance of each of the feature amounts (S1103).

Specifically, the feature extraction engine 152
The feature-to-feature distance between a plurality of query images specified by the user is obtained for each type of feature amount. If the variance of the distance obtained for each feature amount is large, it means that the user is not aware of the feature amount, and if the variance is small, it can be determined that the user values the feature amount. . Therefore, for features with large variance, the importance is reduced,
When the variance is small, the importance is set to be large. For example, the importance of each feature can be determined using Equation 7 shown in the seventh embodiment.

Thereafter, the feature extraction engine 152 passes the feature amount of each query image extracted for each of the three types of feature amounts and the corresponding importance to the matching engine 153.

The matching engine 153 has an image DB
For each of the images to be searched registered in 108, the similarity to each of the query images A, B, and C is obtained using three types of feature amounts, and the obtained similarities are weighted according to the importance. Is performed to select the similarity having the highest similarity from the similarities with the query images A, B, and C, and to perform the process of obtaining the overall similarity of the search target image (S110).
4).

The processing in step S1104 will be specifically described with reference to one image to be searched. The matching engine 153 obtains the similarity between the image to be searched and the query image A using the color feature amount.
That is, the matching engine 153 calculates the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Is determined using Subsequently, the matching engine 153 weights the obtained distance according to the importance determined for the color feature by the feature extraction engine 152. Further, the matching engine 153 obtains distances using the feature amount of the structure and the feature amount of the texture for the image to be searched and the query image A, and for each of the obtained distances,
Weighting is performed according to the importance of the feature of the structure and the importance of the feature of the texture determined by the feature extraction engine 152.

The matching engine 153 also performs
The distance between the image to be searched and the query image B is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the degrees of importance determined for each of the types of feature amounts. Further, the distance between the image to be searched and the query image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the degrees of importance determined for each of the types of feature amounts.

Then, the matching engine 153
The shortest distance is selected from the distances between the search target image obtained for each of the three types of feature amounts and each of the inquiry images A, B, and C, and is set as the overall similarity. Matching engine 153
Calculates the total similarity for each image to be searched in this way, and outputs the total similarity for each image to be searched to the U / I unit 151.

The U / I unit 151 refers to the overall similarity of each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of the overall similarity in FIG.
The image is displayed in the first image display column 202 (S1105). Here, the image to be searched and the query image A,
Since the distance between the feature amounts with either one of B and C is used, the images to be searched are actually displayed in the image display column 202 in ascending order of the value of the overall similarity. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the inquiry images and the feature amount of the image to be searched are equal.

If the user wants to perform a narrowing-down search as a result of referring to the image of the search result displayed in the image display field 202, the user designates the image displayed in the image display field 202 as a query image and performs a similar search. Processing can also be performed.

As described above, according to the image search apparatus of the eleventh embodiment, when the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts, The similarity between the query images is obtained for each type of feature amount so that arbitrary weighting can be performed according to the type of the feature amount, and each similarity is determined based on the obtained similarity degree for each type of feature amount. The importance of each type of feature amount between query images is determined, and for each image to be searched,
The degree of similarity with each query image is obtained for each type of feature, and the similarity with each query image obtained for each type of feature is determined according to the importance of each type of feature determined. By performing weighting, the apparatus determines the importance of each type of feature amount between query images specified by the user, and reflects the determined importance on the similarity between the images. It is possible to return search results. Therefore, search processing desired by the user can be realized, and search results desired by the user can be obtained.

In the description of the eleventh embodiment, three images are used as inquiry images, but the number of inquiry images is not limited to three. Further, the types of the feature amounts are not limited to the three types of color, structure, and texture, and may be any combination of the feature amounts.

[Twelfth Embodiment] An image retrieval apparatus according to the twelfth embodiment enables an image retrieval from a different viewpoint from the first to eleventh embodiments. In the image search device according to the twelfth embodiment, the same components as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. Is omitted.

The image retrieval device according to the twelfth embodiment provides a feature level roughness (fineness of viewpoint) used at the time of retrieval.
Can be specified by the user, and the similarity between the query image and the image to be searched is obtained using the feature amount corresponding to the roughness level specified by the user. Here, the roughness of the feature is defined as a degree indicating how faithfully the feature extracted from the image expresses a predetermined feature of the image to be extracted. Therefore, in the following, 1. Feature extraction processing Processing executed by the image search device according to Embodiment 12 will be described in the order of image search processing.

[0238] 1. Feature amount extraction processing Here, a plurality of feature amounts having different roughnesses are extracted in advance for the image to be searched. We will extract the quantity. As described in the first embodiment, the timing at which the feature amount is extracted from the image to be searched is when the image to be searched is registered in the image DB 108. Therefore, a process of extracting a feature amount from each image will be described with reference to FIG.

The feature extraction engine 152 shown in FIG.
An image to be registered is input via the unit 151 (S1).
1) A process of extracting a feature amount from the input image is executed (S12).

In the twelfth embodiment, for example, based on an extraction criterion in which at least the number of divisions indicating the number of divisions of an image is defined in steps as the level of the roughness of the feature,
The feature extraction engine 152 can extract feature amounts having different roughness from the image. Specifically, for example, the following feature level roughness levels are defined as the extraction criterion, and the feature extraction engine 152 uses the image input in step S <b> 11 to obtain different roughness values for the same feature (color, structure, etc.). The feature level is extracted. Note that the extracted feature amounts differ only in roughness, and the types of the feature amounts (eg, color, structure, etc.) are the same.

Level 1: As shown in FIG.
The feature amount 1 is extracted from the entire image. Level 2: As shown in FIG. 22 (b), the image is divided into four parts to generate divided images A to D, and the feature amounts A to D are extracted from each of the divided images A to D, so that one A feature value 2 (feature value group) including feature values A to D is extracted from the image. Level 3: As shown in FIG. 22 (c), the image is divided into eight to generate divided images A to H, and feature amounts A to H are extracted from each of the divided images A to H, thereby forming one image. A feature amount 3 (feature amount group) including feature amounts A to H is extracted from the image. Level 4: As shown in FIG. 22D, the image is divided into 16 to generate divided images A to P, and by extracting the feature amounts A to P from each of the divided images A to P, one level is obtained. A feature amount 4 (feature amount group) including feature amounts A to P is extracted from the image.

Here, the process of extracting the feature values of levels 1 to 4 from the image will be specifically described by taking the process of extracting the structure (edge) and color (color histogram) of the image as the feature value as an example.

(1) Structure Level 1 Edge information is extracted from the image input in step S11 using an edge extraction filter such as sobel. From the extracted edge information, the position of the edge, the strength and direction of the edge can be obtained. The edge strength is binarized with an appropriate threshold value to obtain edge image data of edge components in up, down, left, and right directions. As an example, FIGS. 24A to 24D show edge image data of edge components in the up, down, left, and right directions obtained from the image shown in FIG.

With respect to the edge image data of FIGS. 24A to 24D, it is assumed that the edge portion is a black pixel and the others are white pixels. Edge information such as the average of the sizes of the connected rectangles of the black pixels and the dispersion of the size of the connected rectangles of the black pixels is extracted from each edge image data, and this is set as the feature amount 1 of the structure of the entire image.

Level 2 The edge image data shown in FIGS. 24A to 24D are respectively divided into divided images A to D (four divisions) as shown in FIG. The edge information described in (1) to (4) is extracted to obtain feature amounts A to D of the structure in units of divided images. As a result, the feature amount 2 including the feature amounts A to D can be extracted from the image.

Levels 3 and 4 Level 3 is shown in FIGS. 24 (a) to 24 (d).
Is divided into divided images A to H as shown in FIG. 22C (8 divisions), and the level 4 is divided into divided images A to P as shown in FIG. 22D. (16 divisions), and in the same manner as in the case of level 2, the feature amounts 3 and A to P
Can be extracted from the image.

(2) Color Next, the process of extracting the color (color histogram) of an image as a feature will be described.

Level 1 An appropriate color space (for example, Lab, Luv, HSV, etc.) is selected and divided into a plurality of regions, and it is determined to which region in the color space each pixel of the image input in step S11 corresponds. By investigating and normalizing the number of pixels for each region with the number of pixels of the entire image, the feature amount 1 for the entire image shown in FIG. 22A is extracted.

Level 2 An appropriate color space (for example, Lab, Luv, HSV, etc.) is selected and divided into a plurality of regions, and the image input in step S11 is divided into divided images A as shown in FIG.
To D (four divisions). Then, a feature amount A can be extracted by checking which region in the color space each pixel of the divided image A corresponds to, and normalizing the number of pixels for each region with the number of pixels of the entire divided image A. . The same processing is performed on the other divided images B to D to extract the characteristic amounts B to D, respectively, and the characteristic amount 2 including the characteristic amounts A to D is extracted from the image input in step S11.

Level 3 and Level 4 For Level 3, the image input in step S11 is divided into divided images A to H as shown in FIG.
22), the level 4 is divided into divided images A to P (16 divisions) as shown in FIG. 22D, and the feature amount 3 including the feature amounts A to H in the same manner as in the case of level 2. Then, a feature amount 4 including feature amounts A to P is extracted from the image.

As described above, by extracting the feature amount from the same image by changing the number of divisions, for example, a certain image is entirely blue, but when viewed in detail, blue and a little red are mixed. It is possible to extract a feature amount that depends on the fineness of the viewpoint such as For example, it is possible to extract a feature amount including information indicating where red is dispersed in an image centered on blue.

In this way, after extracting the feature amount of each level from the image input in step S11, as shown in the flowchart of FIG.
The extracted feature amount is registered in the image feature DB 109 in association with the original image registered in the image DB 108 (S1).
3).

Although the description has been given of the case where the features of the structure and color of the image are extracted as the feature amounts, other types of features may be extracted as the feature amounts. Further, for example, even when the image is divided into four parts, various methods other than the division method shown in FIG. 22B can be considered (for example, the image is divided into four parts vertically or horizontally). Therefore, the division method can be set arbitrarily.

[0254] 2. Image Search Processing FIG. 25 is an explanatory diagram illustrating an example of a search window 200g used in the image search device according to Embodiment 12. The search window 200g includes, in addition to the image specification columns A to C, the search button 201, and the image display column 202, a feature specification column 209 for specifying the type of a feature amount used for the search.
There is provided a roughness level specification column 210 for specifying which level (roughness) of the types of characteristic amounts specified in the characteristic specification column 209 is to be used. Feature specification column 2
09, for example, one or both of the structural characteristic amount and the color characteristic amount can be designated, and the roughness level designation column 210 can designate, for example, at least one of levels 1-4. And The levels that can be specified in the roughness level specification column 210 correspond to the respective levels of the feature amount extraction criterion described above.

If no feature value is specified in the feature specification field 209, it is assumed that search processing is performed using both the structure and color feature quantities, and the roughness level specification field 210 is used. If no level is specified, it is assumed that search processing is performed using the feature amounts of all levels. Also, in order to be able to specify all usable types of feature amounts, a feature specification column 2
A button “all” may be prepared in 09. Similarly, to be able to specify all levels,
A button “all” may be provided in the roughness level designation field 210.

Next, an image search process according to the twelfth embodiment will be described. Here, since different processing is performed depending on how to specify the search condition in the search window 200g, the outline of the image search processing will be described first with reference to the flowchart of FIG. The description will be made separately for each search condition.

(1) Outline of Image Retrieval Processing First, the outline of the image retrieval processing according to the twelfth embodiment will be described. The user designates at least one query image in the image designation field shown in FIG.
Then, in the roughness level designation field 210, the type of feature quantity used at the time of retrieval and the roughness level of the feature quantity used are designated (S1201).

Subsequently, when the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 inputs the query image, the type of feature amount and the type of roughness level to the feature extraction engine 152, and the feature extraction engine 152 The feature amount is extracted from the query image according to the level of the degree (S1202). However, if the feature amount has already been extracted from the query image specified by the user, the process of step S1202 can be skipped. If no feature has been extracted from the image to be searched, the feature may be extracted at this point. After that, the feature extraction engine 152 matches and extracts the feature amount of the extracted query image.
It is passed to the engine 153 (mapped to the feature space in FIG. 4).

The matching engine 153 has an image DB
For each of the search target images registered in the search window 108, a process of calculating the similarity with the query image using the feature amount of the level specified in the search window 200g is executed (S
1203). That is, the matching engine 153
Calculates the distance between the image to be searched and the query image plotted in the feature amount space shown in FIG. 4 based on the feature amount of the corresponding level using a distance definition formula defined for each feature.

The feature value F is usually expressed as a vector F = ｛f
1, f2, f3,..., Fn}, and assuming that the feature amounts extracted from the images a and b are Fa and Fb, respectively, a difference distance formula D (Fa, Fb) represented by the following Expression 11 is generally obtained. Used as a distance definition formula.

[0261]

[Equation 11]

This distance formula can be used in the case of the feature amounts of “structure” and “color”. If the “ratio of black pixels to the entire image” in “structure” is used as a feature amount as a histogram, An appropriate similarity can be obtained when there are enough edges in the image, but an appropriate similarity cannot be obtained when there are few edges in the image.

In the sense of similarity to a human edge, for example, an image having no edges in an image is slightly different from an image having some edges, but if the image has sufficient edges, the amount is small. Even if they are slightly different, they are insensitive. Therefore, this problem can be solved by normalizing the histogram with the number of all edge pixels (normalized distance formula) in order to increase the accuracy of the distance more than the difference distance formula.

That is, the elements of the histogram are given by the following equations. foi = fi / N where foi is a normalized histogram element. Further, the total number of pixels N is given by Expression 12.

[0265]

(Equation 12)

When the vector elements are all 0 and S = 0, normalization is not performed. That is, the normalized distance formula is given by the formula (13).

[0267]

(Equation 13)

When the number of edges in an image is small, the above formula is effective. However, when the number of edges is large, the conventional histogram can obtain a more appropriate similarity. Therefore, the following formula 14 has been devised so that when the number of edges is large, the calculation formula is close to the conventional formula, and when the number of edges is small, the calculation formula is close to the above formula.

[0269]

[Equation 14]

It should be noted that the value for normalization is varied as in the following Expression 15 according to the total number of pixels.

[0271]

(Equation 15) k is a constant, and is appropriately specified depending on the target image set.

In addition to performing the above calculations at the time of retrieval, by converting the vectors into vectors obtained by normalization or semi-normalization at the time of extracting feature amounts and registering them in the database, a simple difference expression can be obtained at the time of retrieval. Can be used to determine the distance, and a high-speed search can be performed.

Although the above-described semi-normalization formula can calculate an appropriate similarity irrespective of whether there are many or few edges in an image, when calculating at the time of retrieval, there is a problem that it takes time because of the complexity of the formula. is there. Therefore, the conventional difference distance formula and the normalized distance formula are selected and calculated according to the number of edges and the required search speed.

For example, the following cases can be classified. 1) When the edge ratio of both images is equal to or more than a certain threshold A, select the difference distance formula. 2) When either one is below the threshold A. a) When high speed is required. Select the difference distance formula. B) High speed. When is not required Use the normalized distance formula

However, in this method, a different equation is selected depending on the case, and the resulting metric space does not satisfy the axiom of distance. When it is necessary to satisfy the distance axiom, a semi-normalized expression is preferable.

Subsequently, U / I section 151 performs matching
The engine 153 refers to the degree of similarity to the query image obtained for each image to be searched, and displays, for example, images to be searched in the image display column 202 of FIG. 25 in descending order of similarity (S1204). However, since the distance between the feature amounts of the image to be searched and the query image is used as the similarity here, the images to be searched are actually displayed in the image display column 202 in ascending order of the value of the similarity. Will be. That is, the similarity has a value of 0 or more, and when the similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

(2) Specific Example of Image Retrieval Processing Next, while associating with the flowchart of FIG. 26 described above, one query image, “color” as a feature amount, and “level 1” as a roughness level were designated. When one query image, "color" as feature value and "level 2" are specified as roughness level One query image, "color" as feature value and two or more levels as roughness level are specified The case where a plurality of types of feature amounts are specified The case where a plurality of query images are specified The image search process according to the twelfth embodiment will be specifically described in this order.

When “Color” is specified as one query image and the feature amount and “Level 1” is specified as the roughness level In the search window 200g shown in FIG. It is assumed that “level 1” is designated as the “color” and the roughness level (S
1201).

The feature extraction engine 152 operates as shown in FIG.
As shown in (1), for the entire query image, “feature amount 1”, which is a feature amount corresponding to “level 1”, for the feature amount “color” is extracted (S1202).

Then, the matching engine 153
Based on the query image extracted by the feature extraction engine 152 and the feature amount 1 of the search target image, the similarity to the query image is obtained for each search target image (S120).
3). Specifically, the distance between the image to be searched and the query image plotted in the feature amount space shown in FIG. 4 based on the corresponding feature amount 1 is determined using a distance definition formula defined for each feature. Ask.

After that, the U / I section 151
The engine 153 refers to the similarity with the query image obtained for each image to be searched, and displays the images to be searched in the image display column 202 of FIG. 25 in the order of, for example, the highest similarity (smallest value) (S1204). ).

As described above, by making it possible to execute the image search process by selecting a feature amount having a roughness such as level 1, for example, a search can be performed using a color feature amount. Thus, it is possible to execute a search process such as searching for a blue image as a whole.

Although a detailed description will be omitted,
Even when “structure” or another type of feature is designated as the feature, search processing can be performed by the same processing.

When “Color” is specified as one query image and feature level and “Level 2” is specified as the roughness level In the search window 200g shown in FIG. 25, the user inputs one query image and feature level as “ It is assumed that “level 2” is designated as the “color” and the roughness level (S
1201).

The feature extraction engine 152 operates as shown in FIG.
, The query image is divided into divided images A to D,
The feature values A to D of “color” are extracted from each of the divided images A to D (S1202). As a result, the feature amount “color” is a feature amount corresponding to “level 2”, and the feature amount A
The "feature amount 2" consisting of .about.D is extracted from the query image.

Then, the matching engine 153
Based on the query image extracted by the feature extraction engine 152 and the feature amount 2 of the search target image, the similarity to the query image is obtained for each search target image (S120).
3). Specifically, the similarity between the divided images A, the divided images B, the divided images C, and the divided images D of the image to be searched and the query image is obtained. More specifically, based on the feature amount 2 extracted by the feature extraction engine 152, the divided images A, B, C, and C plotted in the feature space shown in FIG. The distance between D is determined using a distance definition formula defined for each feature. Here, the similarity is obtained for each of the divided images A to D, so that four similarities are obtained.

Subsequently, the matching engine 153
The similarity between the search target image and the query image is determined using the obtained four similarities. As a method of determining the similarity, for example, the following method can be considered. -1 The similarity between the image to be searched and the query image is determined by the sum of squares of the four similarities. -2 Determine the similarity between the image to be searched and the query image by summing the four similarities (see AND search). -3 Highest (smallest value) among the four similarities
By selecting the similarity, the similarity between the image to be searched and the query image is determined (see OR search). It should be noted that any one of the methods -1 to -3 may be designated in the search window 200g.

After that, the U / I section 151 sets the matching
The engine 153 refers to the similarity with the query image obtained for each image to be searched, and displays the images to be searched in the image display column 202 of FIG. 25 in the order of, for example, the highest similarity (smallest value) (S1204). ).

As described above, by making it possible to execute the image search processing by selecting a roughness feature amount such as level 2, for example, when performing a search using a color feature amount Therefore, it is possible to execute a search process such as searching for an image in which red is mixed in a part where blue is present as a whole. In other words, search processing can be performed based on which color exists at which position in the image,
The search process can be performed from a fine viewpoint that cannot be performed by a search using the feature amount (level 1) extracted for the entire image.

Although detailed description is omitted,
Even when “structure” or another type of feature is specified as the feature, the search process can be performed by the same process. Further, even when the roughness levels 3 and 4 are specified, the search processing can be performed by the same processing as the processing of the level 2 described above, and thus the description is omitted here.

[0291] When one query image, "color" is specified as the feature amount, and two or more levels are specified as the roughness level. Here, as an example of the case where two or more levels are specified as the roughness level, The image search processing will be described based on the case where all the roughness levels are specified. Therefore, FIG.
In the search window 200g shown in FIG.
It is assumed that one query image, “color” and all roughness levels are designated as the feature amounts (S1201). As described above, when no level is specified in the roughness level specification field 210 or when all levels are specified, all roughness levels are specified.

The feature extraction engine 152 operates as shown in FIG.
~ As shown in FIG.
The feature values 1 to 4 are extracted from the query image specified in the above according to the levels 1 to 4 (S1202). In addition, the feature amount 1 corresponding to the level 1 is obtained by the method described in the section.
The feature quantities 2 to 4 corresponding to the levels 2 to 4 can be extracted by the method described in the section.

Then, the matching engine 153
Using the query image extracted by the feature extraction engine 152 and the feature amounts 1 to 4 of the search target image, the similarity to the query image is obtained for each search target image (S120).
3). Here, the similarity between the search target image and the query image is determined for each feature amount (level), and as a result, four similarities to the query image are obtained for each search target image. The process of calculating the similarity between the image to be searched and the query image using the feature amount 1 (level 1) is the same as the process described in the section.
The process of obtaining the similarity between the image to be searched and the query image using the feature amounts 2 to 4 (levels 2 to 4) is the same as the process described in each section.

Subsequently, the matching engine 153
The final similarity with the query image is determined using the four similarities with the query image obtained for each level for each image to be searched. To determine the final similarity,
For example, the following method can be considered. -1 The final similarity with the query image is determined by the sum of squares of the four similarities (a modification of the AND search). -2 Determine the final similarity with the query image by summing the four similarities (see AND search). -3 Highest (smallest value) among the four similarities
By selecting the similarity, the final similarity with the query image is determined (see OR search).

When the final similarity to the query image is to be determined by the methods -1 and -2, for example, an image which is entirely blue and a certain portion is mixed with red Search processing (AND search)
Search). That is, since all (two or more) features of the roughness level are used, it is possible to search for a desired image with higher accuracy than in a case where a search is performed using only the features of a specific level. it can.
In the case where the final similarity with the query image is determined by the method of -3, for example, an image having an overall blue image and an image having an overall blue portion mixed with red are used. Search processing such as searching for one (O
R search).

It should be noted that any one of the methods -1 to -3 may be specified in the search window 200g.

After that, the U / I unit 151
The engine 153 refers to the similarity with the query image obtained for each image to be searched, and displays the images to be searched in the image display column 202 of FIG. 25 in the order of, for example, the highest similarity (smallest value) (S1204). ).

As described above, by making it possible to select a plurality of feature amounts having different roughnesses and execute the image search processing, the retrieval amount can be reduced as compared with the case where one feature amount having a specific roughness is used. It is possible to execute the search processing by various methods having different viewpoints for. Although detailed description is omitted, even when “structure” or another type of feature is designated as the feature, the search process can be performed by the same process.

When a plurality of types of feature values are specified When a plurality of types of feature values are specified in the feature specification field 209, the feature extraction engine 152 sets the search window 2
The feature amount of the type and level designated by 00g is extracted from the query image (S1202). For example, when a structure and a color feature amount are specified, the feature extraction engine 152 extracts a feature amount of a corresponding roughness level for the structure feature amount from the query image, and extracts a corresponding level of the color feature amount. 22 is extracted (FIG. 22).
reference).

Then, the matching engine 153
For each image to be searched, the similarity to the query image is obtained for each type of feature quantity, and the final similarity to the query image is determined using the obtained similarity for each type of feature quantity (S12).
03).

As a final similarity determination method, for example, the following method can be considered. -1 The final similarity with the query image is determined by the sum of squares of the similarity obtained for each type of feature amount (AND
Search variant). -2 The final similarity with the query image is determined by summing the similarities obtained for each type of feature amount (A
See ND search). When this method is used, it is possible to execute an AND search corresponding to a plurality of types of feature amounts. -3 Determine the final similarity with the query image by selecting the highest (smallest) similarity from among the similarities obtained for each type of feature amount (see OR search). When this method is used, it is possible to execute an OR search according to a plurality of types of feature amounts. It should be noted that any one of the methods -1 to -3 may be designated in the search window 200g.

For example, when one query image is specified, and the feature amounts of the structure and color and the roughness levels 1 and 2 are specified, the feature amount of the level 1 structure is used for each image to be searched. Similarity obtained using the level 2 structure feature, similarity obtained using the level 1 color feature, and similarity obtained using the level 2 color feature Is obtained. In such a case, all the obtained similarities may be summed to determine the final similarity with the query image (see the method of -2 (-1)), or the obtained similarity may be obtained. The highest (smallest) similarity may be selected from among the similarities to determine the final similarity with the query image (see method -3). Alternatively, the similarity of each level is totaled for each type of feature amount, or the highest (smallest) similarity is selected from the similarities of each level, and the similarity is selected for each type of feature amount. After calculating the degree of similarity to the query image,
-3 may be used to determine the final similarity to the query image.

After that, the U / I section 151
The engine 153 refers to the overall similarity obtained for each image to be searched, and sorts the images to be searched in the order of, for example, the highest overall similarity (smallest value) in the image display column 202 of FIG.
(S1204).

When a plurality of query images are specified When a plurality of query images are specified, the feature extraction engine 152 extracts the feature amount of the type and level specified in the search window 200g from each query image ( S1202). And the matching engine 15
3 obtains the similarity with each query image for each image to be searched as described in any of the above items, and then obtains the overall similarity of the images to be searched (S1203).

As a method for obtaining the overall similarity, for example, there is the following method. -1 For each image to be searched, the sum of squares of the similarity with each query image is set as the overall similarity (a modification of the AND search). -2 For each image to be searched, the similarity with each query image is summed to obtain a total similarity (see AND search).
When this method is used, an AND search can be executed. -3 For each image to be searched, the highest (smallest) similarity is selected from among the similarities to each query image, and is set as the overall similarity (see OR search). When this method is used, an OR search can be performed.

It should be noted that any one of the methods -1 to -3 may be specified in the search window 200g. If

After that, the U / I section 151
The engine 153 refers to the overall similarity obtained for each image to be searched, and sorts the images to be searched in the order of, for example, the highest overall similarity (smallest value) in the image display column 202 of FIG.
(S1204).

As described above, according to the image retrieval apparatus of the twelfth embodiment, at least one query image is specified, and the level of the feature amount extracted from the image is changed from coarse to fine. Based on the extraction criteria defined stepwise, at least one level of roughness of the feature amount used for the search is specified, and based on the extraction criteria, at least the coarse level specified from the query image and the plurality of search target images. The feature amount corresponding to the search level is extracted at a predetermined timing, and for each image to be searched, the similarity with the query image is obtained using the feature amount corresponding to the specified roughness level, Since the search result is output based on the similarity obtained for each image, it is possible to execute a plurality of types of search processes having different search accuracy. That is, since the user can specify at least one level of roughness of the feature amount used for the search, the fineness of the user's viewpoint for the search can be expressed as the search condition, and the user's intention can be accurately determined as the search condition. Can be expressed. Therefore, it is possible to perform an image search process as desired by the user.

[0309] If it is desired to perform a refined search as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed. In addition, although the types of the feature amounts are the structure and the color, only the structure is
Alternatively, only the color may be used, or another type of feature amount may be used. When a plurality of inquiry images and a plurality of types of feature amounts are specified, the importance may be specified as described in the other embodiments.

Further, in the twelfth embodiment, the roughness of the feature is defined by the number of divisions of the image. However, not only the image but also the objects existing in the image are to be extracted. Alternatively, the roughness of the feature amount for each object may be defined according to the number of divisions of the object. The search processing in the case where the object existing in the image is also targeted for extracting the feature amount is the same as the above-described search processing, and thus the detailed description is omitted here.

Further, as described above, it is also possible to realize the following image retrieval processing by utilizing the feature amount extracted for each divided image obtained by dividing the image.
Here, for convenience of explanation, a modified example of the image search device according to Embodiment 12 will be described with an example in which an image is divided into four parts.

(Modification 1) In the twelfth embodiment, the query image and the search target image are obtained by comparing the divided images of the query image and the search target image on a one-to-one basis to determine the similarity. Was determined for the similarity. On the other hand, in the first modification, for example, as shown in FIG. 27, one of the divided images of the query image is designated, and the similarity between the designated divided image and each divided image of the search target image is obtained. And an image having a divided image similar to a specific divided image of an inquiry image can be searched for from images to be searched.

(Modification 2) Modification 2 is, for example, shown in FIG.
As shown in (1), by obtaining the similarity between the entire query image and each divided image of the search target image, an image including the query image can be searched for from the search target images. .

Although a detailed description will be omitted, by enabling such a search process as in Modification 1 and Modification 2, it is possible to search for an image by various methods.

[Thirteenth Embodiment] The image retrieval apparatus according to the thirteenth embodiment is capable of searching for an image including a query image from among images to be searched, thereby obtaining the first embodiment described above. This makes it possible to perform an image search from a viewpoint different from that of 〜12. In the image search device according to the thirteenth embodiment, the same components as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. Is omitted.

Specifically, the image search device according to the thirteenth embodiment specifies at least one query image, and determines, for each search target image, a region of a predetermined size in the search target image and the query image. By repeatedly executing processing for obtaining a similarity using a predetermined feature amount extracted in advance over the entire search target image, an image including the query image can be searched for from the search target images. It is.

In the image search apparatus according to the thirteenth embodiment, there are various methods for obtaining the similarity between a region of a predetermined size in an image to be searched and a query image using a predetermined feature amount extracted in advance. Can be used, but here, a method as shown in FIGS. 29 and 30 will be used. For example, as shown in FIG. 29, the query image is divided into four to generate divided images a to d, a predetermined type of feature is extracted from each divided image, and the image to be searched is divided into 16 parts. The divided images A to P are generated, and the same kind of feature amount as in the case of the inquiry image is extracted from each divided image. However, the number of divisions of the query image and the image to be searched is not limited to 8 and 16, respectively, and it is sufficient that the relationship is such that the number of divisions of the query image <the number of divisions of the image to be searched. Then, using the feature amount extracted from each divided image, as shown in FIGS. 30 (a) to 30 (i),
The similarity between the query image and a part of the search target image is sequentially obtained. FIG. 30 shows the images to be searched shown in FIG. 29, respectively, and the hatched portions in the images to be searched correspond to the area compared with the query image. If the number of divisions of the query image and the image to be searched is reversed (for example, the query image is divided into 16 and the image to be searched is divided into 8. That is, the number of divisions of the query image> the division of the image to be searched) Number), and images included in the query image can be searched.

Hereinafter, the image search processing by the image search apparatus according to the thirteenth embodiment will be specifically described. In addition,
As a premise, it is assumed that at least with respect to an image to be searched, a feature amount has been extracted from each divided image as shown in FIG. The specific process of extracting the feature amount is as described in the twelfth embodiment, and a detailed description thereof will not be repeated. Further, the type of the feature amount to be extracted may be any type such as a color and a structure.

FIG. 31 is a flowchart of an image search process according to Embodiment 13, and FIG. 32 is an explanatory diagram showing an example of a search window 200h. The user sees FIG.
At least one query image is designated in any of the image designation fields A to C shown in FIG. 2 (S1301). Here, it is assumed that one inquiry image is specified for the sake of simplicity.

Subsequently, the user presses the search button 20 in FIG.
When 1 is specified, the U / I unit 151 inputs the query image to the feature extraction engine 152, and
Extracts a predetermined type of feature amount from the query image (S
1302). As shown in FIG. 29, the feature extraction engine 152 divides the query image to generate divided images a to d, and extracts a predetermined type of feature amount from each divided image. However, if the feature amount has already been extracted from the query image specified by the user, this step S130
Step 2 can be skipped.

Then, the matching engine 153
For each image to be searched registered in the image DB 108,
Using the feature amount extracted by the feature extraction engine 152, a process of obtaining a similarity between a predetermined region in the search target image and the query image is executed over the entire search target image to obtain a similarity with the query image. (S1303).

One example of the processing executed in step S1303 will be described in detail with reference to FIG. For example, focusing on FIG. 30A, the matching engine 153 includes a feature extraction engine 152
The similarity between the divided image a of the query image and the divided image A of the search target image is calculated by using the feature amounts of the divided images of the query image and the search target image extracted in advance. Similarity between the search target image and the divided image B ・ Similarity between the query image divided image c and the search target image divided image E ・ Similarity between the query image divided image d and the search target image F The similarity is calculated for each. In other words, the distance between the divided images plotted in the feature amount space shown in FIG. 4 is obtained by using a distance definition formula defined for each feature. Here, since the similarity is obtained for each of the four divided images, four similarities are obtained.

Subsequently, the matching engine 153
Using the obtained four similarities, the similarity between the region of the search target image shown in FIG. 30A and the query image is determined. As a method of determining the similarity, for example, the following method can be considered. -1 The similarity between the region of the image to be searched and the query image is determined by the sum of squares of the four similarities. -2 The similarity between the region of the image to be searched and the query image is determined by summing the four similarities (AN
D search). -3 Highest (smallest value) among the four similarities
By selecting the similarity, the similarity between the region of the image to be searched and the query image is determined (see OR search). Note that the search window 200h may be used to specify which of the methods -1 to -3 to use.

The matching engine 153 is shown in FIG.
The processing described above is also performed for the cases shown in (b) to FIG. 30 (i) to determine the similarity between each area of the search target image and the query image.

As shown in FIG. 30 (a) to FIG. 30 (i), the similarity between each region of the search target image and the query image is obtained. The similarity is obtained. Therefore, the matching engine 153 executes a process of determining a final similarity with the query image using nine similarities for each image to be searched. As a final similarity determination method, for example, the following method can be considered.

-1 The final similarity between the image to be searched and the query image is determined from the sum of squares of the nine similarities (a modification of the AND search). -2 The final similarity between the image to be searched and the query image is determined by summing the nine similarities (A
See ND search). -3 Highest (smallest value) among 9 similarities
By selecting the similarity, the final similarity between the search target image and the query image is determined (see OR search). Note that the search window 200h may be used to specify which of the methods -1 to -3 to use.

Returning to the description of the flowchart in FIG.
The / I unit 151 refers to the final similarity with the query image obtained for each image to be searched by the matching engine 153 and, for example, sorts the images to be searched in descending order of similarity in the image display column of FIG. 202 (S130).
4). Here, since the distance between the feature amounts of the image to be searched and the query image is used as the similarity, the images to be searched are actually displayed in the image display column 20 in ascending order of the value of the similarity.
2 will be displayed. That is, the similarity has a value of 0 or more, and when the similarity is 0, it means that the feature amount of each query image and the feature amount of the image to be searched are equal.

In the above description, one specified query image is used and only one type of feature amount is used for the search. However, a plurality of types of feature amounts can be used. You can specify more than one.

An example of a process when a plurality of types of feature amounts are used will be briefly described. Matching engine 1
As described above, 53 obtains the similarity between the image to be searched and the query image for each feature of the same type. Then, the matching engine 153 can determine the similarity with the query image for each image to be searched using the similarity obtained for each of the different types of feature amounts by the following method, for example.

-1 The final similarity with the query image is determined by the sum of squares of the similarity obtained for each type of feature amount (a modification of the AND search). -2 The final similarity with the query image is determined by summing the similarities obtained for each type of feature amount. When this method is used, an AN corresponding to a plurality of types of feature amounts is used.
D search can be executed. -3 The final similarity with the query image is determined by selecting the highest (smallest) similarity from among the similarities obtained for each type of feature amount. When this method is used, it is possible to execute an OR search according to a plurality of types of feature amounts. Note that the search window 200h may be used to specify which of the methods -1 to -3 to use.

[0331] An example of a process when a plurality of inquiry images are specified will be briefly described. As described above, the matching engine 153 calculates the similarity between the search target image and each query image. Then, the matching engine 153 can determine the similarity of the search target image to each query image for each search target image using, for example, the following method.

-1 For each image to be searched, the sum of squares of similarities with each query image is set as the overall similarity of the images to be searched (a modification of the AND search). By using this method, it is possible to search for an image including a plurality of images as query images. -2 For each image to be searched, the similarity to each query image is summed to obtain the overall similarity to the image to be searched. When this method is used, AND using multiple query images
A search can be performed. That is, it is possible to search for an image including a plurality of images as query images. -3 For each image to be searched, the highest (smallest) similarity is selected from among the degrees of similarity to each query image, and is set as the overall similarity of the images to be searched. When this method is used, it is possible to execute an OR search using a plurality of query images. That is, it is possible to search for an image including any one of the plurality of images as the inquiry image. Note that the search window 200h may be used to specify which of the methods -1 to -3 to use.

As described above, according to the image search apparatus of the thirteenth embodiment, at least one query image is specified, and for each search target image, a region of a predetermined size in the search target image is specified. A process of obtaining a similarity with a specified query image using a predetermined feature amount extracted in advance is repeatedly executed over the entire search target image, and a search result is output based on the similarity obtained for each search target image. To do
From the images to be searched, an image including the query image can be found. Therefore, it is possible to provide one technique for realizing an image search intended by the user.

[0334] If it is desired to perform a refined search as a result of referring to the image of the search result displayed in the image display field 202, the image displayed in the image display field 202 is designated as a query image and a similar search is performed. Processing can also be performed. When a plurality of inquiry images and a plurality of types of feature amounts are specified, the importance may be specified as described in the other embodiments.

[Embodiment 14] Next, an image classification apparatus according to the present invention will be described. This image classification device
Similar to the above-described image search device, images are classified using similarity between images. By classifying the images in advance, it is possible to search for the image desired by the user only by following the classification. This makes it possible to easily search for an image desired by the user by a method different from the image search.

An image classification device according to the fourteenth embodiment of the present invention classifies images by applying the image search processing described in the first embodiment. Therefore, Embodiment 1
The description of the points common to those described in Sections will be appropriately omitted.

FIG. 33 is a block diagram of an image classification device according to the fourteenth embodiment. 33, the same components as those of the image search device shown in FIG. 1 are denoted by the same reference numerals, and only different points will be described here. FIG.
The image classification device shown in FIG. 3 differs from the image retrieval device shown in FIG. 1 in that image classification software 301 for causing the CPU 101 (computer 100) to execute the function as the image classification device of the present invention and classification items ( A folder to be described later), a classification condition for classifying images to be classified into respective classification items, and a classification tree DB 302 in which image information of the classified result is registered.

FIG. 34 is a block diagram showing processing executed in the image classification device shown in FIG. 33, and corresponds to FIG. User interface (U /
I) section 151 (corresponding to the setting means, designating means, and display means of the present invention) displays a classification item setting screen, a classification result display screen, and the like for performing image classification for a user, and displays classification conditions. It executes input and output of classification results. The feature extraction engine 152 executes a process of extracting a feature amount of an image from an image (image to be classified) registered in the image DB 108 and an image set as a classification criterion described later. Further, the matching engine 153 (corresponding to the calculating means, the determining means, the classifying means, and the searching means of the present invention) calculates the similarity between the reference image and the image to be classified, and classifies the classification target based on the obtained similarity. (For example, classification tree DB30)
2 and the image ID is registered in the classification item registered in step 2 and the image is classified).

Next, the operation of the image classification device having the above-described configuration will be described in the order of (1) classification item setting processing and (2) image classification processing. Here, a plurality of images registered in the image DB 108 are set as images to be classified. The feature amounts used for classifying the images are the color, structure, and texture of the image. From the images to be classified in the image DB 108,
It is assumed that each type of feature has already been extracted by the processing described in the first embodiment with reference to FIG.

(1) Classification Item Setting Process FIG. 35 is an explanatory diagram showing an example of the image classification window 350a for performing the setting process of the classification item of the image and the display of the image after the classification. When the display of the image classification window 350a is designated by the user, the U / I unit 151 of FIG.
Is displayed, and processing such as creation of a folder and setting of classification conditions, which will be described later, is executed in response to a user instruction.

FIG. 35 shows a state in which items for managing images (hereinafter referred to as “folders”) are already set, but here, only the image DB folder 351 corresponding to the root exists. Assuming that
Processing for setting each folder will be described.

The image DB folder 351 in FIG. 35 is a folder for managing all the images registered in the image DB 108. By changing the setting of the image DB folder 351, the image to be classified can be changed from the image in the image DB to another image. For example, it is also possible to specify and classify an image obtained as a result of search by the image search device described in the first to thirteenth embodiments.

The user sets a new folder below the image DB folder 351 as a classification destination for classifying the images in the image DB 108. For example, the user selects “create folder” from a menu (not shown) and creates a folder 352 under the image DB folder 351.

Then, the user selects the folder 352 in order to set the classification condition for classifying the images to be classified (the images in the image DB 108) in the folder 352, and selects “set classification condition” from a menu (not shown). Is selected, and the classification condition setting window 3 shown in FIG.
60a is displayed.

The classification condition setting window 360a is shown in FIG.
The configuration is almost the same as a part of the search window 200a shown in FIG. 7, and the classification condition can be set in the classification condition setting window 360a by the same method as the search condition described in the first embodiment. Classification condition setting window 360
a is a plurality of reference images (corresponding to a query image of an image search) used as references for classifying images to be classified for each feature amount of, for example, color, structure, and texture used when classifying images. ), And a setting button 361 for specifying the setting of the classification condition for a desired folder.

As the image designated as the reference image, for example, an image read by a scanner, an image taken by a digital camera, an image downloaded via the network 112, an image previously stored in the hard disk 110, and registered in the image DB 108 Any image, such as an image being displayed, may be used.

As described in the first embodiment, the user associates, for example, three images with each type of feature and designates them as reference images A, B, and C in image designation columns A, B, and C, respectively. I do. Although not shown, a threshold value of the similarity between the reference image and the image to be searched can be set for each folder from the classification condition setting window 360a. As will be described later, the images to be classified whose similarity with the reference image set in the folder exceeds the threshold value are classified into the folder. Here, the threshold is set for each folder, but one threshold may be set for the entire folder.

When the user selects the setting button 361, the U / I unit 151 sets the designated reference image and threshold value in the folder 352 as classification conditions. In the case where the feature amount has not been extracted from the image set as the reference image, the feature extraction engine 152 may perform a process of extracting the feature amount when setting the classification condition in the folder. . Needless to say, a necessary feature amount may be extracted from each reference image when performing a process of classifying images.

Note that the user can select folders 354 to 357
Folder created (for example, folder 353)
(For example, folders 354 and 3 for folder 353)
55), and classification conditions can be set similarly.

In this manner, a plurality of folders 352 to 358 as shown in FIG. 35 are set as classification items for classifying images to be classified. The U / I unit 151
Using the created folder and the corresponding classification conditions as folder information, the classification tree D together with information on the hierarchical structure of the folder
Register in B302.

As described in the first embodiment, in addition to simply designating a plurality of reference images for each folder, each query image is associated with each type of feature used in classifying the images. Since it is possible to specify, it is possible to represent one virtual reference image that the user considers appropriate, and it is possible to accurately represent the user's intention for classification as a classification standard. Therefore, by the classification process described below,
It is possible to obtain an image of the classification result reflecting the intention of the user.

(2) Image Classification Processing FIG. 36 is a flowchart showing an image classification processing according to the fourteenth embodiment. When the user instructs the start of the classification process, the matching engine 153 of FIG. 34 calculates the similarity between each reference image and the image to be classified using the corresponding type of feature amount for each of the folders 352 to 358. Then, a process of calculating the total similarity of the images to be classified by adding the obtained similarities is performed (S1401).

The processing in step S1401 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A set in the folder 352 (see the classification condition setting window 360a in FIG. 35) and the classification target image by the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Similarly, the matching engine 153 calculates the distance between the image to be classified and the reference image B based on the feature amount of the structure, and calculates the distance between the image to be classified and the reference image C based on the feature amount of the texture. Ask.

Subsequently, the matching engine 153
The distances between the image to be classified and each of the reference images A, B, and C are totaled, and the total value of the distances is used as the overall similarity (see Equation 1). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Then, the matching engine 153
For each of the folders 352 to 358, it is determined whether or not the total similarity of the images to be classified obtained in step S1401 exceeds a threshold (S1402). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

The matching engine 153
The images to be classified are classified into folders in which the overall similarity of the images to be classified exceeds a threshold (S1403). That is,
When the overall similarity of the image to be classified exceeds the threshold, the matching engine 153 determines the folder as a classification destination of the image to be classified and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302. ID of the image to be classified (assumed to be preset)
Is registered in the classification tree DB 302.

Steps S1401 to S14 described above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. It should be noted that the images to be classified may be classified into two or more folders according to the overall similarity obtained for each folder.

FIG. 37 is an explanatory diagram showing the state of the image classification window 350a after the images in the image DB 108 have been classified into the folders 352 to 358. Each folder 35
2 to 358, representative images selected by the matching engine 153 are displayed. When the user selects a specific folder, the images classified into the folder are displayed in a list on the right side of the image classification window 350a.

As described above, according to the image classification device of the fourteenth embodiment, the classification items are set, and the similarity between the reference image and the image to be classified is determined for each of the set classification items. One reference image is set for each type of feature amount, and for each classification item, the similarity between the set reference image and the image to be classified is determined using the corresponding type of feature amount, and the similarity is determined. The overall similarity of the image to be classified is calculated by summing the similarities obtained by the classification, and based on the overall similarity of the image to be classified determined for each classification item and a preset classification destination determination criterion, By determining the classification item to be classified, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately conveyed to the device and the user's intention for the classification can be reliably determined. It is possible to convey to the apparatus, it is possible to obtain a classification result that accurately reflects the intention of the user for the classification, it is possible to realize the classification process as the user desires.

For example, even when there is no suitable image as a reference image, an image having a similar color atmosphere to an image desired as a classification result or an image having a similar object shape atmosphere is designated as a reference image. A combination of these images can be used as one query image.

Although three types of feature amounts, that is, color, structure, and texture, are used here, these are merely examples, and are not intended to limit the types of feature amounts.

Although the representative image is displayed for each folder as shown in FIG. 37, instead of this, a folder name is displayed or a reference image set in the folder is displayed. Is also good.

Although a detailed description is omitted, when a new image is registered in the image DB 108, the image is also classified by performing a classification process.
The images in 108 can always be kept in a classified state. For example, if the U / I unit 151 monitors the state of the image DB 108, the image D
The image in B108 can always be kept in a classified state.

[Embodiment 15] An image classifying apparatus according to Embodiment 15 classifies images by applying the image search processing described in Embodiment 2. In the image classification device according to the fifteenth embodiment, the components already described in the fourteenth embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and the operation of the image classification device is common. A description will be appropriately omitted.

The operation of the image classification device according to the fifteenth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Processing FIG. 38 is an explanatory diagram showing an example of an image classification window 350b for performing processing for setting image classification items and displaying images after classification. Here, the user can select the folder 352
Is created, and a classification condition setting window 360b as shown in FIG. 38 is displayed in order to set the classification condition in the folder 352.

A classification condition setting window 36 shown in FIG.
0b has almost the same configuration as a part of the search window 200b shown in FIG.
For b, a classification condition can be set in the same manner as the search condition described in the second embodiment. The classification condition setting window 360b includes image specification columns A, B, and C for specifying a plurality of reference images (corresponding to a query image for image search) used as references for classifying images to be classified. For each of the reference images specified in the image specification field, a feature amount designation button 362 for designating a type of feature amount to be used when obtaining the similarity with the image to be classified, and setting of classification conditions for the corresponding folder. And a setting button 361 for designation.

As described in the second embodiment, the user designates, for example, three images as reference images A, B, and C in image specification columns A, B, and C, respectively, and sets each of the reference images A, B, and C For each C, the type of the feature amount used when obtaining the similarity with the image to be classified is designated. In the classification condition setting window 360b shown in FIG. 38, for example, a desired feature amount can be designated from colors, structures, and textures.

Although not shown, a threshold value of the similarity between the reference image and the image to be searched can be set for each folder from the classification condition setting window 360b. As will be described later, the images to be classified whose similarity with the reference image set in the folder exceeds the threshold value are classified into the folder. Here, the threshold is set for each folder, but one threshold may be set for the entire folder.

When the user selects the setting button 361, the U / I unit 151 sets the specified reference image, the type of the feature amount, and the threshold value in the folder 352 as the classification condition. In the case where the feature amount has not been extracted from the image set as the reference image, the feature extraction engine 152 may perform a process of extracting the feature amount when setting the classification condition in the folder. . Needless to say, a necessary feature amount may be extracted from each reference image when performing a process of classifying images.

[0371] It is to be noted that the user can select folders 354-357.
Folder created (for example, folder 353)
(For example, folders 354 and 3 for folder 353)
55), and classification conditions can be set similarly.

In this manner, a plurality of folders 352 to 358 as shown in FIG. 38 are set as classification items for classifying images to be classified. The U / I unit 151
Using the created folder and the corresponding classification conditions as folder information, the classification tree D together with information on the hierarchical structure of the folder
Register in B302.

As described in the second embodiment, not only a plurality of reference images can be simply designated for each folder, but also a feature amount corresponding to a feature of an image emphasized by the user can be designated for each designated reference image. Therefore, it is possible to represent one virtual reference image that the user considers appropriate, and to accurately represent the user's intention for classification as a classification criterion. Therefore, it is possible to obtain an image of a classification result reflecting a user's intention by a classification process described later.

(2) Image Classification Processing The image classification processing of the fifteenth embodiment will be described with reference to FIG. When the user instructs to start the classification process,
The matching engine 153 in FIG.
For every 2 to 358, the similarity between each reference image and the image to be classified is obtained using the feature amount of the type specified by the user, and the obtained similarities are summed to obtain the overall similarity of the image to be classified. The required processing is executed (S1401).

The processing in step S1401 will be specifically described with reference to the folder 352 as an example. The matching engine 153 refers to the folder information of the classification tree DB 302, and determines the similarity between the reference image A set in the folder 352 (see the classification condition setting window 360b in FIG. 38) and the classification target image by using the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Similarly, the matching engine 153 calculates the distance between the image to be classified and the reference image B based on the feature amount of the structure, and calculates the distance between the image to be classified and the reference image C based on the feature amount of the texture. Ask.

After that, the matching engine 153
The distances between the image to be classified and each of the reference images A, B, and C are summed, and the sum of the distances is used as the overall similarity (see Equation 2). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Subsequently, the matching engine 153
For each of the folders 352 to 358, it is determined whether or not the total similarity of the images to be classified obtained in step S1401 exceeds a threshold (S1402). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

Then, the matching engine 153
The images to be classified are classified into folders in which the overall similarity of the images to be classified exceeds a threshold (S1403). That is,
When the overall similarity of the image to be classified exceeds the threshold, the matching engine 153 determines the folder as a classification destination of the image to be classified and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302. ID of the image to be classified (assumed to be preset)
Is registered in the classification tree DB 302.

Steps S1401 to S14 described above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. It should be noted that the images to be classified may be classified into two or more folders according to the overall similarity obtained for each folder. When the image classification process is executed, FIG.
8 of the image classification window 350b
At 358, a representative image is displayed as shown in FIG.

As described above, according to the image classification apparatus of the fifteenth embodiment, a folder is set, a plurality of reference images are set for each set folder, and an image to be classified is set for each set reference image. The type of the feature amount used when obtaining the similarity with the target image is set. For each folder, the similarity between each reference image and the image to be classified is obtained using the set type of the characteristic amount, and the obtained similarity is determined. By calculating the total similarity by summing the degrees and determining the folder to which the image to be classified is to be classified based on the total similarity obtained for each folder and a preset classification destination determination criterion, the user's intention is determined. Can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the device, and the user's intention for the classification can be reliably transmitted to the device. It is possible to obtain a classification result that accurately reflects the intention of the user, it is possible to realize the classification process as the user desires.

For example, even if there is no suitable image as a reference image, an image having a similar color atmosphere to an image desired as a classification result or an image having a similar object shape atmosphere is designated as a reference image. A combination of these images can be used as one query image.

Although the three types of feature amounts of color, structure, and texture are used here, these are merely examples, and are not intended to limit the types of feature amounts.

Although a detailed description will be omitted, when a new image is registered in the image DB 108, the image is also classified by performing a classification process.
The images in 108 can always be kept in a classified state. For example, if the U / I unit 151 monitors the state of the image DB 108, the image D
The image in B108 can always be kept in a classified state.

[Embodiment 16] The image classifying apparatus according to the sixteenth embodiment differs from the image classifying apparatus according to the fifteenth embodiment in that the image search processing described in the third embodiment is applied to enable the user to classify the image. It is intended to express the intention more accurately. Embodiment 16
In the description of the configuration already described in the fourteenth embodiment, the same reference numerals are used to omit a detailed description, and a description of the common operation of the image classification device is omitted as appropriate.

The operation of the image classification device according to the sixteenth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Process FIG. 39 is an explanatory diagram showing an image classification window 350c used in the image classification device according to the sixteenth embodiment. Here, it is assumed that the user has created the folder 352, and displays a classification condition setting window 360c as shown in FIG. 39 to set the classification condition in the folder 352.

A classification condition setting window 36 shown in FIG.
0c indicates the relative importance of each reference image specified in the image specification columns A, B, and C in addition to the one shown in FIG. 38 (as shown in the search window 200c of the third embodiment). A designated importance designation bar 363 is provided.

[0387] As shown in FIG.
The images are designated as reference images A, B, and C in image designation columns A, B, and C, respectively, and the designated reference images A, B, and C are designated.
For each of B and C, the type of the feature amount used when obtaining the similarity with the image to be classified is designated.

Subsequently, the importance designation bar 2 of the third embodiment
As described with reference to FIG.
Is used to designate the importance for each reference image. The importance designation bar 363 enables designation of importance of “+” and importance of “−” centering on “0” for each reference image. In addition, only the "+" direction or only the "-" direction may be designated as the importance, not both the "+" direction and the "-" direction.

As described above, by specifying the importance for each reference image, it is possible to accurately express the user's intention for classification. That is, the importance is reflected as a weight to the similarity by the classification processing described later, so that it is possible to obtain an image of the classification result reflecting the intention of the user.

Further, when the user specifies the threshold value of the folder 352 and selects the setting button 361, the U / I unit 15
1 sets the reference image, the type of feature amount, the importance, and the threshold value in the folder 352 as classification conditions.

By performing such processing for each folder, a plurality of folders 352 as shown in FIG.
358 are set as classification items for classifying images to be classified. The folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing FIG. 40 is a flowchart showing an image classification processing according to the sixteenth embodiment. When the user instructs the start of the classification process, the matching engine 153 of FIG. 34 uses each of the folders 352 to 358 to determine the similarity between each reference image and the image to be classified using the type of feature amount designated by the user. The obtained similarities are weighted according to the importance, and the similarities are totaled to execute a process of obtaining the overall similarity of the image to be classified (S1601).

The processing in step S1601 will be specifically described with reference to the folder 352 as an example. The matching engine 153 refers to the folder information of the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360c in FIG. 39) set in the folder 352 and the classification target image by using the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the reference image A specified by the user. Matching engine 153
Similarly, the distance between the image to be classified and the reference image B is obtained based on the feature amount of the structure, the obtained distance is weighted according to the importance of the reference image B, and the feature amount of the texture is obtained. , The distance between the image to be classified and the reference image C is obtained, and the obtained distance is weighted according to the importance of the reference image C.

Then, the matching engine 153
The distance between the image to be classified and each of the reference images A, B, and C is totaled, and the total value of the distances is used as the overall similarity (see Equation 3). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Subsequently, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S1601 exceeds a threshold (S1).
602). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S1603). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S1601-S16 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification processing is executed, the folders 352 to 358 of the image classification window 350c in FIG.
Displays a representative image as shown in FIG.

As described above, according to the image classification apparatus of the sixteenth embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the reference image. It is possible to set the degree of importance, for each folder, determine the similarity of each reference image and the image of the classification target, by weighting the obtained similarity according to the set importance, Since the user's intention can be accurately expressed as a classification condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the image reflecting the user's intention is reflected. Can be obtained.

Although detailed description is omitted,
Also in the image classification device described in the fourteenth embodiment, the importance may be specified for each query image as described in the sixteenth embodiment.

[Embodiment 17] The image classification device according to the embodiment 17 applies the AND search described in the embodiment 4 as one method for accurately expressing the user's intention as the classification condition. Image classification processing is performed. In the image classification device according to the seventeenth embodiment, the components already described in the fourteenth embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and the operation of the image classification device is common. A description will be appropriately omitted. Further, in the image classification device according to the seventeenth embodiment, one type of feature amount is used when obtaining the similarity between images, but it is needless to say that a plurality of types of feature amounts may be used. .

The operation of the image classification device according to the seventeenth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Process FIG. 41 is an explanatory diagram showing an image classification window 350d used in the image classification device according to the seventeenth embodiment. Here, it is assumed that the user has created the folder 352, and displays a classification condition setting window 360d as shown in FIG. 41 to set the classification condition in the folder 352.

A classification condition setting window 36 shown in FIG.
0d is provided with an AND designation button 364 for designating an AND condition and an OR designation button 365 for designating an OR condition, in addition to the image designation fields A, B and C and the setting button 361. . Note that OR
The conditions will be described in detail in Embodiment 21 described later.

The user, as shown in FIG.
The images are designated as reference images A, B, and C in image designation columns A, B, and C, respectively, and an AND designation button 36
Select 4 to specify the AND condition. Further, when the user specifies the threshold value of the folder 352 and specifies the setting button 361, the U / I unit 151 sets the specification of the reference image and the AND condition and the threshold value as the classification condition in the folder 352.

By performing such processing on each folder, a plurality of folders 352 as shown in FIG.
358 are set as classification items for classifying images to be classified. The folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing The image classification processing of the seventeenth embodiment will be described with reference to FIG. When the user instructs to start the classification process,
The matching engine 153 in FIG.
For each of 2 to 358, a process of calculating the similarity between each reference image and the image to be classified using a predetermined feature amount, and summing up the obtained similarities to obtain the overall similarity of the image to be classified is executed ( S1401).

[0408] The processing in step S1401 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image by a predetermined feature amount. Is determined using That is, the matching engine 153
Calculates the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount using a distance definition formula of a vector such as a Euclidean distance. Similarly, the matching engine 153 calculates the distance between the classification target image and the reference image B based on the predetermined feature amount, and also calculates the distance between the classification target image and the reference image C.

Then, the matching engine 153
The distances between the image to be classified and each of the reference images A, B, and C are summed, and the sum of the distances is used as the overall similarity (see Equation 4). When the similarity between each reference image and the image to be classified is determined using a plurality of types of feature amounts, the similarity (distance) between the image to be classified and each reference image is determined for each type of feature amount, The total similarity may be obtained by summing all the similarities for each reference image for each image to be classified. The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Next, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S1401 exceeds a threshold (S1).
402). However, here, distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, the total similarity is actually equal to or less than (or less than) the threshold. Will be determined.

[0411] Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S1403). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S1401 to S14 described above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, a search target image having the highest overall similarity (the smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358. , A classification tree D associated with the corresponding folder information
Register in B302. When the image classification process is performed,
Each folder 35 in the image classification window 350d in FIG.
In 2 to 358, representative images are displayed as shown in FIG.

As described above, according to the image classification device of the seventeenth embodiment, a folder is set, a plurality of reference images are set for each set folder, and each set reference image and classification are set for each folder. The similarity of the target image is obtained, and the obtained similarities are summed to obtain an overall similarity. The classification of the image to be classified is performed based on the overall similarity obtained for each folder and a preset classification destination determination criterion. In order to determine the destination folder, it is possible to perform image classification processing applying AND search processing. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

[0414] In the seventeenth embodiment, the types of the characteristic amounts are not particularly shown. However, any type of characteristic amounts such as colors, structures, and textures can be used alone or in combination.

[0415] Further, although a detailed description is omitted, when a new image is registered in the image DB 108, the image is also classified by performing a classification process.
The images in 108 can always be kept in a classified state. For example, if the U / I unit 151 monitors the state of the image DB 108, the image D
The image in B108 can always be kept in a classified state.

[Embodiment 18] The image classifying apparatus according to the eighteenth embodiment differs from the image classifying apparatus according to the seventeenth embodiment in that the image search processing described in the fifth embodiment is applied so that the user It is intended to express the intention accurately.

The operation of the image classification device according to the eighteenth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Process FIG. 42 is an explanatory diagram showing an image classification window 350e used in the image classification device according to the eighteenth embodiment. Here, it is assumed that the user has created the folder 352, and displays a classification condition setting window 360e as shown in FIG. 42 to set the classification condition in the folder 352.

A classification condition setting window 36 shown in FIG.
0e is the classification condition setting window 360 shown in FIG.
In addition to d, an importance designation bar 366 for designating relative importance between the respective reference images designated in the image designation columns A, B and C is provided.

[0420] As shown in FIG.
The images are designated as reference images A, B, and C in image designation columns A, B, and C, respectively, and the importance is designated for each of the reference images A, B, and C using an importance designation bar 366. This importance designation bar 366 is used in the fifth embodiment.
As described with reference to the importance designation bar 207, it is possible to designate the importance of "+" and the importance of "-" centering on "0" for each reference image. In addition,
As the degree of importance, not only the "+" direction and the "-" direction, but also the "+" direction or only the "-" direction may be specified.

As described above, by specifying the importance for each reference image, the user's intention for classification can be accurately expressed. That is, the importance is reflected as a weight to the similarity by the classification processing described later, so that it is possible to obtain an image of the classification result reflecting the intention of the user.

[0422] Further, the user can select the AND button 364.
And specify the AND condition, and
When the setting button 361 is specified after specifying the threshold value of 2, the U / I unit 151 sets the reference image, the importance, the specification of the AND condition, and the threshold value in the folder 352 as the classification condition.

By performing such processing for each folder, a plurality of folders 352 as shown in FIG.
358 are set as classification items for classifying images to be classified. The folder information of each set folder is registered in the classification tree DB 302.

(2) Image Classification Processing The image classification processing of the eighteenth embodiment will be described with reference to FIG. When the user instructs to start the classification process,
The matching engine 153 in FIG.
For each 2 to 358, a similarity between each reference image and the image to be classified is obtained using a predetermined feature amount, and each obtained similarity is weighted according to importance, and each similarity is summed. (S160)
1).

[0425] The processing in step S1601 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360e in FIG. 42) set in the folder 352 and the classification target image by a predetermined feature amount. Is determined using That is, the matching engine 153
Calculates the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount using a distance definition formula of a vector such as a Euclidean distance. Then, the matching engine 153
Performs weighting on the obtained distance in accordance with the importance of the reference image A specified by the user. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on the predetermined feature amount, weights the obtained distance according to the importance of the reference image B, The distance between the image to be classified and the reference image C is obtained based on the predetermined feature amount, and the obtained distance is weighted according to the importance of the reference image C.

Thereafter, the matching engine 153
The distances between the image to be classified and each of the reference images A, B, and C are summed, and the sum of the distances is used as the overall similarity (see Equation 5). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Subsequently, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S1601 exceeds a threshold (S1).
602). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

[0428] Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S1603). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S1601-S16 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, the folders 352 to 358 of the image classification window 350e in FIG.
Displays a representative image as shown in FIG.

As described above, according to the image classification device of the eighteenth embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the reference image. It is possible to set the degree of importance, for each folder, determine the similarity of each reference image and the image of the classification target, by weighting the obtained similarity according to the set importance, Since the user's intention can be accurately expressed as a classification condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the image reflecting the user's intention is reflected. Can be obtained.

[Embodiment 19] The image classifying apparatus according to the nineteenth embodiment differs from the image classifying apparatus according to the seventeenth embodiment in that the image search processing described in the sixth embodiment is applied to enable the user to classify the image. It is intended to express the intention accurately. Note that, in the image classification device according to the nineteenth embodiment, for example, classification processing is performed using three types of feature amounts of color, structure, and texture.

The operation of the image classification device according to the nineteenth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Process FIG. 43 is an explanatory diagram showing an image classification window 350f used in the image classification device according to the nineteenth embodiment. Here, it is assumed that the user has created the folder 352, and displays a classification condition setting window 360f as shown in FIG. 43 to set the classification condition in the folder 352.

A classification condition setting window 36 shown in FIG.
0f is the classification condition setting window 360 shown in FIG.
In addition to d, for each of the reference images specified in the image specification columns A, B, and C, to specify the importance of each type of feature amount used when calculating the similarity with the image to be classified. The bar 367 is designated.

As shown in FIG. 43, the user
The images are designated as reference images A, B, and C in image designation columns A, B, and C, respectively, and the designated reference images A, B, and C are designated.
For each of B and C, the importance of each type of feature amount is designated.
As described in the sixth embodiment, the importance designation bar 367 indicates the importance of “+” and the importance of “−” centering on “0” for each type of feature amount of each reference image. It is possible to specify. In addition, only the "+" direction or only the "-" direction may be designated as the importance, not both the "+" direction and the "-" direction.

As described above, by specifying the importance of each type of feature for each reference image, it is possible to accurately express the user's intention for classification. In other words, the importance is reflected as a weight to the similarity obtained for each type of feature amount by the classification processing described later, so that it is possible to obtain an image of the classification result reflecting the user's intention.

[0437] Further, the user operates the AND designation button 36.
4 and specify the AND condition,
When the setting button 361 is selected after designating the threshold value of 52, the U / I unit 151 sets the specification of the reference image, the importance, the AND condition, and the threshold value in the folder 352 as the classification condition.

By performing such processing on each folder, a plurality of folders 352 as shown in FIG.
358 are set as classification items for classifying images to be classified. The folder information of each set folder is registered in the classification tree DB 302.

(2) Image Classification Processing Image classification processing according to the nineteenth embodiment will be described with reference to FIG. When the user designates the start of the classification process, the matching engine 153 of FIG. 34 calculates the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358. Weighting is performed on each similarity according to the importance, and the processing of obtaining the total similarity of the image to be classified by summing each similarity is executed (S1601).

[0440] The processing in step S1601 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information of the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360f in FIG. 43) set in the folder 352 and the classification target image by using the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the color feature specified by the user with respect to the reference image A. further,
The matching engine 153 obtains the distance between the image to be classified and the reference image A by using the feature amount of the structure and the feature amount of the texture. The user designates the distance to the reference image A for each obtained distance. Weighting is performed according to the importance of the structural feature and the importance of the texture feature.

The matching engine 153 also performs
The distance between the image to be classified and the reference image B is obtained for each of three types of feature amounts, and the obtained distances are weighted according to the importance of each of the feature amounts specified for the reference image B. .
Further, the distance between the image to be classified and the reference image C is obtained for each of three types of feature amounts, and the obtained distances are weighted according to the importance of each of the feature amounts specified for the reference image C. I do.

Subsequently, the matching engine 153
The distances between the classification target image obtained for each of the three types of feature amounts and each of the reference images A, B, and C are summed, and the total value of the distances is used as the overall similarity (see Equation 6). Matching engine 1
53 obtains the overall similarity for each of the folders 352 to 358 in this manner.

Next, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S1601 exceeds a threshold (S1).
602). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

[0444] Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S1603). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S1601-S16 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, the folders 352 to 358 of the image classification window 350f in FIG.
Displays a representative image as shown in FIG.

As described above, according to the image classification device of the nineteenth embodiment, when the similarity between the reference image and the image to be searched is obtained using a plurality of types of feature amounts, It is possible to set the importance for each type of feature amount for each reference image so that any weighting can be performed according to the type of the reference image and the feature amount. By calculating the similarity between each reference image and the image to be classified for each feature amount of, and by weighting the similarity obtained for each type of feature amount according to the set importance, the user's intention Can be accurately expressed as a classification condition, so that the user's intention can be accurately conveyed to the device, and the importance is reflected as a weight for the similarity, so that the image in which the user's intention is reflected It is possible to obtain a classification result.

Note that the types of feature amounts are not limited to the three types of colors, structures, and textures, and any combination of feature types may be used.

[Embodiment 20] The image classifying apparatus according to the twentieth embodiment differs from the image classifying apparatus according to the seventeenth embodiment in that the image search processing described in the seventh embodiment is applied so that the user The intention is to allow the device to pump the intention. It is assumed that the image classification device according to the twentieth embodiment performs a classification process using three types of feature amounts of color, structure, and texture, for example.

In Embodiments 18 and 19 described above, it is necessary for the user to specify the importance of the reference image or the importance of each feature amount in each reference image. In the method, the similarity between the reference images designated for each folder by the user is obtained using the feature amount used when obtaining the similarity between the reference image and the image to be classified, and the similarity degree for each obtained feature amount is obtained. The feature is that the degree of importance of each feature amount between the reference images is determined on the device side based on the reference image and reflected as a weight for the similarity.

The operation of the image classification device according to the twentieth embodiment will be described below in the order of (1) classification item setting processing and (2) image classification processing.

(1) Classification Item Setting Process The classification item setting process will be described with reference to FIG. The user has created the folder 352, and displays a classification condition setting window 360d as shown in FIG. 41 to set the classification condition in the folder 352.

As shown in FIG. 41, the user
The images are designated as reference images A, B, and C in image designation columns A, B, and C, respectively, and an AND designation button 36
Select 4 to specify the AND condition. Further, when the user specifies the threshold value of the folder 352 and selects the setting button 361, the feature extraction engine 152 determines the distance between the reference images A, B, and C for each of the three types of feature amounts of color, structure, and texture. (Similarity) is calculated, and the importance of each feature value among the reference images A, B, and C is determined based on the obtained distance of each feature value.

Specifically, the feature extraction engine 152
The distance between the features of the reference images is obtained for each type of feature amount. If the variance of the distance obtained for each feature amount is large, it means that the user does not know the feature amount, and if the variance is small, the user attaches importance to the feature amount as a classification criterion. Can be determined. Therefore, the importance is reduced when the feature amount has a large variance, and the importance is increased when the variance is small (see Equation 7).

The feature extraction engine 152 inputs the importance for each of the three types of feature amounts to the U / I unit 151, and
1 sets the reference image, the importance determined by the feature extraction engine 152, the designation of the AND condition, and the threshold value as the classification condition in the folder 352.

By performing such a process for each folder, a plurality of folders 352 as shown in FIG.
358 are set as classification items for classifying images to be classified. The folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing Next, the image classification processing according to the twentieth embodiment will be described with reference to FIG.
This will be described with reference to FIG. When the user designates the start of the classification process, the matching engine 153 of FIG. 34 calculates the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358. Weighting is performed on each similarity according to the importance, and the processing of obtaining the total similarity of the image to be classified by summing each similarity is executed (S1601).

[0457] The processing in step S1601 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image by the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Subsequently, the matching engine 153 weights the obtained distance according to the importance determined for the color feature by the feature extraction engine 152. Further, the matching engine 153 obtains a distance using the feature amount of the structure and the feature amount of the texture for the image to be classified and the reference image A, and for each of the obtained distances, obtains the distance of the structure determined by the feature extraction engine 152. Weighting is performed according to the importance of the feature and the importance of the texture.

The matching engine 153 also performs
The distance between the image to be classified and the reference image B is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the importance determined for each type of feature amount. Further, the distance between the image to be classified and the reference image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the importance determined for each of the types of feature amounts.

[0459] Thereafter, the matching engine 153
The distances between the classification target image obtained for each of the three types of feature amounts and each of the reference images A, B, and C are totaled, and the total value of the distances is used as the overall similarity. The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Next, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S1601 exceeds a threshold (S1).
602). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S1603). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S1601-S16 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity expressed in terms of distance) among the images classified into the folders 352 to 358,
The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, the folders 352 to 358 of the image classification window 350d in FIG.
Displays a representative image as shown in FIG.

As described above, according to the image classification apparatus of the twentieth embodiment, when the similarity between the reference image and the image to be searched is obtained using a plurality of types of feature amounts, For each classification item, the degree of similarity between the set reference images is obtained for each type of feature amount so that arbitrary weighting can be performed according to the type of feature amount, and for each type of feature amount, Based on the obtained similarity, the importance of each feature amount between the reference images is determined, and for each classification item, the similarity of each reference image and the image to be classified is obtained for each type of feature amount. By performing weighting according to the determined importance on the similarity obtained for each type of feature, the importance of each type of feature between the reference images specified by the user on the device side is determined, Determining the importance determined against the similarity between images In order to, it is possible to return the classification result of the image matching the intention of the user. Therefore,
The classification process desired by the user can be realized, and the classification result desired by the user can be obtained.

[0464] Note that the types of feature amounts are not limited to the three types of color, structure, and texture, and any combination of feature types may be used.

[Embodiment 21] The image classification device according to the embodiment 21 applies the OR search described in the embodiment 8 as one method for accurately expressing the user's intention as the classification condition. Image classification processing is performed.

The image classification process performed by the image classification device according to the twenty-first embodiment is the same as that described in the seventeenth embodiment with reference to FIG.
1 in the classification condition setting window 360d, AND
This is executed when the OR designation button 365 is selected instead of the designation button 364. Therefore, in the description of “(1) Classification item setting process” in the seventeenth embodiment, O
Assuming that the R designation button 365 has been selected,
The image classification processing based on the R condition will be described.

In addition to the specification of the OR condition, it is assumed that the reference images A, B, and C, and the threshold value are set as the classification conditions in the folder 352 shown in FIG. 41 as in the seventeenth embodiment. .

FIG. 44 is a flowchart showing the image classification processing according to the twenty-first embodiment. When the user instructs to start the classification process, the matching engine 153 in FIG.
Calculates the similarity of each reference image and the image to be classified using a predetermined feature amount for each of the folders 352 to 358, and selects the similarity indicating the highest similarity from the obtained similarities. A process for calculating the overall similarity of the images to be classified is executed (S2101).

[0469] The processing in step S2101 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image by a predetermined feature amount. Is determined using That is, the matching engine 153
Calculates the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount using a distance definition formula of a vector such as a Euclidean distance. Similarly, the matching engine 153 calculates the distance between the classification target image and the reference image B based on the predetermined feature amount, and also calculates the distance between the classification target image and the reference image C.

Then, the matching engine 153
The shortest distance is selected from the distance between the image to be classified and each of the reference images A, B, and C, and the overall similarity is obtained (see Expression 8). The matching engine 153 selects the similarity having the highest similarity for each of the folders 352 to 358 in this way, and obtains the overall similarity.

Subsequently, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the similarity selected in step S2101 exceeds a threshold (S21).
02). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

[0472] Then, the matching engine 153
The images to be classified are classified into folders in which the selected similarity exceeds the threshold (S2103). That is, matching
The engine 153 determines a folder whose similarity exceeds the threshold value as a classification destination of the image to be classified, and generates a classification tree D
The ID of the image to be classified (predetermined) is registered in the classification tree DB 302 in association with the folder information of the corresponding folder registered in B302.

Steps S2101 to S21 described above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects the highest similarity from among the images classified into the folders 352 to 358 (step S210).
1, the image having the smallest similarity expressed in distance) is selected as the representative image, and the image is selected in association with the corresponding folder information.
Register in 2. When the image classification processing is executed, FIG.
Folders 352-3 of the image classification window 350d
At 58, a representative image is displayed as shown in FIG.

In the image classification process according to the twenty-first embodiment, the classification target image is similar to any one of the reference images set in a certain folder so that a similarity greater than or equal to a threshold is obtained. If it is, it will be classified into that folder. Therefore, the image classification process of the twenty-first embodiment is
It can be seen that the R search is applied.

As described above, according to the image classification apparatus of the twenty-first embodiment, a folder is set, a plurality of reference images are set for each set folder, and each set reference image and classification are set for each folder. The similarity of the target image is obtained, and a similarity having the highest similarity is selected from the obtained similarities, and classification is performed based on the similarity selected for each folder and a preset classification destination determination criterion. Since the folder to which the target image is to be classified is determined, it is possible to perform an image classification process using an OR search process. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

Although detailed description is omitted,
When a new image is registered in the image DB 108, the image DB 10 is also subjected to a classification process so that the image DB 10
8 can always be kept in a classified state. For example, if the U / I unit 151 monitors the state of the image DB 108, the image DB 10
8 can always be kept in a classified state.

[Embodiment 22] The image classification apparatus according to the twenty-second embodiment differs from the image classification apparatus according to the twenty-first embodiment in that the image search processing described in the ninth embodiment is applied so that a user It is intended to express the intention accurately.

The image classification process performed by the image classification device according to the twenty-second embodiment is the same as that shown in FIG.
In the second classification condition setting window 360e, AND
This is executed when the OR designation button 365 is selected instead of the designation button 364. Therefore, in the description of “(1) classification item setting process” in the eighteenth embodiment,
Assuming that the R designation button 365 has been selected,
The image classification processing based on the R condition will be described.

In addition to the designation of the OR condition, in the folder 352 shown in FIG. 42, similarly to the eighteenth embodiment, the reference images A, B, and C, the importance of each reference image and the threshold are set as the classification conditions. It shall be set.

FIG. 45 is a flowchart showing an image classification process according to the twenty-second embodiment. When the user instructs to start the classification process, the matching engine 153 in FIG.
Calculates the similarity between each reference image and the image to be classified for each of the folders 352 to 358 using a predetermined feature amount,
Weighting is performed on each of the obtained similarities in accordance with the degree of importance, a similarity having the highest similarity is selected from among the similarities, and a process of obtaining the overall similarity of the image to be classified is performed ( S2201).

[0481] The processing in step S2201 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360e in FIG. 42) set in the folder 352 and the classification target image by a predetermined feature amount. Is determined using That is, the matching engine 153
Calculates the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount using a distance definition formula of a vector such as a Euclidean distance. Then, the matching engine 153
Performs weighting on the obtained distance in accordance with the importance of the reference image A specified by the user. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on the predetermined feature amount, weights the obtained distance according to the importance of the reference image B, The distance between the image to be classified and the reference image C is obtained based on the predetermined feature amount, and the obtained distance is weighted according to the importance of the reference image C.

Thereafter, the matching engine 153
The shortest distance is selected from the distances between the image to be classified and each of the reference images A, B, and C, and the overall similarity is obtained (see Expression 9). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

Subsequently, the matching engine 153
It is determined for each of the folders 352 to 358 whether or not the total similarity calculated in step S2201 exceeds a threshold (S2).
202). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold (S2203). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S2201-S22 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity in terms of distance) from the images classified into the folders 352 to 358, respectively. Classification tree D in association with the corresponding folder information
Register in B302. When the image classification process is performed,
Each folder 35 in the image classification window 350e of FIG.
In 2 to 358, representative images are displayed as shown in FIG.

As described above, according to the image classification apparatus of the twenty-second embodiment, the obtained similarity can be arbitrarily weighted in accordance with the type of the reference image. It is possible to set the degree of importance, for each folder, determine the similarity of each reference image and the image of the classification target, by weighting the obtained similarity according to the set importance, Since the user's intention can be accurately expressed as a classification condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the image reflecting the user's intention is reflected. Can be obtained.

[Twenty-third Embodiment] The image classification apparatus according to the twenty-third embodiment is the same as the image classification apparatus according to the twenty-first embodiment, except that the image search processing described in the tenth embodiment is applied. It is intended to express the intention accurately.

The image classification process performed by the image classification device according to the twenty-third embodiment is similar to that of the nineteenth embodiment shown in FIG.
3 in the classification condition setting window 360f, AND
This is executed when the OR designation button 365 is selected instead of the designation button 364. Therefore, in the description of “(1) classification item setting process” in the nineteenth embodiment,
Assuming that the R designation button 365 has been selected,
The image classification processing based on the R condition will be described.

In addition to the specification of the OR condition, in the folder 352 shown in FIG. 43, the reference images A, B, and C, and the respective reference images are specified for each type of feature amount similarly to the nineteenth embodiment. It is assumed that the importance and the threshold are set as the classification conditions.

Next, an image classification process according to the twenty-third embodiment will be described with reference to FIG. When the user designates the start of the image classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358, and obtains the similarity. A weighting process is performed on each similarity according to the importance, and a similarity having the highest similarity is selected from the similarities, and a process of obtaining the overall similarity of the image to be classified is performed (S2201). ).

The process in step S2201 will be specifically described by taking the folder 352 as an example. The matching engine 153 refers to the folder information of the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360f in FIG. 43) set in the folder 352 and the classification target image by using the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the color feature specified by the user with respect to the reference image A. further,
The matching engine 153 obtains the distance between the image to be classified and the reference image A using the feature amount of the structure and the feature amount of the texture, and for each of the obtained distances, the structure specified by the user for the reference image A. Are weighted according to the importance of the feature amount of the texture and the importance of the feature amount of the texture.

The matching engine 153 also performs
The distance between the image to be classified and the reference image B is obtained for each of three types of feature amounts, and the obtained distances are weighted according to the importance of each of the feature amounts specified for the reference image B. .
Further, the distance between the image to be classified and the reference image C is obtained for each of three types of feature amounts, and the obtained distances are weighted according to the importance of each of the feature amounts specified for the reference image C. I do.

Thereafter, the matching engine 153
The shortest distance is selected from the distances between the classification target image obtained for each of the three types of feature amounts and each of the reference images A, B, and C, and is set as the overall similarity (see Expression 10). The matching engine 153 thus operates the folders 352 to 35
The overall similarity is calculated for each of the eight.

Subsequently, the matching engine 153
Referring to the classification tree DB 302, folders 352 to 358
For each time, it is determined whether or not the overall similarity calculated in step S2201 exceeds a threshold (S2202). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

[0495] Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold value (S2203). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S2201-S22 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity in terms of distance) from the images classified into the folders 352 to 358, respectively. Classification tree D in association with the corresponding folder information
Register in B302. When the image classification process is performed,
Each folder 35 of the image classification window 350f of FIG.
In 2 to 358, representative images are displayed as shown in FIG.

As described above, according to the image classification device of the twenty-third embodiment, when the similarity between the reference image and the image to be searched is obtained using a plurality of types of feature amounts, It is possible to set the importance for each type of feature amount for each reference image so that any weighting can be performed according to the type of the reference image and the feature amount. By calculating the similarity between each reference image and the image to be classified for each feature amount of, and by weighting the similarity obtained for each type of feature amount according to the set importance, the user's intention Can be accurately expressed as a classification condition, so that the user's intention can be accurately conveyed to the device, and the importance is reflected as a weight for the similarity, so that the image in which the user's intention is reflected It is possible to obtain a classification result.

[Twenty-fourth Embodiment] The image classification device according to the twenty-fourth embodiment is different from the image classification device of the twenty-first embodiment in that the image search processing described in the eleventh embodiment is applied to allow the user The intention is to allow the device to pump the intention.

The image classification process performed by the image classification device according to the twenty-fourth embodiment is similar to that of the twentieth embodiment shown in FIG.
1 in the classification condition setting window 360d, AND
This is executed when the OR designation button 365 is selected instead of the designation button 364. Therefore, in the description of “(1) Classification item setting process” in the twentieth embodiment, O
Assuming that the R designation button 365 has been selected,
The image classification processing based on the R condition will be described.

In addition to the specification of the OR condition, in the folder 352 shown in FIG. 41, the feature extraction engine 152 determines each of the reference images A, B and C, and the type of the feature amount in the same manner as in the twentieth embodiment. It is assumed that the set importance and the threshold are set as the classification conditions.

Next, an image classification process according to the twenty-fourth embodiment will be described with reference to FIG. When the user designates the start of the image classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358, and obtains the similarity. A weighting process is performed on each similarity according to the importance, and a similarity having the highest similarity is selected from the similarities, and a process of obtaining the overall similarity of the image to be classified is performed (S2201). ).

[0502] The processing in step S2201 will be specifically described using the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302, and determines the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image by the color feature amount. Is determined using That is, the matching engine 153
The distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on the color feature amount is obtained using a vector distance definition formula such as a Euclidean distance. Subsequently, the matching engine 153 weights the obtained distance according to the importance determined for the color feature by the feature extraction engine 152. Further, the matching engine 153 obtains a distance using the feature amount of the structure and the feature amount of the texture for the image to be classified and the reference image A, and for each of the obtained distances, obtains the distance of the structure determined by the feature extraction engine 152. Weighting is performed according to the importance of the feature and the importance of the texture.

[0503] Similarly, the matching engine 153
The distance between the image to be classified and the reference image B is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the importance determined for each type of feature amount. Further, the distance between the image to be classified and the reference image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted in accordance with the importance determined for each of the types of feature amounts.

[0504] Thereafter, the matching engine 153
The shortest distance is selected from the distances between the classification target image obtained for each of the three types of feature amounts and each of the reference images A, B, and C, and the overall similarity is obtained. Matching engine 153
Calculates the total similarity for each of the folders 352 to 358 in this manner.

Subsequently, the matching engine 153
Referring to the classification tree DB 302, folders 352 to 358
For each time, it is determined whether or not the overall similarity calculated in step S2201 exceeds a threshold (S2202). However, here, the distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, in practice, whether the total similarity is equal to or less than (or less than) the threshold value It will be determined whether or not.

Then, the matching engine 153
The images to be classified are classified into folders in which the calculated overall similarity exceeds the threshold value (S2203). In other words, the matching engine 153 determines a folder in which the calculated overall similarity exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 to determine the classification target. An image ID (preset) is registered in the classification tree DB 302.

Steps S2201-S22 explained above
By executing the process 03 on each image registered in the image DB 108, the images in the image DB 108 can be classified. Further, the matching engine 153 selects, as a representative image, an image having the highest overall similarity (smallest overall similarity in terms of distance) from the images classified into the folders 352 to 358, respectively. Classification tree D in association with the corresponding folder information
Register in B302. When the image classification process is performed,
Each folder 35 in the image classification window 350d in FIG.
In 2 to 358, representative images are displayed as shown in FIG.

As described above, according to the image classification device of the twenty-fourth embodiment, when the similarity between the reference image and the image to be searched is obtained using a plurality of types of feature amounts, For each classification item, the degree of similarity between the set reference images is obtained for each type of feature amount so that arbitrary weighting can be performed according to the type of feature amount, and for each type of feature amount, Based on the obtained similarity, the importance of each feature amount between the reference images is determined, and for each classification item, the similarity of each reference image and the image to be classified is obtained for each type of feature amount. By performing weighting according to the determined importance on the similarity obtained for each type of feature, the importance of each type of feature between the reference images specified by the user on the device side is determined, Determining the importance determined against the similarity between images Order to the user can be realized from the image classification processing as desired, it is possible to obtain a classification result of the image desired by the user.

[Twenty-Fifth Embodiment] Next, a twenty-second embodiment will be described.
5 will be described. Embodiment 2
The image classification device according to 5 dynamically classifies images based on a reference desired by a user, and enables a user to search for a desired image.

The structure of the image classification device and the structure of the image classification software according to the twenty-fifth embodiment are shown in FIGS.
4, the description thereof is omitted here, and only the image classification processing will be described. The image to be classified is the image DB1 here.
08, and color, structure, and texture feature amounts are extracted from these images and registered in the image feature DB 109.

FIGS. 46 to 48 are explanatory diagrams of the image classification processing in the twenty-fifth embodiment. When the user selects “image classification processing” from a menu (not shown), U
/ I unit 151 is an image classification window 40 shown in FIG.
0 is displayed on the screen.

In the image classification window 400 shown in FIG. 46, as the initial state, the classification items 401 for all the images in the image DB 108 and the images registered in the image DB 108 are dynamically classified. Items 402a, 403a, and 404a representing the types of feature amounts used at the time
Are shown. When the user selects the classification items 402a and 403
By selecting one of a and 404a, the images in the image DB 108 are dynamically classified.

[0513] Here, the classification item 402a is an item used when classifying an image using the color feature amount, and the classification item 403a is used when classifying the image using the structural feature amount. The classification item 404a is an item for classifying an image using a feature amount of a texture.

In FIG. 46, the user enters a classification item 402
Assuming that a is specified, the U / I unit 151 notifies the matching engine 153 that the color feature has been specified. The matching engine 153 converts the image in the image DB 108 to the Near in the color feature space (see FIG. 4).
Clustering is performed using a general clustering method such as an est Neighbor method or a K-means algorithm (classification into groups of similar images), and a representative image is selected from each cluster. As for the representative image, any selection method may be used as long as the representative images are not similar, such as selecting the image at the center of the cluster as the representative image.

The U / I unit 151 performs matching
The clustering result is input from the engine 153, and the representative images 405 to 408 are displayed in association with the classification item 402a selected by the user, as shown in FIG. In FIG. 47, since four representative images are displayed, it is shown that the images in the image DB 108 are classified into four clusters based on the color feature amounts. That is, in the twenty-fifth embodiment, the cluster generated by designating the type of the feature amount is a substantial classification item. The user can display and view images belonging to each cluster other than the representative image as necessary.

FIG. 48 shows the result of further clustering and clustering the clusters of the representative image 407 in the color feature space. As described above, the user can reach the target image by designating the types of the feature amounts step by step.

[0517] The detailed description of the feature values other than the color is omitted, but even when the feature value of the structure or the texture is selected, the clustering is similarly performed in the feature space to classify the image. Will go on.

[0518] As described above, according to the image classification device of the twenty-fifth embodiment, when the type of the feature is specified, the similarity between the images to be classified is determined using the specified type of feature. Based on the obtained degree of similarity between the images to be classified, the images to be classified are classified into groups of similar images, and a representative image representing each group is selected from each of the classified groups. On the screen, when one of the displayed representative images is designated and the type of the feature amount is designated, each of the classes classified into the group to which the representative image belongs using the designated type of feature amount By classifying images into groups of similar images, selecting a representative image representing each group from each of the classified groups and displaying the selected image on the screen, the image can be classified according to a standard desired by the user when the image is to be classified. It becomes possible to continue to dynamically classify, using the classification of the image it is possible to find a desired image. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

Also, the type of the feature quantity is classified into the classification items 402 and 4.
03 and 404, and by displaying the classified representative images of each group on the screen in association with the specified classification items, as shown in FIG.
08 can be displayed on the screen in a tree structure, so that the user can recognize the state of the image classification processing at a glance,
The convenience of the dynamic image classification process can be improved.

[0520] The result of classifying images by the image classifying process of the twenty-fifth embodiment can be reused by registering it in the classification tree DB 302 as a classification history, for example.

[0521] The images to be classified are not limited to the images in the image DB 108.
A plurality of images obtained as a result of search by the image search devices of (1) to (13) can be used as images to be classified.

[Twenty-Sixth Embodiment] Next, a twenty-second embodiment will be described.
The sixth image classification device will be described. The image classification device according to the twenty-sixth embodiment is obtained by integrating some or all of the image classification devices described in the fourteenth to twenty-fifth embodiments into one image classification device. That is, Embodiments 14 to
As is clear from the description at 25, these devices can be implemented with a common configuration as shown in FIGS. 33 and 34, so that they can be easily integrated.

By integrating the image classification devices described in the fourteenth to twenty-fifth embodiments into one, classification conditions can be set by different methods for each folder. For example, in FIG. 35, the folder 352 stores the classification conditions described in the fourteenth embodiment, and the folder 353 stores the classification conditions described in the seventeenth embodiment.
Can be set in the folder 358, and the classification condition described in Embodiment 21 can be set in the folder 358.

[0524] The classification condition setting processing and the image classification processing are the same as those described in the respective embodiments, and thus description thereof is omitted here.

As described above, by integrating the image classification devices described in the fourteenth to twenty-fifth embodiments, classification conditions can be set from various viewpoints, and the degree of freedom in setting the classification conditions is increased. be able to.

For the same reason as in the case of the image classification device, the image search devices described in the first to thirteenth embodiments can be integrated as one device. This allows
It is possible to search for a desired image by combining different search methods.

[Embodiment 27] Further, Embodiment 1
It is also possible to integrate a part or all of the image search device and the image classification device described in 〜 to 25 as one device (the image search device and the matching engine 153 of the image classification device perform both the image search process and the image classification process). That it has the function of). That is, in the image search processing and the image classification processing described in the embodiment of the present invention, the similarity between images is obtained by using the feature amounts extracted from the images by the matching engine 153 shown in FIGS. Therefore, the image search device and the image classification device described in Embodiments 1 to 25 can be easily integrated into one device.

[0528] Although detailed description is omitted, the following processing can be performed by integrating the image search device and the image classification device described in Embodiments 1 to 25 into one device.

(1) An image selected from images classified into each folder (classification item) by any one of the image classification processes described in Embodiments 14 to 25 is designated as a query image, and Embodiment 1 is specified. Image search can be performed by any of the image search processes described in (1) to (13). (2) One of the image classification processes described in any one of Embodiments 14 to 25 (or 26) is set such that an image of a search result searched in any one of the image retrieval processes described in Embodiments 1 to 13 is to be classified. Can be classified. (3) A plurality of images classified into a specific folder (classification item) in any one of the image classification processes described in Embodiments 14 to 25 are searched, and any of the images described in Embodiments 1 to 13 is used. A desired image can be searched by the image search processing.

As described above, by integrating some or all of the image search device and the image classification device described in Embodiments 1 to 25 into one device, the image search process and the image classification process can be executed freely. It becomes possible.

[0531] As described above, the image processing apparatus and the image classification apparatus according to the present invention are realized by executing a prepared program on a computer in accordance with the procedure based on each flowchart. This program can be provided for each embodiment described above or in any combination.

This program is provided by being recorded on a computer-readable recording medium such as a hard disk, a floppy disk, a CD-ROM, an MO, and a DVD. Then, as shown in FIG. 49A, the program is transferred from the recording medium 450 to the CPU 101 (computer 1).
00), and the CPU 101 executes processing based on the program, thereby realizing the image search device and / or the image classification device according to the present invention.

This program can also be executed in the form of being downloaded from a recording medium of the server 451 via a network such as the Internet or a LAN, and read into the CPU 101, as shown in FIG. is there. Also, the program can be downloaded and executed by distributing the program not by a network but by a broadcast wave, for example.

Although not specifically mentioned in the above embodiments, the image search device and the image classification device of the present invention can be applied not only to still images but also to moving images. This is because a moving image is composed of continuous still images.

The image search processing described in the first to third embodiments can be said to use the same method as the AND search described in the fourth to seventh embodiments. The search windows 200a and 200b can be applied to the OR search described in
And 200c (FIGS. 5, 7 and 9)
The ND condition and the OR condition may be designated. Similarly, in the image classification device described in the fourteenth to sixteenth embodiments, an AND condition and an OR condition may be designated as one of the classification conditions.

The image search device described in the fourth to seventh embodiments, the image search device described in the eighth to eleventh embodiments,
In the image classification devices described in the seventeenth to twentieth embodiments and the image classification device described in the twenty-first to twenty-fourth embodiments, an AND condition or an OR condition is specified, respectively. Specification is not mandatory. In other words, the apparatus can be configured to execute the processing described in each embodiment without specifying an AND condition or an OR condition.

[0537] Although detailed description is omitted,
In the image classification apparatus described in the fourteenth to twenty-fourth embodiments, for example, the timing of executing the process of classifying images into the folder 354 shown in FIG. 35 is after the image is classified into the folder 353. This is because the folder 354 is for further classifying the images classified into the folder 353. In addition, it is possible to freely perform processing such as addition / deletion of a folder for classifying images and change of classification conditions set for the folder.

Although detailed description is omitted, for example, in a client-server system, the server is provided with the function of the image search device or the image classification device described in each embodiment, and the search conditions are set by the client. It is apparent that it is possible to specify the image or to display the classified image on the screen of the client.

[0539]

As described above, according to the image search apparatus of the present invention (claim 1), for each type of feature amount used when obtaining the similarity between the query image and the image to be searched. Each query image is designated, and for each image to be searched, the degree of similarity to each designated query image is obtained using the corresponding type of feature quantity, and the obtained degree of similarity to each query image is totaled. By calculating the overall similarity of the images to be searched and outputting the search results based on the overall similarity obtained for each image to be searched, the intention of the user can be accurately expressed as a search condition, so that Since the intention can be accurately transmitted to the device and the user's intention for the search can be reliably transmitted to the device, a search result that accurately reflects the user's intention for the search can be obtained. Accurate image retrieval processing as desired can be carried out. Specifically, for example, even when an appropriate image does not exist as an inquiry image, an image having a similar color atmosphere to an image desired as a search result or an image having an atmosphere similar to an object shape is designated as an inquiry image. Then, a combination of these images can be used as one query.

Also, the image search device of the present invention (Claim 2)
According to this, a plurality of query images are specified, and a type of a feature amount used when obtaining a similarity with the search target image is specified for each specified query image. The similarity with each query image is calculated using the feature amounts of the different types, and the similarity with each query image obtained is summed to obtain the overall similarity of the search target image. By outputting search results based on the total similarity, the user's intention can be accurately expressed as a search condition, so that the user's intention can be accurately conveyed to the device and the user's intention for the search can be reliably determined. Since the information can be transmitted to the apparatus, a search result that accurately reflects the user's intention for the search can be obtained, and a highly accurate image search process desired by the user can be performed. Specifically, for example, even when an appropriate image does not exist as an inquiry image, an image having a similar color atmosphere to an image desired as a search result or an image having an atmosphere similar to an object shape is designated as an inquiry image. Then, a combination of these images can be used as one query.

Also, the image search device of the present invention (Claim 3)
According to the method, a plurality of query images are designated, the similarity with each designated query image is obtained for each of the search target images, and the obtained similarity with each query image is summed to obtain the search target image. By obtaining the overall similarity and outputting the search result based on the overall similarity obtained for each image to be searched, it is possible to perform an AND search, so that the user's intention is accurately expressed as a search condition. One approach can be provided.

Also, the image search device of the present invention (Claim 4)
According to the method, a plurality of query images are designated, a similarity with each designated query image is obtained for each image to be searched, and the highest similarity among the obtained query images is shown. The OR search can be performed by selecting the similarity, obtaining the overall similarity of the search target image, and outputting the search result based on the total similarity obtained for each search target image. One method for accurately expressing the user's intention as a search condition can be provided.

Also, the image search device of the present invention (Claim 5)
According to the image retrieval apparatus according to the second, third or fourth aspect, an important weight is assigned to each query image so that the obtained similarity can be arbitrarily weighted according to the type of the query image. The degree of similarity can be specified, the similarity with each query image is obtained for each image to be searched, and the obtained similarity with each query image is weighted according to the specified importance. After calculating the overall similarity, the user's intention can be accurately expressed as a search condition, so that the user's intention can be accurately transmitted to the device, and the importance is reflected as a weight for the similarity. Therefore, it is possible to obtain a search result image in which the user's intention is reflected.

[0544] Further, the image retrieval device of the present invention (Claim 6)
According to the image retrieval apparatus according to claim 3 or 4, when the similarity between the query image and the image to be searched is obtained using a plurality of types of feature amounts, the query image and the search image are compared with the obtained similarity. It is possible to specify the importance of each type of feature amount for each query image so that any weight can be assigned according to the type of feature amount. After calculating the similarity with each query image for each feature amount, weighting the similarity with each query image obtained for each type of feature amount according to the specified importance, and then performing overall similarity By calculating the degree, the user's intention can be accurately expressed as a search condition, so that the user's intention can be accurately conveyed to the apparatus. In addition, the importance is reflected as a weight to the similarity, and thus the user's intention is reflected. The intent of The becomes possible to obtain an image of the search results are reflected.

[0545] Further, the image retrieval apparatus of the present invention (Claim 7)
According to the third aspect of the present invention, in a case where the similarity between the query image and the image to be searched is obtained by using a plurality of types of feature amounts, the feature amount of the feature amount is compared with the obtained similarity. The similarity between the specified query images is calculated for each type of feature amount so that arbitrary weighting can be performed according to the type, and based on the obtained similarity degree for each type of feature amount. Determine the importance of each type of feature value between query images, determine the similarity with each query image for each type of feature value for each image to be searched, and determine the similarity for each type of feature value. After weighting the similarity with each query image according to the degree of importance for each type of determined feature amount, the overall similarity is obtained, so that a query between the query images specified by the user on the device side is performed. Determine the importance of each type of feature To reflect the importance of determining the similarity between images, it is possible to return search results that match the user's intention. Therefore, search processing desired by the user can be realized, and search results desired by the user can be obtained.

[0546] Further, the image retrieval apparatus of the present invention (claim 8)
According to the method, at least one query image is specified, and the feature level used for the search is determined based on the extraction criterion in which the roughness level of the feature quantity extracted from the image is defined stepwise from coarse to fine. At least one roughness level is specified, and a feature amount corresponding to at least the specified roughness level is extracted at a predetermined timing from the query image and the plurality of search target images based on the extraction criterion. For each of the images, a similarity with the query image is obtained using a feature amount corresponding to the specified roughness level, and a search result is output based on the similarity obtained for each image to be searched. It is possible to execute a plurality of types of search processing with different precisions. That is, since the user can specify at least one level of roughness of the feature amount used for the search, the fineness of the user's viewpoint for the search can be expressed as the search condition, and the user's intention can be accurately determined as the search condition. Can be expressed. Therefore, it is possible to perform an image search process as desired by the user.

[0547] Further, the image retrieval apparatus of the present invention (claim 9)
According to the image retrieval apparatus according to the eighth aspect, the extraction criterion defines the number of divisions indicating at least how many images are divided in steps as a roughness level, and is defined stepwise. When extracting a feature value according to one of the roughness levels, the image is divided based on the corresponding number of divisions to generate a plurality of divided images, and the feature value is extracted from each of the generated divided images. By extracting a feature amount group composed of feature amounts in divided image units from the query image and the search target image, and using the feature amount group to determine the similarity between the query image and the search target image, the query image The similarity between the query image and the search target image is obtained by calculating the similarity of each of the corresponding divided images of the search target image using the corresponding feature amount. In the level of the feature amount of roughness can be easily defined, it is possible to introduce the concept that the position relationship between the image features in the image search by determining the similarity for each divided image. Therefore, it is possible to execute highly accurate image search processing.

Further, the image search device of the present invention (Claim 1)
According to 0), the extraction criterion sets the level at which the entire image is to be extracted as a feature amount without dividing the image, as a level indicating the coarsest feature amount among the roughness levels defined stepwise. When extracting feature values according to the level at which feature values are extracted for the entire image, ignore the positional relationship between image features because the feature values are extracted for the entire image from the query image and the image to be searched. It is possible to include a search process at a simple level.

Also, the image search device of the present invention (Claim 1)
According to 1), in the image retrieval device according to any one of claims 8, 9 and 10, when the extraction criterion is to extract an object existing in an image as a feature amount, the extraction is performed from the object. Since the definition of the level of roughness of the feature quantity to be included is included, it is possible to specify the search condition from a further different viewpoint, and the intention of the user can be accurately expressed as the search condition.

[0550] Further, the image retrieval apparatus of the present invention (claim 1)
According to 2), in the image search device according to any one of claims 8 to 11, for a plurality of images to be searched, feature amounts corresponding to all roughness levels defined by extraction criteria are set. Extracted in advance, and among the specified query images, for a query image from which no feature amount has been extracted, extract a feature amount corresponding to the specified roughness level before executing the process of obtaining similarity. As a result, the speed of the image search process can be increased.

The image retrieval apparatus of the present invention (Claim 1)
According to 3), at least one query image is specified, and for each search target image, a predetermined feature in which a similarity between a region of a predetermined size in the search target image and the specified query image is extracted in advance. In order to output the search result based on the similarity obtained for each image to be searched by repeatedly executing the process using the amount over the entire image to be searched, the image including the query image is selected from among the images to be searched. Can be found. Therefore, it is possible to provide one technique for realizing an image search intended by the user.

Further, the image search device of the present invention (claim 1)
According to 4), in the image search device according to any one of claims 1 to 13, the image of the search result can be designated as the query image, so that a more detailed refined search can be performed, and the user desires It is possible to realize the same search processing.

According to a computer-readable recording medium of the present invention (claim 15), a program for causing a computer to function as each means of the image search device according to any one of claims 1 to 14 By executing this program on a computer, it is possible to reliably convey the user's viewpoint to the device when specifying search conditions, and to return search results that match the user's viewpoint. Can be realized. That is, it is possible to realize an image search device that can realize search processing as desired by the user.

The image classification device according to the present invention (Claim 1)
According to 6), classification items are set, and for each of the set classification items, one reference image is provided for each type of feature amount used when calculating the similarity between the reference image and the image to be classified. For each classification item, the similarity between each of the set reference images and the image to be classified is obtained by using the feature amount of the corresponding type, and the obtained similarities are totaled to obtain the overall similarity of the image to be classified. By determining the classification item to which the classification target image is to be classified based on the overall similarity of the classification target image obtained for each classification item and a preset classification destination determination criterion. Since the intention can be accurately expressed as the classification condition, the user's intention can be accurately transmitted to the device, and the user's intention for the classification can be reliably transmitted to the device. Figure can be obtained classification results that accurately reflects, it is possible to realize the classification process as the user desires. In other words, even when there is no suitable image as the reference image, for example, an image having a similar color atmosphere to an image desired as a classification result or an image having a similar object shape atmosphere is designated as a reference image. Since the combined image is used as one reference image, the user's viewpoint can be reliably transmitted to the device when specifying the classification condition, and the classification process is performed using such a reference image. Therefore, it is possible to appropriately classify the images into the classification destination desired by the user.

[0555] The image classification device of the present invention (claim 1)
According to 7), the classification items are set, a plurality of reference images are set for each of the set classification items, and the feature amount used when obtaining the similarity with the image to be classified for each of the set reference images is set. Is set, and for each classification item, the similarity between each reference image and the image to be classified is obtained using the feature amount of the set type, and the obtained similarities are summed to obtain a total of the images to be classified. The similarity is obtained, and the classification item to which the classification target image is classified is determined based on the total similarity of the classification target image obtained for each classification item and a preset classification destination determination criterion. Since the intention of the user can be accurately expressed as a classification condition, the user's intention can be accurately transmitted to the device, and the user's intention of the classification can be reliably transmitted to the device. Classification result reflecting can be obtained, it is possible to realize the classification process as the user desires. In other words, even when there is no suitable image as the reference image, for example, an image having a similar color atmosphere to an image desired as a classification result or an image having a similar object shape atmosphere is designated as a reference image. Since the combined image is used as one reference image, the user's viewpoint can be reliably transmitted to the device when specifying the classification condition, and the classification process is performed using such a reference image. Therefore, it is possible to appropriately classify the images into the classification destination desired by the user.

Further, the image classification device of the present invention (claim 1)
According to 8), a classification item is set, a plurality of reference images are set for each set classification item, and the similarity between the set reference image and the image to be classified is calculated for each classification item. The similarity is summed to determine the overall similarity of the image to be classified, and based on the overall similarity of the image to be classified determined for each classification item and a preset classification destination determination criterion,
In order to determine the classification item to which the image to be classified is to be classified, it is possible to perform an image classification process using an AND search process. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

The image classification device of the present invention (claim 1)
According to 9), a classification item is set, a plurality of reference images are set for each set classification item, and the similarity between each set reference image and the image to be classified is calculated for each classification item. The similarity having the highest similarity is selected from the similarities obtained, and the overall similarity of the image to be classified is obtained, and the overall similarity of the image to be classified obtained for each classification item and a preset classification destination Since the classification item to which the image to be classified is classified is determined based on the determination criterion, it is possible to perform the image classification processing applying the OR search processing. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

The image classification device according to the present invention (Claim 2)
According to (0), in the image classification device according to claim 17, 18 or 19, each of the reference images is assigned such that the obtained similarity can be arbitrarily weighted according to the type of the reference image. It is possible to set the degree of importance, and for each classification item, the similarity between each reference image and the image to be classified is obtained, and the obtained similarity is weighted according to the set importance. Thereafter, by obtaining the overall similarity, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the device, and the importance is reflected as a weight for the similarity. Therefore, it is possible to obtain an image classification result reflecting the user's intention.

Also, the image classification device of the present invention (Claim 2)
According to 1), in the image classification device according to claim 18 or 19, when the similarity between the reference image and the image to be classified is determined using a plurality of types of feature amounts, the reference similarity is determined based on the determined similarity. It is possible to set the importance for each type of feature amount for each reference image so that any weight can be given according to the type of image and feature amount,
For each classification item, the similarity between each reference image and the image to be classified is obtained for each type of feature amount, and the similarity obtained for each type of feature amount is weighted according to the set importance. After that, by calculating the overall similarity, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately conveyed to the device, and the importance is reflected as a weight for the similarity. Therefore, it is possible to obtain an image classification result reflecting the user's intention.

The image classification device of the present invention (claim 2)
According to 2), in the image classification device according to claim 18 or 19, when the similarity between the reference image and the image to be classified is further determined using a plurality of types of feature values, The similarity between each set reference image is determined for each type of feature amount for each classification item so that arbitrary weighting can be performed according to the type of feature amount, and for each type of feature amount. Determine the importance of each feature amount between each reference image based on the similarity obtained in, and, for each classification item, obtain the similarity of each reference image and the image to be classified for each type of feature amount, After weighting the similarity obtained for each type of feature amount in accordance with the determined importance, and obtaining the overall similarity, each type of reference image specified by the user on the device side is obtained. Judgment of importance of features and judgment To reflect the importance of the similarity between images, it is possible to return the classification result of the image matching the intention of the user. Therefore, the classification processing desired by the user can be realized, and the classification result desired by the user can be obtained.

Also, the image classification device of the present invention (Claim 2)
According to 3), in the image classification device according to any one of claims 16 to 22, a classification item for further classifying an image classified for each classification item can be set, so that detailed image classification is performed. Processing can be executed.

The image classification device of the present invention (claim 2)
According to 4), when the type of the feature amount is designated, the similarity between the images of the classification target is calculated using the feature amount of the designated type, and the similarity between the images of the classification target is calculated. On the basis of,
The images to be classified are classified into groups of similar images, a representative image representing each group is selected from each of the classified groups, displayed on a screen, and one of the displayed representative images is designated. At the same time, when the type of the feature amount is designated, each image classified into the group to which the representative image belongs is classified into a group of similar images using the feature amount of the designated type, and a group is classified from each of the classified groups. By selecting a representative image representing the image and displaying it on the screen, it becomes possible for the user to dynamically classify the image based on the criteria desired by the user when classifying the image,
It is possible to search for a desired image using the image classification. Therefore, it is possible to provide an image classification device capable of classifying images by a new classification method.

Further, the image classification device of the present invention (Claim 2)
According to the fifth aspect, in the image classification device according to the twenty-fourth aspect, a type of a characteristic amount that can be specified is displayed on a screen, and a representative image of each group is displayed on a screen in association with the type of the specified characteristic amount. Since the result of the classification of the plurality of target images is displayed on the screen in a tree structure, the user can recognize the state of the image classification processing at a glance, and the convenience of the dynamic image classification processing can be improved.

[0564] The image classification device of the present invention (claim 2)
According to 6), in the image classification device according to any one of claims 16 to 25, an arbitrary image classified for each classification item is used as a query image, and the same or similar to the query image. Since the image is searched, the convenience of the device can be improved.

The image classification device of the present invention (Claim 2)
According to 7), in the image classification device according to claim 26, a search is performed using the image search device according to any one of claims 1 to 14. The viewpoint can be reliably transmitted to the device, and a search result that matches the user's viewpoint can be returned, so that the convenience of the device can be improved.

Also, the image classification device of the present invention (Claim 2)
According to 8), in the image classification device according to any one of claims 16 to 25, since the image to be classified is a search result image, for example, image classification is applied to narrow down the search result. It becomes possible.

Further, according to the computer-readable recording medium of the present invention (claim 29),
8. Since a program for causing a computer to function as each means of the image classification device according to any one of 8 is recorded, by executing this program, the user's viewpoint can be surely provided to the device when specifying classification conditions. It is possible to realize an image classifying device that can convey information and classify images into classification items that match the user's viewpoint. That is, it is possible to realize an image classification device that can appropriately classify images into classification destinations desired by the user.

[Brief description of the drawings]

FIG. 1 is a block configuration diagram of an image search device according to Embodiment 1 of the present invention.

FIG. 2 is a block diagram showing processing executed in the image search device according to Embodiment 1 of the present invention.

FIG. 3 is a flowchart illustrating an outline of an image registration process executed in the image search device according to the first embodiment of the present invention.

FIG. 4 is an explanatory diagram showing a state in which each image is mapped to a feature space according to a feature amount extracted from the image in the image search device according to the present invention.

FIG. 5 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the first embodiment of the present invention.

FIG. 6 is a flowchart illustrating an image search process executed in the image search device according to the first embodiment of the present invention.

FIG. 7 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the second embodiment of the present invention.

FIG. 8 is a flowchart showing an image search process executed in the image search device according to the second embodiment of the present invention.

FIG. 9 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the third embodiment of the present invention.

FIG. 10 is a flowchart showing an image search process executed in the image search device according to Embodiment 3 of the present invention.

FIG. 11 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the fourth embodiment of the present invention.

FIG. 12 is a flowchart showing an image search process executed in the image search device according to Embodiment 4 of the present invention.

FIG. 13 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the fifth embodiment of the present invention.

FIG. 14 is a flowchart showing an image search process executed in the image search device according to Embodiment 5 of the present invention.

FIG. 15 is an explanatory diagram showing a search window used at the time of a search in the image search device according to the sixth embodiment of the present invention.

FIG. 16 is a flowchart showing an image search process executed in the image search device according to Embodiment 6 of the present invention.

FIG. 17 is a flowchart showing an image search process executed in the image search device according to the seventh embodiment of the present invention.

FIG. 18 is a flowchart showing an image search process executed in the image search device according to the eighth embodiment of the present invention.

FIG. 19 is a flowchart showing an image search process executed in the image search device according to Embodiment 9 of the present invention.

FIG. 20 is a flowchart showing an image search process executed in the image search device according to Embodiment 10 of the present invention.

FIG. 21 is a flowchart showing an image search process executed in the image search device according to Embodiment 11 of the present invention.

FIG. 22 is an explanatory diagram illustrating a level of roughness of a feature extracted from an inquiry image and an image to be searched in the image search device according to Embodiment 12 of the present invention;

FIG. 23 is an explanatory diagram illustrating a feature amount extraction process (extraction of edge features) in the image search device according to the twelfth embodiment of the present invention.

FIG. 24 is an explanatory diagram illustrating a feature amount extraction process (edge feature extraction) in the image search device according to the twelfth embodiment of the present invention.

FIG. 25 is an explanatory diagram showing a search window used at the time of search in the image search device according to Embodiment 12 of the present invention.

FIG. 26 is a flowchart showing an image search process executed in the image search device according to Embodiment 12 of the present invention.

FIG. 27 is an explanatory diagram showing a first modification of the image search process executed in the image search device according to the twelfth embodiment of the present invention.

FIG. 28 is an explanatory diagram showing a second modification of the image search process executed in the image search device according to the twelfth embodiment of the present invention.

FIG. 29 is an explanatory diagram for describing an outline of an image search process executed in the image search device according to Embodiment 13 of the present invention;

FIG. 30 is an explanatory diagram for describing an outline of an image search process executed in the image search device according to the thirteenth embodiment of the present invention.

FIG. 31 is a flowchart showing an image search process executed in the image search device according to Embodiment 13 of the present invention.

FIG. 32 is an explanatory diagram showing a search window used at the time of search in the image search device according to Embodiment 13 of the present invention.

FIG. 33 is a block diagram of an image classification device according to Embodiment 14 of the present invention.

FIG. 34 is a block diagram showing processing executed in the image classification device according to Embodiment 14 of the present invention.

FIG. 35 is an explanatory diagram showing an example of an image classification window for performing an image classification item setting process and displaying images after classification in the image classification device according to Embodiment 14 of the present invention;

FIG. 36 is a flowchart showing an image classification process executed in the image classification device according to Embodiment 14 of the present invention.

FIG. 37 is an explanatory diagram illustrating a state of an image classification window after images are classified in the image classification device according to Embodiment 14 of the present invention;

FIG. 38 is an explanatory diagram showing an example of an image classification window for performing an image classification item setting process and displaying images after classification in the image classification device according to Embodiment 15 of the present invention;

FIG. 39 is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and display of images after classification in the image classification device according to Embodiment 16 of the present invention;

FIG. 40 is a flowchart showing an image classification process executed in the image classification device according to Embodiment 16 of the present invention.

FIG. 41 is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and displaying classified images in the image classification device according to Embodiment 17 of the present invention;

FIG. 42 is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and classified image display in the image classification device according to Embodiment 18 of the present invention;

FIG. 43 is an explanatory diagram showing an example of an image classification window for performing an image classification item setting process and displaying images after classification in the image classification device according to Embodiment 19 of the present invention;

FIG. 44 is a flowchart showing an image classification process executed in the image classification device according to Embodiment 21 of the present invention.

FIG. 45 is a flowchart showing an image classification process executed in the image classification device according to Embodiment 22 of the present invention.

FIG. 46 is an explanatory diagram showing an image classification process executed in the image classification device according to Embodiment 25 of the present invention.

FIG. 47 is an explanatory diagram showing an image classification process executed in the image classification device according to Embodiment 25 of the present invention.

FIG. 48 is an explanatory diagram showing an image classification process executed in the image classification device according to Embodiment 25 of the present invention.

FIG. 49 is an explanatory diagram showing a recording medium on which a program for realizing the image search device and / or the image classification device according to the present invention is recorded, and a distribution form of the program.

[Explanation of symbols]

REFERENCE SIGNS LIST 100 computer 107 image search software 108 image DB 109 image feature DB 151 U / I section 152 feature extraction engine 153 matching engine 200a to 200h search window 201 search button 202 image display column 203,362 feature amount designation button 204,207,208 , 363, 366, 367
Importance designation bar 205, 364 AND designation button 206, 365 OR designation button 209 Feature designation column 210 Roughness level designation column 301 Image classification software 302 Classification tree DB 350a-350f Image classification window 351-358 Folder 360a-360f Classification condition setting Window 361 setting button 400 image classification window 401, 402a to 404a, 402b to 404b
Classification item 405-412 Representative image 450 Recording medium 451 Server

Claims (29)

[Claims] 1. A method for extracting at least one type of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one type of feature quantity to determine a similarity between the query image and the search target image. In an image search device that searches for images that are the same as or similar to the query image from the plurality of search target images by determining the degrees of search, each of the plurality of images to be used when calculating the similarity between the query image and the search target image A designating unit that designates one query image for each type of feature amount, and a type of feature amount corresponding to the degree of similarity with each query image designated by the designating unit for each image to be searched. A search means for obtaining the total similarity of the images to be searched by summing up the similarities with each of the obtained query images, and for each of the images to be searched by the search means An output means for outputting a search result based on the obtained overall similarity, and an image search device. 2. A method for extracting at least one kind of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one kind of feature quantity to obtain a similarity between the query image and the search target image. An image search device that searches for the same or similar image as the query image from the plurality of search target images by calculating degrees, the query target is specified for each of the specified query images, and A designating unit for designating a type of a feature amount to be used when obtaining a degree of similarity with the image of the image, and for each image to be searched, a feature amount of the type designated by the designating unit is used for each query image. Searching means for obtaining the similarity and summing up the obtained similarities with the query images to obtain an overall similarity of the image to be searched; Output means for outputting a search result based on the overall similarity obtained for each image. 3. Extracting at least one kind of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one kind of feature quantity, resembling the query image and the search target image. In an image search device that searches for the same or similar image as the query image from the plurality of search target images by obtaining degrees, a specifying unit that specifies a plurality of query images, for each of the search target images, A search unit that calculates the similarity with each of the query images specified by the specifying unit, sums up the obtained similarities with each of the query images, and obtains the overall similarity of the search target image; An output unit that outputs a search result based on the overall similarity obtained for each of the images to be searched. 4. A method for extracting at least one kind of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one kind of feature quantity to obtain a similarity between the query image and the search target image. In an image search device that searches for the same or similar image as the query image from the plurality of search target images by obtaining degrees, a specifying unit that specifies a plurality of query images, for each of the search target images, The similarity to each query image specified by the specifying means is obtained, and the similarity indicating the highest similarity is selected from the obtained similarities to each query image to determine the overall similarity of the search target image. Search means for obtaining a degree, and output means for outputting a search result based on the overall similarity obtained for each of the images to be searched by the search means. Image search apparatus. 5. The designating means designates a degree of importance for each query image so that the similarity obtained by the search means can be arbitrarily weighted according to the type of the query image. The search means obtains a similarity with each of the query images for each of the search target images, and determines the similarity with each of the obtained query images by the importance specified by the specification means. 5. The image search device according to claim 2, wherein the overall similarity is obtained after performing weighting according to the degree. 6. When the similarity between the query image and the image to be searched for is determined using a plurality of types of feature amounts, the specifying unit compares the query image and the feature amount with the similarity determined by the search unit. It is possible to specify the importance for each type of feature amount for each query image so that any weighting can be performed in accordance with the type of the search target image. For each of the types of feature amounts, a similarity with each of the query images is obtained, and the similarity with each of the query images obtained for each of the types of feature amounts is assigned a degree of importance specified by the specifying unit. The image retrieval apparatus according to claim 3, wherein the total similarity is obtained after performing weighting according to the following. 7. When the similarity between the query image and the image to be searched is determined using a plurality of types of feature amounts, the similarity determined by the search means may be arbitrarily determined according to the type of the feature amount. So that the similarity between the query images specified by the specifying means is obtained for each type of feature amount, and the similarity between each type of obtained feature amount is determined based on the obtained similarity degree for each type of feature amount. The image processing apparatus further includes an importance determination unit that determines importance of each type of feature amount between query images, wherein the search unit determines, for each of the search target images, a similarity to each query image for each type of feature amount. After determining the degree, the similarity with each query image obtained for each type of feature amount is weighted according to the importance level for each type of feature amount determined by the importance level determination unit, Calculating the overall similarity Image retrieval apparatus according to claim 3 or 4, symptom. 8. A method for extracting at least one kind of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one kind of feature quantity to obtain a similarity between the query image and the search target image. An image search device that searches the plurality of search target images for an image that is the same as or similar to the query image, by specifying at least one of the query images and a rough feature amount to be extracted from the image. Specifying means for specifying at least one level of roughness of a feature used in a search, based on an extraction criterion that defines a level of roughness stepwise from coarse to fine, and based on the extraction criterion, A feature amount corresponding to at least the roughness level specified by the specifying unit is extracted at a predetermined timing from the query image and the plurality of search target images. Symbol extraction means, for each of the images to be searched, search means for obtaining a degree of similarity with the query image using a feature amount corresponding to a roughness level specified by the specification means, Output means for outputting a search result based on the degree of similarity obtained for each image to be searched. 9. The extraction criterion defines stepwise the number of divisions indicating at least how many parts of an image are to be divided as the level of roughness, and the feature extracting means defines the stepwise defined roughness. When extracting the feature amount according to any one of the following levels, the image is divided based on the corresponding number of divisions to generate a plurality of divided images, and the feature amounts are respectively extracted from the generated divided images. By extracting, a feature group consisting of feature amounts in divided image units is extracted from the query image and the search target image, and the search means uses the feature amount group to extract the query image and the search target image. When the similarity of the query image and the search target image is obtained, the similarity of each of the corresponding divided images of the query image and the search target image is obtained using the corresponding feature amount. Image retrieval apparatus according to claim 8, characterized in that to determine the degree of similarity. 10. The extraction criterion is a level indicating the coarsest feature amount among the stepwise defined roughness levels, and is a level at which the entire image is extracted as a feature amount without dividing the image. The feature extraction means extracts the feature amount from the query image and the image to be searched for the entire image when extracting the feature amount according to the level at which the entire image is to be extracted. The image search device according to claim 9, wherein: 11. The method according to claim 1, wherein the extraction criterion includes a definition of a roughness level of the feature extracted from the object when the object existing in the image is a feature extraction target. The image search device according to any one of 8, 9, and 10. 12. The feature extracting unit extracts feature amounts corresponding to all roughness levels defined by the extraction criterion for the plurality of search target images in advance, and designates the feature amounts by the designation unit. Of the query images obtained, for a query image from which the feature value has not been extracted, a feature value corresponding to the roughness level specified by the specifying device before the search unit executes the process of obtaining the similarity. The image retrieval apparatus according to any one of claims 8 to 11, wherein the image retrieval is performed. 13. A method for extracting at least one type of feature quantity representing a feature of an image from a query image and a plurality of search target images, and using the extracted at least one type of feature quantity to obtain a similarity between the query image and the search target image. An image search device that searches for the same or similar image as the query image from the plurality of search target images by determining the degree; a specifying unit that specifies at least one of the query images; A process of obtaining a similarity between a region of a predetermined size in the image to be searched and the query image specified by the specifying unit by using a predetermined feature amount extracted in advance is repeated over the entire image to be searched. A search unit to be executed, and an output unit that outputs a search result based on the similarity obtained for each image to be searched by the search unit. Image retrieval apparatus characterized by. 14. The image search device according to claim 1, wherein the specifying unit can specify an image of a search result as the query image. 15. A computer-readable recording medium on which is recorded a program for causing a computer to function as each means of the image search device according to claim 1. Description: 16. A reference image serving as a classification reference of the classification target image is set for each classification item to which the classification target image is to be classified, and the reference image set for each of the classification items and the classification target The similarity between the reference image and the image to be classified is obtained for each of the classification items by using at least one type of feature amount representing the characteristics of the image extracted from the image, and the classification is performed based on the obtained similarity. An image classification device for classifying a target image into one of the classification items, wherein the classification item is set, and for each set classification item, the similarity between the reference image and the classification target image is obtained. Setting means for setting one reference image for each type of feature amount to be used; and similarity of each reference image and the image to be classified set by the setting means for each of the classification items. Calculating means for calculating the overall similarity of the image to be classified by summing up the obtained similarities and the classification target obtained for each of the classification items by the calculating means. Determining means for determining a classification item to which the image to be classified is to be classified based on the overall similarity of the images and a preset classification destination determination criterion. 17. A reference image serving as a classification reference of the classification target image is set for each classification item to which the classification target image is to be classified, and the reference image set for each of the classification items and the classification target The similarity between the reference image and the image to be classified is obtained for each of the classification items by using at least one type of feature amount representing the characteristics of the image extracted from the image, and the classification is performed based on the obtained similarity. An image classification device that classifies a target image into any of the classification items, wherein the classification items are set, and for each set classification item, a plurality of the reference images are set, and for each of the set reference images. Setting means for setting the type of feature amount used when obtaining the degree of similarity with the image to be classified; and, for each of the classification items, the reference image and the reference image using the type of feature amount set by the setting means. Calculating means for calculating the similarity of the images to be classified and calculating the total similarity of the images to be classified by summing the obtained similarities; and calculating the classification target for each of the classification items by the calculating means. Determining means for determining a classification item to which the image to be classified is to be classified based on the overall similarity of the images and a preset classification destination determination criterion. 18. A reference image serving as a classification reference of the classification target image is set for each classification item to which the classification target image is classified, and the reference image set for each of the classification items and the classification target The similarity between the reference image and the image to be classified is obtained for each of the classification items by using at least one type of feature amount representing the characteristics of the image extracted from the image, and the classification is performed based on the obtained similarity. An image classification device for classifying a target image into any of the classification items, wherein the classification item is set, and for each set classification item, setting means for setting a plurality of the reference images; Calculating the similarity between each reference image set by the setting means and the image to be classified,
Calculating means for summing up the obtained similarities to obtain an overall similarity of the images to be classified; and a total similarity of the images to be classified obtained for each of the classification items by the arithmetic means and a preset classification destination. Determining means for determining a classification item to which the image to be classified is to be classified based on a determination criterion. 19. A reference image serving as a classification reference of the classification target image is set for each classification item to which the classification target image is classified, and the reference image set for each of the classification items and the classification target The similarity between the reference image and the image to be classified is obtained for each of the classification items by using at least one type of feature amount representing the characteristics of the image extracted from the image, and the classification is performed based on the obtained similarity. An image classification device for classifying a target image into any of the classification items, wherein the classification item is set, and for each set classification item, setting means for setting a plurality of the reference images; Calculating the similarity between each reference image set by the setting means and the image to be classified,
Calculating means for selecting a similarity having the highest similarity from the obtained similarities to determine the overall similarity of the classification target image; and the classification target image obtained for each of the classification items by the calculation means. Determining means for determining a classification item to which the image to be classified is to be classified, based on the overall similarity of the image and a preset classification destination determination criterion. 20. The setting means sets the importance for each of the reference images so that the similarity obtained by the calculation means can be arbitrarily weighted according to the type of the reference image. It is possible that, for each of the classification items, the calculating means calculates the similarity between each of the reference images and the image to be classified, and calculates the similarity according to the importance set by the setting means with respect to the obtained similarity. 20. The image classification device according to claim 17, wherein the total similarity is obtained after performing the weighting. 21. The method according to claim 19, wherein the setting unit calculates the similarity between the reference image and the image to be classified using a plurality of types of feature amounts. It is possible to set the degree of importance for each type of feature amount for each reference image so that any weighting can be performed according to the type of The similarity between the reference image and the image to be classified is obtained for each type of feature amount, and the similarity obtained for each type of feature amount is determined according to the importance set by the setting unit. 20. The image classification device according to claim 18, wherein the total similarity is obtained after performing weighting. 22. When calculating the similarity between the reference image and the image to be classified using a plurality of types of feature amounts, the similarity determined by the calculation means may be arbitrarily determined according to the type of the feature amount. For each of the classification items, the similarity between the reference images set by the setting means is obtained for each type of feature amount, and the similarity between the reference images is obtained for each type of feature amount. An importance determining unit that determines an importance for each feature amount between the reference images based on the similarity; wherein the calculation unit includes, for each of the classification items, each of the types of feature amounts; The similarity between the image and the image to be classified is obtained, and the similarity obtained for each type of feature amount is weighted in accordance with the importance determined by the importance determining means, and then the overall similarity is calculated. It is characterized by seeking the degree Image classification apparatus according to claim 18 or 19 that. 23. The apparatus according to claim 16, wherein the setting unit can set a classification item for further classifying the images classified for each of the classification items. Image classification device. 24. At least one kind of feature amount representing a feature of an image is extracted from a plurality of images to be classified, and each image to be classified is classified into a group of similar images using the extracted feature amount. An image classification device, comprising: a designation unit that designates a type of the feature amount; and when the type of the feature amount is designated by the designation unit, each image to be classified using the designated type of feature amount. Classifying means for classifying each image of the classification target into a group of similar images based on the obtained degree of similarity between the images of the classification target; and Display means for selecting a representative image representative of each group from the group and displaying the selected image on a screen, wherein the specifying means further specifies one of the representative images displayed by the display means, and The type of quantity can be specified, and when the type of the representative image and the characteristic amount is specified by the specifying means, the group to which the representative image belongs by using the specified type of characteristic amount Classifying each image classified into a group of similar images, wherein the display unit selects a representative image representing each group from each group classified by the classification unit and displays the selected image on a screen. Image classification device. 25. The display means displays on a screen the types of feature amounts that can be designated by the designation means, and associates each of the groups classified by the classification means with the type of feature quantity designated by the designation means. 25. The image classification device according to claim 24, wherein a representative image is displayed on a screen, and a result of classification of the plurality of images to be classified is displayed on a screen in a tree structure. 26. The image processing apparatus according to claim 26, further comprising a search unit that searches for an image that is the same as or similar to the query image by using an arbitrary image classified for each classification item as a query image. The image classification device according to any one of 16 to 25. 27. The search device according to claim 1, wherein
The image classification device according to claim 26, wherein the image classification device is the image search device according to any one of the above. 28. The classification target image according to claim 2,
The image according to any one of claims 16 to 25, wherein the image is an image retrieved by the retrieval means according to claim 6 or the image retrieval device according to any one of claims 1 to 14. Classifier. 29. A computer-readable recording medium having recorded thereon a program for causing a computer to function as each means of the image classification device according to any one of claims 16 to 28.