JP4674257B2 - Image classification apparatus and computer-readable recording medium storing a program for causing a computer to function as the apparatus - Google Patents

Description

  The present invention relates to an image search device for searching for an image desired by a user from a large number of images and an image classification device for classifying a large number of images into a predetermined classification destination. The present invention relates to an image classification apparatus capable of performing search processing and image classification processing.

  In recent years, with the development of computer technology and image processing technology, attempts to build an image database by accumulating a large amount of electronic images have been actively conducted. What is important when constructing an image database is an image search device for easily obtaining an image to be used from among a large amount of accumulated images.

  As one of the image search devices, a search key using words is assigned to each image, and the input keyword is matched with the search key attached to the image in the same way as the document search to match the keyword. In some cases, an image with a search key to be returned is returned as a search result.

  However, since the impression that a person receives from an image varies from person to person, it cannot be said that the search key attached to the image expresses all the impression that the person receives from the image. Therefore, in the image search using the search key by words, In many cases, a satisfactory image cannot be obtained. This is because the shape features and positional relationships of things (hereinafter referred to as “objects”) such as humans, animals, and buildings that appear in the image cannot often be expressed in appropriate words.

  Therefore, instead of searching for an image by matching a keyword with a search key by words, a specific image (hereinafter referred to as “query image”) is given as a search condition, and an image similar to the query image is stored in the image database. It is desirable to be able to search from within. In order to realize such a search device, feature quantities representing image features are extracted from the query image and the search target image, respectively, and the similarity between the query image feature quantity and the search target image feature quantity is determined. A method has been proposed in which an image having a feature amount similar to the feature amount of the inquiry image is returned as a search result. That is, this image search apparatus searches for similar images by regarding the similarity between features as the similarity of the images themselves.

  Also, instead of using the image itself as an inquiry image, the image color (color histogram), texture, object shape, etc. are extracted as a database and stored in a database, and when searching for an image, A search device that searches for an image having a specified feature amount by specifying the feature amount of the image in the database as a search condition has also been proposed (Myron Flickner et al, “Query by Imageand Video Content: The QBIC System , ”Computer, September 1995, pp23-32).

  However, in an image search apparatus that searches for an image similar to the query image by specifying the query image, for example, if there is no query image similar to the image that the user wants as a search result, it is specified as the query image. Therefore, there is a problem in that it is difficult to perform the search operation.

  Further, even in an image search apparatus that searches for an image by designating the feature amount of the image, for example, the user selects a color scheme of the image desired as the search result, or the shape of the object present in the image desired as the search result Since it is necessary to specify the feature amount of the image as a search condition by drawing, there is a problem that the burden on the user when executing the search is large. In addition, since the feature amount of the image is specified as the search condition, it is difficult to accurately express the image desired by the user as the search result in the search condition, and the user's intention for the search cannot be properly transmitted to the apparatus. There was a point. As a result, it is difficult to obtain a search result desired by the user.

  In other words, the conventional image search apparatus is inconvenient in that there is no degree of freedom in specifying search conditions, and the user's intention cannot be accurately expressed as search conditions. Therefore, it is difficult to accurately convey the user's intention to the apparatus side, and it is difficult to accurately obtain an image desired by the user. Furthermore, in order to accurately convey the user's intention to the device side and perform the search process as intended by the user, various search methods are prepared so that the user can perform the search process in a desired manner. There must be.

  An object of the present invention is to provide an image classification device that can perform image classification processing by a new method by applying the image retrieval processing in the image retrieval device as described above to image classification processing.

In order to achieve the above object, an image classification apparatus according to the present invention is an image that serves as a classification criterion for the classification target image for each classification item that is a classification destination of the classification target image, and represents a plurality of image characteristics. And determining whether or not to classify the image into the classification items based on the similarity between the image and one or more reference images associated with any one of the feature amount types. A setting means for setting the threshold value , a calculating means for calculating the similarity of the type of feature quantity associated with each of the reference images with respect to the image to be classified, and the calculated similarity based on the said threshold value, an image of the classified and a determination unit configured to determine whether to classify the classification items, the setting means, for each category of the grouping destination of the classification target image, Further similarity The determination means calculates a total similarity obtained by weighting and summing the similarities calculated for each of the reference images with the importance, and based on the total similarity and the threshold value, It is determined whether or not the images to be classified are classified into the classification items .

  In the image classification device of the present invention, the setting means can set classification items for further classifying the images classified for each classification item.

  The recording medium of the present invention is a computer-readable recording medium in which a program for causing a computer to function as each means of any one of the image classification apparatuses of the present invention is recorded.

  According to the image classification device of the present invention, classification items are set, a plurality of reference images are set for each set classification item, and the similarity between each set reference image and classification target image is set for each classification item. The total similarity of the images to be classified is obtained by summing the obtained similarities, and the classification is performed based on the overall similarity of the images to be classified obtained for each classification item and a preset classification destination determination criterion. In order to determine a classification item to be a classification destination of the target image, it is possible to perform an image classification process using an AND search process.

1 is a block configuration diagram of an image search device according to Embodiment 1 of the present invention. It is a block diagram which shows the process performed in the image search device which concerns on Embodiment 1 of this invention. It is a flowchart which shows the outline of the image registration process performed in the image search device which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows a mode that each image was mapped to the feature space according to the feature-value extracted from the image in the image search device which concerns on this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 1 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 2 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 2 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 3 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 4 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 4 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 5 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 5 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 6 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 6 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 7 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 8 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 9 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 10 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 11 of this invention. In the image search device concerning Embodiment 12 of this invention, it is explanatory drawing explaining the level of the roughness of the feature-value extracted from a query image and the image of search object. It is explanatory drawing explaining the feature-value extraction process (extraction of an edge feature) in the image search device which concerns on Embodiment 12 of this invention. It is explanatory drawing explaining the feature-value extraction process (extraction of an edge feature) in the image search device which concerns on Embodiment 12 of this invention. It is explanatory drawing which shows the search window used at the time of the search in the image search device concerning Embodiment 12 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 12 of this invention. It is explanatory drawing which shows the modification 1 of the image search process performed in the image search device which concerns on Embodiment 12 of this invention. It is explanatory drawing which shows the modification 2 of the image search process performed in the image search device which concerns on Embodiment 12 of this invention. It is explanatory drawing for demonstrating the outline of the image search process performed in the image search device concerning Embodiment 13 of this invention. It is explanatory drawing for demonstrating the outline of the image search process performed in the image search device concerning Embodiment 13 of this invention. It is a flowchart which shows the image search process performed in the image search device which concerns on Embodiment 13 of this invention. It is explanatory drawing which shows the search window used at the time of a search in the image search device which concerns on Embodiment 13 of this invention. It is a block block diagram of the image classification device based on Embodiment 14 of this invention. It is a block diagram which shows the process performed in the image classification device which concerns on Embodiment 14 of this invention. It is explanatory drawing which shows an example of the image classification window for performing the setting process of the classification item of an image, and the display of the image after a classification | category in the image classification device based on Embodiment 14 of this invention. It is a flowchart which shows the image classification process performed in the image classification device based on Embodiment 14 of this invention. It is explanatory drawing which shows the mode of the image classification | category window after the image is classified in the image classification device based on Embodiment 14 of this invention. In the image classification device according to the fifteenth embodiment of the present invention, it is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and display of images after classification. It is explanatory drawing which shows an example of the image classification window for performing the setting process of the classification item of an image, and the display of the image after a classification | category in the image classification device based on Embodiment 16 of this invention. It is a flowchart which shows the image classification process performed in the image classification device based on Embodiment 16 of this invention. In the image classification device according to the seventeenth embodiment of the present invention, it is an explanatory diagram showing an example of an image classification window for performing processing for setting image classification items and displaying images after classification. In the image classification device according to the eighteenth embodiment of the present invention, it is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and display of images after classification. In the image classification device according to the nineteenth embodiment of the present invention, it is an explanatory diagram showing an example of an image classification window for performing image classification item setting processing and display of images after classification. It is a flowchart which shows the image classification process performed in the image classification device based on Embodiment 21 of this invention. It is a flowchart which shows the image classification process performed in the image classification device based on Embodiment 22 of this invention. It is explanatory drawing which shows the image classification process performed in the image classification device based on Embodiment 25 of this invention. It is explanatory drawing which shows the image classification process performed in the image classification device based on Embodiment 25 of this invention. It is explanatory drawing which shows the image classification process performed in the image classification device based on Embodiment 25 of this invention. It is explanatory drawing which shows the distribution form of the recording medium which recorded the program which implement | achieves the image search device and / or image classification device based on this invention, and a program.

  Embodiments of an image search device and an image classification device according to the present invention will be described below in detail with reference to the accompanying drawings.

[Embodiment 1]
First, an image search apparatus according to the present invention will be described. FIG. 1 is a block configuration diagram of an image search apparatus according to the first embodiment. The image search apparatus shown in FIG. 1 includes a computer 100 such as a personal computer or a workstation, and includes a CPU 101 that controls the whole, a ROM 102 that stores a boot program, and a RAM 103 that is used as a work area of the CPU 101. A keyboard 104 and a mouse 105 for inputting an inquiry image, a search execution command, etc., a monitor 106 such as a CRT or a liquid crystal display for displaying an image of a search result, and the image search apparatus according to the present invention Image search software 107 for causing the CPU 101 (computer 100) to execute a function, an image DB 108 in which a plurality of images are registered, an image feature DB 109 in which feature amounts extracted from images in the image DB 108 are stored, and the like Enter the image An interface (I / F) 111 for connecting a scanner or a digital camera for performing the operation, a network adapter 113 for connecting the computer 100 (image search device) to a network 112 such as the Internet or a LAN, and each of the above components And a bus 114 for connecting the two.

  FIG. 2 is a block diagram showing processing executed in the image search apparatus shown in FIG. Although details of the processing will be described later, in FIG. 2, a user interface (U / I) unit 151 (corresponding to the designation unit and output unit of the present invention) A search condition input screen, a search result display screen, etc. are displayed, and search condition input, search result output processing, etc. are executed. The feature extraction engine 152 (corresponding to the importance determination unit and the feature extraction unit of the present invention) extracts the feature amount of the image from the image (search target image) registered in the image DB 108 and the query image specified at the time of the search. Execute the process. Further, the matching engine 153 (corresponding to the search means of the present invention) compares the feature quantity extracted from the query image with the feature quantity of the image in the image DB 108 registered in the image feature DB 109, and determines the query image. A process of searching for an image having a feature corresponding to the feature is executed.

  In FIG. 2, a U / I unit 151, a feature extraction engine 152, and a matching engine 153 mean modules that constitute the image search software 107 shown in FIG. The processing by these units is realized by the CPU 101 shown in FIG.

Next, regarding processing executed by the image search apparatus having the above-described configuration,
(1) Feature amount extraction processing (2) Image search processing will be described in this order.

(1) Feature Amount Extraction Processing First, processing for extracting feature amounts representing image features from each image will be described by taking as an example processing for registering an image to be searched in the image DB 108. Here, as an example, it is assumed that color features (color histogram) of images, edge features of images, and texture features in images that are well known in the past are extracted as feature amounts.

  FIG. 3 is a flowchart showing an outline of an image registration process executed in the image search apparatus according to the first embodiment. The feature extraction engine 152 shown in FIG. 2 inputs an image to be registered via the U / I unit 151 (S11). The input image may be any image such as an image read by a scanner, an image captured by a digital camera, an image downloaded via the network 112, or an image stored in the hard disk 110 in advance.

  Then, the feature extraction engine 152 executes a process of extracting a feature amount from the input image (S12). In step S12, the three types of feature values are extracted from each image. These three types of feature amounts can be extracted from each image using a conventionally known method as described below.

  An image color histogram (hereinafter simply referred to as “color”) is divided into a plurality of regions by selecting an appropriate color space (eg, Lab, Luv, HSV, etc.), and each pixel of the image is colored. It can be obtained by examining which region in the space corresponds and normalizing the number of pixels in each region by the total number of pixels. An edge of an image (hereinafter referred to as “structure”) can be obtained by using an edge extraction filter such as a sobel. Furthermore, the texture can be obtained by texture extraction based on the co-occurrence matrix (reference: Mikio Takagi, supervised by Yoshihisa Shimoda, “Image Analysis Handbook”, The University of Tokyo Press, 1991).

  After extracting the feature amount from the image in step S12, the feature extraction engine 152 registers the extracted feature amount in the image feature DB 109 in association with the original image registered in the image DB 108 (S13). The original image is also registered in the image DB 108.

  From the viewpoint of speeding up the image search, it is preferable to extract the feature amount in advance and register it in the image feature DB 109 at the time of registration for the search target image registered in the image DB 108. However, when feature amounts are to be extracted from each search target image during image search, it is not necessary to extract feature amounts in advance.

  By executing the processing shown in FIG. 3, the image registered in the image DB 108 is mapped to a feature space as shown in FIG. 4 for each type of extracted feature quantity. At the time of image search, the feature amount of the image is extracted from the inquiry image as will be described later, and is mapped to the feature space shown in FIG. The points (black dots) shown in FIG. 4 indicate images mapped to the feature space, and the distance between the points of the inquiry image and the points of each image is the image similarity. Note that most image feature amounts are vector data, and vector distance definition formulas such as Euclidean distance are generally used to calculate the distance between points that are similarities. It is also possible to define a unique distance according to the type of feature amount of the image.

(2) Image Search Process Next, the image search process executed in the image search apparatus according to Embodiment 1 will be described. The image search device according to the first embodiment takes into consideration that the similarity point between the query image and the image desired by the user as a search result cannot be transmitted to the device simply by specifying the query image as a search condition. By designating an inquiry image for each type of feature amount used for search, a virtual inquiry image considered appropriate by the user can be expressed. As described above, the types of feature quantities that can be used here are three types of color, structure, and texture, and the search target image is an image registered in the image DB 108.

  FIG. 5 is an explanatory diagram showing a search window used at the time of search, and FIG. 6 is a flowchart showing image search processing according to the first embodiment. The U / I unit 151 in FIG. 2 displays the search window 200a shown in FIG. 5 on the monitor 106 shown in FIG. 1 when executing the image search process. The search window 200a is used to specify the image designation fields A, B, and C for designating one query image for each color, structure, and texture feature amount used for image retrieval, and to designate execution of image retrieval processing. A search button 201 and an image display field 202 for displaying a search result image are provided.

  The user designates an inquiry image for each type of feature amount in each image designation field shown in FIG. 5 (S101). The image designated as the inquiry image may be any image such as an image read by a scanner, an image captured by a digital camera, an image downloaded via the network 112, or an image stored in the hard disk 110 in advance. .

  Here, a method for specifying an inquiry image will be described in detail. As a premise, there is no single image that appropriately represents an image desired by the user as a search result. For example, an image A, an image structure (object that is similar to the image color desired in the search result, and an object It is assumed that an image B having a similar atmosphere and an image C having a similar texture atmosphere in the image are found.

  As shown in FIG. 5, the user designates these three images A, B, and C as inquiry images in the image designation columns A, B, and C, respectively. Specifically, an inquiry image A whose image color atmosphere is similar to the desired image as a search result is designated in the image designation field A corresponding to the feature amount “color”. Similarly, an inquiry image B whose image structure atmosphere is similar to the desired image as a search result is displayed in the image designation field B corresponding to the feature quantity “structure”, and an inquiry image C whose texture atmosphere in the image is similar. Is designated in the image designation field C corresponding to the feature amount “texture”.

  In the following description, the image designated in the image designation field A is the query image A, the image designated in the image designation field B is the query image B, and the image designated in the image designation field C is the query image C. I will call it.

  In this way, not only simply specifying multiple query images, but also specifying a query image for each type of feature quantity used for image search, a single virtual query image considered appropriate by the user is expressed. In addition, the user's intention for the search can be accurately transmitted to the device side. Therefore, it is possible to obtain an image of a search result reflecting the user's intention by a search process described later.

  Returning to the flowchart of FIG. When the user designates the search button 201 in FIG. 5, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 extracts the query images A, B, and C from each query image A, B, and C. A process of extracting the corresponding type of feature quantity is executed (S102). That is, the feature extraction engine 152 extracts a color feature amount from the query image A, a structure feature amount from the query image B, and a texture feature amount from the query image C. After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  Note that if the feature amount extracted from the query images A, B, and C is registered in the image feature DB 109, the process of step S102 in FIG. 6 can be skipped.

  Subsequently, the matching engine 153 obtains the similarity to each of the query images A, B, and C using the corresponding feature amount for each search target image registered in the image DB 108 and obtains each obtained query. A process of calculating the total similarity of the search target images by summing up the similarities with the images A, B, and C is executed (S103).

  The processing in step S103 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. The distance between the images corresponds to the similarity between the two images, and the shorter the distance is, the more similar the two images are in color characteristics. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B based on the feature amount of the structure, and calculates the distance between the image to be searched and the query image C based on the feature amount of the texture. Ask.

Thereafter, the matching engine 153 sums the distances between the search target image and each of the query images A, B, and C, and sets the total value of the distances as the total similarity. Here, if the number of types of feature quantity is N, the query image of feature quantity j is Q _j , the search target image is I _i , and the distance formula between the feature quantity j images is D _j (), The total similarity S _{i for} each image is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  The U / I unit 151 refers to the total similarity obtained for each search target image by the matching engine 153, and displays the search target images in the image display column 202 in FIG. (S104). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, the search target is actually in ascending order of the value of the overall similarity. An image is displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  Note that it is not necessary to display all the images to be searched in the image display field 202. Only a predetermined number of images are displayed, or less than a predetermined threshold (total similarity is represented by the distance between images). Only the images that have obtained the overall similarity may be displayed. That is, it is possible to arbitrarily set / change how the search result is displayed (the same applies to the following embodiments).

  As a result of referring to the image of the search result displayed in the image display field 202, if it is desired to perform a narrowed search, the image displayed in the image display field 202 is designated as an inquiry image and the same search process is performed. You can also. It is also possible to perform a process of enlarging and displaying the image by clicking the image displayed in the image display field 202 with the mouse 105.

  Thus, according to the image search device according to the first embodiment, one query image is designated for each type of feature amount used when obtaining the similarity between the query image and the search target image. For each search target image, the similarity to each specified query image is obtained using the corresponding type of feature amount, and the similarity to each obtained query image is totaled to obtain the overall similarity of the search target image Since the user's intention can be accurately expressed as a search condition by outputting the search result based on the total similarity obtained for each image to be searched, it is possible to accurately convey the user's intention to the device. In addition, the user's intention for the search can be surely communicated to the apparatus, so that a search result accurately reflecting the user's intention for the search can be obtained, and a highly accurate image as desired by the user can be obtained. It is possible to perform the search process.

  For example, even if an appropriate image does not exist as an inquiry image, an image having a color atmosphere similar to an image desired as a search result or an image having an object shape atmosphere is specified as an inquiry image. Can be used as one query.

  In the description of the first embodiment, the query image is specified corresponding to all types of feature amounts, but it is not always necessary to specify the query image corresponding to all types of feature amounts. . In addition, although three types of feature quantities of color, structure, and texture are used, these are merely examples, and are not intended to limit the types and number of feature quantities.

[Embodiment 2]
As in the case of the first embodiment, the image search device according to the second embodiment gives the apparatus a similar viewpoint between the query image and the image that the user desires as a search result simply by specifying the query image as a search condition. In consideration of the fact that it cannot be transmitted, by specifying multiple query images and the types of features used for the search, it is possible to express one virtual query image that the user thinks appropriate It is a thing. Note that in the image search device according to the second embodiment, the same reference numerals are used for the configurations already described in the first embodiment, and detailed description thereof is omitted, and common points regarding the image search processing are also described as appropriate. Is omitted.

  FIG. 7 is an explanatory view showing a search window used at the time of search, and FIG. 8 is a flowchart showing image search processing according to the second embodiment. The U / I unit 151 in FIG. 2 displays the search window 200b shown in FIG. 7 on the screen 106 shown in FIG. 1 when executing the image search process. The search window 200b is used to obtain the similarity between the image designation columns A, B, and C for designating a plurality of query images and the query target image for each query image designated in each image designation column. A feature amount designation button 203 for designating the type of feature amount, a search button 201 for designating execution of image search processing, and an image display column 202 for displaying an image of the search result are provided.

  As described in the first embodiment, as a premise, there is no single image that appropriately represents an image that the user desires as a search result. For example, an image color arrangement atmosphere exists in the image that is desired as a search result. Assume that a similar image A, an image B having a similar image structure atmosphere, and an image C having a similar texture atmosphere in the image are found.

  The user designates a plurality of inquiry images in each image designation field shown in FIG. 7 and designates the type of feature amount used for obtaining the similarity with the image to be searched for each designated inquiry image ( S201).

  Specifically, the user designates three images A, B, and C as inquiry images in the image designation fields A, B, and C as shown in FIG. Then, the user designates “color” with the feature amount designation button 203 for the inquiry image A in which the atmosphere of the color arrangement of the image is similar to the desired image as the search result. Similarly, “Structure” is designated for the query image B whose image structure atmosphere is similar to the desired image as the search result, and “Texture” is designated for the query image C whose texture atmosphere in the image is similar. .

  In this way, not only simply specifying a plurality of query images, but specifying a feature quantity corresponding to the feature of the image that the user places importance on for each specified query image, a virtual one that the user considers appropriate It is possible to express an inquiry image and accurately convey the user's intention for the search to the apparatus side. Therefore, it is possible to obtain an image of a search result reflecting the user's intention by a search process described later.

  However, in FIG. 7, different types of feature amounts are designated for each inquiry image, but the same type of feature amounts may be designated. That is, the same type of feature amount may be specified for all of the plurality of inquiry images.

  Then, when the user designates the search button 201 in FIG. 7, the U / I unit 151 inputs the query images A, B, and C and the type of feature amount to the feature extraction engine 152, and the feature extraction engine 152 A process of extracting a specified type of feature quantity from the query images A, B, and C is executed (S202). That is, the feature extraction engine 152 extracts a color feature amount from the query image A, a structure feature amount from the query image B, and a texture feature amount from the query image C. After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  If the feature amounts extracted from the query images A, B, and C are registered in the image feature DB 109, the process of step S202 in FIG. 8 can be skipped.

  Subsequently, the matching engine 153 obtains the similarity with each of the query images A, B, and C using the type of feature specified by the user for each search target image registered in the image DB 108. A process of calculating the total similarity of the search target images by summing up the similarities with the obtained inquiry images A, B and C (S203).

  The processing in step S203 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. The distance between the images corresponds to the similarity between the two images, and the shorter the distance is, the more similar the two images are in color characteristics. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B based on the feature amount of the structure, and calculates the distance between the image to be searched and the query image C based on the feature amount of the texture. Ask.

Thereafter, the matching engine 153 sums the distances between the search target image and each of the query images A, B, and C, and sets the total value of the distances as the total similarity. Here, the number of feature amounts is N, the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between images of the feature amount j is D _j (), k. If the presence / absence of designation of the feature value j of the second image is t _jk (1 if designated, 0 if not designated), the total similarity S _i for each image to be searched is expressed as follows. It is.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  The U / I unit 151 refers to the overall similarity obtained for each search target image by the matching engine 153, and displays the search target images in the image display column 202 in FIG. (S204). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity, the search target is actually in ascending order of the value of the overall similarity. An image is displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  Note that it is not necessary to display all the images to be searched in the image display field 202. Only a predetermined number of images are displayed, or less than a predetermined threshold (total similarity is represented by the distance between images). In this case, it is possible to display only the images that have obtained the overall similarity.

  As a result of referring to the image of the search result displayed in the image display field 202, if it is desired to perform a narrowed search, the image displayed in the image display field 202 is designated as an inquiry image and the same search process is performed. You can also. It is also possible to perform a process of enlarging and displaying the image by clicking the image displayed in the image display field 202 with the mouse 105.

  As described above, according to the image search device according to the second embodiment, a plurality of query images are specified, and the type of feature amount used when obtaining the similarity with the search target image for each specified query image For each image to be searched, the similarity with each query image is calculated using the specified type of feature amount, and the total similarity is obtained by summing the similarities with each obtained query image. The user's intention can be accurately expressed as a search condition by outputting the search result based on the total similarity obtained for each image to be searched, so that the user's intention can be accurately transmitted to the apparatus, Since the user's intention for the search can be reliably transmitted to the apparatus, a search result that accurately reflects the user's intention for the search can be obtained, and the image search process can be performed with high accuracy as desired by the user. It is possible.

  For example, even if an appropriate image does not exist as an inquiry image, an image having a color atmosphere similar to an image desired as a search result or an image having an object shape atmosphere is specified as an inquiry image. Can be used as one query. In addition, since the feature quantity desired by the user can be specified for each specified inquiry image, the search condition can be freely set as compared with the image search apparatus according to the first embodiment.

  In the description of the second embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. That is, even if only one inquiry image is specified, the search process can be performed in the same way, but in the second embodiment, it is meaningful to specify a plurality of images. In addition, although three types of feature quantities of color, structure, and texture are used, these are merely examples and are not intended to limit the types of feature quantities.

[Embodiment 3]
The image search apparatus according to the third embodiment is configured so that the user's intention for the search can be expressed more accurately in the image search apparatus according to the second embodiment.

  FIG. 9 is an explanatory diagram showing a search window 200c used in the image search device according to the third embodiment. In addition to the configuration of the search window 200b shown in FIG. 7, the search window 200c shown in FIG. 9 is important for specifying the relative importance for each inquiry image specified in the image specification fields A, B, and C. A degree designation bar 204 is provided.

  As described in the first and second embodiments, as a premise, there is no single image that appropriately represents an image desired by the user as a search result. Assume that an image A having a similar atmosphere, an image B having a similar image structure atmosphere, and an image C having a similar texture atmosphere in the image are found.

  The user designates these three images as inquiry images A, B, and C in the image designation columns A, B, and C, respectively, as shown in FIG. Next, the user designates “color” with the feature amount designation button 203 for the inquiry image A in which the atmosphere of the color scheme of the image is similar to the desired image as the search result. Similarly, “Structure” is designated for the query image B whose image structure atmosphere is similar to the desired image as the search result, and “Texture” is designated for the query image C whose texture atmosphere in the image is similar. .

  Subsequently, the user uses the importance designation bar 204 to designate the importance for each inquiry image. As an example, the importance level designation bar 204 is configured so that the importance level of “+” and the importance level of “−” can be designated centering on “0” for each inquiry image. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  For example, among the query images A, B, and C, the query image A having a color scheme similar to the desired image as the search result is the most important, that is, an image with a color scheme such as the query image A as the search result is particularly important. If the user feels desired, the user moves the importance designation bar 204 to a desired position in the “+” direction and designates that the inquiry image A is important. When the importance is specified in this way, “+” corresponding to the importance specified for the query image A is used for the similarity between the image to be searched and the query image A at the time of a search described later. Is weighted.

  For example, when the user feels that a texture image such as the query image C is not desired as a search result, the user moves the importance designation bar 204 to a desired position in the “−” direction. That is, the NOT condition can be given to the inquiry image by designating the importance in the “−” direction. Therefore, an image that the user thinks that an image having such characteristics is not desired as a search result can be designated as an inquiry image. In this case, the similarity between the search target image and the query image C is weighted with “−” according to the importance specified for the query image C.

  As described above, by enabling the user to designate the importance level for each inquiry image, the user's intention for the search can be accurately expressed as the search condition. That is, since the importance is reflected as a weighting on the similarity by a search process described later, it is possible to obtain a search result image reflecting the user's intention. Note that it is not always necessary to specify the degree of importance. When the user desires, the degree of importance may be specified for the desired inquiry image.

  Next, image search processing by the image search apparatus according to Embodiment 3 will be described. FIG. 10 is a flowchart showing search processing according to the third embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 9 and designates the type of feature amount used when obtaining the similarity with the image to be searched for each designated inquiry image, The importance is designated as necessary (S301). As shown in FIG. 9, it is assumed that the user designates inquiry images A, B, and C, and designates the type and importance of the feature amount for each inquiry image A, B, and C.

  Subsequently, when the user designates the search button 201 in FIG. 9, the U / I unit 151 inputs the query images A, B, and C and the type of feature quantity to the feature extraction engine 152. A process of extracting a specified type of feature amount from each of the query images A, B, and C is executed (S302). That is, the feature extraction engine 152 extracts a color feature amount from the query image A, a structure feature amount from the query image B, and a texture feature amount from the query image C. After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  Subsequently, the matching engine 153 obtains the degree of similarity with each of the query images A, B, and C using the type of feature amount designated by the user for each image to be searched that is registered in the image DB 108. Then, weighting is performed on each similarity according to the importance, and a process of obtaining the total similarity of the search target images by summing the similarities with the query images A, B, and C is executed (S303).

  The processing in step S303 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the inquiry image A designated by the importance designation bar 204. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B based on the feature amount of the structure, weights the obtained distance according to the importance of the query image B, and The distance between the image to be searched and the inquiry image C is obtained based on the feature amount of the texture, and the obtained distance is weighted according to the importance of the inquiry image C.

  Since the similarity to be weighted is the distance between the feature amounts of the images, the weighting of “+” based on the importance of “+” works in the direction of shortening the distance between the images, and “−”. The weighting of “−” based on the importance level of the function works in the direction of increasing the distance between images.

Thereafter, the matching engine 153 sums the distances between the search target image and each of the query images A, B, and C, and sets the total value of the distances as the total similarity. Here, the number of feature amounts is N, the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between images of the feature amount j is D _j (), k. If the importance of the feature quantity j of the second image is w _jk (0 if the corresponding feature is not specified), the total similarity S _i for each image to be searched is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  The U / I unit 151 refers to the total similarity obtained for each search target image by the matching engine 153, and displays the search target images in the image display column 202 in FIG. (S304). Here, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the total similarity, in practice, the search targets are actually searched in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also. It is also possible to perform a process of enlarging and displaying the image by clicking the image displayed in the image display field 202 with the mouse 105.

  As described above, according to the image search device according to the third embodiment, the degree of importance is set for each inquiry image so that the obtained similarity can be arbitrarily weighted according to the type of inquiry image. By specifying the degree of similarity with each query image for each image to be searched and weighting the degree of similarity with each obtained query image according to the specified importance Since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the apparatus, and the importance is reflected as a weight for the similarity, so the user's intention is reflected. An image as a search result can be obtained.

  In the description of the third embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. However, in the third embodiment, it is meaningful to specify a plurality of inquiry images because the importance is specified for each inquiry image.

  Although detailed description is omitted, the image search apparatus described in the first embodiment can also specify the importance for each inquiry image as described in the third embodiment. good.

[Embodiment 4]
The image search device according to the fourth embodiment enables AND search to be executed as one technique for accurately expressing the user's intention as a search condition. Note that in the image search device according to the fourth embodiment, the same reference numerals are used for the configurations already described in the first embodiment, and detailed description thereof is omitted, and common points regarding the image search processing are also described as appropriate. Is omitted.

  FIG. 11 is an explanatory diagram showing a search window 200d used in the image search device according to the fourth embodiment. The search window 200d includes an AND designation button 205 for designating execution of an AND search and an OR for designating execution of an OR search in addition to the image designation columns A, B and C, the search button 201 and the image display column 202. A designation button 206 is provided. The OR search will be described in detail in an embodiment 8 to be described later, and only the AND search will be described here. In addition, although the feature amount used in the search is assumed to be one type, naturally, a plurality of types of feature amounts may be used.

  Next, image search processing by the image search apparatus according to Embodiment 4 will be described. FIG. 12 is a flowchart showing image search processing according to the fourth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 11, selects the AND designation button 205, and designates an image search process based on an AND condition (S401). As shown in FIG. 11, it is assumed that the user designates three inquiry images A, B, and C.

  Subsequently, when the user designates the search button 201 in FIG. 11, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 selects each query image A, B. And a process of extracting a predetermined feature amount from C (S402). After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  The matching engine 153 obtains a similarity to each of the query images A, B, and C using a predetermined feature amount for each search target image registered in the image DB 108, and obtains each query image A, A process of calculating the total similarity of the search target images by summing the similarities with B and C is executed (S403).

  The processing in step S403 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount. That is, the matching engine 153 determines the distance between the search target image plotted in the feature amount space shown in FIG. 4 and the query image A based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Find using. Similarly, the matching engine 153 obtains the distance between the image to be searched and the inquiry image B based on a predetermined feature amount, and obtains the distance between the image to be searched and the inquiry image C.

Thereafter, the matching engine 153 sums the distances between the search target image and each of the query images A, B, and C, and sets the total value of the distances as the total similarity. Here, if the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between the images is D (), the total similarity S for each search target image is S. _i is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  When performing a search using a plurality of types of feature amounts, the similarity (distance) between the search target image and each query image is obtained for each type of feature amount, and for each feature amount for each search target image. The total similarity can be obtained by summing all the similarities with the respective query images obtained in the above.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. (S404). The total similarity here means the similarity of each image to be searched with respect to all of the query images A, B, and C. Therefore, an AND search of the images is executed. However, since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the total similarity here, the search target is actually in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrow search is to be performed, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the fourth embodiment, a plurality of query images are specified, and for each search target image, the similarity with each specified query image is obtained, and each obtained query image is obtained. Since the total similarity is obtained by summing the similarities and the search result is output based on the overall similarity obtained for each image to be searched, an AND search can be performed. Can be provided as a search condition.

  In the description of the fourth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Further, although the type of feature quantity is not particularly shown, any type of feature quantity such as color, structure, texture, etc. can be used alone or in combination.

[Embodiment 5]
The image search device according to the fifth embodiment is the image search device according to the fourth embodiment, in which the user's intention for the search can be expressed more accurately.

  FIG. 13 is an explanatory diagram showing a search window 200e used in the image search device according to the fifth embodiment. In addition to the search window 200d shown in FIG. 11, the search window 200e shown in FIG. 13 is an importance designation bar for designating relative importance for each inquiry image designated in the image designation fields A, B and C. 207.

  As described in the third embodiment, the importance designation bar 207 can designate the importance of “+” and the importance of “−” with “0” as the center for each inquiry image. To do. Therefore, when an importance level of “+” is designated for an inquiry image that the user considers important, and when an image similar to such an image is not desired as a search result, “−” is obtained for the corresponding inquiry image. The importance of can be specified. The designated importance is reflected in the search result as a weight for the obtained similarity. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  Next, image search processing by the image search apparatus according to Embodiment 5 will be described. FIG. 14 is a flowchart showing search processing according to the fifth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 13 and designates the importance as necessary for each designated inquiry image. Further, the user selects an AND designation button 205 and selects the AND condition. An image search process is designated (S501). As shown in FIG. 13, it is assumed that the user designates inquiry images A, B, and C, and designates an arbitrary importance degree for each designated inquiry image A, B, and C.

  When the user designates the search button 201 in FIG. 13, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 A process of extracting a predetermined feature amount from C is executed (S502). After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  The matching engine 153 obtains the similarity with each of the query images A, B, and C using a predetermined feature amount for each search target image registered in the image DB 108, and for each obtained similarity Weighting is performed in accordance with the importance, and the processing for obtaining the total similarity of the search target images by summing the similarities with the query images A, B, and C is executed (S503).

  The processing in step S503 will be specifically described taking one search target image as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount. That is, the matching engine 153 determines the distance between the search target image plotted in the feature amount space shown in FIG. 4 and the query image A based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the query image A specified in the search window 200e. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B, weights the obtained distance according to the importance of the query image B, and also searches the image to be searched and the query image. The distance to C is obtained, and the obtained distance is weighted according to the importance of the inquiry image C.

Thereafter, the matching engine 153 sums the distances between the search target image and each of the query images A, B, and C, and sets the total value of the distances as the total similarity. Here, if the number of query images is M, the k-th query image is Q _k , the search target image is I _i , the distance formula between the images is D (), and the importance of the k-th image is w _k , The total similarity S _i for each image to be searched is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. (S504). Since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity here, the images to be searched are actually in ascending order of the value of the overall similarity. It will be displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the fifth embodiment, the importance is set for each inquiry image so that the obtained similarity can be arbitrarily weighted according to the type of the inquiry image. By specifying the degree of similarity with each query image for each image to be searched and weighting the degree of similarity with each obtained query image according to the specified importance Since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the apparatus, and the importance is reflected as a weight for the similarity, so the user's intention is reflected. An image as a search result can be obtained.

  In the description of the fifth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Further, although the type of feature quantity is not particularly shown, any type of feature quantity such as color, structure, texture, etc. can be used alone or in combination.

[Embodiment 6]
The image search device according to the sixth embodiment is such that the user's intention for the search can be expressed more accurately in the image search device of the fourth embodiment.

  FIG. 15 is an explanatory diagram showing a search window 200f used in the image search apparatus according to the sixth embodiment. The search window 200f shown in FIG. 15 is a feature used for the search for each of the inquiry images A, B, and C specified in the image specification fields A, B, and C in addition to the search window 200d shown in FIG. An importance level designation bar 208 for designating the importance level for each type of quantity is provided. Note that, as an example of the feature amount used for the search, three types of color, structure, and texture are used.

  The importance level designation bar 208 enables designation of the importance level for each type of feature amount for each inquiry image. The importance level designation bar 204 (see FIG. 9) of the third embodiment is used. As described above, it is possible to designate the importance of “+” and the importance of “−” around “0”. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  Specifically, the importance designation method will be described with reference to the inquiry image A in FIG. For example, when it is considered that the color arrangement in the inquiry image A is close to the desired image as a search result, the user operates the color importance designation bar 208 to designate the importance of “+”. On the other hand, when the structure in the query image A is considered to be almost similar to the desired image as a search result, the user operates the structure importance designation bar 208 to designate the importance of “−”.

  In the search processing described later, the similarity between the query image A and the search target image is obtained for each color feature, structure feature, and texture feature. In the above example, the importance level of “+” is specified for the color feature amount, and the feature amount of “−” is specified for the structure feature amount. “+” Is weighted according to the importance of “+”, and “−” is weighted according to the importance of “−” with respect to the similarity obtained using the structural feature amount. This process is executed for the similarity to each image to be searched.

  As for the query images B and C, the process like the query image A is executed by designating the importance for each feature amount described above.

  In this way, by specifying the importance for each type of feature amount for each inquiry image, it is possible to reliably convey the user's intention for the search to the apparatus side. Then, since the importance is reflected as a weighting on the similarity obtained for each type of feature amount by the search process, it is possible to obtain an image of a search result reflecting the user's intention. Note that it is not always necessary to specify the importance level. If the user desires, the importance level may be specified for an arbitrary feature amount of the desired query image.

  Next, image search processing by the image search apparatus according to Embodiment 6 will be described. FIG. 16 is a flowchart showing search processing according to the sixth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 15, designates the importance for each type of feature amount for each designated inquiry image, and further selects an AND designation button 205 to select AND. An image search process according to a condition is designated (S601). As shown in FIG. 15, it is assumed that the user designates inquiry images A, B, and C, and designates the importance for each type of feature amount for each inquiry image A, B, and C.

  Then, when the user designates the search button 201 in FIG. 15, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152. Processing for extracting three types of feature values of color, structure, and texture from C is executed (S602). Thereafter, the feature extraction engine 152 passes the feature amounts of each inquiry image extracted for each of the three types of feature amounts to the matching engine 153 (each mapped to the feature space of FIG. 4).

  Subsequently, the matching engine 153 obtains similarities between the query images A, B, and C using the three types of feature amounts for each search target image registered in the image DB 108, and obtains each similarity obtained. The degree of importance is weighted according to the degree of importance, and the degree of similarity between each of the query images A, B, and C is summed up to obtain the total degree of similarity of the search target images (S603).

  The processing in step S603 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the color feature amount designated for the inquiry image A in the search window 200f. Further, the matching engine 153 obtains the distance using the structural feature amount and the texture feature amount for the image to be searched and the query image A, and the search window 200f for the obtained distance to the query image A. On the other hand, weighting is performed in accordance with the importance of the feature quantity of the designated structure and the importance of the feature quantity of the texture.

  Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B for each of the three types of feature amounts, and for each feature amount designated for the query image B for each of the obtained distances. Weighting is performed according to the importance of. Further, the distance between the image to be searched and the inquiry image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted according to the importance for each feature amount designated for the inquiry image C. I do.

Thereafter, the matching engine 153 sums the distances between the search target images obtained for each of the three types of feature quantities and the query images A, B, and C, and sets the total distance as the total similarity. Here, the number of feature amounts is N, the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between images of the feature amount j is D _j (), k. When the importance of the feature quantity j of the second image is w _jk , the total similarity S _i for each image to be searched is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  The U / I unit 151 refers to the total similarity obtained for each search target image by the matching engine 153, and displays the search target images in the image display column 202 in FIG. (S604). Since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity here, the images to be searched are actually in ascending order of the value of the overall similarity. It will be displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the sixth embodiment, when the similarity between the query image and the search target image is determined using a plurality of types of feature amounts, the query image and Importance for each feature type can be specified for each query image so that arbitrary weighting can be performed according to the type of feature amount. By calculating the similarity with each query image for each feature amount, and weighting the similarity with each query image determined for each type of feature amount according to the specified importance, Since the intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the apparatus, and the importance is reflected as a weight for the similarity, so that the search result reflecting the user's intention is reflected. Images can be obtained of.

  In the description of the sixth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Also, the types of feature values are not limited to the three types of color, structure, and texture, and any kind of combination of feature values may be used.

[Embodiment 7]
The image search device according to the seventh embodiment is the image search device according to the fourth embodiment, in which the device side can capture the user's intention for the search.

  In Embodiments 5 and 6 described above, the user needs to specify the importance of the query image or the importance of each feature amount for each query image. In the image search apparatus according to Embodiment 7, The similarity between the query images specified by the user is obtained using the feature amount used for the image search, and the importance for each feature amount between the query images is calculated on the device side based on the obtained similarity for each feature amount. This is characterized in that it is determined and reflected as a weight for similarity. Here, as an example, it is assumed that search processing is performed using three types of feature quantities of color, structure, and texture, and the search window for specifying search conditions is the same as the search window 200d shown in FIG. And

  Next, image search processing by the image search apparatus according to Embodiment 7 will be described. FIG. 17 is a flowchart showing search processing according to the seventh embodiment. The user displays the search window 200d shown in FIG. 11, designates a plurality of inquiry images in each image designation field, and selects an AND designation button 205 to designate an image search process based on an AND condition (S701). It is assumed that the user designates inquiry images A, B, and C.

  When the user designates the search button 201 in FIG. 11, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 Processing for extracting three types of feature values of color, structure, and texture from C is executed (S702).

  Subsequently, the feature extraction engine 152 obtains the distance (similarity) between the query images A, B, and C for each of the extracted three types of feature amounts, and performs each query based on the obtained distance for each feature amount. The importance for each feature amount between the images A, B and C is determined (S703).

  Specifically, the feature extraction engine 152 obtains the inter-feature distance between the query images for each of the three types of feature amounts. If the variance of the distance obtained for each feature quantity is large, it means that the user is not aware of the feature quantity, and if the variance is small, it can be judged that the user attaches importance to the feature quantity. . Therefore, the importance is set to be small when the feature amount is large and the importance is set to be large when the variance is small.

For example, if σ ² is the variance of the distance between all the query images in the feature i, the importance w _k of the i-th feature amount can be determined by the following equation. k is an arbitrary coefficient for making w _k fall within an appropriate value.

  Note that, as described in the third, fifth, and sixth embodiments, the importance level of “+” and the importance level of “−” can be set according to the degree of dispersion of the distance.

  Thereafter, the feature extraction engine 152 passes the feature amount of each inquiry image extracted for each of the three types of feature amounts and the corresponding importance to the matching engine 153.

  The matching engine 153 obtains the similarity to each of the query images A, B, and C using three types of feature amounts for each search target image registered in the image DB 108, and for each obtained similarity Then, weighting is performed in accordance with the importance, and the similarity with each of the query images A, B, and C is summed to execute processing for obtaining the total similarity of the search target images (S704).

  The processing in step S704 will be specifically described by taking one image to be searched as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. Subsequently, the matching engine 153 performs weighting on the obtained distance according to the importance determined by the feature extraction engine 152 for the color feature amount. Further, the matching engine 153 obtains the distance using the structural feature amount and the texture feature amount for the search target image and the query image A, and the structure determined by the feature extraction engine 152 for each obtained distance. The weighting is performed according to the importance of the feature quantity and the importance of the texture feature quantity.

  Similarly, the matching engine 153 obtains the distance between the image to be searched and the inquiry image B for each of the three types of feature amounts, and sets the importance determined for each type of feature amount for each of the obtained distances. Perform the corresponding weighting. Further, the distance between the image to be searched and the inquiry image C is determined for each of the three types of feature amounts, and the determined distances are weighted according to the importance determined for each type of feature amount.

  Thereafter, the matching engine 153 sums the distances between the search target images obtained for each of the three types of feature quantities and the query images A, B, and C, and sets the total distance as the total similarity. In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. (S705). Since the total distance between the feature amounts of the image to be searched and the query images A, B, and C is used as the overall similarity here, the images to be searched are actually in ascending order of the value of the overall similarity. It will be displayed in the image display field 202. That is, the total similarity has a value of 0 or more, and when the total similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the seventh embodiment, when the similarity between the query image and the search target image is obtained using a plurality of types of feature amounts, the feature amount is compared with the obtained similarity. The degree of similarity between each query image is calculated for each type of feature quantity so that arbitrary weighting can be performed according to the type, and between each query image based on the obtained degree of similarity for each type of feature quantity The degree of importance for each type of feature quantity is determined, and for each search target image, the similarity to each query image is obtained for each type of feature quantity, and each query image obtained for each type of feature quantity is obtained. By determining the importance of each type of feature amount between the query images specified by the user on the device side, by weighting the degree of similarity with the degree of importance for each type of feature amount determined, Reflect the importance of the judgment in the similarity between images Thereby Therefore, it is possible to return search results that match the user's intention. Therefore, the search process as desired by the user can be realized, and the search result desired by the user can be obtained.

  In the description of the seventh embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Also, the types of feature values are not limited to the three types of color, structure, and texture, and any kind of combination of feature values may be used.

[Embodiment 8]
The image search device according to the eighth embodiment enables an OR search to be executed as one technique for accurately expressing the user's intention as a search condition. The image search process by the image search apparatus according to the eighth embodiment is executed when the OR specification button 206 is specified instead of the AND specification button 205 in the search window 200d of FIG. 11 used in the description of the fourth embodiment. Is done.

  FIG. 18 is a flowchart illustrating image search processing according to the eighth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 11, selects the OR designation button 206, and designates an image search process based on the OR condition (S801). Assume that the user designates three inquiry images A, B, and C as shown in FIG.

  When the user designates the search button 201 in FIG. 11, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 A process of extracting a predetermined feature amount from C is executed (S802). After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  The matching engine 153 obtains a similarity to each of the query images A, B, and C using a predetermined feature amount for each search target image registered in the image DB 108, and obtains each query image A, A process of selecting the similarity indicating the highest similarity from the similarities to B and C and executing the process of obtaining the total similarity of the search target images is executed (S803).

  The processing in step S803 will be specifically described taking one search target image as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount. That is, the matching engine 153 determines the distance between the search target image plotted in the feature amount space shown in FIG. 4 and the query image A based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Find using. Similarly, the matching engine 153 obtains the distance between the image to be searched and the inquiry image B based on a predetermined feature amount, and obtains the distance between the image to be searched and the inquiry image C.

Thereafter, the matching engine 153 selects the shortest distance from the distances between the search target image and each of the query images A, B, and C to obtain the overall similarity. In other words, the matching engine 153 selects the similarity indicating the highest similarity from the similarities between the search target image and each of the query images A, B, and C. Here, if the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between the images is D (), the total similarity S for each search target image is S. _i is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  When searching using a plurality of types of feature amounts, the similarity (distance) between the search target image and each query image is obtained for each type of feature amount, and each type of feature amount is determined for each search target image. What is necessary is just to select the similarity (shortest distance) which shows the highest similarity from the similarity with each inquiry image calculated | required for every feature-value, and let it be a total similarity.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. (S804). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the total similarity, in practice, the search target is in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the query images and the feature amount of the search target image are equal.

  In the eighth embodiment, if the search target image is similar to any one of the query images A, B, and C, an overall similarity indicating high similarity is obtained. Therefore, OR search is realized.

  As a result of referring to the image of the search result displayed in the image display field 202, if it is desired to perform a narrowed search, the image displayed in the image display field 202 is designated as an inquiry image and the same search process is performed. You can also.

  Thus, according to the image search device according to the eighth embodiment, a plurality of query images are specified, and for each search target image, the degree of similarity with each specified query image is obtained, and each obtained query image is obtained. By selecting the similarity that shows the highest similarity from among the similarities and outputting the search result based on the selected similarity, it is possible to perform an OR search, so the user's intention is searched It is possible to provide one method for accurately expressing as a condition.

  In the description of the eighth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Further, although the type of feature quantity is not particularly shown, any type of feature quantity such as color, structure, texture, etc. can be used alone or in combination.

[Embodiment 9]
The image search device according to the ninth embodiment is such that the user's intention for the search can be expressed more accurately in the image search device of the eighth embodiment. The image search process by the image search apparatus according to the ninth embodiment is executed when the OR specification button 206 is specified instead of the AND specification button 205 in the search window 200e of FIG. 13 used for the description of the fifth embodiment. Is done.

  FIG. 19 is a flowchart illustrating image search processing according to the ninth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 13 and designates the importance as necessary for each designated inquiry image. Further, the user selects an OR designation button 206 according to the OR condition. An image search process is designated (S901). Assume that the user designates inquiry images A, B, and C as shown in FIG. 13, and designates an arbitrary importance degree for each designated inquiry image A, B, and C.

  When the user designates the search button 201 in FIG. 13, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 A process of extracting a predetermined feature amount from C is executed (S902). After that, the feature extraction engine 152 passes the extracted feature amount of each inquiry image to the matching engine 153 (maps to the feature space of FIG. 4).

  The matching engine 153 obtains the similarity with each of the query images A, B, and C using a predetermined feature amount for each search target image registered in the image DB 108, and for each obtained similarity Weighting is performed in accordance with the importance, and the processing for obtaining the overall similarity of the search target images by selecting the similarity indicating the highest similarity from the similarities to the query images A, B, and C is executed. (S903).

  The processing in step S903 will be specifically described taking one search target image as an example. The matching engine 153 obtains a similarity between the search target image and the query image A using a predetermined feature amount. That is, the matching engine 153 determines the distance between the search target image plotted in the feature amount space shown in FIG. 4 and the query image A based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the inquiry image A specified in the search window 200e. Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B, weights the obtained distance according to the importance of the query image B, and also searches the image to be searched and the query image. The distance to C is obtained, and the obtained distance is weighted according to the importance of the inquiry image C.

Thereafter, the matching engine 153 selects the shortest distance from the distances between the search target image and each of the query images A, B, and C to obtain the overall similarity. Here, if the number of query images is M, the k-th query image is Q _k , the search target image is I _i , the distance formula between the images is D (), and the importance of the k-th image is w _k , The total similarity S _i for each image to be searched is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. Is displayed (S904). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the total similarity, in practice, the search target is in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the query images and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the ninth embodiment, the degree of importance is set for each inquiry image so that the obtained similarity can be arbitrarily weighted according to the type of inquiry image. By specifying the degree of similarity with each query image for each image to be searched and weighting the degree of similarity with each obtained query image according to the specified importance Since the user's intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the apparatus, and the importance is reflected as a weight for the similarity, so the user's intention is reflected. An image as a search result can be obtained.

  In the description of the ninth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Further, although the type of feature quantity is not particularly shown, any type of feature quantity such as color, structure, texture, etc. can be used alone or in combination.

[Embodiment 10]
The image search apparatus according to the tenth embodiment is such that the user's intention for the search can be expressed more accurately in the image search apparatus of the eighth embodiment. The image search process by the image search apparatus according to the tenth embodiment is executed when the OR specification button 206 is specified instead of the AND specification button 205 in the search window 200f of FIG. 15 used in the description of the sixth embodiment. Is done.

  FIG. 20 is a flowchart showing image search processing according to the tenth embodiment. The user designates a plurality of inquiry images in each image designation field shown in FIG. 15, designates the importance for each type of feature amount in each designated inquiry image, and further selects an OR designation button 206 to select an OR condition. The image search processing by is designated (S1001). Assume that the user designates query images A, B, and C as shown in FIG. 15, and designates the importance for each type of feature amount for each query image A, B, and C.

  Then, when the user designates the search button 201 in FIG. 15, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152. Processing for extracting three types of feature values of color, structure, and texture from C is executed (S1002). Thereafter, the feature extraction engine 152 passes the feature amounts of each inquiry image extracted for each of the three types of feature amounts to the matching engine 153 (each mapped to the feature space of FIG. 4).

  The matching engine 153 obtains the similarity to each of the query images A, B, and C using three types of feature amounts for each search target image registered in the image DB 108, and for each obtained similarity Weighting according to importance, and selecting the similarity indicating the highest similarity from the similarities to the query images A, B, and C, and executing the process of obtaining the overall similarity of the search target images (S1003).

  The processing in step S1003 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the color feature amount designated for the inquiry image A in the search window 200f. Further, the matching engine 153 obtains the distance using the structural feature amount and the texture feature amount for the image to be searched and the query image A, and the search window 200f for the obtained distance to the query image A. On the other hand, weighting is performed in accordance with the importance of the feature quantity of the designated structure and the importance of the feature quantity of the texture.

  Similarly, the matching engine 153 obtains the distance between the image to be searched and the query image B for each of the three types of feature amounts, and for each feature amount designated for the query image B for each of the obtained distances. Weighting is performed according to the importance of. Further, the distance between the image to be searched and the inquiry image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted according to the importance for each feature amount designated for the inquiry image C. I do.

After that, the matching engine 153 selects the shortest distance from the distances between the search target images obtained for each of the three types of feature amounts and the query images A, B, and C to obtain the total similarity. Here, the number of feature amounts is N, the number of query images is M, the k-th query image is Q _k , the search target image is I _i , and the distance formula between images of the feature amount j is D _j (), k. When the importance of the feature quantity j of the second image is w _jk , the total similarity S _i for each image to be searched is expressed as follows.

  In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  Subsequently, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and for example, the images to be searched are displayed in the image display column 202 in FIG. (S1004). Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the total similarity, in practice, the search target is in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the query images and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the tenth embodiment, when the similarity between the query image and the search target image is determined using a plurality of types of feature amounts, the query image and Importance for each feature type can be specified for each query image so that arbitrary weighting can be performed according to the type of feature amount. By calculating the similarity with each query image for each feature amount, and weighting the similarity with each query image determined for each type of feature amount according to the specified importance, Since the intention can be accurately expressed as a search condition, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity, so that the search reflecting the user's intention is performed. It is possible to obtain an image of the fruit.

  In the description of the tenth embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Also, the types of feature values are not limited to the three types of color, structure, and texture, and any kind of combination of feature values may be used.

[Embodiment 11]
The image search device according to the eleventh embodiment is such that the device side can capture the user's intention for the search in the image search device of the eighth embodiment. The image search processing by the image search apparatus according to the eleventh embodiment is performed by using the OR specification button 206 instead of the AND specification button 205 in the search window 200d of FIG. 11 used for the description in the eighth embodiment (fourth embodiment). It is executed when specified.

  FIG. 21 is a flowchart showing image search processing according to the eleventh embodiment. The user designates a plurality of inquiry images in each image designation column shown in FIG. 11, and selects the OR designation button 206 to designate image search processing based on the OR condition (S1101). Assume that the user designates inquiry images A, B, and C as shown in FIG.

  Thereafter, when the user designates the search button 201 in FIG. 11, the U / I unit 151 inputs the query images A, B, and C to the feature extraction engine 152, and the feature extraction engine 152 Processing for extracting three types of feature values of color, structure, and texture from C is executed (S1102).

  Subsequently, the feature extraction engine 152 obtains the distance (similarity) between the query images A, B, and C for each of the extracted three types of feature amounts, and performs each query based on the obtained distance for each feature amount. The importance for each feature amount between the images A, B and C is determined (S1103).

  Specifically, the feature extraction engine 152 obtains a distance between features of a plurality of inquiry images designated by the user for each of the three types of feature amounts. If the variance of the distance obtained for each feature quantity is large, it means that the user is not aware of the feature quantity, and if the variance is small, it can be judged that the user attaches importance to the feature quantity. . Therefore, the importance is set to be small when the feature amount is large and the importance is set to be large when the variance is small. For example, the importance of each feature amount can be determined using Equation 7 shown in the seventh embodiment.

  Thereafter, the feature extraction engine 152 passes the feature amount of each inquiry image extracted for each of the three types of feature amounts and the corresponding importance to the matching engine 153.

  The matching engine 153 obtains the similarity to each of the query images A, B, and C using three types of feature amounts for each search target image registered in the image DB 108, and for each obtained similarity Weighting according to importance, and selecting the similarity indicating the highest similarity from the similarities to the query images A, B, and C, and executing the process of obtaining the overall similarity of the search target images (S1104).

  The processing in step S1104 will be specifically described taking one search target image as an example. The matching engine 153 obtains the similarity between the search target image and the query image A using the color feature amount. That is, the matching engine 153 determines the distance between the image to be searched and the query image A plotted in the feature amount space shown in FIG. Find using. Subsequently, the matching engine 153 performs weighting on the obtained distance according to the importance determined by the feature extraction engine 152 for the color feature amount. Further, the matching engine 153 obtains the distance using the structural feature amount and the texture feature amount for the search target image and the query image A, and the structure determined by the feature extraction engine 152 for each obtained distance. The weighting is performed according to the importance of the feature quantity and the importance of the texture feature quantity.

  Similarly, the matching engine 153 obtains the distance between the image to be searched and the inquiry image B for each of the three types of feature amounts, and sets the importance determined for each type of feature amount for each of the obtained distances. Perform the corresponding weighting. Further, the distance between the image to be searched and the inquiry image C is determined for each of the three types of feature amounts, and the determined distances are weighted according to the importance determined for each type of feature amount.

  After that, the matching engine 153 selects the shortest distance from the distances between the search target images obtained for each of the three types of feature amounts and the query images A, B, and C to obtain the total similarity. In this way, the matching engine 153 obtains the total similarity for each image to be searched, and outputs the total similarity for each image to be searched to the U / I unit 151.

  The U / I unit 151 refers to the total similarity for each image to be searched by the matching engine 153, and displays, for example, the search target images in the image display column 202 in FIG. 11 in descending order of the total similarity (S1105). . Here, since the distance between the feature amounts of the image to be searched and any one of the query images A, B, and C is used as the total similarity, in practice, the search target is in ascending order of the value of the total similarity. An image is displayed in the image display field 202. That is, the overall similarity has a value of 0 or more, and when the overall similarity is 0, it means that the feature amount of any one of the query images and the feature amount of the search target image are equal.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also.

  As described above, according to the image search device according to the eleventh embodiment, when the similarity between the query image and the search target image is obtained using a plurality of types of feature amounts, the feature amount is compared with the obtained similarity. The degree of similarity between each query image is calculated for each type of feature quantity so that arbitrary weighting can be performed according to the type, and between each query image based on the obtained degree of similarity for each type of feature quantity The degree of importance for each type of feature quantity is determined, and for each search target image, the similarity to each query image is obtained for each type of feature quantity, and each query image obtained for each type of feature quantity is obtained. By determining the importance of each type of feature amount between the query images specified by the user on the device side, by weighting the degree of similarity with the degree of importance for each type of feature amount determined, Determining the degree of importance against the similarity between images In order to, it is possible to return search results that match the user's intention. Therefore, the search process as desired by the user can be realized, and the search result desired by the user can be obtained.

  In the description of the eleventh embodiment, three images are used as inquiry images. However, the number of inquiry images is not limited to three. Also, the types of feature values are not limited to the three types of color, structure, and texture, and any kind of combination of feature values may be used.

[Embodiment 12]
The image search apparatus according to the twelfth embodiment makes it possible to perform an image search from a viewpoint different from those of the first to eleventh embodiments. Note that in the image search device according to the twelfth embodiment, the same reference numerals are used for the configurations already described in the first embodiment, and detailed description thereof is omitted, and common points regarding the image search processing are also described as appropriate. Is omitted.

The image search apparatus according to the twelfth embodiment enables the user to specify the roughness level (the fineness of the viewpoint) of the feature amount used during the search, and the feature amount corresponding to the roughness level specified by the user Is used to obtain the similarity between the query image and the search target image. Here, the roughness of the feature amount is defined as indicating the degree to which the feature amount extracted from the image faithfully represents a predetermined feature of the image to be extracted. So, in the following,
1. Feature amount extraction processing Processing executed by the image search device according to the twelfth embodiment will be described in the order of image search processing.

1. Feature amount extraction processing Here, a plurality of feature amounts having different roughness are extracted in advance for an image to be searched, and among query images, a query image from which no feature amount is extracted is featured at the time of search execution. We will extract the quantity. The timing for extracting the feature amount from the search target image is set when the search target image is registered in the image DB 108 as described in the first embodiment. A process for extracting feature amounts from each image will be described with reference to FIG.

  The feature extraction engine 152 of FIG. 2 inputs an image to be registered via the U / I unit 151 (S11), and executes a process of extracting a feature amount from the input image (S12).

  In the twelfth embodiment, for example, the feature extraction engine 152 calculates the roughness of an image from the image based on an extraction criterion that defines stepwise the number of divisions indicating how many images are to be divided as the roughness level of the feature amount. Different feature values can be extracted. Specifically, for example, the following feature level roughness levels are defined as extraction criteria, and the feature extraction engine 152 determines different roughness for the same feature (color, structure, etc.) from the image input in step S11. The feature amount of the level is extracted. Note that the extracted feature quantities differ only in roughness, and the types of feature quantities (for example, color, structure, etc.) are the same.

Level 1: As shown in FIG. 22A, feature value 1 is extracted from the entire image.
Level 2: As shown in FIG. 22B, the image is divided into four to generate divided images A to D, and feature amounts A to D are extracted from the divided images A to D, respectively. A feature amount 2 (a feature amount group) composed of feature amounts A to D is extracted from the image.
Level 3: As shown in FIG. 22C, the image is divided into eight to generate divided images A to H, and feature amounts A to H are extracted from the divided images A to H, respectively. A feature amount 3 (a feature amount group) composed of feature amounts A to H is extracted from the image.
Level 4: As shown in FIG. 22 (d), an image is divided into 16 to generate divided images A to P, and feature amounts A to P are extracted from the divided images A to P, respectively. A feature amount 4 (a feature amount group) composed of feature amounts A to P is extracted from the image.

  Here, the process of extracting the feature quantities of levels 1 to 4 from the image will be specifically described by taking as an example the process of extracting the structure (edge) and color (color histogram) of the image as the feature quantity.

(1) Structure (1) Level 1
For example, edge information is extracted from the image input in step S11 using an edge extraction filter such as sobel. From the extracted edge information, the edge position, edge strength and direction can be obtained. The edge intensity is binarized with an appropriate threshold value, and edge image data of edge components in the vertical and horizontal directions is obtained. As an example, edge image data of edge components in the vertical and horizontal directions obtained from the image shown in FIG. 23 are shown in FIGS.

24A to 24D, assuming that the edge portion is a black pixel and the others are white pixels,
• Extraction of edge information from each edge image data such as the ratio of black pixels to the entire image, the center of gravity of the black pixels, the dispersion of the black pixels, the average of the sizes of the connected rectangles of the black pixels This is the feature amount 1 of the structure of the entire image.

▲ 2 ▼ Level 2
The edge image data shown in FIGS. 24A to 24D is divided into divided images A to D (four divisions) as shown in FIG. 22B, and each divided image A to D is changed to (1). The described edge information is extracted to obtain the feature amounts A to D of the structure in units of divided images. As a result, the feature amount 2 composed of the feature amounts A to D can be extracted from the image.

(3) Levels 3 and 4
For level 3, the edge image data in FIGS. 24A to 24D is divided into divided images A to H as shown in FIG. 22C, respectively (8 divisions). As shown in FIG. 22 (d), the image is divided into divided images A to P (16 divisions), and from the feature quantity 3 and the feature quantities A to P each consisting of the feature quantities A to H in the same manner as in level 2. The feature quantity 4 can be extracted from the image.

(2) Color Next, processing for extracting the color (color histogram) of the image as a feature amount will be described.

▲ 1 ▼ Level 1
An appropriate color space (for example, Lab, Luv, HSV, etc.) is selected and divided into a plurality of regions, and it is examined which region in the color space each pixel of the image input in step S11 corresponds to. The feature quantity 1 for the entire image shown in FIG. 22A is extracted by normalizing the number of pixels with the number of pixels of the entire image.

▲ 2 ▼ Level 2
An appropriate color space (for example, Lab, Luv, HSV, etc.) is selected and divided into a plurality of regions, and the image input in step S11 is divided into divided images A to D as shown in FIG. (4 divisions). Then, it is possible to extract the feature amount A by examining which region in the color space each pixel of the divided image A corresponds to and normalizing the number of pixels for each region with the number of pixels of the entire divided image A. . Similar processing is performed on the other divided images B to D to extract the feature amounts B to D, respectively, and the feature amount 2 including the feature amounts A to D is extracted from the image input in step S11.

(3) Level 3 and Level 4
For level 3, the image input in step S11 is divided into divided images A to H as shown in FIG. 22C (eight divisions), and for level 4, the divided images are shown as shown in FIG. A feature amount 3 consisting of feature amounts A to H and a feature amount 4 consisting of feature amounts A to P are extracted from the image in the same manner as in the case of level 2 (16 divisions).

  In this way, by extracting feature values from the same image by changing the number of divisions, for example, an image is entirely blue, but blue and a little red are mixed in detail. Thus, it is possible to extract a feature amount depending on the fineness of the viewpoint. For example, it is possible to extract a feature amount including information indicating at which position in the image centered on blue that red is dispersed.

  In this manner, after extracting the feature amount of each level from the image input in step S11, the feature extraction engine 152 extracts the image in association with the original image registered in the image DB 108 as shown in the flowchart of FIG. The obtained feature amount is registered in the image feature DB 109 (S13).

  Here, the description has been made by taking as an example the extraction of the image structure and the color feature as the feature amount, but other types of features may be extracted as the feature amount. For example, even when an image is divided into four, various methods other than the dividing method shown in FIG. 22B are conceivable (for example, an image is divided into four vertically or horizontally). Therefore, the division method can be arbitrarily set.

2. Image Search Processing FIG. 25 is an explanatory diagram showing an example of a search window 200g used in the image search device according to the twelfth embodiment. The search window 200g includes, in addition to the image designation columns A to C, the search button 201, and the image display column 202, a feature designation column 209 that designates the type of feature amount used for the search, and the types designated in the feature designation column 209. And a roughness level designation field 210 for designating which level (roughness) of the feature quantity is to be used. In the feature designation field 209, for example, one or both of the structural feature quantity and the color feature quantity can be designated. In the roughness level designation field 210, for example, at least one of levels 1 to 4 can be designated. Shall be possible. The levels that can be designated in the roughness level designation column 210 correspond to the levels of the feature quantity extraction criteria described above.

  If no feature quantity is designated in the feature designation field 209, the search processing is performed using both the structure quantity and the feature quantity, and any level in the roughness level designation field 210 is assumed. If neither is specified, the search process is performed using the feature amounts of all levels. In addition, in order to be able to specify all of the available types of feature quantities, a button “all” may be prepared in the feature specification field 209. Similarly, a button “all” may be prepared in the roughness level designation field 210 so that all levels can be designated.

  Subsequently, an image search process according to Embodiment 12 will be described. Here, since different processing is performed depending on how the search condition is specified in the search window 200g, the outline of the image search processing is first described with reference to the flowchart of FIG. Each search condition will be explained separately for each case.

(1) Outline of Image Search Process First, an outline of the image search process according to the twelfth embodiment will be described. The user designates at least one inquiry image in the image designation field shown in FIG. 25, and in the feature designation field 209 and the roughness level designation field 210, the type of feature quantity used at the time of search and the rough value of the feature quantity to be used. A level is designated (S1201).

  Subsequently, when the user designates the search button 201 in FIG. 25, the U / I unit 151 inputs the inquiry image, the type of feature amount, and the type of roughness level to the feature extraction engine 152, and the feature extraction engine 152 The feature quantity is extracted from the inquiry image in accordance with the designated feature quantity type and roughness level (S1202). However, when the feature amount has already been extracted from the inquiry image designated by the user, the processing in step S1202 can be skipped. Further, when the feature amount is not extracted for the search target image, the feature amount may be extracted at this point. Thereafter, the feature extraction engine 152 passes the extracted feature amount of the query image to the matching engine 153 (maps to the feature space of FIG. 4).

  The matching engine 153 executes a process for obtaining the similarity with the query image using the feature amount of the level specified in the search window 200g for each image to be searched registered in the image DB 108 (S1203). That is, the matching engine 153 determines the distance between the image to be searched and the query image plotted in the feature amount space shown in FIG. 4 based on the feature amount of the corresponding level for each feature. Obtained using an equation.

  The feature amount F is normally expressed as a vector F = {f1, f2, f3,..., Fn}, and the feature amounts extracted from the images a and b are represented by the following formula 11, respectively. The differential distance formula D (Fa, Fb) is generally used as the distance definition formula.

  This distance formula can also be used in the case of “structure” and “color” feature quantities. However, when the “ratio of black pixels to the whole image” in “structure” is used as a feature quantity as a histogram, An appropriate similarity can be obtained when there are enough edges, but an appropriate similarity cannot be obtained when there are few edges in the image.

  In the sense of similarity to human edges, for example, an image with no edges is slightly different from an image with little edges, but if the image has enough edges, the amount is slightly different. But the difference is insensitive. Therefore, this problem can be solved by normalizing the histogram with the number of all edge pixels (normalized distance equation) in order to increase the accuracy of the distance than the difference distance equation.

That is, the elements of the histogram are given by the following formula.
foi = fi / N
Note that foi is a normalized histogram element. Further, the total number of pixels N is given by the equation (12).

Note that normalization is not performed when the vector elements are all 0 and S = 0. That is, the normalized distance formula is given by the formula (13).

  When the number of edges in the image is small, the above formula is effective, but when the number is large, the conventional histogram can obtain a more appropriate similarity. Therefore, the formula of formula 14 is devised so that the formula becomes close to the conventional formula when there are many edges, and the formula becomes close to the above formula when there are few edges.

  Note that the value for normalization is changed as shown in the following equation 15 depending on the total number of pixels.

ｋは定数であり、対象とする画像集合によって適宜指定する。

k is a constant and is appropriately specified depending on the target image set.

  In addition to performing the above calculation at the time of search, it is converted into a vector obtained by normalization or semi-normalization at the time of feature extraction in advance and registered in the database. It is also possible to perform a high-speed search.

  Although the above-mentioned semi-normalized expression can calculate an appropriate similarity regardless of the number of edges in the image, there is a problem that it takes time because the expression is complicated when calculating at the time of retrieval. Therefore, the conventional differential distance formula and the normalized distance formula are selected and calculated according to the number of edges and the required search speed.

For example, cases can be classified as follows.
1) When the edge ratio of both images is greater than or equal to a threshold A Select the differential distance formula 2) When either one is below the threshold A a) When high speed is required Select the differential distance formula b) High speed When is not required Use normalized distance formula

  However, in this method, different formulas are selected depending on the case, so that the resulting metric space does not satisfy the distance axiom. If it is necessary to satisfy the distance axiom, a semi-normalized expression is desirable.

  Subsequently, the U / I unit 151 refers to the similarity with the query image obtained for each image to be searched by the matching engine 153, and displays the images to be searched in the order of the similarity, for example, in the image display of FIG. It is displayed in the column 202 (S1204). However, since the distance between the feature amounts of the search target image and the query image is used as the similarity here, the search target images are actually displayed in the image display column 202 in ascending order of the similarity value. Will be. That is, the similarity has a value of 0 or more. When the similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

(2) Specific Example of Image Search Process Next, while corresponding to the flowchart of FIG.
(1) When one query image, “color” as the feature quantity and “level 1” as the roughness level are designated (2) One query image, “color” as the feature quantity and “level 2” as the roughness level ”Is designated (3) One inquiry image,“ color ”is designated as the feature quantity, and two or more levels are designated as the roughness level. The image search processing according to the twelfth embodiment will be specifically described in the order in which a plurality of inquiry images are designated.

(1) When “Color” is specified as one inquiry image and feature level and “Level 1” is specified as the roughness level In the search window 200g shown in FIG. It is assumed that “level 1” is designated as the “color” and roughness level (S1201).

  As shown in FIG. 22A, the feature extraction engine 152 extracts “feature amount 1” that is a feature amount corresponding to “level 1” for the feature amount “color” for the entire inquiry image (S1202). ).

  Then, the matching engine 153 obtains a similarity to the query image for each search target image based on the query image extracted by the feature extraction engine 152 and the feature quantity 1 of the search target image (S1203). Specifically, the distance between the image to be searched and the query image plotted in the feature amount space shown in FIG. 4 based on the corresponding feature amount 1 is used using a distance definition formula defined for each feature. Ask.

  Thereafter, the U / I unit 151 refers to the similarity with the query image obtained for each image to be searched by the matching engine 153, and displays the images to be searched in the order of, for example, the highest similarity (smallest value). 25 images are displayed in the image display field 202 (S1204).

In this way, by selecting a feature value of roughness such as level 1 and performing an image search process, for example, in the case of a search using a color feature value, Search processing such as searching for a blue image as a whole can be executed.

  Although a detailed description is omitted, even if “structure” or another type of feature quantity is specified as the feature quantity, the search process can be performed by the same process.

(2) When one query image, “color” as the feature quantity and “level 2” as the roughness level are designated In the search window 200g shown in FIG. It is assumed that “level 2” is designated as the “color” and roughness level (S1201).

  As shown in FIG. 22B, the feature extraction engine 152 divides the inquiry image into divided images A to D, and extracts feature values A to D of “color” from the divided images A to D, respectively (S1202). ). As a result, the feature amount corresponding to “level 2” for the feature amount “color” and “feature amount 2” including the feature amounts A to D is extracted from the inquiry image.

  Then, the matching engine 153 obtains the similarity with the query image for each search target image based on the query image extracted by the feature extraction engine 152 and the feature quantity 2 of the search target image (S1203). Specifically, the similarity between the divided images A, the divided images B, the divided images C, and the divided images D of the search target image and the inquiry image is obtained. More specifically, based on the feature amount 2 extracted by the feature extraction engine 152, the divided images A, the divided images B, the divided images C, and the divided images plotted in the feature amount space shown in FIG. The distance between D is obtained using a distance definition formula defined for each feature. Here, since the similarity is obtained for each of the divided images A to D, four similarities are obtained.

Subsequently, the matching engine 153 determines the similarity between the search target image and the query image using the obtained four similarities. As a method for determining the similarity, for example, the following method can be considered.
(2) -1 The similarity between the search target image and the query image is determined by the sum of squares of the four similarities.
{Circle around (2)}-2 The sum of the four similarities is used to determine the similarity between the search target image and the query image (see AND search).
(2) -3 By selecting the highest (smallest value) similarity among the four similarities, the similarity between the search target image and the query image is determined (see OR search).
Note that it may be possible to specify in the search window 200g which method (2) -1 to (2) -3 is to be used.

  Thereafter, the U / I unit 151 refers to the similarity with the query image obtained for each image to be searched by the matching engine 153, and displays the images to be searched in the order of, for example, the highest similarity (smallest value). 25 images are displayed in the image display field 202 (S1204).

  In this way, by selecting the feature quantity of roughness such as level 2 and performing the image search process, for example, when performing a search using the color feature quantity, Thus, it is possible to execute a search process such as searching for an image in which red is mixed in a portion where blue is present as a whole. In other words, search processing based on which color is located at which position in the image is possible, and search processing is performed from a fine viewpoint that cannot be performed by search using the feature amount (level 1) extracted for the entire image. It can be carried out.

  Although detailed description is omitted, even when “structure” or another type of feature quantity is designated as the feature quantity, the search process can be performed by the same process. Further, even when the roughness levels 3 and 4 are designated, the search process can be performed by the same process as the process of the level 2 described above, and thus the description thereof is omitted here.

(3) When one query image, “color” as the feature quantity and two or more levels are specified as the roughness level Here, as an example when two or more levels are specified as the roughness level, The image search process will be described based on the case where all the roughness levels are designated. Therefore, it is assumed that in the search window 200g shown in FIG. 25, the user designates one inquiry image, “color” as a feature amount, and all roughness levels (S1201). As described above, when none of the levels is designated in the roughness level designation column 210 or when all levels are designated, all the roughness levels are designated.

  As shown in FIGS. 22A to 22D, the feature extraction engine 152 extracts feature quantities 1 to 4 according to levels 1 to 4 from the query image specified in the search window 200g (S1202). ). The feature quantity 1 corresponding to level 1 can be extracted by the method described in the section (1), and the feature quantities 2 to 4 corresponding to levels 2 to 4 can be extracted by the method described in the section (2). .

  Then, the matching engine 153 uses the query image extracted by the feature extraction engine 152 and the feature amounts 1 to 4 of the search target image to obtain a similarity with the query image for each search target image (S1203). . Here, the similarity between the search target image and the query image is obtained for each feature amount (level), and as a result, four similarities with respect to the query image are obtained for each search target image. The process for obtaining the similarity between the search target image and the query image using the feature amount 1 (level 1) is the same as the process described in the item (1). The processing for obtaining the similarity between the search target image and the query image using the feature amounts 2 to 4 (levels 2 to 4) is the same as the processing described in the item (2).

Subsequently, the matching engine 153 determines the final similarity with the query image using the four similarities with the query image obtained according to each level for each image to be searched. As a final similarity determination method, for example, the following method can be considered.
(3) -1 The final similarity with the query image is determined by the sum of squares of the four similarities (deformation of AND search).
(3) -2 The final similarity with the query image is determined by summing the four similarities (see AND search).
(3) -3 By selecting the highest (smallest value) similarity from the four similarities, the final similarity with the query image is determined (see OR search).

  When the final similarity to the inquiry image is determined by the methods (3) -1 and (3) -2, for example, the whole image is blue and a certain portion is mixed with red. It is possible to execute a search process (AND search) such as searching for a current image. In other words, since feature values of all (two or more) roughness levels are used, a desired image can be searched with higher accuracy than when searching using only feature values of a specific level. it can. Also, in the case of determining the final similarity with the inquiry image by the method of (3) -3, for example, red is mixed in an overall blue image and an overall blue portion. It is possible to execute a search process (OR search) such as searching for any of the existing images.

  It should be noted that any one of methods (3) -1 to (3) -3 may be used in the search window 200g.

  Thereafter, the U / I unit 151 refers to the similarity with the query image obtained for each image to be searched by the matching engine 153, and displays the images to be searched in the order of, for example, the highest similarity (smallest value). 25 images are displayed in the image display field 202 (S1204).

  As described above, by selecting a plurality of feature quantities having different roughnesses and performing the image search process, the viewpoint for the search can be compared with a case where one feature quantity having a specific roughness is used. Search processing can be executed by various different methods. Although detailed description is omitted, even when “structure” or another type of feature quantity is designated as the feature quantity, the search process can be performed by the same process.

(4) When multiple types of feature quantities are specified When multiple types of feature quantities are specified in the feature specification field 209, the feature extraction engine 152 inquires about the feature quantities of the types and levels specified in the search window 200g. Each is extracted from the image (S1202). For example, when structure and color feature quantities are specified, the feature extraction engine 152 extracts feature quantities having a roughness level corresponding to the structure feature quantities from the query image, and also applies levels corresponding to the color feature quantities. Are extracted (see FIG. 22).

  Then, the matching engine 153 obtains the similarity to the query image for each type of feature amount for each image to be searched, and finally uses the similarity for each type of feature amount to obtain the final query image. The similarity is determined (S1203).

As a final similarity determination method, for example, the following method can be considered.
(4) -1 The final similarity with the query image is determined based on the sum of squares of the similarity obtained for each type of feature quantity (deformation of AND search).
{Circle around (4) -2} The final similarity with the query image is determined by summing up the similarities obtained for each type of feature amount (see AND search). When this method is used, an AND search according to a plurality of types of feature amounts can be executed.
(4) -3 The final similarity with the query image is determined by selecting the highest similarity (smallest value) from the similarities obtained for each type of feature amount (see OR search) ). When this method is used, an OR search corresponding to a plurality of types of feature amounts can be executed.
Note that it may be possible to specify in the search window 200g which method (4) -1 to (4) -3 is to be used.

  Further, for example, when one query image is specified and the feature amount of structure and color and roughness levels 1 and 2 are specified, the feature amount of the structure of level 1 is obtained for each image to be searched. Similarity, similarity obtained using level 2 structure features, similarity obtained using level 1 color features, and similarity obtained using level 2 color features It will be. In such a case, all the similarities obtained may be summed to determine the final similarity with the query image (the method of (4) -2 (4) -1). Reference), the highest similarity (smallest value) may be selected from the obtained similarities, and the final similarity with the inquiry image may be determined (method (4) -3). reference). Alternatively, the similarity of each level is totaled for each type of feature quantity, or the highest (smallest value) similarity is selected from the similarity levels of each level, and each feature quantity type is selected. After obtaining the similarity to the query image, the final similarity to the query image may be determined using any one of the methods (4) -1 to (4) -3 described above.

  Thereafter, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and, for example, selects the images to be searched in descending order of the overall similarity (value is small). The image is displayed in the image display field 202 (S1204).

(5) When a plurality of query images are designated When a plurality of query images are designated, the feature extraction engine 152 extracts the feature quantity of the type and level designated in the search window 200g from each query image ( S1202). Then, the matching engine 153 obtains the similarity to each inquiry image for each image to be searched as described in any one of items (1) to (4), and then combines the images to be searched. The similarity is obtained (S1203).

As a method for obtaining the overall similarity, for example, there are the following methods.
(5) -1 For each image to be searched, the sum of squares of the similarity with each query image is set as the total similarity (deformation of AND search).
{Circle around (5)}-2 For each image to be searched, the similarity with each query image is totaled to obtain the total similarity (see AND search). When this method is used, an AND search can be executed.
(5) -3 For each image to be searched, the highest similarity (smallest value) is selected from the similarities to the respective query images to obtain the total similarity (see OR search). When this method is used, an OR search can be executed.

  It should be noted that any one of methods (5) -1 to (5) -3 may be specified in the search window 200g.

  Thereafter, the U / I unit 151 refers to the overall similarity obtained for each image to be searched by the matching engine 153, and, for example, selects the images to be searched in descending order of the overall similarity (value is small). The image is displayed in the image display field 202 (S1204).

  As described above, according to the image search device according to the twelfth embodiment, at least one inquiry image is specified, and the roughness level of the feature amount extracted from the image is gradually increased from coarse to fine. Based on the defined extraction criteria, specify at least one roughness level of the feature quantity used for the search, and based on the extraction criteria, specify at least the roughness level specified from the query image and multiple search target images The feature amount corresponding to is extracted at a predetermined timing, the similarity with the query image is obtained for each search target image using the feature amount according to the specified roughness level, and for each search target image. Since the search result is output based on the obtained similarity, it is possible to execute a plurality of types of search processes with different search accuracy. In other words, since the user can specify at least one level of roughness of the feature amount used for the search, the fineness of the user's viewpoint for the search can be expressed as a search condition, and the user's intention can be accurately set as the search condition Can be expressed. Therefore, it is possible to perform image search processing as desired by the user.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also. Further, although the type of feature amount is the structure and color, only the structure or only the color may be used, or another type of feature amount may be used. Further, when a plurality of inquiry images or a plurality of types of feature quantities are designated, the importance may be designated as described in other embodiments.

  In the twelfth embodiment, the roughness of the feature amount is defined depending on how many images are divided. However, not only the image but also an object existing in the image is a feature amount extraction target, You may decide to define the roughness of the feature-value of an object unit by how many parts are divided | segmented. The search process when an object existing in an image is also a feature amount extraction target is the same as the search process described above, and thus detailed description thereof is omitted here.

  Further, as described above, it is possible to realize the following image search processing by using the feature amount extraction for each divided image obtained by dividing the image. Here, for convenience of explanation, a modification of the image search device according to the twelfth embodiment will be described using an example in which an image is divided into four.

(Modification 1)
In the above-described twelfth embodiment, the similarity between the inquiry image and the search target image is obtained by comparing the inquiry image and the divided images of the search target image in a one-to-one correspondence to obtain the similarity. On the other hand, in the first modification, for example, as shown in FIG. 27, one of the divided images of the inquiry image is designated, and the similarity between the designated divided image and each divided image of the search target image is obtained. Thus, an image having a divided image similar to the specific divided image of the inquiry image can be found from the search target images.

(Modification 2)
In the second modification, for example, as shown in FIG. 28, an image including the query image is searched from the search target images by obtaining the similarity between the entire query image and each divided image of the search target image. Is to be able to.

  Although detailed description is omitted, by enabling such search processing as in the first modification and the second modification, it is possible to search for an image by various methods.

[Embodiment 13]
The image search device according to the thirteenth embodiment enables an image search from a different viewpoint from the above-described first to twelfth embodiments by enabling an image including an inquiry image to be found from among search target images. It is possible to do. Note that in the image search device according to the thirteenth embodiment, the same reference numerals are used for the configurations already described in the first embodiment, and detailed description thereof is omitted, and common points regarding the image search processing are also described as appropriate. Is omitted.

  Specifically, the image search device according to the thirteenth embodiment designates at least one inquiry image, and for each search target image, determines the similarity between a region of a predetermined size in the search target image and the query image. By repeatedly executing the processing to be obtained using a predetermined feature amount extracted in advance over the entire image to be searched, an image including an inquiry image can be found from the images to be searched.

  In the image search device according to the thirteenth embodiment, various methods are used for the process of obtaining the similarity between a predetermined size area in a search target image and a query image using a predetermined feature amount extracted in advance. Here, a technique as shown in FIGS. 29 and 30 is used. For example, as shown in FIG. 29, the query image is divided into four to generate divided images a to d, a predetermined type of feature amount is extracted from each divided image, and the search target image is divided into 16 parts. The divided images A to P are generated, and the same type of feature quantity as that of the query image is extracted from each divided image. However, the number of divisions of the query image and the search target image is not limited to 8 and 16, respectively, but may be a relationship that the number of query image divisions <the number of search target image divisions. Then, using the feature amount extracted from each divided image, as shown in FIGS. 30A to 30I, the similarity between the query image and a part of the search target image is sequentially obtained. FIG. 30 shows the image to be searched shown in FIG. 29, and the hatched portion in the image to be searched corresponds to an area compared with the query image. Note that if the number of divisions of the query image and the search target image is reversed (for example, the query image is divided into 16 and the search target image is divided into 8. That is, the number of query image divisions> division of the search target image Number), it is possible to search for an image included in the inquiry image.

  The image search process performed by the image search apparatus according to Embodiment 13 will be specifically described below. As a premise, it is assumed that at least an image to be searched has feature values extracted from each divided image as shown in FIG. The specific processing for extracting the feature amount is as described in the twelfth embodiment, and thus detailed description thereof is omitted here. Further, the type of feature quantity to be extracted may be any color, structure, or the like.

  FIG. 31 is a flowchart of image search processing according to the thirteenth embodiment, and FIG. 32 is an explanatory diagram showing an example of a search window 200h. The user designates at least one inquiry image in any of the image designation fields A to C shown in FIG. 32 (S1301). Here, in order to simplify the description, it is assumed that one inquiry image is designated.

  Subsequently, when the user designates the search button 201 in FIG. 32, the U / I unit 151 inputs an inquiry image to the feature extraction engine 152, and the feature extraction engine 152 extracts a predetermined type of feature amount from the inquiry image. (S1302). As shown in FIG. 29, the feature extraction engine 152 divides the inquiry image to generate divided images a to d, and extracts a predetermined type of feature amount from each divided image. However, when the feature amount has already been extracted from the inquiry image designated by the user, the processing in step S1302 can be skipped.

  Then, the matching engine 153 uses the feature amount extracted by the feature extraction engine 152 for each search target image registered in the image DB 108, and uses the similarity between a predetermined region in the search target image and the query image. Is performed over the entire search target image, and the similarity to the query image is obtained (S1303).

An example of the process executed in step S1303 will be described in detail with reference to FIG. For example, with reference to FIG. 30A, the matching engine 153 uses the feature amount of each divided image of the query image and the search target image extracted in advance by the feature extraction engine 152, and
The similarity between the divided image a of the inquiry image and the divided image A of the search target image The similarity between the divided image b of the inquiry image and the divided image B of the search target image The divided image c of the inquiry image and the search target The similarity between the divided image E and the divided image E of the inquiry image and the divided image F of the image to be searched are obtained. In other words, the distance between the divided images plotted in the feature amount space shown in FIG. 4 is obtained using the distance definition formula defined for each feature. Here, since the similarity is obtained for each of the four divided images, four similarities are obtained.

Subsequently, the matching engine 153 determines the similarity between the area of the search target image shown in FIG. 30A and the query image using the obtained four similarities. As a method for determining the similarity, for example, the following method can be considered.
(1) -1 The similarity between the area of the image to be searched and the query image is determined by the sum of squares of the four similarities.
{Circle around (1)}-2 The sum of the four similarities is used to determine the similarity between the image area to be searched and the query image (see AND search).
(1) -3 By selecting the highest (smallest value) similarity among the four similarities, the similarity between the area of the image to be searched and the query image is determined (see OR search).
Note that it may be possible to specify in the search window 200h which method (1) -1 to (1) -3 is to be used.

  The matching engine 153 also performs the above-described processing for the cases shown in FIGS. 30B to 30I, and determines the similarity between each region of the search target image and the query image.

  As shown in FIGS. 30A to 30I, as a result of obtaining the similarity between each region of the search target image and the query image, nine similarities with the query image are obtained for each search target image. Will be obtained. Therefore, the matching engine 153 executes a process of determining the final similarity with the query image using nine similarities for each image to be searched. As a final similarity determination method, for example, the following method can be considered.

{Circle around (2)}-1 The final similarity between the search target image and the query image is determined based on the sum of squares of the nine similarities (deformation of AND search).
(2) -2 The final similarity between the search target image and the query image is determined by adding the nine similarities (see AND search).
(2) -3 The final similarity between the search target image and the query image is determined by selecting the highest (smallest value) similarity among the nine similarities (see OR search). .
Note that it may be possible to specify in the search window 200h which method (2) -1 to (2) -3 is to be used.

  Returning to the description of the flowchart in FIG. 31, the U / I unit 151 refers to the final similarity with the query image obtained for each image to be searched by the matching engine 153, and performs a search, for example, in descending order of similarity. The target image is displayed in the image display field 202 of FIG. 32 (S1304). Here, since the distance between the feature quantities of the search target image and the query image is used as the similarity, the search target images are actually displayed in the image display column 202 in ascending order of the similarity value. become. That is, the similarity has a value of 0 or more. When the similarity is 0, it means that the feature amount of each query image and the feature amount of the search target image are equal.

  In the above description, only one specified query image is used and one type of feature amount is used for the search. However, a plurality of types of feature amounts can be used, and a plurality of query images are specified. You can also.

  Therefore, an example of processing when a plurality of types of feature amounts are used will be briefly described. As described above, the matching engine 153 obtains the similarity between the search target image and the query image for each feature quantity of the same type. Then, the matching engine 153 can determine the similarity to the query image for each image to be searched, using the similarity obtained for each of different types of feature amounts by the following method, for example.

(3) -1 The final similarity with the query image is determined based on the sum of squares of the similarity obtained for each type of feature quantity (deformation of AND search).
(3) -2 The final similarity to the inquiry image is determined by summing up the similarities obtained for each type of feature quantity. When this method is used, an AND search according to a plurality of types of feature amounts can be executed.
(3) -3 The final similarity with the query image is determined by selecting the highest similarity (smallest value) from the similarities obtained for each type of feature quantity. When this method is used, an OR search corresponding to a plurality of types of feature amounts can be executed.
Note that it may be possible to specify in the search window 200h which method (3) -1 to (3) -3 is to be used.

  An example of processing when a plurality of inquiry images are specified will be briefly described. As described above, the matching engine 153 obtains the similarity between the search target image and each query image. The matching engine 153 can determine the similarity of the search target image with respect to each query image for each search target image, for example, using the following method.

(4) -1 For each image to be searched, the sum of squares of the similarity with each query image is set as the total similarity of the images to be searched (deformation of AND search). By using this method, it is possible to search for an image including a plurality of images as an inquiry image.
(4) -2 For each image to be searched, the similarity with each query image is summed to obtain the total similarity of the images to be searched. When this method is used, an AND search using a plurality of query images can be executed. That is, it is possible to search for an image including a plurality of images as an inquiry image.
(4) -3 For each image to be searched, the highest similarity (with a small value) is selected from the similarities to each query image to obtain the total similarity of the images to be searched. When this method is used, an OR search using a plurality of inquiry images can be executed. That is, it is possible to search for an image including any one of a plurality of images as an inquiry image.
It should be noted that any one of the methods (4) -1 to (4) -3 may be specified in the search window 200h.

  As described above, according to the image search device according to the thirteenth embodiment, at least one query image is specified, and each search target image is specified as a region of a predetermined size in the search target image. In order to repeatedly execute the process of obtaining the similarity with the query image using a predetermined feature amount extracted in advance over the entire image to be searched, and to output a search result based on the similarity obtained for each image to be searched. An image including the inquiry image can be found from the search target images. Therefore, it is possible to provide one method for realizing an image search as intended by the user.

  As a result of referring to the image of the search result displayed in the image display column 202, if a narrowed search is desired, the image displayed in the image display column 202 is designated as an inquiry image and the same search process is performed. You can also. Further, when a plurality of inquiry images or a plurality of types of feature quantities are designated, the importance may be designated as described in other embodiments.

[Embodiment 14]
Next, the image classification device according to the present invention will be described. This image classification device classifies images using similarity between images, as in the image search device described above. By classifying images in advance, it is possible to search for a desired image by simply following the classification. By doing so, it is possible to easily search for an image desired by the user by a method different from the image search.

  The image classification device according to the fourteenth embodiment of the present invention classifies images by applying the image search processing described in the first embodiment. Therefore, description of points that are the same as those described in Embodiment 1 will be omitted as appropriate.

  FIG. 33 is a block configuration diagram of an image classification device according to the fourteenth embodiment. 33, the same components as those in the image search apparatus shown in FIG. 1 are denoted by the same reference numerals, and only different points will be described here. The image classification apparatus shown in FIG. 33 differs from the image search apparatus shown in FIG. 1 in that image classification software 301 for causing the CPU 101 (computer 100) to execute the function as the image classification apparatus of the present invention, and classification items (Folder described later), a classification condition for classifying images to be classified into classification items, and a classification tree DB 302 in which image information of the classification result is registered.

  34 is a block diagram showing processing executed in the image classification apparatus shown in FIG. 33, and corresponds to FIG. A user interface (U / I) unit 151 (corresponding to the setting means, specifying means, and display means of the present invention) displays a classification item setting screen, a classification result display screen, etc. for performing image classification for the user. Then, classification condition input, classification result output processing, and the like are executed. The feature extraction engine 152 executes a process of extracting image feature amounts from images registered in the image DB 108 (classification target images) and images set as classification criteria described later. Further, the matching engine 153 (corresponding to the calculation means, determination means, classification means, and search means of the present invention) obtains the similarity between the reference image and the image to be classified, and classifies the object based on the obtained similarity. A process for determining a classification item to be a classification destination of the image (for example, registering and classifying an image ID in a classification item registered in the classification tree DB 302) is executed.

Next, regarding the operation of the image classification apparatus having the above-described configuration,
(1) Classification item setting processing (2) Image classification processing will be described in this order. Here, a plurality of images registered in the image DB 108 are classified images. Further, the feature amounts used when classifying the images are the color, structure and texture of the image. From the images to be classified in the image DB 108, the types of features have already been obtained by the processing described in the first embodiment with reference to FIG. It is assumed that the feature amount is extracted.

(1) Classification Item Setting Processing FIG. 35 is an explanatory diagram showing an example of an image classification window 350a for performing image classification item setting processing and display of images after classification. When the display of the image classification window 350a is designated by the user, the U / I unit 151 in FIG. 34 displays an image classification window 350a as shown in FIG. Processing such as classification condition setting is executed.

  FIG. 35 shows a state in which items for managing images (hereinafter referred to as “folders”) are already set. Here, only the image DB folder 351 corresponding to the root exists. Assuming that this is the case, a process for setting each folder will be described.

  An image DB folder 351 in FIG. 35 is a folder for managing all the images registered in the image DB 108. Note that by changing the setting of the image DB folder 351, the image to be classified can be changed from the image in the image DB to another image. For example, it is also possible to specify and classify the images obtained as a result of retrieval by the image retrieval apparatus described in the first to thirteenth embodiments.

  The user sets a new folder as a classification destination for classifying the images in the image DB 108 under the image DB folder 351. For example, the user selects “create folder” from a menu (not shown), and creates a folder 352 under the image DB folder 351.

  Then, the user selects the folder 352 and sets “classification condition setting” from a menu (not shown) in order to set a classification condition for classifying the images to be classified (images in the image DB 108) in the folder 352. Then, the classification condition setting window 360a shown in FIG. 35 is displayed.

  The classification condition setting window 360a has substantially the same configuration as that of a part of the search window 200a shown in FIG. 5, and the classification condition setting window 360a has the same classification method as the search condition described in the first embodiment. Can be set. The classification condition setting window 360a includes, for example, a plurality of reference images (image search queries) used as a reference for classifying images to be classified for each feature amount of color, structure, and texture used when classifying images. Image designation fields A, B, and C for designating (corresponding to an image), and a setting button 361 for designating classification condition settings for a desired folder.

  As an image to be designated as a reference image, for example, an image read by a scanner, an image taken by a digital camera, an image downloaded via the network 112, an image stored in advance on the hard disk 110, or registered in the image DB 108 Any image or the like may be used.

  As described in the first embodiment, the user associates, for example, three images with each type of feature amount, and designates them as the reference images A, B, and C in the image designation fields A, B, and C, respectively. Although not shown, a threshold value of similarity between the reference image and the search target image can be set for each folder from the classification condition setting window 360a. As will be described later, an image to be classified whose similarity with a reference image set in a folder exceeds a threshold is classified into that folder. Here, a threshold value is set for each folder, but one threshold value may be set for the entire folder.

  When the user selects the setting button 361, the U / I unit 151 sets the designated reference image and threshold value in the folder 352 as a classification condition. In the case where the feature amount is not extracted from the image set as the reference image, the feature extraction engine 152 may perform the process of extracting the feature amount when setting the classification condition in the folder. . Of course, a necessary feature amount may be extracted from each reference image when performing the process of classifying the images.

  The user creates folders (for example, folders 354 and 355 with respect to the folder 353) for further classifying the images classified into the created folder (for example, the folder 353) like the folders 354 to 357, Similarly, classification conditions can be set.

  In this way, a plurality of folders 352 to 358 as shown in FIG. 35 are set as classification items for classifying the images to be classified. The U / I unit 151 registers the created folder and the corresponding classification condition as folder information in the classification tree DB 302 together with information related to the hierarchical structure of the folder.

  As described in the first embodiment, it is possible not only to specify a plurality of reference images for each folder, but also to specify each inquiry image corresponding to each type of feature amount used when classifying the images. It is possible to express one virtual reference image that the user considers appropriate, and to accurately express the user's intention for classification as a classification reference. Therefore, it is possible to obtain an image of the classification result reflecting the user's intention by the classification process described later.

(2) Image Classification Processing FIG. 36 is a flowchart showing image classification processing according to the fourteenth embodiment. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the classification target image using the corresponding type of feature amount for each of the folders 352 to 358. Then, a process for obtaining the total similarity of the images to be classified by summing the obtained similarities is executed (S1401).

  The processing in step S1401 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360a in FIG. 35) set in the folder 352 and the image to be classified as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Find using. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on the feature amount of the structure, and calculates the distance between the image to be classified and the reference image C based on the feature amount of the texture. Ask.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C, and sets the total distance as the total similarity (see Equation 1). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity of the classification target images obtained in step S1401 exceeds a threshold value (S1402). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose total similarity of the images to be classified exceeds a threshold value (S1403). That is, when the total similarity of the images to be classified exceeds the threshold, the matching engine 153 determines the folder as a classification destination of the images to be classified, and folder information of the corresponding folder registered in the classification tree DB 302. The ID of the image to be classified (assumed to be set in advance) is registered in the classification tree DB 302 in association with.

  By executing the processes in steps S1401 to S1403 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. Note that the images to be classified may be classified into two or more folders according to the total similarity obtained for each folder.

  FIG. 37 is an explanatory diagram showing the state of the image classification window 350a after the images in the image DB 108 are classified into folders 352 to 358. In each of the folders 352 to 358, representative images selected by the matching engine 153 are displayed. When the user selects a specific folder, the images classified into the folder are displayed in a list on the right side of the image classification window 350a.

  As described above, according to the image classification device according to the fourteenth embodiment, the classification items are set, and each type used when obtaining the similarity between the reference image and the classification target image for each of the set classification items. For each feature item, one reference image is set, and for each classification item, the similarity between each set reference image and the classification target image is obtained using the corresponding type of feature amount, and the obtained similarity To determine the total similarity of the images to be classified, and based on the total similarity of the images to be classified obtained for each classification item and a preset classification destination determination criterion, By determining the classification item, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the device, and the user's intention for the classification can be reliably transmitted to the device. It is possible, it is possible to obtain a classification result that accurately reflects the intention of the user for the classification, it is possible to realize the classification process as the user desires.

  For example, even when an appropriate image does not exist as a reference image, an image having a color atmosphere similar to a desired image or an image having an object shape atmosphere similar to a desired image as a classification result is designated as the reference image. Can be used as one inquiry image.

  Here, three types of feature quantities of color, structure, and texture are used, but these are merely examples, and are not intended to limit the types of feature quantities.

  In addition, as shown in FIG. 37, the representative image is displayed for each folder, but instead of this, the folder name may be displayed or the reference image set for the folder may be displayed.

  Further, although detailed description is omitted, when a new image is registered in the image DB 108, the image in the image DB 108 can always be kept in a classified state by performing the classification process on the image. For example, if the U / I unit 151 monitors the state of the image DB 108, it is possible to keep the images in the image DB 108 always in a classified state without requiring human intervention.

[Embodiment 15]
The image classification apparatus according to the fifteenth embodiment classifies images by applying the image search process described in the second embodiment. Note that in the image classification device according to the fifteenth embodiment, the same reference numerals are used for the configurations already described in the fourteenth embodiment, and detailed description thereof is omitted, and the operations of the image classification device are also common. The description will be omitted as appropriate.

Hereinafter, the operation of the image classification device according to the fifteenth embodiment will be described.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Processing FIG. 38 is an explanatory diagram showing an example of an image classification window 350b for performing image classification item setting processing and display of images after classification. Here, it is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, the classification condition setting window 360b as shown in FIG. 38 is displayed.

  The classification condition setting window 360b shown in FIG. 38 has substantially the same configuration as part of the search window 200b shown in FIG. 7, and the classification condition setting window 360b has the same search conditions as those described in the second embodiment. Classification conditions can be set by the method. The classification condition setting window 360b includes image designation fields A, B, and C for designating a plurality of reference images (corresponding to query images for image search) used as a reference for classifying images to be classified. For each reference image designated in the image designation field, a feature quantity designation button 362 for designating the type of feature quantity used when obtaining the similarity to the image to be classified, and setting of classification conditions for the corresponding folder A setting button 361 for designating.

  As described in the second embodiment, the user designates, for example, three images as the reference images A, B, and C in the image designation fields A, B, and C, respectively, for each reference image A, B, and C. Specifies the type of feature amount used when obtaining the similarity to the image to be classified. As an example, the classification condition setting window 360b shown in FIG. 38 can specify a desired feature amount from among color, structure, and texture.

  Although not shown, a threshold value of similarity between the reference image and the search target image can be set for each folder from the classification condition setting window 360b. As will be described later, an image to be classified whose similarity with a reference image set in a folder exceeds a threshold is classified into that folder. Here, a threshold value is set for each folder, but one threshold value may be set for the entire folder.

  When the user selects the setting button 361, the U / I unit 151 sets the designated reference image, the type of feature amount, and the threshold value in the folder 352 as the classification condition. In the case where the feature amount is not extracted from the image set as the reference image, the feature extraction engine 152 may perform the process of extracting the feature amount when setting the classification condition in the folder. . Of course, a necessary feature amount may be extracted from each reference image when performing the process of classifying the images.

  The user creates folders (for example, folders 354 and 355 with respect to the folder 353) for further classifying the images classified into the created folder (for example, the folder 353) like the folders 354 to 357, Similarly, classification conditions can be set.

  In this way, a plurality of folders 352 to 358 as shown in FIG. 38 are set as classification items for classifying the images to be classified. The U / I unit 151 registers the created folder and the corresponding classification condition as folder information in the classification tree DB 302 together with information related to the hierarchical structure of the folder.

  As described in the second embodiment, the user can specify not only a plurality of reference images for each folder but also a feature amount corresponding to the feature of the image that the user attaches importance to for each specified reference image. It is possible to express one virtual reference image that is considered appropriate, and to accurately express the user's intention for classification as a classification reference. Therefore, it is possible to obtain an image of the classification result reflecting the user's intention by the classification process described later.

(2) Image Classification Processing The image classification processing according to the fifteenth embodiment will be described with reference to FIG. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 determines the similarity between each reference image and the classification target image for each of the folders 352 to 358 using the type of feature amount designated by the user. At the same time, a process of calculating the total similarity of the images to be classified is performed by summing the calculated similarities (S1401).

  The processing in step S1401 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360b in FIG. 38) set in the folder 352 and the image to be classified as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on the feature amount of the structure, and calculates the distance between the image to be classified and the reference image C based on the feature amount of the texture. Ask.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C, and sets the total distance as the total similarity (see Expression 2). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity of the classification target images obtained in step S1401 exceeds a threshold value (S1402). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose total similarity of the images to be classified exceeds a threshold value (S1403). That is, when the total similarity of the images to be classified exceeds the threshold, the matching engine 153 determines the folder as a classification destination of the images to be classified, and folder information of the corresponding folder registered in the classification tree DB 302. The ID of the image to be classified (assumed to be set in advance) is registered in the classification tree DB 302 in association with.

  By executing the processes in steps S1401 to S1403 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. Note that the images to be classified may be classified into two or more folders according to the total similarity obtained for each folder. When the image classification processing is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350b of FIG. 38 as shown in FIG.

  As described above, according to the image classification device according to the fifteenth embodiment, a folder is set, and a plurality of reference images are set for each set folder, and each set reference image is similar to an image to be classified. Set the type of feature quantity used when calculating the degree, and for each folder, use the set type feature quantity to calculate the similarity between each reference image and the image to be classified, and total the calculated similarity The user's intention is determined based on the classification condition by determining the folder that is the classification destination of the image to be classified based on the total similarity calculated for each folder and a preset classification destination determination criterion. Since the user's intention can be accurately transmitted to the device and the user's intention for the classification can be reliably transmitted to the device, the user's intention for the classification can be expressed. It is possible to obtain a classification result that accurately reflects the, it is possible to realize the classification process as the user desires.

  For example, even when an appropriate image does not exist as a reference image, an image having a color atmosphere similar to a desired image or an image having an object shape atmosphere similar to a desired image as a classification result is designated as the reference image. Can be used as one inquiry image.

  Here, three types of feature quantities of color, structure, and texture are used, but these are merely examples, and are not intended to limit the types of feature quantities.

  Further, although detailed description is omitted, when a new image is registered in the image DB 108, the image in the image DB 108 can always be kept in a classified state by performing the classification process on the image. For example, if the U / I unit 151 monitors the state of the image DB 108, it is possible to keep the images in the image DB 108 always in a classified state without requiring human intervention.

[Embodiment 16]
The image classification apparatus according to the sixteenth embodiment can express the user's intention for classification more accurately by applying the image search process described in the third embodiment to the image classification apparatus according to the fifteenth embodiment. It is a thing. Note that in the sixteenth embodiment, the same reference numerals are used for the configurations already described in the fourteenth embodiment, and detailed descriptions thereof are omitted, and descriptions of points common to the operations of the image classification device are omitted as appropriate. .

Hereinafter, the operation of the image classification device according to the sixteenth embodiment will be described.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Processing FIG. 39 is an explanatory diagram showing an image classification window 350c used in the image classification device according to the sixteenth embodiment. Here, it is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, the classification condition setting window 360c as shown in FIG. 39 is displayed.

  The classification condition setting window 360c shown in FIG. 39 is in addition to that shown in FIG. 38 (as shown in the search window 200c of the third embodiment), and each reference image designated in the image designation columns A, B, and C. An importance level designation bar 363 for designating a relative importance level is provided for each.

  As shown in FIG. 39, the user designates, for example, three images as reference images A, B, and C in the image designation fields A, B, and C, respectively, and classifies them for each designated reference image A, B, and C. Specifies the type of feature value used when obtaining the similarity to the image.

  Subsequently, as described in the importance level designation bar 204 of the third embodiment, the user designates the importance level for each reference image using the importance level designation bar 363. The importance level designation bar 363 is capable of designating the importance level of “+” and the importance level of “−” with “0” as the center for each reference image. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  As described above, when the user designates the importance level for each reference image, it is possible to accurately express the user's intention for the classification. That is, since the importance is reflected as a weighting on the similarity by the classification process described later, an image of the classification result reflecting the user's intention can be obtained.

  Further, when the user designates the threshold value of the folder 352 and selects the setting button 361, the U / I unit 151 sets the reference image, the type of feature amount, the importance level, and the threshold value in the folder 352 as classification conditions.

  By performing such processing for each folder, a plurality of folders 352 to 358 as shown in FIG. 39 are set as classification items for classifying the images to be classified. The folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing FIG. 40 is a flowchart showing image classification processing according to the sixteenth embodiment. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 determines the similarity between each reference image and the classification target image for each of the folders 352 to 358 using the type of feature amount designated by the user. The obtained similarity is weighted according to the importance, and a process for obtaining the total similarity of the images to be classified is executed by summing up the similarities (S1601).

  The processing in step S1601 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360c in FIG. 39) set in the folder 352 and the classification target image as a color feature amount. Find using. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the reference image A designated by the user. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on the feature amount of the structure, weights the obtained distance according to the importance of the reference image B, and The distance between the image to be classified and the reference image C is obtained based on the texture feature amount, and the obtained distance is weighted according to the importance of the reference image C.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C, and sets the total distance as the total similarity (see Expression 3). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity obtained in step S1601 exceeds a threshold value (S1602). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S1603). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S1601 to S1603 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. When the image classification process is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350c of FIG. 39 as shown in FIG.

  Thus, according to the image classification device according to the sixteenth embodiment, the degree of importance is set for each reference image so that the obtained similarity can be arbitrarily weighted according to the type of the reference image. It is possible to set the user's intention by calculating the similarity between each reference image and the image to be classified for each folder and weighting the obtained similarity according to the set importance. Can be accurately expressed as a classification condition, so it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weighting to the similarity, so the image classification result that reflects the user's intention Can be obtained.

  Although not described in detail, the image classification apparatus described in the fourteenth embodiment also allows the importance to be specified for each inquiry image as described in the sixteenth embodiment. good.

[Embodiment 17]
The image classification apparatus according to the seventeenth embodiment performs an image classification process applying the AND search described in the fourth embodiment as one technique for accurately expressing the user's intention as a classification condition. Note that in the image classification device according to the seventeenth embodiment, the same reference numerals are used for the configurations already described in the fourteenth embodiment, and detailed descriptions thereof are omitted, and the operations of the image classification device are also common. The description will be omitted as appropriate. In the image classification device according to the seventeenth embodiment, one feature amount is used when obtaining the similarity between images. Of course, a plurality of feature amounts may be used. .

Hereinafter, the operation of the image classification device according to the seventeenth embodiment will be described.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Processing FIG. 41 is an explanatory diagram showing an image classification window 350d used in the image classification device according to the seventeenth embodiment. Here, it is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, the classification condition setting window 360d as shown in FIG. 41 is displayed.

  A classification condition setting window 360d shown in FIG. 41 includes an AND designation button 364 for designating an AND condition and an OR designation button 365 for designating an OR condition in addition to the image designation fields A, B and C and the setting button 361. Is a point provided. The OR condition will be described in detail in a 21st embodiment to be described later.

  As shown in FIG. 41, the user designates, for example, three images as reference images A, B, and C in the image designation fields A, B, and C, respectively, selects an AND designation button 364, and designates an AND condition. . Further, when the user designates the threshold value of the folder 352 and designates the setting button 361, the U / I unit 151 sets the reference image, the designation of the AND condition, and the threshold value as the classification condition in the folder 352.

  By performing such processing for each folder, a plurality of folders 352 to 358 as shown in FIG. 41 are set as classification items for classifying the images to be classified. Note that the folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing The image classification processing according to the seventeenth embodiment will be described with reference to FIG. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified using a predetermined feature amount for each of the folders 352 to 358, and the obtained similarity. Processing for obtaining the total similarity of the images to be classified by summing the degrees is executed (S1401).

  The processing in step S1401 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the image to be classified as a predetermined feature amount. Find using. That is, the matching engine 153 determines the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on a predetermined feature amount, and obtains the distance between the image to be classified and the reference image C.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C, and sets the total distance as the total similarity (see Expression 4). In addition, when obtaining the similarity between each reference image and the classification target image using a plurality of types of feature amounts, the similarity (distance) between the classification target image and each reference image is obtained for each type of feature amount, For each image to be classified, all similarities with respect to each reference image may be summed to obtain an overall similarity. The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity obtained in step S1401 exceeds a threshold value (S1402). However, here, distance is used as the similarity, and the shorter the distance (the smaller the value of the distance), the higher the similarity. Therefore, the total similarity is actually less than (or less than) the threshold. It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S1403). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S1401 to S1403 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects, as a representative image, a search target image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358. , It is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350d in FIG. 41 as shown in FIG.

  Thus, according to the image classification device according to the seventeenth embodiment, a folder is set, a plurality of reference images are set for each set folder, and each set reference image and classification target image are set for each folder. The total similarity is obtained by summing the obtained similarities, and becomes a classification destination of the image to be classified based on the total similarity obtained for each folder and a preset classification destination determination criterion. In order to determine the folder, it is possible to perform image classification processing using AND search processing. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  In the seventeenth embodiment, the type of feature quantity is not particularly shown, but any type of feature quantity such as color, structure, texture, etc. can be used alone or in combination.

  Further, although detailed description is omitted, when a new image is registered in the image DB 108, the image in the image DB 108 can always be kept in a classified state by performing the classification process on the image. For example, if the U / I unit 151 monitors the state of the image DB 108, it is possible to keep the images in the image DB 108 always in a classified state without requiring human intervention.

[Embodiment 18]
The image classification apparatus according to the eighteenth embodiment can accurately express the user's intention for classification by applying the image search processing described in the fifth embodiment to the image classification apparatus according to the seventeenth embodiment. Is.

The operation of the image classification device according to Embodiment 18 is as follows.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Processing FIG. 42 is an explanatory diagram showing an image classification window 350e used in the image classification device according to the eighteenth embodiment. Here, it is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, the classification condition setting window 360e as shown in FIG. 42 is displayed.

  The classification condition setting window 360e shown in FIG. 42 designates the relative importance between the reference images designated in the image designation fields A, B and C in addition to the classification condition setting window 360d shown in FIG. An importance level designation bar 366 is provided.

  As shown in FIG. 42, the user designates, for example, three images as reference images A, B, and C in the image designation fields A, B, and C, respectively, and uses the importance degree designation bar 366 to reference image A, The importance is designated for each of B and C. As described with respect to the importance level designation bar 207 of the fifth embodiment, the importance level designation bar 366 indicates the importance level of “+” and the importance level of “−” with respect to “0” for each reference image. It is possible to specify. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  As described above, when the user designates the importance level for each reference image, it is possible to accurately express the user's intention for the classification. That is, since the importance is reflected as a weighting on the similarity by the classification process described later, an image of the classification result reflecting the user's intention can be obtained.

  Further, when the user selects the AND designation button 364 to designate the AND condition and designates the threshold value of the folder 352 and then designates the setting button 361, the U / I unit 151 causes the reference image, the importance, the AND condition to be specified. And a threshold value are set in the folder 352 as a classification condition.

  By performing such processing for each folder, a plurality of folders 352 to 358 as shown in FIG. 42 are set as classification items for classifying the images to be classified. Note that the folder information of each set folder is registered in the classification tree DB 302.

(2) Image Classification Processing The image classification processing according to the eighteenth embodiment will be described with reference to FIG. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified for each folder 352 to 358 using a predetermined feature amount. Each similarity is weighted according to the importance, and a process for obtaining the total similarity by summing up the similarities is executed (S1601).

  The processing in step S1601 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360e in FIG. 42) set in the folder 352 and the image to be classified as a predetermined feature amount. Use to find. That is, the matching engine 153 determines the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the reference image A designated by the user. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on a predetermined feature amount, weights the obtained distance according to the importance of the reference image B, The distance between the image to be classified and the reference image C is obtained based on a predetermined feature amount, and the obtained distance is weighted according to the importance of the reference image C.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C, and sets the total distance as the total similarity (see Expression 5). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity obtained in step S1601 exceeds a threshold value (S1602). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S1603). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S1601 to S1603 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. When the image classification processing is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350e in FIG. 42 as shown in FIG.

  Thus, according to the image classification device according to the eighteenth embodiment, the degree of importance is set for each reference image so that the obtained similarity can be arbitrarily weighted according to the type of the reference image. It is possible to set the user's intention by calculating the similarity between each reference image and the image to be classified for each folder and weighting the obtained similarity according to the set importance. Can be accurately expressed as a classification condition, so it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weighting to the similarity, so the image classification result that reflects the user's intention Can be obtained.

[Embodiment 19]
The image classification apparatus according to the nineteenth embodiment can accurately express the user's intention for classification by applying the image search processing described in the sixth embodiment to the image classification apparatus according to the seventeenth embodiment. Is. In the image classification apparatus according to the nineteenth embodiment, for example, the classification process is performed using three types of feature amounts of color, structure, and texture.

The operation of the image classification device according to Embodiment 19 is as follows.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Processing FIG. 43 is an explanatory diagram showing an image classification window 350f used in the image classification device according to the nineteenth embodiment. Here, it is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, a classification condition setting window 360f as shown in FIG. 43 is displayed.

  The classification condition setting window 360f shown in FIG. 43 is similar to the image to be classified for each reference image designated in the image designation fields A, B and C in addition to the classification condition setting window 360d shown in FIG. An importance level designation bar 367 is provided for designating the importance level for each type of feature amount used when determining the level.

  As shown in FIG. 43, the user designates, for example, three images as reference images A, B, and C in the image designation fields A, B, and C, respectively, and features for each designated reference image A, B, and C. Specify the importance for each type of quantity. As described in the sixth embodiment, the importance level designation bar 367 displays the importance level of “+” and the importance level of “−” with “0” as the center for each type of feature amount of each reference image. It is possible to specify. Note that it is possible to designate not only both the “+” direction and the “−” direction but also the “+” direction or the “−” direction as the importance.

  Thus, by specifying the importance for each type of feature amount for each reference image, it is possible to accurately represent the user's intention for the classification. That is, since the importance is reflected as a weighting on the similarity obtained for each type of feature amount by the classification process described later, an image of the classification result reflecting the user's intention can be obtained.

  Further, when the user selects the AND designation button 364 to designate an AND condition, designates the threshold value of the folder 352, and then selects the setting button 361, the U / I unit 151 causes the reference image, importance, AND The designation of the condition and the threshold value are set in the folder 352 as the classification condition.

  By performing such processing for each folder, a plurality of folders 352 to 358 as shown in FIG. 43 are set as classification items for classifying the images to be classified. Note that the folder information of each set folder is registered in the classification tree DB 302.

(2) Image Classification Processing Image classification processing according to the nineteenth embodiment will be described with reference to FIG. When the user designates the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358. Weighting is performed on each similarity according to importance, and processing for obtaining the total similarity of the images to be classified is performed by summing up the similarities (S1601).

  The processing in step S1601 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360f in FIG. 43) set in the folder 352 and the classification target image as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the feature amount of the color designated by the user for the reference image A. Further, the matching engine 153 obtains the distance between the image to be classified and the reference image A by using the structural feature amount and the texture feature amount, and the user calculates the distance from the reference image A for each of the obtained distances. Weighting is performed according to the importance of the feature amount of the specified structure and the importance of the feature amount of the texture.

  Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B for each of the three types of feature amounts, and for each feature amount designated for the reference image B for each of the obtained distances. Weighting is performed according to the importance of. Further, the distance between the image to be classified and the reference image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted according to the importance for each feature amount designated for the reference image C. I do.

  Thereafter, the matching engine 153 sums the distances between the images to be classified obtained for each of the three types of feature amounts and the reference images A, B, and C, and the total value of the distances is used as the total similarity (Equation 6 See). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity obtained in step S1601 exceeds a threshold value (S1602). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S1603). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S1601 to S1603 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. When the image classification processing is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350f of FIG. 43 as shown in FIG.

  As described above, according to the image classification device according to the nineteenth embodiment, when the similarity between the reference image and the search target image is obtained using a plurality of types of feature amounts, Importance for each type of feature can be set for each reference image so that arbitrary weighting can be performed according to the type of feature, and each type of feature for each folder The similarity of each reference image and classification target image is obtained for each type, and the user's intention is classified according to the set importance by weighting the similarity obtained for each type of feature amount. Therefore, it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weight for the similarity. Therefore, the classification result of the image reflecting the user's intention is displayed. Rukoto is possible.

  Note that the types of feature quantities are not limited to the three types of color, structure, and texture, but may be any kind of combination of feature quantities.

[Embodiment 20]
In the image classification device according to the twentieth embodiment, by applying the image search process described in the seventh embodiment to the image classification device according to the seventeenth embodiment, the device side can fetch the user's intention for the classification. It is a thing. Note that, for example, the image classification device according to the twentieth embodiment performs classification processing using three types of feature amounts of color, structure, and texture.

  In Embodiments 18 and 19 described above, the user needs to specify the importance of the reference image or the importance for each feature amount in each reference image. In the image classification device according to Embodiment 20, The similarity between the reference images designated for each folder by the user is obtained using the feature amount used when obtaining the similarity between the reference image and the classification target image, and based on the obtained similarity for each feature amount. The feature is that the importance for each feature amount between the reference images is determined on the device side and reflected as a weighting for the similarity.

The operation of the image classification device according to Embodiment 20 is as follows.
(1) Classification item setting processing (2) Image classification processing will be described in this order.

(1) Classification Item Setting Process The classification item setting process will be described with reference to FIG. It is assumed that the user has created the folder 352, and in order to set the classification condition in the folder 352, a classification condition setting window 360d as shown in FIG. 41 is displayed.

  As shown in FIG. 41, the user designates, for example, three images as reference images A, B, and C in the image designation fields A, B, and C, respectively, selects an AND designation button 364, and designates an AND condition. . Further, when the user designates the threshold value of the folder 352 and selects the setting button 361, the feature extraction engine 152 causes the distance between the reference images A, B, and C for each of the three types of feature amounts of color, structure, and texture. (Similarity) is obtained, and the importance for each feature amount between the reference images A, B, and C is determined based on the obtained distance for each feature amount.

  Specifically, the feature extraction engine 152 obtains the distance between the features of the reference images for each of the three types of feature amounts. If the variance of the distance obtained for each feature amount is large, it means that the user is not aware of the feature amount. If the variance is small, the user attaches importance to the feature amount as a classification standard. It can be judged. Therefore, the importance is decreased when the feature amount has a large variance, and the importance is increased when the variance is small (see Equation 7).

  The feature extraction engine 152 inputs the importance for each of the three types of feature amounts to the U / I unit 151. The U / I unit 151 specifies the reference image, the importance determined by the feature extraction engine 152, and the AND condition. The threshold value is set in the folder 352 as the classification condition.

  By performing such processing for each folder, a plurality of folders 352 to 358 as shown in FIG. 41 are set as classification items for classifying the images to be classified. The folder information of the set folder is registered in the classification tree DB 302.

(2) Image Classification Processing Next, image classification processing according to the twentieth embodiment will be described with reference to FIG. When the user designates the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified using three types of feature amounts for each of the folders 352 to 358. Weighting is performed on each similarity according to importance, and processing for obtaining the total similarity of the images to be classified is performed by summing up the similarities (S1601).

  The processing in step S1601 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Subsequently, the matching engine 153 performs weighting on the obtained distance according to the importance determined by the feature extraction engine 152 for the color feature amount. Further, the matching engine 153 obtains a distance using the structural feature quantity and the texture feature quantity for the classification target image and the reference image A, and for each of the obtained distances, the structure of the structure determined by the feature extraction engine 152 is obtained. Weighting is performed according to the importance of the feature quantity and the importance of the texture feature quantity.

  Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B for each of the three types of feature amounts, and sets the importance determined for each type of feature amount for each of the obtained distances. Perform the corresponding weighting. Further, the distance between the image to be classified and the reference image C is determined for each of the three types of feature amounts, and the determined distances are weighted according to the importance determined for each type of feature amount.

  Thereafter, the matching engine 153 sums the distances between the images to be classified and the reference images A, B, and C obtained for each of the three types of feature amounts, and sets the total value of the distances as the overall similarity. The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity obtained in step S1601 exceeds a threshold value (S1602). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S1603). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S1601 to S1603 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from among the images classified in the folders 352 to 358 as a representative image. It is registered in the classification tree DB 302 in association with the folder information. When the image classification process is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350d in FIG. 41 as shown in FIG.

  Thus, according to the image classification device according to the twentieth embodiment, when the similarity between the reference image and the search target image is obtained using a plurality of types of feature amounts, the feature amount is compared with the obtained similarity degree. For each classification item, the similarity between each set reference image is obtained for each type of feature quantity, and the similarity obtained for each type of feature quantity so that arbitrary weighting can be performed according to the type. The degree of importance for each feature amount between each reference image is determined based on the degree, and for each classification item, the similarity between each reference image and the classification target image is obtained for each type of feature amount, and each type of feature is determined. The importance obtained by determining the importance of each type of feature between the reference images specified by the user on the device side by weighting the similarity obtained for each quantity according to the determined importance. To reflect the degree in the similarity between images You can return the result of classification suits image the user's intention to become. Therefore, the classification process as desired by the user can be realized, and the classification result desired by the user can be obtained.

  Note that the types of feature quantities are not limited to the three types of color, structure, and texture, but may be any kind of combination of feature quantities.

[Embodiment 21]
The image classification apparatus according to the twenty-first embodiment performs an image classification process using the OR search described in the eighth embodiment as one method for accurately expressing the user's intention as a classification condition.

  In the image classification processing by the image classification device according to the twenty-first embodiment, the OR designation button 365 is selected instead of the AND designation button 364 in the classification condition setting window 360d of FIG. 41 used for the description in the seventeenth embodiment. If executed. In view of “(1) classification item setting process” in the seventeenth embodiment, assuming that the OR designation button 365 is selected, the image classification process based on the OR condition will be described below.

  It is assumed that reference images A, B, and C, and threshold values are set as classification conditions in the folder 352 shown in FIG. 41 together with the designation of the OR condition, as in the case of the seventeenth embodiment.

  FIG. 44 is a flowchart showing image classification processing according to the twenty-first embodiment. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified for each folder 352 to 358 by using a predetermined feature amount. A process of selecting the similarity indicating the highest similarity from the similarities and obtaining the total similarity of the images to be classified is executed (S2101).

  The processing in step S2101 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the image to be classified as a predetermined feature amount. Find using. That is, the matching engine 153 determines the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on a predetermined feature amount, and obtains the distance between the image to be classified and the reference image C.

  Thereafter, the matching engine 153 selects the shortest distance from the distances between the images to be classified and the reference images A, B, and C to obtain the overall similarity (see Expression 8). In this way, the matching engine 153 selects the similarity indicating the highest similarity for each of the folders 352 to 358 and obtains the total similarity.

  Subsequently, the matching engine 153 determines whether the similarity selected in step S2101 exceeds the threshold for each of the folders 352 to 358 (S2102). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose selected similarity exceeds the threshold (S2103). That is, the matching engine 153 determines the folder whose selected similarity exceeds the threshold as the classification destination of the image to be classified, and associates it with the folder information of the corresponding folder registered in the classification tree DB 302 to classify the image. Are registered in the classification tree DB 302.

  By executing the processes in steps S2101 to S2103 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 has the highest similarity (the similarity selected in step S2101 from among the images classified in each of the folders 352 to 358, and the lowest similarity expressed in terms of distance). ) Are selected as representative images and registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350d in FIG. 41 as shown in FIG.

  In the image classification process according to the twenty-first embodiment, if the image to be classified is similar to a reference image set in a certain folder so that a similarity equal to or higher than a threshold is obtained, the folder It will be classified into. Therefore, it can be seen that the OR search is applied to the image classification process of the twenty-first embodiment.

  Thus, according to the image classification device according to the twenty-first embodiment, a folder is set, a plurality of reference images are set for each set folder, and each set reference image and classification target image are set for each folder. And selecting the similarity showing the highest similarity from the obtained similarities, and selecting the image to be classified based on the similarity selected for each folder and a preset classification destination determination criterion Therefore, it is possible to perform image classification processing using OR search processing. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  Although a detailed description is omitted, when a new image is registered in the image DB 108, the image in the image DB 108 can always be kept in a classified state by performing a classification process on the image. For example, if the U / I unit 151 monitors the state of the image DB 108, it is possible to keep the images in the image DB 108 always in a classified state without requiring human intervention.

[Embodiment 22]
The image classification apparatus according to the twenty-second embodiment can accurately represent the user's intention for classification by applying the image search processing described in the ninth embodiment to the image classification apparatus according to the twenty-first embodiment. Is.

  In the image classification processing by the image classification apparatus according to the twenty-second embodiment, the OR designation button 365 is selected instead of the AND designation button 364 in the classification condition setting window 360e of FIG. 42 used for the explanation in the eighteenth embodiment. If executed. Thus, in the description of “(1) Classification item setting process” in the eighteenth embodiment, assuming that the OR designation button 365 is selected, the image classification process based on the OR condition will be described below.

  In addition to the designation of the OR condition, in the folder 352 shown in FIG. 42, as in the case of the eighteenth embodiment, the reference images A, B and C, the importance level and threshold value of each reference image are set as the classification conditions. Shall.

  FIG. 45 is a flowchart showing image classification processing according to the twenty-second embodiment. When the user instructs the start of the classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified for each folder 352 to 358 using a predetermined feature amount. Weighting is performed on each similarity according to the importance, and processing for obtaining the overall similarity of the images to be classified is executed by selecting the similarity indicating the highest similarity among the similarities (S2201). .

  The processing in step S2201 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360e in FIG. 42) set in the folder 352 and the image to be classified as a predetermined feature amount. Use to find. That is, the matching engine 153 determines the distance between the image to be classified and the reference image A plotted in the feature amount space shown in FIG. 4 based on a predetermined feature amount, for example, a vector distance definition formula such as a Euclidean distance. Use to find. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the reference image A designated by the user. Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B based on a predetermined feature amount, weights the obtained distance according to the importance of the reference image B, The distance between the image to be classified and the reference image C is obtained based on a predetermined feature amount, and the obtained distance is weighted according to the importance of the reference image C.

  Thereafter, the matching engine 153 selects the shortest distance from the distances between the images to be classified and the reference images A, B, and C to obtain the overall similarity (see Expression 9). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 determines, for each of the folders 352 to 358, whether or not the total similarity calculated in step S2201 exceeds a threshold value (S2202). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S2203). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S2201 to S2203 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from the images classified in the folders 352 to 358 as a representative image, The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification processing is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350e in FIG. 42 as shown in FIG.

  As described above, according to the image classification device according to the twenty-second embodiment, the importance is set for each reference image so that the obtained similarity can be arbitrarily weighted according to the type of the reference image. It is possible to set the user's intention by calculating the similarity between each reference image and the image to be classified for each folder and weighting the obtained similarity according to the set importance. Can be accurately expressed as a classification condition, so it is possible to accurately convey the user's intention to the device, and the importance is reflected as a weighting to the similarity, so the image classification result that reflects the user's intention Can be obtained.

[Embodiment 23]
The image classification device according to the twenty-third embodiment can accurately express the user's intention for classification by applying the image search processing described in the tenth embodiment to the image classification device of the twenty-first embodiment. Is.

  In the image classification processing by the image classification device according to the twenty-third embodiment, the OR designation button 365 is selected instead of the AND designation button 364 in the classification condition setting window 360f of FIG. 43 used for the explanation in the nineteenth embodiment. If executed. In view of “(1) classification item setting process” in the nineteenth embodiment, assuming that the OR designation button 365 is selected, an image classification process based on an OR condition will be described below.

  In addition to the specification of the OR condition, in the folder 352 shown in FIG. 43, as in the case of the nineteenth embodiment, the reference images A, B and C, the importance specified for each type of feature amount for each reference image In addition, it is assumed that a threshold is set as a classification condition.

  Next, an image classification process according to Embodiment 23 will be described with reference to FIG. When the user designates the start of the image classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified by using three types of feature amounts for each of the folders 352 to 358. Then, weighting is performed on each similarity according to the importance, and the processing for obtaining the total similarity of the images to be classified by selecting the similarity that shows the highest similarity from the similarities is executed (S2201). ).

  The processing in step S2201 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360f in FIG. 43) set in the folder 352 and the classification target image as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Find using. Subsequently, the matching engine 153 weights the obtained distance according to the importance of the feature amount of the color designated by the user for the reference image A. Further, the matching engine 153 obtains the distance between the image to be classified and the reference image A using the structural feature amount and the texture feature amount, and the user designates the reference image A for each of the obtained distances. Weighting is performed according to the importance of the feature amount of the structure and the importance of the feature amount of the texture.

  Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B for each of the three types of feature amounts, and for each feature amount designated for the reference image B for each of the obtained distances. Weighting is performed according to the importance of. Further, the distance between the image to be classified and the reference image C is obtained for each of the three types of feature amounts, and the obtained distances are weighted according to the importance for each feature amount designated for the reference image C. I do.

  Thereafter, the matching engine 153 selects the shortest distance from the distances between the images to be classified obtained for each of the three types of feature amounts and the reference images A, B, and C to obtain the total similarity (several 10). The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 refers to the classification tree DB 302 and determines whether or not the total similarity obtained in step S2201 exceeds the threshold for each of the folders 352 to 358 (S2202). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S2203). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S2201 to S2203 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from the images classified in the folders 352 to 358 as a representative image, The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification processing is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350f of FIG. 43 as shown in FIG.

  Thus, according to the image classification device according to the twenty-third embodiment, when the similarity between the reference image and the search target image is obtained using a plurality of types of feature amounts, the reference image and Importance for each type of feature can be set for each reference image so that arbitrary weighting can be performed according to the type of feature, and each type of feature for each folder The similarity of each reference image and classification target image is obtained for each type, and the user's intention is classified according to the set importance by weighting the similarity obtained for each type of feature amount. Therefore, it is possible to accurately convey the user's intention to the device, and since the importance is reflected as a weight for the similarity, the classification result of the image reflecting the user's intention is displayed. Rukoto is possible.

[Embodiment 24]
In the image classification device according to the twenty-fourth embodiment, by applying the image search process described in the eleventh embodiment to the image classification device according to the twenty-first embodiment, the device side can fetch the user's intention for the classification. It is a thing.

  In the image classification processing by the image classification device according to the twenty-fourth embodiment, the OR designation button 365 is selected instead of the AND designation button 364 in the classification condition setting window 360d of FIG. 41 used in the description of the twentieth embodiment. If executed. Thus, in the description of “(1) classification item setting process” in the twentieth embodiment, it is assumed that the OR designation button 365 is selected, and the image classification process based on the OR condition will be described below.

  In addition to the designation of the OR condition, the folder 352 shown in FIG. 41 contains the important images determined by the feature extraction engine 152 for each type of the reference images A, B, and C and the feature amount as in the case of the twentieth embodiment. It is assumed that the degree and the threshold are set as classification conditions.

  Next, the image classification process according to the twenty-fourth embodiment will be described with reference to FIG. When the user designates the start of the image classification process, the matching engine 153 in FIG. 34 obtains the similarity between each reference image and the image to be classified by using three types of feature amounts for each of the folders 352 to 358. Then, weighting is performed on each similarity according to the importance, and the processing for obtaining the total similarity of the images to be classified by selecting the similarity that shows the highest similarity from the similarities is executed (S2201). ).

  The processing in step S2201 will be specifically described taking the folder 352 as an example. The matching engine 153 refers to the folder information in the classification tree DB 302 and uses the similarity between the reference image A (see the classification condition setting window 360d in FIG. 41) set in the folder 352 and the classification target image as a color feature amount. Use to find. That is, the matching engine 153 determines the distance between the classification target image plotted in the feature amount space shown in FIG. 4 based on the color feature amount and the reference image A, for example, a vector distance definition formula such as a Euclidean distance. Find using. Subsequently, the matching engine 153 performs weighting on the obtained distance according to the importance determined by the feature extraction engine 152 for the color feature amount. Further, the matching engine 153 obtains a distance using the structural feature quantity and the texture feature quantity for the classification target image and the reference image A, and for each of the obtained distances, the structure of the structure determined by the feature extraction engine 152 is obtained. Weighting is performed according to the importance of the feature quantity and the importance of the texture feature quantity.

  Similarly, the matching engine 153 obtains the distance between the image to be classified and the reference image B for each of the three types of feature amounts, and sets the importance determined for each type of feature amount for each of the obtained distances. Perform the corresponding weighting. Further, the distance between the image to be classified and the reference image C is determined for each of the three types of feature amounts, and the determined distances are weighted according to the importance determined for each type of feature amount.

  Thereafter, the matching engine 153 selects the shortest distance from the distances between the images to be classified obtained for each of the three types of feature amounts and the reference images A, B, and C to obtain the overall similarity. The matching engine 153 obtains the total similarity for each of the folders 352 to 358 in this way.

  Subsequently, the matching engine 153 refers to the classification tree DB 302 and determines whether or not the total similarity obtained in step S2201 exceeds the threshold for each of the folders 352 to 358 (S2202). However, here, distance is used as the similarity, and the shorter the distance (the smaller the distance value), the higher the similarity. Therefore, is the total similarity actually less than or equal to the threshold (or less)? It will be determined whether or not.

  Then, the matching engine 153 classifies the images to be classified into folders whose calculated overall similarity exceeds the threshold (S2203). That is, the matching engine 153 determines a folder whose total similarity obtained exceeds the threshold as a classification destination of the image to be classified, and associates the folder with the folder information of the corresponding folder registered in the classification tree DB 302 as a classification target. The image ID (assumed to be set in advance) is registered in the classification tree DB 302.

  By executing the processes in steps S2201 to S2203 described above for each image registered in the image DB 108, the images in the image DB 108 can be classified. In addition, the matching engine 153 selects an image having the highest overall similarity (the lowest overall similarity in terms of distance) from the images classified in the folders 352 to 358 as a representative image, The information is registered in the classification tree DB 302 in association with the corresponding folder information. When the image classification process is executed, representative images are displayed in the folders 352 to 358 of the image classification window 350d in FIG. 41 as shown in FIG.

  As described above, according to the image classification device according to the twenty-fourth embodiment, when the similarity between the reference image and the search target image is obtained using a plurality of types of feature amounts, the feature amount is compared with the obtained similarity degree. For each classification item, the similarity between each set reference image is obtained for each type of feature quantity, and the similarity obtained for each type of feature quantity so that arbitrary weighting can be performed according to the type. The degree of importance for each feature amount between each reference image is determined based on the degree, and for each classification item, the similarity between each reference image and the classification target image is obtained for each type of feature amount, and each type of feature is determined. The importance obtained by determining the importance of each type of feature between the reference images specified by the user on the device side by weighting the similarity obtained for each quantity according to the determined importance. To reflect the degree in the similarity between images User can be realized from the image classification processing as desired, it is possible to obtain a classification result of the image desired by the user.

[Embodiment 25]
Next, an image classification apparatus according to Embodiment 25 will be described. The image classification apparatus according to the twenty-fifth embodiment is configured to dynamically classify an image based on a criterion desired by a user and to search for a desired image.

  Note that the configuration of the image classification apparatus and the configuration of the image classification software according to the twenty-fifth embodiment are as shown in FIGS. 33 and 34, so the description thereof will be omitted here, and only the image classification processing will be described. To do. Here, the images to be classified are images registered in the image DB 108, and color, structure, and texture feature amounts are extracted from these images and registered in the image feature DB 109. And

  46 to 48 are explanatory diagrams of image classification processing in the twenty-fifth embodiment. When the user selects “image classification processing” from a menu (not shown), the U / I unit 151 in FIG. 34 displays an image classification window 400 shown in FIG. 46 on the screen.

  The image classification window 400 shown in FIG. 46 is used when the classification items 401 for all the images in the image DB 108 and the images registered in the image DB 108 are dynamically classified as an initial state. Classification items 402a, 403a, and 404a representing types of feature quantities to be displayed are shown. When the user selects any of the classification items 402a, 403a, and 404a, the images in the image DB 108 are dynamically classified.

  Here, the classification item 402a is an item used when classifying an image using color feature amounts, and the classification item 403a is an item used when classifying an image using structural feature amounts. The classification item 404a is an item for classifying an image using a texture feature amount.

  In FIG. 46, if the user designates the classification item 402a, the U / I unit 151 notifies the matching engine 153 that the color feature amount is designated. The matching engine 153 clusters images in the image DB 108 in a color feature space (see FIG. 4) using a general clustering method such as the Nearest Neighbor method or the K-average algorithm (to group similar images). And a representative image is selected from each cluster. For the representative image, any selection method may be used as long as the representative images are not similar to each other, for example, the center image of the cluster is selected as the representative image.

  Then, the U / I unit 151 receives the result of clustering from the matching engine 153, and displays representative images 405 to 408 in association with the classification item 402a selected by the user, as shown in FIG. In FIG. 47, since four representative images are displayed, it is shown that the images in the image DB 108 are classified into four clusters by the color feature amount. That is, in the twenty-fifth embodiment, a cluster generated by designating the type of feature amount is a substantial classification item. The user can display and view images belonging to each cluster other than the representative image as necessary.

  FIG. 48 shows the result of further clustering and classifying the clusters of the representative image 407 in the color feature space. In this way, the user can reach the target image by designating the type of feature value step by step.

  Although detailed descriptions of feature quantities other than colors are omitted, clustering is similarly performed in the feature space to classify images even when structure or texture feature quantities are selected. become.

  As described above, according to the image classification device according to the twenty-fifth embodiment, when the type of feature amount is designated, the similarity between the images to be classified is obtained by using the feature amount of the designated type. Based on the similarity between the images to be classified, the images to be classified are classified into groups of similar images, and representative images representing the groups are selected from the classified groups and displayed on the screen. Further, when one of the displayed representative images is specified and the type of feature amount is specified, the images classified into the group to which the representative image belongs are similar using the specified type of feature amount. The images are classified into groups of images to be classified, and representative images representing the groups are selected from the classified groups and displayed on the screen, so that the images are dynamically classified according to the criteria desired by the user when the images are to be classified. Since it is possible gradually, using the classification of the image it is possible to find a desired image. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  Further, the type of feature quantity is displayed on the screen as classification items 402, 403, and 404, and a representative image of each classified group is displayed on the screen in association with the designated classification item, thereby displaying an image as shown in FIG. Since the result of classifying the images in the DB 108 can be displayed on the screen in a tree structure, the user can recognize the state of the image classification processing at a glance, and the convenience of the dynamic image classification processing can be improved.

  Note that the result of classifying the image by the image classification process of the twenty-fifth embodiment can be reused by registering the result in the classification tree DB 302 as a classification history, for example.

  Further, the images to be classified are not limited to the images in the image DB 108, and for example, a plurality of images obtained as a result of retrieval by the image retrieval apparatus according to the first to thirteenth embodiments can be used as the images to be classified.

[Embodiment 26]
Next, an image classification apparatus according to Embodiment 26 will be described. The image classification device according to the twenty-sixth embodiment is obtained by integrating a part or all of the image classification devices described in the fourteenth to twenty-fifth embodiments as one image classification device. That is, as is apparent from the descriptions in the fourteenth to twenty-fifth embodiments, these devices can be realized with a common configuration as shown in FIGS. 33 and 34, and can be easily integrated.

  By integrating the image classification apparatuses described in the fourteenth to twenty-fifth embodiments into one, classification conditions can be set by different methods for each folder. For example, in FIG. 35, the folder 352 has the classification conditions described in the fourteenth embodiment, the folder 353 has the classification conditions described in the seventeenth embodiment, and the folder 358 has the classification conditions described in the twenty-first embodiment. Each can be set.

  Note that the classification condition setting process and the image classification process are the same as those described in each embodiment, and thus the description thereof is omitted here.

  Thus, by integrating the image classification apparatuses described in Embodiments 14 to 25 into one, classification conditions can be set from various viewpoints, and the degree of freedom regarding the setting of classification conditions can be increased. .

  For the same reason as in the case of the image classification device, the image search devices described in the first to thirteenth embodiments can be integrated as one device. This makes it possible to search for a desired image by combining different search methods.

[Embodiment 27]
Furthermore, part or all of the image search device and the image classification device described in Embodiments 1 to 25 can be integrated as one device (the image search process is performed in the matching engine 153 of the image search device or the image classification device). And image classification processing). That is, in the image search process and the image classification process described in the embodiment of the present invention, the similarity between images is obtained using the feature amount extracted from the image by the matching engine 153 shown in FIGS. Therefore, the image search device and the image classification device described in Embodiments 1 to 25 can be easily integrated into one device.

  Although detailed description is omitted, the following processing can be performed by integrating the image search device and the image classification device described in Embodiments 1 to 25 into one device.

(1) In the first to thirteenth embodiments, an image selected from images classified into each folder (classification item) by any one of the image classification processes described in the fourteenth to twenty-fifth embodiments is designated as an inquiry image. An image search can be performed by any of the described image search processes.
(2) Any of the image classification processes described in the fourteenth to twenty-fifth (or twenty-sixth) embodiments, with the search result image retrieved in any one of the image retrieval processes described in the first to thirteenth embodiments as a classification target. It can be classified by.
(3) One of the images described in the first to thirteenth embodiments, with a plurality of images classified into a specific folder (category item) in any one of the image classification processes described in the fourteenth to twenty-fifth embodiments as search targets. A desired image can be searched by image search processing.

  As described above, by integrating a part or all of the image search apparatus and the image classification apparatus described in Embodiments 1 to 25 into one apparatus, it is possible to freely execute the image search process and the image classification process. It becomes.

  As described above, the image processing apparatus and the image classification apparatus according to the present invention are realized by executing a program prepared in advance by a computer according to a procedure based on each flowchart. This program can be provided for each of the embodiments described above or in any combination.

  This program is provided by being recorded on a computer-readable recording medium such as a hard disk, floppy (registered trademark) disk, CD-ROM, MO, DVD or the like. As shown in FIG. 49A, the program is read from the recording medium 450 by the CPU 101 (computer 100), and the CPU 101 executes processing based on the program, whereby the image search apparatus according to the present invention and / or An image classification device is realized.

  Further, as shown in FIG. 49B, this program can also be executed in the form of downloading from the recording medium of the server 451 via a network such as the Internet or a LAN and reading it into the CPU 101. Further, the program can be downloaded and executed not by the network but also by, for example, distributing the program by broadcast waves.

  Although not particularly mentioned in the above-described embodiments, the image search device and the image classification device according to the present invention can be applied not only to still images but also to moving images. This is because a moving image consists of continuous still images.

  In addition, it can be said that the image search processing described in the first to third embodiments uses the same technique as the AND search described in the fourth to seventh embodiments. Since the described OR search can be applied, the AND condition and the OR condition may be specified in the search windows 200a, 200b, and 200c (FIGS. 5, 7, and 9). Similarly, in the image classification apparatuses described in the fourteenth to sixteenth embodiments, an AND condition and an OR condition may be designated as one of the classification conditions.

  Also, the image search device described in the fourth to seventh embodiments, the image search device described in the eighth to eleventh embodiments, the image classification device described in the seventeenth to twenty-first embodiments, and the embodiments 21 to 24 described. In the image classification apparatus described above, an AND condition or an OR condition is designated, but the designation of the AND condition or the OR condition is not essential. In other words, the apparatus can be configured to execute the processing described in each embodiment without specifying an AND condition or an OR condition.

  Although detailed description is omitted, in the image classification apparatus described in the fourteenth to twenty-fourth embodiments, for example, when the process of classifying images into the folder 354 shown in FIG. After being classified. This is because the folder 354 is for further classifying the images classified in the folder 353. In addition, it is possible to freely perform processing such as addition / deletion of folders for classifying images and change of classification conditions set in the folders.

  Further, although detailed description is omitted, for example, in a client / server system, the server is provided with the function of the image search device or image classification device described in each embodiment, and the search condition is designated from the client. Obviously, it is possible to display classified images on the client screen.

  (1) As described above, according to the image search apparatus of the above-described embodiment, one inquiry image is obtained for each type of feature amount used when obtaining the similarity between the inquiry image and the image to be searched. For each image to be searched, the similarity to each specified query image is obtained using the corresponding type of feature amount, and the similarity to each obtained query image is totaled to obtain the image to be searched The user's intention can be accurately expressed as a search condition by outputting a search result based on the total similarity obtained for each image to be searched. In addition to being able to convey the user's intention for the search to the device, it is possible to obtain a search result that accurately reflects the user's intention for the search, and to perform as precisely as the user desires. High image retrieval processing of can be performed. Specifically, for example, even when an appropriate image does not exist as an inquiry image, an image having a color atmosphere similar to the desired image or an image having an object shape atmosphere is specified as an inquiry image as a search result. A combination of these images can be used as one query.

  (2) Further, according to the image search device of the above embodiment, a plurality of query images are specified, and the type of feature amount used when obtaining the similarity with the search target image for each specified query image. For each image to be searched, the degree of similarity with each query image is obtained using the specified type of feature amount, and the similarity with each obtained query image is totaled to obtain a total of the images to be searched By calculating the similarity and outputting the search result based on the total similarity obtained for each image to be searched, the user's intention can be accurately expressed as a search condition, so that the user's intention is accurately communicated to the device In addition, the user's intention for the search can be surely transmitted to the apparatus, so that a search result that accurately reflects the user's intention for the search can be obtained, and an image with high accuracy as desired by the user. It is possible to perform the search process. Specifically, for example, even when an appropriate image does not exist as an inquiry image, an image having a color atmosphere similar to the desired image or an image having an object shape atmosphere is specified as an inquiry image as a search result. A combination of these images can be used as one query.

  (3) Further, according to the image search device of the above embodiment, a plurality of query images are specified, and for each search target image, a similarity with each specified query image is obtained, and each obtained query image It is possible to perform an AND search by calculating the total similarity of the images to be searched by summing the similarities of the images and outputting the search result based on the total similarity calculated for each image to be searched. Therefore, it is possible to provide one method for accurately expressing the user's intention as a search condition.

  (4) Further, according to the image search device of the above embodiment, a plurality of query images are specified, and for each search target image, a similarity with each specified query image is obtained, and each obtained query image The similarity that shows the highest similarity is selected from among the similarities, the overall similarity of the image to be searched is obtained, and the search result is output based on the overall similarity obtained for each image to be searched Thus, it is possible to perform an OR search, and therefore it is possible to provide one method for accurately expressing the user's intention as a search condition.

  (5) Further, according to the image search device of the above-described embodiment, the degree of importance for each inquiry image so that the obtained similarity can be arbitrarily weighted according to the type of inquiry image. For each image to be searched, the similarity with each query image was obtained, and the similarity with each obtained query image was weighted according to the designated importance. Later, by calculating the overall similarity, the user's intention can be accurately expressed as a search condition, so that the user's intention can be accurately transmitted to the apparatus, and the importance is reflected as a weight for the similarity. Therefore, it is possible to obtain a search result image reflecting the user's intention.

  (6) Further, according to the image search device of the above embodiment, when the similarity between the query image and the search target image is further determined using a plurality of types of feature amounts, the query image is compared with the calculated similarity. In addition, it is possible to specify the importance for each type of feature quantity for each query image so that arbitrary weighting can be performed according to the type of feature quantity. After obtaining the similarity with each query image for each type of feature quantity, weighting according to the specified importance for the similarity with each query image obtained for each type of feature quantity, Since the user's intention can be accurately expressed as a search condition by obtaining the similarity, it is possible to accurately convey the user's intention to the apparatus, and the importance is reflected as a weight for the similarity. The intent of The becomes possible to obtain an image of the search results are reflected.

  (7) Further, according to the image search apparatus of the above embodiment, when the similarity between the query image and the search target image is obtained using a plurality of types of feature quantities, the feature quantity is compared with the obtained similarity degree. The degree of similarity between each specified query image is obtained for each type of feature quantity so that arbitrary weighting can be performed according to the type of the type, and based on the obtained degree of similarity for each type of feature quantity The degree of importance of each type of feature amount between each query image is determined, and for each search target image, the similarity to each query image is obtained for each type of feature amount, and for each type of feature amount. Between the query images specified by the user on the device side by weighting the degree of similarity with each obtained query image according to the importance for each type of feature amount determined, and then obtaining the overall similarity The importance of each type of feature To reflect the importance of determining the similarity between images, it is possible to return search results that match the user's intention. Therefore, the search process as desired by the user can be realized, and the search result desired by the user can be obtained.

  (8) According to the image search device of the above embodiment, at least one inquiry image is specified, and the level of roughness of the feature quantity extracted from the image is defined stepwise from coarse to fine. Based on the criteria, specify at least one roughness level of the feature quantity used for the search, and based on the extraction criteria, it corresponds to at least the roughness level specified from the query image and multiple search target images The feature amount is extracted at a predetermined timing, the similarity with the query image is obtained for each image to be searched using the feature amount according to the specified roughness level, and the similarity obtained for each image to be searched Since the search result is output based on the degree, it is possible to execute a plurality of types of search processes with different search accuracy. In other words, since the user can specify at least one level of roughness of the feature amount used for the search, the fineness of the user's viewpoint for the search can be expressed as a search condition, and the user's intention can be accurately set as the search condition Can be expressed. Therefore, it is possible to perform image search processing as desired by the user.

  (9) Further, according to the image search device of the above-described embodiment, the extraction criterion further defines stepwise the number of divisions representing at least how many images are divided as the level of roughness. When extracting feature values according to one of the defined roughness levels, the image is divided based on the number of divisions to generate a plurality of divided images, and feature values are generated from the generated divided images. To extract a feature quantity group consisting of feature quantities in units of divided images from the query image and the search target image, and obtain the similarity between the query image and the search target image using the feature quantity group The similarity between the query image and the search target image is obtained by using the corresponding feature amount to obtain the similarity for each of the corresponding divided images of the query image and the search target image. Level of the division number by the feature amount of roughness can be easily defined, it is possible to introduce the concept that the position relationship between the image features in the image search by determining the similarity for each divided image. Therefore, it is possible to execute an image search process with high accuracy.

  (10) Further, according to the image search device of the above-described embodiment, the extraction criterion is further set to a level indicating the roughest feature amount among the stepwise defined roughness levels without dividing the image. When a feature amount is extracted according to a level where the entire image is a feature amount extraction target and the entire image is a feature amount extraction target, the feature amount is targeted for the entire image from the query image and the search target image. Therefore, it is possible to include a search process at a simple level ignoring the positional relationship of image features.

  (11) Further, according to the image search device of the above-described embodiment, when the extraction criterion is to extract an object existing in the image as a feature amount extraction target, the level of roughness of the feature amount extracted from the object Therefore, the search condition can be specified from a different viewpoint, and the user's intention can be accurately expressed as the search condition.

  (12) Further, according to the image search device of the above embodiment, for a plurality of search target images, feature amounts corresponding to all the roughness levels defined by the extraction criteria are extracted in advance. For query images for which no feature value has been extracted from the specified query images, the feature value corresponding to the specified roughness level is extracted before executing the processing for obtaining the similarity. Processing speed can be increased.

  (13) Further, according to the image search device of the above-described embodiment, at least one inquiry image is specified, and for each search target image, a specified size area in the search target image is specified. A process for repeatedly executing a process for obtaining a similarity with an image using a predetermined feature amount extracted in advance over the entire image to be searched, and outputting a search result based on the similarity obtained for each image to be searched. An image including an inquiry image can be found from the target images. Therefore, it is possible to provide one method for realizing an image search as intended by the user.

  (14) Further, according to the image search device of the above embodiment, since the image of the search result can be specified as the inquiry image, further detailed search can be performed, and the search process as desired by the user is realized. can do.

  (15) According to the computer-readable recording medium of the above embodiment, the program for causing the computer to function as each unit of the image search device according to any one of the above embodiments is recorded. An image search device that enables a computer to execute a search condition so that a user's viewpoint can be reliably transmitted to a device when a search condition is specified and a search result that matches the user's viewpoint can be returned. It can be realized. That is, it is possible to realize an image search apparatus that can realize search processing as desired by the user.

  (16) Also, according to the image classification device of the above-described embodiment, classification items are set, and for each type of classification item set, each type of image used when obtaining the similarity between the reference image and the classification target image. One reference image is set for each feature amount, and for each classification item, the similarity between each set reference image and the classification target image is obtained using the corresponding type of feature amount, and the obtained similarity is obtained. The total similarity of the images to be classified is calculated in total, and the classification target images are classified based on the total similarity of the images to be classified obtained for each classification item and a preset classification destination determination criterion. By determining the classification item, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the device, and the user's intention for the classification can be reliably transmitted to the device. Since it is, it is possible to obtain a classification result that accurately reflects the intention of the user for the classification, it is possible to realize the classification process as the user desires. That is, even when an appropriate image does not exist as a reference image, for example, an image having a similar color atmosphere to an image desired as a classification result or an image having an object shape atmosphere is designated as a reference image. Since the combined image is used as one reference image, the user's viewpoint can be reliably transmitted to the apparatus when specifying the classification condition, and the classification process is executed using such a reference image. Therefore, it is possible to appropriately classify the image into the classification destination desired by the user.

  (17) Further, according to the image classification apparatus of the above embodiment, classification items are set, a plurality of reference images are set for each set classification item, and each set reference image is compared with an image to be classified. Set the type of feature quantity used to calculate the similarity, and for each category item, use the set type of feature quantity to obtain the similarity between each reference image and the image to be classified, and the obtained similarity To determine the total similarity of the images to be classified, and based on the total similarity of the images to be classified obtained for each classification item and a preset classification destination determination criterion, By determining the classification item, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the device, and the user's intention for the classification can be reliably transmitted to the device. Therefore, it is possible to obtain a classification result that accurately reflects the intention of the user for the classification, it is possible to realize the classification process as the user desires. That is, even when an appropriate image does not exist as a reference image, for example, an image having a similar color atmosphere to an image desired as a classification result or an image having an object shape atmosphere is designated as a reference image. Since the combined image is used as one reference image, the user's viewpoint can be reliably transmitted to the apparatus when specifying the classification condition, and the classification process is executed using such a reference image. Therefore, it is possible to appropriately classify the image into the classification destination desired by the user.

  (18) According to the image classification device of the above embodiment, a classification item is set, a plurality of reference images are set for each set classification item, and each set reference image and classification target are set for each classification item. The similarity of the images to be classified, and the obtained similarities are totaled to obtain the overall similarity of the images to be classified, and the overall similarity of the images to be classified obtained for each classification item and a predetermined classification destination determination Since the classification item that is the classification destination of the image to be classified is determined based on the reference, it is possible to perform image classification processing using AND search processing. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  (19) Further, according to the image classification device of the above embodiment, classification items are set, a plurality of reference images are set for each set classification item, and each set reference image and classification target are set for each classification item. The similarity of the images to be classified is determined, and the similarity indicating the highest similarity is selected from the obtained similarities to obtain the overall similarity of the images to be classified. Based on the total similarity and a preset classification destination determination criterion, a classification item to be a classification destination of the image to be classified is determined, so that it is possible to perform an image classification process using an OR search process. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  (20) Further, according to the image classification device of the above-described embodiment, the degree of importance is further determined for each reference image so that the obtained similarity can be arbitrarily weighted according to the type of the reference image. For each classification item, the similarity between each reference image and the image to be classified is obtained, and the obtained similarity is weighted according to the set importance, Since the user's intention can be accurately expressed as a classification condition by obtaining the similarity, the user's intention can be accurately transmitted to the apparatus, and the importance is reflected as a weight for the similarity. It is possible to obtain an image classification result reflecting the intention.

  (21) Further, according to the image classification device of the above embodiment, when the similarity between the reference image and the image to be classified is obtained using a plurality of types of feature amounts, the reference image is compared with the obtained similarity. In addition, it is possible to set the importance for each type of feature quantity for each reference image so that arbitrary weighting can be performed according to the type of feature quantity. The similarity between each reference image and the image to be classified is calculated for each feature amount, and the similarity obtained for each type of feature amount is weighted according to the set importance, and then the overall similarity is calculated. As a result, the user's intention can be accurately expressed as a classification condition, so that the user's intention can be accurately transmitted to the apparatus, and the importance is reflected as a weight for the similarity. It is possible to obtain a classification result reflects images.

  (22) Further, according to the image classification device of the above embodiment, when the similarity between the reference image and the classification target image is obtained using a plurality of types of feature quantities, the feature quantity is compared with the obtained similarity degree. For each classification item, the similarity between each set reference image is obtained for each type of feature quantity, and for each type of feature quantity so that arbitrary weighting can be performed according to the type of The importance for each feature amount between each reference image is determined based on the similarity, and for each classification item, the similarity between each reference image and the classification target image is obtained for each type of feature amount. After weighting according to the determined importance for the similarity obtained for each feature quantity, the overall similarity is obtained, so that each type of feature quantity between the reference images specified by the user on the device side is obtained. Judgment of importance and display the judged importance To reflect the degree of similarity between, it is possible to return the classification result of the image matching the intention of the user. Therefore, the classification process as desired by the user can be realized, and the classification result desired by the user can be obtained.

  (23) Further, according to the image classification device of the above embodiment, since the classification items for further classifying the images classified for each classification item can be set, the detailed image classification processing can be executed. It becomes possible.

  (24) Further, according to the image classification device of the above embodiment, when the type of feature amount is designated, the similarity between the images to be classified is obtained by using the designated type of feature amount. Based on the similarity between the images to be classified, the images to be classified are classified into groups of similar images, and representative images representing the groups are selected from the classified groups and displayed on the screen. Furthermore, when one of the displayed representative images is specified and the type of feature amount is specified, each image classified into the group to which the representative image belongs using the specified type of feature amount is compared between similar images. By classifying the images into groups and selecting a representative image representing each group from the classified groups and displaying them on the screen, the user can dynamically classify the images according to the criteria desired by the user. Since it is possible, by using the classification of the image it is possible to find a desired image. Therefore, it is possible to provide an image classification device that can classify images by a new classification method.

  (25) Further, according to the image classification device of the above embodiment, the types of feature quantities that can be specified are further displayed on the screen, and the representative images of each group are displayed on the screen in association with the types of the specified feature quantities. Since the result of classifying a plurality of images to be classified is displayed on the screen in a tree structure, the user can recognize the state of the image classification process at a glance, and the convenience of the dynamic image classification process can be improved.

  (26) Further, according to the image classification device of the above embodiment, since an arbitrary image classified for each classification item is used as an inquiry image to search for an image that is the same as or similar to the inquiry image, Convenience can be improved.

  (27) Further, according to the image classification device of the above embodiment, since the search is further performed using the image search device according to any one of the above embodiments, the viewpoint of the user when specifying the search condition Can be reliably transmitted to the apparatus, and a search result suitable for the user's viewpoint can be returned, so that the convenience of the apparatus can be improved.

  (28) Further, according to the image classification device of the above embodiment, since the image to be classified is an image of a search result, for example, it is possible to apply image classification to search result narrowing.

  (29) According to the computer-readable recording medium of the above embodiment, since the program for causing the computer to function as each unit of the image classification device according to any one of the above embodiments is recorded, the program By executing the above, it is possible to reliably convey the user's viewpoint to the apparatus when specifying the classification condition, and to classify the image into the classification items suitable for the user's viewpoint Can be realized. That is, it is possible to realize an image classification device that can appropriately classify images into classification destinations desired by the user.

100 Computer 107 Image Search Software 108 Image DB
109 Image feature DB
151 U / I unit 152 Feature extraction engine 153 Matching engine 200a to 200h Search window 201 Search button 202 Image display field 203, 362 Feature amount designation button 204, 207, 208, 363, 366, 367 Importance level designation bar 205, 364 AND designation buttons 206 and 365 OR designation button 209 Feature designation column 210 Roughness level designation column 301 Image classification software 302 Classification tree DB
350a-350f Image classification window 351-358 Folder 360a-360f Classification condition setting window 361 Setting button 400 Image classification window 401, 402a-404a, 402b-404b Classification item 405-412 Representative image 450 Recording medium 451 Server

Claims (8)

For each classification item that is the classification destination of the classification target image, the classification target image is a classification reference for the classification target image, and one of the plurality of characteristic amount types representing the feature of the image Setting means for setting one or more reference images associated with each other and a predetermined threshold value for determining whether to classify the image into the classification items based on the similarity of the images;
For each of the reference images, calculation means for calculating a similarity with respect to the classification target image with respect to the feature type of the associated type;
Based calculated similarity and the said threshold value, an image of the classified and a determining means for determining whether or not to classify the categories,
The setting means further sets the importance of the similarity for each classification item that is the classification destination of the image to be classified,
The determining means calculates a total similarity obtained by weighting and summing the similarities calculated for each of the reference images with the importance, and based on the total similarity and the threshold value, determines the images to be classified. An image classification apparatus for determining whether to classify into the classification items .   The image classification apparatus according to claim 1, wherein the type of the feature amount includes an image color, an image structure, and an image texture. The setting means sets the importance for each of the reference images;
The determining means calculates a total similarity obtained by weighting and summing the similarities calculated for each of the reference images with the importance set for the reference image, and calculating the total similarity and the threshold value . The image classification apparatus according to claim 1 , wherein the image classification apparatus determines whether to classify the image to be classified into the classification items based on the classification. The setting means, the image classification apparatus according to any one of claims 1 to 3, characterized in that it is possible to set the categories for further classifying the classified images for each of the classification items. The determining means determines whether to classify the image to be classified into the classification items based on the highest similarity calculated for each of the reference images and the threshold value. The image classification device according to claim 1. Selecting means for selecting a representative image from the images classified into the classification items;
Display means for displaying the representative image for each classification item,
The image classification device according to claim 1. For each of the reference images, further comprising importance level determining means for determining the importance level of the similarity for each type of the feature amount,
The image classification device according to claim 1. A computer-readable recording medium having recorded thereon a program for causing a computer to function as each unit of the image classification apparatus according to any one of claims 1 to 7 .

Images