JP2007317133A - Image classification method, device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely classify images corresponding to concepts on the Internet. <P>SOLUTION: When a keyword serving as a query is given, images corresponding to a plurality of concepts of the keyword are gathered by referring to explanation text around the images. In a feature space configured of feature amounts extracted from the images, an identification function for identifying the images corresponding to the respective concepts is found by extracting proper learning data from a group of images corresponding to the respective concepts, and based on the identification function, the images matching the respective concepts are classified. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image classification method, apparatus, and program, and in particular, image classification for classifying an image as a search result according to image content when searching for an image by a keyword for multimedia information expressed by text and an image. The present invention relates to a method, an apparatus, and a program.

  With advances in computer and internet technology, it is becoming easier to generate and store image data in the real world. On the other hand, an efficient management technique for enormous amounts of image data has been demanded. As one of the core technologies, Internet image classification technology has attracted attention. Conventional image classification methods are classified into the following three approaches.

(1) Classification by text matching:
This is a method of classifying images by manually assigning keywords to images in advance and determining similarity of images by keyword matching.

(2) Classification by image matching:
There is a method of extracting feature quantities such as color, texture, and shape from an image and classifying images by collation using these feature quantities (see, for example, Non-Patent Document 1).

(3) Classification by text and image integration processing:
When searching for an Internet image by text, a method of classifying and presenting images having similar image contents when presenting an image related to a query word as a search result has been studied (for example, Non-Patent Document 2). reference). This method is realized by the following two steps. Web sites including Internet images are often composed of images and explanatory texts that describe the images. First, images related to the query words are collected by matching the query words with the explanatory texts. Next, the collected images are classified using the feature quantities related to the color and texture of the image. This method differs from the above technique (2) in that an image obtained by collecting related images using explanatory text is used as a classification target, so that images having many similar contents can be classified.
Yixin Chen, James Z Wang, Robert Krovets, “content-based image retrieval by clustering”, Multimedia Information Retrieval 2003, pp.193-200 Deng Cai, Xiaofei He, Zhiwei Li, Wei-Ying Ma and Ji-rong Wen, “Hierarchical Clustering of WWW Image Search Results Using Visual, Textual and Link Analysis”, 12th ACM International conference on Multimedia, New York City, USA, Oct . 2004

  However, in the collation by the text of the prior art (1), it is necessary to manually add a text (index) representing the meaning and contents of the image to each image in advance. The problem is that it takes a lot of time and cost to apply the above.

  In the matching by the image of the prior art (2), classification is performed only by physical feature amounts such as color, texture, and shape extracted from the image. Since it is not possible to clearly define the correspondence relationship, it is difficult to perform semantic classification of images.

  The classification based on the integration of the text and the image in the prior art (3) is performed in the order of first obtaining an image by text collation and then classifying by image collation. Since collation by image has the same problem as the prior art (2), it is necessary to collect images that are as easy to classify as possible by text collation performed first. However, with image collection by simple text matching, all images corresponding to various concepts included in the query word are collected, so that image classification based on image contents is difficult as in the case of the prior art (2). The classification accuracy is low.

  The present invention has been made in view of the above points, and an object thereof is to provide an image classification method, apparatus, and program capable of accurately classifying images corresponding to a concept.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is an image classification method for classifying a search result image according to image content when searching for an image by a keyword for multimedia information expressed by text and an image,
A concept / thesaurus acquisition unit searches a word dictionary storing a plurality of concepts of a word and a thesaurus of each concept based on an inputted query keyword, and acquires n concepts and a thesaurus of the concept A thesaurus acquisition step (step 1);
The image collecting means searches the web by the AND condition of the concept thesaurus and query keyword, collates the explanatory text around the image, collects the query keyword and the image related to the thesaurus, and stores the image in the storage means A collection step (step 2);
For each concept n (n = 1, 2,..., N), the image filtering means has a high similarity when the query keyword and the concept thesaurus are AND-searched from the image group collected by the storage means. An image filtering step (step 3) for extracting the image of (2) as a positive case candidate and the other images as unlabeled images;
A positive case generation unit extracts an image feature amount for a positive case candidate and sets a positive case as an example of an image whose distance from the center of the distribution in the image feature space is equal to or less than a preset threshold ( Step 4) and
The negative case generation means selects a feature amount whose variation is smaller than a preset threshold from the image feature amounts extracted from each image of the positive case, and obtains a negative case from the distribution of the unlabeled image group for the feature amount. A case generation step (step 5);
Discriminant function calculating means for determining a discriminant function for discriminating between positive and negative cases using the positive case obtained in the positive case generating step and the negative case obtained in the negative case generating step as learning data (step 6) and
An identification step (step 7) in which an identification unit obtains an image corresponding to the concept n from the positive case candidate and the unlabeled image group using an identification function;
The processing after the image collection step is repeated for all concepts n (step 8).

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 2) is an image classification device for classifying an image of a search result according to image contents when searching for an image by a keyword for multimedia information expressed by text and an image.
A word dictionary 107 storing a plurality of concepts of words and a thesaurus of each concept;
A concept / thesaurus acquisition unit 100 that searches the word dictionary 107 based on the input query keyword and acquires n concepts and a thesaurus of the concept;
An image collecting unit 101 that performs a web search using an AND condition of a conceptual thesaurus and a query keyword, collates explanatory texts around the image, collects the query keyword and an image related to the thesaurus, and stores them in a storage unit;
For each concept n (n = 1, 2,..., N), M images having a high degree of similarity when an AND search of a query keyword and a concept thesaurus is performed on a group of images collected by the storage means is a positive example. Candidate, image filtering means 102 for other images as unlabeled images,
A positive case generation unit 103 that extracts an image feature amount for a positive case candidate and sets an image whose distance from the center of distribution in the image feature space is equal to or less than a preset threshold value as a positive case;
A negative case generation unit 104 that selects a feature amount whose variation is smaller than a preset threshold value from image feature amounts extracted from each image of the positive case, and obtains a negative case from a distribution of unlabeled image groups with respect to the feature amount;
A discriminant function calculating unit 105 for obtaining a discriminant function for discriminating between a positive case and a negative case using the positive case obtained by the positive case generating unit 103 and the negative case obtained by the negative case generating unit 104 as learning data;
An identification unit 106 for obtaining an image corresponding to the concept n from the positive case candidate and the unlabeled image group using an identification function;
For all concepts n, the image acquisition means 101, the image filtering means 102, the positive case generation means 103, the negative case generation means 104, the discrimination function calculation means 105, and the means for repeating the discrimination means 106 are provided.

  The present invention (Claim 3) is an image classification program for causing a computer to execute each means of the image classification apparatus according to Claim 2.

In the present invention, the Internet image is
・ Various images taken by various people for various purposes;
The meaning of the image is expressed by surrounding text information and image information:
First, when a keyword to be queried is given, images corresponding to a plurality of concepts possessed by the keyword are collected by collating explanatory texts around the image, and from the image In a feature space composed of extracted feature quantities, an identification function for identifying an image corresponding to each concept is obtained by extracting appropriate learning data from an image group corresponding to each concept, and based on the identification function To classify the images corresponding to each concept. Thereby, the image corresponding to the concept can be classified with high accuracy.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a system configuration according to an embodiment of the present invention.

  The system shown in the figure has a configuration in which a client terminal 1, a web image search engine 2, and a website 3 are connected to the Internet 4.

  The client terminal 1 accepts a keyword serving as a query that is a user search request, and presents an image searched based on the query word.

  The web image search engine 2 performs text matching between the query word and the explanatory text of the website 3, and searches for a web image associated with the query word.

  The website 3 is a large number of websites published on the Internet, and the published information is composed of images and surrounding explanatory text.

  The client apparatus 1 when the system shown in FIG. 3 is assumed is described as an image classification apparatus.

  FIG. 4 shows the configuration of the image classification device in one embodiment of the present invention.

  The image classification apparatus shown in the figure includes a concept / thesaurus acquisition unit 100, an image collection unit 101, an image filtering unit 102, a positive case generation unit 103, a negative case generation unit 104, an identification function calculation unit 105, an identification processing unit 106, a word The dictionary storage unit 107 is configured.

  The concept / thesaurus acquisition unit 100 reads each concept of the query keyword and the thesaurus of each concept from the word dictionary stored in the word dictionary storage unit 107 based on the query keyword input from the user, Each read concept and thesaurus is output to the image collection unit 101.

  The word dictionary storage unit 107 is an electronic dictionary that summarizes a plurality of concepts possessed by a word and a thesaurus of each concept. When a word is received from the concept / thesaurus acquisition unit 100, the concept and thesaurus possessed by the word are converted into a concept / thesaurus acquisition unit. Output to 100. An example of a word dictionary is shown in FIG.

  When the image collection unit 101 receives the query keyword and the thesaurus related to each concept from the concept / thesaurus acquisition unit 100, the web search engine uses the query keyword and the thesaurus related to each concept as a search condition (AND condition) around the web image. It collates with a certain explanatory text, collects images related to the query word and the thesaurus, and stores them in storage means (not shown) such as a memory. The text similarity in collation between the collected image and the explanatory text when searched by the web image search engine 2 is output to the image filtering unit 102.

  When the image filtering unit 102 receives the text similarity with the image from the image acquisition unit 101, the image filtering unit 102 divides the M images having high text similarity into two groups of positive case candidate images and the other images as unlabeled images. The positive case candidate image is output to the positive case generation unit 103 and the identification processing unit 106, and the unlabeled image is output to the negative case generation unit 104 and the identification processing unit 106.

  When the positive case generation unit 103 receives the positive case candidate image from the image filtering unit 102, the positive case generation unit 103 extracts a representative feature amount that characterizes the positive case image, and detects the positive case image using the extracted representative feature amount. The detected positive case image is output to the discriminant function calculation unit 105, and the extracted positive case feature amount (representative feature amount) is output to the negative case generation unit 104. Details of the extraction of the correct case image will be described later with reference to the flowchart of FIG.

  When the negative case generation unit 104 receives the unlabeled image from the image filtering unit 102 and the representative feature amount from the positive case generation unit 103, the negative case generation unit 104 uses the representative feature amount to obtain a negative value from the unlabeled image received from the image filtering unit 102. A case image is extracted, and the extracted negative case image is output to the discrimination function calculation unit 105. Details will be described later in the processing flow of FIG.

  The discriminant function calculating unit 105 obtains a discriminant function for discriminating between the negative case and the positive case by using the positive case image received from the positive case generating unit 103 and the negative case image received from the negative case generating unit 104 as learning data, The obtained discrimination function is output to the discrimination processing unit 106. The discrimination function can be calculated using, for example, a conventional classifier called BDA (Reference 1: Xiang Sean Zhou, Thomas S. Huang, “Comparing Discriminating transformations and SVM for learning during multimedia retrieval”, ACM Multimedia 2001. , pp. 137-146).

  The identification processing unit 106 classifies the positive case candidate image and the unlabeled image input from the image filtering unit 102 based on the identification function input from the identification function calculation unit 105. The result classified as the positive case is output as an image corresponding to the corresponding concept of the query keyword.

  Next, the operation in the above configuration will be described.

  FIG. 6 is a flowchart of the process of the image classification device according to the embodiment of the present invention.

  Step 201) The concept / thesaurus acquisition unit 100 acquires a keyword to be a query.

  Step 202) The concept / thesaurus acquisition unit 100 accesses the word dictionary storage unit 107 based on the query keyword, and reads N concepts possessed by the query word and a thesaurus corresponding to each concept.

  Step 203) In the image collection unit 101, images are retrieved by using the web image search engine 2 via the Internet 4 under the search condition (AND condition) in which the nth concept thesaurus is added to the query word. Then, the image and the similarity are stored in a storage means (not shown) such as a memory. The image search may be performed by collating texts such as the description text or title around the web image with the search condition, and detecting images on the Internet in descending order of similarity.

  Step 204) The image filtering unit 102 classifies the top M images having high similarity in text matching of the images collected by the image collecting unit 101 as positive case image candidates and the other images as unlabeled images.

  Step 205) The positive case generation unit 103 obtains a positive case image from the positive case image candidate images input from the image filtering unit 102. The processing contents will be described with reference to FIG.

First, the correct case generation unit 103 reads a correct case candidate image (step 301). Next, P feature amounts such as color (f ^C ), texture (f ^T ), shape (f ^S ), etc. are extracted from the acquired positive case candidate images (step 302). Here, the feature quantity of each color, texture, and shape is a multidimensional feature vector obtained by the prior art.

  For each of the P feature amounts, the variation of the entire positive case candidate image is obtained (step 303). First, for the color feature space, the standard deviation of the entire positive case candidate image is obtained using the following function:

Ｃは、正事例候補画像全体の中心、Ｍは正事例候補画像の総数、ｆ_m ^Cはｐ次元色特徴ベクトルである。

C is a positive case candidate entire image center, M is the total number of positive case candidate image, f _m ^C is a p-dimensional color feature vector.

  Similarly, for P feature spaces such as texture and shape, the standard deviation in the entire positive case candidate image is calculated. By this process, P standard deviations are obtained.

Next, the feature quantity having the smallest P standard deviation is detected and set as the representative feature quantity (I) (step 304). For example, if the standard deviation of the color feature quantity (f ^C ) is the smallest among the P feature quantities such as color (f ^C ), texture (f ^T ), and shape (f ^S ), the representative feature quantity ( I) is a color feature amount.

  The center C of the positive case candidate image in the feature space of the representative feature quantity (I) determined in step 304 is obtained (step 305).

  Next, the distance between the positive case candidate image n and the center C in the feature space of the representative feature quantity (I) is calculated (step 306). Here, the distance is calculated using a general Euclidean distance calculation formula.

  It is determined whether the distance obtained in step 306 is equal to or less than a preset threshold value. If it is equal to or smaller than the threshold value, the process proceeds to step 308. If it is greater than the threshold value, the process proceeds to step 309.

  If it is less than or equal to the threshold value, the positive case candidate image n is assigned to the positive case (step 308).

  If it is larger than the threshold value, it is determined whether or not the processing from step 306 to step 308 has been performed on all the positive case candidate images. If not, the process proceeds to step 310, and if not, the processing ends.

  n is incremented and the routine proceeds to step 306 (step 310).

  Step 206) The negative case generation unit 104 extracts a negative case image. Hereinafter, the operation of the negative case generation unit 104 will be described in detail with reference to FIG.

  First, the negative case generation unit 104 extracts the representative feature quantity (I) from the unlabeled image. For example, if the representative feature amount (I) obtained in step 205 is a color feature amount, the color feature amount is also extracted from the unlabeled image (step 41).

  Among the unlabeled images, those having a distance larger than the threshold from the center C of the positive case candidate image in the feature space of the representative feature amount (I) obtained in step 305 are detected as negative cases (step 42).

Step 207) The discrimination function calculation unit 105 uses the positive case image generated by the positive case generation unit 103 and the negative case image generated by the negative case generation unit 104 as a positive case and negative case learning data as a negative case. An identification function for identifying a positive case is obtained. As an example of obtaining the discriminant function, a method using a conventional classifier called BDA (Biased Discriminate Analysis) is effective (see Document 1). The BDA classifier determines a weighting factor that minimizes the variance of the positive case group and maximizes the variance of the negative case group and the positive case group. FIG. 9 shows an example of the BDA classifier. A matrix representing a weighting coefficient for each dimension of the representative feature quantity (I) is represented by W, a positive case image represented by the representative feature quantity (I) is represented by _fl , a total number of positive case images is represented by L, and a representative feature quantity (I). If the negative case image is nf _j , the total number of negative case images is J, and the center of the positive case image in the representative feature space (I) is C, the optimal weighting factor for achieving the above goal is )

式（１）より、Ｗは、

From equation (1), W is

により求められる。

Is required.

  Step 208) Using the learned classifier, the discrimination processing unit 106 discriminates between the correct case candidate image and the unlabeled image, and outputs the sample determined as the correct case as an image for the nth concept of the query keyword. . The weighting W obtained by the learned BDA classifier is applied to the image feature amount to obtain the classification degree S by the following equation. If S is equal to or less than a preset threshold value, S is an image for the nth concept.

S = W ^T · f _i
Step 209) The identification processing unit 106 determines whether the image classification processing from Step 203 to Step 208 has been performed on the N concepts related to all query keywords. If not, the processing from step 203 to step 208 is repeated. Otherwise, go to Step 210.

  Step 210) The identification processing unit 106 presents an image corresponding to the N concepts related to the query keyword as a result.

  Further, the present invention constructs each function of the configuration of the image classification apparatus shown in FIG. 4 as a program and installs and executes it on a computer used as the image classification apparatus or distributes it via a network. Is possible.

Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed on a computer used as an image classification apparatus.
The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for classifying images existing on the Internet.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. 1 is a system configuration diagram according to an embodiment of the present invention. It is a block diagram of the image classification device in one embodiment of the present invention. It is an example of the word dictionary in one embodiment of this invention. It is a flowchart of the process of the image classification device in one embodiment of this invention. It is a flowchart of the detailed process of the positive case image extraction s205 in one embodiment of this invention. It is a flowchart of the process of negative case image extraction s206 in one embodiment of this invention. It is a figure explaining the BDA classifier in one embodiment of this invention.

Explanation of symbols

1 Client Terminal 2 Web Image Search Engine 3 Website 4 Internet 100 Concept / Thesaurus Acquisition Unit, Concept / Thesaurus Acquisition Unit 101 Image Collection Unit, Image Collection Unit 102 Image Filtering Unit, Image Filtering Unit 103 Positive Case Generation Unit, Positive Case Generation Unit 104 negative case generation unit, negative case generation unit 105 identification function calculation unit, identification function calculation unit 106 identification unit, identification processing unit 107 word dictionary, word dictionary storage unit

Claims (3)

An image classification method for classifying multimedia information expressed in text and images according to image content when searching for images by keywords,
A concept / thesaurus acquisition unit searches a word dictionary storing a plurality of concepts of a word and a thesaurus of each concept based on an inputted query keyword, and acquires n concepts and a thesaurus of the concept A thesaurus acquisition step;
The image collection means searches the web based on the AND condition of the concept thesaurus and the query keyword, collates the explanatory text around the image, collects the query keyword and the image related to the thesaurus, and stores them in the storage means An image collection step to
The degree of similarity when the image filtering means performs an AND search of the query keyword and the concept thesaurus from the image group collected in the storage means for each concept n (n = 1, 2,..., N). An image filtering step of extracting M images having a high value as candidates for positive cases and other images as unlabeled images;
A positive case generation unit that extracts an image feature amount for the positive case candidate and sets a positive case as an image whose distance from the center of the distribution in the image feature space is a predetermined threshold or less. When,
A negative case generation means selects a feature amount whose variation is smaller than a preset threshold value from the image feature amounts extracted from each image of the positive case, and selects a negative case from the distribution of the unlabeled image group for the feature amount. The negative case generation step to be sought,
Discrimination function calculating means for obtaining a discrimination function for identifying a positive case and a negative case using the positive case candidate obtained in the positive case generation step and the negative case obtained in the negative case generation step as learning data A function calculation step;
An identification step for identifying an image corresponding to the concept n using the identification function from the positive case candidate and the unlabeled image group;
An image classification method, wherein the processing after the image collection step is repeated for all concepts n. An image classification device that classifies images of search results according to image contents when searching for images by keywords for multimedia information expressed in text and images,
A word dictionary that stores multiple concepts of words and the thesaurus of each concept;
A concept / thesaurus acquisition means for searching the word dictionary based on the input query keyword and acquiring n concepts and a thesaurus of the concept;
An image collecting means for performing a web search according to an AND condition of the conceptual thesaurus and the query keyword, collating explanatory texts around the image, collecting the query keyword and an image related to the thesaurus, and storing the collected information in a storage means; ,
For each concept n (n = 1, 2,..., N), M images having a high degree of similarity when an AND search of the query keyword and the concept thesaurus is performed on the collected image group of the storage means. An image filtering means for setting the image as a positive case candidate and the other images as unlabeled images;
For the positive case candidate, a positive case generation unit that extracts an image feature amount and sets an image whose distance from the center of distribution in the image feature space is equal to or less than a preset threshold as a positive case;
Negative case generation means for selecting a feature amount whose variation is smaller than a preset threshold from the image feature amount extracted from each image of the positive case, and obtaining a negative case from a distribution of unlabeled image groups for the feature amount; ,
An identification function calculating means for obtaining an identification function for identifying a positive case and a negative case by using the positive case determined by the positive case generating means and the negative case determined by the negative case generating means as learning data;
Identification means for obtaining an image corresponding to the concept n from the positive case candidate and the unlabeled image group using the identification function;
For all concepts n, the image collection means, the image filtering means, the positive case generation means, the negative case generation means, the discrimination function calculation means, means for repeating the identification means,
An image classification apparatus comprising: On the computer,
An image classification program for causing each means of the image classification apparatus according to claim 2 to be executed.

Images