JP2000112956A - Information retrieval device/method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely take out objective multimedia information against a given retrieval word. SOLUTION: A retrieval request input processing 201 holds an inputted retrieval word in a retrieval word holding part 202. An associated word development processing 208 refers to an associated word dictionary 211 and obtains an associated word on the retrieval word held in the retrieval word holding part 202. A picture content retrieval processing 210 by the associated word retrieves the keyword of picture information based on the associated word obtained and the retrieval word. On the other hand, a sensibility pattern development processing 212 obtains a sensibility pattern corresponding to the retrieval word and the associated word. A sensibility pattern retrieval processing 214 retrieves picture information by using the feature quantity of the obtained sensibility pattern. A retrieval result integration processing 217 integrates retrieval results obtained by the picture content retrieval processing 210 and the sensibility pattern retrieval processing 214.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information retrieval apparatus and method for retrieving information based on a given retrieval word.

[0002]

2. Description of the Related Art Conventionally, in an information retrieval apparatus for retrieving multimedia information, for example, image information, data (keywords) derived from subjective evaluation by one or more persons for each image to be retrieved or extracted from each image Search has been performed using physical image features and the like.

[0003] For example, there is a method in which impressions received by a person as a result of viewing an image and key information associated with the impression are added to the image management information and used for retrieval. Key information may include words that express the impression received from the image, such as "warm feeling" or "cold feeling", or information drawn on the image, such as "kitten,""sea," or "mountain." Some words have words added as keywords. In some cases, some image characteristic elements drawn in an image are subjectively evaluated and added as key information. For example, information on colors such as "red" and "blue", information on shapes such as "round", "triangle", and "pointed", and information on textures such as "gritty feeling" Some are expressed using icons, added to images as key information, and used for search.

[0004] On the other hand, there is also a system that extracts a physical image feature amount from an image and uses it for image retrieval. Image features include, for example, some colors and overall tones depicted in the image,
There are shapes, compositions, textures, etc. of what is depicted in the image. The image feature amount is extracted from a divided region / block obtained by, for example, dividing the entire image into regions based on color information, dividing the image into blocks of a fixed area, or the like.
Or extracted from the entire image. Physical image features include, for example, color information, image density distribution, texture, edge, area, area, position, frequency distribution, and the like.

[0005]

However, in the above conventional example, when an image including a keyword that matches the input search word is searched, an image that does not meet the search request of the searcher may be obtained. In particular, when an image search is performed using an abstract search word such as a "fresh" image, the searched image is significantly limited. In order to solve this, it has been proposed to expand the search term "fresh" to a keyword associated with it and perform a search, but using such a method, non-fresh images are mixed in the search results. Could be done.

It has also been proposed to provide a search image instead of a search word and perform a search using the feature amount. In this case, a search image reflecting the intention of the searcher is prepared. Operability is poor, for example, it is necessary to select a search image.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and provides an information retrieval method and apparatus capable of accurately extracting target information from a given retrieval word. With the goal.

[0008]

An information retrieval apparatus according to one aspect of the present invention for achieving the above object has, for example, the following configuration. That is, an information search device that searches for information based on an input search word, a first search unit that determines a search keyword based on the search word, and searches for information based on the search keyword, A second search unit that determines a feature amount of a pattern corresponding to the search word and searches for information based on the feature amount;
And integration means for integrating the search results obtained by the second search means.

An information search method according to another aspect of the present invention for achieving the above object includes, for example, the following steps. That is, an information search method for searching for information based on an input search word, wherein a first search step of determining a search keyword based on the search word and searching for information based on the search keyword, A second search step of determining a feature amount of a pattern corresponding to the search word and searching for information based on the feature amount;
An integration step of integrating search results obtained in the search step.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram showing an example of the configuration of an information processing device for configuring the image search device of the present embodiment.

In FIG. 1, reference numeral 11 denotes a microprocessor (hereinafter referred to as a CPU), which performs an operation and a logic judgment for image information retrieval in accordance with a control program, and also executes an address bus AB, a control bus CB and a data bus D
B controls each component connected to those buses. The address bus AB transfers an address signal indicating a component to be controlled by the CPU 11. The control bus CB transfers a control signal to each component to be controlled by the CPU 11. Data bus D
B performs data transfer between the constituent devices.

Reference numeral 12 denotes a read-only fixed memory (hereinafter referred to as R
OM), and stores a boot processing program executed by the CPU 11 when the apparatus is started. Also,
Reference numeral 13 denotes a writable random access memory (hereinafter referred to as a RAM) composed of 16 bits per word, which is used for temporarily storing various data from each component. In particular, in the present embodiment, the RAM 13 includes a search word holding unit 202, a search viewpoint holding unit 203, a search weight holding unit 204, a decision weight holding unit 207, a developed associative word holding unit 209, and a developed sentiment pattern holding unit described later with reference to FIG. 213, various data holding units such as a search result holding unit 216 are provided.

Reference numeral 14 denotes an external memory (DISK) which stores a concept discrimination dictionary 205 and an associative word dictionary 211, which will be described later with reference to FIG. 2, as well as an image word and sensitivity pattern correspondence holding unit 215, an image content word holding unit 219, Image holding unit 218, Kansei pattern holding unit 220, Image feature amount holding unit 222, Image feature amount and Kansei pattern correspondence holding unit 2
23 is provided. Note that as a recording medium of the external memory 14, a ROM, a floppy disk, a CD-ROM, a memory card, a magneto-optical disk, or the like can be used.

The external memory 14 includes a search request input process 201 and a weight determination process 20 described later in FIG.
6, associative word expansion processing 208, image content word search processing 210 using associative words, kansei pattern expansion processing 212, kansei pattern search processing 214, search result integration processing 217, image feature extraction processing 221, and kansei pattern determination processing 224 Is stored.

Reference numeral 15 denotes a keyboard (KB) which includes a character / character input key such as an alphabet key, a hiragana key, a katakana key, and a punctuation mark, and a search key for instructing a search (function keys of a general keyboard are used as substitutes). And various function keys such as a cursor movement key for instructing cursor movement. Although not shown, a pointing device such as a mouse may be connected.

Reference numeral 16 denotes a display video memory (hereinafter referred to as VRA).
M) and stores a pattern of data to be displayed.
Reference numeral 17 denotes a CRT controller (hereinafter, CRTC),
It plays a role of displaying the contents stored in the VRAM 16 on the CRT 18. Reference numeral 18 denotes a display device using a cathode ray tube or the like (CR
T), the display pattern of the dot configuration on the CRT 18 and the display of the cursor are controlled by the CRTC 17. It is needless to say that various displays such as liquid crystal can be used as the display device. Reference numeral 19 denotes a network interface card (NIC), which plays a role of connecting the image search device to a network such as Ethernet.

The image retrieval apparatus according to the present embodiment comprising the above-described components operates according to various inputs from the keyboard 15 and various inputs via the network supplied from the network controller. When there is an input from the keyboard 15 or an input from the network controller 19, an interrupt signal is sent to the CPU 11.
Upon receiving the interrupt signal, the CPU 11 reads out various control data stored in the external memory 14 and
Various controls are executed according to the control data. Note that a recording medium storing a control program for realizing the processing specified in the present embodiment is supplied to a system or an apparatus, and a computer of the system or the apparatus reads out and executes the program code stored in the recording medium. Thus, the present invention is achieved.

FIG. 2 is a block diagram schematically illustrating a processing configuration in the image retrieval apparatus according to the present embodiment.

In FIG. 2, reference numeral 201 denotes a search request input process, which is a request related to a desired search target (here,
(Search terms, search viewpoints, search weights, details will be described later.)
Enter Reference numeral 202 denotes a search term holding unit that stores the search term input by the search request input process 201. 2
Reference numeral 03 denotes a search viewpoint holding unit, and a search request input process 201
Is stored. A search weight holding unit 204 stores the search weight input by the search request input processing 201.

Reference numeral 205 denotes a concept discrimination dictionary. As shown in FIG. 8, a search viewpoint related to the concept to be searched, a dissenting concept having a meaning opposite to the concept to be searched, and a concept for searching for a concept. Has two types of coefficients (associative weight and sensibility pattern weight) for weight determination. Note that the concept discrimination dictionary 2
05 will be described later with reference to FIG. 20
Reference numeral 6 denotes a weight determination process, which is performed when searching for a search word stored in the search word holding unit 202.
08) or a sentiment pattern (obtained by the sentiment pattern development process 212) is given a weight (associative weight, sentiment pattern weight) indicating whether to search with emphasis. A determination weight holding unit 207 stores the search weight determined in the weight determination processing 206.

Reference numeral 208 denotes an associative word expansion process, which expands the search word stored in the search word holding unit 202 into an associative word with reference to the associative word dictionary 211, and converts a dissenting concept corresponding to the search word into a concept discrimination dictionary. With reference to the associative word dictionary 211 obtained from 205, the obtained opposite meaning concept is developed into an associative word. Reference numeral 209 denotes an associative word holding unit that stores the associative words (including the associative words of the negative concept) expanded by the associative word expansion processing 208. Reference numeral 210 denotes an image content word search process using an associative word (hereinafter, referred to as an image content word search process 210), which refers to the expanded associative word storage unit 209 and expands the image content words stored in the image content word storage unit 219. Search for a word that matches the associated word. Reference numeral 211 denotes an associative word dictionary, which stores words associated with each concept as a heading in association with an associative viewpoint (details will be described later with reference to FIG. 9).

Reference numeral 212 denotes a sentiment pattern developing process, which develops the search word stored in the search word holding unit 202 into a sentiment pattern by referring to the image word-kansei pattern correspondence holding unit 215, and performs the opposite of the search word. The concept is obtained from the concept discrimination dictionary 205, and the obtained antisense concept is developed into a sensitivity pattern with reference to the image word-sensitivity pattern correspondence holding unit 215.

Reference numeral 213 denotes a development sensitivity pattern holding unit.
The sentiment patterns developed in the sentiment pattern development process 212 are temporarily stored. The storage of data in the development sensitivity pattern holding unit 213 will be described later with reference to FIG. Reference numeral 214 denotes a sentiment pattern search unit, which refers to the developed sentiment pattern holding unit 213 and
A search is made for a sentiment pattern stored in 20 that is similar to the developed sentiment pattern.

Reference numeral 215 denotes an image word-kansei pattern correspondence holding unit which stores a sentiment pattern ID corresponding to a set of an image word and each associative word associated with the image word, so that the correspondence between the image word and the sentiment pattern is stored. Is stored. Note that the image word-sensitive pattern correspondence holding unit 21
Details of 5 will be described later with reference to FIG.

Reference numeral 216 denotes a search result holding unit which stores the image IDs searched by the image content word search processing 210 and the affective pattern search processing 214. A search result integration process 217 obtains the search result of the image content word and the search result of the sensibility pattern by the associative word stored in the search result storage unit 216 by the weight determination process 206. Are integrated based on the decision weights stored in.

An image holding unit 218 stores image information to be searched. Reference numeral 219 denotes an image content word holding unit that describes and stores, in a language, a concept given to describe the content of the image stored in the image holding unit 218. Reference numeral 220 denotes a feeling pattern holding unit that stores the degree of matching between the image information of each image stored in the image holding unit 218 and the feeling pattern. That is, the sensitivity pattern holding unit 2
20 stores the degree of matching with each emotion pattern for each image ID. The data content of the affective pattern holding unit 220 will be described later with reference to FIG.

An image feature extraction process 221 extracts a physical image feature amount from each piece of image information stored in the image holding unit 218. Physical image features include areas divided based on color information, blocks divided by a certain area,
Or a visual feature extracted from the entire image. For example, the image feature is digitized information such as a color distribution and a histogram of the image, a density distribution of the image, a texture, an edge, and a frequency distribution. The details of the image feature amount are shown in FIG.
It will be described later with reference to FIG.

Reference numeral 222 denotes an image feature storage, which stores the image feature of each image obtained by the image feature extraction processing 221. Reference numeral 223 denotes an image feature-feeling pattern correspondence holding unit, which stores the correspondence between the image feature quantity and the feeling pattern. The image feature amount-kansei pattern correspondence holding unit 223 stores a Kansei pattern ID and image feature amount data corresponding to each ID. Note that the data configuration of the image feature-sensitivity pattern correspondence holding unit 223 will be described later with reference to FIG.

Reference numeral 224 denotes a feeling pattern determination process, which refers to the image feature amount holding unit 222 and the image feature amount-sensitive pattern correspondence holding unit 223, and extracts the image feature amount extracted from the feeling pattern and the image extracted from the image information. The degree of matching is obtained by comparing / collating with the feature amount, and the result is registered in the above-described emotion pattern holding unit 220.

FIG. 3 is a diagram showing an example in which a search viewpoint related to the search request word input in the search request input processing 201 is displayed. If the search term is keyboard 15
Is operated, the concept discriminating dictionary 205 as shown in FIG. 8 is searched.
The corresponding search viewpoint is extracted. In FIG. 3, there are three search viewpoints “color tone”, “taste”, and “general atmosphere” with respect to the search request word “bitter”. In this case, the shaded “color tone” is selected. It indicates that When the OK button is pressed (clicked) in this state,
The search viewpoint “tone” is selected and stored in the search viewpoint storage unit 203.

By pressing the cursor movement key on the keyboard 15, the shaded display section changes from "color tone" to "taste" or "general atmosphere", so that a desired search viewpoint can be selected. Can be. The search request word “bitter” is stored in the search word storage unit 202, and the selected search viewpoint is stored in the search viewpoint storage unit 203.

FIG. 4 shows the search request input processing 201.
It is a figure which shows the example of a display of the weight designation | designated operation panel for designating the search weight by an association word, and the search weight by a feeling pattern. As described above, in the present embodiment, the search based on the associative word and the search based on the feature amount of the image (kansei pattern) based on the search word are performed to integrate the two search results. Is weighted. On the weight designation operation panel, it is possible to designate the search weight based on the association word and the search weight based on the emotion pattern. That is, it is possible to specify which of the search based on the associative word and the search based on the sensibility pattern is emphasized.

In FIG. 4, when the slider button 41 is slid to the left, an instruction is given to attach importance to the search by the associative word, and when the slider button 41 is slid to the right, an instruction is attached to attach importance to the search by the sensibility pattern. After the weight is designated by the slider button 41, the OK button 43 is pressed to issue a weight setting instruction. The button 42 is pressed when the weight is not explicitly specified. When the button 42 is selected, a predetermined weight value (obtained from the association weight 83 and the sensitivity pattern weight 84 of the concept discrimination dictionary 205) is used. The weight set in this way is stored in the search weight holding unit 204. It should be noted that pressing of each of the buttons 41 to 43 on the weight designation operation panel may be a click operation using a pointing device (not shown).

FIG. 5 is a diagram showing a data configuration of the image holding unit 218 in which image IDs and image file storage paths are associated with each other. In FIG. 5, reference numeral 51 denotes an image ID, which is an identification number uniquely assigned to one image file in the image database. Reference numeral 52 denotes a file path indicating the storage location of the image file corresponding to the image ID in the external memory 14, which corresponds to the directory and file name of the MS-DS.

Although not shown, each image file is divided into a header section and an image data section, as is well known. The header section stores information necessary for reading data from the image file and additional information describing the content of the image, an image format identifier indicating the image format name, a file size, and a width and height of the image. Sa depth
Information such as the presence / absence of compression, color palette information, resolution, and offset to the storage position of image data is stored.
The image data section is a section that stores image data sequentially. In the present embodiment, the BMP format of Microsoft Corporation is used as such an image format.
The same configuration can be achieved by using other compression formats such as G and Flash Pix.

FIG. 6 is a diagram showing an example of the data configuration of the image content word holding unit 219 in which image IDs are associated with image content words. 6, reference numeral 61 denotes an image ID, which corresponds to the image ID 51 in FIG. Reference numeral 62 denotes an image content word, which stores a word for describing each image represented by the image ID 61. An image content word is a keyword that expresses image features described in an image as words, and a character code (for example, unicod
Stored in e). A plurality of keywords may be stored for one image, and the image content word holding unit 219 stores the image I
It is composed of a list of image content words 62 using D61 as a key.

FIG. 7 is a diagram showing a table in which the data of the image content word holding unit shown in FIG. 6 is stored in a list of image IDs using the image content words as keys. In FIG. 7, all image IDs including each of the image content words 71 as a keyword
Is stored in the image ID 72. Note that FIG. 6 shows the classification based on the image ID, and FIG. 7 shows the classification based on the image content word. Since the contents are the same, it is not necessary to hold both.

FIG. 8 is a diagram showing an example of the data configuration of the concept discrimination dictionary 205. The concept discrimination dictionary 205 provides information on a search term that is a search request, and includes a headword 80 corresponding to an evaluative word, a search viewpoint 81 related to the headword, and a search word opposite to the headword for each search viewpoint. A disagreeable concept 82 having a meaning, an associative word weight 83 when searching for a headword,
It is a table in which a sentence pattern weight 84 at the time of searching for a headword is stored in correspondence.

FIG. 9 is a diagram showing an example of the data structure of the associative word dictionary 211. The associative word dictionary 211 relates to an associative ID 90 for uniquely numbering a set of associative words for the headword, an associative word 92 serving as a starting point of the associative word, an associative word 92 associated with the associative word 91, and the association of the associative word 92. Associative viewpoint 9
3. An association strength 94 indicating the strength of association between the headword 91 and the association word 92.

In this example, the association strength 94 is 0 to 10
The sign indicates the application state of the associative word.
That is, if the association strength is a positive number, the larger the association strength value, the stronger the association by the association (higher co-occurrence). Conversely, if the association strength is a negative number,
It indicates that the associative relationship does not hold as the associative strength increases. For example, in FIG. 9, in the association data of association ID = 126533, the association word “folk art” corresponding to the headword “simple” is a word that can be associated with a strength of “6”, but the association ID = 126536. The associative data "chandelier" is a word that cannot be associated with a strength of "9" because the associative strength is a negative number.

FIG. 10 is a view for explaining a data holding mode in the search result holding section 216. As mentioned above,
The search result holding unit 216 stores the image IDs searched by the image content word search process 210 using the associative word and the emotion pattern search process 214.

In FIG. 10, reference numeral 100 stores the searched image ID, and reference numeral 101 stores the number of associative words having a positive associative strength matched in the image content word search processing 217. The associative word ID list 102 stores up to 20 associative word IDs 90 (FIG. 9) in the associative word dictionary 211 of the associated associative words. If the associative word match count 101 is zero, the associative word ID list 102 is filled with null codes. Reference numeral 103 stores the association degree of the image ID 100 by the association word. If the associative word match count 101 is zero, zero is also stored in the associative match degree 103.

Reference numeral 104 stores the number of emotion patterns having the highest similarity searched in the emotion pattern search processing 223. A sentiment pattern ID list 105 stores up to 20 sentiment pattern IDs of the matched sentiment patterns. When the number of emotion pattern matches 104 is zero,
A null code is embedded in the sensitivity pattern ID list 105. Reference numeral 106 denotes a sentence pattern matching degree obtained by a sentiment pattern search for the image ID 100. When the number of emotion pattern matches 104 is zero, zero is stored in the emotion pattern matching degree 106. In 107, the integrated matching degree of the image ID 100 in response to the search request (by the search result integrating process 217) obtained by using the associative matching degree 103 and the sentiment pattern matching degree 106 as parameters is stored.

FIG. 11 shows the developed sensitivity pattern holding unit 2 shown in FIG.
13 is a diagram for explaining a data configuration example of No. 13. FIG. In FIG. 11, reference numeral 110 denotes an image word from which the sentiment pattern is developed, and the same image word as that stored in the search word holding unit 202 is stored. In this example, a character string “fresh” is stored, and the character string is NULL.
End with code. Reference numeral 111 denotes the number of kansei patterns obtained by developing the image word 110 with reference to the image word-kansei pattern correspondence holding unit 215. For example, the contents of the image word-sensitive pattern correspondence holding unit 215 are as shown in FIG.
In this case, the number of emotional patterns developed from the image word “refreshing” is seven. 112
Is an address indicating a storage destination area of data in which an image word “refreshing” is actually developed. In the example shown in FIG. 11, this storage destination address 112 is
Linking to the data indicated by 15.

In the data 115, "fresh"
, The expanded data actually developed, that is, the sets of the associative words and the sensibility patterns are stored by the number indicated by the number of sensibility patterns 111. In this example, as the data 115,
Seven sets of association words and sentiment patterns are stored. For example, 114 is an associative word of an image word “fresh”. In this example, a character string “forest” is stored, and the character string ends with a NULL code. Reference numeral 113 denotes a feeling pattern ID corresponding to the associative word “forest” between the image words “fresh” and “fresh”. In this example, 5 is stored. The same applies to other pairs of associative words and sensitivity patterns.

FIG. 12 is a diagram showing an example of the data structure of the image word-sensitive pattern correspondence holding unit 215 in FIG. In FIG. 12, reference numeral 120 denotes an image word serving as a development source of the sensibility pattern. In the figure, "Refreshing"
A character string such as "tropical" is stored, and the character string ends with a NULL code. Reference numeral 121 denotes an associative word developed from each of the 120 image words. In this example, an associative word such as "forest", "plateau", and "blue sky" is stored for "fresh", and the character string ends with a NULL code. When a character string is not stored in the portion 121, that is, when only a NULL code is used, this sensibility pattern can be applied to all the image words “refreshing”, and particularly indicates that no association word is specified. ing.

Reference numeral 122 denotes a feeling pattern ID corresponding to 120 image words and 121 associative words. In this example,
005, 006 are stored as sensitivity pattern IDs for the image word “fresh” and its associated word “forest”. In addition, a feeling pattern for “not refreshing” is also stored as an opposite concept to “fresh”. In this example, “not refreshing” does not have an associated word, and 001 and 010 are registered as sensibility pattern IDs.

FIG. 13 is a diagram showing a data structure of the affective pattern holding unit 220 in FIG. In FIG.
Reference numeral 131 denotes an image ID for identifying an image to be registered. As the image ID 131, the same number as the number stored in the image holding unit 218 is used, and uniquely defines an image in this system. 132 stores a sensitivity pattern ID. In the present embodiment, for each image, the image feature amount-kansei pattern correspondence holding unit 2
Since the degree of conformity to all the emotion patterns stored in the memory pattern 23 is calculated, all the emotion pattern IDs (1 to
m) is stored. 133 is a numerical value indicating the degree of conformity between each image and each emotion pattern. The degree of conformity is a number from 0 to 1, where 0 indicates that the image did not conform to the emotional pattern at all, and closer to 1 indicates higher suitability. For example, in the present embodiment, the image I
For the image of D001, the degree of conformity with Kansei pattern 1 is 0.10, and the degree of conformity with Kansei pattern 2 is 0.

FIG. 14 is a diagram showing an example of data obtained when an image feature is extracted from one image by the image feature extraction process. In FIG. 14, the image features are X1, X
2, X3,..., Xn, regions / blocks for extracting image feature amounts are represented by B1, B2,..., Bm, and feature amounts extracted from each region / block are represented by x11 to xmn. In other words, feature amounts relating to physical image features of X1 to Xn are obtained for each area / block.

FIG. 15 is a diagram showing an example in the case of extracting color image feature values. This is a representative color extracted for each region or block of the image. FIG.
According to 5, the physical image features are “representative color” and “feature amount”.
, And the representative colors extracted from the areas or blocks B1, B2,..., Bn are C1 (R1, G1, B1), C2 (R2,
G2, B2),..., Cn (Rn, Gn, Bn), and the respective characteristic amounts are c1 to cn.

FIG. 16 shows the image feature amount holding unit 2 in FIG.
22 is a diagram illustrating a storage example of No. 22. FIG. In FIG.
Is an image ID for identifying an image to be registered. The image ID 161 is the I ID stored in the image holding unit 218.
The same as the D number is used. Reference numeral 162 denotes a number indicating a block or area from which the image feature amount has been extracted. In this example, the number of each area or block is represented by B1, B2,.
It is represented by Reference numeral 163 denotes information (in this example, a representative color is used) representing an image feature extracted from each area or block (B1, B2,..., Bm) shown in 162. This example is an example in which a color image feature is extracted. For example, information C11 (R11, G11, B11),..., Cn1 (R
n1, Gn1, Bn1) are stored. Reference numeral 164 denotes a feature amount of an image feature extracted from each area or block. In this example, C11 (R11, G11, B11),.
.., Cn1 as features of n1 (Rn1, Gn1, Bn1)
Is stored.

FIG. 17 is a diagram showing an example of data storage in the image feature amount-sensitive pattern correspondence holding unit 223 in FIG. In FIG. 17, reference numeral 171 denotes a feeling pattern ID, which is a number for uniquely identifying a feeling pattern. 172 is image feature amount data associated with each emotion pattern ID. In the present embodiment, the sensibility pattern is represented by a chromatic feature amount, and the color component (R
A combination of values in a color space such as GB or HVC is stored. In this example, values in the RGB color space are registered as color components. RGB value is 0 to 255
Is an integer up to. The maximum number of colors associated with each pattern is m.

The sensibility pattern determination process 224 uses the image feature amount holding unit 222 and the image feature amount-kansei pattern correspondence holding unit 223 to perform each kansei operation for each image data registered in the image holding unit 218. The degree of matching with the pattern is obtained and registered in the affective pattern holding unit 220 (described later in step S87 of FIG. 22).

The processing in the image retrieval apparatus of the present embodiment having the above configuration will be described below.

FIG. 18 is a flowchart for explaining the operation of the image search processing according to this embodiment. In FIG. 18, step S1 is a processing module that executes the search request processing 201 in FIG. 2, and inputs a search request. The details of the processing will be described later with reference to FIG.

Step S2 is a weight judgment process 20 shown in FIG.
6, which refers to the search weight stored in the search weight storage unit 204 by the search request input process in step S1, and if the weight is specified, the value is stored in the determination weight storage unit 207. Store. If no weight is specified in the search weight holding unit 204,
Referring to the concept discrimination dictionary 205 (FIG. 8), the search term holding unit 2
The search term stored in the search term 02 is searched from the headword 80, the corresponding associative weight 83 and the affective pattern weight 84 are read out, and stored in the decision weight holding unit 207. If there is no word that matches the search word holding unit 202 in the headword 80, 5 (both are 5) are stored in the determination weight holding unit 207 as default values of the association weight and the sensitivity pattern weight.

In step S3, it is determined whether or not the associative weight is 0 with reference to the decision weight holding unit 207. If the associative weight is 0, the search by the associative word is unnecessary, so the process branches to step S5. If not, the process of step S4 is performed. Step S4 is a processing module that performs an associative word expansion process 208 and an image content word search process 210 using an associative word in FIG. 2, and details of the process will be described later with reference to FIG.

In step S5, the decision weight holding unit 207
To determine whether or not the sentiment pattern weight is 0. If the sentiment pattern weight is 0, the search by the sentiment pattern is unnecessary, and the process branches to step S7. Otherwise, the process of step S6 is performed. Step S6 is a processing module that performs the affective pattern development process 212 and the affective pattern search process 214 in FIG. 2, which will be described later with reference to FIG.

Step S7 is a processing module for performing the search result integration processing 217 in FIG. 2, and the details of the processing will be described later with reference to FIG.

In step S8, the image file corresponding to the image ID stored in the search result holding unit 216 as the search result obtained in step S7 is stored in the image holding unit 21.
Take out from 8 and display. The display of the search result in step S8 is a known process widely performed in the same type of search device.

FIG. 19 shows a search request input process 201 (FIG. 1).
8 is a flowchart illustrating details of step S1) of FIG. In step S21, a search word serving as a search request is input. Search term input is performed by storing the character code input from the KB 15 in the search term holding unit 202 on the RAM 13.

Next, in step S 22, the search terms stored in the search term holding unit 202 are used as a search request, and a search viewpoint related to the search request is extracted from the concept determination dictionary 205. All search viewpoints 81 for the headword 80 (FIG. 8) that matches the search word in the search word holding unit 202 are extracted. For example, when the search word is "cool", three search viewpoints "color", "taste", and "general atmosphere" can be obtained.

Next, in step S23, it is determined whether or not a search viewpoint has been found by the search viewpoint extraction in step S22. If a search viewpoint has been found, the process branches to step S24. If not, the process branches to step S26. .

If a search viewpoint is found in step S22, the search viewpoint is displayed together with the search word in step S24 as shown in FIG. Step S2
In step 5, a desired search viewpoint is selected from the displayed search viewpoint using the user interface described with reference to FIG. The search viewpoint selected and instructed in this manner is stored in the search viewpoint holding unit 203.

Step S26 relates to a search processing method for the search request, and inputs a search weight for designating which of the search based on the associative word and the search based on the sentiment pattern is to be emphasized. In the present embodiment, the weight is set using a user interface as shown in FIG. The user operates the slider bar shown in FIG. 4 to specify the ratio of emphasizing the association word and the ratio of emphasizing the sensitivity pattern (desired search weight) based on the length of the left and right bars of the slider (the position of the button 41). . When the weight is not specified, the use of the default value of the weight in FIG. 4 is specified.

In step S27, step S26
It is determined whether or not a weight has been specified. Then, when it is instructed to use the default value of the weight, the present process is terminated. On the other hand, if the search weight has been designated, step S2
Then, the process proceeds to step S8, where the search weight based on the associative word specified in step S26 and the search weight based on the affective pattern are stored in the search weight holding unit 204, and the process ends.

FIG. 20 is a flowchart showing details of the associative word expansion process 208 and the image content word search process 210 (step S4 in FIG. 18) using the associative words.

In FIG. 20, a step S41 is a search word stored in the search word holding unit 202 and the associative word dictionary 21
1 to obtain associative word data. That is, a search is made for a headword 91 (FIG. 9) in the associative word dictionary 211 that matches the search word, and registered associative word data is extracted. If a word that matches the search word exists in the headword 91, its association I
D are all stored in the expanded associative word holding unit 209.

Next, in step S42, the concept discrimination dictionary 205 is searched, and if there is a headword that matches the search word 202, the search viewpoint 81 (FIG. 8) corresponding to the headword is extracted. Then, the retrieved search viewpoint is compared with the search viewpoint stored in the search viewpoint holding unit 203 in step S25 described above, and if both match, the dissenting concept 82 corresponding to the headword is extracted. . If the search viewpoints do not match, the search for data in which the search word matches the headword is continued. When the above check is completed for all the extracted search viewpoints in this way, step S4
Proceed to 3.

In step S43, an associative word having a headword that matches the antisense concept found in step S42 is searched from the associative word dictionary 211. If there is a headword that matches the negative meaning, the associative ID is assigned a status code indicating that the word is a negative meaning, and stored in the expanded associated word holding unit 209.

Next, in step S44, an associative word is extracted from the associative ID stored in the expanded associative word holding unit 209, and an image content word that matches the associative word is searched from the image content word holding unit 219, and the result is retrieved. It is stored in the holding unit 216.

Here, first, the expanded associative word holding unit 209
Is extracted from the associative word dictionary 21
1, the corresponding associative data is extracted. Then, the association strength 94 of the extracted association data is extracted, and R
It is set in a work memory ASCF (not shown) on the AM 13. Here, if the associative ID extracted from the expanded associative word holding unit 209 is provided with a state code of a disagreeable concept, the sign of the associative strength is inverted to be a negative number.
However, if the association strength is already a negative number, the associative data is discarded and the next associative data is checked.

Next, the associative viewpoint corresponding to the associative ID is taken out, compared with the search viewpoint stored in the search viewpoint holding unit 203, and if they match, a predetermined value α is stored in the work memory VPF (not shown) secured in the RAM 13. And if they do not match, a value of α × 0.1 is set in the work memory VPF.

Next, an image content word matching the association word corresponding to the association ID is searched from the image content word holding unit 219, and if found, the image ID corresponding to the image content word is assigned to the image ID.
72 (FIG. 7) and set in the searched image ID 100 (FIG. 10) of the search result holding unit 216. Then, 1 is set to the associative word match count 101 and the associative word ID1
02 is set to the associative ID to be searched. Then, a value obtained by multiplying the value of the work memory ASCF by the value of the VPF to the predetermined value β of the score by associative word matching is stored in the associative matching degree 103 as the associative matching degree. Image I already
If D is stored, +1 is added to the value of the number 101 of association words, and a new association word ID is added to the association word ID 102;
The value of the stored associative match 103 is added to the obtained associative match to update the value as a new associative match.

FIG. 21 shows a sentiment pattern development process 212,
Kansei pattern search processing 214 and search result integration processing 21
7 is a flowchart showing details of a seventh example.

As described above, the search request input processing 201
The user inputs a search request for searching for an image. The search request includes one or more search words, a search range, and the like. The search term input in this embodiment is
It is an abstract image word that expresses the impression of an image such as a "fresh" feeling, a "warm" feeling, and the like. The input search instruction word is stored in the search word holding unit 202. Here, the description will be made assuming that the image word “refreshing” is stored.

Steps S61 and S62 are processes executed in the sentiment pattern development process 212. In step S61, the image word held in the search word holding unit 202 is developed into a sentiment pattern with reference to the image word / kansei pattern correspondence holding unit 215. In the present embodiment, the image word “refreshing” is stored in the search word holding unit 202.
However, the associative words “forest”, “plateau”, “blue sky”, etc. developed from “refreshing” are stored in the expanded associative word holding unit 209. Expand to For example, a sensitivity pattern ID for an image word “fresh” -associative word “forest”
005 and 006, and 007 as the affective pattern ID for the image word “fresh” -associative word “Kogen”.

Then, the process proceeds to a step S62, wherein a step S6 is executed.
The set of the sentiment pattern ID and the image word / association word developed in step 1 is stored in the developed sentiment pattern holding unit 213. The storage of data in the development sensitivity pattern holding unit 213 is as described above with reference to FIG.

Next, the process proceeds to a step S63. Step S6
3 and step S64 are the emotion pattern search processing 214
This is the process executed by First, in step S63, the emotion pattern ID stored in the developed emotion pattern holding unit 213 is referred to with reference to the emotion pattern holding unit 220.
All image IDs of images having a degree of conformity to 0 greater than 0 are acquired. This process is performed for all the emotion patterns held in the developed emotion pattern holding unit 213. In the affective pattern search process 214, an image ID having a degree of conformity to the affective pattern ID developed from each of the search term and the negative concept that is greater than 0 is acquired.

Next, in step S64, step S
The set of the sentiment pattern ID, the image ID, and the matching degree acquired in 63 are stored in the search result holding unit 216.

Next, the process proceeds to a step S65. Step S
The processing from 65 to S69 indicates the processing executed by the search result integration processing 217. That is, referring to the image content word search result by the associative word and the sentiment pattern search result held in the search result holding unit 216, based on the weight held in the decision weight holding unit 207, the two search results As one search result. If there is a sentiment pattern based on the opposite meaning to the search term in the sentiment pattern search result, the image is excluded from the integrated result. Alternatively, the degree of coincidence of the affective pattern of the image including the affective pattern of the opposite concept may be reduced during the integration processing.

In the integration processing, a method of taking a common element of both search results for each associative word (an AND of the search results) and a weight of each search are used to determine an integrated fitness. Search results. In the present embodiment, the integration processing will be described by taking a method of obtaining the integration suitability as an example.

The search result storage unit 216 stores
For example, the associative fitness of an image that matches the associative word “forest” is A, and the sensitivity pattern ID 005 corresponding to the associative word “forest” is
Is the sensitivity adaptation degree of the image conforming to
7 are weighted by w1, w2 (w1
When + w2 = 1), the integrated fitness can be obtained as follows: integrated fitness = w1 × A + w2 × B or integrated fitness = (w1 × A ^ 2 + w2 × B ^ 2) ^ 1/2. Here, X ^ Y represents X to the power of Y.
The integrated relevance is obtained for all kansei patterns of all associative words. If one image ID has a degree of fitness greater than 0 for a plurality of emotional pattern IDs, a plurality of integrated degrees of fitness are obtained for one image. In this case, the largest degree of fitness is used as a search result. Adopt (Step S6
5).

This processing is performed for all the images in which one of the search results is larger than 0, and the images having the integrated suitability higher than the threshold value are made the integrated search results (step S6).
6, S67, S68).

In step S 69, the set of the image ID and its matching degree is stored in the search result holding unit 216, thereby ending the search processing.

FIG. 22 is a flowchart showing an example of pre-processing of a search performed at the time of image registration. This process
It is controlled according to a processing program stored in the disk 14.

First, in step S81, an image to be registered is designated by the user. The image to be registered is DISK14
And an image stored in an external storage device, an image input device, or an image database server connected to the image processing device. In the present embodiment, images to be searched are stored in advance, and an image is selected from the images.

Next, the process proceeds to a step S82. Step S8
2, the image I corresponding to the specified image file name
D and various kinds of image information necessary for registration are obtained and given to the image feature extraction processing 221. The image ID is
In order to manage the image, the data in the image holding unit 218 is searched for and acquired from the image file name using the ID stored in association with the image file name. The various information of the image is, for example, when the file format of the image is a bitmap format, pixel values of the width and height of the image,
The information includes the number of bits per pixel, the size of an image (in units of bytes), the address of an area where a bitmap image is actually stored, and the like. Since these pieces of information are stored in the header of the image file, they can be obtained by referring to the header. The image file format is not bitmap, but JFIF or Flash Pix
In the case of (1), necessary information can be similarly obtained from the header of the file. Alternatively, these pieces of information may be stored in the image holding unit 218, and the information may be acquired by referring to the information at the time of registration.

Next, the process proceeds to a step S83. Step S
Reference numeral 83 denotes a process performed in the image feature amount extraction process 221 which is a step of analyzing image information corresponding to a designated image ID and extracting a physical image feature amount. In FIG. 15 described above,
In the example of the image feature amount in the present embodiment, a representative color is extracted for each area or block of the image. The representative color is obtained by analyzing an actual bit image of an image one pixel at a time using various types of information of a given image, and calculating color components (RGB, HVC, etc.) used in each area or block.
For example, a method of obtaining the average value of the values in a color space (eg, color space) and the color component having the highest frequency of appearance and using the color component as a representative color is used.

Next, the process proceeds to a step S84. In step S84, the image feature amounts c1 to cn extracted in step S83 are associated with the image ID of the image and stored in the image feature amount holding unit 222. The data storage form in this case is as described above with reference to FIG.

Then, the process proceeds to step S85, where all the feeling pattern IDs stored in the image feature quantity-kansei pattern correspondence holding section 223 and the kansei patterns thereof are referred to by referring to the image feature quantity-kansei pattern correspondence holding section 223. The image feature amount associated with is acquired. In the present embodiment, FIG.
As described above, the color feature amount is associated with each sensibility pattern.

Then, the process proceeds to a step S86, wherein a step S86 is executed.
The matching degree between the sensitivity pattern acquired in step 85 and the image feature amount for this image is obtained. This is a process performed in the affective pattern determination process 224. For each of the affective patterns acquired in step S85, the color feature amount associated with the affective pattern is compared with the image feature amount extracted in step S83. And determine its fitness. The matching degree is obtained for all the emotion patterns stored in the image feature amount-sensibility pattern correspondence holding unit 223.
In order to calculate the degree of conformity, a technique such as a vector operation using a cosine measure or a statistical process is used.

Next, the process proceeds to a step S87. Step S
At 87, the suitability of all the sensibility patterns and images obtained in step S86 is stored in the sensibility pattern holding unit 220 in association with the image ID of this image. An example of storing an image in the affective pattern holding unit 220 is as described above with reference to FIG.

The above processing is performed for all the images to be registered.

As described above, according to the present embodiment, a search using the feature data of the multimedia information itself based on the associative word associated with the search word,
A search is performed using the content word added to the multimedia information, and a final search result can be obtained from the results of both searches. For this reason, it is possible to accurately extract desired desired image information.

For example, conventionally, if the word “sea” is obtained as a related word to the search word “fresh”,
Although there is a possibility that "rough sea" may be searched, in the present application, it is excluded when integrating with a search result based on a sensitivity pattern obtained from a combination of "fresh"-"sea".

In the above embodiment, the description has been made using the image information as the storage information to be searched. For multimedia information (for example, audio information) other than the image information, the information feature is extracted and the feature is extracted. By associating with the sensibility pattern, it can be handled similarly.

In the above description, the image holding unit 218, the image content word holding unit 219, and the sensibility pattern holding unit 220 to be searched are the DISK 1 constituting a single device.
4, it is also possible to distribute these components to different devices and perform a search process on the network via the NIC 19.

The present invention can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but can be applied to a single device (for example, a copier, a facsimile). Device).

An object of the present invention is to provide a system or an apparatus with a storage medium storing a program code of software for realizing the functions of the above-described embodiments, and to provide a computer (or CPU) of the system or the apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

Examples of a storage medium for supplying the program code include a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, and CD.
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

The functions of the above-described embodiments are implemented when the computer executes the readout program codes, and the OS (Operating System) running on the computer is executed based on the instructions of the program codes. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0106]

As described above, according to the present invention,
Based on the search term, a search using the feature data of the information to be searched and a search using the content word given to the information to be searched are performed to obtain a search result. As a result, it is possible to accurately extract desired information.

[0107]

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example of an information processing apparatus for configuring an image search device according to an embodiment;

FIG. 2 is a block diagram schematically illustrating a processing configuration in the image search device according to the embodiment;

FIG. 3 is a diagram illustrating an example in which a related search viewpoint is displayed for a search request word input in a search request input process 201;

FIG. 4 is a diagram showing a display example of a weight designation operation panel for designating a search weight based on an associative word and a search weight based on a feeling pattern in a search request input process 201;

FIG. 5 is a diagram illustrating a data configuration of an image holding unit 218 in which an image ID is associated with an image file storage path.

FIG. 6 is a diagram illustrating a data configuration example of an image content word holding unit 219 in which an image ID is associated with an image content word.

7 is a diagram showing a table in which data of an image content word holding unit shown in FIG. 6 is stored in a list of image IDs using image content words as keys.

FIG. 8 is a diagram showing a data configuration example of a concept determination dictionary 205.

FIG. 9 is a diagram showing a data configuration example of an associative word dictionary 211.

FIG. 10 is a diagram illustrating a data holding mode in a search result holding unit.

11 is a diagram illustrating an example of a data configuration of a development sensitivity pattern holding unit 213 in FIG.

12 is a diagram showing a data configuration example of an image word-kansei pattern correspondence holding unit 215 in FIG.

FIG. 13 is a diagram illustrating a data configuration of a feeling pattern holding unit 220 in FIG. 2;

FIG. 14 is a diagram illustrating an example of data obtained when an image feature amount is extracted from one image by an image feature amount extraction process.

FIG. 15 is an example of an image feature amount according to the present embodiment, in which representative colors are extracted for each area or block of an image.

FIG. 16 is a diagram illustrating a storage example of an image feature amount holding unit 222 in FIG. 2;

17 is a diagram illustrating an example of data storage in an image feature amount-kansei pattern correspondence holding unit 223 in FIG. 2;

FIG. 18 is a flowchart illustrating the operation of the present invention.

FIG. 19 is a flowchart showing details of a search request input process 201 (step S1 in FIG. 18).

20 is a flowchart showing details of an associative word expansion process 208 and an image content word search process 210 using an associative word (step S4 in FIG. 18).

FIG. 21 is a flowchart illustrating details of a sensitivity pattern development process 212, a sensitivity pattern search process 214, and a search result integration process 217.

FIG. 22 is a flowchart illustrating an example of pre-processing of search performed at the time of image registration.

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B050 CA07 EA07 EA09 FA02 FA13 GA08 5B075 NK07 NK08 NK31 NK46 PP03 PP13 PP28 QM07 QS15 UU40

Claims (15)

[Claims] 1. An information retrieval apparatus for retrieving information based on an input search term, wherein a first search is performed in which a search keyword is determined based on the search term, and information is retrieved based on the search keyword. Means for determining a feature amount corresponding to the search word, and searching for information based on the feature amount; integration for integrating search results obtained by the first and second search means And an information retrieval device. 2. An associative word dictionary in which a headword and an associative word related thereto are registered in association with each other, wherein the first search means searches the associative word dictionary with the search word and the search word. 2. The information search device according to claim 1, wherein the associative word obtained by the search is used as a search keyword. 3. The associative word dictionary has registered therein an associative strength indicating the strength of an associative relationship between each associative word and a headword, and wherein the first search means includes an associative strength of each associative word. On the basis of the,
Obtaining a search relevance of each searched information, wherein the integration means refers to the search relevance obtained by the first search means and performs integration with the search result obtained by the second search means. The information search device according to claim 2, wherein 4. An acquisition means for acquiring a dissociative associative word representing a disagreement concept with respect to the search word, wherein the first search means retrieves an image by the dissociative associative word acquired by the acquiring means. 4. The information retrieval apparatus according to claim 3, wherein, in the case of retrieving, the retrieval relevance of the retrieved image is reduced. 5. A representative pattern dictionary for associating a representative pattern with each pair of the search word and the associative word, wherein the second search means searches the representative pattern dictionary by a combination of the search word and the associative word. 3. The information search apparatus according to claim 2, wherein a corresponding representative pattern is obtained by performing the search, and information is searched based on the feature amount of the obtained representative pattern. 6. The information processing apparatus according to claim 1, wherein the information is image information, and the feature amount is a physical image feature amount obtained by analyzing the image information. Information retrieval device. 7. The information retrieval apparatus according to claim 6, wherein the feature amount includes at least one of color arrangement information, composition information, and shape information of the image. 8. An information search method for searching for information based on an input search word, wherein a first search keyword is determined based on the search word, and information is searched based on the search keyword. Integrating a search result obtained by the first and second search steps; determining a feature amount of a pattern corresponding to the search word, and searching for information based on the feature amount; An information retrieval method, comprising: 9. An associative word dictionary in which a headword and an associative word related thereto are registered in association with each other, wherein the first search step searches the associative word dictionary with the search word and the search word. 9. The information search method according to claim 8, wherein the associative word obtained by the search is used as a search keyword. 10. The associative word dictionary has registered therein an associative strength indicating the strength of an associative relationship between each associative word and a headword. The first search step includes associative strength of each associative word. On the basis of the,
Obtaining a search relevance of each searched information; and integrating the search information with the search result obtained in the second search step by referring to the search relevance obtained in the first search step. The information search method according to claim 9, wherein 11. The method according to claim 11, further comprising an acquisition step of acquiring an anonymous association word representing an anonymous concept with respect to the search word, wherein the first searching step includes searching for an image using the antonymous association word acquired in the acquiring step. 11. The method according to claim 10, wherein when the search is performed, the search relevance of the searched image is reduced.
Information search method described in. 12. A representative pattern dictionary for associating a representative pattern with each pair of the search word and the associative word, wherein the second search step searches the representative pattern dictionary by a combination of the search word and the associative word. 10. The information search method according to claim 9, wherein a corresponding representative pattern is obtained by performing the search, and information is searched based on the feature amount of the obtained representative pattern. 13. The information processing apparatus according to claim 8, wherein the information is image information, and the feature amount is a physical image feature amount obtained by analyzing the image information. Information retrieval method. 14. The information retrieval method according to claim 13, wherein the feature amount includes at least one of color arrangement information, composition information, and shape information of the image. 15. A storage medium for storing a control program for causing a computer to search for information based on an input search word, wherein the control program determines a search keyword based on the search word, A code for a first search step for searching for information based on the search keyword, and a code for a second search step for determining a feature amount of a pattern corresponding to the search word and searching for information based on the feature amount And a code for an integration step of integrating search results obtained in the first and second search steps.