JP2004139246A - Image search system, image search method, program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make classified subsidiary information easily understood visually to enhance operability of image search, in the image search wherein the subsidiary information imparted to an image data is hierarchized, and wherein a search condition is set based on the hierarchized subsidiary information to conduct the search. <P>SOLUTION: A CPU 12 of an image search system 11 classifies the image data based on the registration-objective image data and keywords imparted to the image data, sets the keyword as a search condition sentence to conduct keyword search, sets as a representative image the most specious image data out of an image data group provided by the keyword search, displays on a CRT 15 the keyword conformed with a classification name of the image data, and the representative image corresponding to the keyword, searches the image based on the search condition set according to the display, and display-controls the image data provided by the image search on the CRT 15. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image search device, an image search method, a program, and a storage medium suitable for displaying a plurality of image data shot by an electronic still camera such as an electronic camera or a video camera and searching for the image data.
[0002]
[Prior art]
In recent years, electronic still cameras such as electronic cameras and video cameras that store image data such as photographs as digital signals in a memory card or the like have been provided. By storing image data such as photographs taken by these electronic still cameras in a portable memory such as an optical memory, physical space can be reduced compared to photographs taken by a camera using a conventional silver halide film. It can be held in large quantities without need.
[0003]
Conventionally, as means for searching for desired image data from a large amount of stored image data as described above, a keyword is assigned to each stored image data, and a search condition and image specified by an operator are assigned. There is a method of searching for desired image data by comparing a keyword assigned to data with the keyword. Hereinafter, the search method is referred to as a keyword search. In this search method, an operator inputs search conditions into an input field prepared in advance on a display device. In this case, it is also possible to specify a plurality of keywords by a description in a natural language sentence or a logical operator in the search condition. As a means for specifying a keyword, there is a method in which when an operator inputs a first character, the same keyword as the first character is searched from the image database, and the operator selects from a search result list.
[0004]
As a method of searching for image data using a keyword, there is a method of searching for image data by hierarchizing keywords. That is, in the method of searching for image data by hierarchizing keywords, a directory structure is semantically hierarchized in advance, and the hierarchized name is used as a keyword. These hierarchies are presented and keywords are assigned, or keywords are classified. There is a method of searching for image data by causing the image data to be searched (for example, see Patent Document 1).
[0005]
[Patent Document 1]
JP-A-7-129621
[0006]
[Problems to be solved by the invention]
However, in the image data search method according to the related art, particularly when searching for image data using a keyword, it is necessary to find a keyword suitable for desired image data and specify the keyword in a search condition sentence. For example, in a keyword search, if the keyword of the stored image data is recognized in advance, it is possible to easily search for the desired image data, but the keyword of the stored image data is not recognized. Then, the operator must perform image data search by designating an appropriate keyword through trial and error.
[0007]
In addition, a method of searching for image data by providing a hierarchical structure having a hierarchical structure in advance and using the hierarchical name as a keyword and presenting these hierarchies or assigning a keyword or classifying the keyword is referred to as a classified keyword. If you only display the keyword, you may not know the content of the keyword. In particular, for a keyword or the like assigned to image data handled in the content business, a person who assigns a keyword is different from a person who searches for image data, and such a situation is likely to occur.
[0008]
The present invention has been made in view of the above points, and has been classified in an image search in which additional information attached to image data is hierarchized in advance and a search condition is set and searched based on the hierarchized additional information. It is an object of the present invention to provide an image search device, an image search method, a program, and a storage medium that can improve operability of image search by making supplementary information visually easy to understand.
[0009]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides an image search device for searching image data, comprising: image classification means for classifying image data based on image data to be registered and incidental information added to the image data; A representative image setting means for setting a representative image for, a supplementary information display means for displaying the supplementary information and the representative image based on the classification of the image data, a search condition setting means for setting a search condition according to the display, It is characterized by having image search means for performing an image search based on search conditions, and search result display means for displaying image data acquired by the image search.
[0010]
According to the present invention, in an image search method for searching image data, an image classification step of classifying image data based on image data to be registered and additional information to be added to the image data, and setting a representative image for the additional information A representative image setting step, an additional information display step of displaying the additional information and the representative image based on the classification of the image data, a search condition setting step of setting a search condition according to the display, and an image based on the search condition. An image search step for performing a search and a search result display step for displaying the image data obtained by the image search are provided.
[0011]
In addition, the present invention provides a function of classifying image data based on image data to be registered and additional information to be added to the image data, a function of setting a representative image for the additional information, and the additional data based on the classification of the image data. A function of displaying information and the representative image, a function of setting a search condition according to the display, a function of performing an image search based on the search condition, and a function of displaying image data acquired by the image search, to a computer It is characterized by being a program for realizing.
[0012]
According to the present invention, there is provided a computer-readable storage medium storing a program for executing an image search method for searching for image data, wherein the image search method includes image data to be registered and additional information to be added to the image data. Classifying the image data according to the above, setting a representative image for the additional information, displaying the additional information and the representative image based on the classification of the image data, and setting a search condition according to the display A step of performing an image search based on the search condition; and a step of displaying image data obtained by the image search.
[0013]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0014]
[First Embodiment]
FIG. 1 is a block diagram showing a configuration example of the image search device according to the first embodiment of the present invention. The image search device 11 has a configuration in which a CPU 12, a ROM 13, a RAM 14, and a peripheral controller 16 are provided inside the device main body, and a display device (CRT) 15, an external storage device 17, and an input device 20 are provided outside the device main body. I have. In the figure, reference numeral 18 denotes a digital camera, and 19 denotes a digital video camera.
[0015]
The CPU 12 is a central processing unit that is the center of control of the image search device 11, and executes processing shown in each flowchart described below based on a program stored in the ROM 13. The ROM 13 is a read-only memory for storing a program for performing an image search, a data table, and the like. The RAM 14 is a random access memory that provides a work area and the like for executing a program stored in the ROM 13. The display device 15 is for displaying the processing status of the image search device 11, and includes, for example, a CRT display (hereinafter, CRT). The display device 15 is not limited to a CRT and may be a liquid crystal.
[0016]
The peripheral controller unit 16 is for connecting to an external storage device 17, an input device 20, a digital camera 18, an external input / output device such as a digital video camera 19 and the like. The external storage device 17 is a storage device for storing image data. The digital camera 18 is an imaging device that captures a subject and generates image data. The digital video camera 19 is an imaging device that photographs a subject and generates image data. Image data captured and acquired by the operator with the digital camera 18 or the digital video camera 19 is held in the external storage device 17 or the ROM 13. The input device 20 includes a mouse, a keyboard, and the like, and is used for various instructions and various data inputs.
[0017]
FIG. 2 is a diagram illustrating a configuration example of an image information table in the image search device. This image information table is held in the ROM 13 or the external storage device 17 of the image search device. The image information table includes an image data file name 22 indicating a file name of image data captured and acquired by the operator using the digital camera 18 or the digital video camera 19, and an image ID 21 for identifying the image data. .
[0018]
In the example of FIG. 2, for example, an image ID “001” is assigned to the image data file “000 — 0001.jpg”. In this case, an arbitrary numerical value (serial number) is used for the image ID. However, if the image data file name is unique, the image data file name may be used as the image ID.
[0019]
FIG. 3 is a diagram illustrating a configuration example of a keyword information table in the image search device. This keyword information table is held in the ROM 13 or the external storage device 17 of the image search device. The keyword information table includes an image ID 31 for identifying image data, a keyword 32 assigned to the image data, and a keyword importance 33 indicating the importance of the keyword.
[0020]
In the present embodiment, the importance of the keyword is “3” in the case of the subject word of the image, and “2” if the user is satisfied with the search and display using the word, but is not important. Is a valid keyword, “1” is assigned. In the present embodiment, keywords are assigned importance in such a criterion. However, the present invention is not limited to the above example as long as they indicate the importance of keywords attached to images. Explaining with the example in FIG. 3, keywords “flower”, “red”, and “park” are assigned to the image ID “001”, and “(flower =) 3” and “( Red =) 1 ”and“ (park =) 1 ”are assigned importance.
[0021]
FIG. 17 is a diagram illustrating a configuration example of a semantic classification information table in the image search device. This semantic classification information table is held in the ROM 13 or the external storage device 17 of the image search device. The semantic classification information table includes an image ID 171 for identifying image data, a keyword 172 assigned to the image data, and a classification 173 of the keyword. In the present embodiment, the semantic classification information table and the keyword information table are configured separately, but the semantic classification information and the keyword information may be configured on the same table.
[0022]
FIG. 4 is a diagram showing a configuration example of a thesaurus (Thesaurus) dictionary in the image search device. This thesaurus dictionary is stored in the ROM 13 or the external storage device 17 of the image search device. The thesaurus dictionary is composed of a headword 41 indicating a word notation, a semantic classification 42 indicating a semantic classification thereof, and a part of speech 43.
[0023]
Explaining with the example in FIG. 4, the headword “park” indicates that the part of speech is “noun” and the semantic classification is “specific object: product: abstract object: place”. Also, in this case, it has two meanings, “specific object (first layer), manufactured product (second layer)” and “abstract object (first layer), place (second layer)”. The headword “red” has the part of speech “adjective” and no semantic classification. In the present embodiment, the part of speech is also a constituent element in the thesaurus dictionary, but the part of speech may be configured as a word dictionary separately from the thesaurus dictionary.
[0024]
FIG. 5 is a diagram showing a configuration example of a keyword classification information table in the image search device. This keyword classification information table is held in the ROM 13 or the external storage device 17 of the image search device. The keyword classification information table is a value obtained by adding the classification name 51 indicating the classification content, the keyword 52 applicable to the classification, the image ID 53 of the representative image in the keyword, and the importance of the keyword in the stored image keywords. And a certain level 54. This level 54 increases when the importance is high and the keyword is assigned to many images.
[0025]
In the example of FIG. 5, the keyword “fish” is classified as “animal”, the representative image is “039”, and the level is “12”. As the representative image, one of the images in which the keyword “fish” is present is selected from the stored images. In addition, the keyword “fish” (level 12) is stored with a higher value than the keyword “elephant” (level 10) and the keyword “dog” (level 2).
[0026]
FIG. 6 is a diagram showing a display example of a search screen in the image search device. This search screen is displayed on the CRT 15 under the control of the CPU 12 of the image search device. The display contents of the search screen include a classification name 61 indicating the classification contents, a keyword number 62 indicating the number of keywords corresponding to the classification, a keyword control 63 indicating the keywords corresponding to the classification, and an example of image data for the keywords. It is composed of a representative image control 64, a search condition control 65 for inputting a keyword of the search condition, and a search execution control 66 for instructing execution of the search.
[0027]
In this embodiment, the keyword control 63 is configured by a check box control. However, the configuration of the keyword control 63 is not limited to this example, and may be any control such as a button control that can determine whether a keyword has been designated. Alternatively, the keyword may be automatically input by selecting the keyword with the input device 20 such as a mouse or a keyboard and dragging and dropping the keyword to the search condition control 65. The representative image control 64 also includes a control that can determine whether the corresponding representative image has been selected. In the present embodiment, image data is displayed on the button control.
[0028]
Explaining with the example in FIG. 6, the image data is roughly classified into classification names 61 “animal”, “human”, “plant”, “place”, “manufactured product”, and “abstract object”. The number of keywords 62 that apply to each category is “50” for “animal”, “100” for “human”, “40” for “plant”, “99” for “place”, and “99” for “manufactured product”. 12 ”and“ 50 ”of“ abstract objects ”. The keywords 63 assigned to the images corresponding to the classification name 61 “animal” include “fish”, “elephant”, and “dog”. Next to each keyword control 63, a representative image control 64 indicating a representative image is displayed.
[0029]
FIG. 12 is a diagram illustrating a configuration example of a different notation synonym dictionary in the image search device. This synonym dictionary is stored in the ROM 13 or the external storage device 17 of the image search device. The heterologous synonym dictionary is composed of headwords 121 and synonyms or synonyms 122 that apply to the headwords. In the example illustrated in FIG. 12, for the headword "flower", different notations or synonyms such as "flower", "flower", and "flower" exist.
[0030]
FIG. 13 is a diagram illustrating a configuration example of a lower concept word dictionary in the image search device. This lower concept word dictionary is stored in the ROM 13 or the external storage device 17 of the image search device. The lower concept word dictionary includes a headword 131, a word 132 indicating a lower concept applicable to the headword, and a level 133 indicating the hierarchy. Explaining with the example in FIG. 13, for the headword “car”, words of lower concepts such as “Porsche”, “motorcycle”, and “wheel” exist, and each level has a level of “Porsche (0.8)”. "Motorcycle (0.64)" and "wheels (0.55)".
[0031]
Next, an image search method in the image search apparatus according to the first embodiment will be described in detail with reference to FIGS.
[0032]
<Operation example when keyword registration processing is performed>
FIGS. 7 and 8 are flowcharts showing an operation example when a keyword registration process is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0033]
First, in step S11, the CPU 12 of the image search device determines whether the registration of the keyword has been completed. If it is determined in step S11 that the registration of the keyword has not been completed, in step S12, the CPU 12 stores the image data in the external storage device 17 or the ROM 13 as image data to be registered based on a predetermined operation by the operator from the input device 20. The image data already stored or the image data captured by the digital camera 18 or the digital video 19 is acquired via the peripheral controller 16. Next, in step S13, the CPU 12 acquires a keyword to be added to the image data acquired in step S12 by inputting a keyword from the input device 20 by the operator.
[0034]
Next, in step S14, the CPU 12 acquires the importance of the keyword acquired in step S13 by inputting the importance from the input device 20 by the operator. In the present embodiment, the importance is acquired by the input from the input device 20 by the operator, but the importance may be automatically given according to the input order of the keywords. Then, in steps S15 and S16, the CPU 12 obtains the semantic classification and part of speech for the keyword obtained in step S13 using the thesaurus dictionary of FIG.
[0035]
Next, in step S17, the CPU 12 determines a classification name. This will be described in detail with reference to FIGS. Further, in the present embodiment, the classification items are classified as “human”, “plant”, “animal”, “place”, “manufactured product”, and “abstract object”, but are not limited to this classification. Absent. Then, in step S18, the CPU 12 repeats the processing from step S13 to step S18 until the keyword assigned to the image data acquired in step S12 ends. The image data acquired in step S12, the keyword acquired in step S13, the importance acquired in step S14, and the classification name acquired in step S17 are stored in the RAM 14.
[0036]
If it is determined in step S11 that the image to which the keyword is to be added is completed, in step S19, the CPU 12 registers each data held in the RAM 14 in the database. This database is obtained by registering the table of the present embodiment as a database, and is stored in the external storage device 17 or the ROM 13. In addition, each of the above tables is configured so that access can be speeded up by creating an index.
[0037]
In step S19, first, the CPU 12 registers the image data acquired in step S12 in the image data file name of the image information table shown in FIG. 2, and assigns an image ID. Next, the CPU 12 stores the keyword acquired in step S13 and the importance acquired in step S14 in the keyword information table shown in FIG. The classification name acquired in step S17 is registered in the semantic classification information table shown in FIG. Next, in step S20, the CPU 12 acquires the registered keyword. At this time, duplication is removed for keywords registered in advance and having the same classification name and keyword. Next, in step S21, the CPU 12 determines whether a keyword is registered in the keyword classification information table shown in FIG.
[0038]
If it is determined in step S21 that the keyword has not been registered, the process proceeds to step S24. If it is determined in step S21 that a keyword has been registered, in step S22, the CPU 12 sets the registered keyword as a search condition sentence, performs a keyword search, and acquires a representative image. This will be described in detail with reference to FIG. Next, in step S23, the CPU 12 sets the most plausible image data in the image data group obtained by keyword search as a representative image. The acquired representative image is registered as the representative image 53 in the keyword classification information table shown in FIG. At this time, a classification name and a keyword are also registered.
[0039]
In step S24, the CPU 12 repeats the processing from step S21 to step S24 until the registered keyword ends. Next, in step S25, the CPU 12 performs the semantic classification order processing only for the classification in which the keywords have increased in the classification. This will be described in detail with reference to FIG. The ranking of the keywords acquired in this process is reflected on the level in the keyword classification information table shown in FIG.
[0040]
<Operation example when performing classification determination process>
FIG. 15 and FIG. 16 are flowcharts illustrating an operation example when the classification determination process is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0041]
First, in step S71, the CPU 12 determines whether or not the semantic classification acquired in the semantic classification acquisition process (step S15) in FIG. 7 exists. If it is determined in step S71 that the semantic classification does not exist, in step S72, the CPU 12 determines whether or not the part of speech is to be subjected to the classification setting based on the part of speech acquired in the part of speech acquisition processing (step S16) in FIG. . Classification names are not acquired for words whose parts of speech have no meaning, such as particles, auxiliary verbs, conjunctions, and pronouns. If it is determined in step S72 that the part of speech is to be classified, the CPU 12 sets the classification name to "abstract" in step S73. Here, the classification name is set as "abstract object", but the classification name is not limited and may be any type as long as it can be distinguished from other classifications.
[0042]
If it is determined in step S71 that a semantic class exists, in step S74, the CPU 12 determines whether there are two or more first-level semantic classes in the semantic class acquired in step S15 in FIG. Judge. If there is no semantic classification to be the first hierarchy in step S74, in step S80, the CPU 12 sets the acquired semantic classification to the classification name. If there is only one semantic classification in the first layer in step S74, in step S80, the CPU 12 sets the one semantic classification as the classification name.
[0043]
If there are two or more semantic classifications in the first hierarchy in step S74, in step S75, the CPU 12 obtains a judgment classification name. The judgment classification name indicates the classification of the semantic classification for each first hierarchy in the semantic classification acquired in step S15 of FIG. For example, the semantic classification for the keyword “mouse” is “concrete (first level), animal (second level), specific substance (first level), product (second level)”, and the judgment classification name is , "Animals" and "products".
[0044]
Next, in step S76, the CPU 12 performs a similar content search using the classification name acquired in step S75. The details of the similar content search are performed in the same manner as in the conventional method, and a detailed description thereof will be omitted. Next, in step S77, the CPU 12 determines whether there is an image in which the keyword is set as a result of the similar content search in step S76. If an image exists, in step S78, the CPU 12 sets a classification name as a judgment classification name. If no image exists, the CPU 12 does not set a classification name. In step S79, the processes in steps S75 to S79 are repeated until the judgment classification name existing in the semantic classification ends.
[0045]
<Operation example when keyword search processing is performed>
FIG. 11 is a flowchart illustrating an operation example when a keyword search process is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0046]
First, in step S51, the CPU 12 acquires a search condition sentence (keyword) for performing an image search. Next, in step S52, the CPU 12 divides the search condition sentence into words by natural language processing. For this natural language processing, the existing natural language processing is used, and the description is omitted. As a result of referring to the part of speech in the above-mentioned divided words, words that do not show concrete or abstract objects such as particles and auxiliary verbs are deleted from the divided words. Next, in step S53, the CPU 12 obtains a different notation and a synonym for each word divided in the above step S52 using the different notation synonym dictionary of FIG. Then, in step S54, the CPU 12 acquires the lower concept word and its level from the lower concept word dictionary of FIG.
[0047]
Next, in step S55, the CPU 12 repeats creation of a search key of a synonym of a different notation and creation of a lower-level concept search key until the divided word ends. Next, in step S56, the word divided in step S52, the synonymous word in different notation acquired in step S53, and the lower concept word acquired in step S54 are used as search keys, and the keyword information shown in FIG. By referring to the table, the keyword acquired in step S51 and the search key are compared, and if they are the same, the image data is acquired.
[0048]
Next, in step S57, the CPU 12 calculates a score for the acquired image data. This score calculation is performed by setting the keyword to level 3, the synonym word expanded from the keyword to level 2, and the lower concept word expanded from the keyword to level 1 + the level acquired from the lower concept dictionary. Then, the CPU 12 calculates a score value based on the importance of the keyword assigned to the image data and the value of the level. The method of obtaining the score value here is not limited to the above example. Next, in step S58, the CPU 12 repeats the processing in step S57 until the score calculation of the image data acquired in step S56 is completed. When the score calculation is completed, in step S59, the CPU 12 arranges the image data in descending order according to the score value. Thus, a keyword search is performed under the specified search conditions.
[0049]
<Operation example when semantic classification order processing is performed>
FIG. 14 is a flowchart showing an operation example when the semantic classification order processing of FIGS. 7 and 8 is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0050]
First, in step S61, the CPU 12 obtains a classification name whose order is to be changed. Next, in step S62, the CPU 12 acquires a keyword existing in the classification name from the keyword classification information table in FIG. Next, in step S63, the CPU 12 refers to the keyword information table of FIG. 3 and adds all the degrees of importance of the keywords acquired in step S62 to obtain a total. Next, in step S64, the CPU 12 obtains the total importance until the keywords in the classification are completed. The obtained total value is stored in the level 54 of the keyword classification information table in FIG.
[0051]
Through these processes, a keyword is assigned to the image, and preprocessing for displaying the search screen shown in FIG.
[0052]
<Operation example when image search processing is performed>
9 and 10 are flowcharts showing an operation example when the image search processing is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0053]
First, in step S31, the CPU 12 acquires the classification name, the keyword, and the representative image from the keyword classification information table in FIG. Further, the CPU 12 also obtains the number of keywords corresponding to each classification, and stores the number in the RAM 14. Next, in step S32, the CPU 12 removes the duplication of the category name acquired in step S31 and displays the same on the category name 61 which is a display area of the search screen shown in FIG. Next, in step S33, the CPU 12 acquires the number of keywords for the classification name displayed in step S32, and displays it in the number of keywords 62 which is a display area of the search screen shown in FIG. Next, in steps S34 and S35, the keywords and their representative images are displayed on the keyword control 63 and the representative image control 64 of the search screen shown in FIG. 6 in the order (descending order) obtained in step S31.
[0054]
Next, in step S36, the CPU 12 repeats the processing in steps S34 and S35 until the keyword in the classification in step S32 ends. In step S37, the CPU 12 repeats the processing from step S32 to step S36 until the classification to be displayed ends. With the processing so far, the search screen shown in FIG. 6 is displayed on the CRT 15. The display processing of other controls on the search screen is not particularly described, but is also displayed on the search screen.
[0055]
Next, in step S38, an input wait state is entered until a search condition and a search execution are input by the operator. The following processing is performed when an input is made from the operator. In step S39, the CPU 12 determines whether a keyword has been additionally specified. The determination as to whether or not this keyword has been additionally specified is made by determining whether or not addition of a search condition has been specified in the keyword control 63 on the search screen shown in FIG. Therefore, the processing to be determined differs depending on the control configured on the search screen. In the present embodiment, since the check box control is taken as an example, it is determined that the check box is turned on.
[0056]
If it is determined in step S39 that a keyword has been additionally specified, in step S40, the CPU 12 performs a display for adding the keyword to the search condition control 65 on the search screen. If it is determined in step S39 that the keyword has not been added, the CPU 12 determines in step S41 whether the keyword has been designated for deletion. The determination as to whether or not this keyword has been designated for deletion is made by determining whether or not the deletion of the search condition has been designated in the keyword control 63 on the search screen shown in FIG. Therefore, the processing to be determined differs depending on the control configured on the search screen. In the present embodiment, since the check box control is taken as an example, it is determined that the check box is turned off.
[0057]
If it is determined in step S41 that the keyword has been designated for deletion, in step S42, the CPU 12 displays a display for deleting the keyword on the search condition control 65. If it is determined in step S41 that the keyword is not designated to be deleted, in step S43, the CPU 12 determines whether a representative image has been designated (selected). The determination as to whether or not the representative image has been designated is made based on whether or not the representative image displayed on the search screen shown in FIG. 6 has been instructed (pressed) by the input device 20 (for example, a mouse).
[0058]
If it is determined in step S43 that the representative image has been designated, in step S44, the CPU 12 substitutes the keyword into the search condition control 65 as a search condition sentence. Then, the process proceeds to the keyword search process shown in FIG. 11 in step S47. If it is determined in step S43 that the representative image has not been designated, in step S45, the CPU 12 determines whether the search is to be executed. When the search execution control 66 in the search screen of FIG. 6 is designated in the input device 20 in step S45 and it is determined that the search is to be executed, the CPU 12 acquires the setting of the search condition from the search condition control 65 in step S46, The process proceeds to the keyword search process shown in FIG. 11 in step S47.
[0059]
In step S47, the CPU 12 acquires a search condition from the search condition control 65 on the search screen in FIG. 6, and performs a keyword search under the search condition. Next, in step S48, the CPU 12 acquires image file names from the image IDs in the order acquired in the keyword search, and displays image data as a result of the search on the search screen.
[0060]
<Specific example of keyword registration processing>
Next, a specific operation of the image search device will be described with an example. The process of registering a keyword will be described with reference to the drawings. For example, keywords “mouse”, “keyboard”, and “personal computer” are assigned to image data of “000_0002.jpg”, and keywords “mouse”, “dog”, and “pet shop” are assigned to image data of “001_0002.jpg”. It shall be. In step S12 of FIG. 7, the image data “000_0002.jpg” to be registered is acquired. From the acquired image data, “001 — 0002.jpg” is registered as the image data file name and “002” is registered as the image ID in the image information table of FIG. Next, in step S13, the keyword "PC" to be registered is acquired, and in step S14, the importance "2" is acquired. Then, in step S15, a semantic classification for the keyword "mouse" is obtained from the thesaurus dictionary of FIG.
[0061]
From FIG. 4, since the semantic classification for “mouse” has been acquired, two or more first hierarchical levels exist in the acquired semantic classification “specific object: animal: specific object: product” in step S74 of FIG. Determine whether or not. Since there are two or more first layers of the “specific object”, the flow shifts to step S75 in FIG. In step S75, "animal" of the judgment classification name is obtained. Then, the search condition is set to “animal” + “mouse”, and a similar content search is performed. Here, as the similar content search, image data indicating the content related to “mouse” of “animal” is searched. Therefore, in the case of the image data shown in FIG. 3 of the present embodiment, there is no search result. Then, in a step S77, it is determined whether or not the image ID “002” to be registered as the keyword “mouse” is displayed as a search result in the similar content search in the step S76. Since the image ID “002” does not exist in the search result, the classification “animal” is not assigned.
[0062]
Next, in step S75, "manufactured product" of the judgment classification name is acquired. Then, the search condition is set to “product” + “mouse”, and a similar content search is performed. Here, as a similar content search, image data indicating the content related to “mouse” of “manufactured product” is searched. Therefore, in the case of the image data shown in FIG. 3 of the present embodiment, the image ID “002” is searched. Then, in a step S77, it is determined whether or not the image ID “002” to be registered as the keyword “mouse” is displayed as a search result in the similar content search in the step S76. Since the image ID “002” exists in the search result, the classification “product” is assigned.
[0063]
Similarly, keywords “keyboard” and “personal computer” are assigned to image data of “000_0002.jpg”, and keywords “mouse”, “dog”, and “pet shop” are assigned to image data of “001_0002.jpg”. . In this case, the keyword “mouse” of the image data “001 — 0002.jpg” is given the classification name “animal” and not “product”. In step S19 in FIG. 8, the image information table in FIG. 2 stores the image ID “002” image data file name “000_0002.jpg”, and stores the image ID “006” image data file name “001_0002.jpg”. I do.
[0064]
In the keyword information table of FIG. 3, the image ID “002” is registered with the keyword “mouse” importance “3”, the keyword “keyboard” importance “2”, and the keyword “PC” importance “2”. In the image ID “006”, the keyword “mouse” is registered with importance “2”, the keyword “dog” is registered with importance “2”, and the keyword “pet shop” is registered with importance “1”.
[0065]
In the semantic classification information table of FIG. 17, the image ID “002” is registered with the keyword “mouse” classification “animal”, the keyword “keyboard” classification “manufactured product”, and the keyword “PC” classification “manufactured product”. In the image ID “006”, the keyword “mouse” classification “animal”, the keyword “dog” classification “animal”, the keyword “pet shop” classification “manufactured product”, and the classification “place” are registered.
[0066]
Next, in the keyword classification information table of FIG. 5, the representative image and the keyword in the classification are acquired. In step S20 in FIG. 8, the registered keyword and its classification name are acquired and duplication is removed. Next, in step S21, it is determined whether the keyword in the registered category name is stored in the keyword category information table of FIG. In this case, it is assumed that there is no data stored in the keyword classification information table. Next, in step S22, a keyword search is performed with the search condition set to “mouse”. Next, in step S23, the most plausible image is obtained from the search result. Then, the processing from step S21 to step S24 is repeated. Similarly, a representative image of a keyword for the registered classification name is obtained and registered in the database. In FIG. 5, the classification name "animal" keyword "mouse" representative image "006" and the classification name "product" keyword "keyboard" representative image "006" are repeated.
[0067]
The changed classification and its keywords are classified into the categories "product" keyword "mouse", classification "product" keyword "keyboard", classification "product" keyword "PC", classification "animal" keyword "mouse", and keyword " Animals, dogs, and pet shops. Therefore, in step S24, the semantic classification is changed to "animal" or "manufactured product", and the next changed order is obtained.
[0068]
This will be described with reference to FIG. In step S61, the changed classification name “product” is acquired. Next, in step S62, the keyword “mouse” added and changed is acquired. Next, in step S63, the importance of the image to which the keyword "mouse" is assigned is added to obtain a total. The obtained value is stored in the level of the keyword classification information table shown in FIG. As described above, in steps S61 to S64, the importance is added to obtain the total importance.
[0069]
<Specific example of image search processing>
Next, the image search processing shown in FIGS. 9 and 10 will be described with a specific example. In step S31 in FIG. 9, the classification name is obtained from the keyword classification information table in FIG. 5, and the classification name to be removed and displayed is obtained. In this case, “animal”, “human”, “plant”, “place”, “product”, and “abstract object” are acquired. In addition, a keyword and a representative image are acquired in descending order of level for each classification name. At this time, addition of the keyword in the classification name is also performed. Then, in step S32, first, the category name "animal" is displayed as the category name 61 on the search screen of FIG. In step S33, the number of keywords “50” is displayed as the number of keywords 62.
[0070]
Then, in steps S34 and S35, keywords and representative images present in the classification of "animal" are displayed in the order in which they were acquired. In the case of the classification name "animal", additional information "fish" and its representative image (image file of image ID "039"), additional information "elephant" and its representative image (image file of image ID "091"), additional information “Dog” and its representative image (image file of image ID “033”) are displayed in order. The same display is performed for all classifications. As a result, the search screen shown in FIG.
[0071]
Next, processing on the actual search screen will be described. When “running” is already set in the search condition control 63 on the search screen in FIG. 6 and “child” is added to the search condition, the operator must select the check box in the keyword control 63 on the search screen in FIG. Turn on. In this case, it is determined in step S39 of FIG. 10 that the additional keyword has been specified, and in step S40, the keyword “child” is added to the search condition control 65, and “running child” is displayed.
[0072]
Similarly, in the case where “running child” is already set in the search condition control 63 and the “child” is deleted from the search condition, the operator selects the check box in the keyword control 63 on the search screen of FIG. Turn off. In that case, it is determined in step S39 that the keyword deletion has been designated, and in step S40, the keyword "child" is deleted and displayed as "running" in the search condition control 65.
[0073]
Next, when searching for “flower” using the representative image, the operator selects the representative image control 64 on the search screen of FIG. In this case, in step S43, it is assumed that the search has been started from the representative image, and the keyword “flower” is set in the search condition control 65. Then, in step S46, "flower" is set in the search condition sentence, and a keyword search is performed. Next, in step S47, the searched image data is displayed.
[0074]
When a search is performed using the search execution control 66, the execution of the search is determined in step S45, and the search condition existing in the search condition control 65 is obtained in step S46, and a keyword search is performed. For example, if the keyword control 63 of the keyword “car” is selected while “running in the city” is entered in the search condition control 65, the search condition is “car running in the city”. Is done.
[0075]
As described above, according to the first embodiment, in an image search in which keywords assigned to image data are pre-hierarchized and a search condition is set based on the hierarchized keywords, the classified keywords are used. , The operability of image search can be improved.
[0076]
[Second embodiment]
In the above-described first embodiment, a method of obtaining a representative image from a result of a search using the incidental information as a search condition is used as a method of obtaining a representative image. However, the following method is used as a second embodiment. Take. That is, in the second embodiment, when additional information is registered as a method for obtaining a representative image, image data to which the registered additional information is added is registered as a representative image of the additional information. Note that the configuration of the image search device and the configurations of various tables according to the second embodiment are the same as those of the first embodiment, and a description thereof will be omitted.
[0077]
Next, a keyword registration process in the image search device according to the second embodiment, which is different from the first embodiment, will be described in detail with reference to FIGS. 1 to 5 and FIGS. .
[0078]
<Operation example when keyword registration processing is performed>
FIG. 18 is a flowchart illustrating an operation example when a keyword registration process is performed in the image search device. The processing shown in this flowchart is executed by the CPU 12 of the image search device based on a program stored in the ROM 13.
[0079]
First, in step S81, the CPU 12 determines whether the registration of the keyword has been completed. If the registration of the keyword has not been completed in step S81, in step S82, the CPU 12 has already been stored in the external storage device 17 or the ROM 13 as image data to be registered based on a predetermined operation by the operator from the input device 20. Image data or image data captured by a digital camera 18 or a digital video 19 or the like is acquired via the peripheral controller 16. Next, in step S83, the CPU 12 acquires a keyword to be added to the image data acquired in step S12 by inputting a keyword from the input device 20 by the operator.
[0080]
Next, in step S84, the CPU 12 obtains the importance of the keyword obtained in step S83 by inputting the importance from the input device 20 by the operator. In the present embodiment, the importance is acquired by the input from the input device 20 by the operator, but the importance may be automatically given according to the input order of the keywords. Then, in steps S85 and S86, the CPU 12 acquires the meaning classification and the part of speech for the keyword acquired in step S83 using the thesaurus dictionary of FIG.
[0081]
Next, in step S87, the CPU 12 determines a classification name. The details are the same as the processing shown in FIGS. 15 and 16 in the first embodiment. Then, in step S88, the CPU 12 searches the keyword classification information table of FIG. 5 using the keyword acquired in step S83 and the classification name acquired in step S87. If there is a representative image, the CPU 12 changes the representative image to the image ID of the image acquired in step S82. If the representative image does not exist, the CPU 12 registers the representative image with the image ID of the image acquired in step S82 using the classification name and the keyword.
[0082]
Next, in step S89, the CPU 12 repeats the processing from step S83 to step S89 until the keyword assigned to the image data acquired in step S82 ends. Further, the CPU 12 stores the image data acquired in step S82, the keyword acquired in step S83, the importance acquired in step S84, and the classification name acquired in step S87 in the RAM 14, respectively.
[0083]
If it is determined in step S81 that the image to which the keyword is to be added is completed, in step S90, the CPU 12 registers each data held in the RAM 14 in the database. This database is obtained by registering the table of the present embodiment as a database, and is stored in the external storage device 17 or the ROM 13. In addition, each of the above tables is configured so that access can be speeded up by creating an index.
[0084]
In step S90, first, the CPU 12 registers the image data acquired in step S82 in the image data file name of the image information table shown in FIG. 2, and assigns an image ID. Next, the CPU 12 stores the keyword acquired in step S83 and the importance acquired in step S84 in the keyword information table shown in FIG. The classification name acquired in step S87 is registered in the semantic classification information table shown in FIG. Next, in step S91, the CPU 12 performs the semantic classification order processing only for the classification in which the keywords have increased in the classification. The details are the same as the processing shown in FIG. 14 in the first embodiment.
[0085]
As described above, according to the second embodiment, unlike the first embodiment, the processing time is reduced, and the registered image is displayed as a representative image. Operability can be improved.
[0086]
[Other embodiments]
An object of the present invention is to supply a storage medium storing program codes of software for realizing the functions of the embodiments to a system or an apparatus, and a computer (or CPU, MPU, or the like) of the system or apparatus stores the storage medium in the storage medium. This is also achieved by reading and executing the program code thus set.
[0087]
In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.
[0088]
Examples of a storage medium for supplying the program code include a floppy (registered trademark) disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, and DVD. -RW, DVD + RW, magnetic tape, nonvolatile memory card, ROM, and the like can be used.
[0089]
When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also an OS (Operating System) running on the computer based on the instruction of the program code. This also includes a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.
[0090]
Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. This also includes the case where the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.
[0091]
【The invention's effect】
As described above, according to the present invention, the image data is classified based on the image data to be registered and the supplementary information added to the image data, a representative image for the supplementary information is set, and the supplementary information is set based on the classification of the image data. And a representative image are displayed, a search condition is set according to the display, an image search is performed based on the search condition, and image data obtained by the image search is displayed. Thereby, in the image search in which the additional information attached to the image data is pre-hierarchized and the search condition is set and searched from the hierarchized additional information, the classified additional information is visually easily understood. The operability of image search can be improved.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an image search device according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating a configuration example of an image information table in the image search device.
FIG. 3 is a diagram illustrating a configuration example of a keyword information table in the image search device.
FIG. 4 is a diagram illustrating a configuration example of a thesaurus dictionary in the image search device.
FIG. 5 is a diagram illustrating a configuration example of a keyword classification information table in the image search device.
FIG. 6 is a diagram illustrating a display example of a search screen in the image search device.
FIG. 7 is a flowchart showing an operation example when keyword registration processing is performed in the image search device.
FIG. 8 is a continuation of the flowchart of FIG. 7;
FIG. 9 is a flowchart illustrating an operation example when an image search process is performed in the image search device.
FIG. 10 is a continuation of the flowchart in FIG. 9;
FIG. 11 is a flowchart illustrating an operation example when a keyword search process is performed in the image search device.
FIG. 12 is a diagram illustrating a configuration example of a different notation synonym dictionary in the image search device.
FIG. 13 is a diagram illustrating a configuration example of a lower concept word dictionary in the image search device.
FIG. 14 is a flowchart illustrating an operation example when the image search device performs a semantic classification order process during a keyword registration process.
FIG. 15 is a flowchart illustrating an operation example when a classification name determination process is performed in a keyword registration process in the image search device.
FIG. 16 is a continuation of the flowchart in FIG. 15;
FIG. 17 is a diagram illustrating a configuration example of a semantic classification information table in the image search device.
FIG. 18 is a flowchart illustrating an operation example when keyword registration processing is performed in the image search device according to the second embodiment of the present invention.
[Explanation of symbols]
11 Image retrieval device
12 CPU (image classification means, representative image setting means, image search means)
13 ROM (storage device)
15 CRT (Additional information display means, search condition setting means, search result display means)
16 Peripheral controller
17 External storage device (storage device)
18 Digital Camera (Imaging Device)
19 Digital video camera (imaging device)
20 input device (search condition setting means)
61 Classification name
62 keywords
63 Keyword Control
64 representative image control
65 Search condition control
66 Search execution control

Claims (9)

In an image search device that searches for image data,
Image classification means for classifying image data according to image data to be registered and additional information added to the image data; representative image setting means for setting a representative image for the additional information; and the additional information based on the classification of the image data. And additional information display means for displaying the representative image, search condition setting means for setting a search condition according to the display, image search means for performing an image search based on the search condition, and image data acquired by the image search. An image search device comprising: a search result display unit for displaying. The representative image setting means sets the incidental information as a search condition, performs an image search based on the search condition, and sets the most plausible image data in the image data group obtained by the image search as the representative image. 2. The image search device according to claim 1, wherein the image search device is set. 2. The image according to claim 1, wherein, when registering the additional information to be added to the image data, the representative image setting unit sets, as the representative image, image data to which the additional information to be registered is added. 3. Search device. The said additional information display means displays the said additional information respectively corresponding to the classification | category name of several image data, and displays the said representative image respectively respectively corresponding to several said additional information. 3. The image search device according to any one of 3. 2. The image search device according to claim 1, wherein the image classification unit classifies the image data using a thesaurus dictionary based on the image data to be registered and the supplementary information added to the image data. 2. The image search device according to claim 1, wherein the image data to be registered includes image data obtained from an imaging device or a storage device. In an image search method for searching image data,
An image classification step of classifying image data based on image data to be registered and additional information to be added to the image data; a representative image setting step of setting a representative image for the additional information; and the additional information based on the classification of the image data. And an additional information display step of displaying the representative image, a search condition setting step of setting a search condition according to the display, an image search step of performing an image search based on the search condition, and image data acquired by the image search. And displaying a search result. A function of classifying the image data according to the image data to be registered and the additional information to be added to the image data, a function of setting a representative image for the additional information, and the additional information and the representative image based on the classification of the image data. A program for causing a computer to realize a function of displaying, a function of setting a search condition according to the display, a function of performing an image search based on the search condition, and a function of displaying image data acquired by the image search. A computer-readable storage medium storing a program for executing an image search method for searching for image data,
The image search method includes a step of classifying image data based on image data to be registered and additional information to be added to the image data; a step of setting a representative image for the additional information; and a step of setting the additional image based on the classification of the image data. Displaying information and the representative image, setting a search condition according to the display, performing an image search based on the search condition, and displaying image data acquired by the image search. A storage medium characterized by the above-mentioned.

Images