JP3649264B2 - Image search device, image search method, keyword extraction device, and keyword extraction method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to an image search apparatus., Image search method, keyword extraction device, and keyword extraction methodIn particular, an image search apparatus capable of easily and reliably searching image data by using a keyword, Image search method, keyword extraction device, and keyword extraction methodAbout.
[0002]
[Prior art]
Conventionally, as a method for searching for an image, a method using pattern matching and a method using a keyword are known.
[0003]
In the method based on pattern matching, it is determined whether image data matches on a pixel basis.
[0004]
On the other hand, in the method using a keyword, when a user inputs an appropriate keyword for each image and searches for an image, the keyword is input instead of inputting the image, and the keyword is corresponded. I am trying to search for images.
[0005]
[Problems to be solved by the invention]
However, the pattern matching method has a problem that it takes time to search.
[0006]
On the other hand, in the method using the keyword, the search can be performed quickly, but the administrator who manages the database needs to set the keyword for each image, and creates and maintains the database. It takes a lot of effort. In addition, since the search needs to be performed using the keywords used by the administrator, there is a problem that the search cannot be performed using free terms.
[0007]
Furthermore, if the keyword set by the administrator is not appropriate, it is impossible to search for a desired image, and there is a problem that the keyword setting operation greatly affects the search accuracy.
[0008]
The present invention has been made in view of such a situation, and makes it possible to search for a desired image easily and reliably.
[0009]
[Means for Solving the Problems]
  The image search apparatus according to claim 1, wherein the keyword inputted by referring to the storage means for storing the relationship between the image feature quantity and the keyword, and the relationship between the image feature quantity and the keyword stored in the storage means. Is converted into a corresponding image feature amount, a keyword conversion means, and image data stored in the database,Dividing means for dividing into a plurality of areas, and the image data of each area for the plurality of areasImage conversion means for converting to image features;Of the image data of each areaComparing means for comparing the image feature quantity with the image feature quantity converted by the keyword converting means, and search means for searching for image data from the database corresponding to the comparison result of the comparing means .
[0011]
Priorities can be set for image feature quantities or keywords, and search can be performed in accordance with the priorities.
[0012]
When the image feature amount includes color information and shape information,PredeterminedWhen a mode is set, search should be performed using only one of color information and shape information.Butit can.
[0013]
A single keyword can be associated with a plurality of image feature amounts, and a single image feature amount can be associated with a plurality of keywords.
[0014]
  Claim 5Described inImage search methodIs a memory that stores the relationship between image features and keywordsTo the meansWith reference to the relationship between the stored image feature quantity and the keyword, a keyword conversion step for converting the input keyword into a corresponding image feature quantity, and image data stored in the database,A division step for dividing the image into a plurality of areas, and for each of the plurality of areas, the image data of each area isAn image conversion step for converting into image features;Of the image data of each areaA comparison step for comparing the image feature quantity with the image feature quantity converted in the keyword conversion step, and a search step for retrieving image data from the database in correspondence with the comparison result in the comparison step.includingIt is characterized by that.
[0015]
  Claim 6The image search device described in the above, a dividing unit that divides the image data into a plurality of regions based on colors,For multiple areas, the image data of each areaShape information extraction means for extracting shape information;For multiple areas, the image data of each areaColor information extracting means for extracting color information;Of the image data of each areaShape information comparison means for comparing the shape information with the input shape information;Of the image data of each areaColor information comparing means for comparing the color information with the input color information; and search means for searching for image data from the database in correspondence with the comparison results by the shape information comparing means and the color information comparing means. Features.
[0016]
The dividing unit can reduce the resolution of the image and then perform division for each region of the same color.
[0017]
  Claim 8Described inImage search methodA division step of dividing the image data into a plurality of regions based on the color;For multiple areas, the image data of each areaA shape information extraction step for extracting shape information;For multiple areas, the image data of each areaA color information extraction step for extracting color information;Of the image data of each areaA shape information comparison step for comparing the shape information with the input shape information;Of the image data of each areaA color information comparison step for comparing the color information with the input color information; a search step for retrieving image data from the database in accordance with the comparison results of the shape information comparison step and the color information comparison step;includingIt is characterized by that.
[0018]
  Claim 9Described inKeyword extractorIncludes a table storage means for storing a table representing a correspondence relationship between the keyword and the image feature amount;A dividing unit that divides input image data into a plurality of regions; an image converting unit that converts image data of each region into an image feature amount for the plurality of regions;Browse the tableFor image feature amount of image data of each areaCorresponding keywordSearch means to search forIt is characterized by providing.
[0019]
A keyword selection means for selecting a related keyword corresponding to the keyword can be further provided.
[0020]
  Claim 11Described inKeyword extraction methodIsA table that represents the correspondence between a keyword and an image feature amount, a division step that divides input image data into a plurality of regions, an image conversion step that converts image data of each region into an image feature amount, for a plurality of regions Is stored in the storage meansBrowse the tableFor image feature amount of image data of each areaCorresponding keywordAnd a search step for searchingIt is characterized by that.
[0021]
  The image search device according to claim 1 andClaim 5Described inImage search methodIn, image data stored in the database is converted into image feature values. Further, the relationship between the image feature amount and the keyword is stored, and the input keyword is converted into the corresponding image feature amount with reference to the stored relationship. The image feature amount converted from the image data is compared with the image feature amount converted from the keyword, and the image data is retrieved from the database corresponding to the comparison result.
[0022]
  Claim 6The image search device described inClaim 8DescribedImage search methodIn, the image is divided into a plurality of regions, and shape information and color information of the divided image are extracted. Then, image data is retrieved from the database in correspondence with the extracted shape information and color information.
[0023]
  Claim 9Described inKeyword extractorandClaim 11Described inKeyword extraction methodStores a table showing the correspondence between keywords and image feature values, and the image feature values correspond with reference to this table.Converted to keywordsConvertedKeyword outputIs done.
[0024]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below, but in order to clarify the correspondence between each means of the invention described in the claims and the following embodiments, in parentheses after each means, The features of the present invention will be described with the corresponding embodiment (however, an example) added. However, of course, this description does not mean that each means is limited to the description.
[0025]
  The image search apparatus according to claim 1 includes a storage unit (for example, the image feature database 24 in FIG. 2) that stores a relationship between an image feature amount and a keyword, and an image feature amount and a keyword that are stored in the storage unit. The keyword conversion means (for example, step S2 in FIG. 5) for converting the input keyword into the corresponding image feature amount, and the image data stored in the database are referred toDividing means for dividing into a plurality of regions (for example, step S61 in FIG. 12),For multiple areas, the image data of each areaImage conversion means (for example, step S6 in FIG. 5) for converting into image feature amounts and converted by the image conversion meansOf the image data of each areaComparison means for comparing the image feature quantity with the image feature quantity converted by the keyword conversion means (for example, step S8 in FIG. 5), and search for retrieving image data from the database corresponding to the comparison result of the comparison means Means (for example, step S10 in FIG. 5).
[0027]
  Claim 6The image search device described in 1 is divided by a dividing unit (for example, step S61 in FIG. 12) that divides the image data into a plurality of regions based on the color, and the dividing unit.For multiple areas,Shape information extraction means (for example, step S64 in FIG. 12) for extracting the shape information of the image data and the division meansFor multiple areas,Extracted by color information extracting means (for example, step S63 in FIG. 12) for extracting color information of image data and shape information extracting means.Of the image data of each areaShape information comparing means (for example, step S83 in FIG. 15) that compares the shape information with the input shape information, and color information extracting means.Of the image data of each areaCorresponding to the comparison result by the color information comparison means (for example, step S84 in FIG. 15) for comparing the color information with the input color information, the shape information comparison means and the color information comparison means, the image data is obtained from the database. Search means (for example, step S10 in FIG. 5) for searching is provided.
[0028]
  Claim 9Described inKeyword extractorIncludes a table storage unit (for example, the image feature database 24 in FIG. 2) for storing a table representing a correspondence relationship between the keyword and the image feature amount;A dividing unit (for example, step S61 in FIG. 12) that divides input image data into a plurality of regions, and an image converting unit (for example, FIG. 5) that converts image data in each region into image feature amounts for the plurality of regions. Step S6),Browse the tableOf the image data of each areaCorresponding image featuresConvert to keywordConversion means (for example, step S112 in FIG. 19) to be converted and converted by the conversion meansOutput keywordsOutput means (for example, step S119 in FIG. 19).
[0029]
FIG. 1 shows a configuration example of an image search apparatus according to the present invention. The keyboard 1 is operated by the user when inputting characters. By operating the tablet 2 with the pen 2A, for example, an image to be searched is input. The playback device 3 plays back image data recorded on a recording medium such as a solid-state memory, a disk, or a tape. The display 4 displays an image. The interface 5 performs interface processing between the keyboard 1 to the display 4 and the control circuit 10. The control circuit 10 is configured by, for example, a microcomputer and controls each unit.
[0030]
The keyword image feature amount conversion device 6 performs processing for converting a keyword into an image feature amount, or conversely, converting an image feature amount into a keyword. The image data image feature amount conversion device 7 performs processing for converting image data into image feature amounts. The feature amount matching device 8 performs processing for comparing the image feature amount supplied from the keyword image feature amount conversion device 6 or the image data image feature amount conversion device 7 with the image feature amount read from the image database 9. . The image database 9 stores image data of a plurality of images as a database.
[0031]
Note that the keyword image feature quantity conversion device 6 to the image database 9 can be configured by hardware, but can also be configured by a software program.
[0032]
FIG. 2 shows a configuration example of the keyword image feature amount conversion device 6. The related keyword output device 21 is connected to the related keyword database 23. When a keyword is input, the related keyword output device 21 searches for and outputs a related keyword related to the keyword. In the related keyword database 23, related keywords corresponding to predetermined keywords are registered as a database. The character image conversion device 22 is connected to the image feature database 24. When a keyword is input, the character image conversion device 22 refers to the image feature database 24 and outputs an image feature amount corresponding to the keyword, or vice versa. When an image feature amount is input, a corresponding keyword is searched and output. The image feature database 24 stores a conversion table representing the correspondence between keywords and image feature amounts.
[0033]
FIG. 3 shows a configuration example of the image data image feature amount conversion device 7. The region dividing device 31 divides input image data into a plurality of regions, and outputs the image data of each region to the color information extracting device 32 and the shape information extracting device 33. The color information extraction device 32 and the shape information extraction device 33 extract an image feature consisting of color information or an image feature amount consisting of shape information from the image data for each region input from the region dividing device 31, and output it. Has been made.
[0034]
FIG. 4 illustrates a configuration example of the feature amount matching apparatus 8. The shape information comparison device 41 compares the shape information output from the keyword image feature amount conversion device 6 or the image data image feature amount conversion device 7 with the shape information read from the image database 9, and the comparison result ( (Shape similarity) is output to the similarity integration unit 43. Further, the color information comparison device 42 compares the color information output from the keyword image feature amount conversion device 6 or the image data image feature amount conversion device 7 with the color information read from the image database 9 and compares the color information. The result (color similarity) is output to the similarity integration unit 43. The similarity integration unit 43 integrates the shape similarity supplied from the shape information comparison device 41 and the color similarity supplied from the color information comparison device 42 to obtain an overall similarity and outputs the total similarity to the determination unit 44. ing. The determination unit 44 determines the similarity between two images from the value of the overall similarity, and outputs the determination result to the control circuit 10.
[0035]
Next, image search processing will be described with reference to the flowchart of FIG. Now, in this process, it is assumed that a predetermined keyword is input and an image corresponding to the keyword is searched.
[0036]
First, in step S1, the user operates the keyboard 1 and inputs a predetermined keyword (for example, “natural”). When receiving the input of the keyword via the interface 5, the control circuit 10 supplies the keyword to the keyword image feature amount conversion device 6 in step S2, and executes the keyword image feature amount conversion process. The details of this processing are shown in the flowchart of FIG.
[0037]
That is, the keyword image feature quantity conversion device 6 executes related keyword output processing in step S21. Details of the related keyword output processing are shown in the flowchart of FIG.
[0038]
First, in step S41, when a keyword is supplied from the control circuit 10, the related keyword output device 21 sets this keyword as an initial keyword. In this case, the keyword “nature” is set as the initial keyword.
[0039]
In step S42, the related keyword output device 21 sets the initial keyword set in step S41 as a parent keyword. In step S43, a child keyword is searched for the parent keyword. That is, the related keyword output device 21 refers to the related keyword database 23 and searches for a child keyword having “nature” as a parent keyword. Accordingly, for example, as shown in FIG. 8, three first-ranked child keywords (related keywords) of “sea”, “mountain”, and “sky” with respect to the parent keyword (initial keyword) of “nature” ) Is searched.
[0040]
Next, in step S44, the related keyword output device 21 determines whether or not the child keyword has reached the third rank. In this case, since only the first rank child keyword has been searched, a NO determination is made, and the process proceeds to step S45. In step S45, the child keyword is shifted to the parent keyword. In this case, for example, the child keyword “sea” is set as the parent keyword.
[0041]
Next, returning to step S 43, the child keyword for the parent keyword “sea” is searched from the related keyword database 23. Thereby, for example, as shown in FIG. 8, child keywords of “mountain”, “river”, and “fish” are searched.
[0042]
In step S45, the other keywords “mountain” and “sky” are also set as parent keywords. As a result, in step S43, the child keyword corresponding to the parent keyword “mountain” and the child keyword corresponding to the parent keyword “empty” are searched. In the display example of FIG. 8, child keywords “forest”, “river”, and “nature” are searched for the parent keyword “mountain”, and “blue” is searched for the parent keyword “sky”. , "Sea", and "Earth" child keywords are being searched. Thus, the second rank child keyword is searched.
[0043]
Next, proceeding to step S44, it is determined whether or not the child keyword has reached the third rank. In this case, since it has not yet reached, the processing of step S45 and step S43 is executed again, and as shown in FIG. 8, the third rank child keyword having the second rank keyword as the parent keyword is searched. . In this way, when the search up to the third rank is performed, a total of 40 (= 1 + 3 + 9 + 27) keywords (however, there are some same keywords) are searched.
[0044]
If it is determined in step S44 that child keywords up to the third rank have been searched, the process proceeds to step S46, where the related keywords are weighted and the degree of relevance is calculated.
[0045]
Here, the initial keyword has a relevance level of 8, the first rank related keyword has a relevance level of 4, the second rank keyword has a relevance level of 2, and the third rank keyword has a relevance level of 1. It is said. And when the same keyword appears in multiple ranks, or when the same keyword appears in duplicate in the same rank, each relevance value is added, The degree of relevance of the keyword.
[0046]
For example, in the example of FIG. 8, between the initial keyword and the second rank, the keyword “natural” appears once for the initial keyword and the second rank. Therefore, the relevance level of “natural” is 10 (= 8 + 2).
[0047]
The keyword “sea” appears once in the first rank and once in the second rank. Therefore, the degree of association of this “sea” is 6 (= 4 + 2). Since the keyword “river” appears twice in the second rank, the degree of association is 4 (= 2 + 2).
[0048]
Since the keyword “Earth” appears only once in the second rank, the degree of association is set to 2.
[0049]
As described above, when the relevance level is calculated for all the appearing keywords, the related keyword output device 21 selects five keywords in descending order of relevance level in step S47. In step S48, the selected related keyword is output.
[0050]
As described above, when the related keyword output process in step S21 of FIG. 6 is completed, the related keyword output device 21 outputs the five related keywords selected in step S21 to the character image conversion device 22 in step S22. To do. When the character image conversion device 22 receives five consecutive keywords from the related keyword output device 21, the character image conversion device 22 performs character image conversion processing with reference to the image feature database 24 in step S23.
[0051]
In the image feature database 24, for example, as shown in FIG. 9, a keyword and an image feature amount are associated with each other and stored as a conversion table. The image feature amount is divided into color information and shape information.
[0052]
10 and 11 show examples of the correspondence between the keyword of the image feature database and the image feature amount. In FIG. 10, “green” is associated with the keyword “mountain”, and “triangle” is associated with the shape information.
[0053]
In the example of FIG. 11, “blue” color information and “square” shape information are associated with the keyword “sea”.
[0054]
The character image conversion device 22 searches the image feature database 24 for image feature amounts (shape information and color information) corresponding to the five consecutive keywords supplied from the related keyword output device 21, and the search is performed in step S24. The image feature amount obtained as a result of is output as an image feature amount corresponding to the input related keyword.
[0055]
As described above, when the keyword image feature value conversion process in step S2 of FIG. 5 is performed, the image feature value searched by the character image conversion device 22 in step S3 is supplied to the feature value matching device 8. .
[0056]
On the other hand, when an image search is instructed, the processes of steps S4 to S7 are executed in parallel with the processes of steps S1 to S3. In step S4, the control circuit 10 controls the image database 9 and instructs reading of the image data stored therein. In response to this command, the image database 9 reads the image data in step S5 and supplies it to the image data image feature amount conversion device 7. When receiving the supply of image data, the image data image feature amount conversion device 7 executes an image data image feature amount conversion process in step S6. Details of this image data image feature amount conversion processing are shown in FIG.
[0057]
First, in step S61, when the image data image feature value conversion apparatus 7 receives the supply of image data from the image database 9, in step S61, the image data image feature value conversion device 7 executes region division processing. That is, the area dividing device 31 performs a process of dividing the input image data into a plurality of areas. FIG. 13 shows an example of this area division processing. Now, for example, it is assumed that an image is supplied from the image database 9 and is covered with trees, and the red sun floats in the sky blue sky at the upper left of the green mountain. In this image, a brown earth is spreading under the mountain, and a blue river flows over it.
[0058]
The area dividing device 31 divides this image into a plurality of areas so that the same color belongs to the same area. That is, in this example, the area is divided into five areas: a red area (sun), a sky blue area (sky), a green area (mountain), a brown area (ground), and a blue area (river). The
[0059]
Returning to FIG. 12, the image data for each region obtained by the region dividing process in step S61 is supplied to the color information extracting device 32 and the shape information extracting device 33 in step S62. The color information extraction device 32 and the shape information extraction device 33 respectively perform color information extraction processing or shape information extraction processing in step S63 or step S64, respectively.
[0060]
In the case of the display example of FIG. 13, the color information extraction device 32 executes processing for specifying the representative color of the input image data of each region. In the case of the display example of FIG. 13, for example, the representative color of the sun region is red, the representative color of the sky region is sky blue, the representative color of the mountain region is green, the representative color of the earth region is brown, and the representative color of the river region The representative color is specified as blue.
[0061]
It should be noted that the color used as a reference in the region division processing in step S61 can be used as it is in the processing for specifying the representative color in the region in the color information extraction device 32.
[0062]
On the other hand, the shape information extraction device 33 extracts the shape feature of the image data of each region. For example, in the display example of FIG. 13, the shape of the sun region is a circle, and the shape of the mountain region is a triangle. However, it is difficult to recognize the shape of the river region, the shape of the ground region, or the shape of the sky region as a simple shape. Therefore, the shape extraction device 33 calculates the center of gravity position and area of each region image, superimposes it with a typical predetermined shape having the same area, and obtains the shape with the largest overlapping portion as its shape. Extracted as the shape of the image data of the region.
[0063]
FIG. 14 shows a state in which the river region is overlaid with six typical shapes (triangle, regular square, circle, pentagon, rectangle, cross) prepared in advance. . In this way, the shape of the river is compared with the representative shape prepared in advance, and is recognized as the shape having the largest overlapping portion.
[0064]
As described above, when the extraction process of color information and shape information for each region is completed, the extracted color information and shape information are output from the image data image feature amount conversion device 7 in step S65.
[0065]
As described above, when the image data image feature value conversion processing in step S6 in FIG. 5 is completed, in step S7, the control circuit 10 displays the color information and shape information output from the image data image feature value conversion device 7. And transferred to the feature amount matching device 8.
[0066]
When the feature amount matching device 8 receives supply of image feature amounts from the keyword image feature amount conversion device 6 and the image data image feature amount conversion device 7 as described above, the feature amount matching device 8 executes feature amount matching processing in step S8. To do. Details of this processing are shown in FIG.
[0067]
The feature amount matching device 8 acquires the image feature amount output from the keyword image feature amount conversion device 6 in step S81, and acquires the image feature amount output from the image data image feature amount conversion device 7 in step S82. These image feature amounts are supplied to the shape information comparison device 41 and the color information comparison device 42, respectively. When the shape information comparison device 41 and the color information comparison device 42 are supplied with two pieces of shape information and color information, respectively, the shape information comparison processing or the color information comparison processing is executed in step S83 or step S84.
[0068]
The shape information comparison device 41 has a correspondence table of shape similarities corresponding to combinations of representative shapes, and performs a shape information similarity determination process according to this correspondence table. FIG. 16 shows an example of such a correspondence table. FIG. 16 shows the shape similarity when the representative shapes of a circle, a triangle, and a quadrangle are compared. When the same representative shape is compared, such as a circle and a circle, a triangle and a triangle, or a quadrangle and a quadrangle, the shape similarity is 10. For example, the shape similarity between a circle and a triangle is 4, and the shape similarity between a circle and a rectangle is 5. The shape similarity between the triangle and the quadrangle is 5. Such a correspondence table can be obtained experimentally by allowing a plurality of people to determine the shape similarity when many representative figures are combined.
[0069]
In this example, the shape similarity is determined in 10 stages, and the larger the value is, the more similar the two shapes are.
[0070]
On the other hand, the color information comparison device 42 has a diagram of color information as shown in FIG. 17, and determines the color similarity from the distance on the diagram of the two color information. The smaller the distance between the two color information, the larger the color similarity is set (maximum 10), and the larger the distance is, the smaller the color similarity value is set (minimum value 0).
[0071]
As described above, when the shape information comparison process and the color information comparison process are performed in step S83 and step S84, and the shape similarity and the color similarity are obtained, they are obtained in step S85 or step S86. The information is output from the information comparison device 41 or the color information comparison device 42 to the similarity integration unit 43.
[0072]
When the similarity integration unit 43 receives supply of the shape similarity and the color similarity from the shape information comparison device 41 and the color information comparison device 42, respectively, the similarity integration processing is performed in step S87 to calculate the overall similarity. To do. In step S88, the overall similarity is output to the determination unit 44. This total similarity has a maximum value of 100 and a minimum value of 0. The more similar the two images, the greater the value of the overall similarity.
[0073]
In step S89, the determination unit 44 determines whether or not the overall similarity is greater than the maximum value of the overall similarities obtained so far. If it is not larger than the maximum value, the process proceeds to step S90, where it is determined whether or not matching processing has been performed for all search targets. If there is an image feature amount that has not yet undergone matching processing, the process returns to step S82. The image feature quantity of the image from the image database 9 to be searched next is selected. Then, the matching process between the image feature quantity and the image feature quantity corresponding to the keyword is executed in the same manner as described above.
[0074]
If it is determined in step S89 that the obtained overall similarity is greater than the previous overall similarity, the process proceeds to step S91, and the maximum overall similarity stored up to that point is determined as the newly obtained maximum overall similarity. Update with similarity.
[0075]
As described above, when it is determined in step S90 that matching processing has been performed for all search targets, the process proceeds to step S92, and the determination unit 44 outputs an image feature amount corresponding to the maximum total similarity.
[0076]
As described above, when the feature amount matching process in step S8 of FIG. 5 is completed, the feature amount matching device 8 outputs an image feature amount having the maximum total similarity. When receiving the image feature amount, the control circuit 10 supplies it to the image database 9 in step S9. When receiving the supply of the image feature amount, the image database 9 searches for image data having the supplied image feature amount in step S10. In step S11, the image data obtained as a result of the search is output to the control circuit 10. In this way, the control circuit 10 outputs the image data supplied from the image database 9 to the display 4 via the interface 5 and displays it.
[0077]
FIG. 18 conceptually shows the image search processing from the above keywords. That is, in the above example, the keyword “nature” is input, and this keyword is converted into related keywords such as “forest”, “mountain”, “sea”, “river”, and these related keywords are Further, it is converted into an image feature amount. For example, the keyword “forest” is converted into shape information of “triangle” and color information of “green”. The keyword “mountain” is converted into shape information “square” and color information “green”. The keyword “sea” is converted into “quadrangle” shape information and “blue” color information, and the keyword “river” is converted into “rectangle” shape information and “blue” color information.
[0078]
On the other hand, image data in the image database 9 is read out and divided into a plurality of areas, and image feature amounts are extracted for each divided area. Then, the image feature quantity is compared with the image feature quantity obtained by converting the keyword, and image data matching both is searched.
[0079]
In the image feature database 24, a table representing the correspondence between keywords and image feature amounts created in advance through experiments or the like is recorded, and in the related keyword database 23, free and natural associations obtained in advance through experiments or the like. If the keyword table is recorded, the user does not need to create a new keyword or image feature amount. Therefore, after purchasing the image search apparatus, it is possible to immediately start the search process simply by recording the image data to be searched in the image database 9. As a result, the administrator can search for a desired image quickly and accurately using a keyword that is free and close to nature without having to specify new keywords and image features. Become.
[0080]
In this image search device, not only can predetermined image data be searched from keywords as described above, but also image data can be input and a keyword corresponding thereto can be searched. FIG. 19 shows an example of processing (image recognition processing) in that case. First, in step S111, the user inputs image data. This input is performed, for example, by operating the pen 2A on the tablet 2 or by reproducing a predetermined recording medium by the reproducing device 3. When receiving the supply of the image data input in step S111, the control circuit 10 supplies the image data to the image data image feature amount conversion device 7. In step S112, the image data image feature amount conversion device 7 executes an image data image feature amount conversion process. Details of this processing are as shown in FIG. That is, the image data image feature amount conversion device 7 divides the input image data into a plurality of regions, and extracts color information and shape information for each region as image feature amounts.
[0081]
When the image feature amount is extracted in step S112, in step S113, the control circuit 10 transfers the image feature amount output from the image data image feature amount conversion device 7 to the keyword image feature amount conversion device 6. The keyword image feature amount conversion device 6 executes a character image conversion process in step S114.
[0082]
That is, when receiving the input of the image feature amount output from the image data image feature amount conversion device 7, the character image conversion device 22 searches the corresponding keyword with reference to the image feature database 24. The retrieved keyword is transferred from the character image conversion device 22 to the related keyword output device 21 in step S115. In step S116, the related keyword output device 21 refers to the related keyword database 23 and searches for a related keyword corresponding to the input keyword. Also in this case, as in the case described above with reference to FIG. 8, related keywords up to the third rank are searched for one keyword. As a result, 40 related keywords are obtained in step S117 together with the initial keywords.
[0083]
When the related keyword is obtained in this way, the related keyword output device 21 executes keyword selection processing in step S118. That is, as in the case described above, the degree of relevance is calculated and, for example, five keywords having the largest degree of relevance are selected. In step S119, the five selected keywords are output.
[0084]
When five keywords are output from the keyword image feature quantity conversion device 6, the control circuit 10 outputs them to the display 4 via the interface 5 and displays them.
[0085]
FIG. 20 schematically shows the above image recognition process (keyword extraction process). As shown in the figure, when one piece of image data is input, this image data is divided into image data of a plurality of regions, and image feature amounts are extracted for each region. Each image feature amount is converted into a corresponding keyword. Of the converted keywords, the five most relevant keywords are selected. In the display example of FIG. 20, the keywords “nature”, “mountain”, and “river” are extracted corresponding to the input image data.
[0086]
When dividing one image into a plurality of regions, in the above embodiment, the number of colors expressing the image is reduced (by combining them with representative colors). Not only the color but also the resolution may be lowered. By performing this process, the color of the image is automatically expressed by the representative color. As a result, complicated shapes that are difficult to recognize can be grouped into rough shapes, so that the shape extraction process can be easily performed.
[0087]
A priority can be given to the divided area, and the search can be performed in order from the area with the highest priority. For example, if there is a region located in the center of one image, and the ratio of the region to the total area of the image is 20% or more, the image in that region is a highly important image, that is, Stipulate that this is a high priority area. Conversely, an image in a region that is not in the central portion and occupies a small area is an image with low importance. Thus, the priority may be determined in accordance with the position of the divided area.
[0088]
Further, among the images, a background image and other images (foreground image) may be identified, and an image other than the background image (foreground image) may be set as a high priority image.
[0089]
Whether the image in each region is a background image or other image (foreground image) can be determined from the ratio of the length of the outer frame of one image, for example.
[0090]
For example, in the display example of FIG. 21, a mountain image is arranged as a foreground image on a blue sky background image. The top of the mountain is covered with white snow, and the foot is covered with brown soil. It is assumed that this one image is divided into three areas, that is, an empty area, a peak area, and a skirt area. The screen is L₁, L₂, L_Three, L_FourIt is composed of the outer frame. The empty area is the L of the outer frame of the screen.₇, L₂, L_Three, L_Four, L_FiveIs included. On the other hand, the base region is L in the outer frame.₆The summit area does not include the outer frame. Therefore, the ratio of the outer frame length to the empty area is (L₇+ L₂+ L_Three+ L_Four+ L_Five) / (L₁+ L₂+ L_Three+ L_Four) On the other hand, the proportion of the outer frame in the base region is (L₆) / (L₁+ L₂+ L_Three+ L_Four), And the ratio of the outer frame to the summit area is zero. The ratio of the former is larger than the ratio of the latter two. Therefore, the former empty area is more likely to be a background image than the latter two base areas and the peak area. Therefore, an empty area can be determined as a background image.
[0091]
As described above, when the background image and the other image can be identified, the image is divided into the background image region and the other image region. A region can be divided into regions. That is, as described above, when the region is divided by color, the display example of FIG. 21 is divided into three regions: a blue sky region, a white snow mountaintop region, and a brown soil foot region. However, if it is divided into a background image region and other regions, a mountain region (foreground image) composed of a white snow mountain top region and a brown soil foot region is defined as one region. be able to. In other words, this makes it possible to make “a mountain covered with snow” as one region.
[0092]
In the above, one image feature amount is associated with one keyword, but a plurality of image feature amounts may be associated. Conversely, a plurality of keywords may be associated with one image feature amount. In such a case, priority can be given to a plurality of image feature quantities or keywords. Furthermore, priority can be set only for color information or only for shape information among image feature amounts. Further, when a predetermined search mode is instructed, the search process can be performed using only one of color information and shape information. In this way, more flexible search processing is possible.
[0093]
Note that as a transmission medium for transmitting a program for performing the processing as described above to a user, a communication medium such as a network or a satellite can be used in addition to a recording medium such as a magnetic disk, a CD-ROM, or a solid-state memory.
[0094]
【The invention's effect】
  As described above, the image search device according to claim 1 andClaim 5Described inImage search methodAccording to the above, the image data stored in the database is converted into an image feature amount, the input keyword is converted into an image feature amount with reference to a relationship between the image feature amount and the keyword stored in advance, Each image feature amount is compared and image data is searched from the database corresponding to the comparison result, so that desired image data can be quickly and accurately used using keywords that are free and close to nature. It becomes possible to search.
[0095]
  Claim 6The image search device described inClaim 8Described inImage search methodAccording to the above, the image is divided into a plurality of regions based on the color, the shape information and color information of the divided image are extracted, and predetermined image data is retrieved from the database based on the extracted shape information and color information As a result, desired image data can be retrieved quickly and reliably.
[0096]
  Claim 9Described inKeyword extractorandClaim 11Described inKeyword extraction methodAccording to the above, referring to the table representing the correspondence between the keyword and the image feature amount, the image feature amount is associated.Convert to keywordBecause I tried toThe image data can be input and the corresponding keyword can be searched.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an image search apparatus to which the present invention is applied.
FIG. 2 is a block diagram illustrating a configuration example of the keyword image feature amount conversion apparatus in FIG. 1;
FIG. 3 is a block diagram illustrating a configuration example of the image data image feature value conversion apparatus in FIG. 1;
4 is a block diagram illustrating a configuration example of the feature amount matching apparatus in FIG. 1;
FIG. 5 is a flowchart illustrating image search processing in the image search apparatus of FIG. 1;
6 is a flowchart illustrating keyword image feature amount conversion processing in step S2 of FIG.
FIG. 7 is a flowchart for explaining related keyword output processing in step S21 of FIG.
FIG. 8 is a diagram illustrating related keywords.
FIG. 9 is a diagram illustrating a relationship between a keyword and an image feature amount.
FIG. 10 is a diagram illustrating a relationship between a keyword and an image feature amount.
FIG. 11 is a diagram illustrating a relationship between a keyword and an image feature amount.
FIG. 12 is a flowchart illustrating details of image data image feature amount conversion processing in step S6 of FIG. 5;
FIG. 13 is a diagram illustrating area division.
FIG. 14 is a diagram for explaining shape information extraction processing;
FIG. 15 is a flowchart illustrating details of the feature amount matching process in step S8 of FIG.
FIG. 16 is a diagram illustrating determination of shape similarity.
FIG. 17 is a diagram illustrating color similarity.
FIG. 18 is a diagram illustrating a similar image search process.
FIG. 19 is a flowchart illustrating image recognition processing.
FIG. 20 is a diagram illustrating image recognition processing.
FIG. 21 is a diagram illustrating a background image.
[Explanation of symbols]
1 Keyboard
2 tablets
3 Playback device
4 display
5 Interface
6 Keyword image feature quantity conversion device
7 Image data Image feature conversion device
8 Feature matching device
9 Image database
10 Control circuit
21 Related keyword output device
22 Character image converter
23 Related Keyword Database
24 Image feature database
31 Area dividing device
32 color information extraction device
33 Shape information extraction device
41 Shape information comparison device
42 color information comparison device
43 Similarity Integration Department
44 judgment part

Claims (11)

Storage means for storing the relationship between the image feature quantity and the keyword;
Referring to the relationship between the image feature quantity and the keyword stored in the storage means, the keyword conversion means for converting the input keyword into the corresponding image feature quantity;
Dividing means for dividing the image data stored in the database into a plurality of regions;
For the plurality of regions, image conversion means for converting image data of each region into the image feature amount;
Comparison means for comparing the image feature amount of the image data of each region with the image feature amount converted by the keyword conversion means;
An image search apparatus comprising: search means for searching image data from the database corresponding to the comparison result of the comparison means. Priorities are set for the image feature quantities or keywords,
The image search apparatus according to claim 1, wherein the search unit performs a search according to the priority. The image feature amount includes color information and shape information,
The image search apparatus according to claim 1, wherein when the predetermined mode is set, the search unit performs a search using only one of the color information and the shape information. The plurality of the image feature amounts correspond to one of the keywords, or the plurality of keywords correspond to one of the image feature amounts. Image search device. A keyword conversion step of converting an input keyword into the corresponding image feature amount with reference to the relationship between the image feature amount and the keyword stored in the storage means for storing the relationship between the image feature amount and the keyword; ,
A division step of dividing the image data stored in the database into a plurality of regions;
For the plurality of regions, an image conversion step of converting image data of each region into an image feature amount;
A comparison step of comparing the image feature amount of the image data of each region with the image feature amount converted in the keyword conversion step;
A retrieval step for retrieving image data from the database in correspondence with the comparison result in the comparison step;
An image search method comprising: A dividing means for dividing the image data into a plurality of regions based on the color;
For the plurality of regions, shape information extraction means for extracting shape information of image data of each region ;
Color information extracting means for extracting color information of image data of each area for the plurality of areas ;
Shape information comparison means for comparing the shape information of the image data of each region and the input shape information;
Color information comparing means for comparing the color information of the image data of each region and the input color information;
An image retrieval apparatus comprising: retrieval means for retrieving image data from a database in correspondence with the comparison result by the shape information comparison means and the color information comparison means. The image search apparatus according to claim 6 , wherein the dividing unit performs division for each region of the same color after reducing the resolution of the image. A division step of dividing the image data into a plurality of regions based on colors;
For the plurality of regions, a shape information extraction step for extracting shape information of image data of each region ;
A color information extracting step for extracting color information of image data of each region for the plurality of regions ;
Shape information comparison step for comparing the shape information of the image data of each region and the input shape information;
A color information comparison step for comparing the color information of the image data of each region with the input color information;
A retrieval step for retrieving image data from a database in response to the comparison results of the shape information comparison step and the color information comparison step;
An image search method comprising: Table storage means for storing a table representing the correspondence between keywords and image feature amounts;
A dividing means for dividing the input image data into a plurality of regions;
For the plurality of regions, image conversion means for converting image data of each region into an image feature amount;
A keyword extracting apparatus comprising: a search unit that searches for a keyword corresponding to the image feature amount of the image data of each region with reference to the table. Search for related keyword each keyword searched by the searching means corresponding keyword extracting according to further comprising a keyword selection means for selecting a predetermined one from among the retrieved related keywords in claim 9, wherein Equipment . A division step of dividing the input image data into a plurality of regions;
For the plurality of regions, an image conversion step of converting image data of each region into an image feature amount;
A search step of searching for a keyword corresponding to the image feature amount of the image data of each region with reference to a table stored in a storage unit storing a table representing a correspondence relationship between the keyword and the image feature amount ;
A keyword extraction method characterized by including:

Images