JP4351620B2 - Image search system - Google Patents

Description

The present invention relates to an image search system for searching image data stored in a computer terminal.

  With the development of imaging devices such as digital cameras, digital data of captured images (hereinafter referred to as image data) is stored in computer terminals and posted on the Internet homepage or distributed to friends and acquaintances via e-mail. Things have been done.

  Unlike conventional cameras, an imaging apparatus such as a digital camera can easily delete, store, and correct captured image data, and the number of captured images (amount of image data) tends to increase. Therefore, the captured image data is generally stored and managed in a computer terminal. However, in order to search for necessary image data, a user checks each image data stored in the computer terminal one by one. In addition to searching, searching is also performed by using an image search system.

  In such an image search system, as a method of performing a search using a keyword such as a file name associated with image data, or as disclosed in Patent Document 1, Patent Document 2, and Non-Patent Document 1 below, reference image data There is known a method of searching for target image data (hereinafter referred to as target image data) from the similarity with (hereinafter referred to as reference image data).

JP 2000-285141 A Japanese Patent Laid-Open No. 7-219969 Ricoh Co., Ltd., “Ricoh Technology Front Line” [online], September 26, 2002, Internet <URL: http://www.ricoh.co.jp/tech/23/23_02.html>

  In the method of searching for target image data by keyword, it is necessary to set a keyword for each target image data. However, this keyword setting operation is very complicated. In addition, keyword setting tends to be subject to the keyword setter's subjectivity, and image search systems using keywords are difficult to obtain a certain level of search results.

  On the other hand, in the method of searching for target image data based on the similarity to the reference image data, there is no problem like a search using a keyword. However, if a large number of reference image data is not prepared in advance, the user's request The target image data to be searched cannot be searched. Even if a large number of reference image data can be prepared, a huge storage area is required as a storage area for the reference image data. In addition, it is practically impossible to prepare reference image data that assumes every aspect, and in particular, when searching for a very small number of special target image data from a large number of target image data, It is very difficult to prepare the reference image data itself.

  Therefore, in Patent Document 1, several images that are similar in part to the target image data are selected, feature amounts of similar portions are extracted for each image, and these feature amounts are weighted. A method of searching for target image data by searching is disclosed. However, when this method is used, since the reference image data remains divided into a plurality of images, there is a problem that it is difficult for the user to recognize the final image of the reference image data, and an accurate image search is performed. I can't.

  Non-Patent Document 1 discloses a method of using a handwritten image as reference image data. However, input of handwritten image samples has a large individual difference. In this case, the search results do not always match even if the same operator inputs to search for the same target image data. However, there is a problem that high-accuracy image retrieval cannot be performed.

  Further, in Patent Document 2, by performing natural language analysis on a sentence input in a natural language, an image component that is a set of pixels corresponding to the situation element is extracted, and each image is based on the extracted image component. A method of searching for target image data by searching for similarity to data is disclosed. In this case, it is possible to perform an image search with higher accuracy than in the case of Patent Document 1 or Non-Patent Document 1, but whether or not the extracted image component is similar to the target image data, or a natural language There is a problem that an accurate image component cannot always be extracted from the analysis, and the accuracy is not high. Further, since this method is based on a photographic image, the amount of data that needs to be stored in a computer terminal increases. Therefore, this method is not suitable for a personal computer terminal or the like having a limited storage capacity, and is actually difficult to implement.

  Therefore, in view of the above problems, the present inventor automatically generates reference image data based on image parts, and executes an image search process using the reference image data as a search condition. Invented an image search system that enables highly accurate image search.

The invention of claim 1 is an image search system for searching for target image data from a plurality of image data based on reference image data, and the image search system is an element constituting the reference image data. Storage means for storing at least image components, feature information including name information for the image components, front-rear distance information indicating the front-rear relationship, and shape information; and a plurality of the image data to be searched; and the image components A name input accepting unit that accepts an input of a name, a feature information extracting unit that extracts feature information having the name information corresponding to the accepted name from the storage unit, and the image based on the extracted feature information an image component extracting means for extracting a component from the storage means, Oite the combination of image parts the extracted, extracted context distance of each image component Based on the information, the image component of the object is divided into the image component of the background, and if there is a background image component, the value of the average inclination angle of the line connecting the point sequence in the background image component is used to extract the image component. One or two or more compositions for arranging the respective image parts are determined, and when there is no background image part, the number of the image parts is used to determine one or more compositions for arranging the extracted image parts. And determining the composition, and then combining the extracted image components in accordance with the determined composition and combining them to generate the reference image data for each combination, and the created Based on the selected reference image data, composite image selection receiving means for receiving an input indicating that one or more of the reference image data is selected by the user from the reference image data. Te, perform a search of the image data stored in the storage means, an image retrieval system having the image retrieval means for extracting the target image data based on the similarity between the reference image data.

As in the present invention, the reference image data is synthesized based on the image parts, the reference image data that the user thinks is most similar is selected from the reference image data, and the image data is searched based on the selected reference image data. By executing the above, it is possible to realize an image search system with higher accuracy than before. Here, the “name” is a name associated with the image part, and includes nouns (including proper nouns) and the like.
In addition, useless composition processing can be reduced by determining the composition as in the present invention before performing image composition.

  In the invention of claim 2, the feature information further includes surface information composed of color information or pattern information on the surface of the image component, and the image component is formed with an outline composed of point sequence information, The image composition means adds a color or a pattern based on the surface information to the inside or the outside of the point sequence information forming the outline of the image component, and then combines the extracted image components in combination. An image search system for creating the reference image data.

  The image component is preferably composed of a contour formed from point sequence information. This is because by using an image part composed of contours, the data amount of the image part can be greatly reduced as compared with the conventional case. If the data amount of image parts is reduced in this way, a large number of image parts can be stored even in the same data area, or the required data area can be greatly reduced even with the same number of image parts. As a result, a search system for image data can be realized even with a computer terminal whose specifications are not necessarily high, such as a computer terminal used in a general household.

According to a third aspect of the present invention, the image synthesizing unit selects, for each of the extracted image components, two image components having a large numerical value of the front-rear distance information from among the image components that have not been synthesized, and the selection The outside of the point sequence information of the image component having a small front-rear distance information is set as a mask area, and the image part having the larger front-rear distance information among the selected image parts is set in the mask area. Compositing and setting the inside of the point sequence information of the image component having the smaller context distance information among the selected image parts as a mask region, and the context distance information of the selected image components in the mask region Of the selected image components, and the image component having the larger front-rear distance information among the selected image components is used as a synthesized image component. Update the image component with the smaller context distance information with the synthesized image component, and repeat the process until all the image components have been combined with the image component with the smallest context distance information. An image search system that executes a composition process of the reference image data based thereon.

  As in the present invention, the image components may be synthesized in order from image components having large front-rear distance information (image components located far away).

According to a fourth aspect of the present invention, the storage unit further stores a combination history of reference image data created by combining with the image combining unit, and the image search system extracts a combination history stored in the storage unit. And further comprising a composition history management means, wherein the image composition means inquires of the composition history management means whether there is a composition history using the extracted image parts in the past. In the image search system, the reference image data is synthesized based on the synthesis history, and when there is no synthesis history, a synthesis process is executed based on the front-rear distance information.

  By using the synthesis history as in the present invention, reference image data based on image parts can be created at higher speed. As a result, the overall processing speed can be improved.

According to a fifth aspect of the present invention, there is provided an image component that is an element constituting reference image data, feature information including name information for the image component, contextual distance information indicating the context, and shape information, and a plurality of search targets. An image search program that is executed by a computer terminal having a storage device that stores at least the image data and an arithmetic processing device that executes the processing operation of the program. The image search program is read into the arithmetic processing device. It has a name input reception function, a feature information extraction function, an image component extraction function, an image composition function, a composite image selection reception function, and an image search function. The name input reception function is input from an input device of the computer terminal. The input of the name of the image component is received, and the feature information extraction function is configured to receive the name information corresponding to the received name. The image component extraction function extracts the image component from the storage device based on the extracted feature information, and the image composition function extracts the image component extracted from the storage device. Oite the combination, based on the extracted context distance information of each image part, divided into an image part of the image part and the background object, points in the image part of the background when there is an image part of the background column Using the value of the average inclination angle of the line connecting the two, determine one or more compositions to arrange the extracted image parts, if there is no background image parts, using the number of image parts, the extracts each image component obtained by determining one or more of the composition to be placed, after determining the composition, each image component obtained by said extraction by synthesizing a combination according to the composition described above decision, for each combination The reference image data is generated by combining the reference image data, and the composite image selection receiving function is configured so that one or more reference image data is selected by the user using the input device from the generated reference image data. Receiving the input, the image search function executes a search of the image data stored in the storage device based on the selected reference image data, and the target image based on a similarity to the reference image data An image search program for extracting data.

  The above-described image search system can obtain the same effect as a program when configured as in the present invention.

The invention of claim 6 records an image retrieval program according to claim 5, which is a recording medium recording the image search program.

  The image search program may be recorded on a recording medium such as a CD or a DVD.

By using the image retrieval system of the present invention, the user inputs the name of the image component recognized as a feature of the target image data, and synthesizes the reference image data based on the image component associated with the name. By selecting the reference image data that the user thinks is most similar to the target image data, and executing the image search process of the target image data based on the selected reference image data, the search accuracy can be improved as compared with the prior art. The target image data can be searched for higher.

  An example of the system configuration of the image search system 1 of the present invention will be described with reference to the system configuration diagram of FIG. The image search system 1 includes a name input reception unit 2, a feature information extraction unit 3, an image component extraction unit 4, an image composition unit 5, a composite image selection reception unit 6, an image search unit 7, a display unit 8, and a storage unit 9. Become.

  The image search system 1 is executed on the computer terminal 30. FIG. 16 shows an example of a hardware configuration of a general computer terminal 30. The computer terminal 30 includes an arithmetic processing unit 31 that executes arithmetic processing such as a CPU and a register, an input device 32 to the computer terminal 30 such as a keyboard and a mouse, a display device 33 that is a display, a memory, a cache, a hard disk, and other various types. It has a storage device 34 that stores data in a recording medium, and a communication device 35 that transmits / receives data to / from another computer terminal 30 via a network. Note that the hardware configuration of the computer terminal 30 shown here is merely an example, and other hardware configurations may be used.

  In the image search system 1, the name input accepting means 2, the feature information extracting means 3, the image component extracting means 4, the image synthesizing means 5, the composite image selection accepting means 6, the image search means 7, and the display means 8 are an arithmetic processing unit. The storage means 9 stores the data in the storage device 34.

  The storage unit 9 includes a feature information storage unit 10 that stores image component feature information, an image component storage unit 11 that stores image components, and surface information that stores surface information (color information and pattern information) of the image components. Storage means 12, composite image storage means 13 for storing reference image data created by combining image components, image data captured by an imaging device such as a digital camera, image data saved by downloading, and other searches The image data storage unit 14 stores target image data.

  Here, the feature information is information for designating the shape and arrangement of the image component. For example, name information indicating the type of the image component, point sequence information of the image component, or shape information that is an address for storing the information, an image Surface information, which is texture information (color information and pattern information) on the surface of the part, front-rear distance information obtained by quantifying the front-rear relation and distance between image parts, contour strength information indicating the degree of blurring of the contour of the image part, image part Arrangement information indicating an arrangement rule between them, hue correction information for correcting the hue of an image component, and the like. An example of the data structure of the feature information is shown in FIG.

  The name is information related to the image included in the target image data. For example, names used for landscapes and backgrounds such as “sea”, “mountain”, “sky”, “people”, “animals”, “cars”. There are names indicating objects such as. As shown in FIGS. 4A and 4B, the names preferably have a hierarchical structure.

  In FIG. 4A, “Scenery” as the highest name, “Mountain”, “Sea”, etc. as lower names, “Fuji”, “Iwayama”, “Snowy Mountain”, etc. as lower names of “Mountain”. , “Front”, “Top”, “Steep Mountain”, “Sloppy Mountain”, “Sudden Mountain” as the subordinate name of “Snow Mountain”. , There is a name of "gentle mountain".

  In FIG. 4B, “person” is the highest name, “male” and “woman” are the lower names, and “face”, “upper body”, “whole body”, “women” are the lower names of “male”. The lower names of “face” are “front”, “side”, “lower” of “upper body”, “front”, “side”, “black”, “white”, “ Subordinate names of “whole body” are “stop”, “walking”, and subordinate names of “walking” are “black clothes”, “white clothes”, and “blue clothes”.

  An image component is an image made up of elements (objects) constituting reference image data, which are stored in advance in the image component storage means 11 in order to synthesize reference image data. For example, examples of image parts include “mountain”, “sky”, “river”, “sea”, “tree”, “person”, “animal”, and the like. This image part is associated with name information in the feature information, and data of the image part corresponding to the name information is provided. For example, in the example of FIG. 4A, images of “Mount Fuji from the front”, “Mount Fuji from the top”, “Steep rocky mountain”, “Slow rocky mountain”, “Steep snowy mountain”, “Slow snowy mountain”, etc. The parts are stored in the image parts storage means 11, and in the example of FIG. 4 (b), “the front of the man's face”, “the side of the man's face”, “the front of the upper body of the man wearing black clothes "While wearing white clothes in front of the upper body of the man", "Side of the upper body of the man", "Stopping the whole body of the man", "Walking in black clothes of the whole body of the man", " Image parts such as “Walking in white clothes of the whole body” and “Walking in blue clothes of the whole body of the man” are stored in the image parts storage means 11.

  An example of the image component is shown in FIG. FIG. 5A is an example of an “empty” image part, FIG. 5B is an example of an “mountain” image part, and FIG. 5C is an example of an “tree” image part. .

  The image component may be point sequence information (data of only the contour of the image component) in order to reduce the data amount, but is not limited thereto. For example, when the image size of one image component is 640 pixels wide and 480 pixels high, when using the invention disclosed in Patent Document 1, the amount of data is 640 × 480 × 3 = 921,600 bytes. Requires a large amount of data. In contrast, an example of the point sequence information preferably used in the present invention is 7 items × 4 = 28 bytes in the data structure, the point sequence information is an average of 500 points (the contour of the image component is an average of 500 points). If necessary, 500 × 2 × 4 = 4,000 bytes of data is sufficient (x-axis and y-axis information is required to indicate one point of coordinates, and 4 bytes for each axis. Because information is expressed in units).

  FIG. 6 shows a conceptual diagram of the image component storage means 11. In FIG. 6, each piece of shape information is associated with the name information of the image component, and the information of the point sequence in this case is further associated. Accordingly, in the case of FIG. 6, there are three types of shape information corresponding to the name “sunset sky”, sora01, sora02, and sora03, and point sequence information in the shape information is stored.

  The front-rear distance information in the feature information is information indicating the order relationship of the overlapping between image parts. The order of superposition is preferably arranged on the front side as the shape becomes smaller. For example, if there are “bird” image parts and “sky” image parts, the distance information of the “bird” image parts is set to a smaller value than the distance information of the “sky” image parts. Has been. Further, the farther the background is like “sky”, the larger the front-rear distance information becomes, and the objects such as “trees” and “people” are closer, so the front-rear distance information becomes smaller.

  The name input accepting unit 2 is a unit that accepts input of names of image components constituting the reference image data from the input device 32 such as a keyboard and a mouse for creating the reference image data. It is preferable that a plurality of names can be input. The communication device 35 may accept input via a network such as the Internet.

  The feature information extraction unit 3 is a unit that extracts feature information corresponding to the name from the feature information storage unit 10 based on the name received by the name input reception unit 2.

  The image component extraction unit 4 is a unit that extracts the corresponding image component from the image component storage unit 11 based on the feature information extracted by the feature information extraction unit 3.

  The image synthesizing means 5 is means for synthesizing the image parts selected by the image parts extracting means 4 to create reference image data. The created reference image data is stored in the composite image storage unit 13. When the reference image data is synthesized and created, coloring or a pattern may be given inside or outside the contour formed by the point sequence information of the image parts.

  The display unit 8 is a unit that displays the reference image data generated by combining by the image combining unit 5 on the display device 33 that is a display of the computer terminal 30. When there are a plurality of reference image data created by the image synthesizing means 5, all of them may be displayed in a list display or a sequential display, or after ranking based on a predetermined reference, Various display forms can be taken, such as displaying sequentially from the highest ranking. In the present invention, the displayed reference image data may be corrected by the user. In addition to displaying on the display device 33 of the same computer terminal 30, the communication device 35 may display on the display device 33 of another computer terminal 30 via a network such as the Internet.

  The composite image selection accepting unit 6 generates reference image data that the user thinks is most similar to the target image from among the reference image data created by the image composition unit 5 and displayed on the display device 33 by the display unit 8. It is means for receiving information of selection by the input device 32 such as a mouse and transmitting the reference image data selected by the user to the image search means 7 based on the information.

  Based on the reference image data received from the composite image selection accepting unit 6, the image search unit 7 executes a similarity search with the image data to be searched stored in the image data storage unit 14 and obtains target image data. It is a means for searching and extracting. The target image data extracted here is transmitted to the display unit 8, and the display unit 8 displays it via the display device 33. When extracting the target image data, the image search means 7 may extract one image data having the highest similarity as the target image data, or a plurality of image data as the target image data in descending order of similarity. It may be extracted. In addition, the retrieval of the target image data based on the reference image data can take various conventional methods, and the above-mentioned Patent Document 1, Patent Document 2, and Non-Patent Document 1 can also be used. Furthermore, the present invention can be effectively applied to any conventional technique for retrieving similar images.

  Next, an example of the processing process of the image search system 1 of the present invention will be described with reference to the flowchart of FIG. 2 and the system configuration diagram of FIG.

  In the image search system 1, the search unit 8 displays the search screen via the display device 33 in order to search for target image data assumed as a search target by the user. An example of the search screen is shown in FIG. By inputting the name of the image component that the user recognizes as a feature of the target image data from the search screen, the image search system 1 combines the reference image data based on the image component associated with the name. The reference image data that the user considers to be most similar to the target image data is selected from the generated image data, and the image search process of the target image data is executed based on the reference image data.

  In this embodiment, a case where the user wants to search for “image with Mount Fuji and trees in the sunset sky” as target image data will be described. In such a case, it is considered that “sunset sky”, “Mt. Fuji”, and “tree” are characteristic as the target image data, and these are used as names to input in the name input field 20 of the search screen, such as a keyboard or a mouse. This is performed from the device 32 (S100). This input is received by the name input receiving means 2.

  It should be noted that this name input can be made by associating the input name with its feature information if it is input from the name input field 20 using a pull-down menu or the like. Input is preferred. However, in this case, since it is necessary to associate these in advance, other methods such as text input may be used.

  FIG. 8 shows a case where the user inputs “sunset sky”, “Mt. Fuji”, and “Thu” as names in the name input field 20 on the search screen.

  Based on the name received by the name input accepting means 2, the feature information extracting means 3 extracts feature information corresponding to the name from the feature information storage means 10 (S110). Here, since the names are “sunset sky”, “Mt. Fuji”, and “tree”, the feature information corresponding to these names is extracted from the feature information storage means 10.

  When the feature information storage unit 10 stores the feature information shown in FIG. 3, among the feature information corresponding to the name “sunset sky”, the shape information is stored in the image component storage unit 11 (FIG. 6) with the name “sunset sky”. ”Shape information (file names sora01, sora02, sora03), surface information“ sunset sky ”texture information, front-rear distance information“ 10000 ”, contour strength information “16”, “not specified” as the arrangement information (that is, all arrangement within the frame of the image), and “RED + 10” (adding 10 of red among the RGB values) are extracted as the hue correction information. Note that “RED + 10” is used as the hue correction information here, because in the situation of “sunset sky”, the overall color is often reddish. Therefore, by adding 10 to red among the RGB values, redness is obtained. It has an image part.

  Of the feature information corresponding to the name “Mt. Fuji”, the shape information is stored in the location (address or address) of the shape information (file names yama01, yama02, yama03) corresponding to the name “Mt. Fuji” in the image component storage means 11 (FIG. 6). File name, etc.), the texture information of “forest” as the surface information, “5000” as the front-rear distance information, “22” as the contour strength information, “bottom connection” as the arrangement information (that is, Mt. Fuji touches the lower side of the image) "RED + 10" is extracted as the hue correction information. Similarly, of the feature information corresponding to the name “tree”, the shape information is the location (address) of the shape information (file names ki01, ki02) corresponding to the name “tree” in the image component storage unit 11 (FIG. 6). And file name, etc.), “leaf” texture information as surface information, “10” as front-rear distance information, “500” as contour strength information, and “free placement” as placement information (that is, freely placed in the image) "RED + 10" is extracted as the hue correction information. Note that the hue correction information for “Mount Fuji” and “Thu” is “RED + 10” because the hue correction information for “Sunset Sky” is specified as “RED + 10”. It is because it is reflected in. Accordingly, when there is usually no hue correction information and hue correction information is specified for any of the selected image parts, the hue correction information is also reflected in these image parts. It can be a reddish image part as a whole, such as “Sunset sky”.

  After extracting the feature information in this way, the image component extraction unit 4 extracts the image component from the image component storage unit 11 based on the location of the shape information in the feature information (S120). In the above case, since the location of the shape information corresponding to the name “sunset sky” is sora01, sora02, and sora03 as file names, each file and its point sequence information are extracted from the image component storage means 11. Similarly, since the location of the shape information corresponding to the name “Mt. Fuji” is yama01, yama02, and yama03 as file names, each file and its point sequence information are extracted from the image component storage means 11 and correspond to the name “tree”. Since the location of the shape information is ki01, ki02 as file names, each file and its point sequence information are extracted from the image component storage means 11.

  After extracting the image parts in this way, the image composition means 5 performs an image composition process for creating reference image data based on the extracted image parts (S130). At this time, in order to color the image component composed of the point sequence information, each image component may be colored by referring to the surface information in the feature information. For example, since the surface information in the feature information of the image component with the name “Sunset Sky” is “Sunset Sky”, it is colored with the corresponding color, and the surface information in the feature information of the image component with the name “Mt. Fuji” Since it is “forest”, it is colored with a color corresponding to it, and since the surface information in the feature information of the image component with the name “tree” is “leaf”, it is colored with a color corresponding to it. Since the surface information is information on the color of the image component, a corresponding color code (for example, a color code specified by an RGB value) may be directly specified, or “sunset sky” as in this embodiment. ”,“ Forest ”,“ leaves ”, etc., may be designated. In this case, since a specific color is not specified in concepts such as “sunset sky”, “forest”, “leaf”, etc., by referring to the surface information storage means 12, “sunset sky”, “forest” The code of the color corresponding to “leaf” is extracted and colored with the color. FIG. 10 shows a conceptual diagram of the surface information storage means 12. Since the surface information in the feature information is “sunset sky” as described above, the corresponding color code (RGB value) is extracted from the surface information storage means 12 and colored.

  The surface information storage unit 12 may store a color pattern (such as a mesh) in addition to the color code. In this case, the color corresponding to the color code is colored on the image component by the coloring pattern.

  Further, the color of the colored image component may be further corrected based on the hue correction information in the feature information. In the case of FIG. 3, since “Sunset Sky”, “Mt. Fuji”, and “Thu” are all “RED + 10” as the hue correction information, the RGB color value of the image component is further changed to Red (red). Coloring can also be performed by adding 10 to the value of. By correcting the hue in this way, it can be produced as an image component in the sunset sky state.

  Further, based on the contour strength information in the feature information, a correction process for correcting the clarity (contour strength) of the contour of the image component may be performed. In the case of FIG. 3, the contour intensity of “sunset sky” is 16, the contour intensity of “Mt. Fuji” is 22, and the contour intensity of “tree” is 500. . The larger the contour strength value, the clearer the contour. In the above case, the “tree” contour is the clearest, followed by “Mt. Fuji” and finally “sunset sky”. .

  The image synthesizing unit 5 performs an image synthesizing process on the image component selected as described above (colored / patterned in some cases). In the image composition processing, the composition processing of all combinations of the image composition parts extracted in S120 is executed. In this embodiment, one combination of the name “sunset sky”, the name “Mt. Fuji”, and the name “tree” is used. The synthesis process will be described. This is because the combination processing can be realized with other combinations as well. Assume that one combination of image components described in this embodiment is shown in FIG. FIG. 11A shows one of the image parts with the name “Sunset Sky”, FIG. 11B shows one of the image parts with the name “Mt. Fuji”, and FIG. 11C shows the name “Thu”. One of the image parts.

  A processing process of the image composition processing will be described with reference to a flowchart of FIG. First, when composing each image component, composition is performed from a distant image component (an image component having a large numerical value of the front-rear distance information) based on the front-rear distance information of the feature information. Therefore, rearrangement is performed based on the front-rear distance information (S200). Then, since the distance information about the “sunset sky” is “10000”, the distance information about “Mt. Fuji” is “5000”, and the distance information about the “tree” is “10”, the “sunset sky” They are arranged from the distant view in the order of “Mt. Fuji” and “Thu”.

  Since all image parts are unprocessed at first (S210), in order to perform a process of combining “sunset sky” (image part 1) and “Mt. Fuji” (image part 2), “sunset sky” and “ The image component “Mt. Fuji” is selected (S220).

  Then, the outside of the contour information of the selected image component having the smaller front-rear distance information (here, “Mt. Fuji” (image component 2)) is set as a mask region (S230). The image component “Mt. Fuji” in this state is shown in FIG.

  Then, an image component (here, “sunset sky” (image component 1)) having large front-rear distance information among the selected image components is synthesized in the mask area (S240). Therefore, “sunset sky” is combined with “Mt. Fuji” set as the mask area. This state is shown in FIG.

  For the image component after combining in S240, the inside of the contour information of the image component (here, “Mt. Fuji” (image component 2)) having a small front-rear distance information among the selected image components is set as a mask region ( S250). This state is shown in FIG.

  Among the selected image components, an image component having a small front-rear distance information (here “Mt. Fuji” (image component 2)) is synthesized in the mask area (S260). Therefore, “Mt. Fuji” is combined with “Mt. Fuji” set as the mask area. This state is shown in FIG.

  Among the selected image components, an image component having a large contextual distance information (here, “sunset sky” (image component 1)) is set as synthesized (S270). Then, among the selected image components, an image component having a small front-rear distance information (here, “Mt. Fuji” (image component 2)) is updated with the synthesized image (S280). By performing such processing, the image component “sunset sky” is combined with the image component “Mt. Fuji”.

  Next, since “Mt. Fuji” and “Thu” remain as unprocessed image parts, S220 and subsequent steps are repeated.

  Then, here, two unprocessed image parts ("Mt. Fuji" and "Thu") are selected as two image parts with large distance information ("Mt. Fuji" and "Thu") because there are only two (S220). .

  Of the selected image components, the outside of the contour information of the image component (here “tree” (image component 3)) having a small context distance information is set as a mask region (S230). The image component “tree” in this state is shown in FIG.

  Then, an image component (here, “Mt. Fuji” (image component 2)) having a large front-rear distance information among the selected image components is synthesized in the mask area (S240). Therefore, “Mt. Fuji” is combined with “tree” set as the mask area. This state is shown in FIG.

  Among the selected image components, the inside of the contour information of the image component having a small front-rear distance information (here, “tree” (image component 3)) is set as a mask area for the image component after the synthesis in S240 ( S250). This state is shown in FIG.

  Of the selected image components, an image component having a small front-rear distance information (here, “tree” (image component 3)) is synthesized in the mask area (S260). Therefore, “tree” is combined with “tree” set as the mask area. This state is shown in FIG.

  Among the selected image components, an image component having a large front-rear distance information (here, “Mt. Fuji” (image component 2)) is set as synthesized (S270). Then, among the selected image components, an image component having a small contextual distance information (here “tree” (image component 3)) is updated with the synthesized image (S280). By performing such processing, the image component “Mt. Fuji” is combined with the image component “tree”.

After repeating the processing process from S220 to S280 in this way, the image component having the smallest contextual distance information (here “tree” (image component 3)) is replaced with another image component (here “sunset sky” (image The component 1) and “Mt. Fuji” (image component 2)) are synthesized. Therefore, the combined image component (image component “tree”) is stored in the combined image storage unit 13 as reference image data (S290). The reference image data to be stored is shown in FIG.

Reference image data combining processing using such image parts is performed for each combination of image parts. Further, even in the case of a combination of the same image parts, a plurality of arrangements can be made according to the arrangement position based on the arrangement information in the feature information and the composition of each image part. For example, in the above-described example, “tree” is arranged in the center. However, since the arrangement information of “tree” is “free arrangement”, it can be arranged freely in the image. Therefore, it is possible to create another reference image data in which “tree” is arranged on the right side or the left side. As described above, the arrangement of the image parts may be created in a plurality of ways including the arrangement information in the processing process from S220 to S280. Another example of the reference image data created in this way is shown in FIG.

  Further, when the image composition unit 5 performs composition of the image parts, after selecting the composition for arranging the image parts, the image composition may be composed according to the composition. As such a composition, there are five compositions as shown in FIG. In FIG. 13A, the image is divided into three parts vertically and horizontally, and each position has an object (foreground) (whether the object or the background is a background having a predetermined value or more based on the distance information, and an object having a predetermined value or less is the object The image component (image component “tree” in the above example) is arranged in a three-part composition. FIG. 13B is an oblique composition in which the image component of the object is arranged along the oblique line. c) is an S-shaped composition in which an image component of an object is arranged on an S-shape, FIG. 13D is a circle composition in which the image component of the object is arranged at the center of the image, and FIG. There is a triangular composition to be placed on the vertex.

  An example of a processing process for selecting a composition is shown in the flowchart of FIG. First, based on the front-rear distance information in the feature information of the image parts, each image part is divided into an object (foreground) and a background (distant view), and the quantity is confirmed (S300). As described above, the object and the background are determined by using an image part having front-to-back distance information of a predetermined value or more as a background (distant view) and an image part having front-to-back distance information of a predetermined value or less as an object (foreground). . For example, when the predetermined value is 500, in this embodiment, “sunset sky” and “Mt. Fuji” are the background (distant view), and “tree” is the object (foreground). The number of image parts is 3.

  Next, when the average inclination angle of the line connecting the point sequence (outline) is 40 degrees or less with respect to the background (distant view) image parts (here “sunset sky” and “Mt. Fuji”) (S310), It is determined that the composition to be adopted is a three-part composition, and it is decided that the image parts (here “tree”) of the object (foreground) are arranged in the order from the top to the bottom of the horizontal line of the three-part composition (S320). ).

  Also, when the average inclination angle of the line connecting the point sequence (outline) is 60 degrees or more with respect to the background (distant view) image parts (here “sunset sky” and “Mt. Fuji”), it is adopted (S330). It is determined that the composition to be divided is a three-part composition, and the image component (here “tree”) of the object (foreground) is arranged in the order of the left, right, and center of the vertical line of the three-part composition ( S340).

  Furthermore, when the average inclination angle of the line connecting the point sequence (contour) is 40 degrees or more and 60 degrees or less with respect to the background (distant view) image parts (here “sunset sky” and “Mt. Fuji”) ( S350), it is determined that the composition to be adopted is a diagonal composition, and the image component (here “tree”) of the object (foreground) is arranged from the upper right to the lower left (or from the upper left to the lower right) of the diagonal. Determine (S360).

  The average inclination angle described above can be calculated using various methods without any particular limitation. For example, there is a method of calculating an average inclination angle by calculating an average straight line by a least square method and obtaining an angle of the straight line. As another method, when there are n points in the point sequence, the angle of the first point-2 point line with respect to the first point (first point), and the first point-3 point line It is also possible to use a method of calculating the average inclination angle by taking the average of the angles of the lines connecting the points, such as the angles,.

  For the background (distant view) image parts (here “sunset sky” and “Mt. Fuji”), the width of the surface created by the dot sequence (outline) is 1/5 or less of the screen width at the top, and the dot sequence ( When the average inclination angle of the (contour) changes within 30 degrees to the left and right (S370), it is determined that the composition to be adopted is an S-shaped composition, and the image component of the object (foreground) (here "tree") Decides to arrange along the S-shape (S380).

  Furthermore, when the quantity of the image part of the object (foreground) is 1 and the composition of the image part of the background (distant view) has not been determined (S390), it is determined that the composition to be adopted is the Hinomaru composition, It is determined that the image component (here “tree”) in the (foreground) is arranged at the center (S400).

  When the number of image parts of the object (foreground) is 3 and the composition of the image parts of the background (distant view) has not been determined (S410), it is determined that the composition to be adopted is a triangular composition, and the object (foreground) ) Is determined to be arranged on a triangle (S420).

  When the number of image parts of the object (foreground) is 2 or less and the composition of the image parts of the background (distant view) is determined (S430), a three-division composition is added as the composition to be adopted, and the object (foreground) is added. ) Image parts (here “tree”) are determined to be arranged in the order of the lower right and the lower left of the intersections of the horizontal and vertical lines of the three-part composition (S440).

The image synthesizing unit 5 executes the processing process as described above before the image synthesizing process, thereby determining the composition for synthesizing the image parts and creating the reference image data, and based on the arrangement of the composition. It is possible to decide to combine the image parts. By providing a processing process that determines the composition in advance in this way, it is possible to perform synthesis with a certain degree of order rather than disorderly synthesis, and it is possible to reduce the time required for synthesis processing It becomes.

  In the processing process for determining the composition as described above, it is not necessary to determine one composition to be adopted, and a plurality of compositions can be adopted. Furthermore, the processing process of the composition selection process shown in FIG. 14 is an example, and the present invention is not limited to this, and other processing processes can be used.

Reference image data combining processing using such image parts is performed for each feature information and each composition (preferably a plurality of times), and these are used as reference image data. Then, the image synthesizing unit 5 extracts the reference image data stored in the synthesized image storage unit 13, and the display unit 8 displays the reference image data in the reference image data display column 21 of the search screen via the display device 33. In the search screens shown in FIGS. 7 and 8, the reference image data display column 21 can display seven images, but it may be more or less. Further, when displaying the reference image data, it may be a list display, or may be sequentially displayed from a higher priority by giving a predetermined ranking.

  From the reference image data displayed on the display device 33 in this way, the user selects, using the input device 32 such as a mouse or a keyboard, the reference image data considered to be most similar to the target image data to be searched for. (S140). Here, the information selected by the input device 32 is received by the composite image selection receiving means 6, and based on the information, the reference image data selected by the user is transmitted to the image search means 7.

  The image search means 7 executes a similarity search with the image data to be searched stored in the image data storage means 14 based on the reference image data selected by the user received from the composite image selection accepting means 6. Then, the target image data is searched and extracted (S150). The similarity search between the selected reference image data and the image data stored in the image data storage unit 14 can use various conventional methods. The target image data extracted here is transmitted to the display unit 8, and the display unit 8 displays it via the display device 33. At the time of this display, one image data with the highest similarity may be extracted as the target image data, or a plurality of image data may be extracted as the target image data in descending order of similarity.

  Through the processing process as described above, it is possible to search for target image data that is an object of image search. Unlike the conventional case, when the reference image data is synthesized based on the image parts, the user selects the reference image data that is most similar to the target image data, and the target image data is searched based on the selected reference image data. Can perform high image search. Further, point sequence information is used for image parts when combining and creating reference image data, and coloring is performed based on texture information (color information and pattern information) of feature information at the stage of combining reference image data. As a result, the data amount of each image component can be greatly reduced. As a result, even when the storage capacity is the same as the conventional one, a larger number of image parts can be stored in the image part storage means 11 than before, and the number of combinations when combining the reference image data is dramatically increased. It can be made. Thereby, since various reference image data can be provided to the user, the possibility that the reference image data is created even if it is special target image data is higher than before.

In the image search system 1 according to the first embodiment, when the reference image data combining process is executed, the combining process may be executed based on past combining history information. An example of the system configuration of the image search system 1 in this case is shown in FIG.

  In addition to the system configuration of the first embodiment, the image search system 1 further includes a synthesis history management unit 15 and a synthesis history storage unit 16.

The composition history management means 15 stores the composition history of the reference image data newly synthesized by the image composition means 5 in the composition history storage means 16, and the image composition means 5 newly composes the reference image data. Is executed, the past synthesis history stored in the synthesis history storage unit 16 is extracted and transmitted to the synthesis history management unit 15. Here, as information to be stored as the composition history, information such as which image component is used, the adopted composition, the arrangement position of the image component, coloring, contour strength, and the like may be stored as the composition history.

  The synthesis history storage unit 16 is a unit that stores the history of the reference image data created by the image synthesis unit 5.

  The image composition means 5 in this embodiment inquires of the composition history management means 15 whether or not there is a composition history using the image parts in the past before executing the composition processing of the reference image data from the image parts. . The composition history management means 15 that has received the inquiry collates whether or not the composition history storage means 16 has a composition history for the image component, and if so, transmits the composition history to the image composition means 5. If it does not exist, the fact that there is no composition history is transmitted to the image composition means 5. When transmitting this combination history, it is preferable to transmit what kind of combination, composition, etc. the combination processing has been performed, the positional relationship of each image component, coloring, and the like.

  The image composition means 5 receives from the composition history management means 15 an inquiry result as to whether or not there is a composition history using the component in the past, and if there is, composes the reference image data with the composition history. And it is good also as not performing a new synthetic | combination process like Example 1. FIG. On the other hand, if there is no composition history in the past as a result of the inquiry, image composition processing is executed as in the first embodiment. That is, the image composition process in S130 is executed.

  Thus, by providing the composition history management means 15 and the composition history storage means 16 in the image retrieval system 1, when searching for the same target image data, the time for the composition processing of the reference image data by the image parts is reduced. The display means 8 can immediately display the reference image data on the search screen via the display device 33.

  In the first and second embodiments, in addition to the input of search information by name, the input information by color can be corrected. That is, color correction may be performed for each image component or for the synthesized reference image data. In this case, for example, “RED + 10” is provided on the search screen with an input field for information on colors such as darker red than the image so that the input can be performed. In this way, it is possible to create reference image data that more appropriately represents the user's intention and feeling. In such a configuration, the hue correction information column need not be provided in the feature information.

  Furthermore, in addition to inputting search information by name, arbitrary image data or image data extracted from the image data can be used as a new image component. That is, image data input to the computer terminal 30 from the network via the storage device 34, the input device 32, or the communication device 35 provided in the computer terminal 30 or image data extracted from the image data is selected, and an image component is selected. May be stored in the image component storage means 11.

  Further, when extracting the image data from the above-described image data, a shape selection means (not shown) for recognizing and selecting the shape of the object existing in the image data is provided separately, for example, the image data When there are desired objects such as airplanes and automobiles, the shape selection means (not shown) automatically extracts the object from the image data by specifying them, and the extracted object The image data may be stored in the image component storage unit 11 as a new image component.

  An image search program for realizing the image search system 1 of the present invention in the computer terminal 30 can be realized in the same manner. Such an image search program includes a function (module or program; the same applies hereinafter) for realizing the name input receiving means 2, a function for realizing the feature information extracting means 3, a function for realizing the image component extracting means 4, and an image synthesizing means 5. At least a function for realizing the composite image selection receiving means 6 and a function for realizing the image search means 7, and in some cases, it may have information stored in the storage means 9. In the image search program, each function is read by the arithmetic processing unit 31 of the computer terminal 30, thereby realizing each unit on the computer terminal 30. Further, each storage means 9 stores the information in the storage device 34.

  The image search program is recorded on a recording medium such as a CD or a DVD, and is read into the computer terminal 30 to realize the image search system 1 or from a predetermined server via the network to the communication device 35. The image search system 1 may be realized by being downloaded to the computer terminal 30 and being read into the computer terminal 30.

  Each means in the present invention is only logically distinguished in function, and may be physically or practically the same area.

  In implementing the present invention, a storage medium storing a software program for realizing the functions of this embodiment is supplied to the system, and the computer of the system reads and executes the program stored in the storage medium. Of course.

  In this case, the program itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program naturally constitutes the present invention.

  As a storage medium for supplying the program, for example, a magnetic disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, a nonvolatile memory card, or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system running on the computer is one of the actual processes based on the instructions of the program. It goes without saying that the case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing thereof is also included.

Further, after the program read from the storage medium is written into the nonvolatile or volatile storage means 9 provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the program instructions Therefore, it is natural that the arithmetic processing device 31 provided in the function expansion board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing. is there.

By using the image retrieval system of the present invention, the user inputs the name of the image component recognized as a feature of the target image data, and synthesizes the reference image data based on the image component associated with the name. By selecting the reference image data that the user thinks is most similar to the target image data, and executing the image search process of the target image data based on the selected reference image data, the search accuracy can be improved as compared with the prior art. The target image data can be searched for higher.

It is a system configuration figure showing an example of the system configuration of the present invention. It is a flowchart which shows an example of the processing process of this invention. It is a figure which shows an example of the data structure of the feature information memorize | stored in a feature information storage means. It is a figure which shows an example of the hierarchical structure of a name. It is a figure which shows an example of an image component. It is a figure which shows an example of the data structure memorize | stored in an image components memory | storage means. It is a figure which shows an example of a search screen. It is a figure which shows an example of the search screen at the time of inputting a name into a search screen. It is a flowchart which shows an example of the process of an image composition process. It is a figure which shows an example of the data structure memorize | stored in a surface information storage means. It is a figure which shows an example of an image component. It is a figure which shows the transition of an image component in the case of performing the synthesis process of an image component. It is an example of composition. It is a flowchart which shows an example of the process of a composition selection process. It is a figure which shows an example of the reference | standard image data produced by synthesize | combining an image component. It is a figure which shows an example of the hardware constitutions of a computer terminal. It is a system block diagram which shows another example of the system configuration | structure of this invention.

Explanation of symbols

1: Image search system 2: Name input accepting means 3: Feature information extracting means 4: Image parts extracting means 5: Image composition means 6: Composite image selection accepting means 7: Image search means 8: Display means 9: Storage means 10: Feature information storage means 11: Image component storage means 12: Surface information storage means 13: Composite image storage means 14: Image data storage means 15: Composite history management means 16: Composite history storage means 20: Name input field 21: Reference image data Display field 30: Computer terminal 31: Arithmetic processing device 32: Input device 33: Display device 34: Storage device 35: Communication device

Claims (6)

An image search system for searching for target image data based on reference image data from a plurality of image data,
The image search system includes:
Image information that is an element constituting the reference image data, feature information including name information for the image component, front-rear distance information indicating the front-rear relationship, and shape information, and a plurality of the image data to be searched Storage means for storing at least;
Name input accepting means for accepting input of the name of the image component;
Feature information extracting means for extracting feature information having the name information corresponding to the accepted name from the storage means;
Based on the extracted feature information, an image component extraction unit that extracts the image component from the storage unit;
Oite the combination of image parts the extracted based on the extracted context distance information of each image part, divided into an image component of the object image parts and background, if there is an image part of the background of the background Using the value of the average inclination angle of the line connecting the point sequences in the image part, one or more compositions for arranging the extracted image parts are determined, and if there is no background image part, the quantity of the image parts Is used to determine one or more compositions in which the extracted image parts are to be arranged , and after determining the composition, the extracted image parts are combined and combined according to the determined composition. Image synthesis means for synthesizing and creating the reference image data every time;
Composite image selection receiving means for receiving an input indicating that one or more of the reference image data is selected by the user from the created reference image data;
Based on the selected reference image data, search for the image data stored in the storage means, and image search means for extracting the target image data based on the similarity with the reference image data;
An image search system comprising: The feature information further includes surface information composed of color information or pattern information on the surface of the image component,
The image component is formed with an outline composed of point sequence information,
The image composition means includes
By adding a color or pattern to the inside or outside of the point sequence information forming the outline of the image part based on the surface information, and then combining the extracted image parts, the reference image data is synthesized. create,
The image search system according to claim 1. The image composition means includes
For each of the extracted image parts, select two image parts having a large numerical value of the front-rear distance information from among the image parts that have not been combined,
The outside of the point sequence information of the image component having the small front-rear distance information among the selected image parts is set as a mask area, and the image having the larger front-rear distance information among the selected image parts is set in the mask area. Synthesize the parts,
The inside of the point sequence information of the image component having the small front-rear distance information among the selected image components is set as a mask area, and the image having the small front-rear distance information among the selected image parts is set in the mask area. Synthesize the parts,
Among the selected image parts, an image part having a large front-rear distance information is used as a synthesized image part,
Update the image component having a small front-rear distance information among the selected image components with the synthesized image component.
The process is repeated until all the image components are combined with the image component having the smallest contextual distance information.
The reference image data combining process based on the image parts is thereby performed.
The image search system according to claim 1, wherein the image search system is an image search system. The storage means further stores a synthesis history of the reference image data created by the image synthesis means,
The image search system includes:
Further comprising synthesis history management means for extracting a synthesis history stored in the storage means;
The image composition means includes
Inquires the composition history management means whether or not there is a composition history using the extracted image parts in the past, and if there is a composition history, creates the reference image data based on the composition history, When there is no composition history, a composition process based on the context distance information is performed.
The image search system according to claim 1, wherein the image search system is an image search system. Stores at least a plurality of pieces of image data to be searched, image information that is an element constituting reference image data, feature information including name information for the image parts, front-rear distance information indicating the front-rear relationship, and shape information An image search program to be executed by a computer terminal having a storage device and an arithmetic processing device that executes a processing operation of the program,
The image search program includes:
A name input reception function, a feature information extraction function, an image component extraction function, an image composition function, a composite image selection reception function, and an image search function, which are read into the arithmetic processing unit;
The name input reception function is:
Receiving an input of the name of the image component input from the input device of the computer terminal;
The feature information extraction function is:
Extracting feature information having the name information corresponding to the accepted name from the storage device;
The image component extraction function is:
Based on the extracted feature information, the image component is extracted from the storage device,
The image composition function is
Oite the combination of image parts the extracted based on the extracted context distance information of each image part, divided into an image component of the object image parts and background, if there is an image part of the background of the background Using the value of the average inclination angle of the line connecting the point sequences in the image part, one or more compositions for arranging the extracted image parts are determined, and if there is no background image part, the quantity of the image parts Determining one or two or more compositions in which the extracted image parts are arranged, determining the composition, and then combining the extracted image parts in accordance with the determined composition and combining them. Created by combining the reference image data every time,
The composite image selection receiving function is
From the created reference image data, one or two or more reference image data is accepted by the input device selected by the input device,
The image search function
Searching the image data stored in the storage device based on the selected reference image data, and extracting the target image data based on the similarity with the reference image data.
An image search program characterized by that.   6. A recording medium on which the image search program according to claim 5 is recorded.

Images