JPH0668171A - Picture database, picture database preparation method, picture retrieval method and device therefor - Google Patents

Abstract

PURPOSE:To display an object picture quickly for a retrieval request including layout information. CONSTITUTION:Retrieval layout information for a picture to be retrieved is inputted by using a picture database storing a picture, similarity of an object at each area of the picture and the picture layout information. Based on the inputted retrieval layout information, the picture desired to be retrieved among sets of the picture layout information of the picture database is retrieved, the reliability of layout of the retrieval picture is calculated based on the degree of similarity of the object stored together with the retrieved picture and the information is displayed in the order of higher reliability of the calculated layout.

Description

Detailed Description of the Invention

[0001]

This invention relates to an image database,
More specifically, the present invention relates to an image database creating method, an image searching method, and these devices, and more specifically, an image database creating method including layout information, an created image database, and an image database including the layout information to create an image. The present invention relates to a search method and an image search device.

[0002]

2. Description of the Related Art Conventionally, there has been proposed a method for quickly searching for a target image from an image database. As one of such methods, there is a method in which a keyword for a person to search for an image is added to each image. However, it is difficult to assign an accurate keyword that can efficiently search images, and even if a keyword is added, the subjectivity of the person who assigns the keyword is likely to enter, so a keyword that is out of focus for a third party who searches the image. In many cases, the target image was not found quickly.

Further, in recent years, a worker visually checks layout information of an object included in an image for each image, registers the layout information in an image database by an input means such as a keyboard, and layout information of an image to be searched. An image search method has been proposed in which is used as a keyword for image search so that image search can be performed quickly.

[0004]

However, even in the image retrieval method based on the layout information described above, it is still the case that the operator determines the layout of the object for each image and creates the layout information. There was a problem that required great effort. In addition, when there are many images registered in the image database, there is a problem that it takes time to find a target image from images having the same layout information.

Further, in the conventional image retrieval method using layout information, since the layout information is created based on the manual describing the layout relationship of individual objects, the shape is different if the keyword of the retrieval keyword is different. There was a problem that images could not be searched even for objects that were similar to.

[0006]

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is a method of creating an image database capable of quickly displaying a desired image in response to a search request including layout information and improving the efficiency of image search. , An image database creation device, an image database, an image search method, and an image search device.

[0007]

According to a first aspect of the present invention, there is provided a method for creating an image database, wherein a template of an object and basic layout information between the objects are set and a reference is set for an original image. The target object is extracted and the position of the target object is detected by performing the rough recognition using the template, the similarity between the extracted target object and the template is calculated, and the rough recognition is performed by the rough recognition verification. In addition to deleting the incorrect extraction of the target object, delete the target object that has been erroneously extracted based on the basic layout information and the layout of the target object, and create the layout information of the original image. The degree and the image layout information are stored together with the original image.

The image database according to claim 2 for achieving the above object stores the original image, the similarity between the extracted object and the template, and the image layout information. According to another aspect of the present invention, there is provided an image database creating apparatus which uses initial condition setting means for setting a template of an object and basic layout information between the objects, and a template serving as a reference for an original image. By performing rough recognition to extract candidates for the target object and detecting the position of the target object, similarity calculation means for calculating the similarity between the extracted target object and the template, and rough recognition verification. First recognition verification means for deleting erroneous extraction of the object extracted by the rough recognition means, and second recognition verification for deleting the erroneously extracted object based on the basic layout information and the layout of the object. Means, an image layout creating means for creating layout information of the original image based on the outputs of the first recognition verifying means and the second recognition verifying means, and each target. And an image database for storing together with the original image and the similarity with the image layout information with the template and.

In order to achieve the above object, the image retrieval method of claim 4 uses an image database that stores the original image, the similarity between the extracted object and the template, and the image layout information. , Input the search layout information of the image to be searched, search for the image to be searched based on the input search layout information and the image layout information of the image database, and resemble the objects included in the searched image. The reliability of the layout of the search image is calculated based on the degree, and the display of the search image is controlled based on the calculated reliability of the layout.

According to the image retrieval method of the fifth aspect, the reliability of the layout is calculated by multiplying the similarities of all the objects included in the retrieved image. The image search device according to claim 6 for achieving the above object, an image storage unit for storing an original image, a similarity storage unit for storing the similarity between the extracted object and the template, and image layout information. A search layout information input unit for inputting search layout information of an image to be searched by using an image database having an image layout information storage unit for storing the search layout information and the image layout information storage unit. A search means for searching for an image to be searched for based on the image layout information; a reliability calculation means for calculating the reliability of the layout of the search image based on the similarity of objects included in the searched image; Image display control means for controlling the display of the search image based on the reliability of the created layout.

In the image retrieving apparatus according to the seventh aspect, the reliability calculating means includes a multiplying means for multiplying the similarities of all objects included in the retrieved image.

[0012]

According to the image database creating method of claim 1, the template of the object and the object most suitable for the image database to be created are set by setting the template of the object and the basic layout information between the objects. Basic layout information between can be set. Further, rough recognition of the original image can be performed by performing rough recognition on the original image using a template serving as a reference, extracting candidate objects and detecting the position of the target object. Further, by calculating the degree of similarity between the extracted target object and the template, it is possible to have information indicating the degree of matching between the target object and the template. In addition, by deleting the incorrect extraction of the object extracted by the rough recognition by the rough recognition verification, and by deleting the wrongly extracted target based on the basic layout information and the layout of the target, the reliability can be improved. It is possible to create high image layout information. By storing the similarity between each object and the template and the image layout information together with the original image, it is possible to create an image database having a data structure that makes it easy to search for the target image.

In the image database according to the second aspect, since the original image, the similarity between the extracted object and the template, and the image layout information are included, the image search can be efficiently performed. . According to the image database creating apparatus of claim 3, the initial condition setting means sets the template of the object and the basic layout information between the objects, and the outline recognition means uses the template serving as a reference for the original image. By performing the recognition, the candidate of the target object is extracted and the position of the target object is detected. Then, the similarity calculating means calculates the similarity between the extracted object and the template. Further, the first recognition verification means deletes an erroneous extraction of the object extracted by the rough recognition means by the rough recognition verification, and the second recognition verification means makes a mistake based on the basic layout information and the layout of the object. Delete the extracted object. Then, the image layout creating means creates the layout information of the original image based on the outputs of the first recognition verifying means and the second recognition verifying means. Then, by storing the similarity between each object and the template and the image layout information in the image database together with the original image, it is possible to create an image database that facilitates image search.

According to the image search method of claim 4, the reliability of the image layout specified by the search layout information is calculated based on the similarity of the object of the search image, and the image is searched by the search layout information. The reliability of the captured image can be calculated. Then, by controlling the display of the search image based on the calculated reliability of the layout, it is possible to search for the image according to the search request of the operator.

According to the image retrieval method of the fifth aspect, the reliability of the layout is calculated by multiplying the similarities of all the objects included in the retrieved image. Therefore, the reliability can be calculated quickly by a simple calculation called multiplication. In addition, the search accuracy can be improved by using the reliability of the entire image. According to the image search device of claim 6, an image storage unit that stores the original image, a similarity storage unit that stores the similarity between the extracted object and the template, and image layout information that stores the image layout information. An image database having a storage unit is used, and the search layout information input unit uses the input search layout information to search for an image to be searched based on the image layout information in the image database. The reliability calculation means calculates the layout reliability of the search image based on the similarity of the objects included in the searched image, and the image display control means displays the search image based on the calculated reliability of the layout. To control. Therefore, the image display can be changed according to the reliability of the layout of the searched image in consideration of the search layout information, and the image to be searched can be searched quickly.

According to the image retrieval apparatus of claim 7, the multiplication means calculates the reliability of the retrieved image by multiplying the similarities of all the objects included in the retrieved image, so that the reliability can be calculated quickly. In addition, the search accuracy can be improved by using the reliability of the entire image.

[0017]

Embodiments will be described in detail below with reference to the accompanying drawings showing embodiments. FIG. 1 is a flow chart showing a first embodiment of the image database creating method of the present invention. In this image database creating method, a template of an object to be processed by the image database created in step SP1 and basic layout information between the objects are set, and in step SP2, the images forming the image database are processed by an image scanner or the like. It is used, or is read by transferring an image from an image file or the like, the image read in step SP3 is divided into regions having the same characteristics, and a region serving as a reference template is used for each divided region in step SP4. Each time, matching is performed at the pixel data level as rough recognition, and the candidate of the target object is extracted and the position of the target object is detected. A normal correlation method, a vector correlation method, etc. suitable for position detection handling a multi-valued image can be used for this matching.

Next, in step SP5, the degree of similarity between the object extracted at the pixel data level and the template is calculated based on the correlation coefficient of the above correlation method, and in step SP6 feature point matching for rough recognition verification is performed. Erroneous extraction of the object extracted at the performed pixel data level is deleted, and in step SP7, step SP
In step 1, based on the basic layout information set between the objects, layout verification of the objects is performed to delete the incorrectly extracted objects, and in step SP8 layout information of the image after layout verification is created. , Step S
In P9, the similarity between the object and the template and the image layout information are stored together with the image for each image, and it is determined in step SP10 whether or not all the image storage processes have been completed, and all image storage processes have been completed. If it is determined that, a series of processing ends. Also, step SP1
If it is determined that the storage processing of all the images is not completed at 0, the processing from step SP2 to step SP9 is repeated until it is determined that all the storage processing of the image to be processed is completed, and at step SP10, the image is processed. When it is determined that all the storage processing of (3) are completed, the series of processing is completed.

2, 3 and 4 are views for explaining an embodiment of the image database creating method of the present invention. FIG. 2 shows that the object includes a table, a chair, a cup and a vase. , There is a cup and a vase on the table, and there is a chair next to the table. Predefine this basic layout information,
Templates of objects such as a table, a chair, a vase, and a cup are defined as in Figure 1, Figure 2, Figure 3, and Figure 4 of FIG. 3, respectively. Then, in order to create the image database, consider a case where the processed image is divided into a plurality of regions (region A, region B, region C, region D, region E) having the same characteristics as shown in FIG. .

By matching pixel data levels,
When extracting the object and determining the position of the object, if the table template, the chair template, the vase template, and the cup template are respectively matched to the areas A, ... The correlation coefficient for an object can be calculated, and this is defined as the similarity of each template for each region. Then, in each of the objects, those having a clearly low correlation coefficient are deleted from the area specifying object, and the position of each object is specified. Next, feature point matching is performed to delete the object candidates that have been erroneously extracted for each region, and
Layout verification is performed on the basis of the basic layout information shown in FIG. 4, and the candidate of the object of the impossible layout is deleted to determine the layout information of the processed image shown in FIG.
Image database, the similarity, image layout information,
And the original image.

This processing will be described in more detail by taking the case of the area B of FIG. 4 as an example. By the matching at the first pixel data level, the similarity of each template to the area B is temporarily changed to the numerical value shown in FIG. It is assumed that it is calculated.
And the shape is remarkably different due to feature point matching,
A table having a low degree of similarity is removed from the candidates of region B, and it is unlikely that there is a chair on the table according to the layout verification. Therefore, the chair is removed from the candidates of the object, and finally a candidate of region B is selected. The vase and cup will remain. In this case, the area B has the similarity to the vase template and the similarity to the cup template as data. As described above, the image database has the image itself, the layout information of the object narrowed down by the layout verification in the processed image shown in FIG. 4, and the similarity between the finally remaining object and the template for each area. It is created in the open state.

[0022]

[Embodiment 2] FIG. 5 is a flow chart showing an embodiment of the image retrieval method of the present invention. In this image search method, images are searched using the image database created by the image database creation method described in the first embodiment. First, in step SP1, the searcher inputs the search layout information of an image to be searched by using an input means such as a keyboard, and in step SP2, the search layout information input from the image database is used to generate the image layout information. The image to be searched for is searched based on, and only the similarity of the object included in the search layout information is read from the similarities of the objects stored corresponding to the image searched in step SP3.
The reliability of the layout of the search image is calculated from the similarity of the object read in step SP4, and step S
In P5, the search image is displayed based on the reliability of the layout, and the series of operations is completed.

According to this embodiment, the layout reliability of the image searched by the input search layout information is calculated based on the similarity of the object, and the search image is displayed based on the calculated reliability. Therefore, it is possible to quickly find the image that the searcher is looking for with a high probability.

[0024]

Third Embodiment FIG. 6 is a flow chart showing another embodiment of the image retrieval method of the present invention. This embodiment is a method showing one form of the second embodiment, and when the reliability of the input layout information is calculated from the similarity of the objects, the reliability is obtained by multiplying the similarity of each object. It is characterized by points and points in which search images are rearranged and displayed in the descending order of reliability.

That is, first, in step SP1, the searcher inputs the search layout information of the image to be searched by the input means such as a keyboard, and in step SP2, the search layout information input from the image database is used. Then, the image to be searched for is searched based on the image layout information, and the similarity of the object included in the search layout information is selected from among the similarities of the objects stored corresponding to the image searched in step SP3. Only the degree is read, and the layout reliability of the search image is calculated by multiplication from the similarity of the object read in step SP4, and the images searched in order of the layout reliability are rearranged in step SP5.
In step 6, the search images are displayed in the order in which they are sorted in order of increasing reliability, and the series of operations is completed.

According to this embodiment, in addition to the effect of the second embodiment, the reliability of the search layout information with respect to the search image can be quickly calculated by a simple operation such as multiplication, and the object in the search layout information can be calculated. The accuracy of the search can be improved by multiplying the similarities of all the objects included in the search image. In addition, since the display order of the search images is changed based on the reliability of the layout, there is an advantage that the searcher can quickly find the image searched by the searcher by merely looking at the images in order.

FIG. 7 is a diagram for explaining the image retrieval method of this embodiment, and is a diagram showing a method for retrieving an image using the image database 39 created as described above. In FIG. 7, reference numeral 10 is a keyboard for inputting search layout information, and reference numeral 11 is a CR as display means.
T. First, as shown in FIG. 7 (A), the keyboard 1
When search layout information 12 such as “there is a vase on the table” is input as search information from 0, a large number of images in the image database 39 are searched for an image with a vase on the table in layout. To be done. Then, as shown in FIG. 7B, when three images 14 having a vase on the table are searched, the similarity of the object determined to be the table and the similarity of the object determined to be the vase. The reliability of the input search layout information is calculated based on the degree. Image 1 has a reliability of 90%, image 2 has a reliability of 80%, and image 3 has a reliability of 50%.
If so, the images are displayed in the descending order of reliability, and the search efficiency is improved.

That is, since the previously created image database holds the degree of coincidence of the divided areas with respect to a specific object by the information called similarity, the image layout information configured as an aggregate of objects. It is possible to calculate the reliability of the. As for the reliability of the image layout information, if the similarity of the table included in the image is RA and the similarity of the vase is RB, consider the probability that the table is included in the image as RA and the probability that the vase is included as RB. In this case, the probability that both are included at the same time is given by the product of the respective occurrence probabilities.
It is defined as × RB. When this method is adopted, the product of the similarities is large when both the similarities of the two objects are large, so that the reliability is high. Further, when one of the similarities is small, the product of the similarities is small, and thus the reliability is low. In this embodiment, by such a method, the reliability of the image layout information is defined, the presentation order of the image search is prioritized, and a correct search request is quickly searched.
We are able to respond to incorrect search requests.

That is, as described with reference to FIGS. 3 and 4, in the region B shown in FIG. 4, the vase and the cup are listed as candidates together with the degree of similarity, so that a search for "the vase is on the table" is performed. , It is possible to retrieve the cup having the second lowest similarity in the region B, and as a result, from the search layout information “the vase is on the table”, the image 15 “the cup is on the table” is displayed. (Fig. 7
The search (see (B)) can be performed, and the range of the search can be automatically widened. In particular, when there are many candidates for the object that are listed in a certain area by the layout verification, an efficient image search can be performed by sequentially searching for images including the object in descending order of similarity.

In the conventional search using the keyword, if the word expressions are different, the search cannot be performed. In this embodiment, the similar objects are automatically grouped, and the similar objects among the objects are classified in the grouping. Since the degree can be retained, it has a great effect on improving the accuracy and speed of image retrieval.

[0031]

[Fourth Embodiment] FIG. 8 is a schematic block diagram showing an embodiment of an image database creating apparatus according to the present invention. This image database creation device includes a template holding unit 30 that holds a template of an object that the image database intends to handle, and a basic layout setting holding unit that sets and holds basic layout information between the objects that the image database intends to handle. 31 and an image holding unit 32 for holding an image by reading the image using an image scanner or reading the image from an existing image database
And an area dividing unit 33 that divides the area in the image held in the image holding unit 32 into areas using a feature value such as the color of the area.
And using the template of the object held in the template holding unit 30 for each divided area, matching is performed at the pixel data level as rough recognition for each area, and a selection candidate for each object is determined for each area. Then, the pixel data level matching unit 34 that holds the candidate of the object and specifies the position, and the similarity that calculates and holds the similarity of each object at the pixel data level based on the correlation coefficient such as the normal correlation method. For the candidates selected for each area by the degree calculation holding unit 35 and the pixel data level matching unit 34, the feature point matching is performed as the rough recognition verification, and the erroneous target among the object candidates extracted at the pixel data level. Based on the defined basic layout information, the feature point matching unit 36 that deletes The layout verification of the target image is performed and the layout verification unit 37 that deletes the erroneously extracted target is specified, and the layout of the processed image is specified by the target specified by the layout verification unit 37, and the data is stored in the image database 39. An image layout determination unit 38 to output, image layout information, a similarity between an object and a template, an image database 39 as a storage unit that stores data of the image itself held in the image holding unit 32, and an operator. In response to the instruction, the image holding unit 32 has an image reading unit 40 that sequentially reads images.

The image database 39 specifically includes an image layout storage unit 39a that stores image layout information for each image, and a similarity storage unit 39b that stores the similarity between the object and the template for each image. And an image storage unit 3 for storing the data of the image itself held in the image holding unit
9c and. The operation of the image database creating apparatus having the above configuration is as follows.

When creating an image database, a worker anticipates an object included in an image to be handled, creates a template of the object, or imports it from another image database, and stores the object in the template holding unit 30. The template is stored, basic layout information corresponding to an image to be handled is set, and the basic layout information setting storage unit 31 stores the template. From this state, the image reading unit 40
Is operated to output one screen of the image forming the image database to the image holding unit 32, and the image holding unit 32 outputs the image to the area dividing unit 33. The region dividing unit 33 divides the image to be processed by a predetermined region dividing process into a plurality of regions and outputs the divided regions to the pixel data level matching unit 34.

The pixel data level matching unit 34 performs matching at the pixel data level for each area using the template of the object held in the template holding unit 30 and determines a selection candidate for each object for each area.
The target object is held for each area and the position of the target object is specified. The similarity calculation holding unit 35 calculates the similarity of each object for each divided region based on the output of the pixel data level matching unit 34 based on a correlation coefficient such as a normal correlation method, It is output to the feature point matching unit 36 as data of each area.

The candidates for the object determined for each area by the pixel data level matching unit 34 are further narrowed down by the feature point matching unit 36, and the positional information of the object and the similarity for each area are held. It is output to the layout verification unit 37, and the layout verification unit 37 performs layout verification with the basic layout held in the basic layout setting holding unit 31, and deletes the erroneously extracted target object.

When the layout verification unit 37 deletes the erroneously extracted object, the image layout determination unit 38
The image layout determination unit 38 determines the layout of the image to be processed from the output of the layout verification unit 37 and outputs each information to the image database 39. That is, the image layout information from the image layout determination unit 38 is stored in the image layout storage unit 39 a of the image database 39. Also, image database 3
The image layout determination unit 38 is stored in the similarity storage unit 39b of No. 9.
Since the output of the similarity is from, the similarity of the finally remaining candidate object is stored for each area. Further, the processed image data of the image holding unit 32 is stored in the image storage unit 39c.

According to this image database creating apparatus,
Since the image database is created by automatically calculating the image layout information and the degree of similarity from the processed image, the image database creation work can be performed efficiently.

[0038]

Fifth Embodiment FIG. 9 is a diagram for schematically explaining the data structure of the image database created by the image database creating apparatus according to the seventh embodiment. The image database contains images 1, ... For each image n, the image layout is held together with the image data, and the image layout and the similarity of the final candidate for the object to the template are held in correspondence with each other. With such a data structure, image retrieval can be efficiently performed as described later.

[0039]

[Sixth Embodiment] FIG. 10 is a schematic block diagram showing an embodiment of the image retrieval apparatus of the present invention. This image search device is a device for searching for an image using the image database 39 created by the image database creation device shown in FIG. 8, and search layout information such as a keyboard for allowing the searcher to input search layout information. Input section 50
Based on the layout information of the target object and the target object extracted by the extraction unit 51, the target information and the layout information of the target object are extracted so as to facilitate the search from the input search layout information. , Image database 39
Layout information retrieval unit 52 for retrieving the layout for each image stored in the image layout storage unit 39a.
Then, from among the similarities of the objects included in the image searched based on the search layout information, only the similarity of the objects included in the search layout information is read from the similarity storage unit 39b, and the layout of the search image is calculated. An image layout reliability calculation unit 53 that calculates reliability, and a priority determination unit 54 that determines image display priority according to the layout reliability calculated by the image layout reliability calculation unit 53.
And an image display unit 55 that displays the images in the image storage unit 39c based on the priority order determined by the priority order determination unit 54.

The operation of the image retrieving apparatus having the above configuration is as follows. First, when a searcher inputs search layout information including layout information into the search layout information input unit 50, for example, “there is a vase on the table.”, The search layout information input by the extraction unit 51 is used to search for the target object. "Table" and "vase" as the layout information and "vase on the table" as layout information are extracted, and the image layout storage unit 3 is used for the n images stored in the image database by the search layout information search unit 52.
9a is searched for an image including a predetermined object (“table”, “vase”) and having a predetermined layout (“vase on table”).

Next, the image layout reliability calculation unit 53 calculates the reliability of the layout of the search image with reference to the similarity information of the images searched satisfying the conditions, and the priority order determination unit 54 performs the layout. The image display priority is changed according to the reliability of the image, and the image is displayed on the image display unit 55 in a form that the searcher can easily search. As such a display form, there are a form in which images are displayed in descending order of reliability, and a form in which images with a reliability not less than a predetermined value are collectively displayed.

According to this image search apparatus, the image search apparatus automatically calculates the reliability of the layout from the image database based on the search layout information inputted by the searcher, and displays the image based on the reliability. Therefore, you can quickly search for the desired image. Further, since the similarity of the object is stored, it is possible to search and display the candidates of the object in order of similarity even if the search information is ambiguous, and it is possible to broaden the search range. Become.

[0043]

[Embodiment 7] FIG. 11 is a block diagram showing another embodiment of the image retrieving apparatus of the present invention, and is a diagram showing an embodiment of the image retrieving apparatus. The characteristic point of this embodiment is that the image layout reliability calculation unit 53 in the image search apparatus.
However, in calculating the reliability of the layout from the similarity of each object stored in the similarity storage unit 39a, the first holding unit 53a and the second holding unit 5 that hold the similarity of each object.
3b and a multiplication unit 53c that sequentially and cumulatively multiplies the degrees of similarity of the respective objects held in the first holding unit 53a and the second holding unit 53b, respectively.

The similarity between the object A and the object B held in the first holding unit 53a and the second holding unit 53b is multiplied by the multiplying unit 53c, and the multiplication result is the first holding unit 5.
It is returned to 3a again and multiplied by the object C held in the second holding unit 53b. In this way, the similarities of all the objects included in the search image are multiplied, and the reliability of the layout of the search image is calculated.

[0045]

[Embodiment 8] FIG. 12 is a block diagram showing an integrated image search apparatus for efficiently carrying out the image database, the image database creating apparatus, and the image search apparatus of the present invention with a plurality of processors. When searching for an image whose memory is ambiguous, it is desirable that the image can be searched for by an expression such as "an image in which the object B is drawn on the right side of the object A". A huge amount of processing time is required to perform such a search with one processor. Also, when using multiple processors, the algorithm for performing parallel processing is complicated,
It was difficult to realize. In this embodiment, the purpose is to efficiently describe the layout (positional relationship) of the object drawn in the image with a plurality of processors and to provide an incomplete image representation such as the layout of some objects included in the image. It has a function of searching for an image.

This image retrieval apparatus stores an image input section 70 for inputting a processed image, a preprocessing section 71 for improving the image quality of the input image, rough recognition data, rough recognition verification data, and layout verification data. The rough recognition unit that detects the position of the object one after another by comparing the input image with the data stored in the element database 72 and the data stored in the element database, and stores the detection result in the detection result memory 73. 74, a rough recognition result verification unit 75 that verifies the rough recognition result based on the information stored in the detection result memory 73, and an object detected by the processes of the rough recognition unit 74 and the recognition result verification unit 75. Whether the detection result memory 73 that stores information and the detection result memory 73 are used to check the position of the target object and whether the relative layout is consistent The layout description section 76 to verify,
An image database 77 that stores the similarity of the objects included in the images, the image layout information, and the original image for each image, and an image that matches the search layout information input by the user are searched from the image database 77. It has an image search unit 78.

In the element database 72, data for rough recognition in the rough recognition unit 74, data for verifying the recognition result in the recognition result verification unit 75, and data for layout verification in the layout description unit 76. Is stored. As a specific example of the stored data, the outline information of the object is stored when the rough recognition is performed by the vector correlation method, and the raw image of the object is stored when the rough recognition is performed by the normal correlation method. ing. In addition, the color distribution and feature point information of the object are stored for verification of the recognition result. If there is a layout that cannot occur for layout verification, that information is stored. The template is managed according to the type of input image.

The operation of the image retrieving apparatus having the above structure is as follows. First, when a processed image is input from the image input unit 70, the image quality of the input image is improved in the preprocessing unit 71. Then, the rough recognition unit 74 detects where each object of the input image is included in the image. A correlation method is used as a method of detecting where an object is included in an image. The correlation method is a method for searching the image for a portion that best matches the model (template) of the object created in advance. in this way,
A vector correlation method or a normal correlation method is applied to the image to detect a portion that matches the template well. As shown in FIG. 13, the processing result stores the coordinates where the template 79 matches as the position of the object, and stores the correlation coefficient indicating how much the template 79 matches the similarity as the similarity indicating the degree of matching of the rough recognition. The correlation coefficient takes a value of 0.0 to 1.0, and takes a value close to 1.0 when the degree of coincidence is good. Also,
Since the image may include a plurality of the same objects, in the processing, as a result of scanning the entire image, all the positions where the peak 80 of the correlation coefficient appears are detected as the objects. As a result, it is possible to prevent omission of detection of the object. Note that the object that is erroneously detected is deleted in a later process. Therefore, the detection result memory 73 stores the name and position of the object and the correlation coefficient. In a general form, the template name of the object is: (coincidence position, correlation coefficient), and R is the correlation coefficient. Template A: (x1, y1, R1), (x2, y
2, R2) Template B: (x3, y3, R3).

Although this apparatus has a large number of templates, it is also possible to adopt a method of selecting a template to be used according to the type of input image in order to shorten the processing time. For example, when the indoor recorded picture is an input image, "indoor recorded picture" is specified when the image is input, and a template group for indoor recorded pictures is used. Although the objects that can be detected are limited when the template is limited as described above, the names of the objects used for the image search are often limited, and thus a sufficient function can be achieved. Further, if the objects that can be specified at the time of image search are limited to some extent, the specification method and the selection range become clear, and it becomes easier for the operator to use. In this way, the name and position of the target object can be known from the result of the rough recognition unit 74, and the range in which the target object exists can be known from the size of the template used.

Next, the recognition result verifying unit 75 sets a processing area from the position and existence range of the object, and performs accurate image recognition processing on the processing area. As an example of the image recognition processing used here, color distribution of an object, feature point matching, or the like is performed. As a result of the verification, if it is determined that the rough recognition is incorrect, a flag indicating that the data is invalid is set. If no contradiction occurs in the rough recognition, a flag indicating that the data is valid is set, and information (for example, color distribution) obtained by the recognition result verification unit 75 is added to the detection result memory 73. To do. In the general form, the content of the detection result memory 73 at this stage is template I = (position i, correlation coefficient i, color distribution i).

Next, the layout description unit 76 checks the layout between the objects by referring to the detection result memory 73, and verifies whether or not there is a contradiction with the basic layout described in the element database 72. If there is a contradiction, the process for removing the contradiction is performed. As an example of the process for removing the contradiction, it is determined that an object with a low correlation coefficient among the objects in which the contradiction has occurred is erroneously detected and is invalidated.
When the layout verification and the contradiction data deletion are completed, the correlation coefficient and the image layout are registered in the image database 77 together with the original image.

Next, the processing of the image search section 78 will be described.
Since there are some images of news images, recorded photos, and landscape photos that have complex image contents and cannot be described in words, it is extremely difficult to manually assign appropriate keywords. In addition, human words have a high degree of freedom, and conventional keyword searches are inefficient in searching. The most effective way to search for such an image is to use a layout expression such as "an image of a person standing on the right side of a white desk" that can be intuitively grasped by humans. Therefore, in this embodiment, an image is searched by matching the image layout registered in the image database 77 with the search layout input by the user in the above-described processing. With such a method, when an image is searched, a plurality of images are searched as candidate images. In this embodiment, the reliability of the image layout is calculated by multiplying the correlation coefficient of each object included in the search image, and the search is performed. At times, images with high reliability in layout are preferentially presented to realize quick retrieval.

In the conventional image retrieval system using keywords, there is a problem that the target image cannot be retrieved if the entered keyword is incorrect. However, this method allows the rough recognition even if the names of the objects are different. If there is a certain degree of similarity in (for example, the shape is similar), it is possible to search even though the reliability is low, and you can search the image even if there is a mistake in the expression. There are advantages.

[0054]

[Ninth Embodiment] FIG. 14 is a flowchart showing another embodiment of the image database creating method of the present invention. This image database creating method differs from the image database creating method shown in FIG. 1 in that when a processed image is read and rough recognition is performed, the candidate object extraction and the object extraction are performed without dividing into regions having the same characteristics. It is only a point for position detection.

That is, in this image database creating method, the template of the object to be handled by the image database created in step SP1 and the basic layout information between the objects are set, and step SP2
In, the image forming the image database is read by using an image scanner or the like, or the image is transferred from an image file or the like, and matching is performed at the pixel data level as rough recognition using a template serving as a reference in step SP3. Then, the candidate of the target object is extracted and the position of the target object is detected.

Next, in step SP4, the similarity between the object extracted by the rough recognition and the template is calculated based on the correlation coefficient of the correlation method or the like, and step SP5
In the above, feature point matching is performed as rough recognition verification, and erroneous extraction of the object extracted by rough recognition is deleted.
In step SP6, the layout verification of the objects is performed based on the basic layout information set between the objects in step SP1 to delete the incorrectly extracted objects, and in step SP7, the layout of the image after the layout verification. Information is created, the similarity between the object and the template and the image layout information are stored together with the image for each image in step SP8, and it is determined in step SP9 whether or not the image storage processing has been completed, and the image is stored. If it is determined that all the processing is completed, the series of processing is ended. If it is determined in step SP9 that the image storage processing has not been completed, step S9 is performed.
The processing from P2 to step SP8 is repeated until it is determined that the storage processing of the image to be processed is completed, and when it is determined that the storage processing of the image is completed in step SP9, a series of processing is terminated. .

In this image database creating method, since the object is directly extracted without dividing the processed image into areas, the processing associated with the area dividing processing becomes unnecessary and the same as the image database creating method shown in FIG. The effect is obtained. The object extracted in the rough recognition is, for example, as described above, the template A: (x1, y1, R1), (x2, y
2, R2) Template B: held in the form of (x3, y3, R3).

[0058]

[Embodiment 10] FIG. 15 is a schematic block diagram showing an embodiment of an image database creating apparatus of the present invention. This image database creating apparatus is different from the image database creating apparatus shown in FIG. 8 in that when a processed image is read and rough recognition is performed, the object is not divided into areas having the same characteristics and the object is processed by the pixel data level matching unit 34. Is only for extracting candidates and detecting the position of the object. Therefore, the area dividing unit 33 of the image database creating apparatus shown in FIG. 8 is omitted.

The operation of the image database creating apparatus having the above configuration is as follows. When creating an image database, an operator anticipates an object included in an image to be handled, creates a template of the object, or imports it from another image database to store the template of the object in the template holding unit 30. The basic layout information corresponding to the image to be handled is set and held in the basic layout information setting holding unit 31. In this state, the image reading unit 40 is operated to output one screen of the image forming the image database to the image holding unit 32, and the image holding unit 3
2 outputs the processed image to the pixel data level matching unit 34.

The pixel data level matching unit 34 performs matching at the pixel data level using the template of the object held in the template holding unit 30,
Selection candidates for each object are determined, the candidates for the object are held, and the position of the object is specified. The similarity calculation holding unit 35 calculates the similarity of each target object based on the output of the pixel data level matching unit 34 based on a correlation coefficient such as the normal correlation method, and the feature point matching unit 36 stores it as target object data. Output.

The candidates for the object in the pixel data level matching unit 34 are further feature point matching units 3.
6, the candidates are narrowed down, and output to the layout verification unit 37 in a state in which the positional information of the target object and the degree of similarity are held, and the layout verification unit 37 causes the basic layout setting holding unit 3 to operate.
Layout verification is performed with the basic layout held in 1, and the erroneously extracted target is deleted.

When the layout verification unit 37 deletes the erroneously extracted object, the image layout determination unit 38
The image layout determination unit 38 determines the layout of the image to be processed from the output of the layout verification unit 37 and outputs each information to the image database 39. That is, the image layout information from the image layout determination unit 38 is stored in the image layout storage unit 39 a of the image database 39. Also, image database 3
The image layout determination unit 38 is stored in the similarity storage unit 39b of No. 9.
Since the output of the similarity is from, the similarity of the candidate of the finally remaining object in the processed image is stored. Also,
The image storage unit 39c stores the processed image data of the image holding unit 32.

According to this image database creating apparatus,
Since the object is directly extracted without dividing the processed image into regions, the process associated with the region dividing process becomes unnecessary and the same effect as the image database creating device shown in FIG. 8 can be obtained. The present invention is not limited to the above embodiments, and various design modifications can be made without departing from the scope of the present invention.

For example, in the above embodiment, the case where the similarity to the template of the object is defined from the correlation coefficient by the correlation method in the matching at the pixel data level has been described as an example. Others may be used as long as they can be quantitatively calculated with a certain degree of accuracy, and are not particularly limited. Furthermore, the template of the target object to be set can be configured not only in shape and color, but also in a form in which other identification elements are added.

[0065]

As described above, the invention of claim 1 has a unique effect that it is possible to create an image database having a data structure that makes it easy to retrieve a target image.
According to the invention of claim 2, the image database has the image, the degree of similarity between each object of the image and the template, and the image layout information, so that the image search can be efficiently performed. Play.

The invention of claim 3 has a peculiar effect that an image database having a data structure for easily searching a target image can be created. The invention according to claim 4 has a peculiar effect that a target image can be quickly displayed in response to an ambiguous search request including layout information by using the image database of the present invention.

According to the invention of claim 5, the reliability of the layout can be calculated quickly by a simple operation such as multiplication, the search speed can be improved, and the search accuracy can be improved. Produce the effect of.
The invention according to claim 6 has a peculiar effect that a target image can be quickly displayed in response to an ambiguous search request including layout information by using the image database of the present invention.

According to the invention of claim 7, the reliability of the layout can be quickly calculated by a simple operation such as multiplication, the search speed can be improved, and the search accuracy can be improved. Produce the effect of.

[Brief description of drawings]

FIG. 1 is a flowchart of an embodiment of an image database creating method of the present invention.

FIG. 2 is a diagram showing an example of basic layout information.

FIG. 3 is a diagram showing a similarity between a template of an object and a predetermined area.

FIG. 4 is a diagram showing an example of an image to be read.

FIG. 5 is a flowchart of an embodiment of the image search method of the present invention.

FIG. 6 is a flowchart of another embodiment of the image search method of the present invention.

FIG. 7 is a diagram for explaining an embodiment of the image search method of the present invention.

FIG. 8 is a schematic block diagram showing an embodiment of the image database creation device of the present invention.

FIG. 9 is a diagram for schematically explaining the data structure of the image database of the present invention.

FIG. 10 is a schematic block diagram showing an embodiment of the image search device of the present invention.

FIG. 11 is a schematic block diagram showing another embodiment of the image search device of the present invention.

FIG. 12 is a schematic block diagram showing an embodiment of an integrated image search device of the present invention.

FIG. 13 is a diagram for explaining a correlation method.

FIG. 14 is a flowchart of another embodiment of the image database creating method of the present invention.

FIG. 15 is a schematic block diagram showing another embodiment of the image database creating apparatus of the present invention.

[Explanation of Codes] 30 Template Holding Unit 31 Search Layout Setting Holding Unit 33 Region Dividing Unit 34 Image Data Level Matching Unit 35 Similarity Calculation Holding Unit 36 Feature Point Matching Unit 37 Layout Verification Unit 38 Image Layout Determining Unit 39 Image Database 39a Image Layout storage unit 39b Similarity storage unit 39c Image storage unit 50 Search layout information input unit 52 Search layout information search unit 53 Image layout reliability calculation unit 53b Multiplication unit 54 Priority determination unit 72 Element database 73 Detection result memory 74 Outline Recognition unit 75 Recognition result verification unit 76 Layout description unit 77 Image database 78 Image search unit

Claims (7)

[Claims] 1. A template of an object and basic layout information between the objects are set, and rough recognition is performed using an original image template as a reference to extract candidate objects and detect the position of the object. Then, the similarity between the extracted object and the template is calculated, and the incorrect extraction of the object extracted by the rough recognition by the rough recognition verification is deleted, and based on the basic layout information and the layout of the object. A method for creating an image database, in which a wrongly extracted object is deleted, layout information of an original image is created, and the similarity between each object and the template and the image layout information are stored together with the original image. 2. An image database that stores an original image, a degree of similarity between an extracted object and a template, and image layout information. 3. Initial condition setting means (3) for setting a template of an object and basic layout information between the objects.
0), (31) and (72), and a rough recognition means (3) for roughly recognizing the original image using a template serving as a reference, and extracting a candidate of the target and detecting the position of the target.
3) (34) (74) and a similarity calculation means (35) (7) for calculating the similarity between the extracted object and the template.
4) and rough recognition means (33) by rough recognition verification
(34) First recognition verification means (36) (75) for deleting erroneous extraction of the object extracted by (74), and an object erroneously extracted based on the basic layout information and the layout of the object. Second recognition verification means (37) (76) for deleting an object and first recognition verification means (36) (7)
5) and the image layout creating means (38) (73) (76) for creating the layout information of the original image based on the outputs of the second recognition verification means (37) (76), and the respective objects and templates. An image database (39) (7) that stores the similarity and the image layout information together with the original image.
7) An image database creating device including and. 4. A search layout for an image to be searched using an image database (39) (77) that stores the original image, the degree of similarity between the extracted object and template, and the image layout information. Enter information and enter the entered search layout information and image database (39) (7)
The image to be searched is searched based on the image layout information of 7), the reliability of the layout of the search image is calculated based on the similarity of the objects included in the searched image, and the reliability of the calculated layout is calculated. An image retrieval method characterized by controlling display of retrieval images based on degree. 5. The image search method according to claim 4, wherein the reliability of the layout is calculated by multiplying the similarities of all objects included in the searched image. 6. An image storage section (39c) for storing an original image.
And an image database (39) (77) having a similarity storage unit (39b) that stores the similarity between the extracted object and the template and an image layout information storage unit (39a) that stores the image layout information. ) Is used to input search layout information of the image to be searched, and input search layout information and image layout information storage section (39).
The search means (52) (78) for searching for an image to be searched based on the image layout information in a) and the reliability of the layout of the search image based on the similarity of the objects included in the searched image. Reliability calculation means (53) (7) for calculating
8) and image display control means (54) (7) for controlling the display of the search image based on the calculated reliability of the layout.
8) An image retrieval device including: 7. The image search according to claim 6, wherein the reliability calculation means (53) includes multiplication means (53b) for multiplying the similarities of all objects included in the searched image. apparatus.