JPH08335223A - Image retrieval system - Google Patents

Abstract

PURPOSE: To give a higher degree of freedom in a method for specifying an image by setting a structure by specifying an area of the image and what the area is, and retrieving the object image by using the composition as a retrieval key. CONSTITUTION: The image to be retrieved is displayed in the limit frame 501 of a drawing area. A closed area 502 which indicates the arrangement of the part of an image which is considered to be proper as composition data is set, and the data of the closed area are saved in a field 511. A key word which is allocated to the area is saved in a field 512. For retrieval, a retrieval key that a retrieved person gives can be retrieved fast by scanning the composition data set of the retrieved image. Namely, a field 504 as the key word is scanned and matched to find matching data and a field 503 is checked to retrieve whether or not there is data having a matching area among data having matching key words.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for enabling image retrieval of image data stored in a memory using an electronic computer.

[0002]

2. Description of the Related Art Generally, in the case of retrieving image data, the conventional image retrieval method is as follows. Using the image itself that you want to search as a search key, determine whether it is included in the searched image by matching and retrieve it to retrieve the image, and search the keyword added in advance to the image using the keyword as the search key. This is a method of extracting an image.

The former can be used when an image to be searched is prepared at hand, but if it is not available, the search itself becomes impossible. The latter can search even if the image to be searched is not at hand, but even if it does not match the keyword added to the image completely or partially, it is not possible to obtain even the candidate of the image to be searched. Further, if a keyword representing a more semantically higher concept is added, the number of image candidates becomes so large that the image search also fails.

In order to sort out the problem, when there is no image to be searched at hand, a means for searching by keyword is effective, but the degree of freedom of the image cannot be adequately expressed in order to specify the desired image. This means that the image search will fail due to the cause.

[0005]

For example, in work in the field of creating new images by arranging images, such as cataloging, publicity images, and CM creation, it is possible to easily retrieve image samples that match the image of the creator. Method becomes essential. In this case, since the only search key is the image of the person himself / herself, the search by image matching cannot be used. Then, inevitably, a method of searching by using a keyword or creating a search key and using it is necessary.

The present invention is intended to provide a method and apparatus for enabling image retrieval of image data stored in a memory by using an electronic computer in such a situation.

[0007]

The present invention intends to utilize the composition of an image instead of the above-mentioned keyword. To attach a keyword to an image, you cannot select an appropriate keyword unless you properly understand the concept of attachment at that time, but when you look at the concept of image composition, it is added to the image and the concept of the composition that the person who wants to search has. Since the concept of power composition is common in that it is a composition that means a combination of elements and arrangements that make up the drawing, if composition data is added to the image to be searched, data representing the composition of the image to be searched will be displayed. It is possible to search for an image by creating and using it as a search key.

In this case, there is a possibility that the composition of the searched image is understood, or the way of expressing it is different depending on the person. Or, since it does not differ from the semantic representation of the image itself, it does not matter who added the composition data to the searched image.

[0009]

The image is searched by matching the composition data added to the image, which is stored in the database, with the search key representing the composition given by the searcher. When it is determined that the composition data added to the image matches the composition data as the search key given by the searcher, the search result is output as the image candidate to be obtained. Since the image is not directly used as the search key, it is easy to specify the search key, or it is possible to perform the search even when there is no image that can be used as the search key.

[0010]

Embodiments of the present invention will be described below.

FIG. 1 is a view for explaining an example of composition data input as a search key when searching for two-dimensional still image data stored in a database. The searcher forms a closed region 106 by a straight line 102 representing the side of a polygon within the limit frame 101 of the image drawing region on the screen. Reference numeral 103 is also a vertex of a polygon, 104 is a field for inputting a keyword applicable to the closed region, and 105 is the keyword.

In this example, the general name "mountain" is input as the keyword and the appearance of the mountain "dark" is input as the attribute. This area and set of keywords
It constitutes one of the composition data given by the searcher, and a combination of these is hereinafter referred to as a composition data set. In the composition data of this example, mountains are drawn from the central part of the image to the region slightly to the left, which means that the mountains are dark images. When it is determined that the area of the two-dimensional still image data stored in the database and the keyword match, it is assumed that the image requested by the composition data set specified by the searcher has been retrieved and the two-dimensional still image is retrieved. Image data shall be output.

In this example, the area where "town" is used as a keyword is also shown, and the searcher can also search by designating a composition data set composed of a plurality of composition data.

FIG. 2 illustrates an example of how the search result output is displayed. 201 is a reduced image of the search result. 202 is a field in which a description such as the name and attribute of the image is displayed. As shown in this example, a plurality of search results are obtained, and this number changes depending on how to control the degree of matching of the search, which will be described later. In the obtained reduced image 201, this image itself is a switch on the screen, and when the portion is clicked with the pointer, the image before reduction can be displayed. In this way, the searcher can view the image of the search result in more detail.

FIG. 3 illustrates an example of detailed display of search results. 301 is the limit frame of the display area in which the search image is displayed, 302 is the frame of the field in which the description of the image such as the name and attributes of the image or the place and meaning of the image are displayed, and 303 is the image obtained by the search , 304 are inside the field of 302, and 305 is the limit frame of the drawing area on the screen. It is a state of an image displayed by clicking the reduced image 201 of FIG. 2 with a pointer. When there is only one search result, the reduced image display of FIG. 2 is omitted,
It is also possible to make a setting that the search result details display of FIG. 3 is suddenly performed.

FIG. 4 shows another embodiment of composition data input for retrieval. 401 is a limit frame of the drawing area, 402 is a rectangle indicating the area, 403 is a closed area specified by the rectangle, 404 is an input field of a keyword applied to the closed area, and 405 is the keyword. The difference from FIG. 1 is that the specification of the closed region is not an arbitrary polygon but a rectangle. By designating two points on the image, it is possible to designate a rectangle having diagonal vertices. Since the method of describing the area is only the coordinates of two points, the storage capacity is very small, it is easy to handle, and the time and effort for specifying the area are easy. However, since the degree of freedom of the shape of the closed region is limited, the composition data as the search key is uniform, so it is necessary to loosen the matching tolerance for the search, and as a result, the selected output Therefore, the load of the work of selecting a desired image from the search results may increase.

FIG. 5 illustrates an embodiment of a method of creating composition data of a searched image. The image to be searched is displayed in the limit frame 501 of the drawing area. From the displayed search target image, a closed area 502 indicating the arrangement of the portion of the image considered to be appropriate as composition data is set, and the data of this closed area is stored in the field 511. Also, the keyword assigned to the area is stored in the field 512. 505
Is a line showing the correspondence between the closed region and the closed region data storage field.

That is, to explain in correspondence with FIG. 1, since a dark mountain is drawn in the area 502 of the searched image,
Specifying the area 502 and the keyword "mountain, dark" for this
Is given. This is also done for regions 503 and 504. Field 51 for storing this area data
There are as many combinations of 1 and data in the field 512 for storing keywords as there are closed areas (elements of composition) attached to the image to be searched, and a plurality of them are tabulated side by side to form a composition dataset. .

At the time of search, the search key given by the search target person can be searched at high speed by scanning the composition data set of the search target image rather than the actual image. One method of searching is to scan the field 504 of the keyword to find a match by matching and then check field 503 to see if the region may be determined to match from the keyword matches. It is a method of detecting.

FIG. 6 illustrates an embodiment of the data structure of the search key provided by the searcher. The search key is made up of the entire length of the search key, the number of composition data, the permissible mismatching ratio of composition data, and one or more composition data. The composition data is composed of area data, which is the coordinate data of each vertex of the polygon and the keyword, and data indicating the allowable range of the positional deviation and the area deviation when matching the areas. Data indicating the allowable range of these deviations is set in advance and automatically given and used at the time of search.

If matching is executed for all composition data, and the ratio of the number of data that does not actually fall within the permissible range and the number of composition data sets is less than or equal to the permissible mismatch matching number, It is determined that the composition data match. As another embodiment, there is a method of dynamically changing the allowable range and increasing the matching data when the matching data is too small when the search is executed.

Since the image retrieval system according to the present invention can be realized by a general computer equipped with software capable of inputting an image with a keyboard, a mouse, etc., an example thereof will be described later. First, an example of a concrete processing flow will be explained. .

FIG. 7 is a flow chart of an embodiment of the composition data search device. Reference numeral 801 indicates the input of one composition data. As described in FIG. 1, the composition data is input by designating an area and designating a keyword in the area. It is necessary for the searcher to input as many as necessary to specify the image to be searched, and 802 is an inquiry as to whether or not all composition data has been input. Instead of this inquiry 802, normally, when one composition data is input, it is considered that all the composition data has not been input yet, and the system waits for the input of the closed area. However, pressing the end key on the keyboard or on the screen It is also possible to implement to move to the next image search by clicking the end icon with the pointer.
Reference numeral 803 is an image search by composition data and display of the result. At 804, you will be asked if you want to end the image search. If not finished, return to the beginning.

In the flow chart, the image search and result display of 803 are drawn by the symbol of the subroutine, but the flow chart here does not specify the mounting method. Therefore, it may be implemented in the form of a subroutine or inline expansion. The same applies to the drawings for explaining the following flowcharts.

FIG. 8 is a flow chart showing an embodiment of a method of image retrieval and result display in the case of a two-dimensional still image, which is an example of the process 803 in FIG. Reference numeral 901 denotes a process of preparing an appropriate number of images for image matching based on the composition data, and preparing composition data attached to the images from the database as shown in the table of FIG. A block 902 executes image matching by comparing the composition data attached to the image with the composition data as a search key. 903 determines whether the images match. A block 904 stores the searched image when matching is performed. Reference numeral 905 is a block for displaying the search result. Hereinafter, as long as the search is continued using the same composition data, the process is repeated from the beginning. The condition for stopping the search by 906 is, for example, when the number of matched images has reached the set number, when the time limit is reached, or when the searcher gives an instruction to stop the search.

FIG. 9 is a flowchart showing an embodiment of an image matching method using composition data in the case of a two-dimensional still image, which is an example of the process 902 in FIG. Reference numeral 1001 is a portion for extracting one composition data from a plurality of composition data of the search key shown in FIG. 1002 is a determination as to whether or not there is a portion where the keywords match. If there is a match, proceed to 1003 for that area. Reference numeral 1003 is for matching all the closed areas of the search key with the actual image size. 10
Reference numeral 04 calculates the positional deviation between the closed area centroid of the composition data of one of the search keys and the closed area centroid of the composition data attached to the searched image, and the positional deviation is set in advance. If it is within the range, it is determined that the positions match. Similarly, if the area ratios obtained by dividing a large area by a small area for the areas of both areas are within a preset allowable range, 1005 determines that the areas match. 1006 is a composition data set coincidence counter. It is a counter for counting how many pieces of composition data among a plurality of composition data are matched. 1007 determines whether matching has been performed for all composition data included in the search key. This can be realized by counting the number of composition data. If the ratio of the number of matching counters counted in 1006 divided by the number of composition data in the search key is within the preset allowable range, the 1008 considers the composition to be a match and judges the match. return. Otherwise, a non-matching judgment is returned. As the allowable range set in advance, for example, if the mismatch number allowable range of FIG. 6 is used, one obtained by subtracting the value can be used.

FIG. 10 is a flow chart showing another embodiment of the image matching method using composition data in the case of a two-dimensional still image. 1101 to 1103 are 1001 to 10 respectively
Same as 03. Reference numeral 1104 is a part for obtaining evaluation criteria for the degree of separation of the area ratio of the regions and the degree of separation of the positions. Here, the area size of the composition data of the search key is S1,
If the area of the composition data added to the searched screen is S2, A = max (S1 / S2, S2 / S1) can be selected. Also, S
If the barycentric coordinates of the area of 1 and S2 are G1 (G1x, G1y), G2 (G2x, G2y), B = sqrt ((G1x-G2x) ** 2+ (G1y-G2y) ** 2) can do. Here, sqrt and ** represent square root and power, respectively. 1105 defines the energy function E by giving weights Wa and Wb in advance and defining it as E = Wa * A + Wb * B, and when it becomes smaller than a preset value, the It is determined that the composition datasets match. 11
06 to 1108 are the same as 1006 to 1008, respectively.

FIG. 11 is a schematic diagram of an external view of an embodiment of the image retrieval system by composition. 1201 is a display device, 12
02 and 1207 are composition input screens displayed by the display device,
A search result display screen, 1203, is a storage device used for storing an image database with composition data and for saving images obtained as search results. It is equipped with a hard disk that stores the database and a removable storage device such as a floppy disk or magneto-optical disk that stores the search images. Reference numeral 1204 is a keyboard used when inputting a keyword and the like, and 1205 is a mouse of a coordinate input device used when specifying an area. A printer 1206 outputs a search image and search conditions.

With this apparatus, for example, by the method shown in FIG. 5, image data is added to the searched screen and stored in the hard disk. Further, according to the method described with reference to FIG. 1, image data for search is created and the search is performed.

FIG. 12 illustrates another example of adding composition data to the searched image. 1301 is a limit frame of the drawing area of the screen, and 1302 is a display area in which the searched image is displayed. Reference numeral 1303 is a rectangle designating a closed area of composition data. 1304 is an image element that is a part of the image and is a "cloud".
Is drawn. The closed area specified by the rectangle 1303 specifies the "cloud" of the image element by the area. Reference numeral 1305 is an arrow indicating which area the keyword corresponds to. Reference numeral 1306 indicates a frame of a field for inputting a keyword, and 1307 indicates the field. Closed area 1303
If the field 1306 cannot be placed directly within the closed area, such as the area specified by is not large enough, it can also be indicated as indicated by the arrow at 1305. By repeating such work for each image element, the composition data of the searched image is completed.

FIG. 13 illustrates an embodiment of the structure of the composition data of the searched image. The composition data group is composed of the entire length of the composition data, the number of composition data, and one or more composition data. The composition data is composed of the length of the area data, the area data that is the coordinate data of each vertex of the polygon, the number of keywords, and the keyword string that exists by that number. The image header is
It shows what the image looks like. In fact, the image data format that already exists can be adopted for both the header and the image data.

However, at that time, the name of the image given to the entire image and the attributes such as the place, the author, and the date are added. By the way, in the example of FIG. 13, the image header has its length, the name and attribute of the image data, the number of pixels on the X axis of the image, the number of pixels on the Y axis of the image, the number of color data per pixel, and the length of the color map. , Colormap ,. The number of color data per pixel is two in the case of a bitmap and 24 in the case of a full-color image because the color is represented by 8-bit luminance information for each color of RGB. A color map is a correspondence table showing the correspondence between image data and the colors of an actual full-color image, etc. Even if the number of color data per pixel is eight, one color data is actually obtained. The colors that can be assigned can express a wider variety. Of course, the number of colors that can be developed simultaneously is 256 in that case. In the image data, data having a structure designated by the image header is placed.

By the way, the composition data of the searched image does not necessarily have to be included with the searched image data. For example, an implementation method is possible in which a pointer indicating the hierarchy or position on the recording device in which the searched image data is stored is provided at the beginning of the composition data. In this method, only the image data is stored in the image database, and only the composition data including the pointer indicating the position of the image on the image database is stored in the composition data database. Since it is not necessary to store the image itself in the composition data database, it contributes to saving the storage capacity.

FIG. 14 is a flowchart of another embodiment of the composition data input method. In this example, the area designation is semi-automatic as compared with the example shown in FIG. 1701 is a block that roughly specifies a region by a rectangle. 17
02 removes the noise of the image in the area. 1703 performs edge detection in the same area, and 1704 binarizes the edge image and removes binarized noise. 1705 performs thinning of the image, 1706 interactively removes unnecessary things in the image edge and supplements necessary parts to make corrections.
Interpolate the edges to create a closed area. The 1708 displays the designated area so that it can be confirmed whether a desired image element can be designated in the area. 1709 ends if the result is what you want, and if not, returns to the block of corrections in 1706. 1710 creates and adds a keyword to the area when the closed area is determined and completed.

In this example, a semi-automated complex algorithm was shown. Of course, a method of designating a closed region by using a rectangle, a polygon, a curve, or the like and not using an edge of an image can be given as another embodiment.

FIG. 15 illustrates the appearance of an image tag creation system for adding composition data to an image to be searched. The image tag here means composition data, but hereinafter, the terms image tag or image data will be used as appropriate. Figure 11
Compared to the image retrieval system based on the composition data described in
An image input device is added and an output device is omitted, but there is no essential difference. Reference numeral 1801 denotes a display device, and 1802 denotes an image tag creation screen displayed by the display device. 1803 is an image database and a recording device for images with composition data. An image database with composition data will be built here. 1804 is a keyboard used for inputting a keyword of composition data, 1805 is a mouse used for designating an area of an image element of composition data, and 1806 is an 8 mm video or a means for inputting image data. For example, an 8mm VCR. 1807
Is a VCR with a video tape playback function such as VHS and an LD with an image recording medium playback function such as LD
An LD writing device that can record an image with composition data on a player or an LD is used.

Similar to FIG. 11, this system can be used as an image retrieval system by composition data, but composition data is further added to an image obtained by retrieval by composition data and returned to the image database with composition data. It is possible to The image data with composition data added with the composition data and stored can be searched again by composition data search.

As described above, it is possible to further enhance the data once made into a database by adding information.
In this case, if the database is not for individual use but for groups or groups, the data can be enriched every time various people use it, and the effect of making the database construction faster can be expected. .

FIG. 16 is a schematic diagram for explaining the data input / output portion of the three-dimensional image structure retrieval system. Various operations can be performed using a virtual space created by a computer called virtual reality. 2001 is a data glove that can detect bending of joints by optical fibers. Using this data globe, a three-dimensional area is specified in a three-dimensional image space created by a computer to realize three-dimensional composition input. 2002 is a display device that projects a three-dimensional image. It is a device that creates a parallax between both eyes by presenting a slightly different image to each eye so that a stereoscopic image can be perceived. 2003 is a magnetic sensor. It is used to detect the position and orientation of the hand and head. 2004 is a foot pedal. You can move in the 3D image as if you were in the image, and at that time, you can specify the moving speed and direction with the foot pedal. One or more foot pedals are prepared. In the case of two, the left and right pedals receive the power control of the tires attached to the left and right independently, and can virtually move in the three-dimensional image space. Of course, with the movement, the appearance of the 3D image changes as if it were on the spot. 2005
Is a microphone for voice input. By inputting voice or sound, it is possible to add a keyword or select a condition. In addition, by projecting a virtual cockpit in a three-dimensional image and operating it, it is possible to move the image space, input a composition, and perform other operations.
Manipulation in the video space is realized by detecting human movements, which is constructed from data glove inputs and magnetic sensor inputs.

FIG. 17 is a schematic diagram of an embodiment showing the physical connection of the composition image retrieval network. 2101 is a medium for transmitting information on the network. For example, it is realized by a coaxial cable, an optical fiber, or communication by radio waves. Reference numeral 2102 is another network, and the other networks are connected by an image search gateway 2103. Reference numeral 2104 is a database of tagged images, which has a function of transmitting it to the network. Reference numeral 2105 is a search device for tagged images, which is obtained by adding a search function to the image database 2104. An image search terminal 2106 is used by the user to search for images using this terminal.

The composition data sent from the image search terminal 2106 reaches the image search gateway 2103 and the image search device with tag 2105 and searches for an image. Image search gateway
It is also sent to another network 2102 via 2103.

One route through which the image data to be searched for search is from the image transmitter with tag 2104 to the image search terminal.
It is sent to 2106 and searched.

The other route of the search is to send the matching image searched by the image search device with tag 2105 to the image search terminal 2106, and if necessary, further search to complete the search.

As another route of the search, the image sent from another network 2102 is searched by the image search gateway 2103, transferred to the network 2101, and delivered to the image search terminal 2106. It is a thing.

If the image retrieval gateway 2103 is connected between other physically separated networks, it is efficient due to the traffic mitigation effect of data transfer,
If the efficiency is not considered, it is possible to input the searched image from one network and send the searched image to the same network.

FIG. 18 is a schematic diagram of an embodiment showing the information transmission shape of the composition image retrieval network. Information is transmitted on the tree structure with the image search terminal 2201 as the apex. The information transmission path 2202 connects the image search terminal 2201 and the next image search gateway 2203. A device having an image database, such as a tagged image search device 2204 or a tagged image transmission device, is connected to the end of the tree structure. An image search gateway 2203 is arranged at each branch of the tree structure. When the image data is searched and narrowed down, it climbs the branch of information transmission and reaches the image search terminal at the apex, and is narrowed down to the data set as the search key.

As another embodiment, as the image data, only the composition data attached to the image data is transmitted to the network for searching, and only the image data satisfying all the search conditions is transferred from the database. Is the method.

FIG. 19 is a block diagram showing an embodiment of the image search gateway. Now, the composition data that is the search key is temporarily transferred from the network 2304 to the network communication unit.
Suppose it reaches 2301. It is sent to the composition image search unit 2302 and set. The searched image is the network communication unit 23.
It is sent from 03, matching is performed for each image and search is executed. The searched image data is sent back to the sender of the composition data, which is the search key, through the network communication unit 2301. These operations are performed by the network communication unit.
2301 and 2303 are equivalent, and the same operation is performed even when the search key arrives from the opposite direction. Further, even if the search key and the image to be searched arrive from the same network, the same operation is performed.

FIG. 20 is a diagram for explaining an embodiment of a mechanism for adding new composition data to the search image data by the search requesting user. Reference numeral 2401 is a search request source, and the user searches for an image by setting the composition data sets a and b which are elements of the composition data 2403. Image data 2405 to which a tag of composition data is added is transmitted from the composition image database 2402. G represents image data. The tagged image data 2404 is the search request source after the search.
It reaches 2401 and is presented to the user.

Here, it is assumed that the user further adds composition data to the image. The composition dataset
If 2406 is c, it is returned to the composition database as in 2407, and the original composition data is updated as in 2408. Assuming that it will be used on the network by multiple users, the database will grow rapidly by adding composition data to the images searched by many users.

However, preventive measures are required to prevent the user from adding false information. One of them is to add a mechanism to check composition registration on the composition database side through a person. The other is to set a field that cannot be changed by the user for all composition data, and add information such as the mail address of the person who added the composition and the affiliated organization in the form of a keyword. Then, it is a method that allows the user to search by selecting the search range by the keyword or the like. In the latter case, determining keywords so that it can be distinguished between information added by individuals and information added by public institutions or companies is one useful classification when narrowing the search range. I think it will be.

FIG. 21 is a diagram for explaining an embodiment of a mechanism for transmitting composition data as a search key from a search request source to the image search network in the composition image search network. 2501 is the search request source, and a composition image search terminal or the like is assumed. From here, a search key having five composition image data sets 2508 called abcde is transmitted. Then it will find each search gateway 2502, 2503, 250
It is transmitted to the composition image databases 2505, 2507, etc. via 4, 2506.

In this illustration, it is assumed that this database has a search function. The composition data transmitted to the search gateway adopts one of the composition data sets, which is its element, and transfers the rest to the search means such as the next search gateway on the network. here,
If the search conditions are selected in ascending order, most of the image data that do not meet the search conditions will be sorted first by the ones with the strict search conditions, and the image data that flows through the network will decrease.

This search method using the image search network is assumed to use a network with a very high information transfer rate, and is very effective when the time required for data transfer is shorter than the time required for the search. . As such a network, for example, there is a network of respective arithmetic units on a parallel computer.

As still another embodiment, the search items contained in the search key, the composition of the composition data here, is not fixed to each search means, but it is changed according to the load of the network and the search means. The assigned search items are dynamically assigned to the search means. When the load on the search means is heavy, or when the data transfer load on the input side network is heavy, some of the search items are allocated to the search means on the information source side. Then, since the present search conditions are loosened, the load of the search means is likely to be reduced, and the search conditions on the information source side are also strict, so that the amount of data transferred by the network is reduced and the load of the network is also reduced. When both loads are light, a part of the search items is allocated to the search request source side.

As yet another embodiment, the composition image data from the information source has a check item for the search item, and a mark is added to the searched item. The search means that has received the data checks the item, performs matching on the item that has not been searched yet, adds the check of the search item to the data that passed it, and transfers the data to the next search means. .

As a known example similar to this search method, there is an information search method using a communication language called Telescript of General Magic Co., USA. This allows a job called Telescript to be given to a job unit called an agent, such as image search. Then, it moves or splits a server called Telescript engine on the network and performs searches while increasing it, and when it finds the desired one, it transfers it to the requester.

FIG. 22 illustrates an example in which a hierarchical structure or a hierarchical concept of an image object is used as an example of a name used as a keyword of an image to be searched, and an embodiment thereof.
2601 is a word indicating an animal "dog". This is a common name, and there are various types of dogs. 2602 uses the size of a dog as a method for further distinguishing "dogs". "Small dog""Mediumdog"
This is an example of classification into three types of "large dogs". 2603 is an example in which the type of dog is described in more detail for each classification 2602. There are various ways to divide the structure, and as an example, it is possible to have such a hierarchical structure.

Reference numeral 2604 indicates an example of a keyword input as composition data. 2605 is the name of the image including the hierarchical structure. The names “dog”, “small dog”, and “yorkshire terrier” are combined with delimiters in between.

The method of retrieval is up to the composition retrieval system, but a general retrieval method is to enable retrieval by any of these three names. In other words, you can search by "dog" and "yorkshire terrier". Then, if you search for “dog”, you can also select “Shiba Inu” or “Colly”, but if you search for “Yorkshire Terrier”, only the images with that name will be listed as candidates. Only. In this way, the search range can be specified by selecting the name of the level of this hierarchical structure as the name specified as the keyword of the search key. In 2606, the attribute of the image is written. Although the image is a small dog called "Yorkshire Terrier", information that the dog is "brown" and "naughty" is added.

At the time of search, the range of the search can be narrowed by designating the attribute. In this example, if the name "Yorkshire Terrier" attribute "Brown" is specified as the search key, the field of the keyword name of the searched image is matched to search for "Yorkshire Terrier" and the attribute field is matched. Will be searched for "brown". The 2605 has a hierarchical structure as long as the words in the higher order are included, and it is not necessary to arrange the words in the higher order in the hierarchy from the top. When searching, every word is scanned indiscriminately. Therefore, even a classification that has a different hierarchical structure can have even the names that appear there arranged there. Also, you can put the information of the person who wrote this information in the attribute of 2606. For example, reliable information added by a public institution and untrusted information added by an individual can be distinguished by adding the attribute to the keyword of the search key.

FIG. 23 is a schematic diagram for explaining the structure of the semantic hierarchy of inputting as a keyword with respect to the keyword input of the tag creating device for adding composition data. For example, assume that the structure for the letter K is now as shown in the figure. However, it is assumed that K, 1,2,3, etc. represent meaningful characters. Then, the upper structure of K is considered to be 1-2-3-K of A and 4-5-6-K of B. Such candidates are automatically created by searching the dictionary. An example of a dictionary containing a thesaurus, which is used in this case, is the classification vocabulary table of the National Institute for Japanese Language (Shuei Publishing, 1964).

FIG. 24 shows an example in which the keyword candidates are listed regarding the keyword input of the tag creating apparatus for adding the composition data. When there are a plurality of candidates as shown in FIG. 23, a plurality of candidates to be input as keywords are displayed and a user can select from them.
For example, when A is selected, a keyword including a higher-level structure such as "1: 2: 3: K" is set.

FIG. 25 illustrates a procedure for inputting a character having a higher-level structure as a keyword in inputting a keyword in the tag creating apparatus for adding composition data. First, enter keywords using the 2901. Then 29
In 02, the structure of the meaning including the character is searched from the dictionary and its superstructure is taken out. 2903 is the dictionary. You can use the classification vocabulary table of the National Institute for Japanese Language (Shuei Publishing, 1964). In 2904, it is determined whether or not the candidate is 1, and if there is a plurality of candidates, it is displayed in 2905 as shown in FIG. At 2906, select a structure from it. The keyword input is completed by setting the structure as a keyword in 2907.

The image described in each claim is not limited to a two-dimensional still image, and does not prevent application to data such as a moving image or a three-dimensional image.

[0066]

According to the present invention, the composition is set by designating the area of the image and what the area is,
By searching for the image to be searched using that composition as a search key, it is possible to give a wider degree of freedom by the method of specifying the image, compared to image search by image name or hypercard type image selection such as Mosaic. enable.

Further, the present invention provides a means for constructing a database capable of searching images by composition.

[Brief description of drawings]

FIG. 1 is a diagram for explaining how composition data is input in the case of two-dimensional still image data.

FIG. 2 is a diagram illustrating a display of search results.

FIG. 3 is a diagram illustrating detailed display of search results.

FIG. 4 is a diagram illustrating another example of composition data input.

FIG. 5 is a diagram illustrating an example of a method of saving composition data.

FIG. 6 is a diagram illustrating an example of a data structure of a search key.

FIG. 7 is a flowchart of an example of composition data search.

FIG. 8 is a flowchart showing an example of a method of image search and result display in the case of a two-dimensional still image.

FIG. 9 is a flowchart showing an embodiment of an image matching method using composition data in the case of a two-dimensional still image.

FIG. 10 is a flowchart showing another embodiment of an image matching method using composition data in the case of a two-dimensional still image.

FIG. 11 is a schematic diagram of an external view of an embodiment of an image search system based on composition data.

FIG. 12 is a diagram illustrating a state in which a tag is added to an image of the image tag creation system in a two-dimensional still image.

FIG. 13 is a diagram illustrating an example of a data structure of image data with composition data.

FIG. 14 is a flowchart of an embodiment of a composition data input method.

FIG. 15 is a diagram illustrating the appearance of an image tag creation system that adds composition data to an image.

FIG. 16 is a schematic diagram illustrating a data input / output portion of a three-dimensional image structure search system.

FIG. 17 is a schematic diagram of an embodiment showing a physical connection of a composition image search network.

FIG. 18 is a schematic diagram of an example showing an information transmission shape of a composition image search network.

FIG. 19 is a block diagram showing an embodiment of an image search gateway.

FIG. 20 is a diagram illustrating an example of a mechanism for adding new composition data to search image data by a search requesting user.

FIG. 21 is a diagram illustrating an example of a mechanism for transmitting composition data from a search request source to the image search network in the composition image search network.

FIG. 22 is a diagram illustrating an example in which a hierarchical structure or a hierarchical concept of an image target is used as an example of a name used as a keyword of a searched image and an example thereof.

FIG. 23 is a schematic diagram for explaining a hierarchical structure of a meaning input as a keyword in inputting a keyword of a tag creation device to which composition data is added.

FIG. 24 is a diagram showing an example in which keyword candidates are listed regarding keyword input of a tag creation device that adds composition data.

FIG. 25 is a flowchart showing a procedure for inputting a character having a higher-level structure as a keyword in inputting a keyword in the tag creation device which adds composition data.

[Explanation of symbols]

101: limit frame of image drawing area, 102: straight line representing sides of polygon, 103: vertex of polygon, 104: field for inputting keyword, 105: keyword, 106: closed area,
201: Image of reduced search result, 202: Field displaying explanation of name and attribute of the image, 301: Limit frame of display area, 302: Field displaying explanation of image such as name and attribute of image Frame, 303: image obtained by search, 304: inside field of 302, 305: limit frame of drawing area, 401: limit frame of drawing area, 402: rectangle indicating area, 403: closed area, 404: keyword Input field, 405: keyword, 501: limit frame of drawing area, 502,
503, 504: Closed area, 511: Field for storing data of closed area, 512: Keyword, 505: Curve showing correspondence between closed area and closed area data storage field, 1201: Display device, 1202: Composition input screen Search results, 1203: Storage device, 1204: Keyboard, 1205: Mouse, 1206: Printer, 1301: Limit frame of drawing area, 1302: Display area where image data is displayed, 1303: Composition data closed area Specified rectangle, 1304: "Cloud" is drawn in the image element that is a part of the image, 1305: arrow indicating which area the keyword corresponds to, 1306: frame of the field for entering the keyword, 1307: keyword input field, 1801: display device, 1802: image tag creation screen,
1803: Image database and image recording device with composition data, 1804: Keyboard, 1805: Mouse, 1806: 8mm
VCR with video tape playback function such as 1807: VHS such as video or 8mm VCR, and LD
LD player that has a function of reproducing image recording media such as, or an LD writing device that can record images with composition data onto an LD, 2001: Data Grove, 2002: 3
Display device for projecting 3D images, 2003: Magnetic sensor, 20
04: Foot pedal, 2005: Microphone, 2101: Network information transmission medium, 2102: Other networks, 2103: Image search gateway, 2104: Tagged image database, 2105: Tagged image search device, 2106: Image search Terminal 2201: Image search terminal 2202: Information transmission path 2203:
Image search gateway, 2204: image search device with tag, image transmission device with tag, etc. 2301: network communication unit, 2302: composition image search unit, 2303: network communication unit, 2304: network, 2401: search request source, 2402: Composition image database, 2403: Composition data, 2404: Tagged image data being transmitted, 2405: Tagged image data in composition image database, 2406: Composition data set added by user, 2407: Composition data returned to composition database Set, 2408: updated composition image data, 250
1: Composition image search terminal or other search request source, 2502: Search gateway (has composition data set a for search), 250
3: Search gateway (has composition data set b for search), 2504: search gateway (has composition data set c for search), 2505: search database (has composition data set d, e for search), 2506: Search gateway (having composition data set b for search), 2507: Search database (having composition data sets c, d, e for search), 2508: Composition data for search sent from search request source , 2601:
2602: The word that indicates the animal "dog", 2602: The one that uses the size of the dog as a method to further distinguish "dog", 2603: In the example that describes the type of dog in detail, 2604: Input as composition data 2605: example of image name including hierarchical structure, 2606: image attribute, 2901: input part of character string to be added to image in area, 2902: character string search part, 29
03: Dictionary of hierarchical structure, 2904: Judgment unit for number of structure candidates, 29
05: Multiple candidate display section, 2906: Candidate selection section, 2907: Keyword setting section.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Masaru Takeuchi 2520 Akanuma, Hatoyama-cho, Hiki-gun, Saitama Stock company Hitachi Research Laboratory (72) Inventor Atsushi Hiroike 2520 Akanuma, Hatoyama-cho, Hiki-gun Saitama Inside Hitachi Research Laboratory (72) Inventor Yasuhide Mori 2520 Akanuma, Hatoyama Town, Hiki-gun, Saitama Stock Company Hitachi Research Laboratory

Claims (9)

[Claims] 1. Composition data of an image in which an image element of an arbitrary area of a searched image to be displayed on a display device is specified by area data indicating a position in the image and expression of characteristics of the image element in the area. A database that has, a display device for inputting composition data consisting of area data indicating the position of an image element that the image to be searched should have and a feature expression of the image element, and composition data input through the display device And a display device that displays the detection result and a processing device that detects a match between composition data in the database. 2. An input having a function of designating a region of an image element which is a part of an image displayed on a display device and adding information expressing characteristics of the image element to the image element to obtain composition data. The image retrieval system according to claim 1, further comprising a device. 3. The image search system according to claim 2, wherein the area of the image element is set by designating a closed area within the display limit of the display device. 4. A match is determined by setting a predetermined allowable range for the positional deviation between the closed region included in the composition data of the searched image in the database and the closed region of the search data.
Image retrieval system described. 5. A predetermined allowable range is set for the difference between the size of the area of the closed region included in the composition data of the searched image in the database and the size of the area of the closed region of the search data, and a match is determined. The image search system according to claim 1. 6. The image search system according to claim 1, wherein the information expressing the characteristics of the image element is expressed by a keyword. 7. The image search system according to claim 1, further comprising a function of adding information indicating a feature of a new image element to the searched image and returning the added information to the database. 8. Composition data of an image in which an image element of an arbitrary area of a searched image to be displayed on a display device is specified by area data indicating a position in the image and expression of characteristics of the image element in the area. A database that has, a display device for inputting composition data consisting of area data indicating the position of an image element that the image to be searched should have and a feature expression of the image element, and composition data input through the display device And an image retrieving system for detecting a match between the composition data in the database and a display device for displaying the detection result, and a network connecting the image retrieving systems. An image retrieval network system characterized in that the databases can be used by each other. 9. The image search network system according to claim 8, further comprising a function of adding information indicating characteristics of a new image element to the searched image and returning the information to the original database.