JPH10254903A - Image retrieval method and device therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically and effectively acquire the information on an unknown object based on its image by retrieving an image data base based on the retrieval object image and also the environmental information defined when the object image is acquired. SOLUTION: A user photographs an object via a camera part 1 to retrieve an unknown object. If the composition of the photographed image is inappropriate, the user corrects the composition via a composition change part 3 to acquire an image of optimum composition. Then the user cuts out the contour of the optimum image shown at a display part 14 via a pen input part 4 and extracts the image at an image extraction part 5. The extracted image is converted into an object binary image at an object image conversion part 6. Meanwhile, the object size is calculated from the photographed image, and the data image acquired by retrieving an image data base 11 is converted into a binary image at a data image conversion part 12 as long as the data image has a size equal to the object. Then the binary image is compared with the object binary image at an image comparison part 10. When the highest similarity is decided between both binary images, the data are extracted from the acquired image. Then the retrieval result is outputted as an image via a result display control part 13.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval method and apparatus, and more particularly to an image retrieval method for photographing an unknown object.
The present invention relates to an image search method and apparatus capable of automatically obtaining information on an object from an image.

[0002]

2. Description of the Related Art Conventionally, when a desired image is retrieved from an image database, there is known a method of obtaining a desired image based on the degree of coincidence between two image data, that is, an input image and an image in the image database.

Further, since the input image data includes image data which is not directly related to the object such as the background, it is necessary to search the image database using the input image data. It is necessary to cut out only a simple object, but as a method of extracting the outline of the object from the line drawing information, it has been conventionally performed.

[0004]

However, in a method of obtaining a desired image based on the degree of coincidence between two image data, that is, an input image and an image in an image database, a search is performed using the entire image of the object. However, there is a problem that similar objects are considered as a large number of candidates, and it is difficult to narrow down targets.

In the method of extracting the outline of an object from line drawing information in order to cut out a necessary object, information on the attribute of the object cannot be obtained, so that information for search is insufficient, so that it is expected. There was a problem that it was difficult to obtain a good search result.

Accordingly, an object of the present invention is to provide an image search method and apparatus capable of automatically and efficiently obtaining information on an unknown target image from the target image.

[0007]

In order to achieve the above object, according to a first aspect of the present invention, there is provided an image search method for searching an image database based on a search target image and acquiring information on the search target image. Environment information at the time of acquisition of the search target image is input together with the target image, and the image database is searched based on the search target image and the environment information.

Here, the environment information is information expressing conditions at the time when the search target image is obtained. Information on the location of the acquisition location, information on the acquisition time, information on the device and method used for acquisition, information on the observation target when the image was acquired, environmental conditions such as climate, lighting, temperature and humidity at the acquisition location This includes information about

Further, the invention of claim 2 is characterized in that, in the invention of claim 1, the search target image is a picked-up image obtained by image pick-up by an image pick-up means.

According to a third aspect of the present invention, in the second aspect of the present invention, the environment information is size information indicating an actual size of the captured image calculated based on parameter information of the imaging means. Features.

According to a fourth aspect of the present invention, in the third aspect of the invention, the parameter information is zoom information indicating a zoom ratio of the imaging means, and the size information is the zoom information and the imaging means. The distance is calculated based on distance information indicating a distance from the object to the object to be imaged and relative size information indicating a relative size shown in the image screen captured by the image capturing means.

According to a fifth aspect of the present invention, in the fourth aspect of the present invention, the distance information is calculated based on autofocus information for controlling autofocus of the image pickup means.

According to a sixth aspect of the present invention, in the fourth aspect of the invention, the distance information is obtained based on a measurement output of a distance measuring means for measuring a distance from the imaging means to the object to be imaged. And

According to a seventh aspect of the present invention, in the second aspect, the environment information is altitude information indicating an altitude of an acquisition position of a captured image acquired by the imaging means.

According to an eighth aspect of the present invention, in the second aspect, the environment information is date and time information indicating an acquisition date and time of a captured image acquired by the imaging means.

According to a ninth aspect of the present invention, in the second aspect of the present invention, the environment information is position information indicating an acquisition position of a captured image acquired by the imaging means.

According to a tenth aspect of the present invention, in the second aspect, the environment information is area information indicating an acquisition area of a captured image acquired by the imaging means.

According to an eleventh aspect of the present invention, there is provided an image search apparatus for searching an image database based on a search target image and obtaining information on the search target image, wherein: a search target image obtaining means for obtaining the search target image; Environment information obtaining means for obtaining environment information indicating an environment in which the search target image is obtained by the search target image obtaining means; search target images obtained by the search target image obtaining means; and environment information obtained by the environment information obtaining means And a search means for searching the image database based on the image database.

A twelfth aspect of the present invention is characterized in that, in the eleventh aspect of the present invention, the search target image obtaining means is an imaging means for picking up the search target image.

According to a thirteenth aspect of the present invention, in the twelfth aspect of the present invention, the environment information obtaining means stores the size information indicating the actual size of the captured image calculated based on the parameter information of the imaging means. It is obtained as environmental information.

According to a fourteenth aspect, in the thirteenth aspect, the parameter information is zoom information indicating a zoom ratio of the imaging means, and the size information is
The distance is calculated based on the zoom information, the distance information indicating the distance from the imaging unit to the object to be imaged, and the relative size information indicating the relative size shown in the imaging screen imaged by the imaging unit. .

According to a fifteenth aspect of the present invention, in the fourteenth aspect, the distance information is calculated based on autofocus information for controlling autofocus of the imaging means.

In a sixteenth aspect based on the fourteenth aspect, the environment information acquiring means includes a distance measuring means for measuring a distance from the imaging means to the object to be imaged. , Based on the measurement output of the distance measuring means.

According to a seventeenth aspect of the present invention, in the twelfth aspect of the invention, the environment information acquiring means measures altitude information indicating an altitude of an acquisition position of a captured image acquired by the imaging means. , And wherein the environmental information is altitude information measured by the altitude measuring means.

According to an eighteenth aspect of the present invention, in the twelfth aspect of the present invention, the environment information obtaining means includes clock means for counting date and time information indicating the date and time of acquisition of the captured image acquired by the imaging means. The environmental information is date and time information counted by the clock means.

According to a nineteenth aspect of the present invention, in the twelfth aspect, the environment information acquiring means includes a position measuring means for measuring position information indicating an acquisition position of a captured image acquired by the imaging means. And wherein the environment information is position information measured by the position measurement means.

According to a twentieth aspect of the present invention, in the twelfth aspect, the environment information is area information indicating an acquisition area of a captured image acquired by the imaging means.

According to a twenty-first aspect of the present invention, there is provided an image search method for searching an image database based on a predetermined search algorithm using an image to be searched as a key and acquiring information on the image to be searched, wherein a plurality of search algorithms are stored. A search algorithm suitable for the search is selected from a plurality of search algorithms stored in the algorithm database when searching the image database using the search target image as a key, and the selected search algorithm is selected. Is used to search the image database.

According to a twenty-second aspect of the present invention, in the twenty-first aspect, the selection of the search algorithm is performed in accordance with the type of an object included in the search target image.

According to a twenty-third aspect of the present invention, in the twenty-first aspect, the selection of the search algorithm is performed in accordance with a feature of an object included in the search target image.

According to a twenty-fourth aspect of the present invention, there is provided an image search apparatus which searches an image database based on a predetermined search algorithm using a search target image as a key and acquires information on the search target image, and holds a plurality of search algorithms. An algorithm database, selecting means for selecting a search algorithm suitable for the search from a plurality of search algorithms stored in the algorithm database when searching the image database using the search target image as a key, Search means for searching the image database using the selected search algorithm.

According to a twenty-fifth aspect of the present invention, in the twenty-fourth aspect, the selection means selects the search algorithm in accordance with a type of an object included in the search target image.

According to a twenty-sixth aspect of the present invention, in the twenty-fourth aspect, the selecting means selects the search algorithm in accordance with a feature of an object included in the search target image.

[0034]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

FIG. 1 is a schematic block diagram showing an embodiment of an image retrieval apparatus according to the present invention.

In FIG. 1, the camera unit 1 is, for example, CC
An image including an object for which the user wants to know information is captured by a small camera unit using D.

The image storage section 2 is a memory for storing image data. The camera unit 1 starts photographing in accordance with a user's instruction, and the user adjusts the photographing direction and the photographing range, and operates to take an image including the target into the image storage unit 2 as a photographed image.

The display unit 14 is composed of, for example, a display device using an LCD, and provides various information to the user by displaying image data and additional information stored in the image storage unit 2.

The composition instruction storage section 30 stores information indicating an appropriate composition, which is general or according to the type of object.

The moving object sensor 31 detects a moving object using the ultrasonic drap effect, and a microphone 32
Is to capture surrounding sounds.

The automatic capture control unit 33 controls automatic capture of an image from the camera unit 1 to the image storage unit 2 according to the output of the moving object sensor 31 or the microphone 32.

The ultrasonic distance sensor 36 measures the distance to the object using ultrasonic waves.
Numeral 7 is for measuring the distance to the target using a laser beam.

The photographing magnification calculating section 38 calculates the magnification of the photographed image using a part or all of the optical system information from the camera section 1 or the output from the ultrasonic distance sensor 36 or the laser distance measuring section 37.

The composition changing unit 3 judges the composition of the target from the captured image in the image storage unit 2 captured by the camera unit 1, enlarges, moves, and rotates the target image, and
The image data is converted into an optimal composition image having a composition most easily compared with the above image data.

The pen input unit 4 includes a touch panel and a pen-type pointing device mounted on the display screen of the display unit.

The image extracting section 5 cuts out a portion of interest of the target image using the optimum composition image converted by the composition changing section 3 and the coordinates on the image specified by the pen input section 4, and removes the background from the target image. Is extracted.

The target image converter 6 converts the target image data into 2
The target binary image is generated by the binarization.

The image database 11 stores a plurality of data images necessary for searching for an object. The image to be stored is a natural image, and additional information related to each image is also stored at the same time.

The data image converter 12 converts the image stored in the image database 11 into a data binary image in the same manner as the target image converter 6.

The image comparison unit 10 includes a target binary image generated from an image obtained by capturing a target by a user and the image database 1.
A similarity is calculated by comparing the image in 1 with the converted binary image.

The result display control unit 13 extracts the additional information on the image database 11 corresponding to the search result, and outputs it to the display unit 14.

The control section 100 connects each section of the image search apparatus, transfers data and cooperates with processing.

FIG. 2 is a front view showing an example of the appearance of the image retrieval apparatus shown in FIG. As described above, this image search device has the camera unit 1 that captures an image of an object for which the user wants to know information.
Denotes an objective lens of the camera unit 1 and 1b denotes a shutter button of the camera unit 1.

FIG. 3 is a rear view showing the appearance of the image retrieval apparatus shown in FIG.

As shown in FIG. 3, a display screen 14a of the display unit 14 shown in FIG. 1 is formed on the back of the image search device. In FIG. 3, 14b is a scroll button for scrolling the display screen displayed on the display screen 14a.

Next, the operation of the image retrieval apparatus shown in FIG. 1 will be described with reference to FIG.

FIG. 4 is a flowchart showing the flow of the processing of the image retrieval apparatus shown in FIG.

First, the camera section 1 starts photographing by a user's operation. The image in the range photographed by the camera unit 1 is displayed on the display unit 14. Thus, the user can easily determine whether the object to be known is within the shooting range.

The user looks at the image displayed on the display unit 14, and when the image captured by the camera unit 1 is within the target range, the user can see FIG.
And a still image displayed on the display unit 14 by pressing the shutter button 1b shown in FIG.
Is taken into the image storage unit 2. FIG. 5 shows an example of a captured image of the camera unit 1 at this time.

Further, the photographing magnification calculating section 38 includes the camera section 1
Calculates the distance to the target from the information of the optical system such as the auto focus and the lens characteristics output from the microcomputer or the output of the ultrasonic distance sensor 36 or the output of the laser distance measuring unit 37, and calculates the calculated distance to the target and the lens From the information of the optical system such as characteristics, the ratio of the size of the target image in the captured image to the actual target, that is, the magnification, is calculated.

The magnification calculated by the photographing magnification calculator 38 is sent to the image comparator 10 by the controller 100 and used as a filter when searching the image database 11 for images, as will be described in detail later.

Next, the composition changing unit 3 judges the composition from the photographed image on the image storage unit 2 and changes the composition.

That is, when the object which is the target image picked up by the camera section 1 is not at the center of the screen, the composition is changed to move the object to the center of the screen.

When the object which is the target image captured by the camera unit 1 is smaller than the size of the entire screen, the composition is changed so that the object is enlarged so as to fit in the size of the screen.

The composition changing section 3 performs such processing on the image in the image storage section 2 to generate an optimum composition image. FIG. 6 shows an example of the optimum composition image generated by the composition changing unit 3.

The optimum composition image changed by the composition changing unit 3 is displayed on the display unit 14. The user uses this display unit 14
The user inputs and specifies the coordinates of the object that he or she wants to know while viewing the optimum composition image displayed in (1).

Here, when the pen input unit 4 is a pen-type input device, the pen-type input device is provided with a touch panel mounted on a display screen (LCD) of the display unit 14. By applying pressure to the touch panel with this pen-type input device, it is possible to input position information of a portion of interest. At this time, the locus that the pen has passed is imposed on the screen of the display unit 14 and is displayed in a manner that is easy for the user to understand.

Here, the line surrounding the object does not have to be continuous, and if it is interrupted, a line connecting the nearest end points is virtually assumed.

The image extracting unit 5 uses the outline information of the object input by the pen input unit 4 to extract only the image of the object excluding the background image. This processing is performed on the image storage unit 2, and the obtained image is the target image.
An example of an image specified by the pen input unit 4 is shown in FIG.
FIG. 8 shows an image of a background portion, and FIG. 9 shows an example of a target image extracted by the image extracting unit 5.

The target image converter 6 converts the target image extracted by the image extractor 5 into a binary image by providing a reference value based on brightness. This binary image is called a target binary image, and an example of the target binary image is shown in FIG. In this image search device, the target binary image is used as a search key.

That is, the image database 11 stores the natural images of the target plants and animals and related information such as the names and characteristics of the images. FIG. 11 shows an example of a data image stored in the image database 11.

The image to be input to the image comparing section 10 is a binary image. Therefore, in order to make a comparison in the image comparison unit 10, the data image on the image
Need to change to value image.

The data image 2 on the image database 11
The conversion into the value image is performed by the data image conversion unit 12. The data image conversion unit 12 performs the same image processing as that performed by the target image conversion unit 6 on the data image to obtain a data binary image. FIG. 12 shows an example of the data binary image converted by the data image conversion unit 12.

The data binary image converted by the data image conversion unit 12 is sequentially sent to the image comparison unit 10 and compared with the target binary image converted by the target image conversion unit 6. At this time, the magnification information stored in the search auxiliary information storage unit 39 is used as a filter at the time of image search.

For example, the target image and the image database 11
When the magnification information of the data image registered in the... Has a difference of 10 times or more, the comparison is not performed on the image data. In this way, for example, animals and insects can be classified with relatively high reliability.

In the comparison in the image comparing section 10, two binary images are superimposed, and a judgment is made based on a matching ratio of data. Here, when the ratio of matching data is large, it is determined that the similarity between the two images is high.

This processing is sequentially performed in the same manner on the images to be searched in the image database 11, and the image to be obtained is determined based on the result. That is, the image having the highest similarity is finally selected.

The result display control unit 13 extracts a natural image (data image) and related information from which the binary image of data having a high degree of similarity is based, from the image database 11 and displays it on the display unit 14.

In this way, the user can easily and instantly search for information on an object by photographing an unknown object, as if the user automatically checked the picture book.

Next, the search processing of the image search apparatus will be described in more detail with reference to the flowchart of FIG.

In FIG. 13, first, an object is photographed with an appropriate composition using the camera unit 1 shown in FIG. 1 (step 1).
01), a captured image is obtained (step 102).

Next, it is checked whether the composition of the photographed image is appropriate (step 103). If the composition is not appropriate, the composition changing unit 3 corrects the composition of the acquired photographed image (step 104) to obtain an optimal composition image (step 104). Step 105).

Thereafter, the display unit 14 is displayed using the pen input unit 4.
The outline of the target of the display image displayed in is displayed with a pen, and the target is cut out (step 106), and the target image is extracted by the image extraction unit 5 (step 107).

The target image extracted by the image extractor 5 is converted into a binary image by the target image converter 6 (step 108).
A target binary image is obtained (step 109).

On the other hand, the size of the object is calculated from the photographed image obtained in step 102 (step 110), and the size data of the object is obtained (step 111).

Next, the image database 11 is searched (step 112), and it is checked whether or not the data image obtained by the search of the image database 11 is almost the same size as the object (step 113).

If it is determined that the size is not substantially the same as the size of the object, the process returns to step 112 to search for the next candidate.

If it is determined in step 113 that the size is almost the same as the size of the object, the data image
This data image is converted into a binary image (step 11).
4) Obtain a data binary image.

Next, the target binary image obtained in step 109 and the data binary image obtained in step 115 are compared by the image comparing section 10, and the similarity is calculated (step 116).

Then, it is checked whether or not the similarity calculated in step 116 is the highest similarity (step 117). If it is determined that the similarity is the highest, this similar image is obtained (step 122). Go to 119.

If it is determined in step 117 that the similarity is not the highest, it is checked whether it is the last data image (step 118). If it is not the last data image, the process returns to step 112, and the last data image is returned. If it is, the process proceeds to step 119.

In step 119, data is extracted from the obtained data image (step 119), and the retrieval result is obtained (step 102).

Then, this search result is displayed in the result display control unit 1.
3 to output an image to the display unit 14 (step 121),
This processing ends.

FIG. 14 shows another embodiment of the image retrieval apparatus of the present invention.

In the embodiment shown in FIG. 1, in the photographing magnification calculating section 38, 1) information on an optical system such as autofocus and lens characteristics outputted from the camera section 1 2) ultrasonic distance sensor 36 3) The output of the laser distance measuring unit 37 is used to calculate the distance to the target, and the image of the target in the captured image is calculated from the calculated distance to the target and information of the optical system such as lens characteristics. The ratio of the size of the object and the actual object, that is, the magnification is calculated.
The embodiment shown in FIG. 4 is different from the embodiment shown in FIG. 1 in that a target outer shape measuring device 16 is provided instead of the above configuration.

That is, in the embodiment shown in FIG. 14, the object outline measuring device 16 is configured to obtain information from the camera unit 1 and measure the object outline. FIG.
In FIG. 4, parts performing the same functions as the respective parts shown in FIG. 1 are denoted by the same reference numerals as those used in FIG. 1, and detailed description thereof will be omitted.

Referring to FIG. 14, an image retrieval apparatus according to this embodiment includes a camera unit 1, an image storage unit 2, a composition change unit 3,
Pen input unit 4, image extraction unit 5, target image conversion unit 6, image comparison unit 10, image database 11, data image conversion unit 12, result display control unit 13, display unit 14, target outline measurement device 16, control unit 100 It comprises.

Here, the target outer shape measuring device 16 calculates the outer size of the target from the distance to the target and the zoom magnification obtained from the camera unit 1 at the time of shooting the target, and obtains the image database
It is used as a filter when searching for an image.

In this configuration as well, for example, when the outer dimensions of the object and the outer information of the data image registered in the image database 11 have a difference of, for example, 10 times or more, the comparison is performed on the image data. In this configuration, animals and insects can be classified with relatively high reliability.

In the above-described embodiment, the image database is searched by using the outer shape information of the object. However, other information can be used.

For example, the temperature of a target may be measured using an infrared sensor or the like, and the type of the target may be automatically determined based on the measured temperature.

At this time, the measurement range may be limited to the center in order to limit the object, or the temperature distribution in the imaging range may be measured as in thermography, and the image of the object determined by image processing may be used. The temperature of the matching part may be output as the measurement result.

By using such information as a filter at the time of image search, it is not necessary to perform useless image search, and the accuracy and speed of the search are improved.

Further, an altitude sensor is added to the system,
The altitude at which the target exists may be obtained as additional information. This altitude information is also used as a filter at the time of image search.

With this configuration, it is not necessary to perform useless image search, and the accuracy and speed of the search can be improved.

Further, a clock may be added to this image search apparatus, and the date and time of the target shooting may be obtained as additional information. In the case of this configuration, the image database 11 stores data of animals and plants that can be seen in each season. In addition, data on changes in the form of plants with time, data on animals performing activities with time, and the like are accumulated.

Based on these data, date and time information at the time of shooting can be used as a filter at the time of image search.

Further, this image search device is provided with a GPS (Gl
oval Positioning System), the position where the target exists may be automatically detected using this GPS, and this may be obtained as additional information.

In this case, information on animals and plants that differ depending on the area is stored in the image database 11. The position information and the information are used as a filter at the time of image search.

The user inputs information on the environment where the object exists, for example, the sea, a pond, a mountain, and the like. For the input method, a selection is displayed on the display unit, and this is selected by a pen-shaped input device. This environment information can also be used as a filter when searching for image information.

As described above, by adding various additional information, it is not necessary to perform useless image search, and it is possible to improve search accuracy and speed.

FIG. 15 shows still another embodiment of the image retrieval apparatus of the present invention. In the embodiment shown in FIG. 15, the accuracy and speed of image search are improved by using the background image of the image captured by the camera unit 1. In FIG. 15, parts having the same functions as those shown in FIG. 1 are denoted by the same reference numerals as those used in FIG. 1, and detailed description thereof will be omitted.

Referring to FIG. 15, an image retrieval apparatus according to this embodiment includes a camera unit 1, an image storage unit 2, a composition change unit 3,
Pen input unit 4, image extraction unit 5, target image conversion unit 6, image comparison unit 10, image database 11, data image conversion unit 12, result display control unit 13, display unit 14, background image determination unit 15, control unit 100 It comprises.

Here, the background image judging section 15 estimates the surrounding environment from the background image excluding the target from the image taken by the camera section 1 and provides information to the image comparing section 10.

FIG. 16 is a flowchart showing a flow of processing of the image retrieval apparatus shown in FIG.

First, the camera section 1 starts photographing by a user's operation. The image in the range photographed by the camera unit 1 is displayed on the display unit 14. Thereby, the user can easily determine whether or not the object that the user wants to know is in the shooting range.

When the user looks at the image displayed on the display unit 14 and presses the button when the image captured by the camera unit 1 is within the target range, the still image displayed on the display unit 14 by pressing the button, ie, the camera The captured image of the unit 1 is taken into the image storage unit 2 (see FIG. 5).

Next, the composition change unit 3 judges the composition from the photographed image on the image storage unit 2 and changes the composition.

That is, when the object which is the target image picked up by the camera section 1 is not at the center of the screen, the composition is changed to move the object to the center of the screen.

When the object, which is the target image captured by the camera unit 1, is smaller than the size of the entire screen, the composition is changed so that the object is enlarged to fit in the size of the screen.

The composition changing section 3 performs such processing on the image in the image storage section 2 to generate an optimum composition image (see FIG. 6).

The optimum composition image changed by the composition changing unit 3 is displayed on the display unit 14. The user uses this display unit 14
The user inputs and designates the coordinates of the part of interest of the object that he or she wants to know while viewing the optimum composition image displayed in the above section by using a numeric keypad or the like.

Here, when the pen input unit 4 is a pen-type input device, the pen-type input device is provided with a touch panel mounted on a display screen (LCD) of the display unit 14. By applying pressure to the touch panel with this pen-type input device, it is possible to input position information of a portion of interest. At this time, the locus that the pen has passed is imposed on the screen of the display unit 14 and is displayed in a manner that is easy for the user to understand.

Here, the line surrounding the object does not have to be continuous, and if it is interrupted, a line connecting the nearest end points is virtually assumed.

The image extracting unit 5 uses the outline information of the target input by the pen input unit 4 to extract only the target image excluding the background image. This processing is performed on the image storage unit 2, and the obtained image is the target image (see FIG. 7).

The target image converter 6 converts the target image (see FIG. 9) extracted by the image extractor 5 into a target binary image by providing a reference value based on brightness (see FIG. 10). In this image search device, the target binary image is used as a search key.

The image to be input to the image comparing section 10 is a binary image. Therefore, in order to make a comparison in the image comparison unit 10, the data image on the image
Need to change to value image.

The data image 2 on the image database 11
The conversion into the value image is performed by the data image conversion unit 12. The data image conversion unit 12 performs the same image processing as that performed by the target image conversion unit 6 on the data image (see FIG. 11) to obtain a data binary image (see FIG. 12).

By the way, in the image retrieval apparatus of this embodiment, the surrounding environment is estimated by the background image determining section 15 for the image obtained by removing the object from the image captured by the camera section 1. FIG. 8 shows an example of a background image obtained by removing an object from the captured image.

The judgment by the background image judging section 15 is made based on the ratio of pixels.
It performs a rough classification such as light blue as sky.

The information obtained by the background image determination section 15 is used as a filter when performing an image search on the image database 11. That is, when the surrounding environment is determined to be a mountain or a grassland, the data image of the object existing on the seaside is excluded, and the processing by the image comparison unit 10 is omitted, thereby improving the efficiency of the image search.

The divided data binary images converted by the data image conversion unit 12 are sequentially sent to the image comparison unit 10 and compared with the target binary image converted by the target image conversion unit 6. In this comparison, two binary images are overlapped and determined based on a ratio matching the data. When the ratio of matching data is high, it is determined that the similarity between the two images is high.

This process is sequentially performed on all the images on the image database 11 to calculate the similarity. Finally, the image having the highest similarity is selected.

The result display control unit 13 extracts a natural image (data image) and related information from which the binary image of data having a high degree of similarity is based from the image database 11 and displays it on the display unit 14.

In the image search apparatus according to the above-described embodiment, information for searching the image database 11 is obtained from the background image by using the background image determination unit 15. The above information may be obtained without using it.

In this case, the second target is searched for the background image excluded by the target image converter 6. In this search method, the contour may be input with the assistance of the user, or the contour may be automatically extracted.

An image search similar to that for the first target is performed for the second target extracted in this manner.

Further, a third object is selected from the background image excluding the first and second objects, and the same operation is repeated recursively until no object is detected or a preset number of times.

As a result, an image search is performed for a plurality of objects, and information about the existing environment can be obtained for the second and subsequent objects.

Here, if the object is a plant, it is possible to obtain information on the area, altitude, season, etc. in which it lives. Further, the type may be different from the first target as long as the image search is possible. According to such a configuration, it is possible to obtain more accurate information when searching for the first target.

FIG. 17 shows an example of the data structure of the image database 11 employed in the image search device of each of the above embodiments.

The image database 11 of the image search device
As shown in FIG. 17, the data structure has a structure in which related items are linked around an image. Therefore, it is possible to know the name of the image and additional information (habitat, activity time, outer shape, etc.) from the image searched by the similar image search by the image search device.

Image Database 11 Using Additional Information
Is basically performed by the filtering process as described above. At the same time as capturing an unknown target, the size of the target is calculated using the output of a sensor or the like, and the image database 11 is searched using this size information as a filter.

That is, first, at the stage where an image is selected from the image database 11, the size is compared. For example, if there is a difference of 10 times or more, the two are completely different and no image search is performed. By this processing, useless search time can be omitted, search time can be reduced, and search efficiency can be improved.

The same can be performed for other information (additional information). In addition to the size information, it is possible to use the above-described altitude information, date, time, geographic information, environmental information, and the like. it can. Each piece of information can be registered by a numerical value, and the filter portion can easily perform filter processing by comparing the magnitudes.

FIG. 18 is a table showing an example of the additional information. FIG. 18 shows an example of 1) size, 2) altitude, 3) date, season, 4) geographic information, and 5) environmental information as items of additional information.

FIG. 19 shows still another embodiment of the image retrieval apparatus of the present invention. In the embodiment shown in FIG. 19, the accuracy and speed of image search are improved by accumulating a plurality of natural images for one target in the image database 11. Note that FIG.
In FIG. 2, parts having the same functions as the respective parts shown in FIG. 1 are denoted by the same reference numerals as those used in FIG. 1, and detailed description thereof will be omitted.

Referring to FIG. 19, an image retrieval apparatus according to this embodiment includes a camera unit 1, an image storage unit 2, a composition change unit 3,
Pen input unit 4, image extraction unit 5, target image conversion unit 6, image comparison unit 10, image database 11, data image conversion unit 12, result display control unit 13, display unit 14, control unit 100
It comprises.

Here, the image database 11 stores related information such as natural pictures of plants and animals and their names and characteristics necessary for searching for an object. Further, the image database 11 stores a plurality of natural images for one target.

The plurality of natural images in this case are data images of one object viewed from a plurality of different directions. These natural images are converted into data binary images by the data image conversion unit 12, and are sequentially compared with the target binary images by the image comparison unit 10.

Increasing the number of data images to be stored for one object means that data images viewed from as many directions as possible are prepared, so that the image comparison unit 10 can establish data coincidence. To increase.

As the data image stored in the image database 11, a data image obtained by dividing an object into baht having a specific meaning (hereinafter referred to as a "data divided image") is used. An example of this data division image is shown in FIGS.

Here, FIG. 20 is a data division image composed of only the target flower portion, and FIG. 21 is a data division image composed of the target only leaf portion. Even if only the target flower part is obtained as the target image due to the conditions at the time of shooting, if the data division image of the flower part is stored in the image database 11, the search can be easily performed. .

When a plurality of data divided images such as a flower data divided image and a leaf data divided image are stored in the image database 11, an efficient search can be performed by giving a priority to each of the divided data. become. For example, it is assumed that a plant A is characterized by a flower shape, and a plant B is characterized by a leaf shape.

In this case, for A, the priority of the flower data division image is set high, and for B, the priority of the leaf data division image is set high. The user can enclose the target part he / she wants to specify with a pen-type device and specify attributes such as “flower” and “leaf”.

This input method need not be a pen-type input unit, but may be an input means such as a mouse, a keyboard, or a numeric keypad. When the user determines that the target plant has a characteristic in the flower and the attribute of the flower is designated as shown in FIG. 22, first, the data division image of the flower in the image database 11 is set as a search target. . Then, among the data division images of the flowers, those having higher priority are set as the search targets.

The image database 11 has not only data divided images but also data images showing the entire target. Since the image database 11 has the relationship information between the data division image and the data image indicating the whole, even if a target is found in the data division image, the data image indicating the whole can be obtained from the result and shown to the user. Will be possible.

FIG. 23 shows still another embodiment of the image retrieval apparatus of the present invention. In the embodiment shown in FIG. 23, the most suitable search algorithm can be selected and used according to search conditions such as the type of target and the characteristics of the target image, thereby improving the accuracy and speed of image search. It is intended. Note that FIG.
In FIG. 2, parts having the same functions as the respective parts shown in FIG. 1 are denoted by the same reference numerals as those used in FIG. 1, and detailed description thereof will be omitted.

In FIG. 23, the image retrieval apparatus according to this embodiment comprises a camera unit 1, an image storage unit 2, a composition change unit 3,
Pen input unit 4, image extraction unit 5, target image conversion unit 6, image comparison unit 10, image database 11, data image conversion unit 12, result display control unit 13, display unit 14, algorithm selection unit 20, algorithm database 21 , And a control unit 100.

Here, the algorithm database 21
Holds a plurality of algorithms for the image comparison unit 10 to calculate the similarity between images. The algorithm selection unit 20 selects an algorithm used in the image comparison unit 10 from the algorithm database 20. The calculation of the similarity between the images in the image comparison unit 10 is performed using the algorithm selected by the algorithm selection unit 21.

FIG. 24 is a flowchart showing a flow of processing of the image retrieval system shown in FIG. In FIG.
First, the camera unit 1 starts shooting by a user's operation. Since the photographed range is displayed on the display unit 14, the user can easily determine whether the object to be known is within the photographed range.

When the user is within the shooting range, the user presses a button to load the still image displayed on the display unit 14 into the image storage unit 2 (see FIG. 5).

Next, the composition changing section 3 judges the composition from the captured image on the image storage section 2 and changes the composition. If the object which is the target image is not at the center of the screen, the object is moved to the center of the screen.

When the size of the object is smaller than the size of the entire screen, the object is enlarged so as to fit in the size of the screen.

If it is difficult to take a picture from a position that is easy to compare with the database image due to the positional relationship between the user and the object, a part of the image near the object is subjected to linear conversion to simulate the image from the front. Generate

FIG. 25 shows the principle of this image processing. However, this method changes the direction in which the object can be seen in a simulated manner, so planes can be converted with high accuracy.However, if there are many irregularities on the surface, hidden parts may occur or the plane may be as simple as a plane. Since linear conversion becomes impossible, it is not always possible to generate an image completely from the front.

However, if the distance from the photographer to each point on the surface of the object can be known, the above processing becomes possible.
It is possible to obtain an image that can be used for image search.

Such processing is performed on the image in the image storage unit 2 to generate an optimal composition image (see FIG. 6). The optimum composition image changed by the composition changing unit 3 is displayed on the display unit 14.

The pen input unit 4 can input position information on the screen by mounting a touch panel on the LCD of the display unit 14 and applying pressure to the touch panel with a pen-type input device.

The user surrounds the outline of the object he wants to know with the pen-type input device while viewing the optimum composition image displayed on the display unit 14. At this time, the locus that the pen has passed is imposed on the screen, and is displayed in a manner that is easy for the user to understand. The surrounding lines need not be continuous, and if they are interrupted, a line connecting the nearest end points is virtually assumed (see FIG. 7).

The image extracting unit 5 uses the outline information of the object input by the pen input unit 4 to extract only the image of the object excluding the background image. This processing is performed on the image storage unit 2, and the obtained image is set as a target image (see FIG. 9).

The target image conversion unit 6 converts the target image into a target binary image by setting a reference value based on brightness (see FIG. 10). This target binary image is used as a search key.

The image database 11 stores natural images of plants and animals and related information such as names and characteristics required when searching for an object.

The image to be input to the image comparison unit 10 is a binary image. Therefore, it is necessary to change the data image (see FIG. 11) on the image database 11 to a binary image in order to perform the comparison.

The conversion into a binary image is performed by the data image conversion unit 12.
Do with. The data image conversion unit 12 performs the same image processing as that performed by the target image conversion unit 6 on the data image to obtain a data binary image (see FIG. 12).

The data binary images are sequentially transferred to the image comparing unit 10.
Is compared with the binary image to be sent. At this time, the algorithm selection unit 20 uses the algorithm selected from the algorithm database 21 according to the type of the target.

For example, when the type is plant, the comparison is 2
An algorithm is used in which two binary images are superimposed and a decision is made on the basis of a matching ratio of data. Here, when the ratio of matching data is large, it is determined that the similarity between the two images is high.

If the type is different, another algorithm is used. The correspondence between the type and the algorithm is prepared in a table format in the algorithm database 11.

The similar processing is sequentially performed on all the images on the image database 11 to calculate the similarity. Finally, the image having the highest similarity is selected.

The result display control unit 13 extracts a natural image (data image) and related information from which the data binary image having the high similarity is based from the image database 11 and displays it on the display unit 14.

By photographing an unknown object by the above-described operation, the user can easily and instantly extract information on the object, as in automatically checking a picture book.

In the above embodiment, an algorithm according to the type of the target is selected. However, the contour and color of the object are analyzed by the algorithm selecting unit 20,
A method may be adopted in which which measure can be most differentiated and an algorithm most suitable for the feature of the image is automatically selected.

In the above embodiment, the algorithm selecting unit 20 selects one algorithm according to the type and uses it for calculating the similarity. However, all the prepared algorithms may be used. In this case, the image comparison unit 10 performs image comparison for each prepared algorithm.

The result display control unit 13 sequentially displays the search results on the display unit 14. The display method is not limited to the method of sequentially displaying, but a plurality of display methods may be simultaneously performed if it can be determined. As a result, the user can easily select the image closest to the target, and can have highly reliable options.

Further, even when comparing images for each algorithm, there is a method of giving the algorithm name to the user as information as a result. The result display control unit 13 is a display unit 14
, The data image as the search result and the name of the algorithm used by the user are sequentially displayed.

Here, the display method is not limited to the sequential display method, but a plurality of display methods may be simultaneously performed if it can be determined. As a result, the user can easily select the image closest to the target, and can have highly reliable options. In addition, since the algorithm corresponding to the type of the target can be known at the same time as the search result, it is possible to pay attention to a specific algorithm when it is desired to check the target of the same type.

Furthermore, there is a method of displaying not only the algorithm name but also the characteristics of the algorithm. In this case, the algorithm database 21 stores the features of the algorithms together with the plurality of algorithms. Features describe information about what situations and areas are suitable for searching. The image comparison unit 10 performs image comparison for each prepared algorithm. The result display control unit 13 sequentially displays on the display unit 14 the data image as the search result and the names and characteristics of the used algorithms.

Here, the display method is not limited to the sequential display method, but a plurality of display methods may be simultaneously performed. As a result, the user can easily select the image closest to the target, and can have highly reliable options.

Further, since the algorithm corresponding to the type of the object can be known, it is possible to pay attention to a specific algorithm when it is desired to check the object of the same type. Furthermore, since the characteristics of the algorithm are described, even a user with little search experience can easily grasp which algorithm is most suitable, and can efficiently perform the search.

In the above example, the whole object is taken out by the user's instruction and compared with the data image, but it is not necessary to take out the whole object. For example, a flowering plant
In many cases, comparison of an image centering on petals excluding stems and leaves as an image of interest increases search reliability. If a leaf has characteristics, it is more reliable to refer to that part.

Therefore, similarly to the case where the user specifies the outline of the target, a plurality of portions to be focused on as the target are specified. Since flowers have relatively large differences in color, it is more efficient to search for images by color. In the case of leaves, the color is mainly green, but it is easy to differentiate by shape.

Therefore, a suitable algorithm is applied to the designated portion of interest. The algorithm database 21 has a selection rule for each algorithm, and selects an algorithm according to a portion of interest of a photographed image selected by a user. Then, the image comparison unit 10 compares the images using the algorithm selected by the algorithm selection unit 21.

In the above embodiment, the case where the input target image is mainly a plant is described. However, the present invention is not limited to this. The present invention is also applicable to a case where another image requiring retrieval is targeted. Of course, the same can be applied.

[0194]

As described above, according to the present invention,
The following effects are obtained.

1) Since the most suitable search algorithm can be used depending on the search conditions such as the type of the target and the characteristics of the target image, the accuracy of the search result can be improved.

2) By estimating the surrounding environment from the background image excluding the target in the photographed image and using the information for the search, the search can be performed efficiently.

3) External dimensions of the object obtained from the camera unit at the time of photographing, temperature information of the object obtained by mounting an infrared sensor or the like on this device, altitude information of the object obtained by mounting an altitude sensor, clock Date and time information at the time of shooting the target obtained by mounting the GPS, position information where the target exists obtained by mounting the GPS, and the environment where the target exists such as the sea, pond, and mountain can be obtained by the user inputting. By using information such as environmental information as a filter at the time of search, search accuracy can be improved.

4) A plurality of image databases are prepared. By selecting and using an image database to be searched according to various conditions at the time of search, search efficiency can be improved. In addition, for each image database, Since a search algorithm using the image database is used, the search can be performed efficiently.

5) From the characteristics of the optical system of the camera and the distance information (obtained by the auto-focus information of the camera unit at the time of photographing, the ultrasonic distance sensor, the laser distance sensor, etc.), the ratio of the size of the photographed image to the actual object is calculated By calculating and using it for the search, the accuracy of the search can be improved.

6) The user can specify a target portion of the target image by using a pen-type pointing device and the like, and can further specify its attribute. Therefore, the accuracy of the search result is improved by using the information. be able to.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram showing an embodiment of an image search device according to the present invention.

FIG. 2 is an exemplary front view showing an example of the external appearance of the image search device shown in FIG. 1;

FIG. 3 is a rear view of the appearance of the image search device shown in FIG. 2;

FIG. 4 is a flowchart showing a processing flow of the image search device shown in FIG. 1;

FIG. 5 is a view showing an example of a captured image of the camera unit shown in FIG. 1;

FIG. 6 is a view showing an example of an optimum composition image generated by the composition changing section shown in FIG. 1;

FIG. 7 is a view showing an example of an image specified by the pen input unit shown in FIG. 1;

FIG. 8 is a view showing an image of a background portion specified by the pen input unit shown in FIG. 1;

FIG. 9 is a diagram illustrating an example of a target image extracted by the image extracting unit illustrated in FIG. 1;

FIG. 10 is a view showing an example of a target binary image converted by the target image conversion unit shown in FIG. 1;

FIG. 11 is a diagram showing an example of a data image stored in the image database shown in FIG. 1;

FIG. 12 is a diagram showing an example of a data binary image converted by the data image conversion unit shown in FIG. 1;

FIG. 13 is a flowchart showing details of a search process of the image search device shown in FIG. 1;

FIG. 14 is a block diagram showing another embodiment of the image search device of the present invention.

FIG. 15 is a block diagram showing still another embodiment of the image search device of the present invention.

FIG. 16 is a flowchart showing the flow of processing of the image search device shown in FIG. 15;

FIG. 17 is a diagram showing an example of a data structure of an image database employed in the image search device according to the embodiment of the present invention.

FIG. 18 is a table showing an example of additional information used in the image search device.

FIG. 19 is a block diagram showing still another embodiment of the image search device of the present invention.

20 is a diagram showing a data divided image of a target flower portion stored in the image database of the image search device shown in FIG. 19;

FIG. 21 is a diagram showing a data division image of a target leaf portion stored in the image database of the image search device shown in FIG. 19;

FIG. 22 is a diagram showing a state where a flower portion of a target plant is designated in the image search device shown in FIG. 19;

FIG. 23 is a block diagram showing still another embodiment of the image search device of the present invention.

FIG. 24 is a flowchart showing the flow of processing of the image search device shown in FIG. 23;

FIG. 25 is a view for explaining the operation of the image search device shown in FIG. 23;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Camera part 2 Image storage part 3 Composition change part 4 Pen input part 5 Image extraction part 6 Target image conversion part 7 Attribute name input part 8 Color information input part 9 Priority input part 10 Image comparison part 11 Image database 12 Data image conversion Unit 13 result display control unit 14 display unit 15 background image determination unit 16 target contour measuring device 20 algorithm selection unit 21 algorithm database 30 composition instruction storage unit 31 moving object sensor 32 microphone 36 ultrasonic distance sensor 37 laser distance sensor 38 shooting magnification Calculation unit 39 Search auxiliary information storage unit 100 Control unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Toshiyuki Iwaoka 10 Okayama Todocho, Ukyo-ku, Kyoto-shi, Omron Corporation

Claims (26)

[Claims] 1. An image search method for searching an image database based on a search target image to obtain information on the search target image, comprising: inputting environment information at the time of obtaining the search target image together with the search target image; An image search method, wherein the image database is searched based on a target image and the environment information. 2. The image search method according to claim 1, wherein the search target image is a picked-up image obtained by an image pick-up unit. 3. The image search method according to claim 2, wherein the environment information is size information indicating an actual size of the captured image calculated based on parameter information of the imaging unit. 4. The parameter information is zoom information indicating a zoom ratio of the image pickup means, and the size information is distance information indicating a distance from the image pickup means to the object to be imaged and the image information of the image pickup means. 4. The image search method according to claim 3, wherein the calculation is performed based on relative size information indicating a relative size shown in the image screen captured by the means. 5. The image search method according to claim 4, wherein said distance information is calculated based on autofocus information for controlling autofocus of said imaging means. 6. The image search method according to claim 4, wherein the distance information is obtained based on a measurement output of a distance measurement unit that measures a distance from the imaging unit to the imaging target object. 7. The image search method according to claim 2, wherein the environment information is altitude information indicating an altitude of an acquisition position of a captured image acquired by the imaging unit. 8. The image search method according to claim 2, wherein said environment information is date and time information indicating an acquisition date and time of a captured image acquired by said imaging means. 9. The image search method according to claim 2, wherein the environment information is position information indicating an acquisition position of a captured image acquired by the imaging unit. 10. The image search method according to claim 2, wherein the environment information is area information indicating an acquisition area of a captured image acquired by the imaging unit. 11. An image search apparatus for searching an image database based on a search target image and obtaining information on the search target image, wherein: the search target image obtaining means for obtaining the search target image; and the search target image obtaining means Environment information acquisition means for acquiring environment information indicating an acquisition environment of the search target image, and the image database based on the search target image acquired by the search target image acquisition means and the environment information acquired by the environment information acquisition means. An image search device, comprising: search means for searching. 12. The image search device according to claim 11, wherein the search target image acquisition unit is an imaging unit that captures the search target image. 13. The environment information acquiring unit acquires, as the environment information, size information indicating an actual size of the captured image calculated based on parameter information of the imaging unit. Image retrieval device. 14. The parameter information is zoom information indicating a zoom ratio of the imaging unit. The size information is distance information indicating a distance from the imaging unit to an object to be imaged and the imaging information. 14. The image processing apparatus according to claim 13, wherein the calculation is performed based on relative size information indicating a relative size shown in the image screen captured by the means.
The image search device according to the above. 15. The image search device according to claim 14, wherein the distance information is calculated based on autofocus information for controlling autofocus of the imaging unit. 16. The environment information acquisition unit includes a distance measurement unit that measures a distance from the imaging unit to the imaging target object, wherein the distance information is obtained based on a measurement output of the distance measurement unit. The image search device according to claim 14, wherein: 17. The environment information acquiring means includes altitude measuring means for measuring altitude information indicating an altitude of an acquisition position of a captured image acquired by the imaging means, wherein the environmental information includes the altitude measuring means. 13. The image retrieval apparatus according to claim 12, wherein the information is altitude information measured in step (a). 18. The environment information acquisition unit includes: a clock unit that counts date and time information indicating an acquisition date and time of a captured image acquired by the imaging unit. The environment information is counted by the clock unit. 13. The image search device according to claim 12, wherein the information is date and time information. 19. The environment information acquisition unit includes: a position measurement unit that measures position information indicating an acquisition position of a captured image acquired by the imaging unit. The environment information is measured by the position measurement unit. 13. The image search device according to claim 12, wherein the obtained position information is obtained. 20. The image search device according to claim 12, wherein the environment information is area information indicating an acquisition area of a captured image acquired by the imaging unit. 21. An image search method for searching an image database based on a predetermined search algorithm using a search target image as a key and acquiring information on the search target image, wherein an algorithm database holding a plurality of search algorithms is provided. When searching the image database using the image to be searched as a key, a search algorithm suitable for the search is selected from among a plurality of search algorithms stored in the algorithm database, and the image database is searched using the selected search algorithm. An image search method characterized by searching. 22. The image search method according to claim 21, wherein the selection of the search algorithm is performed in accordance with a type of an object included in the search target image. 23. The image search method according to claim 21, wherein the selection of the search algorithm is performed in accordance with a feature of an object included in the search target image. 24. An image search device for searching an image database based on a predetermined search algorithm using a search target image as a key and acquiring information on the search target image, wherein: an algorithm database holding a plurality of search algorithms; Selecting means for selecting a search algorithm suitable for the search from a plurality of search algorithms held in the algorithm database when searching the image database using a target image as a key; and selecting the search algorithm selected by the selection means. And a search means for searching the image database using the image database. 25. The image search apparatus according to claim 24, wherein said selection means selects said search algorithm in accordance with a type of an object included in said search target image. 26. The image search apparatus according to claim 24, wherein said selection means selects said search algorithm in accordance with characteristics of an object included in said search target image.