JPH10254904A - Image retrieval method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily acquire a proper input image and to perform the image retrieval with high accuracy by referring to the composition designation information on the retrieval object image that is previously set when this object image is fetched. SOLUTION: When a camera part 1 fetches an image through the operation of an operator, the general information showing an appropriate composition stored in a composition designation part 30 is displayed as necessary. The part 30 also stores the information showing the most approximate composition that is different from the above general information in each type, and the composition showing information includes the character information and a template showing a rough external shape of the composition which are selectively shown at a display part 14 in response to the instruction of the user. The template is photographed at the camera part 1 and shown on the image that is displayed at the part 14. Thus, the user photographs the template via the part 1 so as to secure the higher coincidence between the template shape and an object and accordingly can easily secure an optimum photographing composition.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval method and apparatus, and more particularly, to an image retrieval method for photographing an unknown object.
The present invention relates to an image search method and apparatus capable of automatically obtaining information on an object from an image.

[0002]

2. Description of the Related Art Conventionally, when a desired image is retrieved from an image database, there is known a method of obtaining a desired image based on the degree of coincidence between two image data, that is, an input image and an image in the image database.

[0003] Further, a method of photographing a target and using the image as an input image for retrieval has been conventionally performed.

[0004]

However, when the composition of the input image is different from the composition of the image in the image database, there is a problem that the accuracy of the retrieval is reduced.

Accordingly, an object of the present invention is to provide an image search method and apparatus which can easily obtain an appropriate input image and thereby enable a highly accurate search.

[0006]

In order to achieve the above object, according to a first aspect of the present invention, there is provided an image search method for searching an image database based on a search target image and acquiring information on the search target image. At the time of capturing the target image, reference is made to composition instruction information that indicates a composition of the search target image set in advance.

Further, the invention of claim 2 is characterized in that, in the invention of claim 1, the search target image is an image picked up by an image pickup means.

According to a third aspect of the present invention, in the second aspect, the composition instruction information is a template indicating a desired shape corresponding to the search target image, and the template is imaged by the imaging means. A search target image having an appropriate composition is captured by superimposing the search target image on the captured image.

According to a fourth aspect of the present invention, in the third aspect, the template is compared with a captured image captured by the image capturing means, and the captured image is automatically captured based on the degree of coincidence. It is characterized by.

According to a fifth aspect of the present invention, there is provided an image search method for searching an image database based on a search target image to obtain information on the search target image. It is characterized in that, when the setting is made and the capturing condition is satisfied, the captured image captured by the imaging means is automatically captured as the search target image.

According to a sixth aspect of the present invention, in the fifth aspect of the invention, the capturing condition is such that the degree of coincidence between the image captured by the image capturing means and a preset template is higher than a predetermined value. It is characterized by touching.

Further, the invention of claim 7 is characterized in that, in the invention of claim 5, the capturing condition is that the detected sound volume of the surrounding sound exceeds a predetermined value.

[0013] Further, according to an eighth aspect of the present invention, in the fifth aspect of the present invention, the capturing condition is that an image captured by the image capturing means has changed by more than a predetermined amount. .

According to a ninth aspect of the present invention, in the fifth aspect of the present invention, the image capturing means continuously captures an image of an image capturing target and inputs a plurality of image capturing screens. An imaging screen that satisfies the capturing condition is selectively captured.

According to a tenth aspect of the present invention, there is provided an image search apparatus for searching an image database based on a search target image to obtain information on the search target image, wherein the imaging means captures an image of the search target, and the imaging means A composition instruction information setting unit that presets composition instruction information for instructing a composition of a captured image captured by the camera, and an image captured by the imaging unit with reference to the composition instruction information set by the composition instruction information setting unit. Image capturing means for capturing as a search target image.

According to an eleventh aspect of the present invention, in the tenth aspect, the composition instruction information setting means sets the composition instruction information corresponding to the type of the search object. .

According to a twelfth aspect of the present invention, in the tenth aspect of the present invention, the composition instruction information is a template indicating a shape corresponding to the search target.

According to a thirteenth aspect of the present invention, in the twelfth aspect, the image capturing means compares the template with a captured image captured by the image capturing means,
It is characterized in that the captured image is automatically taken in based on the degree of coincidence.

According to a fourteenth aspect of the present invention, in the twelfth aspect of the invention, the image capturing means includes a composition changing means for changing a composition of the captured image to be compared with the template. I do.

According to a fifteenth aspect of the present invention, there is provided an image search apparatus for searching an image database based on a search target image and obtaining information on the search target image, wherein: Capture condition setting means for setting capture conditions of a captured image captured by
Image capturing means for automatically capturing a captured image captured by the image capturing means as the search target image when the capturing condition set by the capturing condition setting means is satisfied. .

According to a sixteenth aspect of the present invention, in the fifteenth aspect, the capture condition set by the capture condition setting means is a combination of a captured image captured by the image capture means and a preset template. The matching degree is higher than a predetermined value.

The invention according to claim 17 is the invention according to claim 15, further comprising sound detection means for detecting surrounding sounds, wherein the capture condition set by the capture condition setting means is such that: It is characterized in that the detected sound volume of the surrounding sounds detected by the sound detecting means has exceeded a predetermined value.

The invention according to claim 18 is the invention according to claim 15, further comprising image change detecting means for detecting a change amount of the image picked up by the image pickup means,
The capture condition set by the capture condition setting means is that the detection output of the image change detection means exceeds a predetermined value.

According to a nineteenth aspect of the present invention, in the fifteenth aspect, the imaging means continuously captures an image of an imaging target and inputs a plurality of imaging screens.
Selective capturing means for selectively capturing an imaging screen that satisfies the capturing condition from among the plurality of imaging screens is provided.

[0025]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

FIG. 1 is a schematic block diagram showing an embodiment of an image retrieval apparatus according to the present invention.

In FIG. 1, a camera unit 1 is, for example, a CC.
An image including an object for which the user wants to know information is captured by a small camera unit using D.

The image storage section 2 is a memory for storing image data. The camera unit 1 starts photographing in accordance with a user's instruction, and the user adjusts the photographing direction and the photographing range, and operates to take an image including the target into the image storage unit 2 as a photographed image.

The display section 14 is composed of, for example, a display device using an LCD, and provides various information to the user by displaying the image data and the additional information stored in the image storage section 2.

The composition instruction storage unit 30 stores information indicating an appropriate composition, which is general or according to a subject type. Specifically, the composition instruction storage unit 30 stores general information indicating an appropriate composition and information indicating the most suitable composition different for each type, and the information indicating the composition includes character information and composition information. Are stored as templates indicating the rough outlines of them, and one or both of them are selectively displayed in accordance with a user's instruction.

The moving object sensor 31 detects a moving object using the ultrasonic drap effect, and a microphone 32
Is to capture surrounding sounds.

The automatic capture control unit 33 controls automatic capture of images from the camera unit 1 to the image storage unit 2 according to the output of the moving object sensor 31 or the microphone 32.

The image change detection unit 34 constantly captures and monitors the image output from the camera 1, compares the image with a predetermined image before a predetermined time or a predetermined reference image, and determines the amount of difference between the two images. Output.

The template coincidence detecting unit 35 compares the template stored in the composition instruction information storage unit 30 and indicating the optimum composition selected by the user's instruction with the image output from the camera 1, and calculates the template coincidence. Output.

The composition changing section 3 judges the composition of the object from the captured image in the image storage section 2 captured by the camera section 1, enlarges, moves, and rotates the image of the object.
The image data is converted into an optimal composition image having a composition most easily compared with the above image data.

The pen input unit 4 includes a touch panel and a pen-type pointing device mounted on the display screen of the display unit.

The image extracting unit 5 cuts out a portion of interest of the target image using the optimal composition image converted by the composition changing unit 3 and the coordinates on the image specified by the pen input unit 4, and removes the background from the target image. Is extracted.

The target image converter 6 converts the target image data into two.
The target binary image is generated by the binarization.

The image database 11 stores a plurality of data images necessary for searching for an object. The image to be stored is a natural image, and additional information related to each image is also stored at the same time.

The data image converter 12 converts the image stored in the image database 11 into a data binary image in the same manner as the target image converter 6.

The image comparing section 10 includes a target binary image generated from an image of the object photographed by the user and the image database 1.
A similarity is calculated by comparing the image in 1 with the converted binary image.

The result display control unit 13 extracts additional information on the image database 11 corresponding to the search result, and outputs it to the display unit 14.

The control section 100 connects each section of the image search apparatus, transfers data and cooperates with processing.

Next, the operation of the image retrieval apparatus shown in FIG. 1 will be described with reference to FIG.

FIG. 2 is a flowchart showing the flow of the processing of the image retrieval apparatus shown in FIG.

In FIG. 2, first, the camera unit 1 starts photographing by a user's operation, and performs an image capturing process (step 101). The image captured by the image capturing process is displayed on the display unit 14. Thereby, the user can easily determine whether or not the object that the user wants to know is in the shooting range.

That is, when photographing by the camera unit 1, the composition instruction storage unit 3 is operated by a user's instruction as necessary.
The general information indicating the appropriate composition stored in 0 is displayed. As a result, the user can easily perform shooting with an appropriate composition. When the user inputs a rough type of a target, a composition most suitable for the target is displayed.

As described above, the composition instruction storage unit 30 stores general information indicating an appropriate composition and information indicating the most suitable composition that differs for each type. A template indicating a rough outline of the character information and the composition is stored. Then, one or both of them are selectively displayed on the display unit 14 according to the instruction of the user.

Here, when the composition instruction is displayed by using a template showing a rough outline, the template is photographed by the camera unit 1 and is displayed on the image displayed on the display unit 14 so as to be superimposed on the image. By taking an image with the camera unit 1 so that the shape of the template matches the object as much as possible, it is possible to easily perform the imaging with the optimal composition.

When the user determines that the object is in the photographing range or in an appropriate composition position, the user can fetch the still image displayed on the display unit 14 into the image storage unit 2 by an operation method such as pressing a button or the like. Alternatively, the automatic capture control unit 33
, The still image displayed on the display unit 14 is automatically loaded into the image storage unit 2. FIG. 3 shows an example of a still image captured in the image storage unit 2 in this manner.

Next, the composition changing unit 3 judges the composition from the photographed image on the image storage unit 2 and performs a composition changing process for changing the composition (step 102).

That is, when the object which is the target image picked up by the camera unit 1 is not at the center of the screen, the composition is changed to move the object to the center of the screen.

When the object which is the target image captured by the camera unit 1 is smaller than the size of the entire screen, the object is enlarged so as to fit in the size of the screen, and the rotational position of the object is appropriately adjusted. If not,
The rotation processing and the left / right inversion or the up / down inversion are performed on the object image so as to be in an appropriate rotation position.

If it is difficult to photograph from a position that is easy to compare with the database image due to the positional relationship between the user and the object, the photographed image is subjected to deformation transformation such as trapezoidal deformation, and is simulated from the front. Generate an image of. Such processing is performed on the image in the image storage unit 2. FIG. 4 shows an example of the optimum composition image generated by the composition changing unit 3.

The optimal composition image changed by the composition change process is displayed on the display unit 14. The user uses this display unit 14
The user inputs and specifies the coordinates of the object that he or she wants to know while viewing the optimum composition image displayed in (1).

Here, when the pen input unit 4 is a pen-type input device, the pen-type input device includes a touch panel mounted on a display screen (LCD) of the display unit 14. By applying pressure to the touch panel with the pen-type input device, a pen input process of inputting position information of a target portion can be performed (step 103). At this time, the locus that the pen has passed is imposed on the screen of the display unit 14 and is displayed in a manner that is easy for the user to understand.

Here, the line surrounding the object does not have to be continuous, and if it is interrupted, a line connecting the nearest end points is virtually set.

The image extracting unit 5 executes an image extracting process for extracting an image of only the target, excluding the image of the background part, using the outline information of the target input by the pen input unit 4 (step 104). . This image extraction processing is performed on the image storage unit 2, and the obtained image is the target image. FIG. 5 shows an example of an image specified by the pen input unit 4, and FIG. 6 shows an example of a target image extracted by the image extraction processing.
Shown in

The target image conversion unit 6 performs a target image conversion process of converting the target image extracted by the image extraction unit 5 into a binary image by providing a reference value based on brightness (step 10).
5). The binary image converted by the target image conversion process is called a target binary image, and an example of the target binary image is shown in FIG.
Shown in

In this image retrieval apparatus, the target binary image is used as a retrieval key. That is, the image database 11 accumulates related information such as natural images of target plants and animals and their names and characteristics. FIG. 8 shows an example of a data image stored in the image database 11.

The image to be input to the image comparison unit 10 is a binary image. Therefore, it is necessary to change the data image on the image database 11 to a binary image in order for the image comparison unit 10 to perform the image comparison.

The data image 2 on the image database 11
The conversion into the value image is performed by a data image conversion process by the data image conversion unit 12 (step 108).

The data image conversion section 12 performs the same image processing as the target image conversion processing performed by the target image conversion section 6 on the data image to obtain a data binary image. FIG. 9 shows an example of the data binary image converted by the data image conversion unit 12.

The data binary images converted by the data image conversion unit 12 are sequentially sent to the image comparison unit 10 and subjected to an image comparison process for comparing with the target binary image converted by the target image conversion unit 6. (Step 106).

In the image comparison processing in the image comparison section 10, for example, two binary images are superimposed and determined based on the ratio of coincidence with data. It is determined that the similarity is high, but another method can be adopted in consideration of the comparison accuracy and the time and cost required for processing.

This image comparison processing is performed in the image database 1
1 is sequentially performed in the same manner on the images to be searched,
The desired image is determined based on the result. That is, the image having the highest similarity is finally selected.

The result display control unit 13 executes a result display process of extracting a natural image (data image) and related information from which the binary image having the high similarity is based on the natural image (data image) and displaying it on the display unit 14. (Step 10
7).

In this way, the user can easily and instantly search for information on an object by photographing an unknown object, as if the user automatically checked the picture book.

FIG. 10 shows the details of the image capturing process in step 101 of FIG. In the image capturing process of step 101 in FIG. 2, 1) direct operation 201 by the user 2) change of the image 202 3) detection of the moving body 203 4) surrounding sound 204 5) difference between the RGB average values of the images inside and outside the template 205 6) Brightness average difference 206 between images inside and outside the template 206 7) Hue average difference 207 between images inside and outside the template 207 8) Number of contour pixels 208 near template boundary 208 or when the combination condition is satisfied Done.

Here, the user's direct operation 201 is a time when the user operates in response to a user's instruction and performs a capturing operation in accordance with a preset operation instruction. This is when the signal from the image change detection unit 34 is larger than the preset motion reference value, and the detection 203 of the moving object means that the signal output from the moving object sensor 031 is the movement of the motion larger than the preset reference value. The time when object detection is indicated, and the surrounding sound 204 is when the signal from the microphone 32 is higher than a preset reference volume.

The conditions 5) to 8) are satisfied when the signal from the template match detecting section 35 is larger than a predetermined matching degree.
5, the pixels of the image from the camera unit 1 are classified into a pixel group belonging to the outside of the template boundary stored in the composition instruction storage unit 30 and a pixel group belonging to the inside thereof, and the red, green, and blue pixels of each pixel in both groups are classified. The difference (205) of the average value of each color component, the difference (206) of the average value of lightness, or the difference (207) of the average value of hue is used as the degree of coincidence, or contour detection is performed on the image from the camera unit 1. The number (208) of pixels within a certain distance range from the template boundary among the pixels of the applied contour image is defined as the matching degree.

In the condition selection processing 209, any one of the above conditions 1) to 8) or a combination thereof is selected, and when the condition selected in the condition selection processing 209 is satisfied, the automatic capture control unit A control signal is output from 33 to instruct the image storage unit 2 to capture an image captured by the camera unit 1, whereby the image captured by the camera unit 1 is captured by the image storage unit 2.

As described above, by automatically performing image capturing, appropriate capturing can be performed even when it is difficult for the user to directly operate and perform appropriate image capturing.

Further, in the above configuration, a plurality of images can be set in accordance with a user's instruction at predetermined time intervals or each time a predetermined capture condition is satisfied.
A predetermined number of images can be stored in the image storage unit 2. In that case, the same conditions as in the case of the single image capture described above are set and used as the conditions for starting continuous capture.

FIG. 11 is a diagram showing the flow of the process of selecting a plurality of images.

In FIG. 11, a plurality of images are selected by: 1) direct operation 301 by a user 2) RGB average difference 302 between images inside and outside the template 3) brightness average difference 303 between images inside and outside the template 4) Template Hue average value difference 304 between the inner and outer images 5) The number of contour pixels 305 near the template boundary 305 or any combination thereof is satisfied.

Here, the direct operation 301 by the user is started when the user performs an operation and performs a capture operation according to a preset operation instruction.

The conditions 2) to 5) are satisfied when the signal from the template match detecting section 35 is larger than a preset matching degree.
5, the pixels of the image from the camera unit 1 are classified into a pixel group belonging to the outside of the template boundary stored in the composition instruction storage unit 30 and a pixel group belonging to the inside thereof, and the red, green, and blue pixels of each pixel in both groups are classified. The difference between the average value of each color component (302), the difference between the average values of lightness (303), or the difference between the average values of hues (304) is used as the degree of coincidence, or contour detection is performed on the image from the camera unit 1. The number (305) of pixels within a certain distance range from the template boundary among the pixels of the applied contour image is defined as the matching degree.

In the condition selection process 306, a selection process is performed under one of the above conditions 1) to 5) or a combination thereof, and when the condition selected in the condition selection process 209 is satisfied, the automatic capture control is performed. A control signal for instructing the image storage unit 2 to capture an image captured by the camera unit 1 is output from the unit 33, whereby a desired image among the plurality of images 307 captured by the camera unit 1 is converted to a captured image. 309.

The captured plural images are stored in the image storage unit 2.
Is displayed on the display unit 14 sequentially or simultaneously according to the user's operation, and one or more of them are selected by the user or the automatic capture control unit 3
3 is selected.

The automatic capture control unit 33 obtains a template matching degree between the template specified by the user and each image by using the template matching detecting unit 35,
One of the data images having the highest template matching degree or a plurality of the images is automatically selected in the descending order.

As described above, even when it is difficult for the user to directly perform the capturing operation at the appropriate timing by capturing the continuous images, the image capturing can be performed at the appropriate timing.

The template coincidence output from the template coincidence detector 35 is obtained by dividing pixels of the image from the camera 1 into a pixel group belonging to the outside of the template boundary and a pixel group belonging to the inside thereof. The difference between the average values of the red, green, and blue color components, the difference between the average values of the lightness and the difference between the average values of the hues, or the contour image obtained by performing the contour detection on the image from the camera 1 , The number of pixels within a certain distance range from the template boundary may be determined as the degree of coincidence.

FIG. 12 is a flowchart showing details of the image capturing process shown in FIG.

In FIG. 12, when the image capturing process is started (step 601), first, a condition is selected (step 602). This condition selection is based on the conditions of the above-described image capture processing, that is, 1) direct operation by the user 2) change of the image 3) detection of the moving object 4) ambient sound 5) difference in RGB average value of images inside and outside the template 6 This is performed by selecting any one of the following:) the average brightness difference between the images inside and outside the template, 7) the average hue difference between the images inside and outside the template, and 8) the number of contour pixels near the template boundary.

Next, a start instruction is given (step 60).
3) It is checked whether or not the condition selected in step 602 is satisfied (step 604). Here, if the condition is not satisfied, the process returns to step 604, and waits until the condition is satisfied. If the condition is satisfied in step 604, image capture is executed (step 605), and this process ends when the image capture is completed. (Step 606).

FIG. 13 is a flowchart showing details of the image selection processing shown in FIG.

In FIG. 13, when this image selection processing is started (step 701), first, a plurality of images to be selected are fetched (step 702).
It is checked whether or not this image selection is automatic selection (step 703).

If it is determined that this image selection is not an automatic selection, the plurality of images taken in step 702 are displayed in a list on the display unit 14 (step 704), and the user selects a desired image. Is performed (step 705), the image selection processing ends (step 7).
09).

When it is determined in step 703 that the selection is automatic, the matching degree is calculated by the template matching degree detecting section 35 (step 706).

The calculation of the degree of coincidence by the template coincidence detecting section 35 includes: 1) RGB average value difference between images inside and outside the template 2) Brightness average value difference between images inside and outside the template 3) Hue average value difference between images inside and outside the template 4) This is performed by using any one of the number of contour pixels near the template boundary.

Next, rearrangement is performed based on the degree of coincidence calculated in step 706 (step 707), and an image having the highest degree of coincidence is selected by this rearrangement (step 708), and this image selection processing ends. (Step 70
9).

FIG. 14 shows the composition instruction storage unit 3 shown in FIG.
9 shows an example of the structure of the composition instruction information stored in “0”.

As is apparent from FIG. 14, the structure of the composition instruction information stored in the composition instruction storage unit 30 is such that the composition instruction character information and the template ID are associated with each other for each type of object. Indicates a storage position of image information indicating the shape of the template.

FIG. 15 shows a data structure of data stored in the image database 11 shown in FIG.

As is clear from FIG. 15, the data structure of the data stored in the image database 11 is such that a plurality of pieces of additional information and image IDs are associated with each other for each target name, and the image ID is Shows the storage position of.

FIG. 16 is a schematic block diagram showing another embodiment of the image retrieval apparatus according to the present invention. In FIG. 16, portions that perform the same functions as those of the image search device shown in FIG. 1 are denoted by the same reference numerals as those used in FIG. 1, and detailed description thereof will be omitted.

In FIG. 16, the image retrieval apparatus includes a camera unit 1, an image storage unit 2, a composition change unit 3, a pen input unit 4, an attribute database 40, a designated contour control unit 41, a color information extraction unit 42, line drawing information Extraction unit 43, image integration extraction unit 4
4, target image conversion unit 6, image comparison unit 10, image database 11, data image conversion unit 12, result display control unit 1
3, a display unit 14, a template database 46, and a control unit 100.

Here, the attribute database 40 stores attribute names of components required for extracting a target image.

The designated contour control unit 41 enlarges, moves, and rotates the contour designated by the pen input unit 4, and changes the contour to a contour most suitable for the target image.

The color information extracting section 42 is provided with a composition changing section 3.
The image is cut out from the optimal composition image generated in step (1) and the coordinates on the image specified by the pen input unit 4 using the color information of the target, the background is removed, and the target image is extracted.

The line image information extracting section 43 includes a color information extracting section 42.
The image is cut out by using the line drawing information in the same manner as in the above.

The image integration / extraction section 44 integrates the attribute information selected from the pen input section 4 and the attribute database 40 and the information obtained from the color information extraction section 42 and the line drawing information extraction section 43 to obtain a more accurate This is to cut out an image having a high image quality.

Further, the template database 46 stores
An outline template is prepared for each type of target, and a plurality of templates are stored.

Next, the operation of the image retrieval apparatus according to this embodiment will be described.

FIG. 17 is a flowchart showing a flow of processing of the image retrieval apparatus shown in FIG.

In FIG. 17, first, the camera section 1 starts photographing by a user's operation, and performs an image capturing process (step 401). The image captured by the image capturing process is displayed on the display unit 14.

Here, when the user determines that the target is in the photographing range or in an appropriate composition position, the user stores the still image displayed on the display unit 14 by an operation method such as pressing a button in the image storage unit 2. take in.

Next, the composition changing unit 3 judges the composition from the photographed image in the image storage unit 2, changes the composition, and executes the composition changing process of generating the optimum composition image (step 40).
2). The processing so far is the same as the operation of the image search device shown in FIG.

By the way, in the image retrieval apparatus of this embodiment, when the target image is cut out from the optimum composition image displayed on the display unit 14, the user can select one of the optimum composition images displayed on the display unit 14. , The attributes of each part can be specified.

That is, a pen input process for enclosing the outline of the target portion that the user wants to specify with a pen-type input device and simultaneously specifying the attribute of each portion based on the attribute name displayed on the display unit 14 from the attribute database 40 is performed. (Step 303). FIG. 19 shows a display example when this attribute is specified.
Shown in

The color information extraction unit 42 performs a color information extraction process for extracting an image of only a target, excluding an image of a background portion, using the target contour input by the pen input unit 4 and color information in the contour. Execute (step 406). FIG. 20 shows an example of a target image obtained by this color information extraction processing.

Similarly, the line drawing information extracting section 43 executes a line drawing information extracting process for extracting information using line drawing information in the same manner as the color information extracting process (step 407). FIG. 21 shows an example of a target image obtained by this line drawing information extraction processing.

The image integration and extraction unit 44 selects information selected from the target image obtained by the attribute information and color information extraction processing for each part and the target image obtained by the line drawing information extraction processing specified by the pen-type input device. Are integrated, and an image integration extraction process for extracting an image with higher accuracy is performed (step 408).

If the target image obtained here is inappropriate, the designated contour control unit 41 enlarges, moves, and rotates the contour of the target image designated by the pen-type input device, and changes the contour to the most suitable contour for the target image. A designated contour control process is executed (step 405). Then, attribute designation and image extraction are newly performed, and this is repeated until an appropriate image is cut out. That is, in this case, the designated contour control unit 41 creates a new image contour several dots inside the designated image contour, and repeats this until an appropriate image can be cut out. FIG. 22 shows an example of an image obtained by the designated contour processing.

In the target image conversion unit 6, the image integration extraction unit 4
In step 409, a target image extracted in the image integration extraction process of step No. 4 is converted into a target binary image by setting a reference value based on brightness.

The conversion of the data image on the image database 11 into a data binary image is performed by a data image conversion process by the data image conversion unit 12 (step 412).

The data binary images converted by the data image conversion unit 12 are sequentially sent to the image comparison unit 10 and subjected to image comparison processing for comparing with the target binary image converted by the target image conversion unit 6. (Step 410).

This image comparison processing is performed in the image database 1
1 is sequentially performed in the same manner on the images to be searched,
The desired image is determined based on the result. That is, the image having the highest similarity is finally selected.

The result display control unit 13 executes a result display process of extracting a natural image (data image) and related information from which the binary image of data having a high degree of similarity is based, from the image database 11 and displaying it on the display unit 14. (Step 41
1).

By the way, the input method in the pen input process in step 403 need not be a pen-type input unit, but may be an input means such as a mouse or a keyboard. Also,
The coordinates on the image may be directly specified using a numeric keypad or the like.

When the contour of the target image is designated, a contour template may be prepared for each type, and the contour may be designated from the contour template database 46 containing the template.

FIG. 18 is a flowchart showing the flow of processing using a template.

The flowchart shown in FIG. 18 is the same as the flowchart in FIG. 17 except that the outline template is used for each type stored in the template database 46 when the outline of the target image is designated. In FIG. 18, each step is denoted by the same reference numeral as used in FIG. 17, and detailed description thereof will be omitted.

In the flowchart shown in FIG. 18, an example of photographing using a template is shown in FIG. In FIG. 23, a rhombus in the center and a line extending downward are templates, and the user adjusts the photographing position so that flowers and stems come to this position, and photographs. The other processing is the same as the processing shown in FIG.

In the above embodiment, the case where the input target image is mainly a plant is shown. However, the present invention is not limited to this, and the present invention is not limited to this case. Of course, the same can be applied.

[0127]

As described above, according to the present invention,
The following effects are obtained.

1) A proper composition instruction is built in the apparatus, and an appropriate composition instruction is given to a user at the time of photographing, thereby enabling photographing with an appropriate composition and performing a search with high accuracy. Can be.

2) An instruction for an appropriate composition according to the type of the object is built in the apparatus, and an instruction corresponding to the type of the object is displayed by inputting the type of the object to be photographed by the user in photographing. By giving an appropriate composition instruction to the user, it is possible to perform photographing with an appropriate composition and perform a search with high accuracy.

3) By incorporating template information indicating an appropriate composition inside the apparatus, displaying the template on a monitor image at the time of photographing, and photographing the object so that the user overlaps the template as much as possible. This makes it possible to perform shooting with an appropriate composition and perform a search with high accuracy.

4) When displaying a template, the user is required to input a rough type of a target in advance, and a template appropriate for the type is displayed in accordance with the input. It is possible to take a picture and perform a search with high accuracy.

5) By detecting surrounding sounds and automatically performing photographing, the operation burden on the user can be reduced, and unskilled operators can photograph with an appropriate composition, and retrieval can be performed with high accuracy. It can be performed.

6) By automatically detecting a change in the image and performing photographing, the operation burden on the user is reduced, and the photographing with an appropriate composition by an unskilled operator is enabled, and the search can be performed with high accuracy. It can be carried out.

7) By constantly monitoring the image and automatically taking a picture when the coincidence with the template indicating the appropriate composition is high, the operation burden on the user can be reduced, and the proper operation by the unskilled operator can be reduced. The composition can be photographed, and the search can be performed with high accuracy.

8) When the captured image does not have an appropriate composition, an image having an appropriate composition can be obtained by subjecting the image to transformation transformation, and a search can be performed with high accuracy.

9) When extracting a target property from a captured image, the user applies a contour template prepared for each type of target to the target object in the captured image,
The target object can be extracted with high accuracy even if the captured composition is not so appropriate, and the search can be performed with high accuracy.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram showing an embodiment of an image search device according to the present invention.

FIG. 2 is a flowchart showing a processing flow of the image search device shown in FIG. 1;

FIG. 3 is a diagram showing an example of a still image captured in the image storage unit shown in FIG.

FIG. 4 is a view showing an example of an optimum composition image generated by a composition changing section shown in FIG. 1;

FIG. 5 is a view showing an example of an image specified by the pen input unit shown in FIG. 1;

FIG. 6 is a view showing an example of a target image extracted by the image extraction processing shown in FIG. 2;

FIG. 7 is a view showing an example of a target binary image converted by the target image conversion processing shown in FIG. 2;

FIG. 8 is a view showing an example of a data image stored in the image database shown in FIG. 1;

FIG. 9 is a diagram showing an example of a data binary image converted by the data image conversion unit shown in FIG. 1;

FIG. 10 is a flowchart showing details of the image capturing process shown in FIG. 2;

FIG. 11 is a flowchart showing the flow of a process of selecting a plurality of images.

FIG. 12 is a flowchart showing details of the image capturing process shown in FIG. 10;

FIG. 13 is a flowchart showing details of the image selection processing shown in FIG. 11;

FIG. 14 is a view showing an example of the structure of composition instruction information stored in a composition instruction storage unit shown in FIG. 1;

FIG. 15 is a diagram showing a data structure of data stored in the image database shown in FIG. 1;

FIG. 16 is a schematic block diagram showing another embodiment of the image search device according to the present invention.

FIG. 17 is a flowchart showing a flow of processing of the image search device shown in FIG. 16;

FIG. 18 is a flowchart showing another flow of the process of the image search device shown in FIG. 16;

FIG. 19 is a view showing a display example when an attribute is designated in the pen input processing shown in FIG. 17;

20 is a diagram showing an example of a target image obtained by the color information extraction processing shown in FIG.

21 is a diagram showing an example of a target image obtained by the line drawing information extraction processing shown in FIG.

FIG. 22 is a view showing an example of an image obtained by the designated contour processing shown in FIG. 17;

FIG. 23 is a view showing an example of photographing using a template adopted in the flowchart shown in FIG. 18;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Camera part 2 Image storage part 3 Composition change part 4 Pen input part 5 Image extraction part 6 Target image conversion part 10 Image comparison part 11 Image database 12 Data image conversion part 13 Result display control part 14 Display part 30 Composition instruction storage part 31 Moving object sensor 32 Microphone 33 Automatic capture control unit 34 Image change detection unit 35 Template match detection unit 40 Attribute database 41 Designated contour control unit 42 Color information extraction unit 43 Line drawing information extraction unit 44 Image integration extraction unit 46 Template database 100 Control unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Toshiyuki Iwaoka 10 Okayama Todocho, Ukyo-ku, Kyoto-shi, Omron Corporation

Claims (19)

[Claims] 1. An image search method for searching an image database based on a search target image and acquiring information on the search target image, wherein a composition of the search target image set in advance is specified when the search target image is captured. An image search method characterized by referring to composition instruction information. 2. The image search method according to claim 1, wherein the search target image is a captured image captured by an imaging unit. 3. The composition instructing information is a template indicating a desired shape corresponding to the search target image. The composition instructing information is displayed by superimposing the template on a captured image captured by the imaging unit. 3. The image search method according to claim 2, wherein a search target image is fetched. 4. The image search method according to claim 3, wherein the template is compared with a captured image captured by the imaging unit, and the captured image is automatically captured based on the degree of coincidence. 5. An image search method for searching an image database based on an image to be searched to obtain information on the image to be searched, wherein an image capturing condition of an image captured by an image capturing means is set in advance, and An image retrieval method characterized in that when the condition (1) is satisfied, a captured image captured by the image capturing means is automatically captured as the search target image. 6. The image capturing apparatus according to claim 5, wherein the capturing condition is that the degree of coincidence between the image captured by the image capturing unit and a preset template is higher than a predetermined value. Image search method. 7. The image retrieval method according to claim 5, wherein the capturing condition is that a detected sound volume of surrounding sounds exceeds a predetermined value. 8. The image search method according to claim 5, wherein the capture condition is that the image captured by the image capturing unit has changed by more than a predetermined amount. 9. The image capturing means continuously captures an image of an image capturing target, inputs a plurality of image capturing screens, and selectively captures an image capturing screen satisfying the capturing condition among the plurality of image capturing screens. The image search method according to claim 5, wherein: 10. An image search apparatus for searching an image database based on a search target image and acquiring information on the search target image, wherein: an image pickup means for picking up an image of the search target; Composition instruction information setting means for setting composition instruction information for instructing composition in advance; and image capturing for capturing the image captured by the imaging means as the search target image with reference to the composition instruction information set by the composition instruction information setting means. And an image search device. 11. The image retrieval apparatus according to claim 10, wherein said composition instruction information setting means sets said composition instruction information in accordance with a type of said search target. 12. The image search apparatus according to claim 10, wherein the composition instruction information is a template that indicates a shape corresponding to the search target. 13. The image capturing device according to claim 12, wherein the image capturing device compares the template with a captured image captured by the image capturing device, and automatically captures the captured image based on the degree of coincidence. The image search device according to the above. 14. The image retrieval apparatus according to claim 12, wherein said image capturing means comprises composition changing means for changing the composition of the captured image to be compared with the template. 15. An image search apparatus for searching an image database based on a search target image and acquiring information on the search target image, wherein: an image pickup means for picking up an image of the search target; Capturing condition setting means for setting a capturing condition; an image automatically capturing a captured image captured by the image capturing means as the search target image when the capturing condition set by the capturing condition setting means is satisfied. An image retrieval device, comprising: a capturing unit. 16. The capturing condition set by the capturing condition setting means is that the degree of coincidence between the image captured by the image capturing means and a preset template is higher than a predetermined value. 16. The image search device according to claim 15, wherein: 17. A sound detecting means for detecting ambient sound, wherein the capturing condition set by the capturing condition setting means is such that the detected sound volume of the surrounding sound detected by the sound detecting means is 2. The method according to claim 1, wherein the predetermined value is exceeded.
5. The image search device according to 5. 18. An image change detecting means for detecting a change amount of a captured image picked up by the image pickup means, wherein the capture condition set by the capture condition setting means is the image change detection means. 16. The image retrieval apparatus according to claim 15, wherein the detection output of the image retrieval apparatus has exceeded a predetermined value. 19. The image capturing means continuously captures an image of an image capturing target and inputs a plurality of image capturing screens. The image capturing means captures an image satisfying the capturing condition among the plurality of image capturing screens. The image retrieval apparatus according to claim 15, further comprising: a selective capturing unit that selectively captures a screen.