JP2005063465A - Image retrieval device, electronic still camera and image retrieval method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image retrieval method, image retrieval device and electronic still camera for retrieving a desired record image based on a picture image inputted by drawing. <P>SOLUTION: The image retrieving device 100 extracts recorded images having highly matching feature elements by comparing feature elements of handwriting-inputted drawings 501, 502, 503 and feature elements of partial images 501', 502', 503', ... of recorded images and displays them as candidate images 111, 112, 113, 114, ... and a user selects an image (e.g. image 111 in Fig.(b')) that the user images, thus retrieving the desired image without keyword input and frame feeding. Here, the feature elements are, for example, an outline feature of a drawing for retrieval and an outline feature of the recorded image, the color inside the outline of the drawing for retrieval and colors of the partial images, the arrangement of the drawing for retrieval and the arrangement of the partial images, etc. However, they are not limited. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image search technique, and more particularly to an image search technique for searching a recorded image based on a drawn and input image image.

  A subject image picked up by a digital camera is recorded in a storage medium as image data through photoelectric conversion, signal conversion, signal processing, and the like by a CCD. In this case, the image data is recorded on the recording medium in the order of imaging.

  Many digital cameras are provided with a display device such as a liquid crystal display and can display a reproduced image read from a recording medium after imaging. However, when displaying a reproduced image with such a digital camera, the image data is read from the recording medium in the order of imaging, and therefore displayed on the screen of the display device in the order of imaging.

  When trying to find a desired image from the recorded images, the playback image is advanced frame by frame until the desired image appears, or a plurality of simple images (thumbnail images) having a small size are displayed on one screen (multi-screen). When a desired simple image is displayed, the original image is reproduced and displayed when the desired simple image is displayed.

  The conventional search method has a problem that frame feeding for searching for a desired image is troublesome when a large number of images are recorded, regardless of whether the number of recorded images is small. In addition to this, when searching for a plurality of desired images (when the desired image is at the end of the recorded image), all the recorded images must be reproduced and frame-by-frame searched. was there.

  In addition to image search by frame advance, it is also possible to search for an image by keyword. In this case, it is necessary to register a keyword in advance corresponding to the image, and it is necessary to add a keyword when inputting an image. There was a problem that the user had to memorize and write down the correspondence between images and keywords.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide an image search method, an image search apparatus, and an electronic still camera that search for a desired recorded image based on a drawn and input image image. .

  In order to achieve the above object, the image search method of the first invention draws an image, and sequentially compares this drawing with each image recorded in the image recording memory, so that a recorded image with high similarity is obtained. Is extracted and displayed.

  In the image search method of the second invention, an image is drawn, and one of similar images obtained by sequentially comparing the drawing and each image recorded in the image recording memory is used as a search image. Compared with each image recorded in the recording memory, a recorded image having a high degree of similarity is extracted and displayed.

  The image search method of the third invention selects one of the images recorded in the image recording memory as a search image, and compares the search image with each image recorded in the image recording memory. A feature is that a recorded image having a high degree of similarity is taken out and displayed.

  According to a fourth aspect of the present invention, there is provided an image search method, wherein in the imaging apparatus, the captured image is used as a search image and the captured image is sequentially compared with each image recorded in the image recording memory to obtain a recorded image having a high similarity. It is extracted and displayed.

According to a fifth aspect of the present invention, in the image search method according to any one of the first to fourth aspects, the comparison is performed by comparing any one of the following a to d.
a: Contour b: Contour and color c: Contour and its position d: Contour, its position and color

  According to a sixth aspect of the present invention, in the image search method according to the fourth aspect of the present invention, a feature element obtained by extracting any one of the a to d from each image recorded in the image recording memory is registered in the memory in advance. And

  According to a seventh aspect of the present invention, in the image search method according to any one of the first to sixth aspects of the present invention, images having a high degree of similarity are reduced in display size in accordance with the number of the images and displayed in multiple displays. .

  According to an eighth aspect of the invention, in the image search method according to any one of the first to fourth aspects of the invention, the search image is sequentially compared with the recorded image after erasing unnecessary portions.

  The image search device of the ninth invention is an image recording means for recording a plurality of images, a drawing input means for drawing and inputting a picture or a figure, and a picture or figure drawn and inputted by the drawing input means and an image recording. An image recognizing unit that obtains a recorded image having a high degree of similarity by comparing each image recorded in the unit, and a notifying unit that notifies a comparison result by the image recognizing unit.

  An image search apparatus according to a tenth aspect of the invention includes an image recording means for recording a plurality of images, a drawing input means for drawing and inputting a picture or a figure, and a picture or figure drawn and inputted by the drawing input means and an image recording. The primary image extraction means for comparing each image recorded in the means to obtain a similar image, and the similar image obtained by the primary image extraction means and the respective images recorded in the image recording means are compared and similar. It has an image recognizing means for obtaining a highly recorded image, and a notifying means for notifying a comparison result by the image recognizing means.

  According to an eleventh aspect of the present invention, there is provided an image search device for recording a plurality of images, an image selection unit for selecting one of the images recorded in the image recording unit as a search image, An image recognizing unit that obtains a recorded image with a high degree of similarity by comparing the search image selected by the selecting unit with each image recorded in the image recording unit, and an informing unit that notifies the comparison result by the image recognizing unit It is characterized by having.

  According to a twelfth aspect of the present invention, in the image search device of the ninth or tenth aspect, the image input means is a handwritten input.

  The thirteenth invention is the image search device according to any one of the ninth to eleventh inventions, wherein the image recognition means is a picture or figure drawn and input by the drawing input means, a similar image obtained by the primary image extraction means, Alternatively, the feature element of the search image selected by the image selection means is extracted and compared with the feature element extracted from each image recorded in the image recording means.

  The fourteenth invention is the image search device according to any one of the ninth to eleventh inventions, wherein the notification means is a display means.

  An electronic still camera according to a fifteenth aspect of the present invention is an electronic still camera comprising an image recording means and a display means. The photographing means for photographing a subject and outputting a photographed image; and the photographed image output from the photographing means; An image recognition unit that obtains a recorded image with a high degree of similarity by comparing each image recorded in the image recording unit, and a notification unit that notifies a recognition result by the image recognition unit.

  As described above, according to the image search method of the first invention and the image search device of the ninth invention, the so-called fuzzy search is performed by comparing the drawn image and the recorded image, so that it is time-consuming and fast to input keywords. Thus, it is possible to find an image that matches the user's image from a large number of images recorded in the image recording memory. In addition, when the color and position are used as search elements, the search accuracy is improved.

  Further, according to the image search method of the second invention and the image search device of the tenth invention, since the recorded image is first searched with the drawn image, the search is further performed with the image obtained by the search. Since the search image after the first image can be searched based on a clearer image than the drawn image (drawing diagram), the search accuracy is improved. Also, the search speed is increased.

  Further, according to the image search method of the third invention and the image search device of the eleventh invention, since a recorded image is searched using a desired recorded image as a search image, a simple, accurate and quick image search can be realized. .

  In addition, according to the image search method of the fourth invention and the electronic still camera of the fifteenth invention, a recorded image is searched by a photographed image, so that it is possible to realize a simple, accurate and quick image search at the photographing site.

  According to the image search method of the fifth invention, image recognition is facilitated by using the contour as a comparison element. Further, the recognition accuracy is improved by using the color and / or position as a comparison element.

  Further, according to the image search method of the sixth aspect of the invention, the comparison element of the recorded image is registered in the memory in advance, and image recognition can be performed based on it, so that not only the search accuracy is improved, but also faster. Search is possible.

  In addition, according to the image search method of the seventh invention, when there are a plurality of candidate images, multi-display can be performed, so that the user can immediately select a desired image from the multi-display screen.

  Further, according to the image search method of the eighth invention, the search can be performed only with the partial images suitable for the search among the search images, so that the search speed is further improved and the search accuracy is further improved.

  According to the image search device of the twelfth aspect of the invention, the image is input by handwriting input, so that the image input is simple and does not give the user a burden on the image input operation.

  Further, according to the image search device of the thirteenth aspect of the invention, the input image or the image feature element and the recorded image feature element are compared, so the search accuracy is high.

  Further, according to the image search device of the fourteenth aspect, since the recognition result is notified by displaying on the screen such as a liquid crystal display, the user can know the recognition result at a glance. In addition, even if the recognition order by the image search device is high, the result may not be the result desired by the user, or the result may be the result desired by the user even if the recognition order is low. The chance of obtaining the desired result is further increased.

<Embodiment (1)>
FIG. 1 is a schematic explanatory diagram of an image search method by drawing input according to the present invention. In the present embodiment, the image retrieval apparatus 100 is a drawing for retrieval (referred to as a diagram depicting an entire image or a partial image of an image to be retrieved) 501, 502, 503 (FIG. 1 ( a recorded image having a high degree of coincidence of characteristic elements is extracted by comparing the characteristic elements of a)) with the partial images 501 ′, 502 ′, 503 ′,... (FIG. 1B). , Candidate images 111, 112, 113, 114,... (FIG. 1C), and an image (image 111 in the example of FIG. 1C) imaged by the user from the displayed candidate images. By selecting, it is possible to easily search for a desired image without inputting a keyword or frame advance.

  Here, examples of feature elements to be compared include the contour feature of the drawing for search and the contour feature of the recorded image (FIG. 8), the color applied to the inside of the contour of the search drawing, and the color of the partial image (FIG. 9). ), Or the arrangement of the drawing for search and the arrangement of the partial images (FIG. 10), but are not limited thereto.

[Circuit configuration example]
FIG. 2 is a block diagram illustrating a configuration example of a digital camera. The image search apparatus 100 includes an imaging unit 1, a drawing input unit 2, an operation unit 3, a control unit 4, an image recognition unit 5, a work memory 6, and an image recording unit. A memory 7, an image reproducing unit 8, a display unit 9, and a liquid crystal display (LCD) 10 are provided.

  The imaging unit 1 includes an imaging device that converts an optical image collected via an imaging system, an optical system such as an imaging lens, a lens driving mechanism, and a diaphragm mechanism, and an imaging device driving unit and the obtained electrical signal. It has a signal converter for converting into digital data.

  The operation unit 2 includes a power switch, a process selection button, and the like. When the user operates a power switch, a process selection button, or the like of the operation unit 2, an electrical signal (code) corresponding to the operation content is sent to the control unit 4. As a result, the user can give the control unit 4 an instruction necessary for starting the image search device 100 or for an image search operation.

The drawing input unit 3 is composed of a member that draws a picture or a line with a pen or a finger and converts the result into an electric signal.
In the present embodiment, the drawing input unit 3 includes a transparent pressure-sensitive input plate (a plurality of piezoelectric elements that detect electric pressure and output an electrical signal on a transparent plate made of a material such as plastic) provided on the liquid crystal display 10. Configured to output electrical signals and coordinate values (x, y) of the touch position when a drawing or point is made with a touch pen or finger, but a flat plate (input tablet) provided with a pressure sensitive element is provided. When a drawing or point is made with a touch pen or a finger, an electric signal and a coordinate value (x, y) of the touch position may be output.
In addition, a transparent plate (heat-sensitive transparent plate) provided with a plurality of heat-sensitive elements that detect heat rays and output electrical signals on the liquid crystal display 10 is provided, and heat rays emitted from the user's finger when a finger draws or points. May be detected, and the electrical signal and the coordinate value (x, y) of the touch position may be output.
In addition, an input pen that emits light and a transparent plate on which a plurality of light detection elements that detect light of a specific wavelength and output an electric signal are provided on the liquid crystal display 10 are provided. It is also possible to detect a light beam having a specific wavelength emitted from and output the electrical signal and the coordinate value (x, y) of the touch position.
Instead of providing the drawing input unit 3, for example, a cursor may be displayed on the LCD 10, and drawing may be performed by arbitrarily moving the cursor with a mouse or a cursor key.

  The control unit 4 includes a CPU, a program storage memory, and a temporary storage memory (not shown), and performs control of the image search apparatus 100 and execution control of the image search process of the present invention. The program storage memory is composed of a nonvolatile memory such as a ROM and stores an image search processing program group in addition to a control program. The temporary storage memory is composed of a RAM and temporarily stores programs and data. When the image retrieval apparatus 100 is activated, a control program, an image retrieval program, and the like are temporarily stored in the working memory from the program storage memory, and are executed and controlled by the CPU to execute an image retrieval process.

  The image recognizing unit 5 performs the recognition process by extracting feature elements of the drawn drawing for retrieval and the recorded image and checking the degree of coincidence between the features. If an image search is performed using the feature elements of the recorded image in advance, an area for recording the feature list as shown in FIG. 12 is secured.

  The work memory 6 comprises a DRAM or the like, and is used for storing image data at the time of image recognition or the like.

  The image recording memory 7 is formed of a recording medium such as a flash memory, a memory card, a magnetic disk, or an optical disk, and records compressed image data.

  The image reproduction unit 8 includes a first decompression unit that decompresses the compressed image data to an image having the same size and the same density as the image before compression in order to perform the decompression process of the read recorded image data (compressed data). A second decompression unit that decompresses the compressed image data to an image having the same size as the image before compression and a density lower than that of the original image; and decompresses the compressed image data to an image having a size smaller than that of the original image. An image reduction unit for reproducing an image is provided.

  In the first decompression unit, the image is restored (reproduced) to the original state. However, when the number of recorded images is large, it takes a search time. In FIGS. 3 and 4, the second decompression unit is used to roughen the pixel density and perform the decompression, thereby speeding up the reproduction speed.

  In addition, when there are a plurality of images having a predetermined degree of coincidence among the images recognized by the image recognition unit 5, the image reduction unit displays 9 images (3 × 3), 16 images (4 When displaying multiple images such as x4), the size of the playback image to be displayed is reduced (1/9 size for nine images on one screen, 1/16 size for 16 images,・ ・） Stretch as shown.

  Note that when the multi-image display is configured and the image search apparatus 100 includes an image recording memory having a sufficiently large memory capacity, the image recording is performed without providing the image reduction unit as described above. Sometimes, the image may be reduced in advance to a plurality of sizes and stored in association with each image so that a reduced image having a size required for display can be extracted.

  The display unit 9 has an image memory such as a video encoder and a VRAM, converts image data and drawing input data into image data, develops the bitmap image in the VRAM, and displays an image on a liquid crystal display (LCD) 9. Display.

[Example 1-1] Image search operation by drawing input (example with image feature extraction)
FIG. 3 is a flowchart showing an operation example of the image search apparatus 100 based on the image search method of the present invention, and FIG. 4 is a flowchart showing an embodiment of the feature extraction process of the drawing diagram in step S6 of FIG. 5 is a flowchart showing an embodiment of the feature extraction process of the recorded image in step S8 in FIG. 3, and FIG. 6 is a flowchart showing an embodiment of the image recognition process in step S10 in FIG. FIG. 7 is a diagram showing an example of a drawing selection screen, FIG. 8 is an explanatory diagram of an image search example by drawing input of only the contour, and FIG. 9 is an explanatory diagram of an image search example by drawing with coloring. FIG. 10 is an explanatory diagram of an example of image search based on the position designation of the drawing.

  In this embodiment, a transparent panel (hereinafter referred to as a screen 110) provided with a pressure sensitive element provided on the liquid crystal display 10 is used as the drawing input unit 3, but as described above, the present invention is not limited to this. The user can touch the screen 110 with the touch pen 3A to input an outline diagram and specify a color and a display position.

Step S1: (Drawing method selection)
In FIG. 3, when the image search process is activated, the screen of the liquid crystal display 10 is cleared and displayed in white. Further, as shown in the example of FIG. 7A, a colorless drawing designation icon 91 and color samples 911 (white), 912 (green), 913 (yellow), 914 (brown),..., 919 (black), etc. A strip-shaped color sample 92 (FIG. 7B) composed of a plurality of color sample designated frames is displayed. In the example of FIG. 7B, the color sample 92 has nine colors. However, the present invention is not limited to this, and a number of color samples including mixed colors may be used. Further, the color sample may be a band composed of a color distribution ranging from white to warm color to intermediate color to cold color to black without being divided into individual color samples. The shape of the color sample may be a pie chart.

  If the user selects the colorless drawing designation icon 91 (points with the touch pen in this embodiment), the process proceeds to S2, and if the color designation icon 92 is selected, the process proceeds to S3. The point result (which icon has been designated) is stored in the RAM (for example, by turning on a flag corresponding to the icon when pointed).

Step S2: (draw (no color specified))
When the user draws a contour drawing while bringing the touch pen 3A into contact with the screen 110 of the image search device 100 (FIG. 8A), the contact trajectory is converted into an electrical signal by the drawing input unit 3 via the control unit 4. It is sent to the VRAM of the display unit 9 and also sent to the image recognition unit 5. The display unit 9 converts the received electrical signal into raster data, develops it in a VRAM as a bitmap image, and displays the image of the drawing for search in black (or a single color other than black). The example of FIG. 8A is an example in which the user draws an image of a large mountain 501, a small mountain 502 and a house 503 as a drawing drawing for search, and is used for searching the drawn large mountain 501, small mountain 502 and house 503. The drawing is converted into an electrical signal. A position designation icon 93 and a drawing end icon 94 are displayed on the screen.

Step S3: (drawing (with color designation))
When the user draws a drawing for search while touching the screen 110 of the image search device 100 with the touch pen 3A, fills the inside of the outline, or draws a diagonal line (FIG. 9A), the image is displayed by the drawing input unit 3. It is converted into an electric signal and sent to the VRAM of the display unit 9 via the control unit 4 and also sent to the image recognition unit 5. The display unit 9 converts the received electrical signal into R, G, and B raster data, develops it in a VRAM as a bitmap image, and displays a drawing image (an outline or an image expressed by an outline and an internal color) in a designated color. . On the screen, a color sample 92, a position designation icon 93, and a drawing end icon 94 are displayed.

When the outline and the internal color are separated, the next color is selected by pointing to the color sample 92 displayed on the screen 110. In the example of FIG. 9A, the color of the image of the large mountain 501, small mountain 502, and house 503 in FIG. 8 (green 912 for the mountain 501, brown 914 for the mountain 502, and yellow 913 for the house 503). ).
The point result is stored in a storage memory (hereinafter referred to as RAM) of the control unit 4. Further, the designated color data (code) is associated with the contour and the inside of the contour for the drawing for retrieval, and the result is stored in the RAM.

  For example, the number of the drawing for search is “1”, the contour code is “1”, the internal code is “2”, the contour and the designated internal color are the same green, and the green color code is “08”. ”Is stored as“ 1108208 ”, and when the contour designated color is blue (code =“ 03 ”) and the internal designated color is blue (code = 06),“ 1103206 ” Can be stored.

Step S4: (Position designation)
When the position designation icon 93 displayed on the screen 110 is pointed, the position designation flag corresponding to the drawing drawing for retrieval is turned on (= 1) among the position designation flags secured in the RAM. . A drawing addition icon 95 and an all drawing end icon 96 are displayed on the screen.

Step S5: (Additional drawing input determination)
Further, when the all drawing end icon 94 (96) displayed on the screen 110 is pointed, the control unit 4 determines that drawing is ended and shifts to the feature extraction process of S6. In addition, when a drawing for search is added (FIGS. 8A and 9A), if the drawing addition icon 95 is pointed, an additional drawing input is performed, so that S2 or S3 depends on the result pointed at step S1. Return to.

Step S6: (drawing drawing feature extraction)
The image recognition unit 5 extracts feature data of the drawing for search. In this embodiment, any one of contour features, contour features and color features, contour features and placement, contour features, color features and placement is extracted as feature data (FIG. 5: steps S6-1 to S6-4). As described above, it is not limited to these.
Which combination is extracted as feature data depends on which combination of icons 91 to 93 is selected in steps S1 to S4. Hereinafter, this step will be described based on the flowchart of FIG. When a plurality of drawing drawings for search are drawn (FIGS. 8A and 9A), feature extraction is performed for each drawing drawing unit (that is, cut out for each drawing and feature extraction is performed sequentially). Do).

-Step S6-1;
In FIG. 4, the image recognition unit 5 cuts out the contour of the drawing for search within the rectangular frame as shown from the raster data developed in the DRAM 6. When a plurality of drawings are drawn as search drawing drawings, they are cut out and stored in drawing units (FIGS. 8B and 9B). Further, the number i of the extracted drawing for search is counted and the search drawing is numbered in order of size or the storage position is switched in order of size (sorted in reverse order). In addition, in the present embodiment, the size of the drawing for search is compared with the size of the circumscribed frame (the length of the diagonal of the circumscribed frame), but is not limited thereto.

-Step S6-2;
A feature vector (represented by a set of direction, starting point of line segment and length) of the extracted drawing for search is extracted and stored in the RAM as contour feature data.
In this embodiment, for example, in the case of the mountain 501, the contour feature is expressed by the length of the line segment from the mountain skirt a (start point) to the peak b and the angle A, and the vector A1 expressed by the angle and the angle from the peak b (start point) to the mountain c. It is given as a set of vectors A2 expressed by the length of the line segment and its angle. When there are a number of peak peaks, vector components are extracted with the peaks as end points (or start points). Further, when the contour is discontinuous as in the house 503, vector components are extracted with the nodes p1, p2,. Further, when the contour includes a curve or an arc, the coefficient and initial value of the function representing the curve, the radius of curvature of the arc, and the like are extracted.

  Step S6-3: When the icon 92 is selected in Steps S1 to S3 (when color is designated), the image recognition unit 5 extracts the color features of the drawing for search (FIG. 9B). To do. The color feature of the drawing for retrieval can be obtained from the color distribution as in the case of extracting the color feature of the recorded image, which will be described later. The stored specified color data (code) is associated with the contour and the result of the contour (color data), and is given to the image recognition unit 5 as a color feature. In this embodiment, the color feature is the ratio of the number of pixels included in the drawing for search and the respective color pixels R (red), G (green), and B (blue), but is not limited to this. If the icon 92 is not selected in steps S1 to S3, the process proceeds to S6-4.

  Step S6-4: The position designation flag secured in the RAM is checked, and if the position designation flag corresponding to the drawing drawing for search is on, the image recognition unit 5 displays the center coordinates of the drawing drawing for searching on the screen. Is extracted and stored in the RAM. The center coordinates can be obtained as the coordinates (a, b) of the center point of the frame circumscribing the drawing drawing for search as shown in the example of FIG. In the example of FIG. 10A, a frame 701A circumscribing the drawing for search 701 (house) and its center are shown. If the position designation flag corresponding to the drawing for search is off, the process proceeds to S6-5.

  Step S6-5: The feature extraction process in steps S6-2 to S6-4 is repeated for the drawing drawing for retrieval extracted in step S6-1, and the feature extraction process is completed for all the drawing drawings for retrieval. Transition to S7.

Step S7: (Simple decompression of image data)
After the feature drawing process for the drawing for retrieval in step S7 is completed, the control unit 4 reads out image data (compressed image data) for one image recorded in the image recording memory 7 and the image reproduction unit 8 In the second decompression unit, the compressed image data is decompressed to an image having the same size as the image before compression and a coarser density than the original image, and is stored in a working memory (hereinafter referred to as DRAM) 5.

Step S8: (feature extraction of recorded image)
The image recognition unit 5 extracts image feature data for one frame stored in the DRAM 6. It should be noted that the feature data to be extracted is the same feature as in the case of the feature extraction of the drawing diagram in step S6. Hereinafter, this step will be described based on the flowchart of FIG. When multiple partial images are included in the image, partial images with a rectangular frame larger than a predetermined size are cut out, and feature extraction is performed in units of partial images until the number becomes the same as the drawing drawing for search in descending order. (I.e., cutting out and extracting features in order of large partial images until the same number as the drawing for retrieval is obtained).

Step S8-1;
The image recognition unit 5 cuts out the outline of the image within the rectangular frame from the raster data developed in the DRAM 6 as shown in FIG.
When a plurality of partial images are included in one frame image, a partial image whose rectangular frame is a predetermined size or larger is cut out and stored in the DRAM 6. When the number of partial images cut out exceeds the number i of drawing drawings for search cut out in step 6-1, i pieces are stored in descending order. In this embodiment, the size of the circumscribed frame (the length of the diagonal line of the circumscribed frame) is used as a reference for comparing the sizes of the partial images. However, the present invention is not limited to this.

-Step S8-2;
Extract the feature vectors (represented by a set of direction and line segment start point and length), the coefficient of the function representing the curve, the initial value, the radius of curvature, etc. of the extracted partial images 501 ′, 502 ′, and 503 ′. And stored in the RAM as contour feature data. The contour feature extraction method may be the same as the search image extraction method.

  Step S8-3: When the icon 92 is selected in steps S1 to S3 (when color is designated), the image recognition unit 5 extracts the color feature of the partial image and stores it in the RAM. The color feature of the drawing for search can be obtained from the color distribution of the extracted partial image. In the present embodiment, the color feature is the ratio between the number of pixels included in the partial image and each of the color pixels R, G, B, but is not limited thereto.

  In the example of FIG. 9 (c), the peaks 501 ′ and 502 ′ and the color distribution peaks 501 ′ of the house 503 ′, which are partial images of the color image, have a green component of 80% or more, and the brown component of the peak 502 ′ is 90%. House 503 ′ is an example of 70% gray component and 30% yellow component). The color feature extraction result is stored in the RAM in association with the contour feature data. If the icon 92 is not selected in steps S1 to S3, the process proceeds to S8-4.

  Step S8-4: The position designation flag secured in the RAM is checked, and if the position designation flag corresponding to the search drawing (FIG. 10B) is on, the image recognition unit 5 The center coordinates of the partial image are extracted and stored in the RAM. As shown in the examples of FIGS. 10C and 10D, the center coordinates are the coordinates (c, d) of the center point of the frame circumscribing the partial image 701 ′ or the coordinates of the center point of the frame circumscribing the partial image 701 ″. If the position designation flag corresponding to the drawing for search is off, the process proceeds to S8-5.

  Step S8-5: The feature extraction process in steps S8-2 to S8-4 is repeated for the partial image extracted in step S8-1. When the feature extraction process is completed for all the extracted partial images, the process proceeds to S9. .

Step S9: (Repeated feature extraction of recorded image)
Steps S7 and S8 are repeated until feature extraction is completed for all the recorded images.

Step S10: (Image recognition processing)
When the feature extraction of the recorded image is completed, the control unit 4 sequentially compares the feature data of the drawing for search with the feature data of each image to calculate the coincidence rate. Then, an image having a matching rate equal to or higher than a predetermined threshold is recognized as a similar image (matching if the matching rate is 100%), and the image number is stored in the RAM as the candidate image. When there are a plurality of drawing drawings for search, the feature data is sequentially compared with the feature data of each partial image in order of size, and the overall matching rate (equal to the matching rate in the case of one drawing drawing for searching). Is recognized as a similar image, and the image number is stored in the RAM as the candidate image. Hereinafter, this step will be described based on the flowchart of FIG.

Step S10-1;
The contour feature data of the j (j = 1 to n) th drawing for retrieval (FIG. 8B, FIG. 9B, or FIG. 10B) stored in the RAM, and one frame stored in the RAM The contour feature data of each partial image (images surrounded by a frame line in FIG. 8C, FIG. 9C, FIG. 10C, or FIG. 10D) The matching rate Γ is obtained.

In this embodiment, the contour feature matching rate (hereinafter referred to as the contour matching rate) is the line vector of the extracted drawing for retrieval and partial image, the coefficient and initial value of the function that expresses the curve, and the radius of curvature of the arc. The matching rate is obtained, and then the mean square of each matching rate is calculated to obtain the contour matching rate Γ.
Specifically, as shown in FIG. 11, when the origin of the extracted drawing for search 501 is set to the lower left, among the partial vectors (feature data) forming the search vector, it is closest to the origin when viewed clockwise. The line segment length A1L of the partial vector (vector A1 in the example of FIG. 11) (FIG. 11 (a)) and the right of the partial vectors forming the partial image 501 ′ when the origin of the partial image 501 ′ is the lower left. A ratio Lr1 (r1 = B1L / A1L) of the partial vector (vector B1 in the embodiment) closest to the origin and the line segment length B1L (FIG. 11B) is obtained, and then the vector A1 and the vector B1 A ratio θr1 of the directions A1θ and B1θ is obtained. Then, the coincidence rate γ1 of the line segment vector A1 is obtained as γ1 = (1− | 1−Lr1 |) × (1− | 1−θr1 |)> ρρ.

When the contour component closest to the origin is a curve or an arc, the ratio of the coefficient of the function representing the curve and the ratio of the initial value or the ratio of the radius of curvature are calculated, and the above-described vector component coincidence ratio calculation is performed. The coincidence rate γ1 is obtained by the same method.
In this case, the closer the match rate is to 1, the higher the degree of similarity is, and γ1 = 1 is a perfect match. Here, the closer the match rate γ is to 0, the lower the similarity.

  Next, when γ1> ρ, the line segment length A2L of the line segment vector A2 on the right side of the vector A1 in the drawing for search 501 and the line segment of the line segment vector B2 on the right side of the vector A1 in the partial image 501 ′. The ratio Lr2 (r2 = B2L / A2L) with the length B2L is obtained, and then the ratio θr2 between the directions A2θ and B2θ of the vector A2 and the vector B2 is obtained to obtain the coincidence rate γ2 of the line segment vector A2.

  When the contour component on the right is a curve or arc, the ratio of the coefficient of the function that expresses the curve and the ratio of the initial value, the ratio of the radius of curvature, and the like are calculated. The coincidence rate γ2 is obtained by the method of

  In the example of FIG. 11, only two line segments A1 and A2 are shown because the search drawing 501 is a mountain. However, as in the house 503 in FIG. In the case of a complicated drawing for retrieval with many arcs, the matching rates γ3, γ4,..., Γm (m is the sum of the number of line vectors and curves) are similarly obtained for each line segment.

-Step S10-2;
Next, the square average of partial vector coincidence rates γ1, γ2, γ3,..., Γm is obtained as the partial image contour coincidence rate Γ, and the coincidence between the f-th partial image and the j-th drawing for search. It is determined whether the rate Γfj exceeds the threshold ρ. If Γfj ≧ ρ, the process proceeds to S10-3, and if Γfj <ρ, the process proceeds to S10-4.

Step S10-3;
If Γfj ≧ ρ in step S10-2, it is recognized that the contour is similar, and the partial image number f, the drawing drawing number j for search, and the contour matching rate Γfj are temporarily stored in the RAM. In this case, a predetermined number of contour matching rates from the contour matching rates Γ11, Γ12,..., Γfj,. If it is less than the predetermined number, only the contour matching rate with the threshold value ρ or higher is extracted and stored in the RAM together with the corresponding partial image number and the drawing drawing number j for search (if it is higher than the predetermined number, it matches) Reject from the lowest rate).

  In this embodiment, the line segment length ± 80% (= error ± 20%) and the direction ± 80% (= error ± 20%) are similar, and the threshold ρ is 0.64 (0.8 × 0). .8 = 0.64), but is not limited to this.

-Step S10-4;
Then, the operations of S10-1 to S10-3 are repeated until the contour coincidence rates Γ11, Γ12,..., Γfj,. When the above processing is completed for the partial images, the process proceeds to S10-5.

  In this embodiment, the contour matching rate is obtained by the above-described method to determine (recognize) similarity or dissimilarity. However, the present invention is not limited to this, and the contour image itself of the drawing drawing for search and the partial image of the recorded image is extracted. These may be compared, and a known contour recognition method may be used.

-Step S10-5;
In step S2, the result of associating the designated color data (code) stored in the RAM with the outline and the inside of the outline is checked (whether or not the icon 92 has been selected (when color is designated)), and color designation is performed. If the color is not designated, the process proceeds to S10-11.

-Step S10-6;
For the partial image having the partial image number stored in the RAM in step S10-4, the image recognition unit 5 stores the color feature data of the jth drawing for search stored in the RAM in step S6-3 and the step S8-3. The color matching rate RGBΓj is obtained by comparing with the color feature data stored in the RAM.

Specifically, in this embodiment, as described in step S6-3 and step S8-3, the color feature of the jth drawing for search is represented by the number of pixels included in the drawing for search and each color pixel R. , G, B ratio (c1j, c2j, c3j): (j = 1 to n), and the color characteristics of each partial image are the number of pixels included in the partial image and the respective color pixels R, G, B (C1f, C2f, C3f: (f = j1 to k))
R, G, B color feature coincidence ratios of the f-th partial image t and the j-th drawing for search,
Rγfj = C1f / c1j, Gγ = C2f / c2j, Bγ = C3f / c3j
And the least mean square of Rγfj, Gγfj and Bγfj is defined as the color matching rate RGBΓfj of the partial image f and the jth drawing for search.

-Step S10-7;
Next, when the color matching rate is extremely low, the partial image number or the like is deleted from the RAM (that is, the partial image is rejected), and the color matching rate RGBΓfj is compared with the threshold ρ ′, and RGBΓfj <ρ ′. The process transitions to S10-8 at the time of, and the process transitions to S10-8 when RGBΓfj ≧ ρ ′.

-Step S10-8;
The contour matching rate corresponding to the partial image number is deleted from the contour matching rate Γ stored in the RAM, and the process proceeds to S10-10.

For example, in FIG. 9B, the color of the drawing for search 503 in FIG. 9B has many brown components, and the partial image 503 ′ in FIG. 9C has many gray components and has different colors, and the color matching rate RGBγ is less than or equal to the threshold ρ ′. Therefore, the partial image 503 ′ is rejected as dissimilar even if the contour drawing matching ratio between the drawing drawing for searching (partial image) 503 and the partial image 503 ′ is high. In the embodiment, ρ ′ = 0.5, but the present invention is not limited to this.
-Step S10-9;
The color matching rate RGBΓfj is stored in the RAM in association with the partial image number f.

-Step S10-10;
Then, the operations of S10-6 to S10-8 or S10-9 are performed until the color matching rates RGBΓ11, RGBΓ12,..., RGBΓfj,..., RGBΓkj of the k partial images and the jth search image are obtained. When the above processing is completed for k partial images, the process proceeds to S10-11.

Step S10-11;
The position designation flag secured in the RAM is checked. If the position designation flag corresponding to the jth drawing for search is on, the process proceeds to S10-12 with position designation, and if it is off, the position designation is not performed. Transition to S10-17.

-Step S10-12;
The image recognition unit 5 compares the center coordinates of the drawing for search stored in the RAM in step S6-4 with the center coordinates of each partial image stored in the RAM in step S8-4, and determines the position of the fth partial image. The coincidence rate XΓfj is obtained.
In this embodiment, the position coincidence rate is a ratio of the mean square of coordinate values (that is, a distance ratio from the origin).

For example, the position coincidence rate between the center coordinates (a, b) in the drawing for search 701 in FIG. 10A and the partial image 701 ′ in FIG.
XΓ701 ′ = (a ² + b ² ) ^1/2 / (c ² + d ² ) ^1/2
The position coincidence ratio between the center coordinates (a, b) in the drawing for search 701 and the partial image 701 ″ in FIG.
XΓ 701 ″ = (a ² + b ² ) ^1/2 / (e ² + f ² ) ^1/2
Given in.

-Step S10-13;
The position coincidence rate XΓfj is compared with the threshold value ρ ″. When XΓfj <ρ ″, the process proceeds to S10-14, and when XΓfj ≧ ρ ″, the process proceeds to S10-15.

-Step S10-14;
If the position coincidence rate XΓfj is equal to or less than the threshold ρ ″, the contour coincidence rate and the color coincidence rate associated with the partial image number f are deleted from the RAM (that is, the partial image is rejected) and the process proceeds to S10-16. To do.

  For example, when the threshold value is Xρ (= 0.8) in FIG. 10, the positions of the partial images 701 ′ in FIG. 10C are substantially the same (XΓ701 ′> ρ ″), but FIG. Since the partial image 701 ″ has a different position (XΓ701 ″ <ρ ″), the partial image is rejected (as a result, the recorded image including the partial image 701 ″ is excluded from the recognition target of the drawing drawing for search 701).

-Step S10-15;
If the position matching rate XΓfj is equal to or greater than the threshold ρ ″, the position matching rate XΓfj is stored in the RAM in association with the partial image number f.

-Step S10-16;
The operations of S10-12 to S10-14 or S10-15 are repeated until the position matching XΓ11, XΓ12,..., XΓfj,. When the above process is completed for k partial images, the process proceeds to S10-17.

-Step S10-17;
Steps S10-1 to S10-16 are repeated until the comparison based on the matching rate is completed for all the drawing drawings for search. When the comparison is completed for all the drawing drawings for search, the process proceeds to S10-17.

-Step S10-18;
Here, the number of partial image numbers (= number of similar partial images) stored in the RAM is checked, and if it is 0, the process proceeds to S10-6.

-Step S10-19;
Further, when the number of partial image numbers stored in the RAM is 1 or more, the total matching rate is obtained in any of the following cases a to d.
a: When there is no color designation or position designation, the square average value of the contour coincidence rate stored in the RAM is set as the total coincidence rate Ξγ.
b: When a color is designated, the mean square value of the contour coincidence rate and the color coincidence rate stored in the RAM is set as an overall coincidence rate Ξγ.
c: When the position is designated, the contour coincidence rate and the square mean value of the position coincidence rate stored in the RAM are set as the overall coincidence rate Ξγ.
d: When there is a color designation and a position designation, the mean square value of the contour coincidence rate, the color coincidence rate and the position coincidence rate stored in the RAM is set as the total coincidence rate Ξγ.

-Step S10-20;
The threshold is determined according to any of the following cases a to d, the overall matching rate Ξγ is compared with the threshold value, and if the total matching rate Ξγ <threshold, this image is rejected and the process proceeds to S10-22. When rate Ξγ ≧ threshold, the process proceeds to S10-21.

a: When there is no color designation and position designation, threshold = ρ (threshold for contour matching rate)
b: When color is specified, threshold = ρ ′ (threshold of color matching rate)
c: threshold value when position is specified = ρ ″ (position specifying threshold value)
d: When color designation and position designation are made, the maximum one of the threshold values ρ, ρ ′, ρ ″ is set as the threshold value.
In the present embodiment, the maximum of the threshold values ρ, ρ ′, ρ ″ is set as the threshold value. However, the present invention is not limited to this, and any one of ρ, ρ ′, ρ ″ is set as the threshold value. It is good.

-Step S10-21;
When the overall matching rate Ξγ exceeds the threshold, the image is set as a candidate image, and the image number is stored in the RAM.

Step S10-22;
The recognition process of steps S1-1 to S1-5 is repeated for the feature data of the recorded image recorded in the RAM, and when the recognition process is completed for all the recorded images, the process proceeds to S11.

Step S11: (Determination of candidate image display method)
The image number of the candidate image stored in the RAM is checked, and if there is one, the process proceeds to S12. If there are a plurality of candidate images, the process proceeds to S13. If there is no candidate image, the process proceeds to S14.

Step 12: (Search image display of one frame size)
The image data of the image number is read from the image recording memory 7, is decompressed to image data for one frame by the first decompression unit of the image reproduction unit 8, reproduces the search image, and sends it to the display unit 9. Displayed on the liquid crystal display 10. In this case, as shown in the example of FIG. 8C, a frame line surrounding the partial images (501 ′, 502 ′, and 503 ′) to be recognized by the drawing drawing for search may be displayed.

Step S13: (Search image multi-display)
When there are a plurality of candidate images, the image data of the search image number is sequentially read out from the image recording memory 7, and the image is reduced by the image reduction unit of the image reproduction unit 8 and sent to the display unit 8 to reproduce the search image image. Then, multi-display is performed on the liquid crystal display 10. In this case, as shown in FIG. 8D, when the number of candidate images is 2 to 4, the reduced size is displayed as 1/4, and when the number is 5 to 9, the reduced size is 1/9, 10 to 10. In the case of 16, the reduction size is automatically selected such that the reduction size is 1/16.

  Further, the reduction size may be fixed to 1/4 (or 1/9), and if there are many candidate images and the single screen multi-display cannot keep up, scrolling to the next screen may be performed.

  When the user points to a desired image from the multi-displayed images with a finger or a touch pen, the process proceeds to S12, and the image is displayed as an image having a size of one frame.

  In addition, when there are a plurality of candidate images, the images may be displayed one by one as an image having a size for one frame instead of being displayed in multiple. In this case, it is desirable to arrange images that have a large overall match rate (sorted in reverse order based on the overall match rate), and sequentially enable scroll display according to the user's points.

Step S14: (message display without search image)
If there is no candidate image, a message to that effect is displayed to prompt the user to input a drawing drawing for search (that is, redo drawing).
According to the above image search method, since the similarity between the handwritten drawing and the recorded image is recognized and the image recording memory is so-called fuzzy search, it is recorded in the image recording memory without the hassle of fast-forwarding and the troublesome keyword input. An image that matches the user's image can be found from a large number of images. In addition, when the color and position are used as search elements, the search accuracy is improved.

  In the flowchart of FIG. 3, the image recognition process (S10) is performed collectively after performing the feature extraction process (S6 to S8) of all the drawing drawings and the recorded image. However, the present invention is not limited to this. The feature extraction process may be sequentially performed as necessary during the image recognition process (that is, S6 to S8 are not performed in the flow of FIG. 3 and the process of S6 to S8 is performed in S10).

  In the flowchart of FIG. 3, the image recognition is performed based on the comparison result of the feature elements of the drawing diagram and the recorded image. However, other methods (for example, the comparison process between the drawing diagram and the image of the recording image, Image recognition may be performed by a known method (in this case, the feature extraction process of S6 to S8 is not performed).

[Example 1-2] Image search operation by drawing input (example using pre-extracted image feature elements)
In the first embodiment, the features of the drawing and the recorded image are extracted for each search by the drawing, and the image recognition is performed. However, in this embodiment, the feature of the recorded image is extracted in advance (when the recorded image is recorded). Then, it is recorded in the feature list secured in the image recording memory 7, and when drawing search is performed, only the feature of the drawing is extracted and compared with the feature of the recorded image registered in the feature list. It is an example which performs image recognition.

  FIG. 12 is a structural diagram of an embodiment of a recorded image feature list. A recorded image feature list 80 stores an image number column 81 for storing an image number of the recorded image and feature data of a partial image of the image. Partial image feature columns 82-1, 82-2,..., 82-n are provided. Each partial image feature column 82 includes a plurality of contour features (for example, feature vector values defined by the length and direction of line segments, coefficients of partial curves, radius of curvature of arcs), and color features (for example, partial It has subdivision columns for storing the color pixel (R, G, B ratio) with respect to the number of pixels of the image and the center coordinates (xi, yj) of the partial image.

These feature values can be automatically stored when a recorded image is input to the image recording memory 7 (when an image is captured or a received image is recorded from an external image processing apparatus).
FIG. 13 is a flowchart showing an operation example during the process of registering image feature data in the feature list.
In step T1 in FIG. 13, image data (usually when an image is taken (when the image search device includes an image pickup unit such as a digital camera) or when an image is received from an external image processing device) Is input), it is recorded in the image recording memory 7 (when the image data is not compressed, it is compressed and recorded).
Next, in step T2, a feature extraction process for the image is performed. The feature extraction processing here is performed on the input image data stored in the DRAM 6, but the same procedure as the above-described image feature extraction processing in FIG. 3 (steps S8 (S8-1 to S8-5 in FIG. 5)). Can be done. However, in this embodiment, since the color feature and the position feature are automatically registered in the feature list 80, it is determined whether or not the color is specified in step S8-3 in FIG. 5 and whether or not the position is specified in step S8-4. The determination operation is not performed (these procedures are deleted at T2).

  In step T3, the feature data obtained in step T2 is stored in the partial image feature columns 82-1, 82-2,..., 82-i for each partial image, and the image number of the image is stored in the image number column 81. And the feature data for one image is registered.

  In step T4, the above steps T1 to T3 are repeated by the number of input images.

FIG. 14 is a flowchart showing an image search operation example of the image search apparatus provided with the feature list.
In step U1 of FIG. 14, drawing input processing such as drawing input by the user, color designation (arbitrary), position designation (arbitrary) and the like is performed. The drawing input process can be performed in the same procedure as steps S1 to S5 in FIG.
In step U2, feature extraction processing is performed on the drawing for search drawn by the user. It should be noted that the feature extraction process can be performed by the same procedure as step S6 (= steps S6-1 to S6-5 in FIG. 4) of the image in FIG.
In step U3, the feature list 80 of the image recording memory 7 is temporarily stored in the RAM in order to improve the processing speed.
In step U4, the feature of the drawing for search extracted in step U2 is compared with the feature of each image registered in the feature list 80, image recognition processing is performed, and a candidate image is obtained. Note that the image recognition process can be performed by the same procedure as step S10 (= steps S10-1 to S10-22 in FIG. 6) of the image in FIG.

In step U5, image display processing is performed according to the number of candidate images. Note that the image display process can be performed in the same procedure as steps S11 to S14 of the image shown in FIG.
As described above, according to the present embodiment, the feature extraction of the recorded image is not performed at the time of the search by the drawing diagram, so that the high-speed search is possible. In addition, even when colors and positions are used as search elements, high-speed search can be performed, so that high-speed and high-precision search is possible.

<Embodiment (2)> Image Search Using Primary Extracted Image In this embodiment, the image search apparatus, as in Embodiment 1 described above, uses the features of the drawing drawing for search input with a finger or the touch pen 3A and the recorded image. A recorded image having a high degree of coincidence of feature elements is extracted as a candidate image by comparison with the feature elements. However, in order to increase the recognition accuracy and the search speed, all the images are characterized as in the first embodiment. A candidate image that is displayed as a candidate image by comparing the feature element with the feature element of the recorded image using one of the images retrieved by the drawing for retrieval without performing element recognition. By selecting an image imaged by the user from among the images, a desired image is retrieved without inputting a keyword or frame advance.

  In this embodiment, the circuit configuration of the image search apparatus is the same as that of the example of FIG. 2 for the sake of explanation, and the feature elements to be compared are also the same as those of the example of the embodiment (1), but are not limited thereto. The same applies to other explanatory diagrams (FIGS. 1, 7 to 10, FIG. 12, and FIG. 13).

[Example 2-1]
FIG. 15 is an explanatory diagram of an image search example using a primary extracted image.
15A is an example of search drawing diagrams 501 and 503 input by handwriting by the user, and FIG. 15B is an image (image 105) searched based on the characteristic elements of the search drawing diagrams 501 and 503. (C) shows that the image 105 is not the user's desired image, but there are partial images 501 'and 503' similar to the search image, so that the user designates it as a search image and is searched. In this example, a plurality of images (images 151 to 154) are displayed in a multi display manner.

FIG. 16 is a flowchart illustrating an operation example of the image search apparatus using the primary extracted image.
Step V1: (Drawing input process)
Here, drawing input processing such as drawing input by the user, color designation (arbitrary), position designation (arbitrary), etc. is performed. The drawing input process can be performed in the same procedure as steps S1 to S5 in FIG.
Step V2: (Characteristic extraction process for search drawing)
Next, the feature extraction process of the drawing for search drawn by the user is performed. It should be noted that the feature extraction process can be performed by the same procedure as step S6 (= steps S6-1 to S6-5 in FIG. 4) of the image in FIG.
Step V3: (Simple decompression of image data)
After the feature drawing process of the drawing for search in step V2 is completed, the control unit 4 reads out image data (compressed image data) for one image recorded in the image recording memory 7 and reads out the image reproduction unit 8. The second decompression unit decompresses the compressed image data into an image having the same size as the image before compression and a coarser density than the original image, and stores the decompressed image data in the DRAM 6.

Step V4: (Image data feature extraction processing)
Next, feature extraction processing is performed on the expanded image data stored in the DRAM 6. This feature extraction process can be performed by the same procedure as the feature extraction process (step S8 (S8-1 to S8-5 in FIG. 5)) of FIG.

Step V5: (Repeated feature extraction of recorded image)
Steps V3 and V4 are repeated until feature extraction is completed for all the recorded images.

Step V6: (Primary extraction process)
The feature of the drawing for search extracted in step V2 is compared with the feature of the image extracted in step V4, image recognition processing is performed, and a candidate image is obtained. The image recognition process can be performed by the same procedure as steps S10-1 to S10-21 in FIG. 6 in step S10 of the image in FIG. 3 described above (step S10-22 is unnecessary in this case).

Step V7: (Image end determination)
If a candidate image is found in V6, the transition is made to V9 as a primary extracted image. If no candidate image is found, the end of the recorded image is determined. If there is still a recorded image, the process returns to V6 and the next candidate image is recognized.

Step V8: (When there is no primary extraction image)
If it is determined in step V7 that the recorded image has been completed, it is determined that the primary extraction image has not been found, and a message “No primary extraction image” is displayed on the liquid crystal display 10 and the drawing is re-input. To return to V1.

Step V9: (Display of primary extracted image)
In order to make the user confirm the candidate image extracted in step V6, the image is expanded to the size of one frame and displayed on the liquid crystal display 10. At this time, confirmation icons (OK icon and NO icon (not shown)) are displayed on the screen.

Step V10: (Confirmation of primary extracted image)
When the user points to the OK icon, the process proceeds to V11 on the assumption that the displayed primary extracted image is used as a search image. When the NO icon is pointed, the process returns to V6 to hit the next extracted image.

Step V11: (Image recognition process using primary extracted image)
Since the feature data of the primary extraction image extracted in step V6 is obtained in step V4, the feature data of the primary extraction image stored in the RAM and the feature of the image extracted in step V4 are compared to perform image recognition processing. To find a candidate image.
Note that the image recognition processing can be performed by the same procedure as step S10 (= steps S10-1 to S10-22 in FIG. 6) of the image in FIG. However, the feature data of the primary extracted image is used instead of the feature data of the drawing for search.

Step V12: (Candidate image display processing)
When the search for all recorded images is completed, image display processing is performed according to the number of candidate images. Note that the image display process can be performed in the same procedure as steps S11 to S14 of the image shown in FIG.

  According to the above method, the second and subsequent search images can be searched using feature data based on an image that is less ambiguous (that is, more accurate) than a handwritten image (drawing diagram). The search accuracy is improved as compared with the case of 1 (Example 1-1). Also, the search speed is increased. The steps V9 and V10 may be omitted.

[Example 2-2]
In this embodiment, it is also possible to perform recognition processing using image features extracted in advance as in Example 1-2. In this case, if a feature list (FIG. 12) in which features of recorded images are registered in advance is recorded in the image recording memory 7, step V10 in FIG. 16 is replaced with steps V10-1 and V10-2 in FIG. In addition, since steps V3 to V5 are not necessary, not only the search accuracy is improved, but also high-speed search is possible.

FIG. 17 shows a flowchart when the recognition process is performed using the primary extracted image and the image feature extracted in advance.
In FIG. 17, in step V10-1, the feature list of the image recording memory 7 is temporarily stored in the RAM in order to improve the processing speed.
In step V10-2, the feature of the primary extraction image extracted in step V6 is compared with the feature of each image registered in the feature list, image recognition processing is performed, and a candidate image is obtained. Note that the image recognition process can be performed by the same procedure as step S10 (= steps S10-1 to S10-22 in FIG. 6) of the image in FIG. The other steps are the same as in FIG.

[Example 2-3]
In the above embodiment, a candidate image is extracted from a recorded image using a drawing for searching for a handwriting, and a candidate image is further searched from the recorded image using the extracted image. A search method including a known image search method such as fast-forwarding is used. The recorded image can be searched using the image obtained by the above. That is, a search including a known search method such as fast-forwarding is performed in step V1 of FIG. 16 or FIG. 17, the feature extraction processing of the image searched in step V2 is performed, and steps V6 to V9 are deleted. Note that the feature extraction processing can be performed by the same procedure as the feature extraction processing (step S8 (S8-1 to S8-5 in FIG. 5)) of FIG. 3 described above. Thereby, an accurate and quick image search can be realized.

[Embodiment 2-4] Addition of Erasing Function of Unnecessary Part Further, when the primary extraction image used for search in FIGS. 16 and 17 is a person (for example, the same person as a person recorded in another recorded image) Even if the face is the same, the clothes and posture are often different. In this case, the recognition efficiency is expected to decrease, so only a specific part (face in the case of a person) with little change is expected. The use of the search image improves the recognition efficiency and does not cause any inconvenience in the search.

Therefore, after an erasure selection button is provided (or an erasure designation icon is displayed on the screen) and the button is operated so that an unnecessary image of the primary extracted image displayed in step V8 can be erased. (Or, after pointing the icon), if you specify the erase range with the cursor or your finger or touch pen, you can erase that part.
As a result, unnecessary feature elements are not used for the search, so that the search speed is further improved and the search accuracy is further improved.

<Embodiment (3)>
In this embodiment, the recorded image is searched using the captured image. In addition, an image capturing device having an image capturing unit and a display unit, such as a digital camera, is used as the image search device.
That is, a recorded image with a high matching rate of feature elements is set as a candidate image by comparing the characteristic elements of the captured image with the characteristic elements of the recorded image, and an image (FIG. 1C) imaged by the user from among the displayed candidate images. In the example, by selecting an image 111), a desired image can be searched without inputting a keyword or frame advance.
In the present embodiment, the feature elements to be compared are the same as in the example of the embodiment (1), but are not limited thereto.

[Circuit configuration example]
FIG. 18 is a block diagram illustrating a circuit configuration example of a digital camera to which the present invention is applied. The digital camera 200 includes an imaging unit 11, an operation unit 12, a control unit 13, an image recognition unit 14, a DRAM 15, and an image recording memory. 16, an image processing unit 17, a display unit 18, and a liquid crystal display (LCD) 19 are provided.

  The imaging unit 11 includes an imaging device that converts an optical image collected via an imaging system, an optical system such as an imaging lens, a lens driving mechanism, and a diaphragm mechanism into an electrical signal, an imaging device driving unit, and the obtained electrical signal. It has a signal converter for converting into digital data.

  The operation unit 12 includes a power switch, a processing mode selection button, and the like, as well as imaging buttons and switches such as a shutter button. When the user operates a switch or button of the operation unit 12, an electrical signal (code) corresponding to the operation content is sent to the control unit 13. As a result, the user can give the control unit 13 instructions necessary for starting up the digital camera 200 and processing operations.

  The control unit 13 includes a CPU, a program storage memory (ROM), and a temporary storage memory (ROM) (not shown), and controls the overall operation of the digital camera 200 and the execution control of processing in each processing mode. In addition to imaging, image recording, reproduction, and display, execution control of various processing modes including image search processing of the present invention is performed. In the image search processing mode, an image search program is temporarily loaded from the program storage memory to the RAM, and execution of the image search processing is controlled.

  The image recognition unit 14 extracts feature elements of the image captured in the image search processing mode, stores them in the RAM, and performs recognition processing by checking the degree of coincidence with the feature elements of the recorded image.

  The DRAM 15 is used as a work buffer for processing in addition to temporary storage of image data captured by the imaging unit 11, temporary storage of reproduced image data, storage of image data during image recognition, and the like.

  The image recording memory 16 is composed of a recording medium such as a flash memory or a memory card, and records compressed image data. It is also possible to record simple image (thumbnail image) data having a smaller size and recording density than the captured image. If an image search is performed using the feature elements of the recorded image in advance, an area for recording the feature list as shown in FIG. 12 is secured. When image management is performed on image data, an image file management list area for managing image data in units of files is secured in addition to image data.

  The image processing unit 17 includes circuits such as an image data expansion / data compression unit, a color processor interface, a communication interface, and a display control interface. The image processing unit 17 is controlled by the CPU to control main operations of the digital camera 200, for example, image processing. Image processing operations such as capture, imaging processing, compression of image data, recording of compressed data, reproduction (extraction of compressed image data, expansion of compressed image data), and the like are executed.

  Also, the image data expansion / compression unit compresses the image data into an image compression unit that compresses the image data, and expands the compressed image data to an image having the same size and the same density as the image before compression in the same manner as the image reproduction unit in FIG. A first decompression unit, a second decompression unit that decompresses the compressed image data into an image having the same size as the uncompressed image and a coarser density (which means that the density is coarser) than the original image, and the compressed image data It has an image reduction unit that reproduces an image that has been expanded and reduced to a smaller size than the original image.

  If the digital camera 200 includes an image recording memory having a sufficiently large memory capacity when configured to display multiple images, the image reduction unit as described above is not provided and the image is recorded in advance. An image can be reduced to a plurality of sizes and stored in association with each image, and a reduced image having a size required for display can be extracted.

  The display unit 18 includes a video encoder and an image memory such as a VRAM, converts image data or drawing input data into image data, develops a bitmap in the VRAM, and displays an image on a liquid crystal display (LCD) 19. .

[Example 3-1]
FIG. 19 is an explanatory diagram of an image search example based on a photographed image. In FIG. 19, an image 191 shown in FIG. 19A is an image of an infant imaged as a search image in order to search an image of a child (infant) taken with the digital camera 200 one year ago from recorded images. is there.

  After the infant is imaged by the digital camera 200, when the processing mode is set to the image search processing mode, the feature element of the infant image 191 taken now is extracted, compared with the feature element of the recorded image, and recognized. The child image 192 shown in (b) and the image 193 together with the mother shown in (c) are examples of images searched and displayed based on the characteristic elements of the infant image 191.

  FIG. 20 is a flowchart showing an example of an image search operation of the digital camera based on a photographed image.

Step W1: (Imaging)
A subject to be used as an image to be used for search in the REC mode (imaging / recording mode) is captured. That is, when the user presses the shutter button of the operation unit 2, the subject image is converted into an electrical signal by the imaging unit 11, and the subject image is taken into the DRAM 15 as image data. The image data taken into the DRAM 15 is subjected to color processing image processing by the image processing unit 17, sent to the display unit 18 as a video signal, and displayed as a still image on the liquid crystal display 19.

Step W2: (Selection of image search processing mode)
Here, when the user operates an image search processing mode selection button (an image search processing mode selection icon may be displayed on the screen) assigned to the function selection button of the operation unit 12, the image search processing mode is set and the process goes to step W3. Transition (if the image search processing mode is not selected within a predetermined time, shift to image recording processing). Note that the image search processing mode may be selected in step W1, and imaging may be performed in step W2. In this case, the process may move to step W3 immediately after the image is taken in step W2, or may be moved to step W4 when a predetermined key is operated after the image is taken.

Step W3: (Imaging feature extraction processing)
Next, a feature extraction process is performed on the image that has just been captured. Note that the feature extraction processing of the captured image can be performed by the same procedure as the above-described step S10 (= steps S8-1 to S8-5 in FIG. 5) of FIG. However, the color designation determination in step S8-3 and the position designation determination in step 8-4 are not performed.

Step W4: (Simple decompression of image data)
After the imaging feature extraction process in step W3 is completed, the control unit 13 reads out the image data (compressed image data) for one image recorded in the image recording memory 16, and the second image processing unit 17 performs the second processing. The decompressing unit decompresses the compressed image data into an image having the same size as the image before compression and a coarser density than the original image, and stores the decompressed image data in the DRAM 15.

Step W5: (Image data feature extraction processing)
Next, a feature extraction process is performed on the expanded image data stored in the DRAM 15. This feature extraction process can be performed by the same procedure as the feature extraction process (step S8 (S8-1 to S8-5 in FIG. 5)) of FIG. However, the color designation determination in step S8-3 and the position determination confirmation in step 8-4 are not performed.

Step W6: (Repeated feature extraction of recorded image)
Steps W4 and W5 are repeated until feature extraction is completed for all the recorded images.

Step W7: (Image recognition process using captured image)
The feature data of the captured image extracted in step W3 and stored in the RAM is compared with the feature of the recorded image extracted in step W5, and image recognition processing is performed to obtain a candidate image. Note that the image recognition processing can be performed by the same procedure as step S10 (= steps S10-1 to S10-22 in FIG. 6) of the image in FIG. However, the feature data of the picked-up image is used instead of the feature data of the drawing for search, and the color designation determination in steps S10-5 to S10-10 and the positions in steps 10-11 to S10-16 are used. Do not make the specified judgment.

Step W8: (Search image display process)
When the search for all recorded images is completed, an image display process is performed in accordance with the number of searched candidate images. Note that the image display process can be performed in the same procedure as steps S11 to S14 of the image shown in FIG. By the above image search method, more accurate search can be performed as compared with the first and second embodiments. In addition, a quick image search is possible.

[Example 3-2] An example using a feature list In addition, it is possible to perform recognition processing using image features extracted in advance as in Example 1-2 in this embodiment. In this case, if a feature list (FIG. 12) in which features of recorded images are registered in advance is recorded in the image recording memory 16, step W7 in FIG. 20 is replaced with steps W7-1 and W7-2 in FIG. Since steps W4 to W6 are not necessary, not only the search accuracy is improved, but also a higher speed search is possible.

FIG. 21 shows a flowchart in the case of performing recognition processing using a captured image and image features extracted in advance.
In FIG. 21, in step W7-1, the feature list of the image recording memory 16 is temporarily stored in the RAM in order to improve the processing speed.
In step W7-2, the feature of the primary extraction image extracted in step W3 is compared with the feature of each recorded image registered in the feature list, and image recognition processing is performed to obtain a search candidate image. Note that the image recognition process can be performed by the same procedure as step S10 (= steps S10-1 to S10-22 in FIG. 6) of the image in FIG. The other steps are the same as in FIG.

[Embodiment 3-3] Addition of Unnecessary Part Erasing Function When the subject used for searching in FIGS. 20 and 21 is a person (for example, the infant 191 in FIG. 19A), it is recorded in a recorded image. Even if the face is the same as the person who is present (for example, the infant 192 in FIG. 19B), the clothes and posture are often different. In such a case, it is expected that the recognition efficiency is lowered. Therefore, if only a specific part (a face in the case of a person) having a small change is used as a search image, the recognition efficiency is improved and the search is inconvenient. There is no point.

Therefore, an erasure selection button is provided (or an erasure designation icon is displayed on the screen) so that an unnecessary portion of the captured image displayed in step W1 can be erased, and a button operation is performed (or When an erasing range is designated with a cursor or a finger or a touch pen, the portion can be erased, and feature extraction from the imaging in step W2 is performed from a specific portion (for example, a face) after erasure.
As a result, unnecessary feature elements are not used for the search, so that the search speed is further improved and the search accuracy is further improved.

  Although several embodiments of the present invention have been described above, it is needless to say that the present invention is not limited to the above-described embodiments and various modifications can be made.

It is outline | summary explanatory drawing of the image search method by the drawing input of this invention. It is a block diagram which shows the structural example of an image search device. It is a flowchart which shows the operation example of the image search device based on the image search method of this invention. It is a flowchart which shows one Example of the feature extraction process of a drawing figure. 6 is a flowchart illustrating an example of a feature extraction process of a recorded image. It is a flowchart which shows one Example of an image recognition process. It is a figure which shows the example of the selection screen for drawing. It is explanatory drawing of the example of an image search by the drawing input of only an outline. It is explanatory drawing of the example of an image search by drawing with a color. It is explanatory drawing of the example of an image search by the position specification of a drawing. It is explanatory drawing which shows an example of a line segment vector. FIG. 10 is a structural diagram of an example of a feature list of recorded images. It is a flowchart which shows the operation example at the time of the feature data registration process of the image to a feature list. It is a flowchart which shows the image search operation example of the image search device provided with the feature list. It is explanatory drawing of the image search example by a primary extraction image. It is a flowchart which shows the operation example of the image search apparatus using a primary extraction image. It is a flowchart in the case of performing a recognition process using a primary extraction image and the image feature extracted previously. It is a block diagram which shows the circuit structural example of the digital camera to which this invention is applied. It is explanatory drawing of the example of an image search by a picked-up image. It is a flowchart which shows the image search operation example of the digital camera by a picked-up image. It is a flowchart in the case of performing recognition processing using a photographed image and image features extracted in advance.

Explanation of symbols

3 Drawing input section (drawing input means)
5,14 Image recognition unit (drawing feature extraction means, primary image extraction means, image feature extraction means, image recognition means, primary image extraction means)
7,16 Image recording memory (image recording means)
9, 18 Display section (notification means)
10, 19 Liquid crystal display (notification means)
100,200 Digital camera (electronic still camera; image search device)
191 Photographed image 501, 502, 503, 701 Retrieval drawing for drawing (drawing diagram; image)

Claims (15)

  An image search method characterized by drawing an image, sequentially comparing the drawing and each image recorded in an image recording memory, and extracting and displaying a recorded image having a high degree of similarity.   An image is drawn, and one of the similar images obtained by sequentially comparing the drawing and each image recorded in the image recording memory is used as a search image, and each image recorded in the image recording memory is An image search method comprising: comparing, extracting and displaying a recorded image having a high degree of similarity.   One of the images recorded in the image recording memory is selected as a search image, the search image is compared with each image recorded in the image recording memory, and a recorded image with high similarity is extracted. An image search method characterized by displaying.   In an imaging apparatus, an image search characterized in that a captured image is used as a search image and the captured image is sequentially compared with each image recorded in an image recording memory, and a recorded image having a high degree of similarity is extracted and displayed. Method. 5. The image search method according to claim 1, wherein the comparison is performed by comparing any of the following a to d.
a: Contour b: Contour and color c: Contour and its position d: Contour, its position and color   5. The image search method according to claim 4, wherein a feature element obtained by extracting any one of a to d from each image recorded in the image recording memory is registered in the memory in advance.   The image search method according to any one of claims 1 to 6, wherein the images having high similarity are displayed in a reduced size according to the number of images.   5. The image search method according to claim 2, wherein the search image is sequentially compared with a recorded image after erasing unnecessary portions. 6. An image recording means for recording a plurality of images, a drawing input means for drawing and inputting a picture or a figure,
Image recognition means for obtaining a recorded image having a high degree of similarity by comparing the picture or figure drawn and inputted by the drawing input means with the images recorded in the image recording means;
An image search apparatus comprising: notification means for notifying a comparison result by the image recognition means. An image recording means for recording a plurality of images, a drawing input means for drawing and inputting a picture or a figure,
Primary image extraction means for comparing the picture or figure drawn and input by the drawing input means with each image recorded in the image recording means to obtain a similar image;
An image recognition means for comparing the similar image obtained by the primary image extraction means with the images recorded in the image recording means to obtain a recorded image having a high degree of similarity;
An image search apparatus comprising: notification means for notifying a comparison result by the image recognition means. Image recording means for recording a plurality of images, image selecting means for selecting one of the images recorded in the image recording means as a search image,
An image recognition means for comparing the search image selected by the image selection means with each image recorded in the image recording means to obtain a recorded image having a high degree of similarity;
An image search apparatus comprising: notification means for notifying a comparison result by the image recognition means.   The image search apparatus according to claim 9 or 10, wherein the image input means is handwritten input.   The image recognizing unit extracts the feature element of the image for search selected by the image selection unit selected by the image selection unit, the similar image obtained by the primary image extraction unit, or the picture or figure drawn by the drawing input unit. The image search device according to claim 9, wherein the image search device is compared with a feature element extracted from each image recorded in the image recording unit.   The image search device according to claim 9, wherein the notification unit is a display unit. In an electronic still camera provided with an image recording means and a display means, a photographing means for photographing a subject and outputting a photographed image;
An image recognizing unit that compares the captured image output from the image capturing unit with each image recorded in the image recording unit to obtain a recorded image having a high degree of similarity;
An electronic still camera comprising: an informing means for informing a recognition result by the image recognition means.

Images