JP2001143080A - Method and device for retrieving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval interface, capable of speedily reflecting user's intention on an image retrieval. SOLUTION: When retrieving desired image from an image database in which plural images are stored, the degree of similarity among a specified image at a retrieval origin and each of the plural images is calculated, based on featured values of the specified image at the retrieval original and featured values of each of the plural image (steps S71 to S77). The similar image is extracted from the plural images, based on the calculated degree of similarity of the images and presented as a retrieval result. Values of A and B are set by setting, to which of brightness and color difference importance is attached in calculation of the degree of similarity here and calculation of the degree of similarity using set weighting is executed, by using the values of A and B in the calculation of the degree of similarity in the step S75.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image search method, apparatus, and medium for searching for desired image data from a plurality of image data.

[0002]

2. Description of the Related Art Various searching means for searching for a desired image from an image database storing a plurality of image data have been proposed. This type of search processing is performed by associating image data with non-image information such as keywords and shooting dates and performing a search based on the image data.Features of the image itself (information such as luminance / color difference information, image frequency, histogram, etc.) ) Based on the search method.

In the latter method, a method of presenting a certain image to an image database and searching for an image using a feature amount of the image as a search key is particularly called similar image search. This is advantageous in that a user who does not have special knowledge about image processing can be provided with an intuitive and easy-to-understand search interface.

[0004]

When there is no image data serving as a search key at hand of a user, for example, a search is performed based on the memory of the user, or a search is performed based on an image idea originally created by the user. May be done. When a user writes an image stored by “handwriting” and performs a similar image search using this image as a key image,
Human memory is ambiguous. In particular, with regard to “color”, in many cases, only the characteristic color is memorized. In many cases, the user often remembers a color close to the primary color, such as “red”. For this reason, it is often difficult to search for the image requested by the user even if the color (RGB) of the image of “handwritten” is directly compared as an image search key. That is, it is very difficult to use the similar image search user interface to perform a search using a handwritten image as an original image,
There was a problem that it could not be used.

[0005] Alternatively, the user needs to search for image data serving as a key by some method, for example, selecting a candidate image by a search using the above-mentioned keyword, and further, using the obtained image as a key image, a similar image search is performed. Tedious operations, such as

[0006] The present invention has been made in view of the above problems, and has as its object to enable an image search that quickly reflects the intention of a user.

[0007]

An image retrieval method according to the present invention for achieving the above object includes, for example, the following steps. That is, an image retrieval method for retrieving a desired image from storage means storing a plurality of images, wherein the retrieval source based on a feature amount of a designated retrieval source image and a feature amount of the plurality of images. Calculating a similarity between an image and each of the plurality of images; extracting similar images from the plurality of images based on the image similarity calculated in the calculating step, and presenting a search result The presenting step, comprising a setting step of setting different weights for each parameter of the feature amount used in the calculating step, in the calculating step,
The similarity is calculated by using the weight set by the setting unit.

Further, an image retrieval apparatus according to the present invention for achieving the above object has, for example, the following configuration. That is, an image retrieval apparatus for retrieving a desired image from a storage unit storing a plurality of images, the retrieval source based on a feature amount of a designated retrieval source image and a feature amount of the plurality of images. Calculating means for calculating a similarity between an image and each of the plurality of images; extracting similar images from the plurality of images based on the image similarity calculated by the calculating means; and presenting a search result Presenting means, and setting means for setting different weights for each parameter of the characteristic amount used in the calculating means, wherein the calculating means calculates the similarity using the weight set by the setting means. Execute.

[0009]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[First Embodiment] FIG. 1 is a block diagram showing the configuration of an image retrieval apparatus (computer system) according to a first embodiment.

In FIG. 1, reference numeral 101 denotes a CPU which controls the entire system. A keyboard 102 is used together with the mouse 102a to input a user's instruction operation to the system. 103 is a display unit, CR
It is composed of T and liquid crystal. 104 is ROM, 10
Reference numeral 5 denotes a RAM, which constitutes a storage device of the system;
01 and the data used by the system. 106 is a hard disk drive, 107
Is a floppy disk drive, which constitutes an external storage device used for the file system of the system. 108
Is a printer.

FIG. 2 is a flowchart showing a general flow of image retrieval according to the first embodiment. Step S
At 21, the user draws an illustration on the display unit 103 that resembles a desired image. In step S22, the amount of the drawn image is calculated. In step S23, a similar image is searched based on the feature amount. Step S24
Then, the similar image data obtained by the above search is displayed on the display unit 103. Hereinafter, each of these steps will be described in detail.

[Description of Step 21] FIG. 3 is a diagram showing an example of an operation screen displayed on the display unit 103 in Step 21. 31 is a user drawing area, 32 is a color designation scroll bar, 33 is a clear button, 34 is an undo button, 35 is a radio button for designating the thickness of the pen,
36 is a tool palette button, 37 is a search execution button,
Reference numeral 38 denotes a setting button for setting a search condition.

The user can write an illustration image similar to an image to be searched in the user drawing area 31 using the drawing tool realized by software and the mouse 102a. The outline of the software operation during drawing of the illustration image is as follows.

The color designation scroll bar 32 is a scroll bar for designating a pen color used for drawing. The R, G, and B values are specified in order from the top. When the clear button 33 is pressed, the entire drawing area 31 is painted white and the drawing area 3
1 can be in the initial state. Undo button 34
Pressing cancels the last operation and returns to the previous state. Further, the thickness of the pen for drawing in the drawing area 31 can be designated using the radio button 35.
Further, when the left “pen” is selected on the tool palette 36, the user
Using a, a free curve can be drawn on the drawing area 31. When the "straight line" at the center of the tool palette 36 is selected, a straight line can be drawn by designating the start point and the end point of the straight line using the pointing device 102a. When "circle" on the right side of the tool palette 36 is selected, a circle can be drawn by designating the center and radius of the circle as in the case of the above "pen" and "straight line".

After drawing an illustration in the drawing area 31 using the drawing tool as described above, a search execution button 37
By pressing, the process ends step S21 and proceeds to step S22.

At this time, the user can operate the setting button 38 before pressing the search execution button 37 to change the conditions for similar image search after step S22. That is, when the user presses the setting button 38, it is possible to select whether to calculate the similarity at the time of searching for a similar image with “emphasis on luminance” or on “emphasis on color difference”. When the setting button 38 is pressed, a setting menu 39 is displayed. In the first embodiment, the user can select any one of three of “emphasis on luminance”, “emphasis on color difference”, and “no emphasis”. .

"Description of Step S22" Step S22
Then, the image feature amount of the illustration image is calculated. FIG. 4 is a diagram for explaining an image division state according to the present embodiment.
As shown in FIG. 4, the size of the drawing area 31 is W pixels in the horizontal direction and H pixels in the vertical direction. In the present embodiment, this is divided into three parts in the horizontal direction and two parts in the vertical direction, that is, a total of six divisions, and the areas (0, 0),... Then, the average value of the R, G, and B values of each of these areas is calculated, and a total of 18 numerical values are used as the image feature amount of the illustration image.

FIG. 5 is a flowchart for explaining the image characteristic amount calculation processing according to the present embodiment. The flow of the feature amount calculation process will be described with reference to the flowchart in FIG. In step S51, a variable k is initialized with a value of 0. Step S
At 52, a variable j is initialized with a value of 0. In step S53, a variable i is initialized with a value of 0. In step S54, the average value of R of the area (i, j) is substituted for the k-th element d (k) of the array d. Similarly, the average value of G is set to d (k + 1), and d
The average value of B is substituted for (k + 2). Note that R, G, B
The method of calculating the average value will be described later with reference to the flowchart of FIG.

In step S55, k is increased by 3.
In step S56, i is increased by one. Step S5
At 7, i is compared with the value 2, and if it is larger than 2, the process proceeds to S58. Otherwise, the process returns to S54. In step S58, j is increased by the value 1. In step S59, j is set to 1
If it is larger than 1, the process is completed. Otherwise, the process returns to S53.

When the above processing is completed, the image feature amount of the illustration image is stored in the array d () having 18 elements. Here, the image is divided into six equal-area rectangular regions for the purpose of calculating the feature amount. However, the dividing method is not limited to a rectangle, and a more complicated shape may be used, or the number of divisions may be increased. When the number of divisions is increased or decreased, the number of elements in the array for storing the feature amount is not 18, but the number of elements is increased or decreased accordingly, and the values used in the determinations in step S57 and step S59 also change. Will do.

FIG. 6 is a flowchart illustrating a method of calculating the average values of the R, G, and B values for each area. It is assumed that image data is stored in three arrays of R (X, Y), G (X, Y), and B (X, Y). However, 0 ≦ X <
W, 0 ≦ Y <H, starting at the upper left corner of the image (0,0)
And In the following flow, X0 ≦ X <X1, Y0 ≦ Y
<Calculate the average density of the partial area of Y1 and obtain the variables DR, D
The average values of the R, G, and B values are substituted for G and DB, respectively.

In step S54, when the area corresponding to the area (i, j) is equally divided into 3 × 2 as shown in FIG. 4, X0 = W × i / 3 and X1 = W × ( i + 1) / 3 Since Y0 = H × j / 2 and Y1 = H × (j + 1) / 2, the constants X0, X1, Y0, and Y1 are initialized with the above values, and are shown in FIG. Execute the flowchart.

First, in steps S61, the variables DR, DG, D
Initialize B with the value 0. In step S62, the variable Y is set to Y0
Initialize with In step S63, a variable X is initialized to X0. In step S64, the value of R (X, Y) is added to DR. Similarly, the value of G (X, Y) is stored in DG, and the value of B is stored in DB.
Add the value of (X, Y). In step S65, the variable X is set to 1
Just increase. In step S66, the variables X and X1 are compared. If they are equal, the process proceeds to step S67, otherwise, the process proceeds to step S64. In step S67, the variable Y is increased by the value 1. In step S68, the variables Y and Y1 are compared. If they are equal, the process proceeds to step S69; otherwise, the process proceeds to step S63. The above steps S63 to S
By repeating 68, the sum of all pixels in the area for each of the RGB color components is obtained by DR, DG, and DB betting.

At step S69, variables DR, DG, DB
Is divided by (X1−X0) × (Y1−Y0). This is the total number of pixels in the area. That is, DR, D
G and DB are average densities obtained by dividing the sum of pixel densities in the area by the number of pixels.

"Description of Step S23" Step S23
Then, a similar image search is performed based on the image feature amount.
It is assumed that N pieces of image data are accumulated in the hard disk device 106, and the feature amount of each image is calculated and stored in advance by the same method as described above. The image data is a well-known JPEG, Flash
It may be stored in a standard file format such as Pix, or may be stored in a so-called RDBMS (relational database management system) in a unique format. The image feature amount is a two-dimensional array D (n, j) having a size of N × 18 (where 0 ≦ n <N, 0 ≦ j
It shall be stored in <18).

At this time, the inter-image distance S (n) between the presentation image and the n-th image is defined by the following equation.

[0028]

(Equation 1)

It is determined that the smaller the distance between the images, the higher the similarity between the images. First, the inter-image distance S (n) (0 ≦ n <N) between all N images and the presentation image is calculated, and then M (0 <M <N) in ascending order of S (n) , A similar image search is performed. Hereinafter, the calculation processing of S (n) will be described with reference to FIG. 7 and the selection processing of M images will be described with reference to the flowchart of FIG.

First, before entering the flowchart of FIG.
The variables A and B are set according to the conditions specified by the list button shown in FIG. In the case of "none", A = B = 50 is set. When “luminance priority” is selected, for example, A = 75 and B = 25 are set. In the case of "color difference emphasis", for example, A = 25 and B = 75 are set. These values of A and B are used in the calculation of the distance between images using the feature amount (described later), and the calculation of the case of emphasizing the luminance and the case of emphasizing the color difference are switched according to the values of A and B.

When the values of A and B have been set as described above, first, in step S71, the feature amount of the key image data is converted into a color space represented by luminance and color difference. In the case of the present embodiment, each average value of each RGB is converted into each value of YCbCr. This conversion formula is, for example, Y = 0.299R + 0.587 + 0.114B Cb = −0.1687R−0.3323G + 0.5B +
128 Cr = 0.5R-0.4187G-0.0813B + 1
28.

At this time, the values of Y, Cb, and Cr are stored in the area where the average values of R, G, and B are stored. In step S72, a variable n is initialized with a value of 0. Step S7
In step 3, the feature amount of the n-th image data stored in the image database is changed to YCbC
Convert to r. In step S74, variables i and S (n) are initialized with a value of 0.

In step S75, D (n, i) and d (i)
Is multiplied by either A / 100 or B / 100, and the square of the result is added to S (n). In this example, the luminance value (Y) is assigned to D (n, i) and d (i).
The color difference value (Cb) is assigned to (n, i + 1) and d (i + 1).
The color difference value (Cr) is stored in (n, i + 2) and d (i + 2). Therefore, (D (n, i) -d (i))
/ 100, and (D (n, i + 1) -d (i + 1))
And (D (n, i + 2) -d (i + 2)) by B / 100.

In step S76, the variable i is increased by the value 3. In step S77, the variable i and 18 are compared. If they are equal, the process proceeds to step S78. Otherwise, the process proceeds to step S7.
Go to 5. In step S78, the variable n is increased by the value 1. In step S79, the variables n and N are compared, and if they are equal, the process ends. Otherwise, the process returns to step S73.

In the array S (n) calculated as described above,
The inter-image distance between the presented image and all stored images was stored. Here, the inter-image distance is calculated by performing weighting corresponding to “emphasis on luminance” and “emphasis on color difference” as described above. Next, a process of selecting M images in ascending order of image distance and storing the image numbers in the array T () will be described with reference to FIG.

In step S81, a variable j is initialized with a value of 0. In step S82, a variable i is initialized with a value of 0. In step S83, the variable min is initialized to a value of 0, and L is initialized to a sufficiently large value. In step S84, S (i) is compared with L,
If S (i) <L, the process proceeds to step S85; otherwise, the process proceeds to step S86.

In step S85, the value i is substituted for the variable min, and S (i) is substituted for L. In step S86, the variable i
Is increased by the value 1. In step S87, i and N are compared, and if they are equal, the process proceeds to step S88; otherwise, the process proceeds to step S83. In step S88, the value m is set to T (j).
Substitute in. In step S89, a sufficiently large value is substituted for S (min). In step S810, j is increased by the value 1. In step S811, j and M are compared, and if they are equal, the processing is completed. Otherwise, step S82
Return to In the above processing, the array T (j) (0 ≦ j <M)
, Image numbers are stored in descending order of similarity with the presentation image.

[Explanation of Step S24] FIG. 9 is a diagram showing an example of an operation screen displayed on the display unit 103 in Step S24. At 91, the presented illustration image is reduced and displayed. At 92a to 92h, the similar images searched by the above processing are reduced and displayed. The image corresponding to the image number having the highest similarity, that is, the image number stored in T (0) is displayed in 92a, the image corresponding to T (1) is displayed in 92b, and the image corresponding to 92 (h) is displayed in 92h. Displays the image with the lowest similarity h among them. For the reduced display, the image data stored on the hard disk may be decoded and reduced and displayed on the screen. Alternatively, low-resolution image data for icons such as FlashPix which is a standard image format may be used. If so, the image data may be decoded and displayed. When the button 93 is pressed, the next candidate, that is, the image corresponding to T (8) to T (15) is similarly reduced to 92a to 92h. This is T (M-1)
Can be repeated until When the reduced display (or icon) of the candidate image is designated, the details (original image) of the corresponding image are displayed. When the button 94 is pressed, the step S24 ends.

Thus, when "priority to luminance" is selected by the list button 39 in FIG. 3, similarity search in which "luminance" is weighted is performed, and when "priority to color difference" is selected,
A similar image search in which weighting is applied to “color difference” can be performed.

In this embodiment, as the values to be set for A and B, the value on the important side is 75, and the other value is 2
It was set to 5, but is not limited to this.

[Second Embodiment] In the first embodiment,
The values of A and B for performing the calculation with weighting are fixedly specified. In the second embodiment, an embodiment will be described in which the values of A and B can be specified by the user.

The user interface in this case is shown in FIG.
Is set to a scroll bar as shown in FIG. Then, control is performed such that only the “luminance” is calculated at the left end, only “color difference” is calculated at the right end, and both “luminance” and “color difference” are calculated with the same weight in the middle. That is, in the second embodiment, the weighted similar image search processing described in the first embodiment (FIG. 7) according to the setting of the scroll bar.
Are controlled in the range of 0 ≦ A <100 and 0 ≦ B <100. For example, the left end is A = 100, B = 0,
The right end is A = 0, B = 100, and the middle is A = B = 50.

In the above embodiment, the feature amount of the image data is calculated by calculating an average value of RGB once, and then calculating the average value of YCbCr
, But is not limited to this, and may be calculated from the beginning using YCbCr.

The layout of the operation screen operated by the user is not limited to the above embodiment. Further, the device for drawing sketch images by the user has been described using a mouse, but the device is not limited to this, and may be an input device such as a pen tablet or a touch panel.

As described above, in each of the above embodiments, the sketch pad is prepared on the computer operation screen, and the user draws an image similar to a desired image on the screen. Then, the database system extracts a feature amount from the image, and performs a similar image search based on the feature amount.

When this similar image search is performed, luminance information and color difference information are generated from the feature amount of the image data, and a similar image search can be performed with emphasis on luminance or color difference, thereby enabling a complicated search. . This makes it possible to provide an image search interface that allows a search to quickly reflect the intention of the user. As described above, according to the above embodiment, a search can be performed without much consideration of color information (color difference) such as “bright” or “dark” based on “brightness”. For this reason, it is possible to cover the ambiguity of the color of the image stored by the user and perform an effective image search. In some cases, the user does not remember much "color" but does remember the shape. Even in such a case, better results can be obtained by performing a search with "emphasis on brightness". Also,
According to the above embodiment, various search conditions can be created from an image once written, and the user can be relieved of the burden of redrawing an image serving as a search key many times.

The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but can be applied to a single device (for example, a copier, a facsimile). Device).

Further, an object of the present invention is to supply a storage medium (or a recording medium) on which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or an apparatus, and a computer (a computer) of the system or the apparatus It is needless to say that the present invention can also be achieved by a CPU or an MPU) reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention. Also,
When the computer executes the readout program code, not only the functions of the above-described embodiments are realized, but also the operating system (OS) running on the computer based on the instructions of the program code.
It goes without saying that a case where the functions of the above-described embodiments are implemented by performing some or all of the actual processing, and the processing performs the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, the program code is read based on the instruction of the program code. Needless to say, the CPU included in the function expansion card or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0050]

As described above, according to the present invention,
It is possible to provide an image search interface capable of performing an image search that quickly reflects the intention of the user.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image search device according to a first embodiment.

FIG. 2 is a flowchart showing a rough flow of image search according to the first embodiment.

FIG. 3 is a diagram showing an example of an operation screen displayed on a display unit 103 in step 21.

FIG. 4 is a diagram illustrating a divided state of an image according to the present embodiment.

FIG. 5 is a flowchart illustrating a feature calculation process of an image according to the embodiment;

FIG. 6 is a flowchart illustrating a method of calculating an average value of R, G, and B values for each region.

FIG. 7 is a flowchart illustrating calculation processing of an inter-image distance according to the embodiment.

FIG. 8 is a flowchart illustrating a process of selecting a similar image.

FIG. 9 is a diagram showing an example of an operation screen displayed on the display unit 103 in step S24.

FIG. 10 is a diagram illustrating a screen configuration for interactive image presentation in the second embodiment.

[Explanation of symbols]

 101 CPU 102 Keyboard 102a Mouse 103 Display 104 ROM 105 RAM 106 Hard Disk 107 Floppy Disk 108 Printer

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Kunihiro Yamamoto, Inventor 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Sumi Kusama 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Incorporated F term (reference) 5B075 ND08 NK06 NK39 PP03 PP13 PQ02 PQ48 PR06 QM08 5L096 AA02 BA08 EA03 HA08 JA03 JA11 9A001 DZ13 FF03 HH23 JZ02 KK42

Claims (17)

[Claims] An image retrieval method for retrieving a desired image from a storage unit storing a plurality of images, the method comprising: determining a feature amount of a designated search source image and a feature amount of the plurality of images; Calculating a similarity between the search source image and each of the plurality of images; extracting similar images from the plurality of images based on the image similarity calculated in the calculation step; And a setting step of setting different weights for each parameter of the feature amount used in the calculation step. In the calculation step, the degree of similarity is calculated using the weight set in the setting step. An image search method characterized by performing a calculation. 2. The image processing apparatus according to claim 1, further comprising a drawing step in which an operator can draw an image interactively, wherein the designated search source image is an image drawn in the drawing step. Image search method. 3. The method according to claim 1, wherein the setting step sets weights corresponding to luminance emphasis and color difference emphasis.
Image search method described in. 4. The image search method according to claim 3, wherein said calculating step includes a converting step of converting the similarity into a value in a YCbCr color space. 5. The image search method according to claim 1, wherein the presenting step displays the extracted image in a reduced size. 6. The method according to claim 1, wherein the presenting step displays an icon image associated with the extracted image. 7. The method according to claim 5, further comprising, when one of the images displayed in the presenting step is selected, displaying a detailed image associated with the selected image. Image search method. 8. The image search method according to claim 5, wherein the presenting step displays the extracted images in order of similarity. 9. An image retrieval apparatus for retrieving a desired image from a storage unit storing a plurality of images, the image retrieval device comprising: Calculating means for calculating the similarity between the search source image and each of the plurality of images; extracting similar images from the plurality of images based on the image similarity calculated by the calculating means; And setting means for setting different weights for each parameter of the feature amount used in the calculating means. In the calculating means, the degree of similarity is calculated using the weight set by the setting means. An image search device for performing calculations. 10. The apparatus according to claim 9, further comprising drawing means for allowing an operator to draw an image interactively, wherein the specified search source image is an image drawn by the drawing means. Image search device. 11. The apparatus according to claim 9, wherein said setting means sets weighting corresponding to importance on luminance and importance on color difference. 12. The calculating means calculates the similarity by YCbCr.
12. The image search device according to claim 11, further comprising a conversion unit configured to convert the value into a value in a color space. 13. The image retrieval apparatus according to claim 9, wherein said presentation means reduces and displays the extracted image. 14. The image search device according to claim 9, wherein the presenting unit displays an icon image associated with the extracted image. 15. The display device according to claim 13, further comprising a display unit that displays a detailed image associated with the image when one of the images displayed on the presentation unit is selected. Image retrieval device. 16. The apparatus according to claim 13, wherein said presenting means displays the extracted images in order of similarity.
An image search device according to claim 1. 17. A storage medium for storing, in a computer, a control program for realizing an image search process for searching for a desired image from storage means storing a plurality of images, wherein the control program is designated by A code of a calculation step of calculating a similarity between the search source image and each of the plurality of images based on the feature amount of the search source image and the feature amount of the plurality of images, A similarity image is extracted from each of the plurality of images based on the obtained image similarity, and a code of a presentation step for presenting a search result; And a code of a step, wherein the calculation step realized by the code of the calculation step includes calculating the similarity using the weight set by the setting unit. Storage medium for the butterflies.