JP2000076268A - Device and method for retrieving image, and computer readable memory - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quickly reflect user's intention and to efficiently perform image retrieval by calculating the image feature value of an inputted image and calculating image similarity based thereon. SOLUTION: A user plots an illustration being a retrieval condition imitating desired image data being a retrieval object to be stored on a hard disk device 106 on a display part 103 with a pointing device 102a. The image feature value of the plotted illustration is calculated. Similar image retrieval is carried out based on the image feature value. A retrieved similar image is shown on the part 103. Thus, the plotting of an illustration by the user and retrieval and display processing can be performed parallelly. In other words, the processing speed of drawing an illustration is prevented from falling extremely and the user can continue the plotting of the illustration without feeling stress.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval apparatus and method for retrieving desired image data from an image database storing a plurality of image data, and a computer readable memory.

[0002]

2. Description of the Related Art Various image retrieval apparatuses have been devised for retrieving desired image data from a database storing a large number of image data. In these image search apparatuses, a method of associating non-image information such as a keyword and a photographing date with image data and performing a search based on the image data is used. )), And can be roughly divided into two methods.

In the latter, certain image data is presented,
A method of searching for image data using the image feature amount of the image data as a search key is particularly called similar image search. this is,
For users who do not have special knowledge about image processing,
There is an advantage that an intuitive search interface can be provided.

[0004]

However, when a handwritten illustration input by a user is used as a search key for a similar image search, and the desired image is not obtained as a result of the search, the illustration is corrected again. It was necessary to execute similar image search processing again. In this case, correction of the illustration cannot be started until the execution of the display of the immediately preceding search result is completed, and it has been difficult to efficiently perform the search operation.

The present invention has been made in view of the above problems, and provides an image retrieval apparatus, a method thereof, and a computer-readable memory capable of promptly reflecting the user's intention and performing an efficient image retrieval. The purpose is to do.

[0006]

The image retrieval apparatus according to the present invention for achieving the above object has the following arrangement.
That is, an image search device that searches for desired image data from an image database storing a plurality of image data,
An input unit that inputs an image to be drawn by the user as a search condition; a first control unit that controls execution of an image feature amount calculation unit that calculates an image feature amount of the image input by the input unit; A storage unit for storing the image data and the image feature amounts of the respective image data in association with each other; an image feature amount calculated by the image feature amount calculation unit; and an image feature of the image data stored in the storage unit. Image similarity calculating means for calculating image similarity based on the amount; and image display means for displaying a list of image data to be a search result based on the image similarity calculated by the image similarity calculating means. Control means for controlling the execution of the first control means and the second control means in parallel with each other to control communication between the first control means and the second control means. 3 control And a stage.

Preferably, the third control means includes:
Further, display control of the input unit and the image display unit is performed on the same screen of a display unit.

Preferably, the image feature quantity calculating means calculates the image feature quantity of the drawn image each time the image is changed by the input means.

[0009] To achieve the above object, an image retrieval apparatus according to the present invention has the following arrangement. That is, a first control means for controlling execution of an image search device for searching for desired image data from an image database storing a plurality of image data, the input means for inputting an image to be drawn by a user as a search condition; An image feature amount calculation unit that calculates an image feature amount of an image input by the input unit; a storage unit that stores the plurality of image data and the image feature amount of each image data in association with each other; An image similarity calculating unit that calculates an image similarity based on the image feature amount calculated by the image feature amount calculating unit and the image feature amount of the image data stored in the storage unit; and the image similarity calculating unit. A second control unit that controls execution of an image display unit that displays a list of image data to be a search result based on the image similarity calculated in the first and second control units; Each control of the second controller in parallel processing, and a third control means for controlling communication with between the first control means and second control means.

Preferably, the third control means includes:
Further, display control of the input unit and the image display unit is performed on the same screen of a display unit.

Preferably, the image feature quantity calculating means calculates the image feature quantity of the drawn image every time the image is changed by the input means.

An image retrieval apparatus according to the present invention for achieving the above object has the following arrangement. That is, a first control means for controlling execution of an image search device for searching for desired image data from an image database storing a plurality of image data, the input means for inputting an image to be drawn by a user as a search condition; A second control unit for controlling execution of an image feature amount calculating unit for calculating an image feature amount of an image input by the input unit; and a plurality of image data, and an image feature amount of each of the plurality of image data. Storage means for associating and storing; image similarity calculated based on the image feature quantity calculated by the image feature quantity calculation means; and image similarity degree based on the image feature quantity of the image data stored in the storage means. A third control for controlling execution of the calculating means and an image display means for displaying a list of image data as search results based on the image similarity calculated by the image similarity calculating means; And the control of the first control means, the second control means, and the third control means is performed in parallel, and the control between the first control means, the second control means, and the third control means is performed. And fourth control means for controlling communication.

[0013] Preferably, the fourth control means includes:
Further, display control of the input unit and the image display unit is performed on the same screen of a display unit.

Preferably, the image feature quantity calculating means calculates the image feature quantity of the drawn image every time the image is changed by the input means.

An image search method according to the present invention for achieving the above object has the following arrangement. That is, an image retrieval method for retrieving desired image data from an image database in which a plurality of image data are stored, wherein a first process and a second process are respectively processed in parallel, and a process between the first process and the second process is performed. Control step of controlling the communication of the first step, an input step of inputting an image to be drawn by a user as a search condition, and an image characteristic amount calculating step of calculating an image characteristic amount of the image input in the input step Wherein the second process comprises: a storage step of storing each of the plurality of image data and an image feature amount of each of the image data in a storage medium in association with each other; and an image calculated in the image feature amount calculation step. An image similarity calculating step of calculating an image similarity based on the characteristic amount and the image characteristic amount of the image data stored in the storage medium in the storing step; Based on the calculated image similarity calculation process, and an image display step of displaying a list of image data to the search result.

An image search method according to the present invention for achieving the above object has the following arrangement. That is, an image retrieval method for retrieving desired image data from an image database in which a plurality of image data are stored, wherein a first process and a second process are respectively processed in parallel, and a process between the first process and the second process is performed. The first process includes an input step of inputting an image to be drawn by the user as a search condition, and the second process includes:
An image feature amount calculating step of calculating an image feature amount of the image input in the input step; a storage step of storing the plurality of image data in a storage medium in association with the image feature amount of each image data; An image similarity calculation step of calculating an image similarity based on the image feature quantity calculated in the image feature quantity calculation step and the image feature quantity of the image data stored in the storage medium in the storage step; An image display step of displaying a list of image data as search results based on the image similarity calculated in the image similarity calculation step.

An image search method according to the present invention for achieving the above object has the following arrangement. That is, an image retrieval method for retrieving desired image data from an image database in which a plurality of image data are stored, wherein a first process, a second process, and a third process are each processed in parallel, and the first process and the second process are performed. A control step of controlling communication between two processes and the third process; and the first process includes:
An input step of inputting an image to be drawn by the user as a search condition; the second process includes an image characteristic amount calculating step of calculating an image characteristic amount of the image input in the input step; A storage step of storing each of the plurality of image data and an image feature amount of each image data in a storage medium in association with each other; an image feature amount calculated in the image feature amount calculation step; An image similarity calculating step of calculating an image similarity based on the image feature amount of the image data stored in the storage medium, and a search result based on the image similarity calculated in the image similarity calculating step. And displaying an image data list.

A computer readable memory according to the present invention for achieving the above object has the following configuration. That is,
A computer-readable memory storing an image search program code for searching for desired image data from an image database storing a plurality of image data, wherein the first process and the second process are each performed in parallel. A program code of a control step for controlling communication between the first process and the second process; a program code of an input step of inputting an image to be drawn by a user as a search condition; An image feature amount calculating step of calculating an image feature amount of an image, wherein the second process associates each of the plurality of image data with an image feature amount of each of the image data and stores the image data in a storage medium in association with each other. A program code of a storage step to be executed, an image feature amount calculated in the image feature amount calculation step, On the basis of the image feature amount of the image data stored in the storage medium, based on the program code of the image similarity calculation step of calculating the image similarity, based on the image similarity calculated in the image similarity calculation step, And a program code for an image display step of displaying a list of image data as search results.

A computer readable memory according to the present invention for achieving the above object has the following configuration. That is,
A computer-readable memory storing an image search program code for searching for desired image data from an image database storing a plurality of image data, wherein the first process and the second process are each performed in parallel. And a program code of a control step for controlling communication between the second process and the first process, and the first process includes a program code of an input step of inputting an image to be drawn by a user as a search condition. And storing the program code of the image feature amount calculating step of calculating the image feature amount of the image input in the input step, the plurality of image data, and the image feature amount of each image data in a storage medium in association with each other. A program code of a storage step to be executed, an image feature amount calculated in the image feature amount calculation step, On the basis of the image feature amount of the image data stored in the storage medium, based on the program code of the image similarity calculation step of calculating the image similarity, based on the image similarity calculated in the image similarity calculation step, And a program code for an image display step of displaying a list of image data as search results.

A computer readable memory according to the present invention for achieving the above object has the following configuration. That is,
A computer-readable memory storing an image search program code for searching for desired image data from an image database storing a plurality of image data, wherein the first process, the second process, and the third process are each processed in parallel. A program code for a control step for controlling communication between the first process, the second process, and the third process; and a program code for an input step for inputting an image to be drawn by a user as a search condition. Wherein the second process comprises a program code of an image feature amount calculating step of calculating an image feature amount of the image input in the input step, and the third process comprises: A program code of a storage step of storing the image data of each image data in a storage medium in association with each other, A program code for an image similarity calculation step for calculating an image similarity based on the image feature amount calculated in the image feature amount calculation step and the image feature amount of the image data stored in the storage medium in the storage step; And a program code for an image display step of displaying a list of image data as search results based on the image similarity calculated in the image similarity calculation step.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an image retrieval apparatus according to an embodiment of the present invention.

In FIG. 1, reference numeral 101 denotes a CPU, which controls the entire system. Reference numeral 102 denotes a keyboard, and reference numeral 102a denotes a pointing device, which is used together with a mouse to draw an illustration serving as a search condition for inputting data into the system or performing a similar image search. A display unit 103 includes a CRT, a liquid crystal, and the like, and displays a user interface for drawing an image serving as a search condition, image data serving as a search result, and the like. A ROM 104 and a RAM 105 constitute a storage device of the system and store programs executed by the system and data used by the system. Reference numeral 106 denotes a hard disk drive, and 107 denotes a floppy disk drive, which constitutes an external storage device used for a file system of the system. Also, the hard disk device 1
In the area 06, a plurality of image data to be searched are stored. Reference numeral 108 denotes a printer, which records an image or the like displayed on the display unit 103 on a recording medium.

Next, an operation screen displayed on the display unit 103 when the search processing executed in the embodiment is executed will be described with reference to FIG.

FIG. 2 is a diagram showing an operation screen displayed on the display unit when executing the search processing according to the embodiment of the present invention.

21 is a user drawing area, 22 is a cursor,
23 is a color designation scroll bar, 24 is a clear button, 2
5 is a search execution button, 27 is a next candidate display button, and 28 is a processing end button. 26a to 26h are areas for displaying a group of icon images corresponding to the image data as a search result.

Using the drawing tool realized by software, a user can draw an illustration in the user drawing area 21 as a search condition that resembles the image to be searched. The outline of the software operation at the time of drawing an illustration is as follows.

The color designation scroll bar 23 is used to designate a pen color used for drawing, and designates R, G, and B values in order from the top. When the clear button 24 is pressed, the entire user drawing area 21 is painted white. The user can draw a free curve on the user drawing area 21 by moving the cursor 22 using the pointing device 102a. When the processing end button 28 is pressed, the operation screen is closed and the processing ends.

Next, an outline of the processing executed by the image search apparatus of the present embodiment will be described with reference to FIG.

FIG. 3 is a flowchart showing the outline of the processing executed by the image search apparatus according to the embodiment of the present invention.

Process A and process B are executed in parallel while communicating with each other. This is realized by a multi-process OS represented by UNIX. It may be realized by a finer control unit such as a known thread, or may be realized by hardware.

First, the processing executed in the process A will be described with reference to FIG.

At step S171, the user operates the display unit 10
An illustration is drawn on 3 as a search condition that resembles desired image data to be searched and stored on the hard disk device 106. In step S172, the image feature amount of the drawn illustration is calculated. In step S173,
Inter-process communication is performed, and the calculated image feature amount is passed to the process B. In step S174, it is determined whether to end the search operation. When continuing (N in step S174)
O), and return to step S171. On the other hand, if not continued (YES in step S174), the process ends.

Next, the processing executed in the process B will be described with reference to FIG.

In step S175, it is determined whether or not the inter-process communication (image feature amount) from the process A has been received. If not received (NO in step S175),
It returns to step S175. That is, a loop for monitoring inter-process communication is formed. On the other hand, if it has been received (YES in step S175), the flow advances to step S176.

In step S176, a similar image search is executed based on the received image feature amount. Step S177
Then, the searched similar image is displayed on the display unit 103. In step S178, it is determined whether to end the search operation. If not terminated (NO in step S178), the process returns to step S175. On the other hand, when ending (step S
(YES at 178), and terminates the process.

With the above processing, drawing of an illustration by a user and search / display processing can be performed in parallel.
In other words, even during the execution of the search / display processing, the processing speed for drawing the illustration does not extremely decrease, and the user can continue drawing the illustration without stress.

Hereinafter, details of the processing executed in each of the above steps will be described. [Explanation of Step S171] In step S171, an illustration as a search condition is drawn in the user drawing area 21 by a user operation, and the process proceeds to the next step S172 at an appropriate timing. The processing at this time will be described with reference to FIG. This timing is a timing at which the system monitors the movement of the mouse 102a at a predetermined timing. Every time the monitoring is performed, the image feature amount of the image drawn so far is calculated.

FIG. 4 shows step S17 of the embodiment of the present invention.
3 is a flowchart showing details of a process executed in Step 1.

Note that x0 and y0 are variables for storing the position of the cursor 22 immediately before, and x1 and y1 are variables for storing the current position of the cursor 22.

First, in step S31, it is determined whether the user has moved the mouse 102a. If not moving (NO in step S31), the process returns to step S31. That is, the processing here forms a loop for monitoring the movement of the mouse 102a. On the other hand, if it is moving (YES in step S31), the process proceeds to step S32.

In step S32, it is determined whether or not the mouse button of the mouse 102a has been pressed. If not pressed (NO in step S32), step S3
The process proceeds to step S4, where the current position (x1, y1) of the cursor 22 is substituted for (x0, y0), and the process returns to step S31. Thereby, the cursor 22 can be simply moved without performing drawing.

On the other hand, when the button is pressed (step S3
If YES in step 2), that is, if dragging is in progress, the process proceeds to step S33. Here, the position (x0, y0) of the previous cursor 22 and the current position (x1,
During y1), a line is drawn with a color determined by the color designation scroll bar 23.

In the following step S35, the current position (x1, y1) of the cursor 22 is substituted for (x0, y0), and the step S171 ends, and the flow advances to the step S172.

In this processing, from the user's point of view, the search is automatically executed each time a little drawing (stroke) is added to the illustration. [Description of Step S172] In step S172, the image feature amount of the illustration drawn in step S171 is calculated.

As shown in FIG. 5, the size of the user drawing area 21 in the present embodiment is W pixels in the horizontal direction and H in the vertical direction.
It is composed of pixels. This is divided into three parts in the horizontal direction and two parts in the vertical direction, for a total of six parts, and the regions (0,
0), region (1, 0),..., Region (2, 1). The average value of the R, G, and B values of each of these areas is calculated, and a total of 18 numerical values are used as the image feature amount of the drawn illustration.

Here, regarding the calculation processing of the image feature amount,
This will be described with reference to FIG.

FIG. 6 is a flowchart showing the calculation processing of the image feature amount according to the embodiment of the present invention.

First, in step S51, a variable k is initialized with a value 0. In step S52, a variable j is initialized with a value 0. In step S53, a variable i is initialized with a value of 0.
In step S54, the k-th element d (k) of the array d is
The average value of the R values in the area (i, j) is substituted. Also, d
The average value of the G value is substituted for (k + 1), and the average value of the B value is substituted for d (k + 2). The details of the method of calculating the average of the R, G, and B values will be described later with reference to the flowchart of FIG.

In step S55, the variable k is increased by the value 3. In step S56, the variable i is increased by the value 1. In step S57, the variable i is compared with the value 2. If the variable i is larger than 2 (YES in step S57), the process proceeds to step S58. On the other hand, if it is 2 or less (NO in step S57), the process returns to step S54.

In step S58, the variable j is increased by the value 1. In step S59, the variable j is compared with the value 1.
If it is larger than 1 (YES in step S59), the process ends. On the other hand, if it is 1 or less (NO in step S59), the process returns to step S53.

When the above processing is completed, the image feature amount of the drawn illustration is stored in the array d () having 18 elements. Here, in order to calculate the image feature amount, the drawn illustration (image) is divided into six rectangular areas having the same area. However, the division is not limited to the rectangular area but may be performed in a more complicated area. Alternatively, the number of divisions may be increased or decreased.
When the number of divisions is increased or decreased, the number of elements of the image feature amount is not 18 but increases or decreases accordingly.

Next, in step S54 of FIG.
Details of the method of calculating the average value of the G and B values will be described with reference to FIG.

FIG. 7 is a flowchart showing the details of the method for calculating the average of the R, G, and B values according to the embodiment of the present invention.

The image data of the drawn illustration is
It is assumed that they are stored in three arrays of R (X, Y), G (X, Y), and B (X, Y). Where 0 ≦ X <W, 0
.Ltoreq.Y <H, and the upper left corner of the image is defined as a starting point (0, 0). In the following flow, X0 ≦ X <X1, Y0 ≦ Y
<Calculate the average of the R, G, and B values in the area of Y1, and calculate the variable D
The average value of the R, G, and B values is returned to R, DG, and DB, respectively. Further, in step S1002, the area (i,
Since the area corresponding to j) corresponds to X0 = W * i / 3 X1 = W * (i + 1) / 3 Y0 = H * j / 2 Y1 = H * (j + 1) / 2, the constants X0, X1, The flowchart is executed after initializing Y0 and Y1 as described above.

First, in steps S61, the variables DR, D
G and DB are initialized with the value 0. In step S62, a variable Y is initialized to Y0. In step S63, the variable X is set to X0
Initialize with In step S64, R (X,
Y) is added. Similarly, G (X, Y) is added to the variable DG, and B (X, Y) is added to the variable DB.

In step S65, the variable X is incremented by one. In step S66, the variables X and X1 are compared.
If equal (YES in step S66), step S6
Go to 7. On the other hand, if they are not equal (N in step S66)
O), and return to step S64.

In step S67, the variable Y is increased by a value of one. In step S68, the variables Y and Y1 are compared. If they are equal (YES in step S68), step S69
Proceed to. On the other hand, if they are not equal (N in step S68)
O), and return to step S63. In step S69, the variables DR, DG, and DB are respectively set to (X1-X0) * (Y1-Y0)
Divide by. This is the number of pixels in the area. That is,
The variables DR, DG, and DB are average densities obtained by dividing the sum of pixel densities in the area by the number of pixels. [Description of Step S173] In step S173, communication between processes is performed, and the image feature amount d () calculated in step S172 of process A is transmitted to process B. Since this process is completed in a very short time, even when the user continues drawing the illustration, the user does not feel stress such as slowing down the process. [Explanation of Step S174] In step S174, it is determined whether or not the processing end button 28 has been pressed, and if it has been pressed, the processing ends. On the other hand, if the button is not pressed, the process returns to step S171, and the user can continue drawing the illustration. [Explanation of Step S175] In step S175, a loop for monitoring whether or not there is communication from the process A is formed. Upon receiving the image feature value d () from the process A, the process proceeds to step S176. "Description of Step S176" In step S176, a similar image search is performed based on the received image feature amount.

It is assumed that N pieces of image data are accumulated in the hard disk device 106, and each image feature amount is calculated and stored in advance by the above-described processing.
The image data may be stored in a standard file format such as a well-known JPEG, FlashPix, etc.
It may be stored in a BMS (relational database management system) original file format.
The image feature amount is a two-dimensional array D having a size of N * 18.
(N, i) (where 0 ≦ n <N, 0 ≦ i <18).

At this time, the drawn illustration (image data) and the n-th
The distance S (n) between the image and the second image data is defined by the following equation.

[0061]

(Equation 1)

It is determined that the smaller the inter-image distance S (n), the higher the similarity of the image data.

First, the image distance S (n) between the image data of all N images stored in the hard disk device 106 and the drawn illustration (image data) (where 0 ≦
n <N). Next, by selecting M pieces (0 <M <N) in ascending order of the image-to-image distance S (n),
A similar image search is performed. The calculation process of the first-half inter-image distance S (n) and the similar image search process of selecting the second half M image data will be described with reference to FIGS. 8 and 9, respectively.

FIG. 8 shows an image distance S according to the embodiment of the present invention.
It is a flowchart which shows the calculation process of (n).

First, in step S71, a variable n is initialized with a value of 0. In step S72, variables i and S (n) are initialized with a value of 0. In step S73, D (n, i) and d
The square of the difference of (i) is added to S (n). Step S
At 74, the variable i is increased by one.

In step S75, the variable i is compared with the value 18. If they are equal (YES in step S75), the flow advances to step S76. On the other hand, if they are not equal (step S7
(NO at 5), and returns to step S73.

In step S76, the variable n is incremented by one. In step S77, the variables n and N are compared. If they are equal (YES in step S77), the process ends. On the other hand, if they are not equal (NO in step S77),
It returns to step S72.

When the above processing is completed, the array S (n) becomes:
The image distance S (n) between the drawn illustration (image data) and all the image data stored in the hard disk device 106 is stored. Subsequently, the image distance S (n)
The details of the similar image search process of selecting M images in order from the smallest image and storing the numbers corresponding to the selected images in the array T () will be described with reference to FIG. 9.

FIG. 9 is a flowchart showing details of the similar image search processing according to the embodiment of the present invention.

First, in step S81, a variable j is initialized with a value of 0. In step S82, a variable i is initialized with a value of 0. In step S83, the variable min is initialized to a value of 0, and L is initialized to a sufficiently large value. In step S84, S (i)
And L. If S (i) is less than L (YES in step S84), the process proceeds to step S85. On the other hand, L
If this is the case (NO in step S84), step S
Proceed to 86.

In step S85, the value i is substituted for the variable min, and S (i) is substituted for L. In step S86, the variable i is increased by the value 1. In step S87, i and N are compared. If they are equal (YES in step S87), the flow advances to step S88. On the other hand, if not equal (NO in step S87), the process returns to step S84.

In step S88, the value min is substituted for T (j). In step S89, a sufficiently large value is substituted for S (min). In step S810, the variable j is increased by the value 1. In step S811, the variable j and M are compared. If they are equal (YES in step S811), the process ends. On the other hand, if they are not equal (NO in step S811), the process returns to step S82.

When the above processing is completed, the images are arranged in the array T (j) (where 0 ≦ j <M) in the order of the image data stored in the hard disk device 106 having a high similarity to the drawn illustration (image data). The number is stored. “Description of Step S177” According to the operation screen of FIG.
The processing will be described.

In the areas 26a to 26h, icon images obtained by reducing and displaying similar images searched by the above processing are displayed. In the area 26a, the highest similarity T
The image corresponding to (0), the image corresponding to T (1),... Are displayed in the area 26b, and the image having the lowest similarity is displayed in the area 26h.

The reduced display is performed on the hard disk drive 10.
6 may be decoded and reduced and displayed on the screen. Further, when low-resolution icon data for an icon is used, such as FlashPix, which is a standard image format, the icon data may be used for display.

Each icon image can be "selected" using the pointing device 102a. When the next candidate display button 27 is pressed, the next candidate, that is,
The icon images of the image data corresponding to T (8) to T (15) are displayed in the areas 26a to 26h. This is T
It can be repeated until (M-1) is reached. [Explanation of Step S178] In step S178, it is determined whether or not the processing end button 28 has been pressed, and if it has been pressed, the processing ends. On the other hand, if the button is not pressed, the process returns to step S175, and waits for inter-process communication from process A.

As described above, according to the present embodiment, in parallel with the similar image retrieval process currently being executed,
Since the user can newly draw an illustration as a search condition, similar image search processing can be performed efficiently.

In the above embodiment, the drawing of the illustration and the calculation of the image feature amount are executed by the same process, but other methods may be used. For example, in the process A, an illustration may be drawn, and in the process B, calculation of an image feature amount, similar image search, and similar image list display may be performed.

Although an example of the system configuration in which the number of CPUs 101 is one has been described, in the case of an application in which a plurality of processes are executed in parallel as in the present invention,
By using a so-called multiprocessor system having a plurality of CPUs, the response can be more effectively improved.

The mouse is used as the pointing device 102a. However, the present invention is not limited to this. For example, if a pen tablet that allows input with a pen is used, the operator can more efficiently draw an illustration. Further, if a touch screen integrated with the display unit 103 so that a user can draw an illustration directly on the screen while watching the screen display is used, more intuitive drawing can be performed. In addition, any pointing device that can input information to the computer system may be used.

In step S177, the reduced images corresponding to the image data as the processing result of the similar image search processing are arranged two-dimensionally and displayed. However, the reduced images may be arranged horizontally (one-dimensionally). Alternatively, a three-dimensional display may be performed in consideration of information in the depth direction. For example,
Higher similarities are displayed in front (larger), and lower similarities are displayed farther (smaller),
A more intuitive list display can be performed.

Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), a device including one device (for example, a copying machine, a facsimile machine) Etc.).

An object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

Examples of a storage medium for supplying the program code include a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, and CD.
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0088]

As described above, according to the present invention,
It is possible to provide an image search device, a method thereof, and a computer-readable memory capable of promptly reflecting a user's intention and performing an efficient image search.

[0089]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image search device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an operation screen displayed on a display unit when a search process according to the embodiment of the present invention is performed.

FIG. 3 is a flowchart illustrating an outline of a process executed by the image search device according to the embodiment of the present invention.

FIG. 4 is a flowchart showing details of processing executed in step S171 according to the embodiment of the present invention.

FIG. 5 is a diagram for explaining division of an image for extracting an image feature amount according to the embodiment of the present invention.

FIG. 6 is a flowchart illustrating calculation processing of an image feature amount according to the embodiment of this invention.

FIG. 7 is a flowchart illustrating details of a method for calculating average values of R, G, and B values according to the embodiment of the present invention.

FIG. 8 is a flowchart illustrating a calculation process of an inter-image distance S (n) according to the embodiment of the present invention.

FIG. 9 is a flowchart illustrating details of a similar image search process according to the embodiment of this invention.

[Explanation of symbols]

 101 CPU 102 Keyboard 102a Pointing device 103 Display unit 104 ROM 105 RAM 106 Hard disk device 107 Floppy disk device 108 Printer

Continued on the front page (72) Inventor Sumi Kusama 3-30-2 Shimomaruko, Ota-ku, Tokyo F-term in Canon Inc. (reference) 5B050 BA10 CA02 CA07 DA09 DA10 EA09 EA12 EA18 FA02 FA03 FA08 FA12 GA08 5B075 ND08 NK10 PP03 PP12 PP30 PQ02 PQ22 PQ46 PQ48 PQ63 PQ64 PQ74 PR06 QM08

Claims (21)

[Claims] An image retrieval apparatus for retrieving desired image data from an image database storing a plurality of image data, comprising: an input unit for inputting an image to be drawn by a user as a search condition; First control means for controlling execution of an image feature quantity calculating means for calculating an image feature quantity of an image obtained, and storage means for storing each of the plurality of image data and the image feature quantity of each image data in association with each other Image similarity calculating means for calculating an image similarity based on the image feature amount calculated by the image feature amount calculating means and the image feature amount of the image data stored in the storage means; Second control means for controlling execution of an image display means for displaying a list of image data as search results based on the image similarity calculated by the degree calculation means; and the first control means. An image search apparatus comprising: a third control unit configured to perform parallel processing of a stage and a control of the second control unit and control communication between the first control unit and the second control unit. 2. The apparatus according to claim 1, wherein the third control unit further controls the display of the input unit and the image display unit on the same screen of a display unit. 3. The image retrieval apparatus according to claim 1, wherein the image feature amount calculation unit calculates an image feature amount of the drawn image every time the image is changed by the input unit. apparatus. 4. An image retrieval apparatus for retrieving desired image data from an image database in which a plurality of image data are stored, wherein the first means controls execution of input means for inputting an image to be drawn by a user as a retrieval condition. Control means; image feature quantity calculation means for calculating an image feature quantity of the image input by the input means; storage means for storing the plurality of image data and the image feature quantity of each image data in association with each other Image similarity calculating means for calculating an image similarity based on the image feature amount calculated by the image feature amount calculating means and the image feature amount of the image data stored in the storage means; Second control means for controlling execution of an image display means for displaying a list of image data as search results based on the image similarity calculated by the degree calculation means; and the first control means. An image search apparatus comprising: a third control unit configured to perform parallel processing of a stage and a control of the second control unit and control communication between the first control unit and the second control unit. 5. The image search device according to claim 4, wherein the third control unit further controls the display of the input unit and the image display unit on the same screen of a display unit. 6. The image retrieval apparatus according to claim 4, wherein said image feature value calculation means calculates an image feature value of the drawn image every time an image is changed by said input means. apparatus. 7. An image retrieval apparatus for retrieving desired image data from an image database in which a plurality of image data are stored, wherein the first means controls execution of input means for inputting an image to be drawn by a user as a retrieval condition. Control means; second control means for controlling execution of an image feature quantity calculation means for calculating an image feature quantity of the image input by the input means; each of the plurality of image data; and an image of each image data A storage unit that stores the feature amounts in association with each other, an image feature amount calculated by the image feature amount calculation unit, and an image similarity calculated based on the image feature amount of the image data stored in the storage unit. An image similarity calculating means for controlling execution of an image display means for displaying a list of image data as search results based on the image similarity calculated by the image similarity calculating means; 3 control means, the respective controls of the first control means, the second control means, and the third control means are processed in parallel, and the control of the first control means, the second control means, and the third control means is performed in parallel. And a fourth control unit for controlling communication with the image retrieval apparatus. 8. The image retrieval apparatus according to claim 7, wherein said fourth control means further controls display of said input means and said image display means on the same screen of a display unit. 9. The image retrieval apparatus according to claim 7, wherein said image feature value calculation means calculates an image feature value of a drawn image each time an image is changed by said input means. apparatus. 10. An image retrieval method for retrieving desired image data from an image database storing a plurality of image data, wherein a first process and a second process are respectively processed in parallel, and the first process and the second process are performed. A control step of controlling communication between processes; an input step of inputting an image to be drawn by a user as a search condition; and an image characteristic of calculating an image characteristic amount of the image input in the input step. An amount calculating step, wherein the second process calculates each of the plurality of image data and an image feature amount of each image data in a storage medium in association with each other, and the image feature amount calculating step The image feature amount calculated, based on the image feature amount of the image data stored in the storage medium in the storage step, an image similarity calculation step of calculating image similarity, Serial based on the calculated image similarity image similarity calculation process, the image retrieval method characterized by and an image display step of displaying a list of image data to the search result. 11. The third control step further includes controlling display of an input screen displayed in the input step and an image display screen displayed in the image display step on the same screen of a display unit. The image search method according to claim 10, wherein: 12. The image characteristic amount calculating step of calculating an image characteristic amount of a drawn image every time an image is changed by the inputting step.
Image search method described in. 13. An image retrieval method for retrieving desired image data from an image database storing a plurality of image data, wherein a first process and a second process are each processed in parallel, and the first process and the second process are performed. A control step of controlling communication between processes; and the first process includes an input step of inputting an image to be drawn by a user as a search condition. The second process includes a step of inputting an image input in the input step. An image feature amount calculation step of calculating an image feature amount; a storage step of storing each of the plurality of image data in association with an image feature amount of each of the image data in a storage medium; and calculating in the image feature amount calculation step The image feature amount calculated, based on the image feature amount of the image data stored in the storage medium in the storage step, an image similarity calculation step of calculating image similarity, Serial based on the calculated image similarity image similarity calculation process, the image retrieval method characterized by and an image display step of displaying a list of image data to the search result. 14. The method of claim 3, wherein the third control step further controls display of an input screen displayed in the input step and an image display screen displayed in the image display step on the same screen of a display unit. The image search method according to claim 13, wherein: 15. The image feature amount calculating step of calculating an image feature amount of a drawn image each time an image is changed by the inputting step.
Image search method described in. 16. An image retrieval method for retrieving desired image data from an image database storing a plurality of image data, wherein the first process, the second process, and the third process are each performed in parallel, and the first process is performed. And a control step of controlling communication between the second process and the third process; and the first process includes an input step of inputting an image to be drawn by a user as a search condition. An image feature amount calculating step of calculating an image feature amount of the image input in the input step, wherein the third process associates each of the plurality of image data with an image feature amount of each image data. A storage step of storing the image data in the storage medium; an image feature quantity calculated in the image feature quantity calculation step; and an image feature of the image data stored in the storage medium in the storage step. An image similarity calculating step of calculating an image similarity based on the amount; and an image displaying step of displaying a list of image data to be a search result based on the image similarity calculated in the image similarity calculating step. An image search method comprising: 17. The display device according to claim 17, wherein the control step further controls display of an input screen displayed in the input step and an image display screen displayed in the image display step on the same screen of a display unit. Item 18. The image search method according to Item 16. 18. The image characteristic amount calculating step of calculating an image characteristic amount of a drawn image every time an image is changed by the inputting step.
Image search method described in. 19. A computer-readable memory storing an image search program code for searching for desired image data from an image database in which a plurality of image data are stored, wherein the computer-readable memory stores a first process and a second process in parallel. A program code of a control step of controlling communication between the first process and the second process; a program code of an input step of inputting an image to be drawn by a user as a search condition; An image feature amount calculating step of calculating an image feature amount of the image input in the step, wherein the second process associates each of the plurality of image data with an image feature amount of each image data. Program code for a storage step of storing in a storage medium, and an image calculated in the image feature amount calculation step A program code for an image similarity calculation step of calculating an image similarity based on the collection amount and the image feature amount of the image data stored in the storage medium in the storage step; And a program code for an image display step of displaying a list of image data as search results based on the image similarity. 20. A computer-readable memory storing an image retrieval program code for retrieving desired image data from an image database storing a plurality of image data, wherein each of the first process and the second process is processed in parallel. A program code of a control step of controlling communication between the first process and the second process; and a program code of an input step of inputting an image to be drawn by a user as a search condition, The second process associates a program code of an image feature amount calculation step of calculating an image feature amount of the image input in the input step, each of the plurality of image data, and an image feature amount of each image data. Program code for a storage step of storing in a storage medium, and an image calculated in the image feature amount calculation step A program code for an image similarity calculation step of calculating an image similarity based on the collection amount and the image feature amount of the image data stored in the storage medium in the storage step; And a program code for an image display step of displaying a list of image data as search results based on the image similarity. 21. A computer-readable memory storing an image search program code for searching for desired image data from an image database storing a plurality of image data, wherein each of the first process, the second process, and the third process is provided. And a program code of a control step of controlling communication between the first process, the second process and the third process, and the first process inputs an image to be drawn by a user as a search condition. A program code for an input step, wherein the second process includes a program code for an image feature amount calculating step of calculating an image feature amount of the image input in the input step, and the third process includes: In the storage step of storing the image data in the storage medium in association with each image data and the image feature amount of each image data. An image similarity calculating step of calculating an image similarity based on a program code, an image feature amount calculated in the image feature amount calculating step, and an image feature amount of image data stored in a storage medium in the storing step. And a program code for an image display step of displaying a list of image data as search results based on the image similarity calculated in the image similarity calculation step. .