JPH11212993A - Picture retrieval device/method and record medium recording picture retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize high similarity in a retrieval result and to improve retrieval precision. SOLUTION: In a picture retrieval device retrieving a picture similar to a key picture based on similarity between a designated key picture and a picture being the object of retrieval, similarity is calculated on a color which a man easily notices, a vivid color, a bright color and the color of an object contained in the picture, for example, (S451, S452 and S454). The similarity of the colors is weighted and final similarity is calculated (S455).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieval apparatus and method for retrieving an image similar to a designated image from images to be managed in an image database for managing image data, and a recording medium storing an image retrieval program.

[0002]

2. Description of the Related Art In an image database for managing image data, image data includes keywords, image shapes, and the like.
It is registered together with search conditions such as the feature amount of an image such as color. The image database searches for images from registered image data based on search conditions specified by the user.

When searching for an image based on image features such as the shape, texture pattern, and color of the image, the above features are obtained from an image designated by the user for search (hereinafter, referred to as a "key image"). The amount is extracted, the extracted feature amount is compared with the feature amount of the image registered in the image database, a similarity indicating the degree of similarity between the two images is calculated, and similarity is calculated based on the similarity. The registered image determined to be present is output as a search result.

For example, in an image database for searching for an image similar in color to a key image, a search is performed using the color of the image as a feature amount. An image is searched using the average color of many colors contained in the image as a feature amount.

[0005]

However, in this case, since the search is not always performed using the color of the user's attention, even if the searched images are similar in color as a whole, the user cannot search for the color. Judging from the sensation, it may not always be said that they are similar. Generally, humans have bright colors,
In many cases, an image is viewed by paying attention to a bright color or a color of an object included in the image. Therefore, it is considered that such a problem can be solved by retrieving an image by giving a particular weight to these colors, and an image closer to the image desired by the user can be retrieved.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide an image search apparatus and method and an image search program which realize high similarity in search results and improve search accuracy. An object of the present invention is to provide a recorded recording medium.

[0007]

According to the present invention, there is provided an image retrieval apparatus, comprising: an image storage unit for storing an image; an image input unit for inputting a designated image; and a color feature from each of the designated image and the stored image. Color feature acquisition means for acquiring
A similarity calculating unit configured to calculate a similarity indicating a degree of similarity between the two images from the color characteristics of the designated image and the color features of the stored image; and an image similar to the designated image based on the similarity. And searching means for searching from the stored images, wherein the similarity is calculated by weighting colors in a predetermined area of a color space. Here, as a color in a predetermined area of the color space, a color that is noticed by a human, for example, a bright color or a bright color is selected.

[0008] Further, when viewing an image, a human can easily pay attention to an object in the image. For this reason, in the image search device,
An object color extracting means for detecting a region of the object included in each of the designated image and the stored image and extracting a color of the detected object may be further provided. At this time, the similarity is calculated by further weighting the color of the object.

An image search method according to the present invention is an image search method for searching an image storing an image similar to a designated image, wherein an image input step of inputting a designated image, and a color feature is extracted from the designated image. A color feature extraction step of acquiring a color feature of the stored image; and a similarity indicating a similarity between the two images is determined from a color feature of the designated image and a color feature of the stored image in a predetermined area of a color space. And a search step of searching the stored image for an image similar to the designated image based on the similarity.

In the image search method, a step of detecting an object region included in each of the designated image and the stored image and extracting a color of the object region may be provided before the similarity calculation step. Good. At this time, in the similarity calculating step, the similarity is calculated by further weighting the color of the object image.

A recording medium according to the present invention is a recording medium on which a computer-readable program is recorded.
The following programs are stored. That is, the recording medium is an image search program for searching an image storing an image similar to the designated image, the image input step of inputting the designated image, extracting a color feature from the designated image, and storing the image. A color feature extraction step of acquiring a color feature of an image, and a color feature of the designated image and a color feature of the stored image, the similarity indicating the similarity between the two images is determined for a color of a predetermined area of a color space. And a search step of searching the stored image for an image similar to the designated image based on the similarity.

Further, in the recording medium, the program has a step of detecting an object region included in each of the designated image and the stored image and extracting a color of the object region before the similarity calculation step. In the similarity calculation step, the similarity may be calculated by further weighting the color of the object image.

Further, in the image search apparatus, the image search method, and the image search program, the color space is an HLS space, and the predetermined area is an area where lightness is higher than a predetermined value or saturation is higher than a predetermined value. Region.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image retrieval apparatus according to the present invention will be described below with reference to the accompanying drawings. The image search device of the present embodiment is an image search device that searches for an image similar to the specified image based on the color characteristics of the image. The similarity (similarity) between images is calculated by weighting colors, bright colors, colors of object images included in the images, and the like, and similar images are searched.

<Overall Configuration of Image Retrieval Apparatus> FIG. 1 is a schematic configuration diagram of an image retrieval apparatus (hereinafter, referred to as a "system") of the present embodiment. As shown in FIG. 1, the system includes a central processing unit (hereinafter, referred to as a “CPU”), and is mainly configured by a control device 1 that controls the entire system.
For the CPU, for example, Pentium manufactured by Intel Corporation is used. The control device 1 includes a display 2 for displaying images, characters, etc., a display for operations, a keyboard 3 and a mouse 4 for performing various inputs and instruction operations, and a data storage medium. Write data /
A floppy disk device 5b and a hard disk device 6 for reading, a printer 7 for printing characters and image data, and a scanner 8 for capturing image data
And a CD-ROM device 9b for reading data stored in the CD-ROM 9a, a speaker 10 for audio output, and a microphone 11 for audio input.

FIG. 2 shows a block diagram of the present system. C
The PU 201 stores, via the data bus 220, a ROM 203 in which a program for controlling the system is stored.
And a RAM 204 for temporarily storing programs and data executed by the CPU 201 for control. A circuit connected to the CPU 201 via the data bus 220 includes a display control circuit 205 for controlling the display 2 for displaying images, characters, and the like, and a keyboard control circuit 2 for controlling transfer of input from the keyboard 3.
06, a mouse control circuit 207 for controlling transfer of an input from the mouse 4, a floppy disk device control circuit 208 for controlling the floppy disk device 5b, and a hard disk device control circuit 20 for controlling the hard disk device 6.
9, a printer control circuit 210 for controlling output to the printer 7, and a scanner control circuit 21 for controlling the scanner 8.
1, a CD-ROM device control circuit 212 for controlling the CD-ROM device 9b, a speaker control circuit 213 for controlling the speaker 10, and a microphone control circuit 214 for controlling the microphone 11. Further, the CPU 201
Is connected to a clock 202 for generating a reference clock required for operating the system.
An expansion slot 215 for connecting various expansion boards is connected via a data bus 220. A SCSI board is connected to the expansion slot 215 and the SCSI
Floppy disk device 5b, hard disk device 6, scanner 8, or CD-ROM device 9 via I board
b or the like may be connected.

In the above system, a floppy disk 5a and a hard disk device 6 are used as image data storage media, but other information storage media such as a magneto-optical disk (MO) may be used. Although the scanner 8 is used as the image data input device, other data input devices such as a still video camera and a digital camera may be used. Further, although the printer 7 is used as an output device, another output device such as a digital copying machine may be used.
In the present system, a program for implementing the control processing is stored in the ROM 203. However, a part or all of the program is stored in an information storage medium such as a floppy disk 5a, a hard disk device 6, or a CD-ROM 9a, and the program and data are read out from the information storage medium to the RAM 204 as necessary, and are read out. It may be executed.

<Image Database and Color Space Area Table> This system stores and manages image data.
The image database includes an image database including image data and additional information serving as a search key for the image data, and a color space area table referred to when extracting color features from the image. The image database and the color space area table are stored in the hard disk device 6
Etc. are logically configured on an information storage medium. FIG. 3 shows an example of the configuration of the image database 50 of the present system. The image database 50 manages “image data” as image information and various search information for the image data. The search information for the image data includes a “keyword” and various feature amounts indicating the features of the image data. The feature amount includes information such as a color feature indicating a feature related to a color of the image, an image shape, a texture pattern, and the like.
In the image search device of the present embodiment, in particular, the color features of the image are used for the search information.

The color space area table 55 has information for dividing a color space into predetermined areas. FIG. 4 shows an example of the color space area table 55. In the present embodiment, an HLS color space as shown in FIG. 5 is used as the color space. In this color space, colors are expressed as hue H (0 ° ≦ H ≦ 360 °), lightness L (0.00 ≦ L ≦ 1.00), and saturation S (0.00 ≦ S
≤ 1.00). In the color space area table 55, the color space is divided into 21 color areas as shown in FIG. Further, in the present embodiment, a color having a saturation of 0.6 or more is defined as a “bright color”, a color having a brightness of 0.5 or more is defined as a “bright color”, and a flag is set for each corresponding color region. Standing up. The “bright colors” and “bright colors” defined in such color regions are selected as colors that are particularly noticeable to human senses. In addition, a color that is easily noticed may be specified by the hue. For example,
A color in a predetermined hue range such as red, yellow, and green may be selected as a color that is easily noticed.

<Control Operation of Image Retrieval Apparatus> A specific control operation of the present system will be described below with reference to a flowchart.

<Main Flow> FIG. 6 is a flowchart showing a main routine of a program executed by the CPU 201 in the present system. When the program is started, first, an initialization process for initializing flags and the like necessary in the subsequent processes and displaying an initial menu screen is performed (S1). On the initial menu screen, selection items for selecting a predetermined process are displayed as icons, and when one of the selection items is selected by the user, the predetermined process is executed. In the present system, on a setting screen such as an initial menu screen displayed on the display 2 or the like, a user selects various processes and inputs setting values via the keyboard 3 or the mouse 4 or the like. . After step S1, it is determined whether a menu is selected by the user on the initial menu screen (S2). If "image registration" is selected in step S2, the process proceeds to an image registration process (S3) for performing a process such as registering image data together with its color characteristics in the image database 50, and then proceeds to step S6. In step S2, "similar image search"
Is selected, the process proceeds to a similar image search process (S4) for performing a process of searching the image database 50 for similar image data based on the designated image data, and then proceeds to step S6. If "other menu" is selected in step S2, other menu processing is performed (S5), and thereafter, the process proceeds to step S6. Step S
If the menu is not selected in step 2, the process proceeds to step S6 without doing anything. In step S6, other processes are executed. When all processes are completed, the process returns to step S2, and thereafter, the same processes are repeated.

The other menu processing (step S5) is basically the same as that of a general search system, and is not directly related to the present invention. (Step S3) and the similar image search process (Step S4) will be described in detail below.

<Image Registration Process> The image registration process (step S3) will be described with reference to the flowchart of FIG. In this process, a process of extracting a color feature from an image specified by the user and registering the color feature and the specified image in the image database 50 is performed.

First, an image designated by the user for registration (hereinafter referred to as "designated image") is read (S31). Thereafter, a feature amount extraction process for extracting a feature amount from the designated image is performed (S32). The feature amount extraction processing will be described later. Thereafter, the extracted feature amount is added to the image data of the designated image (S33), and registered in the image database 50 together with the designated image data (S3).
4) Return.

The feature extraction process (step S32) will be described with reference to the flowchart of FIG. In the feature amount extraction processing, first, as a color feature amount, the occupation ratio of a predetermined color to the entire designated image is obtained (S321). Specifically, the colors of the pixels included in the designated image are counted for each color space area defined in the color space area table 55 shown in FIG. 4, and the ratio of each color space area to the entire image is calculated. I do. FIG. 9 shows an example of the occupancy of each color space area obtained in this manner.

Further, the color of the object area included in the designated image (hereinafter referred to as "object color") and its size are obtained (S322). Specifically, edge detection or the like is performed on the image, an object region included in the designated image is recognized, and the average color of the pixels included in the object region is determined as the object color. Here, as the object color, the color with the largest number of pixels in the object region may be used. Further, a ratio of the object region to the entire designated image (hereinafter, referred to as “occupancy ratio of the object color”) is obtained. FIG. 10 shows an example of the object color and the occupancy of the object color obtained in this way. Here, three object regions are recognized in the designated image, and the object color and the occupancy for each are obtained.

As described above, the image is stored in the image database 5.
When the image data is registered as 0, the occupancy of each color space area of the image (see FIG. 9), the object color of the image and the occupancy thereof (see FIG. 10) are added to the image data as feature amounts. be registered.

<Similar Image Search Process> The similar image search process (step S4) will be described with reference to the flowchart of FIG. In this processing, a color feature is extracted from an image (key image) serving as a search key specified by the user, and a similarity is calculated from the color feature and a color feature of an image registered in the image database 50. A process of searching for an image based on the similarity is performed. At this time, in the present embodiment, the similarity is calculated by giving a particular weight to colors that are easily noticed by humans, for example, bright colors and bright colors. Also, if an image of any object is included in the image,
Generally, attention is likely to be focused on this object. For this reason, the similarity is calculated by weighting the colors of the object area in the image in the same manner as the bright colors and the bright colors.

In FIG. 11, first, a key image designated by a user is read (S41). A feature is extracted from the key image (S42). At this time, the occupancy for each color space area, the object color of the key image and its occupancy are extracted as feature amounts from the key image. The extraction of the feature amount is performed in the same manner as the above-described feature amount extraction process (step S32). Thereafter, a process of comparing the feature amount of the key image with the feature amount of the image registered in the image database 50 and searching for an image similar to the key image from a group of images registered in the image database 50 is performed. Do.

First, it is determined whether or not there is a search target image in the image database 50 for which the similarity has not been calculated with the key image (S43). When there is a search target image in the image database 50, the feature amount of the target image is read from the image database 50 (S
44). Thereafter, a similarity calculation process of calculating a similarity from the feature amount of the key image and the feature amount of the target image is performed (S4).
5) Return to step S43. This similarity calculation process will be described later. Step S4 until the similarity is calculated for all target images in the image database 50.
Repeat steps 3 to S45. If the similarity has been calculated for all the target images in the image database 50 in step S43, search result output processing is performed (S4).
6) Return. Search result output processing (step s4
In 6), based on the similarities calculated for all the target images in the image database 50, images having similarities higher than a predetermined value (threshold) are used as search results in order of decreasing similarity. Displayed on the screen. Further, at this time, a threshold value of the similarity may not be provided, and a predetermined number of images may be displayed on the screen as a search result in descending order of the similarity.

The similarity calculation process (step S45) will be described with reference to the flowchart of FIG. First, referring to the color space area table 55, a color area (bright color) in which a flag indicating a vivid color is set is determined, and based on the occupancy of the key image and the occupancy of the target image with respect to the vivid color, A similar color similarity D1 is calculated (S451).
Next, the color area (bright color) in which the flag indicating the bright color is set is determined with reference to the color space area table 55, and based on the occupancy of the key image and the occupancy of the target image with respect to the bright color, the bright color is determined. Is calculated (S452).
Next, the similarity D3 of the entire image is calculated from the occupancy of the key image and the occupancy of the target image (S453). In FIG.
Vivid color similarity D between key image and target image
1, an example of calculation of similarity D2 of a bright color and similarity D3 relating to the color of the entire image will be described. In this figure, the similarity D1 of the bright color is 5.374, the similarity D2 of the bright color is 9.082, and the similarity D3 of the entire image is 3.097. The values shown in this figure are obtained by the following equations. Difference for each color space area = | (Occupancy of key image) − (Occupancy of target image) | Similarity = 1 / distance where distance = {sum of (square of difference in each color area)}
Is the square root of

Returning to FIG. 12, after that, the similarity D4 of the object color between the key image and the target image is calculated from the object colors of the images and the occupancy thereof (S453). FIG. 14A shows the object color of the key image and its occupancy, and FIG.
4B shows the object color of the target image and its occupancy.
FIG. 14C is a diagram showing the similarity calculated from the object color and the occupancy in each of those images. As shown in FIG. 14, here, three object colors 1, 2, and 3 are extracted from the key image, and two object colors A and B are extracted from the target image. FIG.
In (c) of FIG. 4, the color difference, the occupancy ratio, and the similarity are respectively calculated by the following equations. Color difference = distance in the color space between the object color of the key image and the object color of the target image Occupancy ratio = (smaller value of the occupancy of the key image and the occupancy of the target image) / (occupation of the key image The larger of the ratio and the occupancy of the target image) Similarity = (1 / color difference) × occupancy ratio

Each of the object colors 1, 2, and 3 of the key image,
After calculating the similarity between the object colors A and B of the target image, the similarity having the maximum value among the similarities regarding the object colors is defined as the object color similarity D4. In the example of FIG. 14C, since the similarity 4.037 calculated between the object color 1 of the key image and the object A of the target image is the maximum, this is set as the similarity D4.

Returning to FIG. 12, the similarity between the key image and the target image is calculated by the following equation based on the similarities D1 to D4 calculated in the above steps (S455). Similarity = (D1 · W1 + D2 · W2 + D3 · W3 + D4
(W4) / (W1 + W2 + W3 + W4) Here, D1: similarity of bright colors D2: similarity of bright colors D3: similarity of entire image D4: similarity of object colors W1: weight for similarity of bright colors W2 : Weight for similarity of bright colors W3: Weight for similarity of entire image W4: Weight for similarity of object colors Note that weights W1 to W4 for respective similarities D1 to D4 may use values set in advance. Alternatively, a value set by the user each time may be used.

As described above, in the image retrieval device of the present embodiment, when calculating the final similarity between images,
In addition to the color similarity D3 of the entire image, vivid colors, bright colors, and color similarities D1, D2, and D4 of objects included in the image.
Is calculated with particular weight. in this way,
By judging the similarity between images by giving a particular weight to a color that is easily noticed by humans, a search result of a similar image closer to human perception can be obtained, so that search accuracy can be improved.

[0036]

According to the image retrieval apparatus, the image retrieval method, or the recording medium on which the image retrieval program is recorded according to the present invention, when calculating the similarity, a color that is easily noticed by a human, for example,
Since the calculation is performed with particular weight for the bright and bright colors, a search result close to the similarity of humans can be obtained, a high similarity of the search result is realized, and the search accuracy can be improved. Further, since a human can easily pay attention to the image of the object in the image, the search accuracy can be further improved by calculating the similarity by weighting the color of the image of the object included in the image.

[Brief description of the drawings]

FIG. 1 is a diagram schematically illustrating the configuration of an image search device according to an embodiment.

FIG. 2 is a block diagram mainly showing a control device of the image search device.

FIG. 3 is a diagram showing an example of the configuration of an image database.

FIG. 4 is a diagram showing an example of the configuration of a color space area table.

FIG. 5 is a diagram illustrating an HLS color space.

FIG. 6 is a flowchart of a main flow of the image search device.

FIG. 7 is a flowchart of an image registration process.

FIG. 8 is a flowchart of a feature amount extraction process.

FIG. 9 is a diagram showing an example of the occupancy of a color space area obtained as a color feature.

FIG. 10 is a diagram illustrating an example of an object color of an image obtained as a color feature and an occupancy of the object color.

FIG. 11 is a flowchart of a similar image search process.

FIG. 12 is a flowchart of a similarity calculation process.

FIG. 13 is a diagram showing the occupancy of each color space area of the key image and the target image, and the similarities D1 of bright colors, the similarities D2 of bright colors, and the similarities D3 of the entire image obtained from the occupancies. .

14A shows an object color in a key image and its occupancy, FIG. 14B shows an object color in a target image and its occupancy,
(C) The figure which shows the similarity D4 of the object color between a key image and a target image.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Control device 2 ... Display 3 ... Keyboard 4 ... Mouse 5a ... Floppy disk 5b ... Floppy disk device 6 ... Hard disk 9a ... CD-ROM 9b ... CD-ROM device 50 ... Image database 55 ... Color space area table 201 ... CPU 203 ... ROM 204 ... RAM.

Claims (12)

[Claims] An image storage unit configured to store an image; an image input unit configured to input a designated image; a color feature acquisition unit configured to acquire a color feature from each of the designated image and the stored image; A similarity calculating unit configured to calculate a similarity indicating a degree of similarity between the two images by weighting a color of a predetermined area in a color space from the color characteristics and the color characteristics of the stored image; A search unit for searching an image similar to the designated image from the stored images based on the image. 2. The image retrieval apparatus according to claim 1, wherein an object region included in each of the designated image and the stored image is detected, and an object color extracting unit extracts a color of the detected object. And wherein the similarity is calculated by further weighting the color of the object. 3. The image search device according to claim 1, wherein the color space is an HLS space, and the predetermined area is an area whose brightness is higher than a predetermined value. Search device. 4. The image search device according to claim 1, wherein the color space is an HLS space, and the predetermined area is an area having a saturation higher than a predetermined value. Image search device. 5. An image retrieving method for retrieving an image similar to a specified image from stored images, an image inputting step of inputting a specified image, a color feature of the specified image, and a color of the stored image. A color feature obtaining step of obtaining a feature, and a color feature of the designated image and a color feature of the stored image, and a similarity indicating a similarity between the two images, An image search method, comprising: a similarity calculation step of calculating with a weight; and a search step of searching an image similar to the designated image from the stored images based on the similarity. 6. The image retrieval method according to claim 5, wherein the step of detecting an object region included in each of the designated image and the stored image and extracting a color of the object region is performed as the similarity calculating step. An image retrieval method, wherein the similarity is calculated by further weighting the color of the object image in the similarity calculating step. 7. The image search method according to claim 5, wherein the color space is an HLS space, and the predetermined area is an area whose brightness is higher than a predetermined value. retrieval method. 8. The image search method according to claim 5, wherein the color space is an HLS space, and the predetermined area is an area having a saturation higher than a predetermined value. Image search method. 9. An image retrieval program for retrieving an image similar to a designated image from stored images on a recording medium recording a computer-readable program, wherein: an image inputting step of inputting the designated image; A color feature extraction step of extracting a color feature from the designated image and obtaining a color feature of the image to be stored; and determining a degree of similarity between the two images from the color feature of the designated image and the color feature of the stored image. A similarity calculating step of calculating a similarity indicated by weighting a color in a predetermined area of a color space; and searching the stored image for an image similar to the designated image based on the similarity. A recording medium recording an image retrieval program, characterized by having a retrieval step. 10. The recording medium according to claim 9, wherein the program detects the object region included in each of the designated image and the stored image, and extracts the color of the object region. A recording medium having an image search program recorded therein, which is provided before a degree calculating step, wherein the similarity is calculated by further weighting the color of the object image in the similarity calculating step. 11. The recording medium according to claim 9, wherein the color space is an HLS space, and the predetermined area is an area whose brightness is higher than a predetermined value. A recording medium on which a program is recorded. 12. The image recording apparatus according to claim 9, wherein the color space is an HLS space, and the predetermined area is an area where saturation is higher than a predetermined value. A recording medium that records a search program.