WO2004047026A1

WO2004047026A1 - Image search program

Info

Publication number: WO2004047026A1
Application number: PCT/JP2002/012135
Authority: WO
Inventors: Yusuke Uehara; Daiki Masumoto; Shuichi Shiitani; Susumu Endo; Takayuki Baba
Original assignee: Fujitsu Limited
Priority date: 2002-11-20
Filing date: 2002-11-20
Publication date: 2004-06-03
Also published as: CN100346339C; CN1685365A; JPWO2004047026A1; JP4160050B2

Abstract

An image search program includes a feature amount data extraction unit (103) for extracting feature amount data of a plurality of types (such as layout feature amount, texture feature amount, and shape feature amount) from key image data, adaptation judgment unit (104) for judging presence/absence of adaptation depending on whether the similarity calculation is matched with the human similarity sense for the types of the feature amount data extracted by the feature amount data extraction unit (103), a similarity calculation unit (105) for calculating similarity between the key image data and each image data to be searched, according to the result of judgment of the adaptation judgment unit (104), and a search unit (106) for outputting the image data corresponding to the similarity as a search result.

Description

Image search program Technical field

The present invention relates to an image retrieval program for retrieving desired image data from an image database based on key image data, and more particularly to an image retrieval program capable of improving retrieval accuracy.

Light

Thread

Background Technology Field 1

Food

Conventionally, by extracting and displaying image data similar to key image data given by a user as a search request from an image database storing a large amount of image data in a similar order, the user can efficiently obtain desired image data. Many image search devices have been developed to find them.

A powerful image search device uses a similar image search technology, which numerically calculates the degree of similarity between key image data and the image data to be searched. The feature amount is defined. This feature is used for calculating the similarity of the images.

Here, the feature amount includes the color layout and texture of the image (in the case of graphics, etc., the pattern attached to the surface of the figure and the drawing for expressing the texture) and the image There are types that can be classified according to the viewpoint of capturing features, such as the shape of the contour of an object that exists in the area.

For this reason, many effective definitions and extraction methods have been proposed in the past with the aim of matching the sense of human similarity for each feature type (Norio Katayama, Shinichi Sato). : Index technology for similarity search, IPSJ Journal, Vol. 42, No. 10, pp. 958-964 (2001).

Therefore, the conventional image search device searches for similar images from various viewpoints. A plurality of types of feature values are prepared, and after the user specifies the type of feature value that suits the purpose, the search is executed.

However, in the conventional image retrieval device, it is not easy for a general user who does not have knowledge about the characteristic amount of an image to select a type of the characteristic amount suitable for the purpose of the retrieval.

As a result, it is necessary to repeat the search by deciding the type of the feature amount on a trial basis, and there is a problem that the user's labor required for the search operation is large.

Furthermore, in order for the similarity calculated using these features to match the human sense of similarity, the content of the target image data must satisfy certain conditions that are determined for each type of feature. Needed.

For this reason, if image data that does not meet the conditions is included in the search target, the similarity calculation may deviate from the human sense, and the image that the user feels remarkably lacks similarity with the key image. The data will be included in the search results. This problem does not occur when the search can be restricted to only those that meet the condition of the type of feature using the image data to be searched.

However, if the search target is a collection of image data in which image data of various contents are mixed, such as image data collected using the Internet that has become widespread in recent years, the condition of the feature type will not be met. There is also a problem that a large amount of image data is included and the search accuracy is reduced.

The present invention has been made in view of the above, and an object of the present invention is to provide an image search program capable of improving search accuracy. Disclosure of the invention

In order to achieve the above object, the present invention provides a computer comprising: a feature amount extracting unit configured to extract a plurality of types of feature amounts from a key image; and a type of each feature amount extracted by the feature amount extracting unit. A suitability determining means for determining whether or not the similarity matching the similar sensation is calculated as the presence or absence of the suitability; and a determination result of the suitability determining means. PC leak 002/012135

Three

Based on the result, a similarity calculating unit that calculates a similarity between the key image and each image to be searched, and a search result output unit that outputs an image corresponding to the similarity as a search result. It is an image search program.

According to the present invention, for each type of feature amount extracted from the key image, a force for calculating a similarity degree that matches a human similarity is determined as whether or not there is compatibility. Based on this, the similarity between the key image and each image to be searched is calculated, and the image corresponding to the similarity is output as a search result. This prevents mis-selection of these types, and improves search accuracy. BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing a configuration of an embodiment according to the present invention. FIG. 2 is a diagram showing a table structure of an image database 110 shown in FIG. FIG. 4 is a diagram showing a table structure of a feature amount database 120 shown in FIG. 1, FIG. 4 is a flowchart for explaining an operation of the same embodiment, and FIG. FIG. 6 is a flowchart for explaining the feature amount data storage processing shown in FIG. 6. FIG. 6 is a flowchart for explaining the search processing shown in FIG. 4, and FIG. 7 is a flowchart showing the compatibility determination shown in FIG. FIG. 8 is a flow chart for explaining the processing, FIG. 8 is a view for explaining the search processing shown in FIG. 6, and FIG. 9 is a view for explaining the suitability judgment regarding the color layout feature amount in the embodiment. FIG. 10 shows the colors in the same embodiment. FIGS. 11A and 11B are diagrams for explaining the suitability determination relating to the art feature, FIG. 11 is a diagram illustrating the compatibility determination regarding the color layout feature in the same embodiment, and FIGS. 12A and 12B are the same embodiment. FIG. 13 is a diagram illustrating a color layout feature quantity in FIG. 13. FIG. 13 is a diagram showing a calculation formula used for determining suitability of the color layout feature quantity in the same embodiment. FIG. 15 is a diagram for explaining the suitability judgment regarding the texture feature amount in the embodiment. FIG. 15 is a diagram for explaining the suitability judgment regarding the shape feature amount in the same embodiment. FIG. 16 shows a search result screen 300 when the color layout feature is used in the same embodiment. FIG. 17 shows a texture feature in the same embodiment. FIG. 18 is a diagram showing a search result screen 310 when using a shape feature, and FIG. 18 is a diagram showing a search result screen 320 when using a shape feature in the same embodiment. FIG. 9 is a block diagram showing a configuration of a modification of the same embodiment. BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an embodiment according to the present invention.

The image retrieval apparatus 100 shown in FIG. 1 includes a plurality of feature amounts (color layouts) used to calculate the similarity between key image data and image data in an image database 110 described later. This is a device that selects a feature value that is highly likely to match human similar sensations from among the feature values, texture features, shape features, etc.), and performs an image search using this feature value.

In the image search device 100, the input unit 101 is an input device used for inputting key image data as a search key for similar image search and various key inputs, and includes a digital camera, a scanner, and an Internet communication device. Use a device, keyboard, mouse, etc.

The display unit 102 is an LCD (Liquid Crystal Display), CRT (Cathode Ray Tube), or the like, and displays a key image data and similarity on a search result screen (see FIGS. 16 to 18) related to image search. A predetermined number of search results (image data) are displayed to the user in descending order of the number of search results.

The image database 110 is a database that is configured on a magnetic storage device or a semiconductor memory and stores image data to be searched. Specifically, as shown in FIG. 2, the image database 110 has fields of an image ID and an address. The image ID is an identifier for identifying each image data and key image data to be searched. The address is where the image data is actually stored Represents the place where you are.

For example, the image data whose image ID is “0000001” corresponds to the image data 111 to be searched shown in FIG. 8 (b). This image data 111 # represents a wallpaper image with vertical stripes.

The image data of the image ID is "0000002" corresponds to the image data 111 ₂ to be searched shown in FIG. 8 (c). The image data 111 _2, the same procedure as above Symbol image data 11 represents the wallpaper image of stripes.

The image data of the image ID is "0000003" corresponds to the image data 111 ₃ to be searched shown in FIG. 8 (d). The image data 111 _3, represents the photographic images obtained by photographing the trees and the house of the forest.

Feature database 120, the image data stored in the image database 110 (see FIG. 2) (for example, image data 111 ~: L ll ₃ (FIG. 8 (b) ~ (d) see)) or, the key image It is a database that stores feature data that numerically represents each feature of data (for example, key image data 200 (see FIG. 8 (a))). -Here, in one embodiment, the following three types (1) to (3) are illustrated as types of image feature amounts.

(1) Color layout features

(2) Texture features

(3) Shape features

The (1) color layout feature quantity is a feature quantity representing a spatial distribution state of colors on image data. (2) The texture feature amount is a feature amount representing a pattern in image data or a drawing for representing a texture. (3) The shape feature is a feature representing the shape of the contour of the object existing in the image data.

More specifically, as shown in FIG. 3, the feature amount database 120 includes a color layout feature amount data table 121, a texture feature amount data table 122, and a shape feature amount data table 123. . Each of these color rate feature data table 121, texture feature data table 122, and shape feature data table 123 has fields of image ID and feature data. The image ID corresponds to the image ID in the image database 110 (see Fig. 2).

The feature data in the color layout feature data table 122 is data representing a color layout feature related to image data corresponding to the image ID. The feature amount data of the texture feature amount data table 122 is data representing a texture feature amount regarding the image data corresponding to the image ID. The feature amount data of the shape feature amount data table 123 is data representing a shape feature amount regarding image data corresponding to the image ID.

Returning to FIG. 1, the feature amount data extraction unit 103 extracts the above-described three types of feature amounts (color layout feature amount, color layout feature amount, The feature amount data corresponding to the texture feature amount and the shape feature amount) are extracted and stored in the feature amount database 120 (see FIG. 4).

These feature data are used to calculate the similarity between the key image data and the image data to be searched.

The suitability determination unit 104 calculates the similarity of the feature data used to numerically calculate the similarity between the key image data and the image data to be searched. Whether or not the similarity matching the similar sensation (similarity) of the human being is calculated is determined by whether or not the feature data (feature) is compatible with the given key image data (image data). Is determined.

The similarity calculation unit 105 calculates a vector value corresponding to the feature value data of the key image data determined to be compatible and a feature value data of each image data to be searched. The search unit 106 calculates the similarity of each piece of image data to be searched with respect to the key image data based on the Euclidean distance from the vector value. Next, the operation of the embodiment will be described with reference to the flowcharts shown in FIGS. The image database 1 1 0 shown in FIG. 1, as shown in FIG. ^2, and a plurality of image data to be retrieved is stored in advance. In this state, in step SA1 shown in FIG. 4, it is determined whether or not the user has a request for storing the feature amount data. In this case, the determination result is “No”.

In step S A2, it is determined whether or not the user has a request for image search, and in this case, the determination result is “No”. Thereafter, the determination in step S A1 and step S A2 is repeated until the determination result in step S A1 or S A 2 becomes “Y e s”.

Then, if there is a request from the user for storing the feature data, step S

The judgment result of A1 is "Yes". In step SA3, from the image data stored in the image database 110, feature amount data corresponding to the above three types of feature amounts (color layout feature amount, texture feature amount, and shape feature amount) are extracted. A feature amount data storage process for storing these in the feature amount database 120 is executed.

More specifically, in step SB1 shown in FIG. 5, the feature data extraction unit 103 extracts one image data (for example, image data 1) from the image database 110 (see FIG. 2). 1 (See Fig. 8 (b))). In step SB2, the feature data extraction unit 103 corresponds to three types of features (color layout features, texture features, and shape features) from the image data acquired in step SB1. Extract feature data.

Hereinafter, a method of extracting feature data corresponding to each of the color layout feature, the texture feature, and the shape feature will be described in detail.

(Color layout features)

First, a method of extracting feature data corresponding to the color layout feature will be described. For example, as shown in Fig. 9, the color layout features are obtained by arranging the average of the color values of each partial image data I one-dimensionally when the image data is divided vertically and horizontally into an appropriate number of grids. It is expressed as a value.

Here, the average of the color values of the partial image data I ₅ j is represented by a three-dimensional vector having components of R (red), G (green), and B (blue) in the RGB color space.

Further, the average of the color values of the partial image data I _{i 3} when _{_{(Ru, G n, Β η}} ) and, the color layout _{_{(Ru, G 11} B X1}} , R 12, G 12, B 12J R 44 , G ₄₄ , B

₄₄ ).

In order for the color layout features to be meaningful, a predetermined number or more of partial image data that satisfy the following two conditions (1A) and (2A) are required.

• Condition (1A)

The image data contains a color that is included at a certain percentage or more

• Condition (2A)

Pixels having colors satisfying the condition (1A) are spatially concentrated. These two conditions (1A) and (2A) are based on the average of the color values of the partial image data. This is a condition for indicating that colors are approximated. For example, if many colors exist little by little in image data, averaging the color values is not a good approximation. '

Also, even if a certain percentage or more of the colors exist in the image data, if the colors are divided into small areas and scattered, they are not perceived by humans as remarkable colors. Therefore, the condition is that pixels are spatially concentrated in the image data. Whether or not the above conditions (1Α) and (2Α) are satisfied is determined by the following method. '(How to determine condition (1A))

First, the RGB color space is divided into partial color spaces. Next, after the number of pixels included in the partial image data in each partial color space is counted, a ratio value of the number of pixels of the calculation result to the total prime number of the partial image is calculated for each partial color space.

If there is a partial color space whose ratio value is equal to or greater than a predetermined value, that partial color space is The corresponding color value is set as a representative color of the partial image for the partial image.

As a method of determining the representative color of the partial image, for example, there is a method of determining a color at the position of the center of gravity of the corresponding partial color space. If the representative color of this partial image exists, the condition (1

A) is satisfied.

(How to determine the condition (2Α))

For the representative color of the partial image obtained in the process of judging the condition (1Α), the concentration is calculated by the following method. First, as shown in Fig. 12, for all the pixels on the partial image Μ, a window m with the size of w X h is applied so that the pixels are at the center, and the partial image in the window m is The number _RCxy of pixels corresponding to the representative color is counted. At this time, the degree of concentration SC is calculated by the equation (1) shown in FIGS. In equation (1), RC is the total number of pixels corresponding to the representative color of the partial image, and N is a predetermined value.

When the concentration SC has a representative color of a partial image that is equal to or greater than a predetermined value, the condition (2A) is satisfied.

(Texture feature)

Next, a method of extracting feature data corresponding to the texture feature will be described.

When extracting feature data corresponding to texture features, it is presumed that the entire image data is presumed to be covered with similar textures (patterns). Therefore, in the case of image data that is not covered with a uniform texture, the feature data intended by the calculation method may not be extracted.

As a result, when key image data that is not covered with a uniform texture is given during image search, the image that looks more similar to the human sense in the search target image data In some cases, image data that is not so may show higher similarity than data.

When image search results are displayed in order of similarity, image data that is not similar to human perception is higher than images that are similar to key image data. Would reduce the efficiency of the search.

For example, suppose that key image data 200 shown in FIG. 8 (a) is given as a key image for image search. - How the search target, FIG. 8 (b), and an image data 1 1, 1 1 1 ₂ and 1 1 1 ₃ shown in (c) and (d).

Key image data 2 0 0 and the image data 1 1 1 ₃ is a photographic image obtained by photographing a landscape where there is a house in the grove. On the other hand, the image data 1 1 1 i and 1 1 1 ₂ correspond to a wallpaper image covered with a uniform texture.

The texture features are calculated using the well-known Tamura method (H. Tamura, S. Mori, and T. Yamawaki, lexture Features Corresponding Visual Perception, IEEE Trans. System Man and Cybernetics, Vol. 8, No. 6, (1978). ) Calculated by.

In Tamura's method, texture features are represented by three components: "coarseness", "contrast" J, and "directionality", and the degree of each component is represented by a three-dimensional numerical value. It is extracted from image data as a vector.

“Roughness” indicates the size of the scale of the pattern appearing in the image data, and the larger the scale, the larger the value. “Contrast” indicates the degree of variation in luminance value, and the larger the variation, the larger the value.

“Directivity” indicates the degree to which the direction of the edge component in the image data is concentrated in a certain direction. The larger the frequency in the direction with the highest frequency among the directions of the edge component, the larger the value. It becomes.

When the feature data corresponding to the texture feature is extracted from the key image data 200 (see Fig. 8 (a)) by this Tamura method, the key image data 200 is composed of "roughness" and "contrast". ”And“ direction ”, the image data 1 1 1 and 1 1 1 ₂ show a higher similarity than the image data 1 1 1 ₃ . However, if the user by the image search is looking for landscape images, it is inappropriate for the image data 1 1 1 and 1 1 1 ₂ is included in the search results. If such inappropriate image data is ranked higher in the search results, more unnecessary image data will be seen in order to find the target image data, resulting in lower search efficiency. Will be.

Therefore, for such key image data, the uniformity of the texture is calculated, and if the uniformity is smaller than a predetermined value, it is determined that there is no suitability, and the texture feature can not be used in the search. .

A method of calculating texture uniformity will be described. First, the image to be calculated is divided vertically and horizontally. For example, as shown in FIG. 14 (a) and FIG. 14 (b), image data (key image data 200, etc.) is divided into four equal parts vertically and horizontally.

Next, a texture feature is extracted as a vector from each of the divided partial images. At this time, the method of calculating the texture feature may be the same as the method of calculating the feature used in the similarity calculation, or may be another method.

Here, if the same method, Tamura's method, is used, the feature of the three-dimensional vector is extracted from each partial image. Here, the uniformity of the texture can be represented by the degree of variation of the extracted characteristic vectors.

The more uniform, the smaller the variation. Therefore, the variance is calculated as the degree of variation for the four feature vectors obtained from the partial images, and if the variance is larger than a predetermined value, the uniformity is low, that is, it is determined that there is no compatibility. . In the example of Fig. 14 (a), in the case of the key image data 200, the lower right partial image is significantly different from the other three in "roughness", "contrast", and "direction". Therefore, the variance value is high, and it can be determined that there is no compatibility.

(Shape features)

Next, a method of extracting feature data corresponding to the shape feature will be described. There are two methods for extracting feature data corresponding to shape features: a method in which a pre- contour line is extracted from image data, and a method in which a pre- contour line is not extracted. In most cases, the latter method is used because the contour line has not been extracted in advance.

On the other hand, in the case of arbitrary image data, it is technically difficult to extract the outline of the object existing in the image data, so that the outline of the object is strong in the image data. Utilizing the fact that it becomes an edge component, the shape feature is approximately represented as a frequency distribution for each direction of the local edge component.

In the method of extracting feature amount data corresponding to such a shape feature amount, the following condition (3A) is premised.

Condition (3A)

The presence of an object in a specific part of the image data with a solid color background.

The specific portion is, for example, near the center of the image data.

Here, the method of determining the suitability based on the condition (3A) is as follows. First, as shown in Fig. 15 (a) and (b), pixels are run in the direction toward the center of the image data along a line segment drawn radially from the center of the image data at predetermined angles.查

The difference between the luminance values of successive pixels is sequentially calculated in the scanning process, and the coordinate value ( _X , y) of a point where the difference exceeds a predetermined value is stored.

Repeat this process for each scanning line! / Integrates the distance between the coordinate values between adjacent scan lines for the coordinate values obtained from the results obtained by performing the above steps. Based on this integrated value, if it is larger than the predetermined value, it is determined that there is no compatibility. In the case of the image data 210 that satisfies the preconditions as shown in FIG. 15 (a), the distance between the stored coordinate values becomes small. On the other hand, for the key image data 200 shown in Fig. 15 (b), another object is drawn in the background other than the object (house), and the shape features are extracted from such an image. In this case, meaningful shape features cannot be obtained because the influence of contour lines other than the object is large.

In the case of such image data, since the distance between the stored coordinate values becomes relatively large, it is possible to determine that there is no compatibility by this determination method.

Returning to FIG. 5, in step SB3, the feature data extraction unit 103 converts each feature data (color layout feature, texture feature, and shape feature) extracted in step SB2 into feature data. Stored in database 120 (see Figure 3).

In step SB4, the feature data extraction unit 103 sets the image database 110 It is determined whether or not all the image data in (process of extracting and storing feature amount data) has been processed, and ^^, and the result of the determination is “No”. Thereafter, the above-described processing is performed on the remaining image data.

When the process is completed, the result of the determination in step SB4 is set to “Y e s”, and the feature amount data storage process ends.

Then, when there is a request for image retrieval from the user, the result of the determination in step SA2 shown in FIG. 4 is set to “Yes”. In step SA4, a search process for searching for image data similar to the key image data is executed.

Specifically, in step SC1 shown in FIG. 6, the user inputs, for example, key image data 200 shown in FIG. 8A from the input unit 101. The key image data 200 corresponds to the photographic images taken from the image data 1 1 1 ₃ different angles scenery in the same location as shown in Figure 8 (d).

In step S C2, a suitability determination process is performed for the key image data 200 to determine suitability for each of the color layout features, the texture features, and the shape features.

Specifically, in step SD1 shown in FIG. 7, the suitability determination unit 104 determines suitability of the color layout feature. That is, the suitability determining unit 104 divides the key image data 200 vertically and horizontally.

Here, the key image data 200 is, as shown in FIG. 9, the vertical four equal parts, and those divided transversely to 4 equally divided 1 6 partial image data (I u ~1 ₄₄₎ I do. Next, the suitability determining unit 104 determines, for each of the partial images (I ti to I ₄₄ ) of the key image data 200, each partial color space obtained by dividing R, G, and B in the RGB color space into four equal parts. The ratio value of the number of pixels included in the space is calculated, and the ratio value of the partial color space having the maximum ratio value is obtained as the value shown in FIG.

Here, the ratio value ranges from [0.0, 1.0], and the larger the value, the larger the ratio.

In addition, regarding the color layout feature amount, in the determination method of the condition (1 A) described above, Assuming that the threshold value of the ratio value for the partial image representative color is 0.3, all the partial images have a partial color space having a ratio value equal to or larger than the threshold value, and have the representative color of the partial image.

In addition, assuming that N in Equation (1) (see FIG. 13) for determining the degree of concentration SC in the method of determining the condition (2 A) of the color layout feature is 6, the portion having the ratio value shown in FIG. It is assumed that the calculation result of the degree of concentration s C of the image representative color is a value shown in FIG. The degree of concentration SC is in the range of [0.0, 1.0], and the higher the value, the more concentrated.

Here, assuming that the threshold value of the degree of concentration SC in the determination method of the condition (2A) of the color layout feature amount is 0.6, except for the partial image at the lower right shown in FIG. Satisfy condition (2 A). The partial image at the lower right does not satisfy the condition (2A).

Lastly, when the threshold value of the number of partial images that satisfies both the condition (1A) and the condition (2A), which are criteria for judging the compatibility with the color layout features, is 14, key image data 200 11 Since the conditions (1 A) and (2 A) are both satisfied for the 15 partial images except the lower right partial image shown in FIG. 11, it is determined that there is compatibility with the color layout feature.

In step SD2 shown in FIG. 7, the suitability determining unit 104 determines the suitability of the texture feature. That is, as shown in FIG. 14 (a), the suitability determination unit 104 divides the key image data 200 into two equal parts vertically and horizontally and divides the test feature amount from each partial image into a vector / value. Extract.

Next, the suitability determining unit 104 calculates the variance between the vector values as, for example, 0.7.

Here, the variance value ranges from [0.0, 1.0], and it is assumed that the larger the value, the greater the variance.

Here, assuming that the threshold value for the variance value for judging the compatibility between the key image data 200 and the texture feature amount is 0.6, the variance value of the key image data 200 is Is greater than or equal to the value. Therefore, in this case, it is determined that the key image data 200 does not have the uniformity of the texture, and that the key image data 200 is not compatible with the texture feature amount ^). In step SD3 shown in FIG. 7, the suitability determination unit 104 determines the suitability of the shape feature. That is, as shown in FIG. 15 (b), the suitability judging unit 104 divides the key image data 200 along a line segment radially drawn every 22.5 degrees from the center of the key image data 200. The pixels are scanned in the direction toward the center of the image.

■ Here, the suitability determination unit 104 sequentially calculates the difference between the luminance values of successive pixels in the scanning process, and stores the coordinate value (x, y) of the pixel having a difference exceeding a predetermined threshold. I do. For example, the threshold value of the difference is 0.8 when the range of the difference value is [0.0, 1.0].

It is assumed that the result obtained by integrating the distance between the coordinate values between adjacent scanning lines is 1150 with respect to the coordinate values obtained from the results of performing the above for each scanning line segment. Here, if the threshold value for this integrated value is 10000, the key image data 2000 has an integrated value larger than the threshold value, and it is determined that there is no suitability for the shape feature amount. You.

Based on the above, it is determined that the key image data 200 is compatible with the color layout feature, but is incompatible with the texture feature and the shape feature.

Referring back to FIG. 6, in step SC3, the feature data extraction unit 1◦3 extracts the feature data corresponding to the compatible type (in this case, the color layout feature) from the key image data 200. Is extracted.

In step SC4, the feature data extraction unit 103 converts the feature data (color layout feature) extracted in step SC3 into an image ID (= 0) corresponding to the key image data 200. The data is stored in the color layout feature data table 12 1 (see FIG. 3) of the feature database 120 in association with 0 0 0 0 0 4).

In FIG. 3, in the texture feature amount data table 122 and the shape feature amount data table 123, feature amount data corresponding to the image ID (= 0000 004) is stored. , But not actually stored. In step SC5 shown in FIG. 6, the similarity calculation unit 105 obtains feature amount data (of the key image data 200) corresponding to the image ID (= 0000004) from the color layout feature amount data table 121 shown in FIG. Color layout features). This feature amount data corresponds to the type (color layout special amount) determined to be compatible with the key image data 200 in step SC2.

Next, the compatibility determination unit 104 determines whether or not there is compatibility between the above type (color layout feature amount) and each image data to be searched stored in the image database 110.

Specifically, the suitability determination unit 104 determines whether or not there is compatibility between the color layout feature amount and each image data in the same manner as in step SD1 shown in FIG. If the type is a texture feature, the suitability determination unit 104 determines whether or not there is compatibility between the texture feature and each image data in the same manner as in step SD2 shown in FIG. Is determined.

If the type is a shape feature, the suitability determination unit 104 determines whether or not there is compatibility between the shape feature and each image data in the same manner as in step SD3 shown in FIG. Is determined.

Next, the similarity calculation unit 105 excludes the image data determined to be incompatible by the compatibility determination unit 104 from all the image data to be searched, and removes the image data determined to be compatible. The search target is narrowed down to data (for example, image data corresponding to an image ID (= 0000001 000000 2 0000003)).

Next, the similarity calculating unit 105 obtains the image ID (= 0000001 0000 002 0000003) corresponding feature quantity data (the image data 111 11 1 ₂ and 111 ₃ colors Reiauto features).

Next, the similarity calculation unit 105 calculates the feature data (color layout feature) corresponding to the key image data 200 and the feature data corresponding to the key image data 1 1 1 1 1 1 ₂ and 1 1 1a, respectively. Calculate the critical distance from the data (color layout features). Rounding off to two decimal places, the result (Euclidean distance Is as follows.

• Key image data 200 and image data 1 1 1 i: 1 1 1.6

• Key image data 200 and image data 1 1 1 ₂ : 101.7

• Key image data 200 and image data 1 ₁₁₃ : 7.1

In the above result, the shorter the Euclidean distance, the higher the similarity between the key image data 200 and the image data to be searched.

Therefore, the order of the similarity to the key image data 200 (color layout feature) is as follows: image data 1 1 1 ₃ (see FIG. 8 (d)) in the first place, and image data 1 in the second place.

1 1 ₂ (FIG. 8 (c) refer), the third of the image data 1 1 (Figure 8 (b)) as a reference.

In step SC6 shown in FIG. 6, the search unit 106 obtains the search result from the image database 110 (see FIG. 2) in the order of the similarity obtained in step SC5. Image data 1 1 1 ₃ , image data 1 1 1 ₂ and image data 1 1 1 i are obtained.

Next, the search unit 106 causes the display unit 102 to display the search result screen 300 shown in FIG. This search result screen 300 shows the search results when the color layout feature is used. From left to right in the figure, the key image data 200 and the image data 1 1 1 are arranged in descending order of similarity. ₃ , image data 1 1 1 ₂ and image data 1

1 1! Is displayed.

Further, search results can be seen from the screen 3 0 0, for the key image data 2 0 0 (landscape photograph), image data 1 1 1 ₃ feel is more similar to humans (landscape photograph) is most similar The image data 1 1 1 ₂ and the image data 1 1 1 ₃ which are displayed at a position with a high degree and do not seem similar are displayed thereafter.

Here, suppose that, if the image search is performed using the feature data corresponding to the texture feature without performing the suitability determination described above, the texture feature data table 1 2 2 (3rd The key image data 200 and the search target image data (image data 11 1 to 11 1) are used by using the feature amount data (texture space) stored in the The distance between 1 and ₃ ) is as follows.

• Key image data 200 and image data 1 1 1 i: 9.2

'· Key image data 200 and image data 1 1 1 ₂ : 12.8

• Key image data 200 and image data 1 1 1 ₃ : 64.7

Therefore, the order of the similarity with respect to the key image data 200 (texture feature) is as follows: the image data 1 1 1 is the first place (see FIG. 8 (b)), and the image data 1 1 1 ₂ is the second place the reference 8 view (c)), that Do a third of the image data 1 1 1 ₃ reference (Figure 8 (d)).

In this case, the search unit 106 causes the display unit 102 to display the search result screen 310 shown in FIG. This search result screen 310 shows the search results when the texture feature amount is used. From left to right in the figure, the key image data 200, the image data 1 1 1 Image data 1 1 1 ₂ and image data 1

1 1 3 is displayed.

Also, as can be seen from the search result screen 310, the image data 111, which does not seem similar to humans, has the highest similarity to the key image data 200 (landscape photograph). , And the image data 1 1 1 ₃ which is felt to be more similar is displayed at the position with the lowest similarity.

Also, if an image search was performed using feature data corresponding to the shape feature without performing the suitability determination described above, the shape feature data table 1 2 3 (see FIG. 3) The Euclidean distance between the key image data 200 and the search target image data (image data 1 1 1 1 1 1 ₃ ) is calculated using the feature amount data (shape feature amount) stored in Become like

• Key image data 200 and image data 1 1 1 3 .7

• Key image data 200 and image data 1 1 1 ₂ : 66.3

· Key image data 200 and image data 1 ₁₁₃ : 31.7

Therefore, the order of the similarity with respect to the key image data 200 (shape feature amount) is as follows: image data 1 1 1 is the first place (see Fig. 8 (b)), and image data 1 1 1 ₃ is the second place. No. Figure 8 see (d)), the third largest image data 1 1 1 ₂ (FIG. 8 (c) refer). In this case, the search unit 106 causes the display unit 102 to display the search result screen 320 shown in FIG. The search result screen 320 shows the search results when the shape feature is used. From left to right in the figure, the key image data 200, the image data 1 1 1 丄Image data 1 1 1 ₃ and image data 1 1 1 ₂ are displayed.

In addition, as can be seen from the search result screen 320, the image data 111, which is not felt to be similar to humans, is located at a position with a high similarity to the key image data 200 (landscape photograph). The image data 1 1 1 ₃ which are displayed and felt to be more similar are displayed in the middle position of the similarity.

As described above, according to the embodiment, the types (color layout feature, texture feature, shape feature) of each feature extracted from the key image data 200 correspond to human similarity. It is determined whether or not the matching similarity is calculated as the suitability. Based on the determination result, the similarity between the key image data 200 and each image data to be searched is calculated, and the similarity is calculated. Is output as search results (see FIGS. 16 to 18), so that the selection of the type of the feature amount applied when calculating the similarity is prevented, and Search accuracy can be improved. Also, according to one embodiment, in step S C5 shown in FIG.

0 Determines the compatibility of the type of feature value determined to be compatible with 0 with each image data to be searched, and searches the image data determined to be unsuitable for each image data for the search target , And narrows down each image data determined to be relevant as search targets, and calculates the similarity between the key image 200 and each image data of the search target using the type feature amount. Therefore, the search accuracy and search efficiency can be further improved by narrowing down the search targets.

Although one embodiment according to the present invention has been described in detail with reference to the drawings, a specific configuration example is not limited to this embodiment, and a design within a range not departing from the gist of the present invention. Modifications are included in the present invention. For example, in the above-described embodiment, a program for realizing the functions of the image search device 100 shown in FIG. 1 is recorded on the computer-readable recording medium 500 shown in FIG. Then, the function of the image search device 100 may be realized by causing the computer 400 to read and execute the program recorded on the recording medium 500.

'The computer 400 has a CPU (Central Processing Unit) 410 for executing the above program, an input device 420 such as a keyboard and a mouse, and a ROM (Read Only Memory) 430 for storing various data. , A RAM (Random Access Memory) 440 that stores calculation parameters, etc., a reading device 450 that reads a program from a recording medium 500, an output device 460 such as a display and a printer, and various parts of the device. The connection path is made up of 470.

The CPU 410 reads the program recorded on the recording medium 500 via the reading device 450 and executes the program to realize the function of the image search device 100. Note that examples of the recording medium 500 include an optical disk, a flexible disk, and a hard disk.

Further, in one embodiment, when there are a plurality of types (color layout features, texture features, etc.) determined to be compatible by the compatibility determination unit 104, the similarity calculation unit 1 In step 05, a result obtained by integrating similarities calculated for each type may be used as a similarity calculation result.

According to this configuration, when there are a plurality of types determined to be compatible, the result obtained by integrating the similarities calculated for each type is used as the similarity calculation result. Search accuracy can be improved from the viewpoint.

Further, in one embodiment, when there are a plurality of types (color layout feature amount, texture feature amount, etc.) determined to be compatible by the compatibility determination section 104, one of the plurality of types is selected. The user is allowed to select one type, and the similarity between the key image data 200 and each image data to be searched is calculated by the similarity calculation unit 105 using the feature amount of the type selected by the user. May be configured. Further, in one embodiment, the user is allowed to approve the determination result of the suitability by the suitability determination unit 104, and the key image data 200 is determined based on the determination result that has been approved by the user. The similarity calculation unit 105 may calculate the similarity with each image data to be searched.

According to such a configuration, it is possible to support the user in selecting the type of the optimal feature amount, and it is possible to improve the user interface related to the image search.

As described above, according to the present invention, for each type of feature extracted from a key image, it is determined whether or not a similarity matching a human similar sensation is calculated as the suitability. Based on the determination result, the similarity between the key image and each image to be searched is calculated, and the image corresponding to the similarity is output as the search result. The selection effect of the type of the feature quantity to be prevented can be prevented, and if the search accuracy can be improved, the effect is obtained.

Further, according to the present invention, it is determined whether or not there is compatibility between the type of the feature amount determined to be suitable for the key image and each image to be searched, and it is determined that there is no compatibility for each image. Excluding images from the search target, narrowing down each image determined to be relevant as search targets, and calculating the similarity between the key image and each image of the search target by using the type feature amount. Therefore, by narrowing down the search targets, there is an effect that the search accuracy and the search efficiency can be further improved.

Further, according to the present invention, when there are a plurality of types determined to be compatible, the result obtained by integrating the similarities calculated for each type is used as the similarity calculation result. This brings about an effect that the search accuracy can be improved from a simple viewpoint. Further, according to the present invention, when there are a plurality of types determined to be compatible, the user is allowed to select one type from among the plurality of types, and using the characteristic amount of the selected type, Since the similarity between the key image and each of the images to be searched is calculated, it is possible to assist the user in selecting the optimal feature type. Can be improved Play a fruit.

Also, according to the present invention, the user is allowed to approve the determination result of suitability, and the similarity between the key image and each image to be searched is calculated based on the approved determination result. In this case, it is possible to provide support for selecting an optimal feature type and to improve a user interface related to image search. Industrial applicability

As described above, the image search program according to the present invention is useful for image search using feature amount data of a plurality of types (for example, color layout feature amount, texture feature amount, shape feature amount).

Claims

The scope of the claims

1. Computer

Feature amount extracting means for extracting a plurality of types of feature amounts from the key image,

For each type of feature quantity extracted by the feature quantity extraction means, suitability determination means for determining whether or not a similarity matching human similarity is calculated as compatibility or not,

A similarity calculating unit configured to calculate a similarity between the key image and each image to be searched based on a determination result of the suitability determining unit;

An image search program for functioning as a search result output unit that outputs an image corresponding to the similarity as a search result.

2. The suitability determining means determines whether or not the type of the feature quantity determined to be compatible with the key image is compatible with each image to be searched, and the similarity calculation means includes: For each image, images determined to have no suitability are excluded from the search target, and each image determined to have suitability is narrowed down as a search target, and the key image and the 2. The image search program according to claim 1, wherein a similarity with each image to be searched is calculated.

3. When there are a plurality of types determined to be compatible by the compatibility determination unit, the similarity calculation unit integrates the similarity calculated for each type with the similarity calculation result. 3. The image search program according to claim 1, wherein the image search program performs a search.

4. If there are a plurality of types determined to be compatible, the suitability determining means allows the user to select one type from among the plurality of types, and the similarity calculation means includes: The feature image of the type selected by 3. The image search program according to claim 1, wherein a similarity between each image to be searched and each image is calculated.

5. The suitability judging unit allows the user to approve the judgment result of the suitability, and the similarity calculating unit calculates each of the key image and the search target based on the judgment result approved by the user. 3. The image search program according to claim 1, wherein a similarity to an image is calculated.