KR20030005908A - apparatus and method for representative feature extraction of objects and, its application to content-based image retrieval - Google Patents

Abstract

PURPOSE: An apparatus and a method for content-based image retrieval using extraction of object features of an image are provided to promptly extract representative colors and quality features of objects constructing a still image by using a VQ vector clustering algorithm. CONSTITUTION: An apparatus for content-based image retrieval using extraction of object features of an image includes a first converting part(10) converting image data of RGB color models inputted from the outside into image data of HSV color models; a second converting part(20) converting the image data of RGB models into gray pixel values; a feature extracting part(30) extracting color features of each pixel output from the first converting part, and quality features of each pixel output from the second converting part, a VQ(Vector Quantization) part(50) clustering eight-dimensional vector output from the feature extracting part, and a feature DB(60) storing each cluster clustered at the VQ part.

Description

Apparatus and method for representative feature extraction of objects and, its application to content-based image retrieval}

The present invention relates to an image analysis and retrieval system using digital image processing technology, and more particularly, to an apparatus feature and method for extracting object features of an image through a vector quantization (VQ) vector clustering algorithm and a content-based image retrieval using the same. .

Recently, with the development of computer and communication technology, the demand for multimedia information service is increasing, and VOD (Video On Demand) service, electronic library, medical field (remote medical care, medical service) to be used for still image and video search and broadcasting production on the Internet There is an increasing need for research on multimedia retrieval technology that can be applied.

In the early stages of research, text-based retrieval has been used in which humans directly index all the multimedia data to be searched, and users also search for desired information by using keywords.

However, this method is time-consuming and expensive, and the search efficiency is greatly reduced if the indexer's and searcher's views are inconsistent. In addition, the complex attribute of multimedia data has a disadvantage that can not accurately represent only text.

To compensate for this, there is a need for a content-based retrieval method that extracts features that can represent the contents of multimedia data and performs indexing and detection based on them. This method has the advantage of reducing the time and manpower required to build a database by automatically extracting features from multimedia data and using them in the indexing process.

Typical content-based image retrieval systems include IBM's QBIC, Virabe's Virage, Interpix Software's Image Surfer, Columbia University's VisualSEEK, University of Chicago WebSeer and Excalibur's Excalibur Visual RetrievalWare.

These systems accurately segment the objects constituting an image for content-based image retrieval, extract representative features representing each segmented object, and then use it for image retrieval, thereby clearly identifying the object to be retrieved from the image. Inquiries are made possible.

However, it is not easy to accurately divide the objects that make up an image.

Therefore, to solve this problem, one object in the image is regarded as having similar color and texture features, and the feature values extracted from one image are clustered through a pattern classification algorithm called Expectation-Maximization (EM) and used for searching. A study on base-based queries is underway (Serge Belongie, Chad Carson, Hayit greenspan, and Jitendra Malik, "Color and Texture-Based Image Segmentation Using EM and Its Application to Content-Based Image Retrieval", Sixth Intenational Conference on Computer Vison , pp. 675-682, January 1998).

However, the EM algorithm must predetermine the number of clusters to be clustered, and select the number of clusters that are most appropriately classified as the number of main entities constituting the image among the clustering results for any number of clusters. Therefore, it takes a long time to extract the representative feature of the object, and there is a problem that it is not easy to extract the representative feature value that properly represents the content of one image.

Therefore, the present invention has been made to solve the above problems, by using a fast image extraction and representative object properties that properly represent the content of a video image search, apparatus and method capable of content-based image search The purpose is to provide.

Another object of the present invention is to provide an apparatus and method capable of searching an image regardless of the position, rotation, and size change of an object through content-based image retrieval.

1 is a view showing an apparatus for extracting an object feature of a still image according to the present invention

2 is a diagram illustrating a method for extracting object features of an image according to the present invention.

3 is a flowchart illustrating a VQ vector clustering algorithm according to the present invention.

4 is a view showing a representative feature image according to the present invention;

5 is a flowchart illustrating an image retrieval process according to the present invention.

6 is a view showing a query image having a color table and an image in a DB according to the present invention;

7 is a graph comparing the time required for clustering according to the present invention with the prior art

* Explanation of symbols for main parts of the drawings

10: first conversion unit 20: second conversion unit

30: color feature extraction unit 40: texture feature extraction unit

50: VQ part 60: feature value DB

Features of the object feature extraction apparatus of the image according to the present invention for achieving the above object is a first conversion unit for converting the image data of the RGB color model input from the outside into the image of the HSV color model, and is input from the outside A second converter converting the image data of the RGB model into gray pixel values, a color feature of each pixel converted and output by the first converter, and a texture feature of each pixel converted and output by the second converter It includes a feature extraction unit for extracting a value, a VQ unit for clustering the 8-dimensional vector output from the feature extraction unit, and a feature value DB for storing each cluster clustered in the VQ unit.

A feature of the object feature extraction method of an image according to the present invention for achieving the above object is that when the image data of the RGB model is input from the outside, the image data of the RGB model is converted into image data and gray pixel values of the HSV model Extracting three color features of each pixel using the image data converted into the HSV model, and converting the image data converted into gray pixel values into the ASM (Angular Second Moment) between the pixels; Extracting five texture feature values of contrast, correlation, variance, and entropy, and assigning each of the extracted five texture feature values to all the pixels in the corresponding area. A step of having an eight-dimensional feature vector and clustering each pixel having the eight-dimensional feature vector using a vector quantization (VQ) vector clustering algorithm. Create a cluster by grouping and makin it comprises the step of storing the specific value DB.

In this case, the VQ vector clustering algorithm designates a threshold value for designating a clustering range to be used in the clustering process, and when an 8-dimensional feature vector is input for each pixel, a first determination for determining whether vector data exists in the input pixel. And if the first determination result vector data does not exist, determining that clustering for one image is finished, and ending the execution; and if the first determination result vector data exists, clustering the feature value DB. In the second determination step of determining whether or not there is, and if there is no cluster stored in the feature value DB, a new cluster having a feature value vector of the pixel as a center value is generated and stored in the feature value DB. And if there is a cluster stored in the feature value DB as a result of the second determination, the input data and all stored Measuring the distance between the centers of the studs and searching for a cluster having the closest minimum distance, comparing the retrieved minimum distance with the specified threshold, and as a result of the comparison, a threshold with a defined minimum distance If less, the input data is included as a member of the selected cluster and clustered, and the center value of the cluster is newly calculated. When the comparison results, the minimum distance is greater than the defined threshold value, the input data is used as the center value. There is another feature, which involves creating a new cluster.

A feature of the content-based image retrieval method using object feature extraction of an image according to the present invention for achieving the above object is to prepare a query image, which is an image file of a still image of the object of interest to be searched through a scanner or the Internet. And extracting, selecting, and storing the query image as a representative feature value of a plurality of clusters using a VQ vector clustering algorithm, and storing the selected feature image in a feature value DB, and using only color features among 8-dimensional feature values stored in the feature value DB. Generating a color table, selecting a corresponding color of an object to be searched among a plurality of representative colors displayed in the generated color table, and representing a representative feature value of the selected corresponding color and a plurality of objects stored in a feature value DB. Calculating the similarity between the representative feature values, and the similarity is calculated through the similarity calculation. And providing the user with the image according to the representative feature value in the order of luck.

Other objects, features of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

Exemplary embodiments of an apparatus and method for extracting an object feature of an image and a content-based image retrieval method using the same according to the present invention will be described below with reference to the accompanying drawings.

1 is a diagram illustrating an apparatus for extracting an object feature of a still image according to the present invention.

Referring to FIG. 1, a first converting unit 10 for converting image data of an RGB color model input from an external source into an image of an HSV color model, and a second conversion unit for converting image data of an RGB model input from an external source into gray pixel values. A color feature extraction unit for extracting three color features of hue, saturation, and brightness of each pixel converted and output by the second converter 20 and the first converter 10. 30 and five textures, which are Angular Second Moment, Contrast, Correlation, Variance, and Entropy, between each pixel using a co-occurrence matrix. A texture feature extractor 40 which extracts feature values and assigns them to each pixel, and an 8-dimensional vector which is a color and texture feature value of each pixel output from the color feature extractor 30 and the texture feature extractor 40. A clustered VQ unit 50 and each clustered in the VQ unit 50 It consists of the feature value DB 60 which stores a cluster.

An operation of an apparatus for extracting an object feature of an image configured as described above will be described in detail with reference to the accompanying drawings.

2 is a diagram illustrating a method for extracting object features of an image according to the present invention.

Referring to FIG. 2, when image data of an RGB model is input from the outside (S10), the image data of the RGB model is converted into image data of the HSV model through the first converter 10, and the second converter ( In operation 20, the gray pixel value is converted into a gray pixel value. At this time, the first converter 10 performs the following equations (1, 2, 3).

here, to be.

Next, it is determined whether the image data of the RGB model is converted into the image data of the HSV model by the first converter 10 or the gray pixel value by the second converter 20 (S20).

Subsequently, the color data extractor 30 extracts three color features of each pixel, that is, chromaticity, saturation, and brightness, from the image data converted from the RGB model to the HSV model by the first converter 10 (S40). ).

In addition, the texture data extractor 40 converts the image data of the RGB model into gray pixel values by the second converter 20, and performs an ASM (contrast) and correlation (ASM) between the pixels. Five texture feature values of correlation, variance, and entropy are extracted (S50).

At this time, the method of extracting the texture feature value firstly uses an Angular Second Moment (ASM), contrast, correlation, variance, and congestion between pixels using a co-occurrence matrix. Extract five texture features that are entropy.

In this case, ASM and congestion represent image uniformity, contrast represents contrast and regional variability appearing in the image, and correlation represents linear dependence of gray-tone. The variance represents the scatter plot for the gray-level difference of adjacent pixels.

Therefore, in order to obtain a texture feature, one image is divided into 7 x 7, or 49 regions, and the gray level common matrix in the four normalized directions (0 °, 90 °, 45 °, 135 °) is first calculated for each block. The texture feature values are then extracted and averaged from these matrices to obtain texture feature independent of rotation.

This is expressed as an equation.

First, the value of each entry of the matrix before normalization can be obtained by the following definition.

P (i, j, d, direction): The value of matrix i row j column entry for each direction

Lx = (1,2, ..., Nx), Ly = (1,2, ..., Ny), G = (1,2, ..., Ng)

Image I: Ly × Lx-> G

Nx: the number of horizontal pixels in the image

Ny: Number of pixels vertical

Ng: Gray Level

d: distance from neighboring pixels to consider in calculation

num: the coefficient that satisfies the definition

After calculating each entry value of the matrix in this way, each entry is divided by the following R value for each direction, and normalized to each gray level common in four directions (0 °, 90 °, 45 °, 135 °). The generation matrix can be obtained.

0 °: R = 2Ly (Lx-1) 90 °: R = 2Lx (Ly-1)

45 °: R = 2 (Ly-1) (Lx-1) 135 °: R = 2 (Lx-1) (Ly-1)

And by using the matrix obtained through the above method to calculate each texture feature value as shown in equations (4) to (8).

Where p (i, j) is the (i, j) th entry of the gray level common occurrence matrix

Ng: Gray Level

: Mean and standard deviation of P _x and P _y

mu: mean of the gray level common matrices

P _x : marginal probability matrix obtained by summing the rows of p (i, j)

P _y : The marginal probability matrix obtained by summing the columns of p (i, j)

At this time, the color feature in the color feature extractor 30 is extracted from each pixel, and the texture feature in the texture feature extractor 40 is extracted through the relationship between adjacent pixels.

Therefore, color features are pixel-specific, while texture features are local.

Therefore, in order to use the vector quantization (VQ) vector clustering algorithm, the texture characteristic value must be converted to have the pixel unit feature value by applying the texture feature value to all the pixels in the region (S60).

Thus, by giving five texture feature values of each block to all the pixels in the block, each pixel together with the three color feature values has an eight-dimensional feature value vector (S70).

After clustering is performed using the VQ (Vector Quantization) vector clustering algorithm (S80), the cluster is stored in the feature value DB 60 (S90).

The clustered eight-dimensional feature vector stored in the feature DB DB 60 is used as data for extracting representative feature values of the object later.

In this case, the VQ vector clustering algorithm will be described in detail with reference to the accompanying drawings.

3 is a flowchart illustrating a VQ vector clustering algorithm according to the present invention.

Referring to FIG. 3, first, a threshold value for specifying a clustering range to be used in a clustering process is previously designated (S100).

The threshold value should be determined in consideration of the dimensions and characteristics of the data, the range of the data, and the like.

If the threshold value is too small, clustering multiple objects for an object will result in a bad result. If the threshold value is too large, the characteristics of multiple objects will be clustered in one cluster.

Therefore, it is important to set the most appropriate threshold value, and it is impossible to select a threshold value that satisfies all the threshold values required for each object constituting each image. In the present invention, preferably 0.4 is used as the threshold.

When the threshold value is determined as described above, an 8-dimensional feature vector is received in pixel units (S110).

In operation S120, it is determined whether clustering n-dimensional vector data exists in the input pixel. That is, when clustering is performed for all pixels of one image, there is no longer n-dimensional vector data.

Therefore, if the n-dimensional vector data does not exist as a result of the determination, it is determined that all clustering for one image is finished, and the execution ends.

If the n-dimensional vector data exists as a result of the determination, it is determined whether a cluster exists in the current feature value DB 60 (S130).

As a result of the determination, when there is no cluster stored in the feature value DB 60, a new cluster having a center value of the feature value vector of the pixel is generated and stored in the feature value DB 60. That is, the first input data becomes the first cluster, and the vector value becomes the center value of the cluster (S180).

As a result of the determination, if the cluster is stored in the feature value DB 60, the distance between the input data and the center value of all the stored clusters is measured (S140).

The cluster having the closest distance among the measured input data and the distance of the cluster is searched for (S150).

At this time, the method for finding the cluster having the closest distance is calculated through the following equations (9) and (10).

, here

Where X ^{(p) is the} pth input vector, and N is the vector dimension.

C _j : Center of the j th cluster

C _k : Center of cluster closest to input data

N _x : number of members (data)

S _k : k th cluster set

In operation S160, the distance between the obtained cluster and the feature value vector of the corresponding pixel is compared with a specified threshold value.

As a result of the comparison, if the minimum distance is less than 0.4, which is a defined threshold value, the input data is included as a member of the selected cluster and clustered. The center value of the cluster including the pixel is newly calculated by Equation 11.

Meanwhile, as a result of the comparison, if the distance between the obtained cluster and the feature vector of the pixel is larger than 0.4, which is a defined threshold value, there is no cluster to be clustered with the pixel among the clusters currently stored in the feature value DB 60. In response to the determination, a new cluster having the center of the input data is created. In this case, the feature value vector of the first pixel becomes the center value of the first cluster (S180).

As such, the VQ vector clustering algorithm automatically clusters and classifies the data as much as the appropriate clustering number without predefining the clustering number of the data to be classified.

By using this feature, the VQ algorithm can extract representative features relatively flexibly as the number of main objects constituting the image, and also reduce the feature extraction time, compared to other algorithms including the EM algorithm.

As another method for reducing feature extraction time, when image data for extracting a cluster uses a horizontal 192 pixel and a vertical 128 pixel image, the total number of pixels included in the image is 192 x 128 = 24,576.

Then, the number of input vectors to be performed by the VQ vector clustering algorithm is 24,576.

In this case, since the texture feature and the color feature are similar to each other in the image due to redundancy between adjacent pixels, use of the feature value vector of all the pixels only increases the time required unnecessarily.

Therefore, the image is performed by sampling feature values of a pixel to be used as an input vector while skipping each pixel horizontally and vertically.

Accordingly, since only 2,752 input vectors need to be used per image, the number of input vectors for using the VQ vector clustering algorithm can be reduced to about 1/9.

In this manner, the representative feature values of the four clusters extracted from the still image of FIG. 4 using the VQ vector clustering algorithm are calculated as shown in Table 1 and stored in the feature value DB 60.

It can be seen that the representative feature values of the sky, the house, the straw, and the grass, which are the main clusters constituting the image of FIG. 4, are represented by 8-dimensional vectors.

Object sky House straw grass cluster One 2 3 4 Number of members 748 422 370 290 Center Chromaticity 0.5653 0.0576 0.0904 0.3319 saturation 0.3820 0.3025 0.4357 0.2440 brightness 0.4300 0.4014 0.6291 0.3396 ASM 0.0151 0.0017 0.0013 0.0030 prepare 0.0065 0.0294 0.0399 0.0193 correlation 0.7690 0.6932 0.5969 0.6505 Dispersion 0.0256 0.0995 0.1006 0.0601 Congestion 0.4184 0.5901 0.6014 0.5779

Next, a method of searching for an image based on content based on object feature extraction of an image stored in the feature value DB 60 will be described in detail with reference to the accompanying drawings.

5 is a flowchart illustrating an image search process according to the present invention.

Referring to FIG. 5, first, a user prepares a still image of an object of interest to be searched as an image file through a scanner or the Internet (S190).

The prepared image file, that is, the query image, is extracted, selected as a representative feature value of a plurality of clusters using the VQ vector clustering algorithm described above, and stored in the feature value DB 60 (S200).

Subsequently, the center values of the plurality of clusters thus obtained are detected by the feature value DB 60, and a color table is generated using only hue, saturation, and lightness, which are color features, among the 8D feature values (S210).

That is, as shown in FIG. 6, the user is provided with a color table in which four main feature values are selected using a query image prepared by the user.

Then, the user selects a corresponding color of the object to be searched among the representative colors displayed in the color table (S220).

Subsequently, a similarity calculation between the representative feature value of the corresponding color selected by the user and the representative feature values of the plurality of objects calculated for each still image in the feature value DB 60 is performed (S230).

In this case, distance calculation between the feature values is required for the similarity calculation, which is calculated using a weighted Euclidean distance calculation method as in Equation 12 below.

Where Q is the index for the query image,

D: index about image in feature value DB,

H: hue, S: saturation, V: brightness,

ASM: Angular Second Moment, CON: Contrast,

COR: correlation, VAR: variance,

ETRP: entropy,

w ₁ -w ₈ : weight for each feature, w ₉ : weight for color features,

w ₁₀ : Weight for texture features.

The weight calculated in this way is determined to be the most effective through the test on the various clusters in the feature value DB (60).

In the present invention, as a weight for calculating distances, the weight is uniformly assigned to 0.5 for color and texture features, and 0.44 for chroma features, 0.28 for saturation and brightness, and more importance for chroma. did. The texture features were all weighted equally to 0.2.

The value of this weight is a desirable value for viewing the effective search results.

Similarity S _QD is expressed by a similarity between two object feature values as a value between 0 and 1 by Equation 13. In the case of objects with completely different colors and textures, the value is close to 0, and the similarity between the same objects is expressed as 1.

Through the similarity calculation, a plurality of images according to the representative feature values are provided to the user in the order of similarity (S240).

As a result of comparing the clustering time of the VQ vector clustering algorithm and the EM algorithm in the following manner, it can be seen that EM takes 1.4 times longer than VQ as shown in FIG. 7.

As described above, the object feature extraction of the image and the content-based image retrieval apparatus and method using the same have the following effects.

First, it is possible to quickly extract the representative color and texture features of the objects constituting still images using the VQ vector clustering algorithm in addition to the general color and texture feature extraction methods.

Second, by using the feature value extracted from the first effect in the content-based search, the search based on the contents of the still image can be searched, and by searching by the object, it is possible to search irrespective of the position, rotation and size change of the object. .

Those skilled in the art will appreciate that various changes and modifications can be made without departing from the spirit of the present invention.

Therefore, the technical scope of the present invention should not be limited to the contents described in the embodiments, but should be defined by the claims.

Claims (10)

A first converter converting the image data of the RGB color model input from the outside into the image of the HSV color model; A second converter for converting image data of an RGB model input from the outside into gray pixel values; A feature extraction unit for extracting a color feature of each pixel converted and output from the first converter and a texture feature value of each pixel converted and output from the second converter; A VQ unit for clustering an 8-dimensional vector output from the feature extractor; And a feature value DB for storing each cluster clustered by the VQ unit. The method of claim 1, wherein the feature extraction unit A color feature extraction unit for extracting three color features of hue, saturation, and brightness of each pixel; It includes a texture feature extraction unit that extracts five texture feature values, such as Angular Second Moment (ASM), contrast, correlation, dispersion, and entropy, between each pixel and assigns them to each pixel. The apparatus for extracting object features of an image, characterized in that Converting image data of the RGB model into image data of the HSV model and gray pixel values when the image data of the RGB model is input from the outside; Three color features of each pixel are extracted from the image data converted into the HSV model, and the ASM (Angular Second Moment) and contrast between each pixel is converted into the gray pixel value. Extracting five texture features: correlation, variance, and entropy; Assigning the extracted five texture feature values to all the pixels in the corresponding area so that each pixel has an eight-dimensional feature vector; Performing clustering on each pixel having the 8-dimensional specific value vector using a vector quantization (VQ) vector clustering algorithm to generate a clustered cluster and storing the cluster in a specific value DB. Object feature extraction method. 4. The method of claim 3, wherein the VQ vector clustering algorithm is Specifying a threshold for specifying a clustering range to use during the clustering process, A first determination step of determining whether vector data exists in the input pixel when an 8-dimensional feature vector is input in pixel units; If it is determined that the vector data does not exist, determining that clustering for one image is finished, and ending execution; A second determination step of determining whether a cluster exists in a feature value DB when the first determination result vector data exists; If there is no cluster stored in the feature value DB as a result of the second determination, generating a new cluster having the feature value vector of the corresponding pixel as a center value and storing it in the feature value DB; As a result of the second determination, if there are clusters stored in the feature value DB, measuring a distance between the input data and the center value of all stored clusters and searching for a cluster having the closest minimum distance; Comparing the retrieved minimum distance with the specified threshold value; As a result of the comparison, if the minimum distance is less than a defined threshold, including the input data as a member of the selected cluster, clustering the input data, and newly calculating a center value of the cluster; And generating a new cluster having the input data as a center when the minimum distance is greater than a defined threshold. The method of claim 4, wherein The threshold value is determined in consideration of the dimensions (dimensions) and characteristics of the data, the range of the data, etc. The object characteristic extraction method of the image. The method of claim 4, wherein The threshold value is an object feature extraction method of an image, characterized in that 0.4. The method of claim 3, wherein The extraction of the color features and the extraction of the texture feature values include extracting and extracting feature values of the pixels by skipping two pixels in the horizontal and vertical directions in the input image. Preparing a query image, which is an image file, of a still image of the object of interest to be searched through a scanner or the Internet; Extracting and selecting the query image as a representative feature of a plurality of clusters using a VQ vector clustering algorithm, and storing the query image in a feature DB; Generating a color table using only color features among 8-dimensional feature values stored in the feature value DB; Selecting a corresponding color of an object to be searched from among a plurality of representative colors displayed in the generated color table; Calculating similarity between the representative feature values of the selected corresponding color and the representative feature values of the plurality of objects stored in the feature value DB; And providing an image according to a representative feature value in order of similarity to the user through the similarity calculation. The method of claim 8, The similarity calculation is a content-based image retrieval method using object feature extraction of an image, characterized in that it is calculated using a weighted Euclidean distance calculation method. 10. The method of claim 9, wherein the weight is Content-based image using object feature extraction of image, which is equally weighted with 0.5 for color and texture features, 0.44 for color features, 0.44 for saturation and brightness, 0.28 for saturation and brightness, and 0.2 for all five texture features. Search method.