KR20080018631A - Region weighting mehtod based on spatial location informaiton for relevance feedback in image search engine - Google Patents

Abstract

A space location information-based region weighting method for relevance feedback of an image searching unit is provided to be applied as a core technology for searching images of various fields to thus accurately reflect a user intention and obtain good searching performance. Every image is divided into multiple images by using an image dividing method(step1). A weight value is calculated by extracting a characteristic vector of a size of a region, a location of a region, a distribution of a region, compactness and convexity for weight value information from the divided regions(step2). A region-based image searching process based on a relevance feedback is repeatedly performed by applying the weight value information until the user intention is reflected(step3).

Description

Region Weighting Mehtod Based on Spatial Location Informaiton for Relevance Feedback in Image Search Engine}

1 is a flowchart illustrating an image retrieval process of an area-based image retrieval system based on suitability feedback by applying a spatial position weighting technique according to the present invention.

FIG. 2 is a flow diagram illustrating a region based image retrieval process based on conformance feedback using dynamic region clustering in accordance with the present invention. FIG.

3 illustrates an example of four regions of an animal image according to a size percentage weighting technique, an area frequency weighting technique, and a spatial position weighting technique.

4 is a graph showing an average reproducibility of a spatial position weighting technique, a size percentage weighting technique, and an area frequency weighting technique according to an iterative step.

5 is a graph illustrating a search time of a spatial position weighting technique, a size percentage weighting technique, and an area frequency weighting technique according to an iterative step.

The present invention relates to an area weighting method based on spatial position information for suitability feedback of an image searcher. More specifically, the user's intention is precisely assigned by weighting the area according to the location of the area without weighting the area according to the size of the area. Foreground objects appearing in the center of the image at spatial locations of regions by applying document weighting techniques using term frequency and inverse bibliographic frequency to region-based image retrieval systems based on relevance feedback using dynamic clustering It gives a higher weight to the whole landscape, and is applied as a core technology in various fields such as medical photography, crime prevention and weather forecasting, and image search in various fields such as e-commerce and web image classification, accurately reflecting user's intention, Image search with search performance The present invention relates to a spatial location information-based region weighting method for fitness feedback.

In general, image retrieval is most active with Content Based Image Retrieval, which uses low-level (color, texture, and shape) content attributes to retrieve the user's query image and the most similar images from a large image database. As one of the fields, the development of computer processing speed and the spread of the Internet has made it easy to use digital images of hundreds of thousands of pixels, and the demand for efficient content-based query and image retrieval of images is increasing.

In addition, since content-based image retrieval has low performance in the search system because low-level features of query images do not represent high-level user concepts, the following two problems are solved. Several techniques apply.

1) Relevance Feedback to identify the user's intent.

2) Region-Based Image Retrieval to express the user's perception of the image content as a feature of the segmented area.

Here, Relevance Feedback is an online learning technique that bridges the gap between the low level features of Content Based Image Retrieval and the high level concepts of the user. Evaluate the results by selecting the appropriate images at, where the system improves the results of the next round of search using feedback to calibrate the current query.

In addition, early studies of relevance feedback used single-point and multi-point movement techniques, which are described in the context of MARS [Y. Rui, T. Huang, S. Mehrotra, “Content-based image retrieval with relevance feedback. in MARS ”, Proceedings of IEEE International Conference on Image Processing '97, Santa Barbara, CA, October 1997.] and MindReader [Y. Ishikawa, R. Subramanya, C. Faloutsos, “MindReader. Single point movement techniques such as "Querying databases through multiple examples", Proceedings of the 24th VLDB Conference, 218-227, New York, USA, 1998.] Move the query point in the direction of the determined points.

Then, while moving the query point, query expansion [K. Porkaew, K. Chakrabarti, “Query Re_nement for Multimedia Similarity Retrieval in MARS”, Proceedings of the 7th ACM Multimedia Conference, 235-238, Orlando, Florida, 1999.], Qcluster [D.-H. Kim, CW Chung, K. Barnard, “Relevance feedback using adaptive clustering for image similarity retrieval”, Journal of Systems and Software, 78 (1), 9-23, 2005. As a result, the points judged as suitability are grouped into clusters and the representatives of the clusters are used as new queries.

Region-Based Image Retrieval, on the other hand, represents an image at the object level by applying image segmentation as opposed to the Content Based Image Retrieval approach, which uses only low-level features.

And, the main purpose of using region features is to improve the user's perceptual capturing ability in image content, for which region-based image similarity functions have been proposed. Natsev, R. Rastogu, K. Shim, “WALRUS. A similarity retrieval algorithm for image databases ”, In Proceedings of SIGMOD '99, Philadelphia, PA, 395-406, 1999.] [Y. Rubner, L.J. Guibas, C. Tomasi, “The earth mover's distance, Shimutildimensional scaling, and color-based image retrieval”, Proc. ARPA Image Underatanding Workshop, 661-668, New Orelans, LA, May 1997.]

At this time, the main element of the similarity definition is the region weight, and the region weights presented in the existing region-based image system set the region weights according to the size ratio of the regions, that is, the WALRUS system [A. Natsev, R. Rastogu, K. Shim, “WALRUS. A similarity retrieval algorithm for image databases ”, In Proceedings of SIGMOD '99, Philadelphia, PA, 395-406, 1999.], the similarity between two images is defined as the ratio of the size of the region where the two images coincide. They tend to be large in size, but there is no consistency between this assumption and the user's perception.

In other words, a large area, such as a background area, is not considered as important as a small area, such as a foreground object, but because the user places more importance on the foreground object than the background area in the image. This caused a problem.

The present invention has been made to solve the above problems, it is possible to accurately reflect the user's intention by assigning the weight according to the position of the region without weighting the region according to the size of the region, the term frequency and inverse literature frequency By applying the document weighting technique to the region-based image retrieval system based on relevance feedback using dynamic clustering, we give higher weight to the foreground object appearing in the center of the image by the spatial position of the region. It is applied to image search in various fields such as medical photography, crime prevention and weather forecasting, and e-commerce and web image classification, and accurately reflects user's intentions. Spatial weighting based region weighting It is an object to provide a method.

In order to achieve the object as described above, the present invention provides a region-based image retrieval using fitness feedback, comprising: dividing all images into multiple regions using an image segmentation method; Calculating weights by extracting the size of the area for the weight information, the location of the area, the distribution of the area, the density and the convex feature vectors from the divided areas; And repeating the region-based image retrieval process based on the suitability feedback until the user's intention is reflected by applying the weight information.

And storing the extracted feature vector in a database to measure the distance between the images using the earth's gravity distance in the initial query; Constructing a result set consisting of the top k images close to the query point according to the distance function; Evaluating image suitability of the result set, but introducing attenuation elements in a previous suitable image; Applying a dynamic region clustering technique to correct the query point and distance function, and performing region clustering to merge similar images in a suitable set of images; Generating a representative value of the cluster from the region clustering.

In addition, the initial query is made by including an example image is divided into multiple areas and includes a multiple query point (q), the number of query result image (k), the query point weight (w), the distance function (d).

On the other hand, hierarchical clustering that determines the number of clusters and merges specific clusters of the same level to reduce the number of query points in the next iteration step, in order to group them into fewer clusters than the region of the appropriate image set. Apply Clustering.

At this time, the representatives of the clusters generated from the appropriate image regions of the sorted set form a new query point to calculate the weight, and the generated new query is applied as an input for the next iteration step.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating an image retrieval process of an area-based image retrieval system based on suitability feedback by applying a spatial position weighting technique according to the present invention.

As shown in the figure, as a preprocessing step, all images are divided into multiple regions by applying an image segmentation method (S10).

Then, features such as the size of the area for the weight information, the location of the area, and the dispersion of the area pixels from the area center are extracted from the areas divided in the step S10 (S20).

로 정의되고, 공간 위치 가중치 기법을 위하여 추출된 영역 분산

은 x축과 y축을 따라 계산되므로, 영역 표현은 확대 및 축소에 영향을 받지 않는 특성을 갖는다.In particular, the region position extracted for the spatial position weighting technique is information about the relative position of an object in an image, and the i th region R _ki for the width and width of the image I _k . Generalized coordinates of the center

Domain variance extracted for spatial location weighting

Since is computed along the x- and y-axes, the region representation has the property of being unaffected by zooming in and out.

Finally, in step S20, the region-based image retrieval process based on the relevance feedback reflecting the user's intention is applied by applying the weight information assigned to the region of each image. (S30).

In other words, the present invention grasps the query intention of the user, which is not considered to be important compared to the small area such as the foreground object, and does not have a proportional relationship between the importance of the image and the size of the area. It is done by applying the point.

In order to reduce the semantic gap between high-level user concepts and low-level features appearing in image retrieval, we apply the spatial position weighting technique of region-based image retrieval based on relevance feedback. Since the spatial position weighting technique weights according to the position of the object, the weight of the foreground object is greater than that of the background region based on the assumption that the important region of the image is mainly shown in the center of the image.

2 is a flowchart illustrating a region based image retrieval process based on conformance feedback using dynamic region clustering according to the present invention. As shown in the figure, a flowchart illustrating an image retrieval process of an area-based image retrieval system based on suitability feedback by applying the spatial position weighting technique of FIG. 1 is shown in detail.

Here, the process of when an image is input into the system or transmitted as a query image is shown. When an image is input into the system, the system divides the image into a plurality of areas (S10), and the area size and area center. Feature vectors such as compactness and convexity are extracted, and the extracted feature vectors are stored in a database (S20).

In addition, one example image transmitted by the user in step S10 is analyzed to generate an initial query Q = (q, d, w, k). Multiple query points in space, k is the number of query result images returned by the system, w is the weight of the query points, d is the distance function.

In addition, Earth Mover's Distance is used to measure the distance between two images (S31) [Y. Rubner, L.J. Guibas, C. Tomasi, “The earth mover's distance, Shimutildimensional scaling, and color-based image retrieval”, Proc. ARPA Image Underatanding Workshop, 661-668, New Orelans, LA, May 1997.], The query point (q) is compared with the images stored in the database using the distance function (d), and the distance function (d) As a result, a result set consisting of the number of query result images k close to the query point q is returned to the user (S33).

At this time, the user evaluates the image suitability by distinguishing the suitability of the images of the result (Q), and a suitable image set is constructed based on the suitability evaluation (S35), and the set composed of the suitable images is newly added suitable image and It includes all the appropriate images used in the previous iteration, and the newly added appropriate image reflects the user's concept more accurately by introducing attenuation elements to reduce the effect of the previous suitable image so that the newly added suitable image is more emphasized. (S35a).

Next, in order to correct the query and the distance function, a dynamic region clustering technique consisting of regions based on the clustering process and the clustering merging process is used (S37a).

Here, as the availability of a suitable image increases, the number of query regions is rapidly increased, and the time required to calculate the distance value between the query and the image is proportional to the number of the query regions and reduces the search speed of the system. The region-based clustering process merges similar images in a suitable image set.

The hierarchical clustering algorithm is then applied to a number of cluster groupings that reduce the regions in the appropriate image set, where each cluster matches a new region of the next query, where the cluster merging process determines the number of clusters. In order to reduce the number of query points in the next iteration step, specific clusters at the same level are merged (S37).

공분산(covarinace) 행렬(Si,Sj), 클러스터 안의 요소들의 개수(ni,nj)로 특징지워 질 때, 호텔링(Hotelling)의 T²함수[H. Hotelling, “Multivariate Quality Control In C”, Eisenhart, M. W. Hastay, and W.A. Wallis, eds. Techniques of Statistical Analysis. N.Y., McGraw-Hill, 1947.]는 두 개의 클러스터 위치 동등 여부를 결정하기 위하여 적용된다.For this purpose, two clusters (C _i , C _j ) are each mean vector

Characterized by the covarinace matrix (Si, Sj), the number of elements in the cluster (ni, nj), Hotelling's T ² function [H. Hotelling, “Multivariate Quality Control In C”, Eisenhart, MW Hastay, and WA Wallis, eds. Techniques of Statistical Analysis. NY, McGraw-Hill, 1947.] is applied to determine whether two cluster positions are equal.

Where the two clusters are close enough

는 p와

사이의 자유도를 가지는 F분포의 100(1-α)상위 백분율이다.x _ik is the kth image in C _i ,

With p

100 (1-α) upper percentage of the F distribution with degrees of freedom between.

보다 클 경우에는 두 클러스터의 분리된 상태이고, 전역적인 이미지 표현에 기반한 내용 기반 이미지 검색(Content Based Image Retrieval) 접근 기법에서 동적인 클러스터링을 사용하는 적합성 피드백(Relevance Feedback) 사용은 [D.-H. Kim, C. W. Chung, K. Barnard, “Relevance feedback using adaptive clustering for image similarity retrieval”, Journal of Systems and Software, 78(1), 9-23, 2005.]에 기반을 두고 있다.In other words, if T ²

In larger cases, the use of Relevance Feedback with dynamic clustering in the content-based image retrieval approach based on the global image representation is described in [D.-H]. . Kim, CW Chung, K. Barnard, “Relevance feedback using adaptive clustering for image similarity retrieval”, Journal of Systems and Software, 78 (1), 9-23, 2005.

Finally, the representative of the cluster generated from the appropriate image regions of the sorted set constitutes a new query point, whereby the weight of the representative is calculated, and the new query Q '= (q', d ', w ', k) is generated and used as an input for the next iteration, including a new query point (q'), a new weight (w '), and a distance function (d') to reflect the newly adjusted weight ( S39).

If the user is satisfied with the result after several iterations, the iteration step ends with the last result set.

3 is a diagram illustrating an example of four regions of an animal image according to a size percentage weighting technique, an area frequency weighting technique, and a spatial position weighting technique. As shown in the figure, the spatial position weighting technique can reflect the user's interest by assigning the largest weight to the foreground object, while the foreground, which is the user's interest, by assigning the small weight to the background object. It is made to give a relatively high weight to an object.

Here, a normalized cut segmentation technique that applies each pixel in an image as a point in an arbitrary feature space, grouping similar pixels according to selected color and texture features [J. Shi and J. Malik, “Normalized Cuts and Image Segmentation”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (8), 888-905, August 2000.

인 영역 위치 그리고 x축 및 y축을 따라 계산되어 영역 표현의 확대 및 축소에 영향을 받지 않는

, 영역 중심으로부터 영역 픽셀들의 분산이 사용된다.Each area is expressed using its own color, shape, and weight information, and features such as color standard deviation, density, and convexity in the L * a * b color space are used for color and shape characteristics. The normalized coordinates of the center of the i-th region with respect to the width and width of the image Ik representing information about the area size of the region normalized by the image size for weighting information, and the relative position of an object in an image.

Area position and computed along the x- and y-axes so that they are unaffected by the expansion and contraction of the area representation.

In other words, the dispersion of the area pixels from the area center is used.

On the other hand, the region weighting technique is an important element in region-based image retrieval based on relevance feedback, and an important region is used as a foreground object by using the spatial position of the region within an image. when a is assumed that as shown in the center, each area of the image I _k R _ki With respect to _{the center} of the zone R _ki Image I _k The weighted Euclidean distance between centers is calculated as

으로 나타나고, 영역의 중심

은 영역 위치를 결정하며, 분산

은 영역의 모양과 방향을 나타낸다.Where each image I _k is a set of regions

Appears, the center of the area

Determines the location of the region, and

Indicates the shape and direction of the area.

And, a close area in the center of the image I _k as compared with other areas of the image I _k has a greater weight than the other area weight of each zone is the sum of the normalized as 1, the weight of the region R _ki of images I _k Is defined as follows.

N _k is the number of regions of the image I _k .

In each iteration step, the decay factor β (0 ≦ β ≦ 1) is used to reduce the effect of the suitable image of the previous step, i ₁ , .. Im and the image of the previous step of I _m and I. _{m + 1} ,... , _N suitable images consisting of new images of I _n I ₁ ,. Assuming that I _nk exists, a new region weight is set as follows using a Decay Factor.

Here, the region importance of the cluster C _l is defined as follows.

이 고려되어야 한다.Where v is the number of clusters in each iteration step, and the constraint

This should be considered.

The conformance feedback method according to the present invention consists of a process of region clustering and cluster merging. Through this process, a search for semantically related clusters is performed, and when a user determines that some images are suitable for each iteration of conformance feedback, Cluster a series of points and select the center of the cluster as a representative.

Similar regions are then merged through clustering and form a composite image as a multipoint optimization query, ie Earth Mover's Distance [Y. Rubner, L.J. Guibas, C. Tomasi, “The earth mover's distance, Shimutildimensional scaling, and color-based image retrieval”, Proc. ARPA Image Underatanding Workshop, 661-668, New Orelans, LA, May 1997.] is a composite image of all regions corresponding to the cluster, and the representative and weight of the cluster is the relevance feedback. In the next iteration phase of Region-Based Image Retrieval, it is used as a component of the query point and Earth Mover's Distance function.

The following looks at the experimental example of the present invention.

In order to evaluate a region weighting technique for k-Nearest Neighbor (k-NN) queries in the region-based image database of the present invention, 10,000 general images are selected from a Corel image database. Generate 40 random initial query images from 10 categories (Sunset, Coast, Animals, Airplanes, Birds, Trees, Flowers, Cars, People, Fruits, etc.) and generate Relevance Feedback for simulation. In other words, an image of the same category as the initial query image is considered to be a suitable image, and for each query image, an additional five iterations of feedback are performed in addition to the initial query execution, and all measurements are averaged over 40 queries. K-NN query is used for similarity-based matching, and k is set to 100 (k = 100).

We ran the Pentium-IV Celeron 2.4GHz CPU and 512MB (Mega Byte) memory on the Windows 2003 operating system. The experiment was conducted to answer the following two main questions.

1. How much performance does the method of the present invention improve over size percent weighting and region frequency weighting?

2. How is the average recall and retrieval time improved for a region-based image retrieval system based on relevance feedback using dynamic domain clustering?

Here, the spatial position weighting technique and the size percentage weighting technique using the multipoint relevance feedback [A. Natsev, R. Rastogu, K. Shim, “WALRUS. A similarity retrieval algorithm for image databases ”, In Proceedings of SIGMOD '99, Philadelphia, PA, 395-406, 1999.], [J. Z. Wang, J. Li and G. Wiederhold, “SIMPLIcity. Semantics-Sensitive Integrated Matching for Picture Libraries ”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23 (9), 947-963, 2001] and region frequency weighting techniques, respectively. For this purpose, dynamic domain clustering technique is used, and individual domain frequency weights are replaced with cluster frequency domain weights.

4 is a graph showing an average reproducibility of a spatial position weighting technique, a size percentage weighting technique, and an area frequency weighting technique according to an iterative step. As shown in the figure, the average reproducibility of the region-based image retrieval system of the relevance feedback using dynamic region clustering is determined by the spatial position weighting technique, the size percentage weighting technique, and the region frequency weighting technique of the cluster. Each comparison was made.

In addition, the spatial position weighting technique shows better performance after the first iteration phase, and the average recall of the spatial position weighting technique after 18 iterations is 18% higher than the average recall of the size percentage weighting technique. It is 11% higher than the proposed method, which shows that the spatial position weighting technique improves the performance by 18% and 11%.

After five iterations, the average recall was improved by 39% for the spatial position weighting technique, 38% for the size percent weighting technique, and 37% for the region frequency weighting technique. In addition to higher average reproducibility than weighting technique and region frequency weighting technique, it can provide the fastest performance improvement because it learns the importance of region quickly by using the preference information fed back from Region-Based Image Retrieval. have.

5 is a graph illustrating a search time of a spatial position weighting technique, a size percentage weighting technique, and an area frequency weighting technique according to an iterative step. As shown in the figure, the relevance feedback of region-based image retrieval using dynamic region clustering associated with spatial position to demonstrate the effectiveness of the spatial position weighting technique. The average search time of the spatial position weighting technique, the cluster region frequency weighting technique, and the size percentage weighting technique are compared.

Here, after five iterations, the average search time of the spatial position weighting technique is about one tenth of the region frequency weighting technique, which calculates the similarity value between the region and the associated images that appear in each iteration. This is because it requires additional time to do so.

Therefore, spatial position weighting technique is shown to be superior to size percentage weighting technique and region frequency weighting technique in terms of efficiency and effectiveness.

In other words, in applying the spatial position weighting technique to the relevance feedback including dynamic clustering, weighting of the most recent suitable images to more accurately represent the user's query concept, 10,000 Experimental results with two general-purpose image databases show that the spatial position weighting technique outperforms the spatial positional weighting technique and the size percentage weighting technique in terms of efficiency and effectiveness.

As described above, although described with reference to a preferred embodiment of the present invention, those skilled in the art various modifications of the present invention without departing from the spirit and scope of the invention described in the claims below Or it may be modified.

As described above, the present invention having the configuration as described above can accurately reflect the user's intention by assigning a weight according to the position of the region without giving the region weight according to the size of the region. By applying the document weighting technique to the region-based image retrieval system based on relevance feedback using dynamic clustering, we give higher weight to the foreground object appearing in the center of the image by the spatial position of the region. , Medical photography, crime prevention and weather prediction, and image search in various fields such as e-commerce and web image classification, which can accurately reflect the user's intentions and have excellent search performance. .

Claims (5)

In area-based image retrieval using fitness feedback, Dividing all images into multiple regions using an image segmentation method (S10); Calculating a weight by extracting the size of the area for the weight information, the location of the area, the distribution of the area, the compactness, and the convex feature vector from the divided areas (S20); And Repeating the region-based image retrieval process based on the suitability feedback until the user's intention is reflected by applying the weight information (S30); Spatial location information-based region weighting method for suitability feedback of an image searcher comprising a. The method of claim 1, The step S30 may include storing the extracted feature vector in a database and measuring the distance between the images using the earth's gravity distance in the initial query (S31); Constructing a result set consisting of the top k images close to the query point according to the distance function (S33); Evaluating image suitability of the result set (S35), and introducing attenuation elements into the previous suitable image (S35a); Applying a dynamic region clustering technique (S37a) to correct the query point and distance function (S37a), and performing region clustering to merge similar images in a suitable image set (S37); Generating representative values of clusters from region clustering (S39); Spatial location information-based region weighting method for suitability feedback of an image searcher comprising a. The method of claim 2, The initial query is an image searcher characterized in that the example image is divided into multiple regions and includes multiple query points (q), the number of query result images (k), query point weights (w), and a distance function (d). Spatial Weighting Method Based on Spatial Location Information for Relevance Feedback. The method of claim 2, In order to group the clusters into fewer clusters than the area of the suitable image set in step S37, the number of clusters is determined, and in the next iteration step, merge specific clusters of the same level to reduce the number of query points. Spatial location information based region weighting method for conformance feedback of an image finder, characterized by applying a hierarchical clustering. The method of claim 2, In step S39, representatives of clusters generated from suitable image regions of the sorted set form a new query point to calculate weights, and the generated new query is applied as an input for the next iteration step. A spatial weighting based region weighting method for conformance feedback of an image searcher.

Images