KR100902010B1 - Effcient similarity search method for content based multimedia retrieval with relevance feedback - Google Patents

Abstract

The present invention relates to a content-based multimedia retrieval method including an associated feedback. The multimedia search method comprises the steps of: (a) searching according to the original query object to generate a search result set, calculating distances between the objects and the query object, and storing the calculated results in an approximate distance array (PrevDist), (b) Generating a new query object by performing associative feedback on the search result set; and (c) using the distance information stored in the approximate distance arrangement, the target objects constituting the search result set and the new query object. Calculating approximate distances to (d) generating a candidate search set comprising target objects whose approximate distance is less than a preset similarity r, and (e) filtering the candidate search set. The actual distance between the target objects constituting the new query object and the actual distance are calculated. And a step of generating a correct search set comprised of a small target object than the distance (r). According to the present invention, the retrieval using the associated feedback can filter the low similarity objects by using the distance information obtained from the previous search results, thereby significantly improving the search speed.

Feedback, Search Speed, Multimedia, Query Object

Description

Content-based multimedia retrieval method with associative feedback {EFFCIENT SIMILARITY SEARCH METHOD FOR CONTENT BASED MULTIMEDIA RETRIEVAL WITH RELEVANCE FEEDBACK}

1 is a flowchart sequentially illustrating a multimedia searching method according to an exemplary embodiment of the present invention.

2 is a flow chart illustrating in more detail the processing of the filtering step (step 150) in the multimedia search method according to the preferred embodiment of the present invention.

FIG. 3 is a diagram for explaining a principle of quickly obtaining an approximate distance when re-searching using an associated feedback in a multimedia searching method according to a preferred embodiment of the present invention.

4 is a flowchart illustrating a process of generating a correct answer set (step 160) in a multimedia search method according to an exemplary embodiment of the present invention in more detail.

5 is a diagram showing data used in experiments performed to demonstrate the effectiveness of the search method according to a preferred embodiment of the present invention.

6 and 7 are graphs comparing performances of a search method and conventional methods according to a preferred embodiment of the present invention.

The present invention relates to a multimedia retrieval method, and more particularly, to a multimedia retrieval method for improving retrieval speed when re-searching using relevance feedback in a content-based multimedia retrieval method including relevance feedback.

Recent content-based multimedia retrieval systems use re-search based on associative feedback as a way to reduce the semantic gap. The user can obtain the desired search result through the related feedback, but the search time increases by the number of re-searches until the final result is obtained. Furthermore, the high-level, low-level information used for content-based retrieval requires a very large retrieval time with just one retrieval, so multiple rescans can be a huge burden on the system as a whole. Various indexing methods have been widely studied as a solution for shortening the searching time in the high-dimensional vector space.

Conventional indexing methods can be largely divided into data clustering and filtering methods. KDB-tree, R-tree, R * -tree, X-tree, VP-tree, and M-tree show good performance in multidimensional space. Such data clustering methods are known to show slower search speeds than sequential search in high-dimensional space.

This problem is called "Curse of Dimensionality" and as an effective solution, recently, filtering methods such as VA-file, LPC-file, and HBI have been proposed. Filtering here means that each multimedia object is made into an approximate object of small size, and the approximate distance is used to exclude objects that are far from the query object in a short time. The final search result is in a small set of objects that are not excluded from the filtering step, which can be obtained by calculating the actual distance. The research results show that the filtering methods show about 1.5 ~ 3 times faster search speeds in high-dimensional space than sequential search. However, this indexing method can reduce the absolute time for one-time search, but the problem that the total search time is one-times one-time search time through l re-searches is still not solved.

Accordingly, the present applicant is to propose a search method that can effectively reduce the re-search time in the content-based multimedia search system using the filtering index method and applying the associated feedback.

An object of the present invention for solving the above problems is to provide a multimedia search method that can significantly improve the re-search speed by using the previous search results when re-search by the related feedback in the content-based multimedia search method using the associated feedback It is.

A feature of the present invention for achieving the above-described technical problem relates to a content-based multimedia search method including the associated feedback, the multimedia search method

(a) searching according to the first query object, generating a search result set, calculating actual distances between target objects constituting the search result set and the query object, and storing the result in the approximate distance array PrevDist; ,

(b) performing associative feedback on the search result set and generating a new query object,

(c) calculating approximate distances between the target objects constituting the search result set and the new query object by using distance information stored in the approximate distance array with respect to the search result set;

(d) a filtering step of generating a candidate search set consisting of target objects whose approximate distance is smaller than a preset similarity distance r;

(e) calculating an actual distance between the target objects constituting the candidate search set and the new query object, generating a correct answer search set including target objects whose actual distance is smaller than the similarity distance r; Storing the actual distances of the target objects constituting the correct search set in the approximate distance array PrevDist; and

(f) if the set of correct answers is not satisfactory, repeat steps (b) to (e), wherein the set of search results is the final set of correct answers, and the approximate distance array forms the final set of correct answers. Comprising the actual distances to the target objects to improve the search speed.

In the multimedia retrieval method having the above-mentioned feature, step (c)

(c1) reading distance information of each target object stored in the approximate distance arrangement and a previous query object with respect to the search result set;

(c2) obtaining a distance between an old query object and the new query object;

(c3) calculating approximate distances between the target objects constituting the search result set and the new query object by using the distance between the distance information read from the approximate distance array and the query object. Lose,

In the step (c3), the approximate distances for the respective target objects constituting the search result set may include a difference value between the distance of the corresponding target object read from the approximate distance array and the query objects.

Hereinafter, a content-based multimedia retrieval method including related feedback according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart sequentially illustrating a multimedia searching method according to an exemplary embodiment of the present invention. According to a preferred embodiment of the present invention, a content-based multimedia retrieval method including associative feedback includes (a) a new query object and target objects in a database according to a previous search result based on a previous search result when re-searching by the related feedback. A target object filtering step that quickly calculates an approximate distance between the target objects that are significantly inferior in similarity, and (b) targets objects that are not excluded in the filtering step. By having a final search completion step of calculating the distance to complete the final search result, the search speed can be significantly improved. Hereinafter, a content-based multimedia retrieval method including related feedback according to a preferred embodiment of the present invention will be described with reference to FIG. 1.

First, the search session begins as the first query object q ₀ is input or selected from the outside (step 100). Since the previous query object does not exist when performing the search for the query object q ₀ , the search is performed by calculating the actual distance between the query object and each target object in the multimedia database Λ (step 110). . In this case, when the actual distance between the query object and each target object in the multimedia database Λ is smaller than the preset similarity distance r, the target object is included in the search result set and the actual distance with respect to the target object. Is recorded in the approximate distance arrangement PrevDist (step 120).

The approximate distance arrangement PrevDist is a memory space of 4 x n bytes that is pre-allocated to store the approximate distance calculated for each of the n target objects in the multimedia database Λ for re-search by the associated feedback. .

Next, if the user is not satisfied with the search result, the association feedback is performed (step 130), and according to the result of the association feedback, a new query object q ₁ is defined (step 140), and the new query object q ₁ is generated. To perform rescan. The present invention improves the search speed by introducing a filtering step in the process of performing a re-search using a new query object.

Accordingly, by using the information of the approximate distance array PrevDist, the target objects that are less similar to the new query object among the target objects in the search result set are filtered out (step 150). 2 is a flowchart illustrating in detail the processing of the above-described filtering step (step 150). Hereinafter, the filtering step (step 150) will be described in more detail with reference to FIG.

Referring to FIG. 2, first, target objects in a search result set are read (step 200). After the arbitrary variable i is initialized to 1 (step 210), an approximate distance to the i-th target object is calculated (step 210). Hereinafter, the process and theory of calculating the approximate distance according to the present invention will be described in detail.

First, to calculate an approximate distance for the i-th target object, assume new query object q ₁ , old query object q ₀ and target object p as any three points in the vector space. As shown in Fig. 3, if these three points form a triangle in vector space, then by Triangle Inequality, the length of the two sides, L _p ( q _0, p ) and L _p ( q _{0 ,} q ₁ ), is different. The upper and lower limits of the length L _p ( q _{1 ,} p ) of one side can be calculated. In general, since the re-search is performed by the associated feedback to create a new query object to the default search results for the query object before the new query object q ₁ Is the previous query object q ₀ It is very similar to. Therefore _, it is highly unlikely that the error of distance calculated by estimating the length of L _p ( q _1, p ) and the other two sides is not so large.

Meanwhile, a multimedia database consisting of n multimedia objects Λ = { o _i | Suppose a search on 1≤ i ≤ n}. Assuming that the search space is a d -dimensional vector space, the i th object of Λ , o _i, is _equal to { o _i ¹ , o _i ² ,. , o _i ⁱ ,… , o _i ^d }. In addition, the distance between objects, which is a reference for searching, is calculated using L _p -norm, and the distance L _p ( a _, b ) of any two objects a and b in Λ may be calculated as in Equation 1 below.

To calculate the approximate distance, suppose that we perform a search on a random query object q ₀ in vector space and perform a re-search on a new query object q ₁ created with feedback. Multimedia database (Λ) distance _{L p (q 1, o i} ) with respect to q ₁ within any given object o is _i by the first, second triangle inequality | L _p ( q _{0 ,} o _i ) -L _p ( q _{0 ,} q ₁ ) | ≤ L _p ( q _{1 ,} o _i ) ≤ L _p ( q _{0 ,} o _i ) + L _p ( q _{0 ,} q ₁ ) Exists in the same range as L _p ( q _{0 ,} o _i ) can be obtained without further calculations by keeping the calculation results of the previous search, and L _p ( q _{0 ,} q ₁ ) is the distance between the new query object and the previous query object. Available for all objects in the database. That is, the margin of error for L _p ( q _{1 ,} o _i ) can be calculated with only one subtraction and addition operation, and is an approximate distance not exceeding L _p ( q _{1 ,} o _i ) and A _p ( q _{1 ,} o _i). ) | L _p ( q _{0 ,} o _i ) -L _p ( q _{0 ,} q ₁ ) |

Based on the principle described above, similar search can be efficiently performed when re-search by association feedback with only a little extra memory usage. One search session is defined as the initial query and the associated feedback process until the final search results are generated.

Using the above-described principle, the approximate distance A _p ( q _{1 ,} o _i ) of any i-th target object o _i in the rescanning process is the approximate distance value recorded in the previous searching process, PrevDist [ i ] and L _p. By calculating the difference of ( q _{0 ,} q ₁ ), it can be found quickly. Since the approximate distance A _p ( q _{1 ,} o _i ) of the i-th target object thus obtained is smaller than the actual distance L _p ( q _{1 ,} o _i ) _, A _p ( q _{If 1 ,} o _i )> similarity distance r, then L _p ( q _{1 ,} o _i )> similarity distance r is always satisfied.

Next, when the approximate distance A _p for the i th target object is smaller than the similarity distance r (step 230), the target object is inserted into the candidate search set C _rs (step 240). The approximate distance for the target object is stored in the approximate distance array PrevDist so that the approximate distance can be used for the next re-search (step 250). On the other hand, if not in step 230, the target object is excluded from the candidate search set.

Next, if the current target object is the last target object, the filtering step (step 150) is terminated. Otherwise, i is increased by one (step 270) and the above steps are repeated.

After the above-described filtering step (step 150) is completed, the correct search set is generated by calculating the actual distance between the target objects included in the candidate candidate search set filtered and the new query object filtered from the search result set (step 160). 4 is a flowchart illustrating the processing of the correct answer set generation step (step 160) in more detail. Hereinafter, the generation of the correct answer search set (step 160) will be described in more detail with reference to FIG. 4.

Referring to FIG. 4, target objects in the candidate search set C _rs are read (step 300). After initializing an arbitrary variable i to 1 (step 310), the actual distance between the i-th target object and the new query object in the candidate search set is calculated (step 320).

Next, if the actual distance ( L _p ( q _, o _i )) between the i th target object and the new query object is smaller than the similarity distance r (step 330), it is included in the correct answer search set A _rs . (step 340), and stores the approximate distance arrangement (PrevDist) for the calculation of the distance, and then the search (step 350). Next, if the current target object is the last target object in the candidate search set C _rs (step 360), it ends, otherwise i increments one (step 370) and returns to step 320 to repeat.

After the step of generating the correct answer set is completed (step 160), the correct answer set is provided as the final result (step 170). If the final result is satisfactory (step 180), the search ends, otherwise, the process returns to step 130 and the above-described process is repeated.

According to an efficient similar search method for content-based multimedia search including related feedback according to the present invention having the above-described configuration, a user may obtain a search result more than five times faster than the first search.

Although the present invention has been described above with reference to preferred embodiments thereof, this is merely an example and is not intended to limit the present invention, and those skilled in the art do not depart from the essential characteristics of the present invention. It will be appreciated that various modifications and applications which are not illustrated above in the scope are possible. And differences relating to such modifications and applications should be construed as being included in the scope of the invention defined in the appended claims.

An efficient similarity search method for content-based multimedia retrieval including the associated feedback according to the present invention uses a result calculated in a previous retrieval process to quickly search for a target object within a similar distance (r) with a new quality object. By using this method, it is possible to generate similar search results more than five times faster than the first search.

In order to show the efficiency of the search method according to the present invention, only the search method according to the present invention is applied without using any index method (hereinafter referred to as 'BFS') and HBI (J. Park) which is one of filtering-based index methods. and J. Nang, "A Hierarchical Bitmap Indexing Method for Content Based Multimedia Retrieval," Proceedings of EuroIMSA International Conf., pp.223-228, 2006.). As shown in Fig. 5, k-NN search and r-Range search are performed on four kinds of image data sets. 5 is a diagram showing the data sets used in the experiment. R1 and R2 consisted of MPEG-7 Color Structure descriptor and Edge Histogram descriptor extracted from 25,160 landscape image set provided by Berkeley University, and R3 consists of HSV Color Histogram extracted from 68,040 Corel image set. S1 papers [T. Bozkaya and M. Ozsoyoglu, “Distance based Indexing for High Dimensional Metric Spaces,” Proceedings of ACM SIGMOD Conf. on Management of Data, pp.357-368, 1997. For HBI, we used six bitmaps for R1, ten bitmaps for R2 and R3, and seven bitmaps for S1. For each similar search method, k = 10 (for k-NN search) and r (for r-Range search) are experimented to determine the average number of elements in the search result set to be 20 or less according to the characteristics of each data set. It was.

In order to use the previous search result in association feedback, the search session needs to be maintained. To do this, the connected socket is maintained until the user selects a new query. In addition, the server issued a session ID so that the query for re-search and the query for new search can be distinguished. The query for re-search selects one-third of the search results as an association object and uses Rocchio's method of <Equation 2> [I. Ruthven and M. Lalmas, “A Survey on the Use of Relevance Feedback for Information Access Systems,” The Knowledge Engineering Review, Vol. 18, Issue.2, pp.95-145, 2003.] (α = 0.5, β = 0.25, gamma = 0.25). All experiments were performed on a Microsoft Windows XP platform with an Intel Pentium4 (3.0GHz) CPU and 1GB of memory. In addition, according to the experimental results, the results of the two search methods were averaged and expressed because there was no significant difference between the results of the k-NN search and the r-Range search. Finally, in order to improve the reliability of the experimental results, each of 100 randomly selected queries in the data set was searched and the averaged results were selected as final results.

FIG. 6 shows the ratio and search of the object filtered using the previous distance calculation result without recalculation up to five times using the k-NN and r-Range search methods proposed for R1. These graphs show the time spent on. As shown in (a) of FIG. 6, since the HBI calculates an approximate distance for an object of about 80 to 90% in the initial search, it is natural that the filtering ratio is lower than that of the BFS when rescanning. In addition, in case of BFS, the filtering rate of the 3rd search drops by about 10-15%, and it remains almost constant in the 4 ~ 6th search. This is because the previous distance information used for filtering in the proposed search algorithm is mostly the first query. It can be thought that this is due to an error that occurs because the distance between query objects increases as the number of searches increases while the distance calculated for an object increases.

On the other hand, the HBI shows a relatively constant filtering rate regardless of the number of rescans because the error from the approximate distance is larger than the error. Therefore, it can be seen from FIG. 6 (b) that the search time of HBI takes slightly longer than that of BFS in the second search having a large difference in filtering ratio. However, as the number of rescans increases, the filtering rate of BFS decreases slightly, and HBI performs the second filtering process using indexes. Therefore, HBI shows faster search speed than BFS from the third search. Overall, when the filtering-based indexing method such as HBI is applied together, the filtering ratio decreases slightly, but the overall search time is reduced by the index effect. Therefore, even if the filtering ratio is slightly decreased due to the error caused by the approximate distance calculation, it is more efficient to use the index together.

7 is a graph illustrating a filtering ratio and a search time when the search method according to the present invention is applied to other data sets except R1 together with HBI. In the case of R3 and S1, as in the case of R1, the filtering ratio decreases as the number of rescans increases, but in R2, the filtering ratio increases slightly from the fourth search. This is because when the filtering ratio falls below a certain limit, the number of objects for which the actual distance is newly calculated increases, which reduces the error that increases as the number of rescans increases.

However, referring to FIG. 7 (b), it can be seen that the decrease in the search speed is not large compared to the decrease in the filtering ratio according to the increase in the number of re-searches. This is because many of the unfiltered objects in the one-step filtering process using triangle inequality are filtered by the HBI with a filtering rate of over 90%. As a result, it can be seen that the retrieval speed is increased by 5 times compared to BFS search, which is 4 times higher for R1, 3 times for R2, and 6 times for R3 and S1.

Claims (4)

In the content-based multimedia retrieval method including the associated feedback, (a) searching according to the first query object, generating a search result set, calculating actual distances between target objects constituting the search result set and the query object, and storing the result in the approximate distance array PrevDist; ; (b) performing associative feedback on the search result set and generating a new query object; (c) calculating approximate distances between the target objects constituting the search result set and the new query object using distance information stored in the approximate distance arrangement with respect to the search result set; (d) a filtering step of generating a candidate search set consisting of target objects whose approximate distance is smaller than a preset similarity distance r; (e) calculating an actual distance between the target objects constituting the candidate search set and the new query object, generating a correct answer search set including target objects whose actual distance is smaller than the similarity distance r; Storing the actual distances of the target objects constituting the correct search set in the approximate distance array PrevDist; Including, the multimedia search method for improving the speed of the multimedia search including the associated feedback. The method of claim 1, wherein the multimedia search method (f) repeats steps (b) to (e) when the answer answer set is not satisfactory, wherein the search result set is a final answer search set, The approximate distance arrangement consists of the actual distances to the target objects that make up the final correct search set. The method of claim 1, wherein step (c) (c1) reading distance information of each target object stored in the approximate distance arrangement and a previous query object with respect to the search result set; (c2) obtaining a distance between a previous query object and the new query object; And (c3) using the distance information read in the step (c1) and the distance between the query object obtained in the step (c2) and the new query object, each of the target objects constituting the search result set and the new; Calculating approximate distances to the query object; Multimedia search method characterized in that consisting of. The method of claim 3, In step (c3), the approximate distances between the target objects constituting the search result set and the new query object may include distance information read in step (c1) and the query obtained in step (c2). And calculating a difference value of a distance between an object and the new query object.

Images