CN104077344B - Interactive image retrieval method and system based on adaptive learning region importance - Google Patents

Abstract

The present invention relates to a kind of interactive image retrieval methods and system based on adaptive learning region importance, it is based on average drifting to the retrieval image that user submits and specification cuts into row adaptivenon-uniform sampling, extract the feature of cut zone, importance based on new region importance index zoning, and then the similarity of every piece image in the retrieval image and image library submitted based on general area matching primitives user, according to sequencing of similarity and return to user it is most like before several width images, feedback information based on user, the short-term study of structure and Term Learning strategy, reduce the region importance of counter-example image while being intended to automatically update the region importance of positive example, to quickly, effectively obtain satisfied image.The present invention can mitigate the retrieval burden of user, and the positive example and counter-example image information of study user annotation carry out the importance of automatic update area during retrieval, can be intended to closer to the retrieval of user, to more efficiently improve retrieval performance.

Description

Interactive image retrieval method and system based on adaptive learning region importance

Technical field

The invention belongs to field of image search, more particularly to a kind of interactive image retrieval method and system.

Background technology

With the fast development of multimedia and Internet technology, people touch more and more various information.Image is made For a kind of abundant in content and intuitive multimedia messages of performance, have been favored by people for a long time.It is how quickly and effective Search oneself needs information --- the 1990s content-based image retrieval (Content Based Image Retrieval, CBIR) it comes into being, image retrieval is inquired into from visualization angle.So-called CBIR exactly passes through extraction The features such as the low-level image feature of image, such as color, texture and shape indicate picture material, pass through the similarity system design between feature Complete the matching between image.

Description to picture material includes global description's and local description.Global description's be to image it is whole into The feature of row description extraction, robustness is stronger, affected by noise smaller.However, according to human eye visual perception characteristic, Yong Hujing Some normal being concerned in image or certain target areas, such as a vehicle in image.Because being carried to entire image The feature taken can not preferably express the characteristics of target area.For global description's, local description is to single The description of target is more strong.

Region based CBIR (Region Based Image Retrieval, RBIR) by by graphical representation at Region overcomes the difficulty that global description's subband comes to a certain extent.One RBIR system uses image Segmentation Technology will first Then image segmentation converts the matching of image to the matching between region at several regions.Image table based on regional level Show closer to human eye vision.The Netra systems of UCSB research and development and the Blobworld systems of Berkeley University's research and development are typical RBIR systems, the two systems need user to submit piece image, and relevant range and specified spy are selected from the region of segmentation Levy weight.Due to the complexity of image itself and the inaccuracy of image segmentation, target area is extracted automatically and accurately An or problem.Therefore, how user usually can be because of selecting to be at a loss comprising mesh target area.In order to be supplied to One simple search interface of user, the matching of one-to-many Regional Similarity are suggested.This similarity is related to two images All areas, do not need user and select to participate in matched region, it is only necessary to which user submits a width to retrieve image, subtracts significantly The retrieval burden of user is lacked.The SIMPLIcity systems of Stanford University propose a kind of general area matching (Integrated Region Matching, IRM) method completes images match.IRM allows an area in piece image Multiple regions in domain and another piece image are matched, therefore reduce the influence brought due to the inaccuracy of image segmentation. However, characterizing the importance in region with the area in region in IRM, i.e. region area is bigger, then importance is bigger.Disadvantage exists Assume do not have generality in this.

In general, the low-level image feature for the image that system automatically extracts and the high-level semantic (subjective assessment of user) of image Between there is huge difference (i.e. semantic gap) so that retrieval result is unsatisfactory.In order to efficiently solve The above problem, relevant feedback (Relevance Feedback, RF) technology are introduced into, and produce the figure based on Relevance Feedback Algorithms As retrieval technique.Attempt to establish being associated between the low-level image feature of image and high-level semantic by Relevance Feedback, thus The recall precision of entire searching system is improved after human-computer interaction.RF is used in information retrieval at first, is introduced the 1990s CBIR has been proved that retrieval performance can be effectively improved by numerous researchers.RF strategies are after providing initial retrieval result It is required that user's mark positive example and counter-example image (being known as feedback information), then system learns the retrieval of user according to feedback information and anticipates Figure is to return to the image being intended to closer to user search.

RF is introduced into RBIR by the IDQS systems that Bristol universities propose, system requirements user selects retrieval exemplary plot The area-of-interest of picture, system match image to the mode in region according to region and are then back to most like a series of images.It connects Get off, needs user to indicate that the image of return is positive example or counter-example, Learning vector quantization methods are used The region of feedback image is clustered, these classifications are marked as positive example classification or counter-example classification and the close figure of positive example classification As being returned to user, by iteration until user is satisfied with retrieval result.In IDQS systems, the similarity measurement of use It is the similarity in one-to-one region, is largely influenced by segmentation result and aggravated the negative of user-selected area Load.Feng Jing et al. are according to field feedback come the importance of update area.They are important region at basic assumption The number occurred in positive example image is more than other.So in feedback cycle each time, the only weight in positive example image region The property wanted increases.Then, effect of the counter-example played in feedback searching is had ignored.In fact, increasing positive example image region weight The importance of counter-example image-region is reduced while the property wanted can more effectively improve retrieval performance.

Invention content

The present invention is directed to overcome above-mentioned the deficiencies in the prior art, it is proposed that a kind of interaction of adaptive learning region importance Formula image search method can fast and effeciently retrieve the associated picture more close with user view.

The present invention uses following technical proposals：

A kind of interactive image retrieval method based on adaptive learning region importance, includes the following steps：

1) adaptivenon-uniform sampling is carried out to the retrieval image Q that user submits, obtains cut zone Q={ Q₁,…,Q_m, m here To divide obtained areal；

2) to cut zone Q_i, the 1≤i≤local low-level image feature of m extractions, including color, texture and shape feature；

3) zoning Q_i, region importance (RI) index of 1≤i≤m；

4) according to RI indexs, every piece image B in Q and standard picture library is calculated_l, the similarity S of 1≤l≤N_l,1≤l ≤ N, wherein N are the amount of images in standard picture library；

5) according to S_lTo B_lIt is ranked up, and returns to the most like preceding several width images of user；

6) user participates in feedback, until retrieving satisfied image.

The present invention also provides a kind of image indexing systems, comprise the following modules：

Import image module so that user selects to want the image of retrieval from local file；

Divide module, to show the segmentation result of image；

Weight selecting module has system recommendation and user independently to fill in two kinds of selections；

Show retrieval result module, the display preceding N width image most like with the retrieval image submitted；

Labeling module, the image labeling positive example or counter-example that user returns to system；

Feedback module so that user can select to meet the positive example image that oneself retrieval is intended to.

Beneficial effects of the present invention：The retrieval burden of user is reduced, and makes full use of user's during retrieval Feedback information automatically updates positive example and the region importance of counter-example image by learning in short term and Term Learning, to quickly, Effectively it is intended to close to the retrieval of user.

Description of the drawings

Fig. 1 is the flow chart of the present invention.

Fig. 2 is the systematic schematic diagram of the present invention.

Fig. 3 obtains the image retrieval interface of initial retrieval result.

Fig. 4 user is labeled initial retrieval result.

Feedback searching result of Fig. 5.

Specific implementation mode

The specific implementation of the present invention is as follows：

Step 1: carrying out adaptivenon-uniform sampling to the retrieval image Q that user submits, cut zone Q={ Q are obtained₁,…,Q_m}。

It is two kinds of common image partition methods that average drifting (MS) and specification, which cut (NC), but MS is also easy to produce over-segmentation, NC computation complexities are too high.W.B.Tao etc. combines MS and NC, it is proposed that a kind of new image partition method, MS-Ncut should Method is first split image with MS dividing methods；Then with the side NC in the image basis of the obtained over-segmentation of back Method carries out region merging technique, alleviates over-segmentation and computation complexity to a certain extent.The problem of MS-Ncut methods, is to need Segmentation number is pre-set to terminate merging process, this undoubtedly increases the burden of user.Based on this, it is proposed that a kind of adaptive MS-Ncut dividing methods can automatically determine final segmentation number according to the statistical property of image itself.It is as follows：

Image is carried out MS segmentations by (1a)；

(1b) in view of figure it is theoretical in scheme the definition of G, G=(V, E), wherein V are the vertex of figure, E be vertex and vertex it Between weight.Image after MS is divided regards the figure G in figure theory as, using cut zone as the node V of figure, to figure NC Carry out Cluster merging；

The average color of (1c) in H*S*V color spaces to each extracted region after cluster per channel is as the region 3 dimensional features；

(1d) clusters number d is initialized as 2；

(1e) randomly chooses the feature vector in d region as initial category center, all areas is integrated into nearest In classification, and recalculate class center；

(1f) calculation criterion functionWherein ω₁,…,ω_KIt is K classification, m_sIt is class center, F_tIt is to belong to ω_tThe feature vector in the region of classification, if y be more than or equal to preset threshold epsilon, K=K+1 go to (1e) and (1f), otherwise stops iteration.

Step 2: to cut zone Q_i, the 1≤i≤local low-level image feature of m extractions, including color, texture and shape feature.

Color characteristic：Image is transformed into the spaces L*u*v* by RGB color, extracts L, u, the regions v average color is made 3 for each region tie up color characteristic；

Shape feature：Region 1 dimension density ratio, 2 dimension barycenter, 4 dimension rectangular box, 7 dimension not bending moment as 14 Wei Xingzhuante Sign；

Textural characteristics：The co-occurrence matrix of zoning, four extraction energy, inertia, entropy, evenness statistical properties are as 16 dimensions Textural characteristics.

Step 3: zoning Q_i, region importance (Region importance) RI indexs of 1≤i≤m；

According to the vision system of human eye, the different zones importance in piece image is different.In general, region importance It is related with the area in region.The area in region is bigger, and region is more important.Moreover, important region is often positioned in the center of image Position.Based on assumed above, it is contemplated that region area and regional location, we construct a kind of new RI indexs：If image I tables It is shown as I={ r₁,r₂,…,r_n, wherein r_iIt is the ith zone of image I, n indicates the number in region, then r_iRI be

Wherein A (I) and A (r_i) it is respectively image I and region r_iArea, i.e. the number of pixel, then A (r_i)/A (I) is indicated For region r_iThe shared percentage in image I；(r_ix,r_iy) it is r_iCenter of gravity, (x, y), L (I) and H (I) indicate image I respectively Center of gravity, length and height.Indicate r_iCenter of gravity and I center Euclidean distance.Formula (1) shows：RI (r_i) value is bigger, region r_iIt is more important.

Step 4: according to RI indexs, every piece image B in Q and standard picture library is calculated_l, the similarity S of 1≤l≤N_l, 1≤l≤N, wherein N are the amount of images in standard picture library；

Assuming that two images I₁And I₂It is expressed as I₁={ r_1,1,r_1,2,…,r_1,mAnd I₂={ r_2,1,r_2,2,…, r_2,n, wherein r_1,i、r_2,jImage I is indicated respectively₁And I₂Divide obtained region, m and n indicate the number in region, due to two width figures It is not necessarily the same as dividing obtained areal, without loss of generality, is indicated here using different parameters.RI based on proposition Definition, then calculate I₁And I₂Distance definition be：

Wherein P=(P_i,j)_m×nReferred to as importance matrix, P_i,jIndicate r_1,iAnd r_2,jMatched important coefficient.r_1,i and r_2,jDistance d_i,jIt indicates.Herein, d_i,jIt is defined as：

WhereinIndicate corresponding intrinsic dimensionality, f_i={ f_i,1,…,f_i,33And f_j={ f_j,1,…,f_j,33It is region r_1,i And r_2,jNormalization characteristic vector, i.e., the result that the 33-D feature vectors introduced in step 2 pass through Gaussian normalization.ω_color, ω_texture,ω_shapeIt is the weight of color, texture and shape feature respectively, ω is obtained by experimental check_color=5/8, ω_texture=1/8, ω_shape=2/8.It is in fact possible to by obtaining adaptive weighting with regular hour cost.

Assuming that region r_1,iAnd r_2,jRegion importance RI be RI (r_1,i) and RI (r_2,j), then it needs to meet

Based on (4) and it is proposed that RI indexs, P=(P_i,j)_m×nMost like priority highest can be passed through (MostSimilar Highest Priority,MSHP^[24]) criterion obtains.MSHP criterion attempt to assign more similar region pair Higher matching importance.It is different with the MSHP criterion used in IRM, the region importance in MSHP here be it is proposed that RI indexs.

Step 5: according to S_l, 1≤l≤N is to B_lIt is ranked up, and returns to the most like preceding M width image of user；According to (2) Sequence obtains from small to largeWherein { i₁,…,i_NIt is setting for { 1 ..., N } It changes.Then the image in standard picture library according toBe ranked sequentially and before returning M image to user.

Step 6: user participates in feedback, until retrieving satisfied image.Realize the specific steps are：

(6a) stops retrieving if user is satisfied with retrieval result；Otherwise it follows the steps below

The retrieval result that (6b) user annotation system returns is positive example or counter-example；

(6c) adaptively provides two kinds of similarity thresholds, and by learning to automatically update positive example and anti-with Term Learning in short term The RI values of example image-region, including following procedure：

(6d) provides the definition of similarity threshold first：Given 0≤T≤1, if met

Then T is referred to as similarity threshold.HereIndicate in kth time feedback user annotation it is all just Example and counter-example image；s(R,I_i) indicate region R and image I_iSimilarity, be defined as follows：

s(R,I_i)=max (s (R, r_i,j)),

Wherein s (R, r_i,j) it is region R and image I_iJ-th of region r_i,jSimilarity, be taken as Euclidean distance here Negative exponential function, i.e. s (R, r_i,j)=exp (- d (R, r_i,j)), wherein d (R, r_i,j) it is region R and r_i,jEuclidean distance.It is similar Spending threshold value, we obtain by the way that following strategy is adaptive：

(6c-1-1) constructs similarity matrix P, that is, calculates the region for all positive example images that user submits in kth time feedback The matrix formed with the similarity of all positive and negative illustration pictures.The feature being characterized in due to region after Gauss standard, To the value of similarity belong to [0,1]；

(6c-1-2) calculates the histogram H of similarity matrix P, i.e., [0,1] region is divided into 10 parts, obtains minizoneI=1,2 ..., 10, statistics falls into the element number in the P of each minizone, obtains H (i), i=1,2 ..., and 10；

(6c-1-3) calculates cumulative histogram CH, i.e. CH (i)=CH (i-1)+H (i), i=1,2 ... of H, 10, wherein CH (0)=0；

Then T₁It is defined as：

Similarity threshold T₂It similar can obtain.

The region R of positive example image when (6c-2) k feedback_iWith the region r of counter-example image_iRegion importance RI update For RI_i(k) and rJ_i(k)：

WhereinIt is confactor, T₁And T₂It is similarity Threshold value, and 0≤T₁,T₂≤ 1, n and m are the number of regions of a positive example image and a counter-example image respectively.

If we further assume that retrieval intention of the user in retrieving does not change, then it can be by cumulative each The RI in the positive example region of secondary feedback is to improve retrieval effectiveness.K feedback rear region R_iCumulative RI be defined as：

CRI_i(k)=[CRI_i(k-1)+RI_i(k)]/k, (8)

Wherein RI_i(k) when being the kth time feedback defined by (7) formula positive example image region R_iUpdated RI.When k=1, CRI_i(0) it is defined as the Initial R I in the region that formula (1) defines.

Therefore, the region importance RI for the image that user returns can be automatically updated by formula (7) and (8).

(6d) recalculates the Q and B after update RI values using formula (2)_l, the similarity of 1≤l≤N, and export preceding M Most like image.

(6b)-(6d) is repeated in (6e), until retrieving customer satisfaction system image.

The present invention also provides a kind of searching system based on above-mentioned search method, Fig. 2 gives the structure of the searching system Module, including：Import image module, segmentation module, weight selecting module, display retrieval result module, labeling module and feedback Module.

Import image module so that user selects to want the image of retrieval from local file；

Divide module, to show the segmentation result of image；

Weight selecting module has system recommendation and user independently to fill in two selections, wherein " system recommendation " option indicates The weight of the color, texture and the shape feature that are used when retrieval is the weight obtained by experiment in formula (3), i.e. ω_color= 5/8, ω_texture=1/8, ω_shape=2/8, wherein ω_color,ω_texture,ω_shapeIt is color, texture and shape feature respectively Weight.

Show retrieval result module, which shows retrieves the most like preceding 9 width image of image with what is submitted；

Labeling module, the mark positive example and counter-example that user returns to system；

Feedback module, which, which allows a user to select, meets the positive example image that oneself retrieval is intended to.

The method and system of the present invention give further displaying with emulation experiment, we are in Corel-1000 image libraries The retrieval image that piece image 0.jpg is submitted as user, simulation result are shown in that Fig. 3-Fig. 5 wherein upper left corners Fig. 3 displays are used Family selects 0.jpg as retrieval image from image library, and selects the feature weight of " system recommendation ", on the right side of searching system Region retrieval result indicating template obtains 9 most like width images.Fig. 4 indicates that user meets what retrieval was intended in labeling module selection Positive example image, non-selected image are defaulted as counter-example image, then click feedback module and submit feedback information.Fig. 5 is system The image to reorder that feedback information by learning user returns, the method for study is using feedback searching side of the invention Method.Fig. 4 and Fig. 5 shows that image search method of the invention has preferable initial retrieval result and effective feedback result.

To sum up, the image search method described through the invention can effectively mitigate burden for users and can lead to The feedback for crossing user effectively promotes retrieval performance.Searching system interface based on the search method is easy to learn, is suitble to Non-professional ordinary user comparatively fast adapts to the system.

By above-mentioned specific implementation method as it can be seen that the present invention：1) a kind of interactive image inspection for having user to participate in is proposed Suo Fangfa, what this method was realized by constantly updating the importance in region：It is carried first according to the visual signature of retrieval image itself A kind of region importance index, i.e. formula (1) are gone out, have then been used according to the feedback information of user in the interaction stage of user Short-term study and Term Learning automatically update positive example and the region importance of counter-example image, i.e. formula (7) and formula (8), in turn Matching is re-started using formula (2), the retrieval to fast and effeciently move closer to user is intended to.2) according to the search method Devise corresponding search interface, it is easy to operate, be easily achieved.

Claims (5)

1. a kind of interactive image retrieval method based on adaptive learning region importance, which is characterized in that including walking as follows Suddenly：

1) adaptivenon-uniform sampling is carried out to the retrieval image Q that user submits, obtains cut zone Q={ Q₁,…,Q_m, m is point here The areal cut；

2) to cut zone Q_i, the 1≤i≤local low-level image feature of m extractions, including color, texture and shape feature；

3) the index RI that a characterization region importance is proposed by the area and position that consider region, for calculating area Domain Q_i, the RI of 1≤i≤m；Moreover, in subsequent feedback procedure, RI can be according to the feedback information of user by learning in short term It is automatically updated with Term Learning；

If image I is expressed as I={ r₁,r₂,...r_m, wherein r_iIt is the ith zone of image I, m indicates the number in region, then r_i RI be

Wherein A (I) and A (r_i) it is respectively image I and region r_iArea, i.e. the number of pixel, then A (r_i)/A (I) is expressed as area Domain r_iThe shared percentage in image I；(r_ix,r_iy) it is r_iCenter, (x, y), L (I) and H (I) are indicated in image I respectively The heart, length and height；Indicate r_iCenter and I center Euclidean distance；Formula (1) shows：RI(r_i) value It is bigger, region r_iIt is more important；

4) according to RI indexs, every piece image B in Q and standard picture library is calculated_lSimilarity S_l, 1≤l≤N, wherein N are marks Amount of images in quasi- image library；

Assuming that two images I₁And I₂It is expressed as I₁={ r_1,1,r_1,2,…,r_1,mAnd I₂={ r_2,1,r_2,2,…,r_2,n, wherein r_1,i、r_2,jImage I is indicated respectively₁And I₂Divide obtained region, m and n indicate the number in region；

RI definition based on proposition, then calculate I₁And I₂Distance definition be：

Wherein P=(P_i,j)_m×nReferred to as importance matrix, P_i,jIndicate r_1,iAnd r_2,jMatched important coefficient；r_1,i and r_2,j Distance d_i,jIt indicates, d_i,jIt is defined as：

WhereinIndicate corresponding intrinsic dimensionality, f_i={ f_i,1,…,f_i,33And f_j={ f_j,1,…,f_j,33It is region r_1,iAnd r_2,j Normalization characteristic vector, i.e., the result that the 33-D feature vectors introduced in step 2) pass through Gaussian normalization；ω_color, ω_texture,ω_shapeIt is the weight of color, texture and shape feature respectively；

Assuming that region r_1,iAnd r_2,jRegion importance RI be RI (r_1,i) and RI (r_2,j), then it needs to meet

Based on (4) and RI indexs, P=(P_i,j)_m×nIt is obtained by most like priority highest criterion；

5) according to S_l, 1≤l≤N is to B_lIt is ranked up, and returns to the most like preceding several width images of user；

6) by the feedback information of user, by learning to automatically update the RI of positive example and counter-example image in short term with Term Learning, directly To retrieving customer satisfaction system image.

2. image search method according to claim 1, which is characterized in that the step 1) carries out as follows：

Image is carried out average drifting (Mean Shift, MS) and divided by (1a)；

(1b) in view of the definition of figure G in figure theory, G=(V, E), wherein V are the vertex of figure, and E is between vertex and vertex Weight regards the image as figure G, the node V by the obtained cut zone in (1a) as figure in figure theory, uses figure Specification cuts (Normalized Cuts, NC) and carries out Cluster merging；

3 dimensions of average color as the region of (1c) in H*S*V color spaces to each extracted region after cluster per channel Feature；

(1d) clusters number d is initialized as 2；

(1e) randomly chooses the feature vector in d region as initial category center, and all areas are integrated into nearest classification In, and recalculate class center；

(1f) calculation criterion functionWherein ω₁,…,ω_dIt is d classification, m_sIt is class center, F_tIt is Belong to ω_tThe feature vector in the region of classification, if y be more than or equal to preset threshold epsilon, d=d+1 go to (1e) and (1f), otherwise stops iteration.

3. image search method according to claim 1, which is characterized in that wherein step 2) carries out as follows： Local low-level image feature, including color, texture and shape feature are extracted to cut zone；Color characteristic：By image by RGB color sky Between be transformed into the spaces L*u*v*, extract L, u, the regions v average color ties up color characteristics as the 3 of each region；Shape feature： The 1 dimension density ratio in region, 2 dimension barycenter, 4 dimension rectangular box, bending moment does not tie up shape features to 7 dimensions as 14；Textural characteristics：Calculate area The co-occurrence matrix in domain, four extraction energy, inertia, entropy, evenness statistical properties are as 16 dimension textural characteristics.

4. image search method according to claim 1, which is characterized in that wherein step 3) carries out as follows：

If 6a) user is satisfied with retrieval result, stop retrieving；Otherwise

6b) retrieval result that user annotation system returns is positive example or counter-example；

It 6c) is based on short-term study and Term Learning, automatically updates positive example and the RI values of counter-example image-region；

6d) recalculate Q and B with newer RI values_l, the similarity of 1≤l≤N, and export preceding several most like images；

6b 6e) is repeated) -6d), until retrieving the image for enabling user satisfied.

5. implementing a kind of Interactive Image Retrieval Systems of claim 1 the method, comprise the following modules：

Import image module so that user selects to want the image of retrieval from local file；

Divide module, to show the segmentation result of image；

Weight selecting module has system recommendation and user independently to fill in two selections；

Show retrieval result module, the display preceding several width images most like with the retrieval image that user submits；

Labeling module, the image labeling positive example or counter-example that user returns to system；

Feedback module so that user can select to meet the positive example image that oneself retrieval is intended to.