Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Definitions

• the object of the present invention relates to an image search for finding a visual similarity between images contained in an image database and at least one request image.
• This similarity search is usually carried out by a search engine or indexing engine running on a processor, the images being typically stored in a digital memory, and a terminal makes it possible to present the result to a user of the search method, the latter can also intervene in the process via interfaces (keyboard, mouse, etc.).
• An object of the invention is to try to take into account, in an automatic image search, the subjectivity of the user vis-à-vis the notion of visual perception when it seeks a similarity between images and a request image.
• the main difficulty lies in the fact that the image search algorithms (or others) being deterministic, they always converge towards the same set of results from the same query, whereas a user who uses his subjectivity for comparing images gives a result that can be different from another user.
• a tumor search engine in a medical imaging application will be able to execute a search in a fully automatic way since the subjectivity has only very little space, whereas the classification of images of holidays can make to intervene more subjectivity if the request is generalist.
• any attempt to calculate deterministic visual similarity is therefore doomed to failure more or less depending on the relevance of image comparison processes.
• Di Di, ti (Ii, I 2 ) #D lrt 2 (Ii, I 2 ), where Di ; t i corresponds to the similarity perceived by the user Ui at a time ti, to take into account the fact that the same user can modify his notion of visual similarity over time.
• This example shows the complexity of precisely simulating this notion.
• the user has no idea of the statistical distribution of the signatures in the image database and can not therefore take it into account when setting the parameters;
• a conventional image search with relevance looping includes:
• a first method is that the user selects, among the response images, the images that seem to him to best correspond to his request (see, for example, Y. chen et al., "One-Class SVM for Learning in Image Retrieval", in IEEE International Conference on Image Processing, Thessaloniki, Greece 2001).
• a second method it will be able to contrario specify those which it considers irrelevant (see for example Y. Rui et al., "Relevance Feddback: A Power Tool for Interactive Content-Based Image Retrieval", in Storage and Retrieval for Image. Video Databases (SPIE) pages 25-36, 1998).
• a third step 3 of closure of relevance Since the ways of directing the query are very intuitive to the user, they allow the application to more precisely direct the search during the next relevancy loop.
• the goal of a relevance looping algorithm is to make the best use of the user's feedback to, in a sense, model its subjectivity.
• the relevance loop must therefore allow the application to get closer to the ideal image that is supposed to represent what the user wants.
• Qo the initial request image and Q 0 its signature (or its visual characteristics defined by a set of determined descriptors) in the descriptor space.
• a space descriptors is defined by axes each giving the importance of one of the descriptors determined in an image, the images being generally positioned in this space.
• I p 'and / 1 respectively relevant and irrelevant images specified by the user will be noted.
• a first type of closure of known relevance is that implemented by the Rocchio algorithm (J. Rocchio "Relevance Feedback in Retrieval Information, pages 313-323", in The Smart Retrieval System-Experiments in Automatic Document Processing. , prentice-hall edition, 1971).
• a second type of looping of known relevance is based on a reweighting algorithm, also called the standard deviation method. For example, reference may be made to "Image Retrieval by Examples” by R. Brunelli and O. Mich (IEEE Transactions on Multimedia, 2 (3): 164-171, 2000).
• a third type of closure of known relevance is based on probabilistic models.
• the probability calculation includes the calculation of the following function:
• a second probabilistic model is the Bayesian decision model (see, for example, "Relevance Feedback and Category Search in Image Databases" by C. Meilhac et al in IEEE International Conference on Multimedia Computing and Systems, Florence, Italy, June 1999). categorizes the whole database into two classes: relevant or irrelevant.
• a third probabilistic model is based on Support Vectors Machines (SVM). It is a question here of carrying out the closure of relevance by a classification type approach. We try to separate the base into two groups: relevant images and irrelevant images.
• SVM Support Vectors Machines
• SVM Support Vector Machines
• the algorithm is therefore used as a classifier.
• the user by choosing relevant images (see “One-Class SVM for Learning in Image Retrieval" from
• the Rocchio and re-weighting techniques make a strong assumption: similar images for the user are relatively close in the descriptor space. However, to make this hypothesis requires to have in its possession descriptors perfectly reflecting the human perception, which is never the case. On the other hand, the reweighting is generally done by favoring a direction in the space of the descriptors, that is to say a particular descriptor.
• Bayesian methods and SVM-based methods classify images in the descriptor space. As such, they are heavy learning methods in computational complexity.
• a first objective of the present invention is to achieve a loop of relevance in the context of a similarity search between images and at least one request image, which is not very complex to implement.
• a second objective of the invention is to achieve a relevance looping by means of a non-parametric method, which has no influence on the space of the descriptors or the distances between images.
• a third objective of the invention is the taking into account, during the closure of relevance, negative feedback from the user (i.e. irrelevant returns of images presented to him).
• a fourth objective of the invention is a measurement taken into account, by the algorithm, of the user returns of previous iterations. In particular, the algorithm will take into account to a certain extent possible changes in the user's choice during the search phase.
• a fifth objective of the invention is to have an intelligent presentation of the images retained to the user, so as to have a presentation more relevant than a simple presentation of a list of images.
• the invention achieves these objectives by proposing, according to a first aspect, an image search method for finding a visual similarity between images contained in the image database and at least one request image, the images having a determined signature ( or a set of determined descriptors), elements of the images and at least one element of the request image being positioned in a descriptor space defined by axes each giving the importance of one of the descriptors determined in an element image, characterized in that it comprises the iterative implementation of the following steps: (a) user evaluation of a visual relevance or a visual irrelevance of at least one of a plurality of images. images presented to it, compared to the image request;
• said elements of the images are objects of the images, each image being composed of a plurality of determined objects, and the step (b) further comprises a last operation consisting of a summation of the values of relevance. (previously calculated) of the different objects making up the image considered, thus affecting each image the value of relevance sought for the current iteration;
• the influence field calculated during step (b) has a positive value; In the case where an image is evaluated during step (a) as being relevant, the influence field calculated during step (b) has a negative value.
• Step (b) furthermore comprises the summation, for each pixel, of the relevance values of the current iteration with values of relevance of previous iterations;
• Step (b) further includes, prior to the summation operation of the relevance values of the current iteration with relevance values of previous iterations, a weighting operation, for each pixel, of values of relevance so as to attenuate all their influence on the result of this summation that they come from old iterations;
• the weighting of the relevance values assigned to each element of the request image is different from the weighting of the relevance values assigned to each element of the other images, in that their respective influence on the result of the summation operation is less attenuated according to their seniority;
• Step (b) further comprises a weighting step which assigns a different weight to the influence fields according to whether the associated image has been evaluated in step (a) as relevant or irrelevant;
• step (a) the user furthermore gives a level of relevance or non-relevance to each image that he evaluates, and in that each field of influence calculated during step (b) ) is all the more extensive as this level of relevance or irrelevance is, in absolute value, high;
• step (c) The different images selected during step (c) are presented to the user in an order taking into account the relevance values assigned to them in step (b);
• the method further comprises, prior to the iteration steps, an automatic evaluation of a visual similarity of different images with the request image; and a selection of a determined number of evaluated images as being the most similar with the request image, these evaluated images then being the images presented in step (a).
• the invention proposes a device implementing said method with or without the characteristics previously listed. Also, the invention proposes a computer program comprising coding means for implementing the proposed method.
• FIG. 1 very broadly shows the various steps of an image search method including relevance looping.
• FIG. 2 represents the evolution over time (or during iterations) of the image search region in the descriptor space chosen as a framework for the implementation of the method according to the invention.
• Figures 3 and 4 show an example of implementation of an image search method according to the invention, in the case where the return of the user is positive.
• Figure 3 is a graphical representation of images in a space of 2-dimensional descriptors.
• FIG. 5 represents an exemplary implementation of an image search method according to the invention, in the case where the user's feedback is negative, in a graphic representation of images in a space of the descriptors in 2 dimensions.
• FIG. 6 represents the synthesis of experimental results showing the influence of the nature of the returns (negative and / or positive) of the user on the relevance obtained by the method according to the invention.
• FIG. 7 represents the synthesis of experimental results showing the influence of the change of objective of the user on the relevance obtained by the method according to the invention.
• the images are stored in an image database.
• This image database can be divided into image sub-bases each defining a group of images for a given terrain truth.
• the images or image objects (also generically referred to as "image elements") according to the invention have a specific signature, that is to say that in other words they are described by a set of specific descriptors. .
• image elements are positioned in a space descriptors defined by axes each giving the importance of one of the descriptors determined in a pixel.
• the image elements are thus represented by points in the space of the descriptors, each thus having a position characterizing the signature of the image element considered in the space of the descriptors used (see, for example, FIG. 2).
• the method according to the invention advantageously comprises the following steps, implemented iteratively, until a satisfactory or presumed satisfactory result is obtained: (a) a user's evaluation of a visual relevance or a visual non-relevance at least one of a plurality of images presented to it, with respect to at least one request image;
• step (a) the user is therefore presented, for example on a screen-type display terminal, a number of images to which he must assign a value corresponding to his judgment as to the relevance of answers presented to him.
• a step (a) (consisting of a user intervention in the search loop) will be chosen during which the user will have the choice between declaring a relevant image or a non-image. relevant.
• the user will assign a positive value in case of relevance and negative in the case of irrelevance.
• the invention provides a refinement of the type of choice given to the user, it may also give a level of relevance or non-relevance to each image that it evaluates.
• the relevance step (b) will take into account the relevance assessment of some of the images that are presented to the user to influence the relevance of all the images in the database or sub-base of images considered.
• step (b) directly involves an action by the user who expects an instantaneous return.
• this point is far from being trivial and can quickly lead to practical impossibilities.
• the relevance looping step (b) comprises a calculation of a relevance value assigned to each image, comprising: a calculation of an influence field extending around each element of each image evaluated by the in step (a), so that the absolute value of this influence field decreases as we move away, in the descriptor space, from the evaluated image element considered ; For each image element, a summation of the values of the different fields of influence felt by the image element under consideration, thus affecting each image element a value of relevance for the current iteration.
• relevance looping should be seen as a complementary process to traditional image retrieval. For this, it can act as an independent part in a larger process.
• the invention can thus cause a split of the originally unique search space (centered around the request image) into several (non-related) search spaces, if two elements at one stage of the search are designated as relevant but are distant in the space of the descriptors, thus causing a multi-modal partitioning of the descriptor space.
• N per t be the total number of images designated as relevant by the user and N- rt the total number of negative returns (ie irrelevant images) designated by this same user.
• V, (t ⁇ 0) ⁇ Q .e " ⁇ a > where X Q is a weighting assigned to the query image Q.
• V (i) represents the simple similarity value of the index image i with respect to the request image Q. From this moment, the user has the possibility of designating within the together E pre s. images that he considers relevant or not, before relaunching the search.
• V t (t) ie E tol.
• V t (t) ⁇ ⁇ .e o "-i TM fi.a + V ⁇ , ⁇ - h .e o - * ⁇ i ( ⁇ " ⁇ ) > - ⁇ W e ⁇ > (1)
• ⁇ ⁇ and ⁇ Nk are the respective weights of the images which have respectively been evaluated by the user as relevant and irrelevant.
• a level of relevance for example VA, 3 A , and - 4/4 for three images submitted to it as part of a 4-point relevance rating
• new weighting factors can be introduced for each level of relevance so that the most high in absolute value are the most influential in the final result.
• the images or objects proposed to the user will be the most distant images of the zones created by the irrelevant objects.
• the algorithm in this case does not predict a relevant image, but rather the set of "least irrelevant" images.
• Each picture inserted in E tot . creates a zone or field of influence around its position in the descriptor space. This influence is either positive in the case of a relevant image, or negative in the case of an irrelevant image.
• the calculation of the N pres . new images presented to the user will depend on the topology of the zone of influence created by the summation of the zones associated with the set of images found in E.
• the calculation of the relevance value associated with the index image i then depends on the set of images of E per t. assigned a positive coefficient and images of the set E- ( assigned a negative coefficient reflecting the irrelevance of this group.
• the lifetime associated with an image of E decreases by one unit. When it reaches zero, it is removed from the list.
• the image query continues to play the role of a relevant return.
• the lifetime ⁇ Q (t) can then be different from that of each of the other images of the base or the sub-base.
• the lifetime ⁇ ç (t) of the request image may be greater than the lifetime ⁇ of each of the other images of the database or sub-base, by the specific character of the request image.
• immediate-term memory is defined as opposed to: - a short-term memory, taking into account only the last loop of relevance, as is regularly the case in search engines;
• an image will play a role only temporarily and thus allow the user to change his choice during the image search phase. Indeed, this relevance duration assigned to the images thus affects a learning inertia to the indexing engine that takes into account this possible change of direction of the user. Indeed, in our case, an image designated as relevant at time t may not be at time t + ⁇ , or even in the extreme case become undesirable.
• step (a) the user, during step (a), evaluated the image I p i as being relevant.
• the consequence of the relevance loop is a stretch of the area of influence to the position of I p i in the descriptor space.
• step (a) the user, during step (a), evaluated the images I p2 and I p3 as relevant.
• the consequence of the relevance loop is a stretch of the area of influence to the positions of I p2 and I p3 in the descriptor space.
• step (a) confirms its assessment of the 3rd iteration (relevance of images I and I p2 p3). Finally, we obtain a zone of influence centered on the images I p2 and I p3 , representative of the similarity of images with respect to the request image Q in the meaning understood by the user.
• step (c) of the method according to the invention consists of a selection, by the indexing engine, of the images presenting the values of
• the presentation of the images thus selected is not made randomly, but is presented in a specific order.
• this way of operating can have advantages such as: direct the user more quickly to satisfactory images; - decrease the influence of the neighboring images presented to the user on his choice.
• the notion of similarity is indeed also relative to its environment. The user can indeed designate a first image as being relevant when it is surrounded by certain images, and may be designated as irrelevant by the same user in another context.
• a variant of the invention consists in positioning the images in the space of the descriptors, but the objects of which these images are composed. This relationship is particularly interesting in the context of a process of relevance looping, the notion of similarity between two images being intimately linked to the similarity of the different objects that compose it.
• the relevance looping is therefore the ideal step to link objects and global images. For this purpose, each time the user selects a relevant image P ⁇ in the image space, all the objects composing this image will be considered as relevant and will then be treated as such. The user then has access to all the k objects whose relevance value V (i) is the most important.
• the processing then comprises the two operations mentioned above during the implementation of step (b), said "image elements” then being “image objects”, with in addition a final operation consisting of a summation of the values, of relevance (previously calculated) of the different objects composing the image considered, thus affecting each image the value of relevance sought for the current iteration.
• the algorithm will highlight all the objects common to all the images selected by the user (the summation increasing the area of influence surrounding them).
• a color descriptor based on the colorimetric average of the image, calculated in the HSV color space
• the Applicant has renewed relevance looping experiments for different categories of images.
• Figure 6 shows the evolution of the relevance P (t) as a function of the use of the positive (i.e. relevant) and / or negative (i.e. irrelevant) returns.
• Curve 10 gives the relevance result when the user is allowed (in step (a)) to provide positive and negative responses.
• Curve 20 gives the result of relevance when authorizing the user
• step (in step (a)) to provide only negative responses.
• Curve 30 gives the relevance result when the user is allowed (in step (a)) to provide only positive responses.
• the present invention is not limited to the exemplary image search method as described above, but to any application corresponding to the inventive concept emerging from the present text and the various figures.
• the present invention extends to the image search device capable of implementing the method according to the invention.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Library & Information Science (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Processing Or Creating Images (AREA)

Applications Claiming Priority (1)

Publications (1)

Family

ID=34958538

Family Applications (1)

Country Status (3)

Families Citing this family (17)

Family Cites Families (10)

• 2004
  • 2004-06-23 US US11/630,716 patent/US20070244870A1/en not_active Abandoned
  • 2004-06-23 WO PCT/FR2004/001563 patent/WO2006008350A1/fr active Application Filing
  • 2004-06-23 EP EP04767419A patent/EP1766538A1/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events