FR2861194A1 - Image indexing/search method for multimedia data bases having search methods determining local vectors/indexing method selected and method providing indexing - Google Patents

Abstract

The index and searching method determines (E3) the local description vectors. One of the indexing methods is then selected (E4) from predetermined methods. The selected method is then used to provide the indexing (E5).

Description

1 2861194 5

  The present invention generally relates to image indexing based on the content of the images. It also relates to image search in an indexed image database.

  A description of an image is calculated from the points of interest of this image. Each point of interest is described by a set of local features in a multidimensional local description vector.

  The set of descriptions of a group of images is an image database.

  In order to be exploitable, especially for research purposes, the image database must be efficiently organized. For this, the set of data constituting the image database is indexed.

  The efficiency of the indexing is measured not only by a search time in the database but also by the flexibility of the database.

  The search time is an average search time for an average set of randomly selected query vectors.

  Flexibility is the ease of management of the image base. This includes the ability to add or delete an image in the database, to add or remove a description vector, while the system is running, and without destroying and then re-creating the entire organization of the database. based.

  Flexibility also concerns the possibility of distributing the image database on a set of networked computers. This distribution 2 2861194 can be dynamic depending on the state of the load (memory and calculation) of the different computers.

  A specific problem of indexing images is its size: indeed, the data to be indexed are very numerous and they are generally of great size.

  The article Searching in high-dimensional spaces Index structures for improving the performance of multimedia database by C. Bohm et al, published in ACM Computing surveys, Vol. 33, No. 3, pp. 322-373, September 2001, and the article entitled Searching in Metric Spaces by E. Chavez et al, in ACM Computing Surveys, Vol. 33, No. 3, pages 273-321, September 2001 are review articles dealing with the problem of indexing large data.

  There is a theoretical result called curse of dimensionality, which shows that beyond a certain dimension of description vector, indexing becomes useless and a search must then include the comparison of dimensions. a query vector with all vectors in the database. The search is then sequential.

  However, research has been carried out and has shown that it is possible to optimize and accelerate such research.

  For example, the document Content-based retrieval using local descriptors: problems and issues from a database perspective by L. Amsaleg and P. Gros, published in Pattern analysis and applications, vol. 4, pp. 108-124, 2001, Springer-Verlag deals with indexing 24-dimensional local descriptors for image search. Optimized sequential search speeds up the search.

  The article entitled Visual recognition using local appearances by V. Colin de Verdière et al, published in the Proceedings of the ECCV 1998 Conference, pages 640-654 describes a data structure that allows for accelerated search.

  Nevertheless, for all these indexing methods, the data structure includes all the indexed images. In other words, the descriptors of the images are stored in the same structure. This structure 3 2861194 is then a global structure that is shared between the different descriptors. This structure is shown schematically in Figure 4a.

  Such a structure is not very flexible, for the following reasons. Deleting or moving description vectors is a complex operation that requires tree traversal and complex memory management operations.

  The construction parameters of the structure are chosen to make it optimal for the search, but only for a given number and category of vectors. As a result, when the structure is updated by adding new description vectors corresponding to a new image, the data structure may become suboptimal.

  The present invention aims to remedy the drawbacks of the prior art, by providing a method and an image indexing device which make it possible to build an optimal basis both from the point of view of research in the database but also from the flexibility of the base.

  To this end, the invention proposes a method for indexing images contained in an image database, characterized in that it comprises, for each image to be indexed, the steps of: determination of local description vectors for the image considered, - selecting an indexing method from among a predetermined set of indexing methods, according to a predetermined selection criterion, the selection being made for each image independently of the other images, - indexing of the vectors local description of the image considered by the selected method.

  The invention makes it possible to build an optimal base both from the point of view of research in the database but also from the flexibility of the database.

  Indeed, the indexing according to the invention comprises a first level of indexing indicating the index of the image and a second level which is the actual indexing of the image description vectors.

  The operations of adding, deleting or moving a vector of an image are simple to manage at the level of the first level of indexing since each image is indexed independently.

  In addition, the indexing is adapted to each image, which offers great opportunities for optimization.

  According to a preferred feature, the method further comprises the step of storing an index of the indexed image in question, an index of the selected indexing method and a structure representing the indexing of the vectors of the index. description of the image considered.

  According to a preferred characteristic, the indexing method is selected from a set comprising: a method according to which the vectors are considered sequentially, at least one tree partitioning method by partitioning the space.

  These methods give good experimental results.

  According to a preferred characteristic combined or not with the preceding one, the indexing method is selected from a set further comprising: at least one method of indexing by data partitioning, at least one indexing method using pivots .

  According to a preferred characteristic, the selection criterion comprises an average search time measurement for a predetermined set of test request vectors for each indexing method and the selection of the indexing method for which the average time is the highest. low.

  2861194 Thus, the search time is taken into account in the selection of the indexing method.

  The invention also relates to an image search method in an image database indexed by the method presented above, characterized in that it comprises, for each request vector of a request image, the steps of: reading the index of the indexing method associated with each image of the database, searching in each image of the database according to the search method associated with the indexing method respectively associated with each image of the database, the vector closest to the query vector.

  According to a preferred characteristic, the search comprises: searching for the vector closest to the request vector in each image of the base, searching for the vector closest to the request vector among the vectors determined in the preceding step.

  The research method has advantages similar to those previously discussed.

  Correlatively, the invention relates to a device for indexing images contained in an image database, characterized in that it comprises: means for determining local description vectors for each image to be indexed, means for selecting an indexing method from a predetermined set of indexing methods, according to a predetermined selection criterion, the selection being made for each image independently of the other images, - indexing means of the description vectors local of the image considered by the selected method.

  The invention also relates to an image search device in an image database indexed by the preceding device, characterized in that it comprises: means for reading the index of the indexing method associated with each image of the base, for each request vector of a request image, search means in each image of the database according to the search method associated with the indexing method respectively associated with each image of the database, the vector closest to the query vector.

  The devices according to the invention comprise means for implementing the characteristics previously described. They have advantages similar to those previously presented.

  The invention also relates to a digital apparatus including the device according to the invention or means for implementing the method according to the invention. This digital camera is for example a digital camera, a digital camcorder, a scanner, a printer, a photocopier, a fax machine. The advantages of the device and the digital device are identical to those previously exposed.

  An information storage means, readable by a computer or by a microprocessor, integrated or not to the device, possibly removable, stores a program implementing the method according to the invention.

  A computer program readable by a microprocessor and comprising one or more sequences of instructions is able to implement the methods according to the invention.

  The features and advantages of the present invention will appear more clearly on reading a preferred embodiment illustrated by the accompanying drawings, in which: FIG. 1 is an embodiment of a device implementing FIG. FIG. 2 represents an indexing device of an image according to the invention, FIG. 3 represents an embodiment of an indexing method according to the invention, FIG. indexing structure of an image database according to the prior art, - Figure 4b schematically shows an indexing structure of an image database according to the invention, - Figure 5 shows a method embodiment. for selecting an indexing method for an image according to the invention, - Figure 6 shows an embodiment of image search method in an image database according to the invention, - Figure 7 details theImage search included in the process of the previous figure.

  According to the embodiment chosen and shown in FIG. 1, a device embodying the invention is for example a microcomputer 10 connected to different peripherals, for example a digital camera 107 (or a scanner, or any means of acquisition or image storage) connected to a graphics card and providing information to be processed according to the invention.

  The device 10 comprises a communication interface 112 connected to a network 113 able to transmit digital data to be processed or conversely to transmit data processed by the device. The device 10 also comprises a storage means 108 such as for example a hard disk. It also comprises a disk drive 110 110. This disk 110 may be a diskette, a CD-ROM or a DVD-ROM, for example. The disk 110 as the disk 108 may contain data processed according to the invention as well as the program or programs implementing the invention which, once read by the device 10, will be stored in the hard disk 108. According to a variant, the program allowing the device to implement the invention, can be stored in ROM 102 (called ROM in the drawing). In the second variant, the program can be received to be stored in a manner identical to that described previously via the communication network 113.

  The device 10 is connected to a microphone 111. The data to be processed according to the invention will in this case be an audio signal.

  This same device has a screen 104 making it possible to display the data to be processed or to interface with the user who can thus set up certain modes of treatment, using the keyboard 114 or by any other means (mouse for example) .

  The device 10 is connected to a scanner 105.

  The CPU 100 (called the CPU in the drawing) executes the instructions relating to the implementation of the invention, instructions stored in the read-only memory 102 or in the other storage elements. At power-up, the processing programs stored in a non-volatile memory, for example the ROM 102, are transferred into the RAM RAM 103 which will then contain the executable code of the invention as well as registers for storing the variables. necessary for the implementation of the invention.

  More generally, an information storage means, readable by a computer or by a microprocessor, integrated or not into the device, possibly removable, stores a program implementing the method according to the invention.

  The communication bus 101 allows communication between the various elements included in the microcomputer 10 or connected to it. The representation of the bus 101 is not limiting and in particular the central unit 100 is able to communicate instructions to any element of the microcomputer 10 directly or via another element of the microcomputer 10.

  With reference to FIG. 2, an embodiment of an indexing device according to the invention is intended to index a digital image. The indexing device is integrated in a device, which is for example a digital camera, a digital video camera, a scanner, a printer, a photocopier, a fax machine, a database management system or a computer. .

  The image indexing device according to the invention comprises: means 1 for determining local description vectors 5 for each image to be indexed, means 2 for selecting an indexing method from among a predetermined set of indexing methods, according to a predetermined selection criterion, the selection being made for each image independently of the other images, means 3 for indexing the local description vectors of the image considered by the selected method.

  The result of the indexing is stored in a memory so as to constitute an image base 4.

  The invention also relates to an image search device in an image database indexed by the device presented above, characterized in that it comprises: means for reading the index of the method of indexing associated with each image of the base, for each request vector of a request image, search means 6 in each image of the database according to the search method associated with the indexing method respectively associated with each image of the base, the vector closest to the query vector.

  The operation of the indexing device and the search device is detailed below. The means of these two devices can be integrated in the same device as shown in Figure 2, or in separate devices.

  FIG. 3 represents an embodiment of an indexing method of an image, according to the invention. This method is implemented in the coding device and comprises steps E1 to E6.

  The method is embodied in the form of an algorithm that can be stored in whole or in part in any information storage means capable of cooperating with the microprocessor. This storage means is readable by a computer or a microprocessor. This storage means is integrated or not to the device, and can be removable. For example, it may comprise a magnetic tape, a floppy disk or a CD-ROM (compact disk with frozen memory).

  Consider a set of N images to index IMn, with N a predetermined integer and n an integer varying between 1 and the integer N. Step El is the selection of an image to index IMn.

  The next step E2 is the detection of the points of interest in the current image IMn. The points of interest are detected in a manner known per se. For example, the precise Harris detection technique described in the article entitled "Evaluation of interest points detectors" is published in International journal of computor vision, No. 2, vol. 37, 2000. In another example, using the technique disclosed in French Patent Application No. 03 01545. It is assumed that NB points of interest were found.

  The following step E3 is the calculation of the local description vectors Vkn, with kn between 1 and NBn, respectively associated with the points of interest previously determined in the image IMn. Extended local descriptors are used in the R, G and B color three-component images as described in P.Gros et al., INRIA Research Report No. 3269, Sept. 1997, Color Use for Matching and Matching. image indexing or in the document Content-based retrieval using local descriptors: problems and issues from a database perspective by L. Amsaleg and P. Gros, published in Pattern analysis and applications, vol. 4, pp. 108-124, 2001, Springer-Verlag.

  Each description vector Vkn has a dimension equal to 24.

  The next step E4 is the selection of the best indexing method for the Vkn vectors. The selection is made from a predetermined set of M methods. The selected method is the one that provides the smallest average search time. This step is detailed below. It results in an indexing method Sm, with m an index varying between 1 and M. The next step E5 is the indexing of the local description vectors Vkn by the selected method Sm. The result is a data structure Sm (lMn) which represents the indexing of the description vectors of the image IMn.

  The next step E6 is the storage of the index n of the current image IMn, of the index and of the indexing method used Sm, and of the data structure Sm (lMn).

  The previous indexing operations are repeated successively for all the images of an image database. Thus, we obtain a schematic structure in Figure 4b. The indexing of the image database is at two levels. A first level concerns the respective index of each image. A second level concerns the local description vectors of each of the images. This structure is more flexible than the overall structure according to the prior art, represented in FIG. 4a.

  FIG. 5 shows the step E4 of selecting the best indexing method for the current image, in the form of an algorithm comprising steps E41 to E49.

  A predetermined set of M indexing methods is considered each associated with a search method.

  In a preferred embodiment of the invention, the indexing methods belong to two categories.

  The first category has a simple indexation method in which vectors are considered sequentially in a list. This method has no special setting. In addition, although this is not an actual indexing method, this method may in some cases provide faster searches than other indexing methods, because that sequential processing is well managed by some computer architectures.

  The second category is the method of tree indexing by space partitioning. For example, the method described in the article entitled Local grayvalue invariants for retrieval image of C. Schmid and R. Mohr, published in IEEE transactions on pattern analysis and machine intelligence, Vol. 19, No. 5, pp. 530-534, 1997.

  For these methods, several settings are possible. For example, several slice interval values for each of the dimensions are considered: 2, 4, 8, 10, and so on. Also, several values of maximum occupation of the cells are considered.

  Several values of the threshold which govern the extension of the division to the following dimensions are also considered: 2, 10, 20, etc. The optimal choice of these parameters results from a compromise between the ability to move the vectors away from each other so that a minimum number of cells are considered during the systematic search phase, and the ability to maintain a simple structure that minimizes the number of paths in the tree during the search.

  Depending on the choice of parameters, a set of methods is formed.

  In the following, the M indexing methods are considered sequentially and for each of them an average search time is calculated.

  Step E41 is an initialization in which the parameter m is initialized to the value 1 to consider the first indexing method Si.

  The next step E42 is the indexing of the local description vectors Vkn of the current image IMn by the current method Sm. The result is an indexing structure Sm (lMn).

  The next step E43 is the initialization of a timing device for measuring the search time.

  The next step E44 is the search for a set of vectors 30 test requests Vq in the current indexing structure Sm (lMn).

  The test request vectors Vq are chosen to reflect an average behavior of request vectors. For example, they are drawn randomly from a large number of real vectors. The number of test request vectors is chosen between 1,000 and 10,000 so that the result is statistically valid.

  The search is carried out either at a predetermined distance or closer to the nearest neighbor. In a preferred embodiment, the search is carried out at the nearest neighbor.

  For the sequential indexing method, the distance between each test request vector Vq and each vector Vkn of the indexing structure Sm (IMn) is calculated. The closest vector is thus identified.

  For a method of space-partitioning tree indexing, for example, the closest-neighbor search techniques described in the article Searching in High-Dimensional Spaces Index structures for improving the performance of multimedia database by C. Bohm are used. et al, published in ACM Computing surveys, Vol. 33, No. 3, pp. 322-373, September 15, 2001.

  The next step E45 is the stopping of the timing.

  The next step E46 is the memorization of the timing result for the current method Sm.

  The next step E47 is a test to determine if all 20 indexing methods have been considered.

  If the answer is negative, then this step is followed by step E49 at which the parameter m is incremented by one unit to consider a next indexing method. Step E48 is followed by step E42 previously described.

  When the response is positive in step E47, this step is followed by step E49 to which is determined the indexing method Sm which provides the meanest search time for the current image IMn.

  Thus, the indexing method Sm is associated with the current image Mn.

  FIG. 6 represents the method of searching for a request image in the image database, in the form of an algorithm comprising steps E7 to E1a.

  The method is embodied in the form of an algorithm that can be stored in whole or in part in any information storage means capable of cooperating with the microprocessor. This storage means is readable by a computer or a microprocessor. This storage means is integrated or not to the device, and can be removable. For example, it may comprise a magnetic tape, a floppy disk or a CD-ROM (compact disk with frozen memory).

  In an operational system, this method is executed continuously to process a stream of incoming images resulting for example from an image retrieval from a set of Internet sites.

  The data structure representing the indexing of the images of the database resides in random access memory and can be distributed on a set of computers.

  Step E7 is the input of an IMq request image.

  The next step E8 is the detection of the points of interest of the IMq request image. The detection is performed according to the same method as during the indexing (step E2). It should be noted that if the multi-resolution method described in French Patent Application No. 03 01545 is used, only the points of interest at the resolution of the observed image are calculated.

  The next step E9 is the calculation of the local request Vkq request vectors of the IMq request image. This calculation is similar to that performed during indexing (step E3).

  The next step E10 is the actual search of the request image IMq in the image database, from the request vectors Vkq. The result is the set of K images in the image database that are closest to the IMq request image. This step is detailed below.

  FIG. 7 represents the step El0 of searching the K images of the image database that are closest to the request image IMq, in the form of an algorithm comprising steps E100 to E11.

  For this, the request vectors Vkq are compared with the description vectors Vkn of the database.

  The images IMn of the base are then classified according to the number of their vectors which are mapped to the vectors Vkq of the IMq request image. This number is called score.

  Step E100 is an initialization at which the scores of the images of the base are set to zero.

  The next step E101 is an initialization to which a parameter kq is set to 1 to consider the first request vector.

  The next step E102 is an initialization to which a parameter n is set to 1 to consider the first image of the base.

  The next step E103 is the reading of the index m of the indexing method Sm associated with the current image IMn of the database.

  The next step E104 is the search for the vector Vkn of the current image IMn which is the closest to the current request vector Vkq. This search is performed according to the search method associated with the current Sm indexing method. The possible search methods are the same as those used previously in step E44. The next step E105 is the memorization of the distance dqn between the

  current query vector Vkq and its nearest neighbor Vkn in the current image IMn.

  The next step E106 is a test to determine if all images in the database have been considered.

  If the answer is negative, then this step is followed by step E107 at which the parameter n is incremented by one unit to consider a next image in the database. Step E107 is followed by step E103 previously described.

  When the response is positive in step E106, all the structures of all the images have been scanned. This step is followed by the step E108 at which the image IMn corresponding to the distance dn the weakest is determined. The score for this image is incremented by one.

  The next step E109 is a test to determine if all the request Vkq request vectors have been considered. If the answer is negative, then this step is followed by the step E110 at which the parameter kq is incremented by one unit to consider a next request vector. Step E110 is followed by step E102 previously described.

  When the response is positive in step E109, then this step is followed by step E111 at which the scores of the images of the base are sorted in descending order and the indices of the images corresponding to the K highest scores are given as Result of the research.

  Of course, the present invention is not limited to the embodiments described and shown, but encompasses, on the contrary, any variant within the scope of those skilled in the art.

  In particular, other indexing and search methods can be used in the context of the invention. For example, data partitioning indexing methods such as that described in E.Chavez et al's article "An effective clustering algorithm to index high dimensional metric spaces", published in the proceedings of the SPIRE 2000 conference, are usable.

  Indexing and search methods using pivots, such as that described in the article entitled Searching in Metric Spaces by E. Chavez et al, published in ACM Computing Surveys, Vol. 33, No. 3, pages 273-25,321, September 2001, can also be used.

  Of course, it is possible to use these types of indexing in combination with the indexings described above to index the same database.

Claims (18)

  1. Method for indexing images contained in an image database, characterized in that it comprises, for each image to be indexed, the steps of: determination (E3) of local description vectors for the image 10 considered - selecting (E4) an indexing method from a predetermined set of indexing methods, according to a predetermined selection criterion, the selection being made for each image independently of the other images, - indexing (E5) local description vectors of the image considered by the selected method.   2. Method according to claim 1, characterized in that it further comprises the step (E6) of storing an index of the indexed image considered, an index of the indexing method selected and of a structure representing the indexing of the description vectors of the image considered.   3. Method according to claim 1 or 2, characterized in that the indexing method is selected (E4) from among a set comprising: a method according to which the vectors are considered sequentially, at least one tree indexing method; partitioning of space.   4. Method according to claim 3, characterized in that the indexing method is selected from a set further comprising: at least one method of indexing by data partitioning, at least one indexing method using pivots.   5. Method according to any one of claims 1 to 4, characterized in that the selection criterion (E4) comprises an average search time measurement for a predetermined set of test request vectors for each indexing method and the selecting the indexing method for which the average time is the lowest.   6. An image search method in an image database indexed by the method according to any one of claims 2 and 3 to 5 taken in dependence on claim 2, characterized in that it comprises, for each vector querying a request image, the steps of: - reading (E103) the index (m) of the indexing method (Sm) associated with each image of the database, - searching in each image of the database according to the search method associated with the indexing method (Sm) respectively associated with each image of the base, the vector closest to the request vector.   7. An image search method according to claim 6, characterized in that the search comprises: a first search (E104) of the vector closest to the request vector in each image of the database, a second search (E108) of the vector closest to the request vector among the vectors determined in the previous step. 30 8. A device for indexing images contained in an image database, characterized in that it comprises: means (1) for determining local description vectors for each image to be indexed, means ( 2) selecting an indexing method from a predetermined set of indexing methods, according to a predetermined selection criterion, the selection being made for each image independently of the other images, - means (3) of indexing the local description vectors of the image considered by the selected method.   9. Device according to claim 8, characterized in that it further comprises means (4) for storing an index of the indexed image considered, an index of the selected indexing method and a structure representing the indexing of the description vectors of the image considered.   10. Device according to claim 8 or 9, characterized in that the means (2) of selection are adapted to select the indexing method from a set comprising: - a method according to which the vectors are considered sequentially, - at least one tree indexing method by partitioning the space.   11. Device according to claim 10, characterized in that the means (2) of selection are adapted to select the indexing method from a set further comprising: - at least one method of indexing by partitioning data, - to least an indexing method using pivots.   12. Device according to any one of claims 8 to 11, characterized in that the means (2) of selection are adapted to implement a selection criterion which comprises a measure of average search time for a predetermined set of vectors Test queries for each indexing method and selection of the indexing method for which the average time is the lowest.   13. An image search device in an image database indexed by the device according to any one of claims 9 and 10 to 12 taken in dependence on claim 9, characterized in that it comprises: - means (5) reading the index (m) of the indexing method (Sm) associated with each image of the database, for each request vector of a request image, - search means (6) in each image. of the base according to the search method associated with the indexing method (Sm) respectively associated with each image of the base, of the vector closest to the request vector.   14. An image search device according to claim 13, characterized in that the search means (6) comprise: first search means of the vector closest to the request vector in each image of the database; second search means of the vector closest to the request vector among the vectors determined in the previous step. 25 15. An image indexing device according to any one of claims 8 to 12, characterized in that the means for determining, selecting and indexing are incorporated in: - a microprocessor (100), - a read-only memory (102) comprising a program for processing the data, and - a random access memory (103) comprising registers adapted to record modified variables during the execution of said program.   16. An image search device according to claim 13 or 14, characterized in that the reading and search means are incorporated in: a microprocessor (100), a read-only memory (102) comprising a program for processing the data, and - a random access memory (103) having registers adapted to record modified variables during the execution of said program.   17. Apparatus for processing (10) a digital image, characterized in that it comprises means adapted to implement the method according to any one of claims 1 to 7.   18. Apparatus for processing (10) a digital image, characterized in that it comprises the device according to any one of claims 8 to 16.