EP2417576A1

EP2417576A1 - Determination of descriptor in a multimedia content

Info

Publication number: EP2417576A1
Application number: EP10723226A
Authority: EP
Inventors: Sid Ahmed Berrani; Nicolas Gengembre
Original assignee: France Telecom SA
Current assignee: Orange SA
Priority date: 2009-04-09
Filing date: 2010-04-08
Publication date: 2012-02-15
Also published as: WO2010116093A1; US20120051642A1; US8855420B2

Abstract

The invention relates to a method of determining a descriptor of a region of interest in an image, comprising steps of: - defining (E2) a grid of points for the region of interest; associating (E3) a respective zone of influence with the points of the grid of points; determining (E4) a representative value per point of the grid of points as a function of its respective zone of influence; comparing (E5) the representative value obtained per point of the grid of points with at least one reference value, the result of the comparison making it possible to define a descriptor of the region of interest.

Description

Determination of descriptor in a multimedia content

The field of the invention is that of the transmission or exchange of multimedia content. Multimedia content means text, sound (or audio), images, video or any combination of these elements.

More specifically, the invention relates to the descriptor determination in a multimedia content containing an image or a set of images or a video, in particular for the analysis and control of such contents, such as for example the detection of copies of images. 'a reference content.

The increase in storage capacity of terminals and transmission rates in telecommunications networks has led to the emergence of new services enabling easier consumption of multimedia content.

Thus, content providers offer online services, generally paying, download multimedia content. For content protected by copyright, the respect of these is ensured by the content providers.

Moreover, the number of content exchange sites on which content is put online by the users of these sites continues to increase. Part of this multimedia content is created by the users themselves. Another part consists of protected content and illegally offered for download.

It is therefore necessary to be able to detect illegal copies of protected content.

In general, the detection of copies of multimedia contents consists of searching for the presence or absence of a request content in a reference database of multimedia contents.

Such a base includes descriptors of the multimedia contents of reference. Typically, a descriptor is a numeric value or a set of numeric values that characterizes a portion of the media content. For example, when the multimedia content is a video, a descriptor can be defined for each of the images of the video or for a subset thereof.

To find the presence or absence of a query content in the reference database, we first calculate descriptors for this query content. The calculation mode is identical to the calculation mode of the descriptors of the reference base. Then, we search if the reference database contains descriptors that are identical or similar to those calculated for the query content. If the result is positive, it is deduced that the query content is a copy of the multimedia content whose descriptors have been found in the reference database.

The quality and efficiency of detecting copies of multimedia content is based on the properties of the descriptors. These must be able to be calculated quickly. They must facilitate search in the reference database. These descriptors must also make it possible to detect a copy even if the multimedia content request has undergone important transformations (as for example, a strong compression, a change of resolution, an embedding of text, logo, etc.) with respect to the content. multimedia reference. These transformations can be unintentional, such as transformations due to content recording, transcoding, etc. Some transformations may be intentional in order to make it difficult to detect illegally copied content.

When the multimedia content is an image, a set of images or a video, different types of descriptors can be defined. Some descriptors are calculated globally for an image. Other descriptors are calculated on an image portion called region of interest. For the same image, several regions of interest can be identified and a descriptor calculated for each of them. Descriptors by regions of interest of an image are more efficient than a global descriptor of this image for detecting copies of a video (or an image or a set of images) when it has suffered locally strong transformations. Strong transformations include, for example, partial masking, insertion of a large logo, insertion of a video into an original video, image trimming, etc. Indeed, even if certain regions of a video (or an image or a set of images) are completely missing or hidden, this video remains identifiable thanks to the descriptors of the regions of interest that do not have or little undergone transformation. A global descriptor of a video (or an image or a set of images) is altered by a strong transformation undergone by it.

In the article titled "Feature Extraction and a Database Strategy for

Video Fingerprinting ", Proceedings of the 5 ^th International Recent Advances Conference in Visual Information Systems, 2002, J. Oostveen et al., Provide a global binary descriptor of an image used for the detection of video copies.

A first image is cut into rectangular blocks (for example, 36 blocks on 4 rows and 9 columns). A value is calculated in each of the blocks, such as, for example, the average of the luminances of the pixels of the block.

Then, we calculate the difference between the value obtained in a block and the value obtained in the next block of the same line. We obtain 32 or 4 x 8 values. The same procedure is followed for the next image.

Then, the difference between a value of the first image and the corresponding value of the next image is calculated. This gives 32 new values.

We assign a 1 or a 0 in the descriptor according to the sign of the previously calculated difference.

The previous operations are repeated on the following pairs images for a set of contiguous images of the video.

Then all the descriptors (of 32 binary values in the above example) are concatenated to form a final descriptor.

The disadvantage of such a global detector is that it is inefficient for the detection of copies of a video (or an image or a set of images) when it has undergone strong transformations. as described above.

In the article "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, Vol. 60, No. 2, 2004, D. G. Lowe presents a descriptor defined by region of interest of an image and used for detecting copies of videos.

A descriptor is defined for a circular region of interest. This region is called "scale invariant" (Invariant to changes of scale) insofar as a change in resolution of the image does not change the overall content of the region of interest.

To calculate the descriptor of a region of interest, we define a square encompassing this region. Then, we cut this square into blocks.

In each block, a vector gradient is calculated for each pixel.

The amplitude and orientation of each of these vector gradients are extracted. Then, for each block, a histogram of the orientations of the gradients is created, the value of each orientation being weighted by the corresponding amplitude.

The concatenation of the histograms obtained for the blocks constituting a square encompassing a region of interest defines the descriptor of this region. Such a descriptor is called SIFT (for Scale Invariant Feature

Transform in English).

The components of a SIFT descriptor are real numbers. Therefore, such a descriptor is larger, more complex, and more difficult to exploit than a binary descriptor. One of the aims of the invention is to overcome the disadvantages of the aforementioned prior art.

Thus, according to a first aspect, the present invention relates to a method for determining a descriptor of a region of interest in an image, comprising steps of:

- definition of a grid of points for the region of interest,

associating a respective zone of influence at the points of the grid of points, determining a representative value per point of the grid of points as a function of its respective zone of influence,

- comparing the representative value obtained by point of the grid of points with at least one reference value determined from the values representative of the points of the grid of points, the result of the comparison being expressed in at least one binary value;

concatenation of the results obtained for the points of the grid to define a descriptor of the region of interest of the image.

The method according to the invention makes it possible to define a descriptor by region of interest of the image and not a global descriptor thereof. The descriptor obtained is therefore robust to transformations applied to the image as a whole.

The definition of a grid of points for the region of interest and the association of a zone of influence at these points makes it possible to define a descriptor representative of the visual content of the region of interest. The descriptor obtained being expressed in binary values, it is more compact, simpler and easier to use.

According to a preferred characteristic, during the determination step, the value representative of a point of the grid of points is determined as a function of the weighted values of a measured datum for the pixels of the image contained in the zone d influence of this point. Thus, the descriptor of the region of interest is defined by taking into account all relevant information included in the region of interest of the image.

According to a preferred characteristic, the value representative of a point of the grid of points is equal to the weighted average of the values of the measured data for the pixels of the image contained in the zone of influence of this point.

According to a preferred characteristic, the value representative of a point of the grid of points is equal to the weighted median value of the values of the measured data for the pixels of the image contained in the zone of influence of this point.

Thus, defining the descriptor from the mean or the median value of the measured data for the pixels represents a simple and easy method to set up. According to a preferred characteristic, the value representative of a point of the grid of points is determined by the application of a method based on robust statistics.

The application of a method based on robust statistics makes it possible to attenuate or even cancel the effect of the pixels whose value of the measured data is too far from the representative value. Thus, the result obtained is not distorted by unrepresentative data.

According to a preferred characteristic, the method described above and applied to an original image, thus leading to the determination of the origin descriptor of a region of interest of the original image, also comprises a processing step. additional comprising:

transforming the original image to obtain a transformed image comprising a region of interest which is deduced from a region of interest of the original image as a function of the transformation applied,

determining additional descriptor of a region of interest of the image transformed from the original descriptor of the respective region of interest of the original image, the representation of a region of interest of the original image and the representation of the corresponding region of interest of the transformed image by the same descriptor chosen between the original descriptor and the additional descriptor. By its construction, the descriptor takes into account transformations such as a symmetry along a horizontal axis and / or vertical or a luminance reversal that can undergo the image. Thus, when it is used, it is more robust to this type of transformation.

The invention also relates to a device for determining a descriptor of a region of interest in an image, comprising means for:

- definition of a grid of points for the region of interest,

- association of a respective zone of influence at the points of the grid of points,

determining a representative value per point of the grid of points as a function of its respective zone of influence,

- comparing the representative value obtained by point of the grid of points with at least one reference value determined from the values representative of the points of the grid of points, the result of the comparison being expressed in at least one binary value; concatenation of the results obtained for the points of the grid to define a descriptor of the region of interest of the image.

The invention further relates to a computer program product comprising program code instructions recorded on or transmitted by a computer readable medium, for carrying out the steps of the method described above when said program is running on a computer.

Other features and advantages of the present invention will become apparent from the following description of preferred embodiments described with reference to the figures in which: FIG. 1 represents an embodiment of a descriptor determination method of a region of interest in an image,

FIG. 2 illustrates a first example of definition of a grid of points for a region of interest of an image; FIG. 3 illustrates a second example of definition of a grid of points for a region of interest; 'a picture,

FIG. 4 illustrates an approach based on robust statistics applied to the invention,

FIG. 5 represents an embodiment of a device able to implement the method of FIG. 1.

In the following description, the multimedia content considered is an image, a set of images or a video.

In the case of a set of images or a video, the method according to the invention is applied to the images considered independently of each other. The method can be applied to all the images in a set of images or a video or to a subset of them called keyframes.

Fig. 1 shows an embodiment of a descriptor determining method of a region of interest in an image.

The method comprises a first step E1 of extracting regions of interest from an image. The regions of interest of an image can be extracted with different detectors of regions of interest among which:

a DOG detector (for Difference Of Gaussians in English),

- a Fast Hessian detector,

an MSER detector (for Maximally Stable Extremal Regions),

- a Harris detector. The regions of interest extracted may be of any shape.

From an extracted region of interest, a region of simple shape (e.g., circular, elliptical, square, rectangular, hexagonal, etc.) that encompasses this region of interest is defined.

The following steps E2 to E5 of the method according to the invention apply to the regions of interest extracted during step E1.

These steps make it possible to determine a descriptor by region of interest of the image. The image is described by the set of descriptors of the regions of interest thus obtained.

Step E2 is a step of defining a grid of points relative to a region of interest.

A grid of points is defined for a region of interest extracted during the preceding step E1 or for a region obtained by dilation of a region of interest extracted during the preceding step E1, without moving the center of gravity .

The position of the grid of points corresponds to the position of the region of interest. The size of the grid of points is proportional to the region of interest. The coefficient of proportionality is defined beforehand so that the grid of points covers the region of interest considered, or even exceeds it. Thus the coefficient of proportionality is slightly greater than 1 (for example, of the order of 1, 1 or 1, 2). Depending on the number and position of regions of interest, they may overlap. The corresponding dot grids can also overlap.

The number and distribution of points in the grid is such that the immediate neighborhoods of these points (called zones of influence) encompass the relevant information contained in the region of interest.

The distribution of the points can be any or homogeneous. The points of the grid as well as the centroid of the region of interest do not necessarily coincide with pixels of the image.

FIG. 2 illustrates a first example of defining a grid of points for a region of interest of an image.

The left part of FIG. 2 represents an image I comprising 5 regions of interest Ri to R ₅ .

The right part of FIG. 2 represents a region of interest of the image I, in this case the region R ₅ .

The points Pi to P ₇ represent the points of the grid of points defined for the region of interest R ₅ .

These 7 points are distributed so that the point Pi is positioned at the centroid of the region of interest R ₅ and the points P ₂ to P ₇ represent the vertices of a regular hexagon in which is inscribed the region of interest R ₅ extracted from the image I.

FIG. 3 illustrates a second example of defining a grid of points for a region of interest of an image.

The left part of FIG. 3 represents an image I 'having 4 regions of interest RS to R ₄ .

The right part of FIG. 3 represents a region of interest of the image I ', in this case the region R ₄ on which are positioned 25 points P' ,,, with i and j varying from 1 to 5, d a rectangular grid of dimensions 5x5 points. To lighten the figure, only the points, P'n, P'14 and P _'34 are designated.

During the next step E3, a zone of influence is associated with a point on the grid. An area of influence is a neighborhood of a grid point.

The shape of a zone of influence is arbitrary. Thus, referring to FIG. 2, the zones of influence associated with points Pi to P ₇ of the grid are elliptical. These zones of influence are represented by dashed ellipses surrounding the points Pi to P ₇ .

In FIG. 3, the zones of influence associated with the 25 points of the grid are rectangles. These rectangles are represented in dashed lines around the points PS _1; P'u and PW

The zones of influence of different points of a grid of points may overlap.

The next step E4 is a step of determining a representative value per point of the grid.

This representative value is determined from the values of a measured data for the pixels contained in the zone of influence. For example, the measured data can be the luminance, the average of the channels R, G, B (for red, green, blue in English), the value of a channel, a datum of any system of color representation, etc.

Thus, a pixel contained in the zone of influence of a point of the grid contributes to the determination of the representative value associated with this point.

A pixel can contribute to the determination of the representative value for several points of the grid. This can be seen in FIG. 2, in which the zones of influence represented by dotted ellipses have intersections. Thus a pixel belonging to such an intersection contributes to the determination of the representative values of the points of each of the intersected influence zones. The contribution of a pixel, contained in the zone of influence of a point of the grid, to the determination of the representative value of this point is weighted.

The weighting can be defined, for example, as a function of the distance between the pixel and the point of the grid (center point or centroid of the zone of influence). Thus, the representative value determined for a point of the grid may, for example, be equal to the weighted average of the values of the measured data for the pixels contained in the zone of influence.

As a variant, the average may be replaced by the median value of all the values of the measured data for the pixels contained in the zone of influence.

According to another variant, the determination of the representative value for a point of the grid can be based on robust statistics such as, for example, M-estimators, RANSAC (for RANdom SAmple Consensus in English), etc.

This approach makes it possible to reduce or even eliminate the effect of pixels for which the value of the measured data is very far from the value that one seeks to determine. Taking into account the value of the measured data of this pixel may strongly affect the result obtained. Such an approach may require several iterations.

For example, the M-estimator method based on robust statistics can be applied with multiple iterations.

At the first iteration, a representative value for a point of the grid is determined from the weighted values of a measured datum (for example, the luminance) of the pixels contained in the zone of influence of this point. The weighting applied is a function of the distance between the pixel and the point of the grid considered.

At the next iteration, the weighting applied to the value of a pixel becomes a function of the difference between the representative value determined at the previous iteration for the point of the grid considered and the value of the measured data of this pixel.

The number of iterations is defined by the observation of a criterion representative of the dispersion of the values of the measured data around the determined representative value. The criterion used can be for example the variance, the median value of the deviations, etc. We can choose to stop the iterations when two successive iterations give two results close representative value. It is also possible to predefined a fixed number of iterations.

Figure 4 illustrates an approach based on robust statistics applied to the invention.

The x-axis corresponds to the pixels of a zone of influence taken into account for the determination of the value representative of the point of the grid corresponding to this zone. Thus, Figure 4 shows 7 pixels, Xi to X ₇ .

The y-axis corresponds to the values of the measured data for these pixels and the value representative of the point of the grid considered.

The pixel X ₄ is a pixel whose value of the measured data is very far from the values of the other pixels. The determination of the value representative of the point from, for example, the weighted average of the values of the measured pixel data undergoes the effect of this pixel X ₄ . We then obtain a representative value for the point of the grid considered equal to MC.

The determination of the representative value of the point based on robust statistics makes it possible to rule out the effect of this pixel X ₄ . We then obtain a representative value for the point of the grid considered equal to MR, less than MC. Thus the effect of the pixel is attenuated for the determination of the value representative of the point.

The objective of such an approach is to obtain a descriptor of the region of interest considered less sensitive to transformations of the image.

Thus, with reference to FIG. 4, to significantly modify the value of MC, it is sufficient to modify the value of the measured data for the pixel x _4, whereas to modify the value of MR significantly, the values of the data must be modified. measured by several pixels. For example, the value of the luminance of pixels of a bright spot in a dark area can be strongly modified by a change in resolution of the image (or by another transformation) while the values of the pixels of the dark area are less affected by such a transformation. This transformation becomes non-perceptible for a representative value determined by the application of an approach based on robust statistics. Indeed, in this case, the representative value is that shared by a majority of pixels.

Thus, the use of robust statistics makes it possible to obtain a more robust descriptor for local transformations of the image under consideration.

At the end of step E4, a set of representative values is available, each of these representative values corresponding to a point of the grid defined for a region of interest.

In the next step E5, the representative values obtained in the previous step E4 are compared with at least one reference value.

The order of treatment of the points as well as the order of comparison with at least one reference value are arbitrary. They must nevertheless be the same for any subsequent implementation of the method (for example, for the determination of descriptors of a request content to perform the detection of copies).

By way of example, the at least one reference value may be the representative value determined for the central point of the grid, the average of the representative values obtained for the points of the grid, etc.

According to another example, the at least one reference value is calculated on a set of pixels different from that used to determine the representative value at a point on the grid. This set of pixels comprises, for example, the pixels contained in the zone of influence of a point of the grid and a few neighboring pixels. The result of the comparison is converted into binary values.

For example, if the representative value of a point of the grid is greater than the at least one reference value, the result of the comparison is equal to 1. Conversely, if the representative value of a point of the grid is smaller than at the at least one reference value, the result of the comparison is equal to 0.

The result of the comparison can be expressed over a larger number of binary values in order to refine the deviation with the at least one reference value. For example, if the value representative of a point of the grid is much greater, slightly greater, slightly less, much less than the at least one reference value, the result of the comparison is equal to 11, 10, 01 respectively, 00.

These examples are illustrative of the invention and not limiting thereof. The binary values thus obtained are concatenated according to a predefined order. The result of this concatenation defines a descriptor of the region of interest considered of the image.

The descriptor being obtained by comparing the representative values of the points of the grid with at least one reference value, the latter is independent of global variations in the region of interest considered due to transformations applied to the image.

In the foregoing, at least one identical reference value has been considered for all points of the grid.

As a variant, at least one variable reference value can be considered from one point of the grid to another.

According to another variant, one can consider several reference values for the same point of the grid.

The previous steps E2 to E5 of the method are then applied to the remaining regions of interest extracted during the step E1. A descriptor is thus determined for the regions of interest extracted from the image. The image is described by the set of descriptors of the regions of interest thus obtained.

The method according to the invention also comprises an optional step E6 of additional treatment.

This additional processing step makes it possible to obtain a robust descriptor for simple transformations undergone by an original image comprising at least one region of interest.

A simple transformation is as applied to an original image a first time and then applied a second time to the resulting transformed image the original image is obtained. This is an involution. By way of example, it is a symmetry of the image with respect to a horizontal and / or vertical axis, of a luminance inversion (negative image), etc.

Prior to this step E6, there are descriptors (so-called origin) of the regions of interest of an original image determined by applying the previous steps E2 to E5 of the method. As described above, these original descriptors are in the form of a succession of binary values.

One or more simple transformations are applied to this original image (symmetry with respect to a horizontal and / or vertical axis, luminance inversion, etc.).

Then, the descriptors of the regions of interest of the transformed image are determined. Given the nature of the transformations applied, the regions of interest of the transformed image are not modified in shape and size and are deduced from the regions of interest of the original image as a function of the transformation applied ( for example, by symmetry if the transformation is a symmetry along a horizontal axis and / or vertical). Thus, a region of interest of the transformed image corresponds to a region of interest of the original image. Additional descriptors denote the descriptors of the regions of interest of the transformed image.

An additional descriptor of a region of interest of the transformed image is obtained by permuting and / or taking the complement of certain binary values of the original descriptor of the corresponding region of interest of the original image. Thus, it is not necessary to apply the steps E2 to E5 described above to obtain an additional descriptor of a region of interest of a transformed image.

At the end of step E6, there are then two descriptors, an origin descriptor for a region of interest of the original image and an additional descriptor for the corresponding region of interest of the image. transformed. Only one (for example, the smaller of the two) is retained to represent the two regions of interest considered.

In this way, a region of interest and the region symmetrical with respect to a vertical and / or horizontal axis or region of interest and the inverted region in terms of luminance, etc. have the same descriptor.

The number and nature of the transformations applied to the original image are arbitrary. They must nevertheless be the same for any subsequent implementation of the method (for example, for the determination of descriptors of a request content to perform the detection of copies).

FIG. 5 represents an embodiment of a device able to implement a descriptor determination method of a region of interest in an image as described above.

The device comprises a module M1 for extracting regions of interest from an image.

The module M1 implements the step E1 as described above. The device also comprises an M2 module for defining a grid of points for a region of interest. With reference to step E2, the module M2 makes it possible to define a grid of points for a region of interest extracted by the module M1 or for a region obtained by dilation of a region of interest extracted by the module M1.

With reference to step E3, the module M3 is an association module of a zone of influence at the points of the grid.

The device also comprises a module M4 for determining a representative value per point of the gate as described in step E4.

With reference to step E5, the device also comprises a module M5 for comparing the representative values obtained at the output of the module M4 with at least one reference value.

The device also comprises a module M6 for additional processing of an image as described in step E6.

The device further comprises a central control unit, not shown, connected to each of the modules M1 to M6 and adapted to control their operation.

The modules M1 to M6 may be software modules forming a computer program. The invention therefore also relates to a computer program for a descriptor determining device of a region of interest in an image comprising program code instructions for executing the method previously described by the device.

The different software modules can be stored in or transmitted by a data carrier. This may be a hardware storage medium, for example a CD-ROM, a magnetic diskette or a hard disk, or a transmissible medium such as an electrical signal, optical or radio.

The invention finds in particular, but not only, applications for multimedia content exchange sites.

For example, the invention can be used to detect multiple copies of the same content recorded on such a site. Indeed, the same multimedia content can be recorded several times with a different designation (name, description, etc.) each time. The detection of copies implemented in a content search engine makes it possible to eliminate duplicates and to provide de-duplicated search results.

Some multimedia content is protected by copyright. The invention also makes it possible to detect such content unlawfully made available to the public on content exchange sites.

Claims

A method for determining a descriptor of a region of interest in an image, characterized in that it comprises steps of:

defining (E2) a grid of points for the region of interest,

- association (E3) of a respective zone of influence at the points of the grid of points,

determination (E4) of a representative value per point of the grid of points as a function of its respective zone of influence,

- comparing (E5) the representative value obtained by point of the grid of points with at least one reference value determined from the values representative of the points of the grid of points, the result of the comparison being expressed in at least one value binary; concatenation of the results obtained for the points of the grid to define a descriptor of the region of interest of the image.

2. Method according to claim 1, characterized in that, during the determination step (E4), the value representative of a point of the grid of points is determined according to the weighted values of a measured data for the pixels of the image contained in the zone of influence of this point.

3. Method according to claim 2, characterized in that the value representative of a point of the grid of points is equal to the weighted average of the values of the measured data for the pixels of the image contained in the zone of influence. from this point.

4. Method according to claim 2, characterized in that the value representative of a point of the grid of points is equal to the median value weighted values of the measured data for the pixels of the image contained in the zone of influence of this point.

5. Method according to claim 2, characterized in that the value representative of a point of the grid of points is determined by the application of a method based on robust statistics.

The method according to any one of claims 1 to 5 applied to an original image and leading to the determination of the origin descriptor of a region of interest of the original image, characterized in that it further comprises an additional processing step (E6) comprising:

the transformation of the original image to obtain a transformed image comprising a region of interest which is deduced from a region of interest of the original image as a function of the transformation applied, the descriptor determination addition of a region of interest of the image transformed from the original descriptor of the respective region of interest of the original image,

the representation of a region of interest of the original image and the representation of the corresponding region of interest of the transformed image by the same descriptor chosen between the original descriptor and the additional descriptor.

7. Descriptor determination device of a region of interest in an image, characterized in that it comprises means of: - definition (M2) of a grid of points for the region of interest,

- association (M3) of a respective zone of influence at the points of the grid of points,

determination (M4) of a representative value per point of the grid of points as a function of its respective zone of influence; comparison (M5) of the representative value obtained by point of the grid of points with at least one value of reference determined from the values representative of the points of the grid of points, the result of the comparison being expressed in at least one binary value;

A computer program product comprising program code instructions recorded on or transmitted by a computer-readable medium, for carrying out the steps of the method according to any one of claims 1 to 6 when said program is operating on a computer .