FR2998399A1 - Method for editing video sequence in plane, involves determining series of transformations i.e. homography, for each current image of video sequence, and performing step for temporal filtering of series of transformations - Google Patents

Abstract

The method involves determining a series of transformations (10) i.e. projective transformation or homography, for each current image of a video sequence, where the transformations include a parameter that is representative of movement of a plane between a reference image and the current image. A step for temporal filtering of the series of transformations is carried out (11). Information representative of quality of the transformation, and information that is representative of quality of filtered transformation are provided.

Description

FIELD OF THE EDITION OF A PLANE IN A VIDEO SEQUENCE 1. Field of the invention

The invention relates to the field of video editing. More specifically to a method of editing a clip in a video sequence and to a device implementing this method. 2. State of the art

According to the state of the art, it is known, in the field of video editing, tracking functions of a flat surface (in English "planar tracking") between an image of a video sequence and a set of successive images of the video sequence. These functions of tracking a flat surface are for example used to insert a logo on this flat surface in a video sequence, to change the texture or the color of this flat surface. According to a first method of tracking a planar surface, a simple piece of information representative of the location of a region of interest comprising the plane, for example an all-encompassing box, in the successive images is sufficient for certain applications such as detection and detection. followed by an object. On the other hand, this first method is not sufficiently accurate for more advanced video editing applications as previously described (such as logo insertion). According to a second planar tracking method, a projective transformation is estimated between the image on which the plane is determined, said reference image and each successive image of the video sequence. A projective transformation, also called homography, allows to model the movement of the projected plane of the 3D space in the image plane (for example the rotation of a plane). In "SURF: Speeded Up Robust Features", Herbert Bay, Ess Andreas, Tinne Tuytelaars and Luc Van Gool describe a method for extracting, describing and mapping points of interest in images. From pairs of points mapped between the reference image and each of the other images, the projective transformations are estimated, for example using the robust RANSAC method described by Martin A. Fischler and Robert C. 2 9983 99 2 Bolles "Random Sample Consensus: A Paradigm for Modeling with Applications to Image Analysis and Automated Cartography" (Comm of the ACM, Vol 24, June 1981, 381-395). At least 4 pairs of points belonging to the plane are needed to calculate a homography. Thus, in a texture modification application, an operator indicates the planar texture to be inserted, the location of a plane in the reference image on which to insert the texture, and the planar tracking method provides the motion patterns of the plane. , that is the series of homographies, connecting each image to the reference image. These homographies are then used to automatically insert the texture throughout the sequence from the information for the first image. More precisely, to insert a logo on a flat surface, the estimated projective transformation is applied to each pixel of the current image. If the corresponding point in the reference image belongs to the surface on which the logo has been added, the pixel values of the current image are recalculated by an interpolation method (eg bilinear) of the values of the added logo. in the reference image. However, a disadvantage of this insertion method is that the texture (or logo) inserted is noticeable during the representation of the video sequence. Thus, a "pumping effect" or "flickering" effect can be observed above the surface on which it has been inserted. It is an object of the invention to provide a planar tracking method for smoothing the series of projective transformations and thereby suppressing this floating effect during video editing. 3. Summary of the invention. The invention aims to overcome at least one of these disadvantages of the prior art. The invention relates to a method, implemented in a computer system, of editing a plane in a video sequence comprising a step of determining a series of transformations for each current image of the video sequence, a transformation comprising at least one parameter representative of the movement of the plane between a reference image and the current image. The method is remarkable in that it comprises a step of temporal filtering of the series of transformations.

According to one particular characteristic, the temporal filtering step comprises determining at least one parameter representative of the movement of the plane between the reference image and the current image of the video sequence from a weighted sum of at least a parameter representative of the movement of the plane between the reference image and an image belonging to a determined set of images of a temporal window around the current image. Advantageously, the series of filtered transformations is a function, for each current image, of an information representative of the quality of the transformation and a function of information representative of the quality of the filtered transformation. According to another particular characteristic, the time filtering step comprises a time filtering step of a trajectory representative of the movement of at least one point belonging to the plane between the reference image and the successive current images and a determination step at least one parameter representative of the movement of the plane between the reference image and the current image of the video sequence from the at least one point belonging to the plane in the reference image and at least one corresponding point in the current image where the position of the corresponding point in the current image results from the filtered trajectory. Advantageously, the filtering step of the transformation series comprises: a step of determining the at least one point belonging to the plane in the reference image of the video sequence; a step of determining a trajectory representative of the movement of the at least one point belonging to the plane between the reference image and the successive current images; a step of temporal filtering of the trajectory representative of the movement of the at least one point belonging to the plane between the reference image and the successive current images. Advantageously, a trajectory associated with a point of the reference image comprises a set of motion vectors of the point between the reference image and the current images. The filtered trajectory representative of the movement of the at least one point belonging to the plane of a reference image is a set of motion vectors each corresponding to a weighted sum of neighboring spatial and temporal motion vectors of the trajectory. Advantageously, the filtered trajectory representative of the movement of the at least one point belonging to the plane of a reference image towards the current image is a function of information representative of the quality of the transformation in the vicinity of the at least one point. According to another particular characteristic, the time filtering step of the series of transformations is repeated for a series of filtered transformations. According to another particular characteristic, the transformation is a projective transformation, called homography, the transformation comprising 9 parameters representative of the motion of the projected plane of the three-dimensional space in the image plane. According to another particular characteristic, the method further comprises a step of interpolation or extrapolation of a transformation between the reference image and a current image that is absent from the series of transformations. According to another particular characteristic, the method further comprises a step of modifying the texture of the points belonging to the plane. According to another aspect, the invention relates to a device implementing the method according to one of the previously described variants; a computer program product implementing the method according to one of the previously described variants; and a computer readable medium having computer executable instructions implementing the method according to one of the previously described variants. 4. List of figures. The invention will be better understood, and other features and advantages will appear on reading the description which follows, the description referring to the appended drawings in which: FIG. 1 illustrates a method of editing a plan according to a non-limiting embodiment of the invention; FIG. 2 illustrates a method of editing a plane according to another example of non-limiting implementation that is particularly advantageous of the invention; - Figure 3 schematically illustrates a device implementing the method according to a non-limiting embodiment of the invention. 5. Detailed description of embodiments of the invention. FIG. 1 illustrates a method of editing a plan according to a non-limiting embodiment of the invention. During a first step 10 of determining a series of transformations, a transformation, for example a projective transformation, is determined between a reference image and a current image of the video sequence. The transformation comprises at least one parameter representative of the movement of the plane from a reference image to a current image. According to a particular example, the transformation is a plane projective transformation or homography. A homography H is a linear transformation applied to a vector x comprising 3 components {xh; ; 1}. This linear transformation is represented by a 3 x 3 matrix. According to this example, the transformation comprises 9 parameters. In the remainder of the description, the terms parameters and coefficients of the homography matrix will be used indifferently. To determine this series of transformations of a plane, a set of points x0 belonging to the plane in a first indexed image 0 or reference image is selected. Consider an image sequence (Ii) i = O ... N defined for instants represented by the indices from 0 to N. In the remainder of the description, the terms images and frames will be indifferently used. A point, denoted xi, of an image of the sequence at time i corresponding to xo is obtained by a matching or tracking method, for example described in "SURF: Speeded Up Robust". Features (Computer Vision and Image Understanding (CVIU), Vol 110, No. 3, pp. 346--359, 2008) by Herbert Bay, Andreas Ess, Tinne Tuytelaars and Luc Van Gool. At least 4 pairs of points belonging to the plane are needed to calculate a homography. From pairs of points mapped between the reference image and each of the other images, the projective transformations are estimated, for example using the robust RANSAC method described by Martin A. Fischler and Robert C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography "(Comm of the ACM, Vol 24, June 1981, 381-395).

The matrix representing the projective transformation between the images 0 and i, i> 1 is denoted Hie ,. Thus the point of an image i corresponding to a point in the image 0 is represented by the following equation: xi = Hioxo The motion vector dxi associated with the point xi between the images 0 and i is represented by dxi = xi - xo = (Hio - 1) xo, where I represents the identity matrix 3x3. According to a particularly advantageous characteristic, the method comprises a second time filtering step 11 during which the series of transformation is smoothed. Advantageously, the floating effect of a logo inserted automatically from a reference image using this video editing method is deleted. According to a first embodiment, the temporal filtering is directly applied to the coefficients of the homography Hio. The temporal filtering of the homography series (Hio) i =, ... N is then represented by the homography series (Hiol) i = 1 ... N obtained by the following formula: Ei - Hio '= Ei wi + 1, where: - Hio 'corresponds to the filtered Hio homography, - j is an image index, this index varying in a time window centered on the index i, - - is a normalization coefficient adapted to normalize the displacements on the time interval [0,]] with respect to the displacements on the time interval [0, i],% mi is a weighting coefficient adapted to quantify the influence of the parameters of the homography Hio.

Those skilled in the art will notice that the filtering is applied to the displacement vectors dxi = (Hio -1) xo and not to the positions of the points xi of the plane, which explains the presence of the identity matrix in the formula noted "+ 1 ". According to one variant, the weighting coefficients are defined for example as a function of the absolute distance between the instants i and j, w = exp (-Ii -il). According to another variant, the weighting coefficients are defined according to a quality measure, calculated in advance, of Hio according to known methods of cost calculation, for example sum of the squared differences (in English "sum of squared differences "SSD) defined by: = exp (-ssd (Hio)) with EXoew ij (HjoX0H0 001 2 SSd (11j0) = where (0 corresponds to the region of the image / 0 on which the logo is to be inserted and where Icol corresponds to the cardinal of (.0.

According to other quality measurement variants, the sum of the absolute differences (SAD) or the root mean square error (RMSE) are compatible with the invention.

According to another variant, the weighting coefficient is zero, i.e. wi = 0, when the homography Hio is not defined. This characteristic is particularly advantageous in the case of obscuration of at least part of the plane or if a terminal of the video sequence is reached, that is to say if j <1 or if N.

According to another particular characteristic, the method comprises a third step of validation of the temporal filtering during which a measurement of quality is calculated for each transformation Hio before filtering and for each filtered transformation H'io; and during which the validated transformation H "i0 corresponds to the transformation H'io or Hio for which the measurement of the quality is better, but the invention is not limited to the modeling of the movement of a plane by the homographies, there are also affine models represented in the form of a matrix with 6 coefficients calculated from at least 3 pairs of points making it possible to represent the rigid movements in the image plane such as a translation, a change of scale This is also the case for simplified affine models with 4 parameters Finally, if the model simply represents a translational motion of the plane in the image space, then it is represented in the form of a vector that is that is to say, a single coefficient calculated from a single point of the plane.The latter model is advantageous if the motion of the plane is constrained, that is to say if the degrees of freedom of the plane in space trid This constraint on the movement of the plane is for example introduced in the plane editing method by an operator.

In this first implementation example, even if the filtering weighting coefficients are defined as a function of a quality measure of the transformations, this filtering is defined globally for the whole of the tracked plan.

FIG. 2 illustrates a method of editing a plane according to another example of non-limiting implementation that is particularly advantageous for the invention. In this exemplary embodiment, the filtering advantageously takes into account both local and global criteria. In addition, the filtering is not applied directly to the coefficients of the transformations but to the respective trajectories of points belonging to the whole plane and used to determine the filtered transformations. Advantageously, the filtering is temporal, that is to say that the treatment is applied independently to scattered point trajectories. But the invention is not limited to time filtering, alternatively spatial filtering can be applied locally to these scattered trajectories. All the steps of this second exemplary embodiment will now be described. During a first step 20 of determining a series of transformations of a plane, a set of points x0 belonging to the plane in a first indexed image 0 or reference image is selected. As before, the image sequence (Ii) i = 0 ... N is defined for instants represented by the indices of 0 to N. A point of an image of the sequence at time i corresponding to x0, noted xi, is obtained by a matching or tracking method, for example described in "SURF: Speeded Up Robust Features", Vol 110, No. 3, pp. 346--359, 2008) by Herbert Bay, Andreas Ess, Tinne Tuytelaars and Luc Van Gool. The matrix representing the projective transformation between the images 0 and i, i> 1 is denoted Ha) is determined from these points, for example using the robust RANSAC method described by Martin A. Fischler and Robert C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. "(Comm of the ACM, Vol 24, June 1981, 381-395). According to an advantageous variant, the number of points is 4 and the points are not aligned. During a second step 21, a trajectory is determined for a set of p points tP = (34, ..., x) between the reference image and the image i by using the transformations (Hio) i = 1 ... N. The point of an image i corresponding to a point in image 0 is represented by the following equation x = Hiog. According to one variant, the points are those used in step 20 to determine the homography. According to another variant, the p points are selected from the points of the plane, for example the four vertices of a quadrilateral defining the outline of the plane in the reference image. A trajectory represents the movement of a point xP between a reference image and the successive current images. A trajectory comprises a set of motion vectors dxi = xi - xo associated with the point xP between its position x / o) in a reference image 0 and its positions xr in each of the successive current images i. During a third step 22, the trajectories tP are filtered. A method of filtering trajectories is described, for example, in the international patent application PCT / EP13 / 054165 filed on March 1, 2013 by the same applicant. This method introduces a weighting in the filtering depending on the similarity between the trajectories calculated from the available motion fields between two images of the video sequence. According to a particularly advantageous variant, each motion vector of the filtered trajectory is obtained by calculating a weighted sum of neighboring motion vectors of this trajectory. Advantageously, the neighboring motion vectors will be considered temporally, that is to say vectors belonging to the trajectory of the point of the reference image. In a variant, the motion vectors of the neighboring points spatially around the scattered points xP can also be taken into account in the filtering. According to one variant, each motion vector of a filtered trajectory representative of the movement of a point of the plane of a reference image towards the current image is a function of information representative of the local quality of the displacement corresponding to the transformation. initial (before filtering) in the vicinity of the point. The trajectories are thus filtered temporally on i independently while checking that each new position xr increases the local quality measurement of the displacement around xr. In the opposite case, the unfiltered initial position xii 'is retained for the calculation of the filtered transformation. We can represent the filtering of a point trajectory by the following formula: X. = where j is an index of image, this index varying in a temporal window A of radius r> 0 centered on the index i is A = [max (1, i - r), min (i + r, N)]; - is a normalization coefficient adapted to normalize the displacements between the images on the time interval [0, j], with respect to the displacements on the time interval [0, i], wP is a local weighting coefficient adapted to quantify the influence of homography parameters H () for the p-th trajectory.

This weighting coefficient is based on an estimation of the local quality of the displacement corresponding to the transformation Hp, locally around the points x / o) and x11. According to a variant, the representative information / (Hi0, 34) of this quality measure (also called cost) is calculated from the average of the SAD (sum of the absolute differences) on a spatial neighborhood of dimension 3x3 denoted 19 ( x / c))) around xP - 0 1 (110 'XP) = IyE, 9 (xPEcEtr, g, b) 1-8 (37) - / (Hjoy) 1 0 119 (XDI In this equation 1.19 ( x / 0)) 1 represents the cardinal dei9 (x) so as to take into account the bounds of the image and the plane Typically the cardinal dei9 (x) is equal to 9 (9 pixels on a spatial neighborhood of dimension 3x3 ) but it can be smaller if the point is located on an edge of the image.The weighting coefficient wiP is then defined according to this information representative of the quality measure by the following equation: wP = e Y where y is a positive constant for normalizing the weighting coefficient According to an advantageous variant, if the homography Hjo is not defined, the weighting coefficient wf is naked According to other local quality measurement variants, a normalized cross correlation (NCC) method is compatible with the invention. According to another particular characteristic, the method comprises a step of validating the result of the filtering, ie the x -resultant point of the filtered trajectory, or the resulting x-point of the initial trajectory is selected according to a measurement of local quality of the estimated displacement. From the new filtered position, a new measure of the local quality of the displacement m (dx ', x) is calculated from the motion vector = -: m (dX1, = IyEi9 (xP) E cEfr, b11 I (Y) - ic (y + dx1i '1) 1 1i9 (xIC) I The following equation represents the selection of x1' or x 'according to the measure of local quality of displacement: pfx'si ni (of X10 ) <(I 1 (10, x) + a Xi = xl? Else where a is a positive constant adapted to favor the selection of the filtered position with respect to the unfiltered initial position. During a fourth step 23 of determining the series of filtered transformations, a new homography is estimated Hio 'for each image i as a function of the points p (34) 0. in the image of p = .. P reference and their correspondents in the current images ( e: ") p = 0 ... P 'According to another particular characteristic, the method comprises a step of validation of the temporal filtering during which a global measurement quality is calculated for the series of transformations before filtering Hic) and for the series of filtered transformations H'io. The series of validated H "jo transformations corresponds to the series of transformations H'io or Hiopour for which the measurement of the quality is better.A global quality measurement is, for example, calculated as follows for a transformation between i and 0. on the whole plane n: 9 (FI 0) = yeS2 cE [r, g, b) lay) - Ii (Hioy) ln I and g (Hio ') = EyEn Ece [r, g, b) i ( The quality measurement variants described with the first exemplary embodiment are also compatible therewith.

And the series of retained transformations is represented by H 'rio' if g (Hio ') <g (Hio) + io = Hi () otherwise where / 61 is a positive constant adapted to favor the selection of the filtered transformation with respect to the unfiltered input transformation.

According to another advantageous characteristic, the time filtering step of the series of transformations is repeated for a series of filtered transformations. Thus, the steps of determining the trajectory 21, of filtering 22 and of determining 23 of the series of filtered transformations are iterated from the series of transformations of the first iteration. According to one variant, the filtering step is repeated until a stop criterion is reached, for example a maximum number of iterations, a minimum threshold for improving the overall quality of the transformations estimated between two iterations. consecutive, or a minimum average distance between the estimated transformations between two consecutive iterations. In particular, the improvement of the overall quality of the transformations between two consecutive iterations can be calculated from the following ratio: Ei = 1..Ng (Hi0) g (Hi0 ") And, the distance between two transformations Hi () and Ha) "defined between the images at times 0 and i for a region (.0 of the image at time 0 can be defined as follows: ## EQU1 ## Another advantageous characteristic, the method comprises a step of modifying the texture of the points belonging to the plane. For example, we insert a logo on a plane in the reference image. Then for each successive current image, the homography H, inverse of the filtered homography H, is applied to each pixel of the current image. If the corresponding point in the reference image belongs to the area on which the logo has been added, the pixel values of the current image are recalculated by an interpolation method, for example bilinear, values of the logo added in the reference image. Thus, the floating effect of the logo is reduced because the homography series is smoothed temporally, each homography depending on neighboring homographies. Finally, if homographies are absent from the series of transformations, for example because points used to calculate them are obscured in an image of the sequence, the method described advantageously makes it possible to interpolate a transformation between the reference image and a current image from the transformations for the neighboring images of the current image. If homographies are absent from the series of transformations, for example because they are associated with images outside the boundaries of the video sequence, the method described advantageously makes it possible to extrapolate a transformation between the reference image and a current image from the transformations for images of the sequence.

FIG. 3 schematically illustrates an example of a hardware embodiment of a device adapted to implement one of the methods for editing a plane in a video sequence previously described. The device 3 corresponds for example to a personal computer, a laptop, a tablet, a mobile phone or any image processing unit. The device 3 comprises the following elements, interconnected by a bus 34 of addresses and data which also carries a clock signal: a microprocessor 31 (or CPU (Central Processing Unit); "Central processing unit"); - a non-volatile memory of the ROM type (of the "Read Only Memory") 32; - a random access memory or RAM (of the "Random Access Memory") 33; MMI interface (or human machine interface of the English "Man Machine Interface") 35 that is to say a specific application adapted to the display of information for a user and / or the entry of data or information. settings (eg mouse, keyboard, touch screen, webcam).

It will be observed that the word "register" used in the description of the memories 32 and 33 designates in each of the memories mentioned, as well a memory area of small capacity (some binary data such as the position of a point) that a memory zone large capacity (for storing an entire program or all or part of the data representative of the video sequence). The ROM 22 includes in particular a program "prog" 320. The algorithms implementing the steps of the method specific to the invention and described below are stored in the ROM 32. The microprocessor 31 loads and executes the instructions of these algorithms. The random access memory 33 comprises in particular: in a register 330, the operating program of the microprocessor 31; - encoded data 331 representative of a video sequence; trajectories for scattered distributions of points 332; and - parameters 235 of the series of homographies. The MMI interface is particularly adapted to display the video sequence before and after editing a plan. In addition, the input interfaces are adapted to the selection by a user of the points of the plan to be followed through the video sequence via a displayed graphical interface. In a variant, the points of the plane are automatically selected. Other structures of the device 3 than that described with reference to FIG. 3 are compatible with the invention. In particular, according to variants, the devices compatible with the invention are implemented in a purely hardware embodiment ("hardware" in English), for example in the form of a dedicated component (for example in an ASIC or FPGA or VLSI ) (respectively "Application Specific Integrated Circuit" in English, meaning "Integrated Circuit for a specific application", "Field Programmable Gate Array" in English, meaning "In-Situ Programmable Gate Network", "Very Large Scale" Integration "in English, meaning" very large-scale integration ") or several electronic components integrated in a device or in the form of a mixture of hardware and software elements (" software "in English).

Of course, the invention is not limited to the embodiments described above.

In particular, the invention is not limited to the described method of filtering trajectories, nor to the parametric movement model used in the described embodiments.

Claims (12)

REVENDICATIONS1. A method of editing a clip in a video sequence comprising a step (10, 20) of determining a series of transformations for each current frame of said video clip, a transformation comprising at least one parameter representative of the motion of said clip between a reference image and said current image, said method being characterized in that it comprises a time filtering step (11, 21, 22, 23) of said series of transformations. 2. The method according to claim 1, wherein said step (11) of temporal filtering comprises the determination of said at least one parameter representative of the movement of said plane between the reference image and the current image of said video sequence from a a weighted sum of said at least one parameter representative of the movement of said plane between the reference image and an image belonging to a determined set of images of a temporal window around the current image. 3. Method according to claim 2 wherein the series of filtered transformations is a function, for each current image, information representative of the quality of the transformation and function of information representative of the quality of the filtered transformation. 4. The method of claim 1 wherein said step (21, 22, 23) of temporal filtering comprises a step of filtering (22) temporally a trajectory representative of the movement of at least one point belonging to said plane between said image of reference and said successive current images and a step (23) for determining said at least one parameter representative of the movement of said plane between the reference image and the current image of said video sequence from said at least one point belonging to said plane in said reference image and at least one corresponding point in said current image where the position of the corresponding point in the current image results from said filtered path. 5. The method according to claim 4, wherein the step of temporal filtering of the transformation series comprises: a step (21) of determining a trajectory representative of the movement of at least one point belonging to said plane between said reference image and said successive current images; a step (22) of temporal filtering of the trajectory representative of the movement of said at least one point belonging to said plane between said reference image and said successive current images. 6. Method according to one of claims 4 to 5 wherein a trajectory associated with a point of the reference image comprises a set of motion vectors of said point between the reference image and said current images and in which the trajectory filtered filter representative of the movement of said at least one point belonging to said plane of a reference image is a set of motion vectors each corresponding to a weighted sum of neighboring spatial and temporal motion vectors of said trajectory. 7. Method according to one of claims 4 to 6 wherein the filtered path representative of the movement of said at least one point belonging to said plane of a reference image to said current image is a function of information representative of the quality of the transformation in the vicinity of said at least one point. 8. Method according to one of claims 1 to 7 wherein the step (11, 21, 22, 23) of temporal filtering of the series of transformations is repeated for a series of filtered transformations. 9. Method according to one of claims 1 to 8 wherein said transformation is a projective transformation, called homography, said transformation comprising 9 parameters representative of the motion of the projected plane of the three-dimensional space in the image plane. 10. Method according to one of claims 1 to 9 further comprising a step of interpolation or extrapolation of a transformation between the reference image and a current image absent from the series of transformations. 11. Method according to one of claims 1 to 10 further comprising a step of modifying the texture of the points belonging to said plane. 12. Device (3) implementing the method according to one of claims 1 to 11.