FR2940576A1 - Method for video coding of sequence of digital images, involves coding intermediate images based on global motion compensation, and constructing intermediate image processing zone by fusion of global motion compensation key image zone - Google Patents

Abstract

The method involves coding intermediate images based on global motion compensation in front and rear directions from a key image. An intermediate image processing zone is constructed by fusion of global motion compensation key image zone or by conventional coding, where the fusion of global motion compensation key image zone or the conventional coding is chosen based on coherence measurement result between signals of global motion compensation key image zone. Independent claims are also included for the following: (1) a device for video coding of a sequence of digital images, comprising a coder (2) a device for video decoding of a sequence of digital images comprising a decoding unit.

Description

The invention relates to a video coding method based on global motion compensation and to a device implementing the method. It applies in particular to the fields of video transmission, analysis, decoding and transcoding.

A video sequence has by its very nature a significant statistical redundancy in both the temporal and spatial domains. The desire to use the bandwidth of the transmission media in which these sequences pass through and the objectives of reducing the cost of their storage always made the issue of video compression very early. Conventional video compression techniques can usually be divided into two stages. The first aims to reduce the spatial redundancy and for that to compress a still image. The image is first divided into blocks of pixels (4x4 or 8x8 according to, for example, the MPEG-1/2/4 standards), a passage in the frequency domain and then a quantization to approximate or delete the high frequencies at which the eye is less sensitive, and finally the quantized data are encoded entropically. The second aims to reduce temporal redundancy. This technique makes it possible to predict an image, called an intermediate image in the remainder of the description, from one or more other reference image (s) previously decoded within the same sequence. In other words, a motion estimate is made. This technique consists of searching in these reference images the block that best corresponds to the one to be predicted, and only a motion estimation vector corresponding to the displacement of the block between the two images is preserved, as well as a residual error allowing to refine the rendering. visual. In order to improve the coding performance, especially at low and medium bit rates, and therefore obtain for equivalent bit rates a better video quality of the decoded video, a technique called global motion compensation, designated in the following description by the acronym GMC from the Anglo-Saxon expression Global Motion Compensation, has been proposed. Many global motion models for video compression exist in the state of the art. This type of model has notably been introduced in the MPEG-4 Visual standard, also called MPEG-4 part 2 as well as in DivX or Dirac standards. In the developed approaches, for a given image, the global motion is estimated on the entire image or region, and between said image and its reference images with at least one global motion per reference image. The reference images compensated by the associated global motion then become possible candidates for temporal prediction, in the same way as the reference images compensated by non-global movements, that is to say with the conventional coding methods leading to block motion vectors. The interest of the use of the GMC is a significant reduction in the cost of the movement information on the areas of the image where it applies. The temporal prediction is also improved (pixel motion) with respect to a motion representation based on a block translation vector.

In a conventional encoding scheme, the intermediate images are usually called bidirectional images. In this type of scheme, the construction of GMC-based temporal prediction, even in the case of bi-directional prediction using a mixture of forward and backward prediction, the compensation usually only applies in one direction, before and / or rearward, which can generate temporal inconsistencies between the front and rear compensated versions and degrade the visual rendering by temporal fluctuations on the areas concerned. On the other hand, intermediate images referencing reconstructed areas from a global motion do not actually exploit this temporal inconsistency information between forward and backward predictions. Another disadvantage of the existing coding schemes is that the zones with global motion must be reported. This typically involves the need to code information for each block of the image. In addition, a coding residue is generally coded and if this is not the case, it must be reported to the decoder. It is also important to note that the decoding method based on the GMC technique is totally determined and not adaptable to the complexity of the terminal performing the decoding of the video stream.

An object of the invention is in particular to overcome the aforementioned drawbacks. To this end, the subject of the invention is a video coding method of at least one sequence of digital images, the images of said sequence possibly being intermediate images or key images used as references for the coding by motion compensation of the images. intermediate images. The intermediate images are zone-coded based on a global GMC motion compensation in the forward and reverse directions from the keyframes, the areas of the intermediate image being constructed either by merging the compensated keyframe areas into global motion. or by conventional coding, the choice between merging and conventional coding being carried out according to the result of a measurement of coherence between the signals of the zones of the compensated key images in global movement.

For example, the reference images are encoded before the intermediate images and at least one segmentation map associated with said images is calculated so as to be able to distinguish the GMC pixels from the other pixels of these images. Global motion parameters can be estimated and encoded prior to encoding the intermediate images. According to one aspect of the invention, motion compensated keyframes are derived from the keyframes using at least the global motion parameters. Segmentation maps associated with the moving compensated keyframes can be derived from the segmentation maps associated with the keyframes by transpositions using at least the motion estimation parameters. The intermediate image to be encoded as well as the motion-compensated key images used for its encoding are, for example, divided into processing zones, the processing zones of the intermediate image to be encoded corresponding to the compensated key image processing zones. moving. According to one embodiment of the invention, the processing areas of the motion compensated key images are classified according to their proportion of GMC pixels, said proportion being compared with a threshold r1 between 0 and 1, an area being classified as GMC when said proportion is greater than i and classified non-GMC otherwise. According to another embodiment of the invention, the proportion of GMC pixels per zone of the keyframes compensated in motion is deduced from the segmentation maps. If at least one area of one of the motion compensated images used as references for encoding the area to be encoded of an intermediate image is classified as non-GMC, conventional coding of said area may be performed.

According to another aspect of the invention, if the zones of the motion compensated images used as references for the coding of a zone of an intermediate image are classified as GMC, the coherence of said zones is analyzed by comparison of the signals of the zones of the images. compensated keys in global movement.

The areas of the keyframes compensated in motion by taking into account the global compensation parameters whose coherence must be analyzed are, for example, compensated in motion a second time using a translation vector of a precision at pixel close. The translation vector of the second motion compensation can be calculated using a block matching estimator. According to one embodiment, the mean squared error D of the zone to be coded is computed and is compared with a predefined threshold so as to distinguish the low local gradient zones from the high local gradient zones, the zone being considered at low gradient local and classified as coherent if D is lower than and considered to be a high local gradient in the opposite case. An upper bound S of the mean square error D is computed, for example, by using the local gradient values of the current area and the mean squared error D is compared to the said terminal S, the current area being classified as consistent when D is less than this bound and not consistent otherwise.

When the zone to be coded is classified as coherent, the merging of the corresponding zones of the moving compensated key images can be performed. The merge is performed, for example, using a Graph-cut algorithm. According to one embodiment, when the zone being processed is classified as non-coherent, the conventional coding of said zone is carried out. The invention also relates to a video encoding device of at least one digital image sequence, the images of said sequence possibly being intermediate images or keyframes used as references for the coding by motion compensation of the images. intermediate. The coding device includes means for encoding the intermediate images per processing area based on a global GMC motion compensation in the forward and reverse directions from the keyframes, the processing areas of the intermediate image being encoded either by merging the corresponding areas of the keyframes, or by conventional coding and to automatically choose between fusion and conventional coding by analysis of the area to be coded. The invention also relates to a device for video decoding of at least one sequence of digital images previously coded using the coding method according to the invention, the images of said sequence possibly being intermediate images or key images. used as references for motion compensation decoding of intermediate images. The intermediate images are decoded by zone based on a global GMC motion compensation in the forward and reverse directions from the decoded keyframes, the regions of the intermediate image being reconstructed either by merging the zones of the compensated keyframes in motion. overall, or by conventional decoding, the choice between conventional merging and decoding being performed according to the result of a measurement of coherence between the signals of the zones of the compensated key images in global movement.

The invention has the particular advantage of improving the coding performance by reducing the required bit rate while potentially improving the visual quality of the coded / decoded video sequence. In sequences where only a few foreground objects move in the scene, the use of this method induces a reduced significance of the compressed video stream rate compared to existing techniques. On the other hand, visual artifacts due to temporal fluctuations of the signal on these areas are limited by the use of the method.

Other features and advantages of the invention will become apparent from the following description given by way of nonlimiting illustration, with reference to the appended drawings in which:

FIG. 1 illustrates the principle of temporal dependence between key images and intermediate images; FIG. 2 gives an exemplary implementation of the coding method according to the invention; FIG. 3 presents a method for testing the coherence of an area on two different images.

FIG. 1 illustrates the principle of temporal dependence between reference images, called keyframes in the following description, and intermediate images. The example of the figure considers a group of images, usually designated by the acronym GOP from the Anglo-Saxon Group of Picture, consisting of two key images ICO, ICI and framing one or more intermediate images INT. The coding method according to the invention operates on processing zones which may correspond, for example, to one or more blocks or macroblocks. GOP ICO and ICI keyframes are encoded first. The coding is done according to a conventional approach, the GMC-based coding tools can also be implemented. Thus areas of a keyframe may be encoded or referenced, with GMC prediction and others not. It is then possible to deduce at the level of the coder and the decoder a binary segmentation map indicating whether a zone, and thus the pixels that compose it, is of GMC type or not. The invention relates in particular to the coding of intermediate images INT. For an area to be encoded of a given intermediate image, it is assumed for the rest of the description that the forward and backward global motion parameters denoted GMO and GM1 have been previously estimated and coded. It is also assumed that the key images ICO and ICI are reconstructed to serve as reference images, the intermediate images already coded may also be available as reference images. Finally, a binary segmentation map is calculated for each image by the decision module of the encoder and indicates for each pixel of the reference image whether it is of GMC type or not.

FIG. 2 gives an example of implementation of the coding method according to the invention. The encoding method can be broken down into several steps. A first step 200 performs the GMC global motion compensation of the key images ICO and ICI. Their associated segmentation maps SO and S1, determined beforehand by a decision module of the encoder, or by decoding coding modes at the decoder, are used as input, as well as the GMO and GM1 motion parameters previously determined by a module. The images IGO and IG1 are then obtained, the latter respectively corresponding to the images ICO and ICI compensated in motion according to the motion models GMO and GM1. Moreover, two segmentation maps SGO and SG1 associated with the images IGO and IG1 are transposed from the segmentation maps SO and S1 by motion compensation according to the motion models GMO and GM1. The intermediate image to be encoded is divided into zones of treatment. This cutting can be automatic or adaptive. For example, a processing area may correspond to a macroblock of the image to be encoded. A succession of steps is then applied for each of the zones of the intermediate image to be encoded. In the remainder of the description, the zone being processed is called the current zone. A classification 201 of the corresponding zones is carried out based on the segmentation maps SGO and SG1. Each zone is associated, for example, with one of two possible classes. Each of these classes respectively identifies a GMC or non-GMC zone. Thus, for the current field, a CO variable associated with IGO carries this class information. In the same way, a variable C1 is associated with IG1. By way of example, the two classes can be defined by counting the proportion of GMC-classified pixels of the image in the area considered and comparing this proportion to a given threshold n between 0 and 1, using Segmentation maps SGO and SG1. It is also possible not to use the SO, S1, SGO and SG1 cards. In this case, CO and Cl are, for example, systematically considered as GMC.

It is then checked 202 if CO and Cl are of type GMC. In this exemplary implementation, if CO or Cl are not GMC type, a conventional coding 203 of the zone is performed. This conventional coding may be, for example, spatial prediction type, one-way time prediction or bidirectional time prediction. Conventional coding may still employ GMC prediction, but this will be one of many modes to be reported to the decoder in the bitstream. When CO and Cl are of GMC type, the coherence in the considered zone of the images IGO and IG1 is tested 204. The notion of coherence between image zones is detailed later in the description. If the contents of said images are considered coherent, the signal is generated by merging 205 of the treated area of IGO and IG1, which implies that no information needs to be encoded. The zones constructed by fusion do not require the coding of any additional information and therefore correspond to a zero coding cost, which is obviously very advantageous if said zones are numerous. The encoding mode thus implemented is an implicit mode, which does not require any signaling information. This coding mode, tested on the encoder side, is also tested on the decoder side. The decoder is then itself capable, without signaling information, of knowing whether the current zone is constructed according to this implicit GMC coding mode or not. If the contents are substantially different, and therefore the coherence is not verified, a conventional coding 203 of the zone is performed. GMC prediction can still be used as one of the possible prediction modes.

The intermediate image encoded using the method according to the invention is therefore composed of predicted zones 207 that can not be used as a reference for the coding of other intermediate images and reconstructed zones 206 that can, for its part, be used as references for coding other images.

The encoding method, described above encoder side, can be applied symmetrically to the decoder. The ICO and ICI keyframes of the GOP are decoded first. The decoder, from the decoded encoding modes of the keyframes, constructs for each key image ICO and ICI a binary segmentation map SO and S1 indicating whether a zone, and therefore the pixels that compose it, is of the GMC type or not. The invention relates in particular to the decoding of intermediate images INT. For an area to be decoded from a given intermediate image, the global forward and backward motion parameters denoted GMO and GM1 are previously decoded. The decoding method can be decomposed into several steps. A first step performs GMC global motion compensation of ICO and ICI keyframes. The images IGO and IG1 are then obtained, the latter respectively corresponding to the images ICO and ICI compensated in motion according to the motion models GMO and GM1. In addition, two segmentation maps SGO and SG1 associated with the images IGO and IG1 are transposed from the segmentation maps SO and S1 by motion compensation according to the motion models GMO and GM1. The intermediate image to be decoded is divided into processing zones. This cutting can be automatic or adaptive. For example, a processing area may correspond to a macroblock of the image to be encoded. A succession of steps is then applied for each of the zones of the intermediate image to be encoded. A classification of the corresponding zones is carried out based on the segmentation maps SGO and SG1. It is then checked if CO and Cl are of type GMC. In this exemplary implementation, if CO or Cl are not of GMC type, a decoding of the coding information (coding mode, associated parameters - for example intra prediction direction, motion vectors - prediction residue) for the current area is performed. This information must then be present in the bit stream. When CO and Cl are of GMC type, the coherence in the zone 5 considered of the images IGO and IG1 is tested. If the contents of said images are considered coherent, the signal is generated by merging the processed area of IGO and IG1, which implies that no information needs to be decoded. Fusion-built zones do not require the decoding of any additional information. If the contents are substantially different, and therefore the coherency is not checked, a decoding of the coding information is performed. This information must then be present in the bit stream.

Figure 3 presents a method for testing the coherence of an area on two different images. The notion of coherence between images has been discussed with the previous figure. The coherence measurement of the two signals IGO and IG1 in the current zone can be done by conventional distortion measurements, such as the mean squared error. However, because of the possible limitations of the overall motion estimator and the quantization required in encoding the global motion parameters, the IGO and IG1 signals will never be perfectly aligned and a slight offset is very likely to occur. appear even if both signals are considered consistent. This shift can be all the more important as the motion model moves away from a translational model, that is to say from a model where all the pixels of a zone move according to the same vector. In this case, the motion depends on the position of the pixel. When one is far from the point of origin, a small error of a non-translational component of the model will result in a large deviation of the motion vector from the model. The mean squared error alone does not take into account this possible shift. In order to take this shift into account, an approach is proposed in the context of the invention. A first step 300 has the particular objective of a pixel registration of IG1 by local motion compensation of IG1 relative to IGO. This compensation is done with a translation vector, with a pixel accuracy, and with a maximum excursion excmax limited. The excursion is for example 2 or 3 pixels. To do this a classic block-matching estimator can be used. This type of algorithm aims to find a vector that minimizes the mean squared error. This is implemented in order to correct the large discrepancies due to the errors of the motion model. In a second step, the mean squared error is calculated 301 on the current area. This error D can be expressed with the following expression:

D = I (IGO [p] -IG1, n [p]) 2 (1) pE Z

in which Z designates the zone considered, p a pixel and IG1 mc the motion compensated image of IG1. It is possible to integrate in the equation (1) the variation of the means, which leads to the following expression:

D = I (IGO [p] -, to-IG1 ,, [p1 +, t1) 2 (2) pE Z 20 where 1.10 and g1 are the estimated average luminance values of IGO and IG1 mc on the current area Z. This estimate is followed by a direct signal comparison for low local gradient areas. If D is less than a predefined threshold X, IGO and IG1 are considered consistent on the zone. The threshold may, for example, take the value 52xNz, where Nz is the number of points in the current zone Z. This implies, in this case, that an inter-signal mean deviation of 5 is tolerated. If the previous test 302 is negative, a measurement of the local gradient is performed 303, and this for the zones with high local gradient. The high value of D 30 may be due, for example, to a slight shift, less than the pixel, of a textured zone and therefore with strong gradients. If the two signals are coherent, IG1 mc can be expressed for any pixel p in the current zone with the expression: IG1mc [p] IGO [p + 8] (3)

where 8 = (8x, 8y) is a vector whose two components 8x and 8y are of amplitude less than 1, since a pixel registration has already been done.

It is then possible, after developing Taylor of equation (3) and considering expression (2) to determine an upper bound S of D whose expression is: 2 ~ <S = aIGO [p] 2 + alGO [p] + 2 pE Z ax 'pE Zay i CaIGO [p] aIGO [p] ax ay ~ pE Z

(4) The local gradients are thus calculated 303, then the sum S is compared 304 to D. If D is less than or equal to s, IGO and IG1 are considered coherent on the current zone Z. Otherwise, IGO and IG1 are considered inconsistent on the current area.

Certain parameters such as exc max and intervening in the algorithm can be coded and transmitted to the decoder. It has been seen with the example of Figure 2 that when the compared area is considered consistent, a merger of the two signals IGO and IG1 can be considered. The fusion algorithm aims to mix the two signals satisfactorily, that is to say without displaying echoes due to the slight spatial shift mentioned above. One solution is to use seamless plating algorithms, such as Graph cut. An example of this type of technique is described in the article by Vivek Kwatra et al. titled Graphcut Textures: Image and Video Synthesis Using Graph Cuts, Proc. ACM Transactions on Graphics, Siggraph'03. These algorithms make it possible to assemble plots of texture by limiting visual artifacts of apparent stitching type.

Claims (19)

CLAIMS1- A method for video coding at least one sequence of digital images, the images of said sequence possibly being intermediate images (INT) or keyframes (ICO, ICI) used as references for the coding by motion compensation of intermediate images (INT), the coding method being characterized in that the intermediate images (INT) are zone-coded based on a global motion compensation GMC (200) in the forward (GM1) and backward (GMO) directions from the keyframes (ICO, ICI), the areas of the intermediate image (INT) being constructed either by merging (205) the areas of the global motion compensated keyframes, or by conventional encoding (203), the choice between fusion and conventional coding being performed (201, 202, 204) according to the result of a measurement of coherence between the signals of the zones of the compensated key images in global movement. 2- video coding method according to claim 1 characterized in that the reference images (ICO, ICI) are encoded before the intermediate images and at least one segmentation map (SO, Si) associated with said images is calculated to be able to distinguish the GMC pixels from the other pixels of these images. 3- video coding method according to any one of the preceding claims characterized in that the global motion parameters (GMO, GM1) are estimated and encoded (208) before the coding of the intermediate images (INT). 4- Video coding method according to claim 3 characterized in that key images compensated in motion (IGO, IG1) are derived (200) keyframes (ICO, ICI) using at least the global movement parameters (GMO , GM1). 5- video coding method according to claim 4 characterized in that segmentation maps (SGO, SG1) associated with the motion-compensated keyframes (IGO, IG1) are deduced from the segmentation maps (SO, Si) associated with the keyframes ( ICO, ICI) by transpositions using at least the motion estimation parameters (GMO, GM1). 6. Video coding method according to any one of claims 4 or 5 characterized in that the intermediate image (INT) to be encoded and the keyframes compensated in motion (IGO, IG1) used for its encoding are divided into zones of processing, the processing zones of the intermediate image (INT) to be encoded corresponding to the processing areas of the keyframes compensated in motion (IGO, IG1). 7- video encoding method according to any one of the preceding claims characterized in that the processing areas of the keyframes compensated in motion (IGO, IG1) are classified (CO, Cl) according to their proportion of GMC pixels, said proportion being compared with a threshold r1 between 0 and 1, an area being classified (201) GMC when said proportion is greater than i and classified non-GMC otherwise. 8- Video encoding method according to claim 7 characterized in that the proportion of GMC pixels per area of the key images compensated in motion (IGO, IG1) is deduced from the segmentation maps (SGO, SG1). 9- video encoding method according to any one of claims 7 or 8 characterized in that if at least one area of a motion compensated images (SGO, SG1) and used as references for coding the area to be encoded an intermediate image (INT) is classified non-GMC, a conventional coding of said zone is performed. 10- video encoding method according to any one of claims 7 to 9 characterized in that if the areas of the images compensated in motion (SGO, SG1) used as references for coding an area of an intermediate image (INT) are classified GMC, the coherence of said zones is analyzed (204) by comparison of the signals of the zones of the compensated key images in global movement. 11- video coding method according to claim 10 characterized in that the areas of the keyframes compensated in motion (IGO, IG1) by taking into account the global compensation parameters (GMO, GM1) whose consistency must be analyzed are compensated moving a second time (300) using a translational vector with pixel accuracy. 12- video encoding method according to claim 11 characterized in that the translation vector of the second compensation (300) of movement is calculated using a blocking type estimator. 13- video encoding method according to any one of claims 10 to 12 characterized in that the mean squared error D of the zone to be encoded is calculated and is compared with a predefined threshold (302) so as to distinguish the zones to weak local gradient of zones with high local gradient, the zone being considered at low local gradient and classified as coherent if D is lower than and considered to be a high local gradient in the opposite case. 14. The video coding method as claimed in claim 13, characterized in that an upper bound S of the mean squared error D is calculated by using the values of the local gradients of the current zone and that the mean squared error D is compared ( 304) at said terminal S, the current area being classified coherent when D is less than this bound and not coherent in the opposite case. 15- Video coding method according to any one of claims 13 or 14 characterized in that when the area to be coded is classified as coherent, the merging of the corresponding areas of the keyframes compensated in motion (IGO, IG1) is performed. 16- video encoding method according to claim 15 characterized in that the fusion is performed using a Graph cut algorithm. 17- Video coding method according to any one of claims 15 to 16 characterized in that when the zone being processed is classified as non-coherent, the conventional coding of said zone is carried out. 18- Video coding device of at least one sequence of digital images, the images of said sequence possibly being intermediate images (INT) or key images (ICO, ICI) used as references for the coding by motion compensation of intermediate images (INT), the coding device being characterized in that it comprises means for: - coding the intermediate images (INT) by processing zone based on a global compensation of GMC movement in the forward directions (GM1 ) and back (GMO) from the keyframes (ICO, ICI), the processing areas of the intermediate image (INT) being encoded either by merging the corresponding areas of the keyframes or by conventional coding; - automatically choose between fusion and conventional coding by analysis of the zone to be coded. 19- video decoding device of at least one sequence of digital images previously coded using the method according to any one of claims 1 to 17, the images of said sequence can be intermediate images (INT) or key images; (ICO, ICI) used as references for motion compensation decoding of the intermediate images (INT), the decoding device being characterized in that it comprises means for decoding the intermediate images (INT) by zone based on a overall GMC motion compensation in the forward (GM1) and backward (GMO) directions from the decoded keyframes (ICO, ICI), the areas of the intermediate image (INT) being reconstructed either by merging (205) the zones of the compensated key images in global motion, either by conventional decoding (203), the choice between conventional fusion and decoding being performed according to the result of a coherence measurement between the areas of compensated keyframes in global motion.