KR20020087957A - Preprocessing method applied to textures of arbitrarily shaped objects - Google Patents

Abstract

The present invention relates to a method of preprocessing corresponding to arbitrarily formed objects, comprising a texture portion and an object mask for each object, the method comprising: (1) two-dimensional object plane for each object plane associated with the object; Dividing into blocks, (2) introducing a set of basis vectors selected to represent an estimate of original pixel values as a linear combination of basis vectors, and (3) the original representation of pixel values and the original representation of Defining a cost function ψ to measure the distortion between the estimates, (4) finding coefficients that allow minimizing the cost function ψ, wherein the finding itself is limited to an initialization operation, opaque pixels According to the extraction operation of the basis vectors and the calculation operation of the projection coefficients, the iterations operation, and predetermined criteria Includes an abort operation of the iterations.

Description

Preprocessing method applied to textures of arbitrarily shaped objects}

The MPEG-4 standard, published in 1999, proposed a unified method for efficiently encoding natural objects and composite images. In an encoder that processes these objects (typically made up of several layers which can in turn contain any arbitrarily formed objects), they can consist of two components, binary or gray level pixels, an object mask representing the alpha channel values used by the decoder for the composition and, for example, pixels of the object (white pixels in the mask are layer opaque, with corresponding pixels in the texture part being opaque, Means to replace the pixels of the other object behind it, whereas a black pixel is a form of a texture part that is the values of the corresponding pixel in the texture part is completely transparent, i.e. invisible). Becomes More specifically, the present invention addresses the encoding operation of the texture portion.

To encode moving textures in an MPEG-4 encoder, the convolution method is to use a DCT transform (discrete cosine transform) on the picture blocks. More precisely, the plane to be encoded is divided into macroblocks of size 16 x 16 pixels, and the 16 x 16 luminance information is a two-dimensional 8 x 8 DCT transform (the same 2D transform is an 8 x 8 of 2 containing U and V color difference information). Is further divided into 4 8x8 blocks encoded by (encoded again for blocks). For arbitrarily formed objects, some 8x8 blocks may only contain transparent pixels (not required to encode texture information) or only opaque pixels (standard rectangular 8x8 DCT is used to encode texture information. Used), or at least three kinds, including at least opaque pixels and transparent pixels. The problem to be solved in this third situation is the efficient encoding of this partial texture information regarding bit consumption.

First, empty spaces are filled by extending texture boundary pixels (each sample at the boundary of an opaque area is copied horizontally in the left or right direction to replace with transparent areas), and then the textures are rectangular macroblocks. Typically can be DCT encoded. However, this padding method uses the frequency spectrum (they may be flat in the horizontal direction and may be changed randomly in the vertical direction, resulting in unnecessary frequency components that consume more bits when the macroblocks are DCT encoded). Patterns that cannot be optimal in terms of

Another solution normalized within the MPEG-4 standard is the so-called shape adaptive DCT undertaken in two steps of encoding the patterns of FIG. 1 (given as an example). As shown in FIG. 2, first all opaque pixels are shifted to the best position in the block to be encoded, and then an adaptive one-dimensional n-DCT is applied to each column, where n is the column (in the example of FIG. 2, From left to right, 1, 4, 7, 5, 7 and 1-DCT are applied in the vertical direction, respectively). Similarly, the resulting vertical DCT coefficients are then shifted to the leftmost position in the block that results in the pattern of FIG. 3, and similarly, one-dimensional n-DCT (where n is the number of opaque pixels in the highly regarded rows). Is applied to the row of. Unfortunately, according to this method requiring certain functions in the associated MPEG-4 decoder (as opposed to the typical 8x8 DCT algorithm used for completely opaque blocks), shift operations are typically spatially separated and thus extremely small. Since we connect the correlated coefficients or pixels, we introduce high frequencies.

The present invention relates to a method of preprocessing input data corresponding to image elements (pixels) representing arbitrarily formed objects, the input data being a texture part corresponding to the values of the pixels of the object for each object. And an object mask that subdivides the input data into first and second subsets of data corresponding to opaque and transparent pixels, respectively, completely or partially in the texture portion, wherein the A preprocessing method is provided for determining DCT (Discrete Cosine Transform) coefficients corresponding to the opaque pixels, and for each object considered,

(1) dividing the object plane into two-dimensional blocks,

(2) introducing the set of basis vectors selected to represent an estimate of original pixel values as a linear combination of basis vectors in the picture region defined by the block,

(3) defining a cost function ψ to measure the distortion between the original representation of the pixel values and the estimate of the original representation, and

(4) finding the coefficients that allow to minimize the cost function ψ. The invention of efficiently encoding randomly formed textures is particularly useful with respect to the MPEG-4 standard, but is not limited to this application.

1-3 illustrate prior art methods (shape adaptive DCT) used to encode texture pixels of an arbitrarily formed object.

4 shows a flowchart showing the main steps of the preprocessing method according to the invention.

It is therefore an object of the present invention to provide a preprocessing method that avoids the introduction of these undesirable frequencies and leads to better coding efficiency.

For this reason, the present invention relates to a method as defined at the beginning of the manual, moreover

(a) the cost function

Given as a relation of type,

Where f is a column-vector of pixels of the associated block, ((b _i ), iε (1 to 64)) are basis vectors of 8 × 8 DCT, and f _opaque is f for the _opaque pixels of the block. ((B _opaque ), iε (1 to 64)) is the limit of the basis vectors relative to the position of the opaque pixels of the block, Is called the reconstruction of f _opaque ,

(b) the finding step itself,

An initial estimation of the repetition parameter k = 0, f ^E _opaque = 0, initialization of the parameters including the initial reconstruction coefficients c ⁰ _i = 0,

Extraction of the basis vectors confined to the opaque pixels, and the next {} represents a cross correlation function, i varies from 1 to 64 and (b _opaque ) are projection coefficients, the limited basis vectors

p ^o _i = {(f _opaque -f ^E _opaque ), b _{opaque (i)} }

-Each one,

[a] sub-step of finding an index i * of the basis vector that contributes the most to minimize the cost function;

[b] sub-step of updating the reconstruction of f ^E _opaque according to the relationship f ^E _opaque (k + 1) = f ^E _opaque (k) + p ^k _i .b _{opaque (i)} ;

[c] c ^{k + 1} _i = c ^k _i , and c ^{k + 1} _{i *} = c ^k _{i *} + p ^k _{i *} and projection coefficients P ^{K + 1} _{i *} for the reconstruction coefficients i ≠ _{i *} An iterative (s) operation, which is provided to perform a sub-step of updating

-If the cost function ψ is below a given threshold, or if a predetermined number of iterations is reached, aborting the iterations.

The invention is now described by way of example in conjunction with the accompanying drawings.

The preprocessing method according to the invention is based on a typical 8x8 DCT transform, in order to use the existing decoder structure as defined in MPEG-4, but in relation to transparent pixels, while minimizing the number of nonzero coefficients. Better coding efficiency is provided by calculating DCT coefficients that reconstruct the missing opaque pixels. The method is described in "A projection onto the overcomplete basis approach for block loss recovery" by JHchange of Proceedings ICIP-97, et al., Vol. II, pages 93-96, Santa Barbara, California, October 26-29, 1997. Is an adaptation of the approach.

In the case of the original corrupted MPEG-4 video streams, in this case it is necessary to identify and reconstruct the corrupted picture blocks even when a small error spreads to a large number of blocks. According to this proposed adaptation of the method described in the above document, the base idea represents the estimate for the layer block as a linear combination of basis vectors, and overcomplete basis for finding projection coefficients of the basis vectors to minimize distortion measurements. By introducing the overcomplete basis, one obtains an estimate of the intact original pixel values from a series of intact values. Notations such as

D = damaged block

N = intact neighbor of the block

Using a combination of U = D and N (= larger block), it is proposed to estimate a larger block U containing the relevant corrupted block D from the intact neighboring information N. If f = (f _ij , (i, j)) εU represents the original pixel values i, jεN that are intact, the result is that the task estimates f. Since pixel values are known at N, a distortion measurement of the estimated f ^E of f can be expected, and the distortion measurement is defined by the following squared difference.

(b) If _l = (b ^l _{i, j} ) is the basis of U and the set of basis vectors is chosen to represent the original f as a linear combination of basis vectors, then due to intra-block correlation and taking into account some associated assumptions, The damaged block and its neighbors may have similar spectral characteristics. Therefore, projection coefficient

Is the original coefficient

May be the preferred estimate of.

Thus, if the coefficients a ' _ℓ S make f ^E _N = ∑ _ℓ a _ℓ b _ℓ is a preferred approximation of f _N , then F ^E _U = ∑ _ℓ a _ℓ b _ℓ is a good estimate of f _U (subscript N and U represent a range of vectors).

Next, the problem of reconstructing the damaged block is to find these coefficients a _ℓ 's that allow to minimize ψ as possible with the iterative algorithm described in the document. In accordance with the preprocessing method which is the object of the present invention, it now formulates a problem with changed notations and considerations.

f is a column vector of pixels of the macroblock to be encoded,

f _opaque is the limit of f for the opaque pixels of the macroblock,

B represents the basis functions of the 8x8 DCT transform; B = (b _i ), iε (1 to 64),

B _opaque = (b _{opaque (i)} , iε (1 to 64) represents the limit of these basis vectors to the position of opaque pixels.

Next, the problem is a cost function with the maximum number of zero coefficients,

By minimizing, we find a compact set of coefficients (ci) that maximize the f _opaque at the minimum mean square sense.

If macroblock f was completely opaque, there would have been a finite combination of DCT coefficients that could reconstruct all the pixels (rectangular 8 × 8 DCT). However, if only one wants to reconstruct a certain part of f, there is an infinite number of DCT coefficients that can reconstruct a block containing the same opaque pixels. In fact, the determination of the appropriate DCT coefficients (and the maximum compact set of coefficients) is not specified because the basis functions of the DCT transform are no longer orthonormal when limited to the positions of opaque pixels. In order to find the coefficients that minimize the cost function ψ, the following iterative algorithm is proposed (shown in FIG. 4) which continuously searches for the projection coefficient with the maximum energy:

(1) First step (initialization INIT):

k = 0 (number of repetitions)

f ^E _opaque = 0 (initial reconstruction coefficients)

(2) second stage (extraction sub-stage EXTR and computational sub-stage CALC):

The projection coefficients are calculated P ⁰ _i = {(f _opaque -f ^E _opaque ), b _{opaque (i)} }, where {} represents the correlation function, 1 varies from 1 to 64, and (b _{opaque (i )} ) Are extracted 8 × 8 DCT basis vectors that are confined to opaque pixels (pixels PWT with texture bounded by the shape mask SM).

(3) the third step (k iterations taking into account the estimate), each iteration and for example the k th iteration itself include the following operations, i.e.

(a) Finding (in sub-step FIND) the index i * of the basis vector that captures the remaining maximum energy:

i * = arg.max ∥p ^k _i .b _{opaque (i)} ∥ for ² i = 1 to 64

(b) updating the reconstruction of the f ^e _opaque (in the sub-stage UOEA) according to the following relationship:

f ^E _opaque (k + 1) = f ^E _opaque (k) + p ^k _i .b _{opaque (i)}

(c) updating the reconstruction coefficients c _i ^{k + 1} = c _i ^k for i ≠ i * and c _{i *} ^{k + 1} = c _{i *} ^k + p _{i *} ^k (in sub-step UPDA):

(d) Update the remaining projection coefficients (in sub-stage UPDA):

(4) the fourth step (interruption test or sub-step TEST of the iteration algorithm),

Of which the threshold ε is given,

Below, or when a predetermined number of iterations k _max is reached, stop the iteration process (answer YES to the test), otherwise continue the iteration of the third step (3) until none of these conditions are met. (No to the test). At the end of the algorithm implementation, c _i ^k are 8x8 DCT coefficients that generate opaque pixels of any form.

The preprocessing method described above is typically followed by texture encoding, i. Conventional operations may be provided in which variable length encoding and scanning are provided. Although the present invention has been described above with reference to specific examples of the MPEG-4 standard, it is to be understood that the present invention is not limited thereto. The invention is not limited to any particular coding strategy used in the resulting output bitstream.

Claims (4)

A method of preprocessing input data corresponding to picture elements (pixels) representing randomly formed objects, the input data comprising: for each object, a texture part corresponding to values of pixels of the object, and An object mask that subdivides input data into first and second subsets of data corresponding to completely or partially opaque pixels and transparent pixels in the texture portion, wherein the preprocessing method comprises the opacity Is provided to determine DCT (discrete cosine transform) coefficients corresponding to one pixel, and for each object considered (1) dividing the object plane into two-dimensional blocks, (2) introducing the set of basis vectors selected to represent an estimate of original pixel values as a linear combination of basis vectors in the picture region defined by the block, (3) defining a cost function ψ to measure the distortion between the original representation of the pixel values and the estimate of the original representation, and (4) finding the coefficients that allow to minimize the cost function ψ, wherein the method of preprocessing the input data comprises: (a) the cost function Given as a relation of type, Where f is a column-vector of pixels of the associated block, ((b _i ), iε (1 to 64)) are basis vectors of 8 × 8 DCT, and f _opaque is f for the _opaque pixels of the block. ((B _opaque ), iε (1 to 64)) is the limit of the basis vectors relative to the position of the opaque pixels of the block, Is called the reconstruction of f _opaque , (b) the finding step itself, An initial estimation of the repetition parameter k = 0, f ^E _opaque = 0, initialization of the parameters including the initial reconstruction coefficients c ⁰ _i = 0, Extraction of the basis vectors confined to the opaque pixels, and the next {} represents a cross correlation function, i varies from 1 to 64 and (b _opaque ) are projection coefficients, the limited basis vectors p ^o _i = {(f _opaque -f ^E _opaque ), b _{opaque (i)} } -Each one, [a] sub-step of finding an index i * of the basis vector that contributes the most to minimize the cost function; [b] sub-step of updating the reconstruction of f ^E _opaque according to the relationship f ^E _opaque (k + 1) = f ^E _opaque (k) + p ^k _i .b _{opaque (i)} ; [c] c ^{k + 1} _i = c ^k _i , and c ^{k + 1} _{i *} = c ^k _{i *} + p ^k _{i *} and projection coefficients P ^{K + 1} _{i *} for the reconstruction coefficients i ≠ _{i *} An iterative (s) operation, which is provided to perform a sub-step of updating A stop operation of the iterations if the cost function [psi] is below a given threshold or if a predetermined number of iterations is reached. The method of claim 1, The cost function is a relational expression, Prescribed by The sub-step [a] is residual The index i * of the basis vector that captures the maximum energy of i * = arg.max ∥ p ^k _i .b _{opaque (i)} ∥ ² , i = 1 to 64 A method for preprocessing the input data, characterized in that it is provided to find. A method of coding input data corresponding to a texture of randomly formed objects, the coding method comprising at least a DCT transform of the input data of the texture, quantization of the coefficients obtained from the transform, differential prediction of the data to be coded and the quantized 10. A method of coding input data, comprising a variable length encoding operation of coefficients, the method comprising: And the DCT coefficients are obtained by an implementation of the preprocessing method according to claim 1 and are coefficients c ^k _i corresponding to opaque pixels of the arbitrarily formed objects. An encoding device for coding input data corresponding to a texture of randomly formed objects, the encoding device comprising at least means for implementing a DCT transform of the input data of the texture, quantization of coefficients obtained from the transform, differential prediction of the data to be coded And a variable length encoding operation of the quantized coefficients, comprising: And the DCT coefficients are obtained by an implementation of the preprocessing method according to claim 1, characterized in that the coefficients c ^k _i correspond to opaque pixels of the arbitrarily formed objects.