JP4470440B2 - Method for calculating wavelet time coefficient of image group - Google Patents

Description

In the present invention, each decomposition level includes an n decomposition level including a function M (a _i , b _i ) and a function D (a _i , b _i ) of each input signal pair a _i , b _i. The present invention relates to a method for calculating a wavelet temporal coefficient of a GOP (Group Of Pictures: image group) having a length of 2 ⁿ by recursively applying a time transformation that generates the image. The technique used is based on wavelets and is for digital video coding.

  An image is a matrix of numerical values, and each numerical value represents a color intensity or luminance at a pixel, a value calculated by an algorithm, or the like. If the image is encoded in bytes, each number will contain a value from 0 to 255. An image is called a frame when it represents a matrix of color intensity or luminance at the pixels.

  A digital video signal consists of a series of frames. By displaying a predetermined amount (usually 25 or 30) of frames every second, the mode of operation is shown to the observer. In this sense, the digital video signal is actually a 3-D signal (three-dimensional). Here, the two dimensions represent the image plane (single frame) and the third dimension represents time (a series of consecutive frames (FIG. 1)).

  In order to efficiently compress such a 3-D signal, it is necessary to utilize the redundancy of both space (2-D) and time (3-D). Wavelets have been widely used to exploit the spatial redundancy of video sequences, for example by applying a Haar transform.

  The Haar transform is well known in wavelet theory and is one of the least supported wavelet transforms. Given two input values A and B, the corresponding Haar coefficients are simply their half difference Δ and their mean μ.

In the sense of video coding, Haar transform has been used for both spatial and temporal transforms. In the second case, temporal decomposition is applied to a 2 ⁿ size GOP (image group). The input data may be raw images of video sequences (luminance and chrominance values), or they may be spatially resolved (transform coefficients) using any 2-D linear transform.

When Haar transform is applied as time transform, the scheme shown in FIG. 2 is obtained. Here, the input is represented by _four images (F ₁ , F ₂ , F ₃ , F ₄ ) generated by 2-D Haar transform of the frames of the input sequence.

For a given F ₁ and F ₂ , the corresponding temporal Haar transform consists of an average image μ ₁ and a difference image Δ ₁ , where μ ₁ and Δ ₁ represent the Haar transform of F ₁ and F ₂ . The Haar transform can be applied recursively to generate several levels of decomposition. In the example of FIG. 2, the Haar transform is first applied to the images (F ₁ , F ₂ ) and (F ₃ , F ₄ ), and then applied to the two average images (μ ₁ , μ ₂ ). Two levels of decomposition are shown. The corresponding Haar transform is represented by Δ ₁ , Δ ₂ , Δ ₃ , μ ₃ . It should be noted that Haar decomposition is very efficient for video coding applications when the input images are similar to each other. In this case, since the difference image is close to zero, it is easy to compress with the entropy encoder.

  In Haar transform, the decomposition is applied equally to all inputs for each decomposition.

  The present invention proposes a method for calculating the wavelet time coefficient of a GOP that can efficiently utilize the temporal redundancy of the conversion coefficient calculated by any 2-D linear conversion.

In the method according to the present invention, at the later n-1 decomposition levels, the transform block at each decomposition level corresponds to the sum of two functions D (a _i , b _i ) output from the previous decomposition level. And the corresponding function M (a _i , b _i ) at the previous decomposition level is an input signal to the subsequent conversion block, and when the control signal is zero, The output values are functions M (M (a _i , b _i ), M (a _{i + 1} , b _{i + 1} )) and D (M (a _i , b _i )) of the input signal M (a _i , b _i ). ), M (a _{i + 1} , b _{i + 1} )), and when the control signal is not zero, the output signal is the input signal.

According to a preferred embodiment, the function M (a _i , b _i ) is the average value of the signals a _i , b _i and the function D (a _i , b _i ) is the average of the signals a _i , b _i . The difference is quantized.

  The proposed technique preferably takes advantage of the temporal redundancy of the transform coefficients calculated by 2-D linear transformation using a novel method that the inventors call “dynamic time transformation”. Unlike other solutions, this technique does not require a motion estimation procedure, which is a computationally expensive task.

According to one embodiment, an input sequence of 2 ⁿ frames is first converted individually into 2 ⁿ image GOPs with a 2-D linear transformation, and each image of the GOP has a transform coefficient for each input frame. And the transform coefficients are passed to a time transform that produces a first level temporal decomposition, each decomposition including a time average and a time difference, and at each subsequent decomposition level of n-1 stages, the control signal is The sum of the two time average differences output from the previous decomposition level is quantized.

  The “dynamic time conversion” proposed for the present embodiment is applied to the result of 2-D linear conversion, and is not applied to an unprocessed image. This choice has advantages with respect to encoder complexity.

  According to another embodiment, the input value to the first decomposition level is a raw image and the control signal is encoded and quantized in the two signals output from the previous level, and the quantization is It is the sum of the average differences of the restored and decoded images.

  According to a preferred embodiment, the 2-D linear transform is a 2-D 5.3 wavelet transform, the low-pass filter is a 5-tap filter, the high-pass filter is a 3-tap filter, The time conversion is a Haar time conversion.

  The present invention is presented with the accompanying drawings. Here, the 2-D linear transformation and the time transformation are Haar transformations.

  FIG. 3 shows a schematic scheme of the technique proposed by the first embodiment.

The proposed “dynamic time conversion” is applicable to any GOP of length 2 ⁿ . In FIG. 3, the GOP has a length of 4 (2 ² ). Initially, these images are individually transformed with 2-D wavelets (a parallel implementation may be suitable for this step). As a result, four spatial decompositions containing the wavelet coefficients of each input image are generated. Each decomposition includes the same number of coefficients as the pixels in the input image. These coefficients are then passed to a “dynamic time conversion” that generates four time resolutions. Again, each decomposition includes the same number of coefficients as the pixels in the input image. As will be explained in more detail below, one of these decompositions is a time average and three are time differences.

The length of GOP (2 ⁿ ) determines the number of time resolutions (n). As n increases, temporal redundancy can be utilized more effectively. However, since n also defines the delay due to the encoder, in an actual application example, n cannot be made too large.

  The proposed method defines a procedure for dynamically determining which input coefficients are further transformed and which are simply transmitted.

The defined scheme is shown in FIG. Here, the procedure covers eight input images I _{t to} I _t-7 . The scheme shows that at the first level, the standard Haar transform described above (FIG. 2) is applied to all inputs. The result is two images. The upper side is the average μ, and the lower side is the difference Δ. Q obtained by quantizing the difference image is used as a control signal for the next decomposition level. On the other hand, the average image is one of the inputs for the controlled transform block at the next decomposition level.

The output of the control conversion block is the average and difference of the input signals or directly becomes the input signal according to the control signal. FIG. 5 shows the detailed mechanism of the control conversion block. The input signals are C _n (x, y, t ₁ ) and C _n (x, y, t ₂ ), and _n at respective subband positions x and y at times t ₁ and t _2. Represents the wavelet coefficients generated by the th decomposition.

  Two cases are shown. In the first case, the control signal Q is zero. In this case, the output signal is simply a Haar transform of the input (roughly speaking, the average and difference of the input signal). In the second case, the input signal Q is not zero. In this case, the output signal is an input signal directly.

To illustrate the proposed "dynamic time conversion" mechanism, two examples are presented in FIG. In the first example, a standard Haar transformation mechanism is shown. The input signal is a piece-wise constant that simulates an intensity discontinuity (eg, a moving object) in the time domain. Eight (2 ³ ) input values are decomposed in three stages. At each stage, from the two input signals, two coefficients are obtained as the average of the input and half of the difference (in this example, the input 0,0 becomes 0,0 as the average and half of the difference). , And other inputs 0, 4 are converted to 2, 2). The eight input signals are represented by eight coefficients after three stages of decomposition. One of them is the last calculated average value (5 in this example), and the other seven are calculated differences (3, 2, 0, 0, 4, 0, 0 in this example). is there. From these coefficients, it is possible to reconstruct the input value.

  The proposed "dynamic time conversion" was applied to the same input signal. Differences from the standard Haar transform occur at the second level of decomposition. Here, the input signals 0, 4 are not 2, 2, but 0, 4. This is because the control signal for this conversion block is not zero (in FIG. 6, the non-zero control signal is a dotted line). Because of these differences, the following eight output coefficients are obtained after three levels of decomposition. That is, 0 is the final average, and 8, 4, 0, 0, 4, 0, 0 are all differences. When the two-step quantization is applied to the result obtained by the standard Haar transform, the following coefficients 2,1,1,0,0,2,0,0 are obtained. As a result, 0,0,0,8,6,6,6,6 are obtained after decoding. Similarly, 0, 4, 2, 0, 0, 2, 0, 0 are obtained for the coefficients obtained by applying “dynamic time conversion”, and after decoding, 0, 0, 0 , 8, 8, 8, 8, 8 are obtained. This exactly matches the input. This means that “dynamic time conversion” in the above example is a better coding method than the standard Haar transform.

The proposed “dynamic time transform” significantly improves the performance of the standard Haar transform in the context of video coding. The main advantage is that, at a given rate, the proposed “dynamic time conversion” does not result in troublesome artifacts such as ghosts around moving objects. On the other hand, this happens in the standard Haar transform. Such resistance to artifacts makes it possible to increase the length of the GOP and better utilize temporal redundancy in the input signal. In a standard Haar transform, the presence of artifacts when moving objects are in the scene, the length of the GOP, are limited to 2 ^4. This restriction affects the encoding performance of a standard Haar transform.

  Another important advantage of the proposed implementation is that it can reduce complexity compared to standard approaches such as MPEG-2 / 4. Indeed, the proposed encoding process does not require a preceding decoding procedure in the encoder to take advantage of temporal redundancy.

  The proposed implementation in “Dynamic Time Conversion” does not bring any drawbacks. Since the transformation is symmetric and reversible, there is no need to send any additional control signals to the decoder.

  The proposed “dynamic time conversion” has a major application area in the compression of video surveillance signals. The reason is as follows.

1. In security videos, most of the scene remains fixed. Thus, exploiting temporal redundancy for long GOPs has a significant effect on compression performance. The use of standard Haar transforms for long GOPs is limited by ghost artifacts. This is corrected by using the proposed “dynamic time conversion”.
2. In security applications, real-time constraints are very strong. The proposed “dynamic time conversion” can be easily implemented in hardware because the complexity of computer processing is very low compared to the standard method of MPEG-2 / 4.

  On the other hand, the proposed “dynamic time conversion” can also be used in other application areas that require high computer processing capacity and static scenes are compressed. Examples include video telephony, video forums, video conferencing and the like.

  The method described in the above embodiment is applied to the domain of linear transformation. For example, it is applied to transform coefficients such as wavelet coefficients. However, according to another embodiment of the present invention, it can be expanded and generalized to an image region, and therefore can be applied to color information of an image. In this sense, instead of applying a “dynamic time transform” to the transform domain (eg, the wavelet coefficients shown in FIG. 2), a dynamic Haar transform is applied to the image domain (eg, color intensity, regardless of format). 7. It is possible to apply to RGB, YUV, YcbCr,.

Note that this generalization requires that the encoder be able to access the same information that is also available to the decoder. For this reason, the method of FIG. 4 needs to be generalized as shown in FIG. To make this scheme easier to understand, only a single level of decomposition is shown as an example. Here, the input I _t −I _t−3 is expressed by an unprocessed input image instead of the 2-D conversion as shown in FIG. 4.

  In this generalization, the control transformation block remains as shown in FIG. As described above, two images are generated by the control transform block of the first decomposition. That is, the average difference Δ and the average μ. Δ is encoded using any available encoder (in FIG. 8, the result of encoder 1 is referred to as δ). Δ is decoded using the corresponding decoder 1. The result is Δ ′ approximating the difference between the input images. It will be appreciated that this frame will also be available at the decoder.

  In the example of FIG. 8, two images Δ ′ are obtained with the first decomposition level. These sums at the pixel level become control signals for the subsequent decomposition levels. The control transform block is encoded, quantized, quantized, and encoded average difference and average of the decoded image according to the information whether the difference of the decoded image is 0 or not 0 Or whether to transfer the actual value without time encoding. This selection is made at the pixel level and there is no need to send additional information from the encoder to the decoder. This is because they both get their decisions from the same data. The acquired images μ and Δ are individually encoded by an arbitrary encoder to generate two streams Ω and δ. Finally, the streams sent to the decoder are Ω, which is the final average μ encoded, and three δ streams corresponding to the three average differences encoded. In this way, the decoder can reconstruct the control signal Δ ′ from the corresponding δ.

  In the MPEG-4 standard, a similar result can be obtained by transmitting information on whether a block is encoded internally or externally. In this case, the selection is made at block size resolution and at the additional cost of sending instructions to the encoder.

  While illustrative embodiments in the present invention have been shown and described, a wide variety of modifications, changes and substitutions are contemplated in the above disclosure and some examples, and some features of the invention are It can be used without combining other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the invention.

It is a figure showing the video sequence considered as 3-D signal. It is a figure which shows the system of Haar time decomposition. It is a figure which shows the schematic system of the method of this invention. It is a figure which shows the system of the method of this invention. It is a figure showing a control conversion block. FIG. 4 is an illustration of decomposition according to a standard Haar transform and a transform of the present invention. It is a figure which shows the system of the time Haar transformation applied to the image area | region. FIG. 3 represents the method of the present invention applied to an image area.

Claims (6)

A method of calculating a wavelet time coefficient of a GOP (image group) having a length of 2 ⁿ by recursively applying a time transformation that generates n stages of decomposition levels, wherein each decomposition level corresponds to each input signal In the method including the function M (a _i , b _i ) and the function D (a _i , b _i ) of the pair a _i , b _i , at the later n−1 stage decomposition level, The transformation block is controlled by a control signal corresponding to the sum of two functions D (a _i , b _i ) output from the previous decomposition level, and the corresponding function M (a _i at the previous decomposition level. , B _i ) is an input signal to the transform block, and when the control signal is zero, the output value of the transform block is a function M (M (a _i ) of the input signal M (a _i , b _i ). _{, b i), M (a} i + 1, b i + 1)) and D (M (a _{i b i), M (a i} + 1, b i + 1)) , and when the control signal is not zero, the method in which the output signal is characterized by the said input signal. The function M (a _i , b _i ) is the average value of the signals a _i and b _i , and the function D (a _i , b _i ) is a quantized version of the average difference between the signals a _i and b _i. The method of claim 1 wherein: An input sequence of 2 ⁿ frames is first converted individually into G ⁿ of 2 ⁿ images by an arbitrary 2-D linear transformation (wavelet, DCT,...), And each image has a conversion coefficient of each input frame. And the resulting spatial transform coefficients are passed to a time transform that produces a first level temporal decomposition, each decomposition including a time average and a time difference, followed by each decomposition level in n-1 stages. 3. The method according to claim 2, wherein the control signal is a sum of two time average differences output from the previous decomposition level quantized.   The input value to the first decomposition level is an unprocessed image, and the control signal is encoded and quantized in two signals output from the previous level, quantized, decompressed, and decoded image. 3. The method of claim 2, wherein the sum is the sum of the average differences.   4. The 2-D linear transform is a 2-D · 5.3 wavelet transform, the low-pass filter is a 5-tap filter, and the high-pass filter is a 3-tap filter. the method of.   The method according to claim 1, wherein the time conversion is a Haar time conversion.

Images