KR20010032113A - Apparatus and method for compressing video information - Google Patents

Abstract

A method and apparatus are described for effectively encoding data representing a video image to reduce the amount of data that must be delivered to a decoder. The method includes transforming data sets using a tensor product wavelet transform that can transmit the rests from one subband to another. The vowels of the subbands are weighted in the form of macro-blocks, allowing prioritization of the detected, ranked and transformed data. Motion compensation techniques are performed on subband data generating motion vectors and prediction errors that are position encoded into bit stream packets for transmission to a decoder. Zero subband macro-blocks and subband blocks are identified by themselves in bit stream packets to further reduce the amount of data that must be delivered to the decoder.

Description

Apparatus and method for compressing video information {APPARATUS AND METHOD FOR COMPRESSING VIDEO INFORMATION}

Because of the effective limited bandwidth on the transport channels, only a limited number of bits are valid for encoding audio and video information. Video encoding techniques attempt to encode video information with as few bits as possible while still maintaining the image quality required for the application. Therefore, video compression techniques reduce the bandwidth required to transmit video signals by removing redundant information and displaying the remaining information with the minimum number of bits that can be reconstructed with only a minimal loss of features where the approximation to the original image is important. I'm trying to. In this way, the compressed data can be stored or transmitted in a more efficient manner than the original image data.

There are a number of video encoding techniques that improve coding efficiency by removing statistical overlap from video signals. Many standard image compression techniques are based on block transformation of input images, such as discrete cosine transform (DCT). For example, the well-known MPEG video encoding technique, developed by a group of video experts, correlates and predicts and correlates between pixels (pels) in the spatial domain (through the use of DCT). Correlation between image frames in the time domain (through the use of motion compensation) is used to achieve effective bit rate reduction.

In the well-known orthogonal and bi-orthogonal (subband) transform-based encoding systems (including wrapped orthogonal transforms), the image is transformed into an image. It is converted without the need for the first blocking. The transform encoder is based on the DCT without having to first block the image. DCT-based transform encoders basically block the image for two reasons: 1) The experience is that DCT is known for its best known transform (Kahunen-Luove ') on the 8 x 8 region of the image. Shown to be a good approximation or sequence of difference images; 2) The processing of the DCT becomes 0 (N log N) and the calculation result is limited through blocking of the image.

The end result is that the DCT based means have basic functions that are tightly supported by an 8x8 region (or outside of it) of the image, unless otherwise improved. Orthogonal and bi-orthogonal transforms are predominantly supported at finite intervals of the image under consideration, but have the basic elements sharing range with neighboring spatial regions. For example, subband image encoding techniques use a set of filters to divide the input image into multiple spatial frequency bands to quantize each band or channel. A detailed description of the subband image encoding technique is available. C. Podilchuck and A. See A. Jacquin's Subband Video Coding With Dynamic Bit Allocation and Geometric Vector Quantization, SPIE Vol. 1666 Human Vision, Visual Processing, and Digital Display III, pp.241-52 (February 1992). . At each stage of the subband encoding process, the signal is divided into a lowpass approximation of the image, and a highpass term indicating the detail lost by making this approximation.

In addition, DCT based transform encoders are immutable in that the base elements have a support extending through the entire 8x8 block. This prevents the motion compensation from being executed effectively in the conversion area. Therefore, in use, most motion compensation techniques use adjacent image frames temporarily to form an error term that is a transform coded on 8 × 8 bubble. As a result, these techniques require an inverse transform to be implemented to provide the reference frame from the frequency domain to the time domain. Embodiments of such systems are found in US Pat. No. 5,481,553 to Suzuki et al. And US Pat. No. 5,025,482 to Murakami et al.

1 shows a simplified block diagram of a prior art standard video compression means using DCT. In block 10, the change in image sequence is effectively represented through motion detection techniques, such as one technique used in MPEG, when in prediction mode. In particular, the preceding frames are used as reference frames and the subsequent frames are compared against the preceding frames in forward prediction to remove time overlap and rank the differences between them according to degree. This step sets the stage for motion prediction of subsequent frames and also reduces the data size of subsequent frames. At block 12, a determination is made regarding which portions of the image are moved.

Continuing with the MPEG embodiment using the data set provided by block 10, intraframe motion prediction is performed by applying motion compensation techniques to the reference frame and subsequent frames. The resulting prediction is subtracted from subsequent frames to generate prediction errors / frames. Then, at block 14, the changes are converted to features. In MPEG, this is done by compressing the prediction error using a two-dimensional 8 x 8 DCT.

Most video compression techniques based on DCT or subband encoders concentrate on high precision techniques that attempt to encode video information without loss of precision at the transform stage. However, this high precision precision encoding technique is relatively expensive microprocessors such as Intel's PENTIUM processor, which reduces the adverse conditions to maintain high precision by having dedicated hardware to assist in the manipulation of floating point arithmetic. Depends on

However, in many applications, such relatively expensive hardware is not practical or justified. Therefore, there is also a need for lower cost realization to maintain acceptable image quality levels. Known limited precision conversions that can be realized in lower cost hardware, however, tend to indicate reduced precision as a result of the ″ lossy ″ feature of the encoding process. As used herein, it refers to a system that loses precision through various stages of an encoder and lacks the ability to substantially reconstruct input from transform coefficients in decoding. The inability to compensate for the reduced precision indicated by these low precision transforms has been an obstacle to the use of these transforms.

In view of the foregoing, there is a need for a video encoder that implements motion compensation of the transform domain to eliminate the need for inverse transform of the encoder and to enable a simple control structure for software and hardware devices. There is also a need in the art for a video encoder having a kind of conversion suitable for low precision implementation, including a control structure that enables low cost hardware and high speed software devices.

Summary of the Invention

The present invention relates to a novel and unique apparatus and method for compressing data. More specifically, the apparatus and method of the present invention are suitably formed to, for example, more effectively encode data representing a video image to reduce the amount of data that must be transmitted to the decoder.

The present invention relates to a method of compressing data comprising a first data set and a second data set. The method includes converting the first and second data sets into corresponding first and second transform coefficient sets. Thereafter, data indicating the difference between the first and second transform coefficient sets is generated. The generated data is then encoded for transmission to the decoder.

Transforming the first and second data sets may be implemented using a tensor product wavelet transform. Moreover, the rests resulting from the conversion process can be transmitted from one subband to another.

Data indicative of the difference between the first and second transform coefficient sets is generated by estimating the difference between the first and second transform coefficient sets to provide motion vectors. The motion vectors are applied to the first set of transform coefficients to generate predictions of the second set of transform coefficients. The prediction value is subtracted from the second set of transform coefficients resulting in a set of prediction errors. The first and second transform coefficient sets may be error corrected to ensure synchronization between the encoder and the decoder.

In estimating the difference between the first and second transform coefficient sets, a search region is generated around a subset of the transform coefficients from one of the first and second transform coefficient sets. Thereafter, the subset of relevant transform coefficients is applied to the search region from the remainder of the first and second transform coefficient sets. The subset of relevant transform coefficients is then traversed incrementally within the search region to the position representing the best incremental match. The relevant subset can then be traversed fractionally within the search region up to the position representing the best fractional match.

Another embodiment of a method of compressing data comprising a first data set and a second data set includes converting the first and second data sets into corresponding collections of first and second subsets. Then, data representing the difference between the first and second collections of subbands is generated. The data may be generated, for example, by performing motion compensation techniques. Motion compensation techniques can provide output such as motion vectors and prediction errors. Thereafter, the generated data is encoded for transmission to the decoder.

Embodiments may be subband macro-block packed of a second collection to form subband macro-block grouping. The generated data can then be obtained through the following motion compensation techniques. The difference between the first collection of subbands and the subband macro-block grouping is estimated to provide motion vectors. The motion vectors are applied to a first collection of subbands that yields a prediction of the second collection of subbands. The prediction is then subtracted from the second collection of subbands resulting in a set of prediction errors.

The difference can be estimated between the first collection of subbands below and subband macro-block grouping. A search region is generated for a subset of transform coefficients from the first collection of subbands. The subset of related transform coefficients from the subband macro-block grouping is applied to the search region. The subset of related transform coefficients is traversed incrementally within the search region to the position that represents the best incremental match. The subset of relevant transform coefficients is then traversed fractionally in the search region to the position representing the best fractional match.

The subband macro-block packing method is also described for constructing subband blocks of a collection of subbands derived from the transformation of an image. The method includes separating a set of related subband blocks from a collection of subbands corresponding to an image macro-block of an image. The set of related subband blocks is packed together as a subband macro-block. The steps of separating and packaging the related subband blocks are repeated for each set of related subband blocks of the collection of subbands to form subband macro-block grouping.

The method for macro-block packing can be further refined by aligning the set of related subband blocks within the subband macro-block to the same relative position that the subband blocks occupy in the collection of subbands. The method may also include positioning the subband macro-blocks within the subband macro-block grouping at the same spatial location as the corresponding image macro-block is located within the image block macro-block grouping.

After macro-block packing, a change can be detected between the first subband macro-block grouping (reference) and the subsequent second subband macro-block grouping. Detection is based on distortion assessment according to the following general formula:

Where e _c = measurement of distortion relative to reference value R;

W _i = weight applied;

G = transform coefficients of the second subband macro-block grouping; And

R = reference value (eg, first subband macro-block grouping)

A more specific form of equation for evaluating distortion is:

to be.

Another embodiment of the invention is described in a finite precision method of transforming a data set into transform coefficients, where the data set is transformed using a tensor product wavelet pair and the remainder from it is passed to the opposite filter path. . More specifically, embodiments may include determining low pass components and high pass components of an image. The low pass component is normalized to produce a low pass normalization output and a first remainder rl. Similarly, the high pass component is normalized to produce a high pass normalization output and a second remainder rh. The first operation g (rl, rh) is performed on the first and second remainders (rl, rh) and added to the results therefrom to approximate. The second operation f (rl, rh) is performed on the first and second remainders rl, rh and added to the results from it to detail. It is important to note that the rest of the propagation (the propagation of error) can be used for any transformation, not just a tensor product.

The finite precision method results in overcomplete display of the image. The method may include, for example, downsampling of both the high and low pass components to obtain the necessary and sufficient transform coefficients representing an image of the transform region.

Embodiments of the finite precision method include a low pass filter having a value of -1,2,6,2, -1 and a high pass filter having a value of -1,2, -1. The first operation g (rl, rh) and the second operation f (rl, rh) have the following functions:

g (rl, rh) = rh; And

f (rl, rh) = floor (rh + 1/2), where nh = 1/2.

Certain embodiments of the tensor product wavelet transform including the above are of the form:

Where X _2i = input data;

X _2i-1 = data preceding input data X _2i ;

X _{2i + 1} = data following the input data X _2i ;

D _i = detail term (decimated high pass filter output);

D _{i + 1} = detail term following D _i ; And

A _i = approximation (described low pass filter output).

Also described is an encoder apparatus for predicting a change between sequential frames of a transform region. The apparatuses have inputs configured to receive a first frame and a second frame of the sequence of frames, and further formed to generate corresponding first and second collections of subbands, each supporting a set of transform coefficients. And a conversion device. A motion compensation device having an input coupled to the converter is formed to receive the first and second vowels of the subbands and is further formed to effectively indicate the difference between the first and second vowels of the subbands. Also included is a difference block having an input coupled to the conversion device and an input coupled to the output of the motion compensation device. The input received from the motion compensation apparatus is added from the second collection of subbands in the difference block to generate a prediction error.

The motion compensation device includes a motion estimation device configured for comparing the first and second collections of subbands. A collection of motion vectors is generated therefrom that approximately represents the difference between the first and second collections of subbands. The motion compensation device also has an input coupled to the motion estimation device, configured to receive motion vectors and a first collection of subbands, and to generate therefrom a predictive grouping that represents a prediction of the second collection of subbands. And a motion prediction device further formed. The prediction of the second collection of subbands is subtracted from the second collection of subbands in the difference block resulting in prediction error.

A finite precision conversion device for converting an image frame into a conversion region is also described. The apparatus includes a low pass component and a high pass component that are arranged in parallel and share an input formed to receive an image frame. A low pass normalization device is included that has an input configured to receive an output of the low pass component and is further formed to generate a low pass normalization output and a first remainder rl. The high pass normalization device has an input configured to receive the output of the high pass component and is further formed to generate a high pass normalization output and a second remainder rh. The first computing device has an input formed to receive the first remainder rl and the second remainder rh, and is further formed to calculate the first operation f (rl, rh), the first calculation result. Generates. The second computing device has an input configured to receive the first remainder rl and the second remainder rh and is formed to calculate the second operation f (rl, rh) to generate a second calculation result. . In addition, the first adder has an input configured to receive the first low pass normalization output and the first operation result, the first adder generating a subband approximation. Similarly, the second adder has an input configured to receive the high pass normalization output and the second calculation result, the second adder generating subband details.

The finite precision converter further includes a first downsampler of low pass output and a second downsampler of high pass output. Both downsampling is sufficient to reconstruct the input image and provide the decoder with the necessary transform coefficients.

These and other unique features of the apparatus and method described herein will become more readily apparent from the following detailed description taken in conjunction with the drawings.

This application claims the benefit of Provisional Patent Application No. 60 / 066,638, filed November 14, 1997, which is incorporated herein by reference.

The present invention relates generally to an apparatus and method for encoding and decoding video information. More specifically, the present invention relates to an apparatus and method for motion estimation and motion prediction in a transform domain.

Exemplary embodiments of the invention will be described with reference to the drawings below.

1 is a schematic block diagram of a prior art standard video compression technique using discrete cosine transform (DCT) in which motion compensation is implemented in an image region.

FIG. 2 is a schematic block diagram illustrating a general arrangement of an embodiment of the present invention including preparation for motion compensation to be implemented in the transform domain.

3 is a schematic block diagram of a more detailed arrangement of the embodiment illustrated in FIG. 2.

Fig. 4 (a) illustrates a QCIF image with image macro-blocks (IMB _{X, X} ) 0,0 to 8,10 and Fig. 4 (b) shows that the image frame is converted by a forward wavelet transform. The subband representation of the QCIF image is then illustrated.

FIG. 5 (a) illustrates a subband representation of a QCIF image as illustrated in FIG. 4 (b), and FIG. 5 (b) shows a subband macro-generated from the subband representation illustrated in FIG. 5 (a). illustrates a set of blocks (SMB _{X, X),} and Figure 5 (c) is the subband macro 5 (b) even-by illustrating the structure of the blocks, the sub-band macro-block (SMB _{X, X)} are also Spatially corresponds to their associated image macro-blocks (IMB _{X, X} ) of 4 (a).

6 (a) and 6 (b) are schematic block diagrams illustrating filterbanks for transforming and desmeating the input image and their corresponding vertical and horizontal subbands generated from each filterbank.

FIG. 7 illustrates a structure for delivering finite precision calculations in a filter bank from the high band region to the low band region and vice versa, from the low band region to the high band region.

Fig. 8 shows the search region of the transform region for each subband SB _ij corresponding to the image macro-blocks 2,4 (IMB _2,4 ) of the image region in which the search band is P × P pel. When the image size is in QCIF, the search area for it in SB ₀ is more detailed.

9 (a) to 9 (d) show how the operation is estimated in the transform domain.

Figure 10 shows how the operation is predicted in the transform domain.

FIG. 11 is a schematic block diagram illustrating another detailed arrangement of the embodiment shown in FIG.

Figure 12 is a schematic block diagram illustrating another detailed embodiment of the present invention in which motion estimation is performed in the image region and motion prediction is performed in the transform region.

Figure 13 shows the P x P Pel search area when searching in the image area for the image macro-blocks 2, 4 (IMB ₂ , 4) when the input size is QCIF.

14 is a schematic block diagram illustrating another detailed embodiment of the present invention in which motion estimation and motion prediction are performed in an image region.

Embodiments of the present invention provide an apparatus and method for compressing digital video signals using limited precision conversion techniques. Embodiments improve loss-less or lossy transform based techniques, for example by estimating and predicting motion by motion compensation in the transform domain rather than in the time domain as in the prior art. In this way, improved image quality can be realized on less expensive hardware.

The term motion compensation is intended to be defined in its broadest sense. In other words, although motion compensation is often described and illustrated herein as including motion estimation and motion prediction of a group of pixels, it should also be understood to include rotation and scale, for example. In addition, the term "operation compensation" may include, for example, only generating data indicative of a difference between two sets of data.

Compression efficiency is obtained by converting an image into a feature and first mapping the feature. The description of the text is illustrated as it relates to a sequence of image or video frames. Such an image sequence can be easily understood to be a collection of spatially located data elements (scalars, vectors, or functions), positioned in alignment with each other and indicated by time or some other parameter. The image sequence may be in Cartesian coordinates, but other coordinate systems in the art may also be used.

Moreover, the apparatus and method of the present invention can be used for non-video applications such as voice, audio and electrocardiogram compression. That is, although the invention described herein is illustrated as a two-dimensional system (2D), ie video compression, it is intended that the teachings generally be applicable to any other dimensional system to advance data compression techniques.

For example, the teachings can be applied to one and half dimensional systems 1-1 / 2D, such as ultrasound images. In addition, the teachings can be applied to three-dimensional systems (3D), such as magnetic resonance tomography (MRI).

Through the following description, the term "frame" refers to a sequence transmitted to an encoder regardless of the form of a single image, i.e. whether it is in the time domain, the frequency domain, or any other processing realized thereon. It is called a single image of the merged images. Moreover, the term ″ pel ″ is used for pixels in the time domain, and the terms ″ coefficient ″ and ″ transform coefficient ″ are used for the representation of the pels that are generated after the pel has gone through a forward wavelet transform, for example. do. These terms are used to facilitate the description of the embodiment and are by no means intended to limit the scope of the invention.

Referring to the drawings wherein like reference numerals are the same as like elements of the present invention, a schematic block diagram of an embodiment for compressing a sequence of images or a sequence of frames is shown in FIG. 2. This diagram is one of several embodiments described in the text. More detailed embodiments are described in the paragraphs below.

In FIG. 2, the image is transformed into a collection of features in the transform region of block 20. The feature determined to be valid for that image is selected at block 22, ie this feature determined to have changed significantly from the previous or reference frame. The valid feature is effectively represented at block 24 and sent to the decoder to update the features of the reference frame.

By way of example, the original image is transformed at block 20 and represented by a transform coefficient set. The transform coefficients of the coefficient set are then evaluated in block 22 to determine their validity through various weighting and evaluation techniques and rank them according to their validity. Then, at block 24, motion compensation occurs between the current frame and the past or reference frame. Motion compensation can generate a set of motion vectors, including the operation of evaluating changes between frames. Thereafter, motion vectors are applied to the reference frame during the motion prediction phase. The result of the motion prediction is subtracted from the set of transform coefficients to determine the error of that prediction. The prediction error is then selectively scaled and finally encoded positionally with the motion vectors for delivery to the decoder.

Referring to FIG. 3, a schematic block diagram shows a more specific arrangement of the embodiment that has been described with reference to FIG. In an example, an image sequence or series of video frames 26 encoded in Caltech Intermediate Format (CIF) are transmitted to converter 28. The CIF frame has 288 x 352 pellets. In converter 28, the frames are converted into a quarter CIF (QCIF), eg, a QCIF image 30 as illustrated in FIG. 4A. The QCIF image has 144 x 176 pellets. The CIF is converted to QCIF by lowpass filtering and decimating two (2) in both horizontal and vertical directions. To facilitate processing, the 144 x 176 pellets are divided into image macroblocks (IMB _{x, x} ) each having 16 x 16 pellets. QCIF is used in the text by way of example only and is not intended to be limiting of the invention in any way. The following techniques can be readily adapted to other image (and non-image) formats by methods well known to those skilled in the art.

Referring to Figures 3 and 4, QCIF image 30 is transmitted to blocks 32 and 36, which make up block 20 of Figure 2, where mapping of the image to the feature occurs. More specifically, the QCIF image 30 (FIG. 4 (a)) is sent to block 32 where the forward wavelet transform transforms each frame into a collection of subbands 34 (FIG. 4B). . This configuration of the transformed image, i.e., the collection of subbands 34, is for example used later for motion estimation, motion prediction and stored in memory to determine prediction errors. Suitable forward wavelet transforms that can be used in the present invention are described in more detail below.

The collection of subbands 34 is sent to block 36 for subband macro-block packing. During subband macro-block packing, the subband blocks corresponding to the particular image macro-block are configured to form subband macro-blocks SMB _{X, X.} Each subband macro-block is then placed in the spatial position of the image macro-block it represents in relation. The collection of all subband macro-blocks for a particular frame is referred to as subband macro-block grouping 40.

5 illustrates the processing of subband macro-block packing. During subband macro-block packing, all related blocks of the collection of subband 34 (FIG. 5 (a)) are subband macro-block (as illustrated in FIG. 5 (b) during subband macro-block packing). 38) to reconstruct.

For example, the shaded subband blocks of FIG. 5 (a) corresponding to the image macro-blocks 2,4 (IMBs _2,4 ) of FIG. 4 (a) are subbands such as the illustrated FIG. 5 (b). Reconstructed during subband macro-block packing of block 36 (FIG. 3) to form macro-blocks SMB _2,4 . Subband macro-blocks 38 (SMB _0,0 through SMB _8,10 ) are then composed of subband macro-block grouping 40 as illustrated in FIG. 5 (c) so that each subband macro-block is It is supported by the spatial position of its corresponding image macro-block (IMB _{X, X} ) of the QCIF image 30. In this embodiment, it is recognized that SMB _2,4 is effectively supported by the spatial position of IMB _2,4 as illustrated in Figs. 4 (a) and 5 (c).

Although the embodiment described herein refers only to frame images denoted by QCIF, it is important for those skilled in the art to appreciate that other formats may be used without departing from the scope of the present invention. It is also important to note that a particular grouping of subband blocks of each subband macro-block is used to accommodate the particular wavelet illustrated. There are other groupings of subband data that would be more appropriate for other wavelets.

Collection of image macro-blocks 30 (Fig. 4 (a)), collection of subbands 34 (Fig. 4 (b)), and subband macro-block grouping 40 (Fig. 5 (c)). From the foregoing of the vowel, it should be readily appreciated that there is a correlation between specific image macro-blocks, subband blocks and subband macro-blocks. Examples of such correlations are as follows: (a) image macro-blocks 2,4 (IMB _2,4 ), which are shaded in Figure 4 (a) and identified as image macro-blocks 106; (b) all shaded subband blocks of FIG. 4 (b), eg, subband block 116 of subband 00 (SB ₀ ) and subband block 116 of subband 33 (SB ₃₃ ); And (c) subband macro-blocks 2,4 (SMB _2,4 ), which are shaded in FIG. 5 (c) and identified as subband macro-blocks 117. The description herein, including coefficients having a relationship as illustrated above, may be referred to as 'related'.

Referring to FIG. 3, the subband macro-block grouping 40 may _identify the block 22 recognized in FIG. 2, which determines which feature or subband macro-blocks SB _0,0 to SB _8,10 are to be changed. Are sent to the forming blocks 42, 46, 48, 52. In particular, subband macro-block grouping 40 is adapted to scale each subband macro-block of subband macro-block grouping 40 by an amount that the weight quantizes the perceptual validity of the subband macro-block. Is sent to block 42. The output of weighting block 42 is weighted grouping 44.

Perceptual validity through weighting is, for example, H.261 and H of the International Telegraph and Telephone Advisory Committee (CCITT), which may be determined through a Mean Opinion Score study, or whose standards are incorporated herein by reference. Is determined from the weights used in other coding systems such as those found in .263. For a discussion of average rating scoring, see Discrete Cosine Transform by K.R.Rao & P.Yip, Academic Press, Inc., pp. 165-74 (1990), which is incorporated herein by reference.

After weights are applied to block 42 to scale each subband macro-block, weighted grouping 44 is sent to change detection block 46 to determine the relative amount of change generated. This change is termed 'significance' or distortion of the weighted grouping 44 for video purposes. The effective value can be determined, for example, in relation to the corresponding criteria, such as zero or past weighted grouping. The loop extending from the change-detection block 46 includes a frame delay 48 that returns to the change-detection block 46 for use as a basis for past weighted groupings. The output of change-detection block 46 is change detected grouping 50.

Zero reference is used in the change detection block 46, for example, when initially transmitting frames via the encoder. In this case, the entire frame is related to zero. This is also known as internal frame referencing. As noted above, past weighted groupings may also be used in which macro-block groupings are weighted to block 42 as described above and delayed in delay block 48 of change-detection block 46 for use as a reference. . This recent method, known as internal frame referencing, eliminates the step of repeatedly sending redundant and / or non-critical information to the decoder.

A variant use of zero frame referencing is to reproduce and maintain a relatively accurate reference image at the decoder during system operation. One method employs a method of periodically applying a zero criterion for the entirety of every eight frames of standard 30 frames / second. Alternatively, the image may be stochastically refreshed, such as referencing subband blocks to zero randomly or constantly. In order to facilitate any processing of referencing all or some frames to zero, zero-referenced subband blocks are identified by themselves to prevent the operation compensation operation (hereinafter described) from being implemented on valid blocks. do. Therefore, the identified subband blocks are all played back at the decoder to refresh the entire reference value or some of the reference values, as the case may be.

Referring back to FIG. 3, the collection of subbands 34 whose subband macro-blocks of change-detection grouping 50 are ranked in block 52 according to their validity are stored earlier in memory and according to their amount. It is determined to change. The ranking is based on pre-assigned values by weighting and detecting the subband macro-blocks of blocks 42 and 46, respectively. The output of block 52 includes a ranked subband grouping 53 and a ranked subband macro-block grouping 54 transmitted over line 55.

With continued reference to FIG. 3, the ranked subband grouping 53 and the ranked subband macro-block grouping 54 correspond to blocks 56, 60, 62, 68, 72 corresponding to block 24 of FIG. 2. 76), where the changed macro-blocks are effectively represented. In particular, the ranked subband macro-block grouping 54 ('current' frame) is sent to block 56 for motion estimation. Ranked subband grouping 53 is sent to delay block 62, and then blocks delayed subband grouping 57 ('reference' frame) on line 64 for motion estimation and motion prediction. 56, 60, respectively. The collection of motion vectors 58 is generated in motion evaluation block 56 in the following manner and sent to block 60 for motion prediction and also to block 76 for location encoding.

The motion vector 58 sent to the motion prediction block 60 is used to change the delayed rank subband grouping 57 to produce the predicted grouping 66. Difference block 68 receives the ranked subband grouping 53 and subtracts the predicted grouping 68 therefrom to generate a grouping difference 70, i.e., a prediction error. The grouping difference 70 is scaled further than in block 72 resulting in a scaled grouping difference 74. Those skilled in the art will appreciate that as the number of non-zero grouping differences 70 decreases, it becomes more accurate for the collection of motion vectors 58 to predict changes between this frame and the reference frame. And, the fewer the differences, the fewer bits that must be sent to the decoder to correct for defects in the motion estimation.

The difference of scaled grouping 74 from scaling block 72 and the collection of motion vectors 58 from motion estimation block 56 are position encoded as macro-blocks at block 76. There, the data is effectively organized into bitstreams. The encoded bitstream grouping 78 is output from block 76 and sent to decoder 82 for back processing via transmission line 80. Transmission can be through various media, for example electronic, electromagnetic or optical.

With regard to bitstream formatting, several standard methods are well known in the art for formatting bitstreams. The format used for the H.263 based encoder system is one embodiment. A bitstream is basically a bit packet of consecutive strings. Each packet represents a specific category of data.

For example, the bit packets may include system level data, video, control and audio data. As data is received for position encoding at block 76, it consists of bit packets according to the format used. In general, the collection of bit packets that represent a video frame starts with the bit that identifies it as a new frame. The amount of quantization and other control codes typically follows. Then, the list of macro-blocks representing the scaled grouping difference 74 is encoded. For QCIF, the number of macro-blocks is 99 (see Figure 5 (c)).

To facilitate more efficient transmission of data, each macro-block is followed by a macro-block zero bit (MBZero-bit) indicating the presence or absence of non-zero data in the macroblock. If there is a macro-block, control information for the macro-block that includes the relevant collection of motion vectors 58 is transmitted following the subband data, i.e., the associated scaled grouping difference 74. Including this information substantially reduces the number of bits transmitted over transmission line 80 represented by a single symbol instead of all the bits needed to identify the entire string of macro-block coefficients where the absence of the macro-block is zero. Let's do it.

Another state where additional efficiency may be retained is when only a few subblocks in a subband macro-block are zero. An embodiment includes flagging a subband whose coefficients are zero with a subband zero flag (SBZero flag). The subband from the scaled grouping difference 74 whose coefficients are zero is found that no change exists between the corresponding subband blocks of the ranked subband grouping 53 and the predicted grouping 66. Indicates. Displaying the SBZero flag uses a smaller bit than displaying each coefficient that is zero separately. Of course, the decoder is programmed to recognize both the MBZero-bit and SBZero flags to interpret the symbols introduced during the location encoding of block 76. The zero-running length code for symbolizing a string of zeros is as follows.

With continued reference to FIG. 3, encoded bitstream grouping 78 is received by decoder 82 via transmission line 80 and transmitted to position decoding block 86, which inverts the result of position encoding block 76. do. The collection of motion vectors 58 is extracted from bit stream grouping 78 and sent to prediction block 98. The decoded scaled grouping difference 88, in subband form (FIG. 4B), is supplied to a quantum recovery block 90. In quantum reconstruction block 90, past transform coefficients, and past and present dequantization terms are used to reconstruct the values of the quantization transform coefficients, ie they are used to reproduce the grouping difference 70.

The collection of subbands 92, the frame of reference of the encoder, is transmitted to delay block 94. The collection of delayed subbands 96 is transmitted from delay block 94 to prediction block 98. Similar to the processing performed at the motion prediction block 60 of the encoder, a collection of motion vectors 58 is provided to the delayed collection of subbands 96 at the prediction block 98. There, the delayed vowels of the subbands 96 are changed to produce a subband representation of the updated image that does not include the predicted grouping 100, ie, the grouping difference 70. The grouping difference 70 and the predicted grouping 100 are added at an addition block 102 which generates a collection of subbands 92, ie a collection of new reference frames. Finally, the inverse wavelet transform is implemented at block 104 on the collection of subbands 92. This step is an essential reversal of the forward wavelet transform 32, which is outlined above and described in more detail below. The resulting output from block 104 is reconstructed image 105.

As already described and shown in FIGS. 3 and 4, the QCIF image 30 (FIG. 4 (a)) is adapted to form each video frame to form a collection of subbands 34 (FIG. 4B). A forward wavelet transform 32 is then sent. An embodiment of the transform block 32 uses a tensor product wavelet transform. For a detailed description of tensor product wavelet transformations, see Joel Rossien and Ian Greenshield, Optical Engineering, Vol. 33, Number 8 (August 1994), Standard Wavelet Basis Compression of Images. Other finite precision transforms such as the well-known Mallat, GenLOT, or Harr transform can be used. For a discussion of such a suitable transformed wavelet transform, see Wavelets and Filter Banks, G. Strang and T. Nguyen, Wellesley-Cambridge Press (1997), which is incorporated herein by reference.

Referring to FIG. 4 (b), here is shown a collection of subbands 34 after QCIF image 30 has passed through forward wavelet transform 32. As already indicated, the forward wavelet transform process uses a tensor product wavelet transform or other well known finite precision transform as modified in the text to reduce the result of finite precision realization. In general, the conversion process will consist of m x n stages to generate (m + 1) x (n + 1) subbands. Other embodiments may be made according to the description provided that they are within the scope of this invention.

Referring to Fig. 6 (a), the forward wavelet transform process initially filters the QCIF image frame 30 based on row-by-row using three stages. Each stage includes a low pass filter 108 and a high pass filter 110. In one embodiment, each low pass filter 108 has a value of -1,2,6,2, -1 and each high pass filter has a value of -1,2-1.

After filtering, the low pass components and the high pass components are scaled and deserialized or downsampled at each stage, respectively, by the desistorters 112 and 114 to remove components of the sample values including the discrete signal. In the illustrated embodiment, the input image is downsampled by two factors to discard all other samples. Desampering by two (2) eventually results in necessary and sufficient transform coefficients to enable accurate reconstruction of the input. Thereafter, the downsampled values of the low pass components and the high pass components are normalized at each stage in the manner described in more detail below in connection with FIG. The output of the first stage includes a low pass filter component A _0R and a high pass component D _0R . The low pass component A _0R decomposes into the second time and then to the third time, resulting in further row detail D _1R , D _2R and row mean value A _2R .

The row output (D _0R , D _1R , D _2R , A _2R ) of the row columns shown in FIG. 6 (a) is applied based on column-by-column to the columns shown in FIG. 6 (b). do. Each of the three stages shown in Fig. 6 (b) includes one filter pair, downsampling and normalization process applied in the same manner as described above in combination with Fig. 6 (a). The transform output is a collection of subbands 34 as described above in connection with FIG. 3 and illustrated in FIG. 4 (b).

Referring to FIG. 4 (b) for identification, each subband is identified by a subband notation SB _ij , where i = 0,1,2 or 3 for each row and for each column. j = 0, 1, 2 or 3. Shaded subband blocks, for example, subband block 116 of SB ₀ and subband block 118 of SB ₃₃ correspond to IMB _2,4 of QCIF image 30 of FIG. 4 (a). Because of the degassing process, each corresponding subband block is proportionally such that subband block 116 of SB ₀ contains 8x8 coefficients and subband block 118 of SB ₃₃ contains 2x2 coefficients. Is reduced. As noted above, such subband convexities of the respective subband blocks, e.g., each subband SB ₀ through SB ₃₃ found at the subband positions 2, 4, may facilitate specific processing steps. During the step of subband macro-block packing of block 36 (FIGS. 3 and 5).

In accordance with the features of the described embodiment, referring to Figure 7, the remainder for each stage of the subband encoding process is passed to the opposite filter path to compensate for errors caused by finite precision transform. do. The remainder passed is used to adjust the coefficients on the opposite filter path to account for the loss of precision. Moreover, the process by which the filters are changed thereby may make them neither bi-orthogonal nor orthogonal.

FIG. 7 shows an implementation for delivering the remainder to opposite filter channels for the first stage of the transverse transformation shown in FIG. 6 (a). Similar implementations are included in each row row and column row. The coefficients of the input frame 30 are filtered in a normal manner in the low pass filter 30 and the high pass filter 110. The results are downsampled at samplers 112 and 114, respectively. The decomposed results of the low pass filter 108 are normalized by the low pass normalization output 120 and the low pass normalization processing unit 120 that generates the low pass remainder rl. The decomposed results of the high pass filter 110 are normalized in the high pass normalization output 124 and the high pass normalization processing unit 124 which generates the low pass remainder rh. The remainder (rl, rh) resulting from each normalization processor 120, 124, respectively, passes through the functions g (rl, rh) 128 and f (rl, rh) 130, as illustrated. The result of the function g (rl, rh) 128 is added to the low pass normalization output 122 at the adder 132 resulting in A _0R (average of the first stage). The result of the function f (rl, rh) 130 is added to the high pass normalization output 126 at the adder 133 resulting in D _0R (loss term in the first stage).

For the filters L = {-1,2,6,2, -1} and H = {-1,2, -1}, an embodiment of the remaining functions is: f (rl, rh with nh = 1/2 ) = Lowest value (rh + 1/2); g (rl, rh) = rh. The remaining operations described above are repeated for each filter pair, resulting in a reduced bit allocation of the transform output.

An embodiment of a tensor product wavelet pair is

Where X _2i = input data;

X _2i-1 = data preceding input data X _2i ;

X _{2i + 1} = data following the input data X _2i ;

D _i = detail term (decimated high pass filter output);

D _{i + 1} = detail term following D _i ; And

A _i = approximation (described low pass filter output).

The tensor product wavelet transform described above illustrates two-way splitting into highpass (detailed) and lowpass (approximate) components. In addition, the technique illustrates the possibility of delivering the remainder both from the first band to the second band, from the second band to the first band, or from the first band to the second band and from the second band to the first band. The above described embodiments are intended to illustrate the basic concepts of the present invention and should never be perceived as limiting the scope of the present invention.

As an example, the tensor product wavelet transform may have a first stage in which a three-way split includes a high pass filter, an intermediate band pass filter, and a low pass filter. The output of the low pass filter can be repeated, i.e., a second stage having a three way split can be applied to the output of the low pass filter resulting in a total of five subbands. In this embodiment, the remainders may be passed from the low pass filter and the high pass filter to the intermediate band pass filter. This embodiment is just one embodiment of how the tensor product wavelet transform can be varied and still maintain the scope and spirit of the present invention. Those skilled in the art will readily appreciate that there are a number of other ways in which the input may be split and interlaced at each end and there are also a number of other ways in which the rest may be passed between subbands.

In addition, the transfer of the remainders above is not intended to limit the use for tensor product wavelet transform. It may be used with any transformation. For example, the transfer of the remainders can be used with Discrete Cosine Transform (DCT). Also, the delivery of the remainders can be used in a lossless or lossy manner.

As noted above, the output of the forward wavelet transform 32 may be a complete indication or an over-complete indication of the QCIF image 30. The complete indication of the QCIF image 30 includes a complete indication and duplicate values, variants, or additional subband indications to facilitate motion compensation as described in the text below. Each indication has the value of the described embodiment. By way of example, the over-complete indication may include various image changes, such as translation, rotational movement and scaling. These changes can be recalled as needed during motion compensation, reducing the problem of image change to one indexing.

Although the transformed image frame structures illustrated in the text are for luma components, the structures also maintain for chroma components and therefore are not described separately, and with the forward wavelet transform You should be aware of that.

With respect to the change-detection block 46 described above in connection with Figure 3, how many other reference values, such as zero reference values or, for example, past weighted groupings provided through the delay portion 48, how many? It is appreciated that the weighted grouping 44 can be used to detect if it changes. Embodiments of change-detection block 46 include a change detection metric to which weighted grouping 44 should be applied.

Where e _c = measurement of distortion relative to reference value R;

W _i = weight applied;

G = current grouping of subband transform coefficients; And

R = for example, zero, or previous grouping of subband coefficients obtained via delay block 48.

Change detection metrics are more specific:

Can be taken.

In addition, the change-detector 46 is output from the change-detector 46 in relation to the bit allocation using the information provided by the feedback 132 (FIG. 3) from the encoded bitstream grouping 78. If it is determined to be too expensive, then certain weighted macro-blocks of weighted grouping 44 can be removed. Moreover, change-detection block 46 may be replaced with one feature, e.g., a subband block, with another that is believed to better represent that feature.

As noted above and shown in FIG. 3, the ranked subband grouping 53 and the ranked subband macro-block grouping 54 are delayed block 62 and block 56 over line 55 for motion estimation. Respectively). In block 56, the comparison process involves subband blocks of the ranked subband macro-block grouping 54, i.e., the associated search regions of the 'current' frame and the delayed ranked subband grouping 57, i.e. Reference 'is performed between frames. Those skilled in the art will appreciate certain advantages in using the ranked subband macro-block grouping 54 for the current frame and the delayed ranked subband grouping 57 for the reference frame. However, it should be appreciated that other groupings and combinations may be used to maintain the teachings of the present invention. The comparison process performed at block 56 is passed to block 60 for motion prediction and motion vector 58 to block 76 for location encoding into the bitstream, as outlined above in the text. Results in vowels.

8 and 9, the motion estimation of block 56 and the generation of a collection of motion vectors 58 will be described in more detail. In Fig. 8, delayed ranked subband grouping 57 is shown. Delayed ranked subband grouping 57 is similar to the collection of subbands 34 shown in FIG. 4 (b) but has its subband blocks ranked in block 52 (FIG. 3) and delay block ( Additional processing was performed by delaying at least one frame in step 62). In order to easily determine the individual motion vectors, search regions are defined for the subband blocks of the at least one subbands SB ₀ to SB ₃₃ . The subband blocks in each subband that are selected to have search regions defined for them are those that are limited to those that are important in change-detection block 46. It is sufficient to develop motion vectors based on valid subband blocks in SB ₀ .

8, the search region developed for each subband block corresponding to the image macro-blocks 2 and 4 (IMB ₂ and ₄ ) of the QCIF image 30 (FIG. 4 (a)). Are shown. The size of the search areas may vary. However, search areas for subband blocks will always be proportional to their fractional relationship with the image. By way of example, the basic search region of the P x P Pel in QCIF image 30 (FIG. 13) is subband block 137 of SB ₀ of P / 2 x P / 2 (FIG. 8), as indicated by reference numeral 136. FIG. Is replaced by a search region for subband block 140 of SB ₁ of P / 4 x P / 2 (FIG. 8).

In the embodiments of motion estimation provided herein below, the P x P search region 107 of Figure 13 should include 32 x 32 pels, which is four times the size of IMB _2,4 with 16 x 16 pels. . Therefore, P / 2 by P / 2 search region 136 (FIG. 8) includes 16 by 16 coefficients that are four times the size of subband block 137 (8 by 8 coefficients). The P / 4 × P / 2 search region 139 includes 16 × 8 coefficients that are four times the size of the subband block 140 (8 × 4 coefficients). As described further in the text below, the subband search regions readily determine motion vectors for each valid subband block (0,0 to 8,10) of some or all subbands (SB ₀ to SB ₃₃ ). It is used to

The basic size P x P of the search region may be determined by empirical or statistical analysis under consideration, for example the amount of motion expected between frames. In addition, consideration should be given to the computer actions necessary to perform a search in that search area. Those skilled in the art will appreciate that larger search areas require more computer resources and require more internal frame delay for fixed processors. Conversely, smaller search areas require less computer resources but sacrifice image quality. This is especially true during high image-shift cycles. That is, the quality of the image is reduced because part of the motion is located outside the search area, which prevents precise motion vector selection.

As noted above, the ranked subbadging grouping 53 and the ranked subband macro-block grouping 54 are passed from the block 52 to the local block 62 and the motion estimation block 56 via the line 55. Each is delivered. For example in the text below, the search region is located with respect to the subband blocks 2, 4 of SB ₀ of delayed ranked subband grouping 57 (FIG. 8). Then, subband blocks 2 and 4 of SB ₀ of subband macro-blocks 2 and 4 of the ranked subband macro-block grouping 54 (see subband block 116 of FIG. 5C). ) Is used to traverse the search area during the change. However, as described above, the selection of any subbands or the selection of all subbands can be used according to the method described below.

3, 8, and 9, as described above, the ranked subband grouping 53 is delayed in the delay section 62, which produces a delayed ranked subband grouping 57 ('reference frame'). . Delayed ranked subband grouping 57 is sent to motion estimation block 56 where the search region 136 is identified as having a P / 2 by P / 2 region of SB ₀ for subband block 137. . For this example, the search area is equal to 16 x 16 coefficients. Ranked subband macro-block grouping 54 ('current' frame) is a subband block 138 (Fig. 9 (a)) similar to the shaded region of subband block 116 in Fig. 5 (c). It is also sent to motion estimation block 56, which is retrieved for use in the comparison process described below.

Referring to Figures 9 (a) through 9 (d), the processes in which the motion vectors MV _{x, x} are determined in the motion estimation block 56 of Figure 3 are shown. In the following embodiment, the motion vector for one subband block, i.e. subband blocks 2, 4 of SB ₀ , is determined. However, motion vectors for each valid subband block of each subband SB ₀₀ to SB ₃₃ may be determined.

Referring to Figure 9 (a), the subband block 138 of the ranked subband macro-block grouping 54 is located within the search region 136 of the delayed ranked subband grouping 57 (Figure 8). . Subband block 138 is essentially superimpeded on subband block 137 of delayed subband grouping 57. As mentioned above, the ranked subband macro-block grouping 54 has a structure similar to the subband macro-block grouping 40 shown in Fig. 5C. The delayed ranked subband grouping 57 has a structure similar to the collection of subbands 34 shown in Fig. 4B. Referring again to Figure 9 (a), the coefficients 141 (shown as four circles with 'x' in each) of the search region 136 and the coefficients 142 (4) of the subband block 138 Circles) are used in the text to easily illustrate how to determine the motion vector. Such an embodiment is estimated, although the coefficients 141, 142 are approximately equal in value and the remaining coefficients (not shown) are of different values from the coefficients 141, 142, but approximately equal to each other. The difference in the position of the coefficients 141, 142 represents the change between the two video frames, for example, the translational motion.

Referring to Fig. 9B, the subband block 138 selects a search area 136 to search for to determine the total absolute difference at each step between the subband convex 138 and the search area 136. Search in traverses, that is, in a cascading pattern. Those skilled in the art will appreciate that various traverse patterns can be used. In addition, other criteria than total absolute difference may be used as the basis for the comparison. The initial comparison attempts to find the best match using incremental or full step, movement of the subband block 138. Incremental movement represents a complete shift, or step, in the x or y direction. For example, upon searching the entire search region 136, the subband block 138 shifts the shift coefficients in the search region 136 by ± 4 increments, i.e. by +4 increments in the y direction. . The subband block 138 is ± 4 incremented in the x and y directions because the subband block 138 has 8 x 8 coefficients and the search region has 16 x 16 coefficients.

With continuing reference to Fig. 9 (b), after processing an incremental search, the best match finds three full incremental movements in the positive x direction and two full incremental movements in the y direction. The fractional differences are then determined to more accurately represent the difference between the subband block 138 and the search region 136, as shown in FIG. To facilitate this process, masks representing fractional shifts appropriate for a particular subband are applied to the subband block 138.

For example, because SB ₀ is one-fourth the size of the associated macro-block of the original image (see IMB _{2,4 in} Figure 4 (a)), subband block 138 provides better movement of IMB _2,4 . There are four fractional shifts that can be reproduced more accurately. That is, the subband block 138 may move ± 1/2 of the increment in the x direction and ± 1/2 of the increment in the y direction. Therefore, the four fraction masks 143 are used to find the best match and change the subband block 138.

With continued reference to FIG. 9C, four masks 143 are applied to the subband block 138. The total absolute difference between the coefficients of subband block 138 and search region 136 is determined between the applications of each mask. If a better match is found compared to that determined during the incremental search, a fractional mask is added to the motion vector. In an example, the best match is determined to be +1/2 fraction shifted in the positive x direction. The resulting x and y components of the motion vector are +3 1/2 and +2, respectively.

Those skilled in the art will appreciate that obtaining an exact match to the extent shown in the above embodiments is unusual. In this sense, the 'best match' between the coefficients of the subband block and the coefficients of the search region can be described more accurately as the 'closest approximation' between the two. Motion estimates are later used to compensate for the lack of accuracy.

Referring to Fig. 9 (d), the x and y components of the motion vector are inverted and scaled in their sign. More specifically, -1 times each of the x and y components, and SB ₀ doubles each of the x and y components in this embodiment used for motion estimation. When motion vectors are applied to delayed ranked subband grouping 57 during motion prediction (described in more detail below), the signs of the x and y components are shifted so that the appropriate coefficients are moved from the 'preceding' frame position to the 'current' frame position. Reversed. The x and y components are then scaled up to show the determined (x = 3 1/2, y = 2) shift relative to the associated macro-block of the original QCIF image (IMB _2,4 ). . Scaling allows simpler determination of the x and y components used in shifting the appropriate coefficients of subbands SB ₀₀ to SB ₃₃ during motion prediction.

In an embodiment, the resulting motion vector confirming the movement of subband blocks of SMB _2,4 is x = -7 and y = -4 (MV _2,4 ). Therefore, MV _2,4 specifies the coefficients from each subband of delayed ranked subband grouping 57 ('reference' frame) to predict ranked subband grouping 53 ('current' frame). Mark the movement of the vowels at their new locations. The above process is repeated, for example, for each valid subband block of SB ₀ . Processing typically proceeds in ranking order, ie from macro-blocks with the most amount of movement to macro-blocks with the least amount of movement. Very invalid subband blocks will not be considered at all and therefore will not have an assigned motion vector. This may occur, for example, when there are invalid or no changes at these locations between frames. This may also occur when the subband blocks are zero referenced as described above.

If another subband should be used to calculate the motion vectors, the increment and fractional shifts will be determined in a similar manner as described above using the proportional relationship of the particular subband with respect to the QCIF image 30. For example, if the subband blocks of SB ₁ are used to develop motion vectors, the following criterion will apply: search region size = 16 × 8; x fractional mask = ± 1/4, ± 1/2, and ± 3/4 increments; y fractional mask = ± 1/2 increments; x scaling = 4; And y scaling = 2.

An advantage of using the method is that separable filters can be employed. In other words, filters used for incremental and fractional shifts of one subband block may be used for incremental and fractional shifts of another subband block. For example, the subband blocks of SB ₀ have four possible fractional shifts of x = ± 1/2 and y = ± 1/2. And, the subband blocks of SB ₁ have x = ± 1/4, ± 1/2 and ± 3/4, and have eight possible fractional shifts of y = ± 1/2. Because of the common fractional shift of SB ₀₀ and SB ₁ with x = ± 1/2 and y = ± 1/2, the separable single filters are x = + _1/2 , x = -1 / 2, y in both subbands. = +1/2, and y = -1/2 can be used for fractional shifts. This method can be used for all common fractional shifts of delayed ranked subband grouping 57. The same advantageous use of separable filters may be implemented in motion prediction block 60.

Referring to FIG. 10, after all valid subband blocks have been processed in motion estimation block 56, a collection of motion vectors 58 is output to motion prediction block 60 and position encoding block 76. In motion prediction block 60, motion vectors are derived from each subband of the ranked subband grouping 57 ('reference' frame) that is delayed to predict the ranked subband grouping 53 ('current' frame). It is used to calculate the shift to a new location of certain collections of coefficients.

To determine which mask is used to generate this shift, the x and y components are multiplied by the inverse of the corresponding modulo of each subband block. For example, to determine the x and y components to shift the 8 x 8 collection of coefficients 148 determined to be moved to the 2,4 positions of SB ₀ , the x and y components of MV _2,4 are modulo 2 corresponding. Multiply each by the inverse of The result of this calculation is x = -3 1/2 and y = -2. Therefore, a mask for incremental movement of x = -3, a mask for fractional movement of x = −1 / 2, and a mask for incremental movement of y = −2 is applied to the 8 × 8 coefficients 148.

According to the second embodiment, the x component of MV _2,4 is modulo 4 to determine the x and y components to shift an 8 x 4 collection of coefficients 149 determined to be moved to positions 2,4 of SB ₁ . Multiply by the inverse of and the y components of MV _2,4 are multiplied by the inverse of modulo 2. The result of this calculation is x = -3 3/4 and y = -2. Therefore, a mask for incremental movement of x = -1, a mask for fractional movement of x = -3/4, and a mask for incremental movement of y = -2 is applied.

10 illustrates the movement of all collections of coefficients into subband blocks corresponding to SMB _2,4 . Application of all motion vectors MV _{x, x} from the collection of motion vectors 58 to delayed ranked subband grouping 57 ('reference' frame) results in rank subband grouping 53 ('current' frame). And predicted grouping 66 (FIG. 3).

An alternative embodiment of the masking process for determining fractional movement between frames involves the use of a 3x3 coefficient mask. These masks take a weighted average of the coefficients surrounding the selected coefficient. In the modifying means, a collection of motion vectors 58 containing only incremental movements may be derived for each subband SB ₀ to SB ₃₃ or for a selected number of subbands, e.g. for each valid subband block of only SB ₀ . 9 (a) and 9 (b) described above and determined as shown. The collection of motion vectors 58 is passed to a motion prediction block 60.

In motion prediction block 60, a collection of motion vectors 58 is applied in a manner similar to that shown in FIG. 10 to shift the effective subband blocks of delayed ranked subband grouping 57 to increase. Each coefficient of each shifted vowel of coefficients then has a 3x3 mask applied to it. The mask applied determines the weighted average of the coefficients around each shifted coefficient. The result of the calculation is the predicted value of the shifted coefficient, that is, the coefficients of the new value.

After all motion vectors from the collection of motion vectors 58 are applied to the delayed ranked subband grouping 57 and the mode coefficients shifted by the motion vectors apply to them a 3x3 mask, the result is predicted. The grouping 66 outputs from the motion prediction block 60. Of course, the process is repeated at the prediction block 98 of the decoder 82 to repeat the masking process executed at the motion prediction block 60.

After the prediction is determined by any of the above methods, the predicted grouping 66 is passed to a difference block 68 where the difference between the ranked subband grouping 53 and the predicted grouping 66 is determined. As noted above, difference block 68 generates grouping difference 70.

Although the motion compensation method described herein is illustrated as a function in conjunction with tensor product wavelets, it is important to recognize that the methods can be used with other types of transforms. This involves using other transforms and motion compensation methods in the time domain or transform domain. For example, the data converted to DCT may be a compensated operation in a similar manner as described above. That is, the 64 transform coefficients of each 8x8 block of the DCT may be motion compensated in a manner similar to that used to motion compensate the 64 transform coefficients of each 8x8 subband blocks of SB ₀ of the tensor product wavelet transform. have.

Referring to FIG. 11, another embodiment of a video encoder is shown. According to the embodiment described above and shown in FIG. 3, motion estimation and motion prediction are performed in the transform domain of blocks 150 and 152, respectively. Also, the front part of the embodiment is similar to that described above and shown in FIG. 3. More specifically, CIF image 26 is converted to QCIF image 30 in converter 28. QCIF image 30 is converted to subband macro-block grouping 40 by the image for shape mapping components 20 and converted. Then, the vowels of subband 34 and subband macro-block grouping 40 are ranked by subband grouping 53 and ranked subband macro-block grouping 54 by components associated with determining the changed shape. Respectively).

Similarly to the embodiment shown in FIG. 3, the ranked subband macro-block grouping 54 is passed to the motion estimation block 150 and the ranked subband grouping 53 is passed to the difference block 68. However, instead of using the delayed ranked subband grouping 57 as a reference frame, an error correction subband grouping 171 having a cumulative error added thereto is sent to the delay block 156 to delay the subband grouping 172. Create a 'reference frame'. This variation is necessary when the quantization (or scaling) is so large that it substantially changes the prediction errors 70 generated in the difference block 68.

For example, when a copy of the ranked subband grouping 53 is propagated unchanged through the difference block 68 to deploy the error corrected subband grouping 171, the system is referenced to zero. When it is initialized or when the reference of the decoder is to be refreshed, it is stored in memory. Thereafter, the prediction error 70 accumulates as the prediction error 70 of each subsequent frame passes through the quantization block 158, ie is added to the reference value. The updated reference image is passed to delay block 156 to cause delayed subband grouping 172. Using this method, the reference value of the encoder is kept in sync with the reference value of the decoder. Those skilled in the art will appreciate that this arrangement is useful in maintaining synchronization between the encoder and decoder when a large amount of scaling and / or quantization is performed between predictive and positional encoding.

After the motion estimation block 150 and the motion prediction block 152 receive the delayed subband grouping 172 from the delay block 156, the motion estimation and motion prediction is described above in the text and shown in FIGS. 8-10. It is determined by a similar procedure. In addition, a forward transfer unit 159 is provided between the change detection unit 46 and the quantization block 158 to adjust the amount of quantization to be performed on a particular block according to the amount of change of the block. When a large amount of change is detected in the change detector 46, a plurality of bits are allocated for quantization. And conversely, when a small amount of change is detected in the change detector 46, proportionally fewer bits are allocated for quantization.

Referring to FIG. 12, another embodiment of a video encoder is shown. The front part of this embodiment is similar to the embodiments described above and shown in FIGS. 3 and 11. However, unlike the above embodiments, motion estimation is performed in the image area. This embodiment uses a specific hardware configuration currently available in some processors.

In FIG. 12, CIF image 26 is converted to QCIF image 30 in converter block 28. QCIF image 30 is transformed and transformed into subband macro-block grouping 40 by an image for characterizing mapping components 20. Subband macro-block grouping 40 is processed by the components involved in determining the changed 22 features to determine the subband macro-block ranking. The result is applied to the collection of subbands 34 resulting in ranked subband grouping 53. Ranked subband grouping 53 is then passed to difference block 68.

QCIF image 30, also referred to as a 'current' frame, is also passed to motion estimation block 160 and delay block 166 to determine a collection of motion vectors 162. More specifically, image frame 30 is delayed in delay unit 166, which generates delayed image frame 167, referred to as a 'reference' frame. Referring to FIG. 13, delayed image frame 167 is passed to motion estimation block 160 where a P × P pel search region is developed for each valid image macro-block. By way of example, the P × Ppel search area 107 is set for image macro-blocks 2, 4 (IMB ₂ , ₄ ). Based on the experimental analysis, the search area 107 of 32x32 Fell is used as the search area for the 16x16 Fel image macro-block of the QCIF image frame.

In motion estimation block 160, each valid image macro-block IMB _{X, X of the} current QCIF image 30 frame is corresponding to the search region of image frame 167 that is delayed to determine to determine motion vectors. Is located within. For example, IMB ₂ , ₄ are reconstructed from QCIF image 30 and located within search area 107 of delayed image frame 167. This processing is similar to that performed in the conversion area described above and shown in Figs. 8 and 9 (a).

In a manner similar to that described above and shown in Figure 9 (b), IMB _2,4 is a search for determining the minimum total absolute difference at each step between IMB _2,4 and search area 107. Traverse the region 107. However, unlike the subbad search described above, fractional search is not necessary in the search of the image area. Therefore, after determining the incremental movement of IMB _2,4, the y and y coordinates are inverted (multiplied by -1) and stored in a collection of motion vectors 162 and in memory. The motion vectors are passed to prediction block 154 and position encoding block 76. The motion vectors are then applied to the delayed subband grouping 172 in a manner similar to that described above with respect to FIG. 3 and shown in FIG. 10.

Referring to FIG. 14, another embodiment of a video encoder is illustrated, wherein the front portion is similar to the embodiments described above and shown in FIGS. 3, 11 and 12. However, unlike the embodiments described above, both motion prediction and motion estimation are performed in the image area.

In FIG. 14, a collection of motion vectors 162 is determined in a manner similar to that described above and shown in FIGS. 12 and 13. The collection of motion vectors 162 is passed to block 164 for motion prediction and block 76 for location encoding. In a manner similar to that described above and illustrated in FIGS. 11 and 12, error corrected subband grouping 171 with cumulative errors added to it is passed to delay block 156, thereby delayed subband grouping 172 ('reference). Frame '). However, unlike the embodiments described above, delayed subband grouping 172 is then reconstructed by inverse wavelet transform block 174 to form reconstructed image 176. The reconstructed image has a structure similar to the QCIF image 30 shown in Fig. 4A.

Alternatively, instead of completely reconstructing delayed subband grouping 172, some grouping may be reconstructed to realize validity. For example, 3, 5 filters can be used to obtain reconstituted regions with 48 × 48 pels. The regions are selected based on the validity of the image macro-block 16x16, ie the regions in which they are centered, i.

In motion prediction block 164, the collection of motion vectors 162 is applied to the reconstructed image 176 (or reconstructed 48x48 pel regions once the regions are inverse wavelet transformed). The collection of motion vectors 162 is applied to the reference image 176 reconstructed in a manner similar to that described above and shown in FIG. 10 for shifting the collection of transform coefficients of the subband representation of the QCIF image. The prediction 178 is then passed to a forward wavelet transform block 180 that produces the predicted grouping 66. Predicted grouping 66 is then subtracted from ranked subband grouping 58 at difference block 68 resulting in grouping difference 70. Quantization is performed at block 158 and errors are accumulated to maintain the baseline (described above) and are also passed to position encoding block 76. Position encoding and motion vectors 162 of the quantized error are generated as described above and transmitted to the decoder via transmission line 80.

Although illustrated herein as a software realization, the principles of embodiments of the present invention may be realized through hardware, for example, application specific integrated circuit (ASIC). Preferably, the ASIC implementation comprising the necessary memory requirements iii) minimizes power consumption consistent with the embodiment and ii) allows compression of full color video, such as full CCIR601, as an example of a data rate of less than 13.5 MHz. In order to operate as a pellet. It can be seen that power consumption will be reduced by a factor of 10 by using ASICs compared to conventional software and processor implementations.

Alternatively, optical methods can be employed to save more power. As mentioned above, an approximation to the image is generated at each stage of the wavelet transform and the details lost by making this approximation are recorded. In an opto-electronic or optical implementation, how light is collected and the associated charge is sensed can be adjusted to collect samples of each approximation image. If these approximation images are co-registered in parallel, the detail terms can be calculated from these intermediate values by analog or digital means. Preferably, analog means can be used to calculate the details terms at the output of the analog stage.

The details may be quantized through the use of a bit-series analog-to-digital converter that implements a quantization strategy. In this way, the photonic / optical device acts at a compressed data rate, i.e., the number of digital variations generated, rather than the image data rate (as in the case of ASIC) or processor data rate (as in the case of conventional processor). . This will result in implementations that consume very little current and require less power. It is expected that the implementation of the optical method will further reduce power consumption by about 10 times the factor of that of the ASIC implementation.

It is to be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of the invention and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims (60)

A method of compressing data comprising first and second data sets, the method comprising: Converting the first and second data sets into corresponding first and second transform coefficient sets; Generating data indicative of a difference between the first and second transform coefficient sets; And Encoding the generated data for transmission Method comprising a. 10. The method of claim 1, wherein transforming the first and second data sets is performed using tensor product wavelet transform. 3. The method of claim 2, wherein the remainder from one subband is transmitted to another subband. The method of claim 1, Generating data indicative of a difference between the first and second transform coefficient sets, Estimating the difference between the first and second transform coefficient sets to provide a motion vector; Applying the motion vector to the first transform coefficient set to generate a prediction of the second transform coefficient set; Subtracting the prediction value from the second set of transform coefficients to generate a set of prediction errors Method comprising a. 5. The method of claim 4, wherein the first and second set of transform coefficients are error corrected. 5. The method of claim 4, wherein applying the motion vector to the first set of transform coefficients further comprises applying a mask to each valid transform coefficient to obtain a weighted average of adjacent transform coefficients. Way. The method of claim 4, wherein Estimating the difference between the first and second transform coefficient sets, Generating a search region for a subset of transform coefficients from one of the first and second transform coefficient sets; Applying a relevant subset of transform coefficients to the search region from the remainder of the first and second transform coefficient sets; And Incrementally traversing the relevant subset of transform coefficients in the search region to a location that indicates a best incremental match. 8. The method of claim 7, further comprising fractionally traversing said relevant subset of transform coefficients in said search region to a location that represents a best fractional match. 2. The method of claim 1, wherein transforming the first and second data sets generates the first set of transform coefficients as a first vowel of subbands and generates the second set of transform coefficients as a second vowel of subbands. How to feature. 10. The method of claim 9, further comprising a macro-block packing the second collection of subbands to form subband macro-block grouping. 11. The method of claim 10, further comprising weighting subband macro-blocks in the subband macro-block grouping. 12. The method of claim 10, further comprising detecting a change between the subband macro-block grouping and a reference value. 13. The method of claim 12, wherein detecting a change between the subband macro-block grouping and the reference value is general equation: And based on distortion assessment according to the method. 14. The method of claim 13, wherein detecting a change between the subband macro-block grouping and the reference value is a more specific form of equation: And based on distortion assessment according to the method. The method of claim 10, wherein generating data indicative of a difference between the first and second transform coefficient sets, Estimating the difference between the first collection of subbands and the subband macro-block grouping to provide a motion vector; Applying the motion vectors to the first collection of subbands to generate a prediction of the second collection of subbands; Subtracting the prediction from the second collection of subbands to generate a set of prediction errors Method comprising a. The method of claim 15, Estimating the difference between the first collection of subbands and the subband macro-block grouping, Generating a search region for a subset of transform coefficients from the first collection of subbands; Applying a relevant subset of transform coefficients from the subband macro-block grouping to the search region; And Incrementally traversing the related subset of transform coefficients in the search region to a location that represents the best incremental match Method comprising a. 17. The method of claim 16, further comprising fractionally traversing the associated subset of transform coefficients in the search region to a location that represents the best fractional match. 2. The method of claim 1, wherein encoding the generated data for transmission further comprises identifying subsets of the generated data that are zero. A method of compressing data comprising first and second data sets, the method comprising: Converting the first data set and the second data set into corresponding first and second transform coefficient sets; Estimating a difference between the first and second transform coefficient sets to provide motion vectors; Predicting the second transform coefficient set by applying the motion vectors to the first transform coefficient set; Subtracting the predicted second transform coefficient set from the second transform coefficient set to obtain a prediction error; And Encoding the prediction error and the motion vectors for delivery to a decoder Method comprising a. 20. The method of claim 19, wherein transforming the first data set and the second data set is performed using tensor product wavelet transform. The method of claim 19, Estimating the difference between the first and second transform coefficient sets, Generating a search region for a subset of transform coefficients from one of the first and second transform coefficient sets; Applying a relevant subset of transform coefficients to the search region from the remainder of the first and second transform coefficient sets; And Incrementally traversing the relevant subset of transform coefficients in the search region to a location that indicates a best incremental match. 22. The method of claim 21, further comprising fractionally traversing the relevant subset of transform coefficients in the search region to a location that represents the best fractional match. 20. The method of claim 19, wherein the transforming of the first data set and the second data set generates the first transform coefficient set as a first vowel of subbands and the second transform coefficient set as a second vowel of subbands. Characterized in that the method. 24. The method of claim 23, further comprising a macro-block packing the second collection of subbands to form subband macro-block grouping. 25. The method of claim 24, further comprising weighting subband macro-blocks forming the subband macro-block grouping. 25. The method of claim 24, further comprising detecting a change between the subband macro-block grouping and a reference value. 27. The method of claim 26, wherein detecting a change between the subband macro-block grouping and a reference value is of the general form of: And based on distortion assessment according to the method. 20. The method of claim 19, wherein encoding the prediction error and the motion vector for delivery to the decoder further comprises identifying a subset of the prediction error that is zero. A method of compressing data comprising first and second data sets, the method comprising: Converting the first data set and the second data set into corresponding first and second transform coefficient sets; Estimating a difference between the first and second data sets to provide a motion vector; Predicting the second transform coefficient set by applying the motion vector to the first transform coefficient set; Subtracting the predicted second transform coefficient set from the second transform coefficient set to obtain a prediction error Method comprising a. 30. The method of claim 29, wherein the first set of transform coefficients is error corrected. A method of compressing data of an encoder to reduce the number of bits passed to a decoder, Transforming the first data set and the subsequent second data set to produce corresponding first and second transform coefficient sets; Estimating a difference between the first and second data sets to provide a motion vector; Predicting the second set of transform coefficients by applying the motion vector to the first data set to transform the prediction result; Subtracting the transformed prediction result from the second set of transform coefficients to obtain a prediction error Method comprising a. 32. The method of claim 31, further comprising inversely transforming the first set of transform coefficients and providing the first set of transform coefficients as reference values during prediction. 33. The method of claim 32, wherein the first set of transform coefficients is error corrected. A method of packing subband blocks corresponding to a subset of a data set, the method comprising: Separating the set of related subband blocks from the collection of subbands; Packing the sets of related subband blocks together into a subband macro-block; And Repeating the above separating and packing steps for each set of related subband blocks of the collection of subbands to form subband macro-block grouping Method comprising a. 35. The method of claim 34, wherein the packing step includes aligning a set of related subband blocks in the subband macro-block at the same relative position as the subband blocks located in the collection of subbands. How to. 35. The method of claim 34, wherein the packing step includes placing the subband macro-block within the subband macro-block grouping at the same spatial location that the corresponding subset of data is located within the data set. How to feature. The method of transforming a data set into transform coefficients comprises transforming the data set using a tensor product wavelet transform having at least two filter paths and passing the remainder derived during the conversion between the at least two filter paths. And comprising a step. 38. The method of claim 37, wherein the remainder from the first filter path of the at least two filter paths is passed to a second filter path of the at least two filter paths and the remainder from the second filter path is the first filter path. Characterized in that delivered to. 38. The method of claim 37, wherein the tensor product wavelet transform is a tensor product wavelet pair for determining a high pass component and a low pass component. 40. The method of claim 39, wherein converting the data set and passing the remainder between the filter paths, Determining the low pass component and the high pass component of the data set; Normalizing the low pass component to generate a low pass normalization output and a first remainder rl; Normalizing the high pass component to generate a high pass normalization output and a second remainder rh; Performing a first operation (g (rl, rh)) on the first and second remainders (rl, rh) and adding the output therefrom to the low pass normalization output to approximate a value; And Performing a second operation (f (rl, rh)) on the first and second remainders (rl, rh) and adding the result output therefrom to the high pass normalization output to generate a detail value; Method comprising a. 41. The method of claim 40 further comprising downsampling the low pass component and the high pass component. 40. The method of claim 39, wherein the lowpass component is determined using a filter having values -1,2,6,2, -1; The high pass component is determined using a filter having -1,2, -1; The following function: g (rl, rh) = rh; And f (rl, rh) = floor (rh + 1/2), where nh = 1/2 And further comprising a first operation (g (rl, rh)) and a second operation (f (rl, rh)) having a function. 40. The method of claim 39, wherein the tensor product wavelet pair is of the form: Method consisting of. A method of encoding a data set with transform coefficients comprises transforming the data set using an encoding technique and passing the remainders derived during encoding from the first filter path of the encoder to the second filter path. Way. 45. The method of claim 44, further comprising passing a remainder from the second filter path to the first filter path. 45. The method of claim 44, wherein the encoding technique is a tensor product wavelet transform. 45. The method of claim 44, wherein the encoding technique is a discrete cosine transform. In a method of encoding a data set, Determining a first filter component of the data set of the first filter path; Determining a second filter component of the data set of the second filter path; Normalizing the first filter component to generate a normalized output and remainder; And Delivering the remainder to the second filter path Method comprising a. A method for estimating a change that occurs between a first data set and a second data set, Generating a search region for a subset of data from one of the first and second data sets; Applying a relevant subset of data to the search region from the remainder of the first and second data sets; And Incrementally traversing the relevant subset of data in the search region to a location that represents the best incremental match Method comprising a. 50. The method of claim 49, further comprising fractionally traversing the related subset of data in the search region to a location that represents the best fractional match. In the encoder device, A conversion device having an input configured to receive first and second sets of data and further formed to generate first and second collections of corresponding subbands; And Has an input coupled to the conversion device, is configured to receive the first and second vowels of the subbands, and further formed to effectively indicate a difference between the first and second vowels of the subbands Compensation device An encoder device comprising a. 53. The encoder device of claim 51 wherein the motion compensation device performs all operations on the first and second collections of subbands in the transform region. 53. The apparatus of claim 51, formed to receive a prediction from the motion compensation device and the second collection of subbands from the transform device and between the prediction and the second collection of subbands to generate a prediction error. And a difference block further formed to determine a difference. The method of claim 51, wherein The motion compensation device, A motion estimation device coupled to the transform device, the motion estimation device configured to compare the first and second collections of the subbands to generate motion vectors; And A motion prediction device coupled to the motion estimation device and the transform device, the motion prediction device formed to receive the motion vectors and the first collection of subbands, and further formed to generate a prediction of the second collection of subbands. An encoder device comprising a. In the encoder device for detecting a change, A conversion device having an input configured to receive a first data set and a second data set, and further formed to generate, respectively, a first collection of subbands and a second collection of subbands; And A first subband macro-block indication and a second subband macro-block indication, each having an input coupled to the conversion device and configured to receive a first vowel of the subbands and a second vowel of the subbands, respectively; Macro-block packing device further formed to generate An encoder device comprising a. 56. The apparatus of claim 55, further comprising an input configured to communicate with the macro-block packing apparatus and configured to receive and scale the first subband macro-block indication and the second subband macro-block indication based on sensing effectiveness. An encoder device, further comprising a weighting device. 56. The apparatus of claim 55, having an input configured to communicate with the macro-block packing apparatus and configured to compare the first subband macro-block indication and the second subband macro-block indication to determine a change therebetween. And a change-detection device, wherein the change-detection device is further configured to generate a change-detection grouping that reflects the change. 59. The encoder device of claim 57 further comprising a macro-block ranking device having an input coupled to the change-detection device and configured to rank the change-detected grouping. 59. The method of claim 57, wherein the comparison of the first subband macro-block representation and the second subband macro-block representation is as follows. The encoder device according to the distortion evaluation according to. 60. The method of claim 59, wherein the comparison of the first subband macro-block representation and the second subband macro-block representation is a more specific form of equation: The encoder device according to the distortion evaluation according to.