KR20050019807A - Spatial scalable compression - Google Patents

Abstract

A method and apparatus for providing spatial scalable compression of a video stream is disclosed. The video stream is downsampled to reduce the resolution of the video stream. The downsampled video stream is encoded to produce a base stream. The base stream is decoded and upconverted to produce a reconstructed video stream. The reconstructed video stream is subtracted from the video stream to produce a residual stream. Thereafter, it is determined which pixels or segments in each frame have a predetermined likelihood of having predetermined characteristics. The gain value for the content of each segment or pixel is calculated, where the gain for pixels with a predetermined likelihood of having a predetermined characteristic is biased to 1, and the gain for other pixels is biased to zero. The residual stream and gain values are multiplied to remove bits from the residual stream that do not correspond to predetermined characteristics. The remaining stream of the result is encoded and output as an enhancement stream.

Description

Spatial scalable compression

The present invention relates to a video encoder / decoder.

Due to the inherent large amounts of data in digital video, the transmission of full-motion, high definition digital video signals is a serious problem in the development of high definition televisions. In particular, each digital image frame is a still image formed from an array of pixels depending on the display resolution of a particular system. As a result, the amounts of raw digital information included in high definition video sequences are large. In order to reduce the amount of data that must be transmitted, compression schemes are used to compress the data. Various video compression standards or processes have been established, including MPEG-2, MPEG-4, and H.263.

Many applications have been allowed, in which video may be used at various resolutions and / or qualities within one stream. Methods for achieving this are flexibly referred to as scalability techniques. There are three axes on which to develop scalability. The first axis is scalability on the time axis, often referred to as temporal scalability. The second axis is scalability on the quality axis (quantization), often referred to as signal-to-noise scalability or fine-grain scalability. The third axis is the resolution axis (number of pixels in the image), often referred to as spatial scalability. In layered coding, a bit stream is divided into two or more streams or layers. Each layer can be combined to form a single high quality signal. For example, the base layer may provide a lower quality video signal, while the enhancement layer provides additional information that may enhance the base layer image.

In particular, spatial scalability can provide compatibility between different video standards or decoder capabilities. In addition to spatial scalability, the base layer video can have an input video sequence low resolution, in which case the enhancement layer carries information that can re-store the resolution of the base layer at the input sequence level.

1 illustrates a known spatial scalable video encoder 100. The illustrated encoding system 100 achieves layer compression, whereby a portion of the channel is used to provide a low resolution base layer and the remainder is used to transmit edge enhancement information, whereby two The signals can be recombined to drive the system at high resolution. The high resolution video input Hi-Res is separated by the splitter 102, whereby data is sent to the low pass filter 104 and the subtraction circuit 106. The low pass filter 104 reduces the resolution of the video data and feeds it to the base encoder 108. In general, low pass filters and encoders are well known in the art and are not described in detail herein for the sake of brevity. The encoder generates a low resolution base stream that can be broadcast and received, even if the base stream does not provide the resolution considered as high definition, which can be displayed through the decoder.

In addition, the output of the encoder 108 is supplied to a decoder 112 in the system 100. From there, the decoded signal is provided to the interpolation and upsample circuit 114. In general, interpolation and upsample circuitry 114 reconstructs the filtered resolution from the decoded video stream and provides a video data stream having the same resolution as the high resolution input. However, because of the losses and filtering inherent in encoding and decoding, the loss of information is in the reconstructed stream. The loss is determined in subtraction circuit 106 by subtracting the reconstructed high resolution stream from the original, unmodified high resolution stream. The output of the subtraction circuit 106 is provided to the enhancement encoder 116 which outputs a significant quality enhancement stream.

Although these layered compression schemes work very well, these schemes still have the problem that the enhancement layer requires a high bitrate. One way to improve the efficiency of the enhancement layer is the PCT application IB02 / filed in October 2002 entitled "Spatial Scalable Compression Scheme Using Adaptive Content Filtering". 04297. Briefly, a picture analyzer driven by a pixel based digital metric controls the multiplier gain in front of the enhancement encoder. For regions of less detail, the gain 1-α is biased to zero, which regions are not encoded as the residual stream. For larger detail areas, the gain is biased to 1 and encoded as the residual stream.

Human eyes are attracted to others, so experiments show that human eyes follow people and especially their faces. Thus, this follows that these areas should be encoded as well as possible. Unfortunately, the detail metric is usually not very interested in the minor details of the faces, so usually the alpha value will be relatively high and the faces will be mostly encoded at the low resolution of the base stream. Accordingly, what is needed is a method and apparatus for determining which sections of an entire image should be encoded in an enhancement layer based on human viewing behavior.

1 is a block diagram illustrating a known layered video encoder.

2 is a block diagram of a video encoder layered according to an embodiment of the present invention.

3 is a block diagram of a video decoder layered according to an embodiment of the present invention.

4 is a block diagram of a video decoder layered according to an embodiment of the present invention.

The present invention provides another known layer by using object segmentation to emphasize certain sections of the image of the residual stream, while deemphasize other sections of the image, preferably based on the viewing behavior of the person. Overcomes at least some of the deficiencies of the compressed compression schemes.

In accordance with one embodiment of the present invention, a method and apparatus for providing spatial scalable compression of a video stream is disclosed. The video stream is downsampled to reduce the resolution of the video stream. The downsampled video stream is encoded to produce a base stream. The base stream is decoded and upconverted to produce a reconstructed video stream. The reconstructed video stream is subtracted from the video stream to produce a residual stream. Thereafter, it is determined which segments or pixels within each frame have a predetermined likelihood with predetermined characteristics. The gain value for the content of each segment or pixel is calculated, where the gain for pixels with a predetermined likelihood of having a predetermined characteristic is biased to 1, and the gain for other pixels is biased to zero. The residual stream and gain values are multiplied to remove bits from the residual stream that do not correspond to predetermined characteristics. The remaining stream of the result is encoded and output as an enhancement stream.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

The invention will now be described by way of example with reference to the attached drawings.

2 is a block diagram of a video encoder / decoder 200 layered according to an embodiment of the present invention. Encoder / decoder 200 includes an encoding section 201 and a decoding section. The high resolution video stream 202 is input to the encoding section 201. The video stream 202 is separated by the splitter 204, whereby the video stream is sent to the low pass filter 206 and the second splitter 211. The low pass filter or downsampling unit 206 reduces the resolution of the video stream and sends it to the base encoder 208. Base encoder 208 encodes the downsampled video stream in a known manner and outputs it to base stream 209. In this embodiment, base encoder 208 outputs the local decoder output to upconverting unit 210. The upconverting unit 210 reconstructs the filtered resolution from the local decoded video stream and provides, in a known manner, a reconstructed video stream having essentially the same resolution format as the high resolution input video stream. Alternatively, the base encoder 208 may output the encoded output to the upconverting unit 210, where one of the decoders provided to the upconverting unit 210 or an individual decoder (not shown) may have the encoded signal up. The encoded signal will be decoded first before being converted.

Splitter 211 separates the high resolution input video stream, whereby input video stream 202 is sent to subtraction unit 212 and picture analyzer 214. In addition, the reconstructed video stream is also input to picture analyzer 214 and subtraction unit 212. According to one embodiment of the present invention, picture analyzer 214 includes at least one color tone detector / metric 230 and an alpha modifier control unit 232. In this illustrated example, color tone detector / metric 230 is a skin-color tone detector. Detector 230 analyzes the original image stream and determines which pixel or group of pixels becomes part of a person's face and / or body based on their color tone and / or which pixel or group of pixels is their Determine whether there is at least one predetermined possibility of being part of a person's face or body based on the color tone. The predetermined probability indicates the degree of probability of the pixel or group of pixels with predetermined characteristics. The detector 230 sends this pixel information to the control unit 232. The control unit 232 then controls the alpha value for the pixels, such that the alpha value is biased to zero for pixels with skin tones and one for pixels without skin tones. As a result, the residual stream will include faces and other body parts in the image, whereby the faces and other body parts are enhanced in the decoded video stream.

It will be appreciated that several other tone detectors can be used in the picture analyzer 214. For example, a natural plant detector can be used to detect natural plants in the image for enhancement. Moreover, the control unit 232 can be programmed in various ways how to handle the information from each detector. For example, the pixels detected by the skin-tone detector and the pixels detected by the natural plant detector may be treated identically or weighted in a predetermined manner.

As mentioned above, the reconstructed video stream and the high resolution input video stream are input to the subtraction unit 212. Subtraction unit 212 subtracts the reconstructed video stream from the input video stream to produce a residual stream. Gain values from picture analyzer 214 are sent to multiplier 216, which is used to control the attenuation of the residual stream. The attenuated residual signal is encoded by the enhancement encoder 218 to produce the enhancement stream 219.

In the decoder section 205 shown in FIG. 3, the base stream 209 is decoded in a manner known by the decoder 220, and the enhancement stream 219 is decoded in a manner known by the decoder 222. The decoded base stream is then upconverted in upconverting unit 224. The upconverted base stream and the decoded enhancement stream are combined in the computing unit 226 to produce the output video stream 228.

According to another embodiment of the present invention, the higher resolution area is determined using depth and segmentation information. Larger objects in the foreground of the image are more easily tracked by the viewer's eyes than smaller or smaller objects in the background scene. Thus, the alpha value of pixels or groups of pixels of the object in the foreground is biased to zero, so that the pixels become part of the residual stream.

4 shows an encoder 400 according to an embodiment of the present invention. Encoder 400 is similar to encoder 200 shown in FIG. The entire description of like reference numerals and like elements used for like elements will not be repeated for brevity. The picture analyzer 402 includes a depth calculator 404, a segmentation unit 406, and an alpha modifier control unit 232, among other elements. The original input signal is fed to the depth calculator 404. Depth calculator 404 calculates the depth of each pixel or group of pixels in a known manner, that is, the depth is the distance between the camera and the pixels belonging to the object, and sends information to segmentation unit 406. Segmentation unit 406 then determines other segments of the image based on the depth information. Additionally, motion information in the form of motion vectors 408 from either the base encoder or the enhancement encoder can be provided to the segmentation unit 406 to facilitate segmentation analysis. The results of the segmentation analyzer are fed to the alpha modifier control unit 232. The alpha modifier control unit 232 controls the alpha values for the pixels or groups of pixels so that the alpha value is biased to zero for larger objects or pixels in the foreground of the image. As a result, the resulting residual stream will contain larger objects in the foreground.

It will be appreciated that other elements may be added to the picture analyzer 402. For example, as shown in FIG. 4, the picture analyzer 402 may include a detail metric 410, a skin-tone detector / metric 230, and a natural plant detector / metric 412, but the picture The analyzer is not limited thereto. As mentioned above, the control unit 232 can be programmed in various ways how to handle the information received from each detector when determining how to bias the alpha value for each pixel or group of pixels. . For example, the information from each detector can be combined in various ways. For example, information from skin tone detector / metric 230 may be used by segmentation unit 406 to identify faces and other body parts in the foreground of the image.

The above described embodiments of the present invention enhance the efficiency of known spatial scalable compression schemes by lowering the bitrate of the enhancement layer by using adaptive content filtering to remove unnecessary bits from the residual stream prior to encoding.

It is noted that the above-mentioned embodiments rather limit the present invention, and those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprises' does not exclude the presence of elements or steps other than those listed in a claim. The invention can be implemented by means of a computer suitably programmed by means of hardware comprising various distinguishing elements. In the device claim enumerating various means, these various means may be embodied by the same item of hardware. The simple fact that certain methods are cited in mutually dependent claims does not indicate that a combination of these methods is not used to advantage.

Claims (15)

An apparatus for performing spatial scalable compression of video information captured in a plurality of frames, the method comprising: A base layer encoder for encoding the bit stream, An enhancement layer encoder for encoding a residual signal having a higher resolution than the base layer; A multiplier unit for attenuating the residual signal, wherein the residual signal is a difference between frames upscaled from the base layer and original frames; A picture analyzer for executing segmentation, determining which group of pixels in each frame has at least one predetermined likelihood of having a predetermined characteristic, and calculating a gain value for the content of each pixel, wherein the preliminary The gain for the pixels with the at least one predetermined probability to have a determined characteristic is biased to 1, the gain for other pixels is biased to 0, and the multiplier is adapted to attenuate the residual signal. Spatial scalable compression execution device using gain values. 2. The apparatus of claim 1, wherein the segmentation size is one pixel. 3. The apparatus of claim 1 or 2, wherein the picture analyzer comprises a color-tone detector for detecting pixels having a predetermined color tone. 4. The apparatus of claim 3, wherein the color-tone detector is a skin-tone detector. 4. The apparatus of claim 3, wherein the color-tone detector is a natural vegetation color detector. The method of claim 1, wherein the picture analyzer, A depth calculation unit for determining a depth of each pixel in the frame; A segmentable unit for determining which pixels contain various segments of the images in each frame, wherein the gain for pixels that are part of the objects in the foreground of the image in each frame is biased to 1; Running device. 7. The gain of claim 6, wherein the picture analyzer further comprises at least one color-tone detector, the gain of pixels having a predetermined color-tone or pixels that are part of objects in the foreground of the image in the frame. Wherein the spatially scalable compression execution device is biased to one. A layered encoder for encoding and decoding a video stream, A downsampling unit for reducing the resolution of the video stream; A base encoder for encoding a low resolution base stream, An upconverting unit for decoding and increasing said resolution of said base stream to produce a reconstructed video stream; A subtractor unit for subtracting the reconstructed video stream from the original video stream to produce a residual signal; A picture analyzer for performing segmentation, determining which groups of pixels in each frame have at least one predetermined likelihood of having a predetermined characteristic, and calculating a gain value for the content of each pixel, wherein the preliminary A picture analyzer, wherein the gain for the pixels with the at least one predetermined likelihood having the determined characteristic is biased to 1, and the gain to other pixels is biased to zero; A first multiplier unit that multiplies the residual signal and the gain values to remove from the residual signal bits that do not have the predetermined likelihood having the predetermined characteristic; And an enhancement encoder for encoding the resulting residual signal from the multiplier to output an enhancement stream. The layered encoder of claim 8, wherein the segmentation size is one pixel. 10. The layered encoder of claim 8 or 9, wherein the picture analyzer comprises a color-tone detector for detecting pixels with a predetermined color tone. 11. The layered encoder of claim 10 wherein the color-tone detector is a skin-tone detector. The layered encoder of claim 10, wherein the color-tone detector is a natural plant color detector. The method of claim 8, wherein the picture analyzer, A depth calculation unit for determining a depth of each pixel in the frame; And a segmentation unit for determining which pixels comprise various segments of the images in each frame, wherein the gain for pixels that are part of objects in the foreground of the image in each frame is biased to one. 14. The picture analyzer of claim 13, wherein the picture analyzer further comprises at least one color-tone detector, the gain for pixels having a predetermined color-tone or pixels that are part of objects in the foreground of the image in the frame. Is a 1 biased layered encoder. A method for providing spatial scalable compression using adaptive content filtering of a video stream, the method comprising: Downsampling the video stream to reduce the resolution of the video stream; Encoding the down sampled video stream to produce a base stream; Decoding and upconverting the base stream to produce a reconstructed video stream; Subtracting the reconstructed video stream from the video stream to produce a residual stream; Determining which pixels or segments in each frame have at least one predetermined likelihood of having a predetermined characteristic; Calculating a gain value for the content of each segment or pixel, wherein the gain for the at least one predetermined likelihood pixel having the predetermined characteristic is biased to 1, and the gain for other pixels is The calculation step, biased to zero, Multiplying the gain value with the residual stream to remove bits from the residual stream that do not have the predetermined likelihood having the predetermined characteristic; And encoding the resultant residual stream to output an enhancement stream.