CN101964910B

CN101964910B - Video spatial resolution conversion method based on code-rate type transcoding assistance

Info

Publication number: CN101964910B
Application number: CN 201010257652
Authority: CN
Inventors: 郭红星; 王洁; 余胜生; 田婷; 孙伟平; 周会娇
Original assignee: Huazhong University of Science and Technology
Current assignee: Huazhong University of Science and Technology
Priority date: 2010-08-20
Filing date: 2010-08-20
Publication date: 2012-08-22
Anticipated expiration: 2030-08-20
Also published as: CN101964910A

Abstract

The invention discloses a video spatial resolution conversion method based on code-rate type transcoding assistance, which comprises two key technologies. The first key technology comprises the following steps of: performing code-rate type transcoding on a video stream obtained by decoding a code rate corresponding to the target spatial resolution, actively setting high-frequency information to zero, ensuring the accuracy of motion vector and macroblock pattern multiplexing and benefiting the reduction of calculation amount and transmission bandwidth by maintaining the uniform video spatial resolution of a coding end and a decoding end. The second key technology comprises the following steps of: decoding at a user end and simultaneously resampling the decoded video steam according to the target resolution. Because the resolution of resampling is matched with the resolution of intermediate-frequency domain sub-sampling in code-rate type transcoding, the aliasing effect caused by resampling can be effectively inhibited, and video quality is ensured. In addition, because a large amount of display terminals have the function of resampling per se at present, the method has low extra expense at the decoding end. The invention has simple operation and is applicable to the adaptive real-time video transmission of wired network and mobile wireless network.

Description

Video spatial resolution conversion method based on code rate type transcoding assistance

Technical Field

The invention relates to the field of wireless network video transmission, in particular to a spatial resolution conversion method of a compressed video.

Background

With the development of wireless communication technology, mobile multimedia computing plays an increasingly important role in improving the quality of daily life and the work efficiency of people. However, due to the different screen sizes of different receiving terminals, the wireless video network must meet the need for different terminals to receive video streams that are displayed at different spatial resolutions. Spatial resolution transcoding is an effective way to solve this problem: the method can convert the code stream with one resolution into the code stream with another resolution, thereby gaining wide attention in academia and industry. Many spatial resolution transcoding methods have been proposed, such as [ Vetro a, Christopoulos C, Sun huifang. Ieee signal Processing Magazine, 2003, 20 (2): 18-29] and [ Xin J, Lin C W, SunM T.digital video transcoding.Proc.of the IEEE, 2005.93 (1): 84-97. Among them, the simplest method is the tandem transcoding. Firstly, an input compressed video stream is decoded, and then a reconstructed video obtained by decoding is re-encoded according to a target requirement to obtain a compressed video with a required spatial resolution. Although this method can achieve good image quality, it is not practical because it does not use the information in the original compressed code stream at all when re-encoding, but directly encodes it, so that the computation of the whole transcoder is too large. To solve this problem, pixel domain transcoding and compression (herein referred to as discrete cosine transform-DCT) domain transcoding methods have been developed. The pixel domain transcoding can avoid partial motion estimation and macro block mode decision process by reusing the information such as motion vector and macro block mode obtained when decoding the original compressed code stream. Since motion estimation and mode decision usually account for more than 60% of the computational complexity of the whole encoding process, pixel domain transcoding can greatly reduce the computational complexity. The DCT domain transcoding only needs to decode the compressed code stream to the DCT domain, then directly performs resampling in the DCT domain, and simultaneously re-encodes the motion vector and the macro block mode obtained by the multiplexing decoding end, so that the method can save more calculation amount while ensuring the image quality.

However, when performing spatial resolution conversion, it is difficult to efficiently multiplex information such as a motion vector and a macroblock mode obtained by decoding an original compressed video stream, regardless of whether pixel domain transcoding or DCT domain transcoding is performed. Since they both multiplex motion vectors and macroblock mode information when re-encoding after re-sampling the decoded video stream, as shown in fig. 1. This results in a disparity in the spatial resolution of the video streams at the decoding and encoding ends. For a spatial resolution transcoder, when a block in a video is coded, it is usually necessary to synthesize a new motion vector for coding using the motion vectors of many neighboring blocks of the block. However, these neighboring motion vectors are often not in the same direction, which makes the synthesized target motion vector less accurate, resulting in a significant degradation of video quality compared to a tandem transcoder. Although many improved methods can be used to improve the efficiency of motion vector multiplexing and macroblock mode selection, video quality still drops above 1dB, see [ guo hong star, cheng li, xiao jian, zhou jing. The third harmonic human-computer environment joint academic conference (HHME2007) corpus [ C ], beijing: university of qinghua press, 2007: 383-390]. If better image quality is desired, the resultant motion vector is inevitably refined to obtain a more accurate motion vector, which consumes a large amount of computation. In order to solve these problems, it is highly desirable to propose a new video spatial resolution conversion method to improve the video quality after spatial resolution conversion, while maintaining relatively low computational complexity.

Disclosure of Invention

The invention aims to provide a video spatial resolution conversion method based on code rate type transcoding assistance, which can reduce the calculated amount and improve the image quality.

The proposed video spatial resolution conversion method based on code rate type transcoding assistance processes each frame image of a compressed video stream according to the following method:

(1) carrying out code rate type transcoding on the frame image:

(11) and decoding the frame image to a pixel domain to obtain the video spatial resolution M multiplied by N of the frame image, and the pixel value, the motion vector and the macro block coding mode of each macro block contained in the frame image.

(12) And (3) encoding the video obtained by decoding in the step (11) at a code rate corresponding to the target spatial resolution L multiplied by K, specifically:

(121) for each macro block of the frame image, obtaining the best reference block of the macro block by using the corresponding motion vector, and calculating the pixel value difference between the macro block and the best reference block to obtain a residual error;

(122) respectively carrying out A multiplied by A discrete cosine DCT (discrete cosine transform) on the residual error of each macro block to obtain a DCT coefficient block, wherein the A value is determined by the compression standard of a compressed video stream;

(123) calculating the number X and Y of coefficients which should be reserved by the DCT coefficient block of each macro block:

(124) the X Y coefficient values of the DCT coefficient block of each macro block are kept unchanged, and the rest coefficient values are set to 0.

(125) And (4) quantizing the DCT coefficient block processed by the step (124) for each macro block, and then performing variable length coding on the quantization result, the motion vector and the macro block mode.

(2) And (4) transmitting the coding result of the step (125) to the user terminal.

(3) And (4) decoding the encoding result obtained in the step (125) by the user end, and resampling the decoded video according to the size of a display screen of the user end while decoding.

The step (124) leaves the coefficient values of the ith row and the jth column of each DCT coefficient block of each macroblock unchanged, i being 1, 2, …, and X, j being 1, 2, …, Y.

Said step (124) of retaining the first X Y coefficient values unchanged, in zig-zag scanning order, for the coefficients of each block of DCT coefficients of each macroblock.

The technical effects of the invention are as follows:

the invention provides a code rate type transcoding-assisted video spatial resolution conversion method, which comprises two key technologies: firstly, recoding a video stream obtained by decoding according to a code rate corresponding to a target spatial resolution, and actively setting high-frequency information to be 0; and secondly, resampling the decoded video stream according to the target resolution while decoding at the receiving end.

The first key technology is that a code rate type transcoding technology is utilized, a high-frequency coefficient of a decoded video stream is actively set to be 0 while the decoded video stream is recoded according to a code rate corresponding to a target spatial resolution, and the frequency domain sub-sampling is implicitly achieved while the spatial resolution of a video signal is kept unchanged. By means of the processing mode, the video spatial resolution of the encoding end and the video spatial resolution of the decoding end can be kept consistent when the motion vector and the macro block mode are multiplexed, so that the multiplexing accuracy of the motion vector and the macro block mode is ensured; meanwhile, the high-frequency coefficient actively set to 0 does not need to be calculated and coded, so that the calculation amount and the transmission bandwidth are reduced.

And secondly, the compressed video is re-sampled according to the target spatial resolution while the receiving end decodes the compressed video. The resolution of resampling is matched with the resolution of frequency domain sub-sampling in code rate type transcoding, so that the aliasing effect caused by resampling can be effectively inhibited, and the video quality is ensured. In addition, because modules adaptive to the size of the screen of many display terminals are integrated at present, and the modules have the resampling function, the extra overhead generated by the method at the decoding end is small.

Since aliasing effect is generated in the process of resampling the video, anti-aliasing filtering needs to be performed on the video in order to obtain good image quality. Therefore, in the present invention, the anti-aliasing filtering and the video resampling can be realized in different stages, that is, the anti-aliasing filtering is performed on the video at the encoding end, so as to prepare for the video resampling at the receiving end.

In summary, the present invention provides a general method for implementing spatial resolution conversion of video stream, which is not only suitable for resampling the video spatial resolution in pixel domain, but also suitable for resampling the video spatial resolution in compressed domain; the method is not only suitable for resampling at a ratio of 2: 1, but also suitable for resampling at other ratios.

Drawings

Fig. 1 illustrates a conventional video spatial resolution conversion method that first performs spatial resolution resampling, then multiplexes motion vectors and macroblock modes.

Fig. 2 shows a schematic diagram of the 0 setting method of each 8 × 8DCT block according to the raster scan direction.

Figure 3 shows a system architecture of the present invention.

Fig. 4 shows an 8 x 8 block of DCT coefficients and the numbering of each of them.

Detailed Description

Example one

The invention is further described below in conjunction with the specific example of 2: 1 spatial resolution conversion of an MPEG-4 compressed CIF (352X 288) format video stream to QCIF (176X 144) format, each frame of image of the compressed video stream being processed as follows:

(1) performing code rate type transcoding on the MPEG-4 compressed video stream according to the code rate requirement corresponding to the target spatial resolution, so that the MPEG-4 compressed video stream is converted into the MPEG-4 compressed video stream under the code rate corresponding to the target spatial resolution:

(11) decoding a frame of MPEG-4 compressed input video stream to a pixel domain, and obtaining the video spatial resolution of the frame of image as 352 × 288, and the pixel information, motion vector and macroblock coding mode of each macroblock contained in the frame of image; because the frame image size is 352 × 288, it contains 22 × 18 macroblocks of size 16 × 16.

(12) Re-encoding the video decoded in step (11) under the control of the code rate corresponding to the target spatial resolution 176 × 144, specifically:

(121) and (3) directly multiplexing the motion vector and the macro block mode obtained by the decoding end in the step (11) for each macro block, refining the motion vector with 2 whole pixels through a window to obtain the best reference block of the current macro block, and subtracting the best reference block corresponding to each macro block to obtain a residual error.

(122) And (4) respectively carrying out 8 multiplied by 8 Discrete Cosine Transform (DCT) on the residual error of each macro block calculated in the step (121) to obtain 4DCT coefficient blocks.

(123) Then, the values of X and Y are both 4 calculated from the formula (I). As shown in fig. 2, a partial DCT coefficient of each 8 x 8DCT coefficient block is retained: keeping the reserved DCT coefficient as C_ijWhere i is the number of rows and j isThe number of columns, i 1, 2, …, 4, j 1, 2, …, 4, and other coefficients are set to 0.

(124) And (4) quantizing the 8 × 8DCT blocks of each macro block processed in the step (123), and performing variable length coding together with the motion vector and the macro block mode to obtain the MPEG-4 recompressed video stream.

(2) Transmitting the video recompressed by MPEG-4 in step (1) to the user end through the network.

(3) And (3) the user terminal receives the CIF resolution video stream recompressed by the step (1). The compressed video stream is decoded and resampled 2: 1, wherein the resampling method can adopt a pixel domain method or a compression domain method. The video stream can then be played on a display terminal that supports QCIF (176 x 144) resolution.

Example two

The invention can realize the spatial resolution conversion of 2: 1 and the spatial resolution conversion of video in any proportion. The present invention is further described below with reference to specific examples of spatial resolution conversion from a CIF (352 × 288) format video stream compressed by h.264 to 320 × 240 resolution, i.e., 11: 10 (wide) and 6: 5 (high), wherein each frame of image of the compressed video stream is processed as follows:

(1) performing code rate type transcoding on the input H.264 compressed video stream according to the target code rate requirement, so that the input H.264 compressed video stream is converted into the H.264 compressed video stream at the code rate corresponding to the spatial resolution required by the receiving end:

(11) decoding a frame of H.264 compressed input video stream to a pixel domain to obtain 352 × 288 video spatial resolution, which contains pixel information of each macroblock, a motion vector and a macroblock mode; since the frame size is 352 × 288, it contains 22 × 18 (16 × 16) macroblocks.

(12) Recoding the video stream obtained by decoding in the step (11) under the control of the code rate corresponding to the target spatial resolution of 320 × 240, specifically:

(121) and (3) directly multiplexing the motion vector and the macro block mode obtained by the decoding end in the step (121) for each macro block, refining the motion vector of 3 whole pixels through a window to obtain the optimal reference block of the current block, and subtracting the optimal reference block corresponding to each macro block from each other to obtain a residual error.

(122) Respectively carrying out 4 x 4DCT on the residual error of each macro block obtained in the step (121) to obtain 16 4 x 4DCT coefficient blocks D₀₀，D₀₁，D₀₂，D₀₃，D₁₀，D₁₁，D₁₂，D₁₃，D₂₀，D₂₁，D₂₂，D₂₃，D₃₀，D₃₁，D₃₂，D₃₃Wherein each D_ijRepresenting a 4 x 4 block of DCT coefficients. Then, the 16 4 × 4DCT coefficient blocks are combined into 4 8 × 8DCT coefficient blocks D 'according to formula (II)'₀₀，D′₀₁，D′₁₀，D′₁₁：D′_st＝L₁·D_(s×2)(t×2)·R₁+L₁·D_{(s×2)(t×2+1)}·R₂+L₂·D_{(s×2+1)(t×2)}·R₁+L₂·D_{(s×2+1)(t×2+1)}·R₂

(II)

Wherein,

L_{1} = [\begin{matrix} I \\ 0 \end{matrix}],

L_{2} = [\begin{matrix} 0 \\ I \end{matrix}],

R₁＝[I 0]，R₂＝[0 I]i is a 4 × 4 unit matrix, D'_stIs the synthesized target 8 x 8 block of DCT coefficients, s ∈ [0, 1 ∈]，t∈[0，1]。

(123) Then, the 8 × 8DCT coefficient block synthesized in step (122) is set to 0. The values of X and Y were both 7 calculated from formula (I). The 7 x 7 coefficients of each 8 x 8DCT block are stored in zig-zag scan order. The scanning order here may depend on the scanning mode adopted when reordering the quantized DCT coefficients in a specific video stream compression standard. For the DCT coefficient block shown in fig. 4, if the scanner used in the reordering is in a frame format, the corresponding scanning order is: d₀₀，d₀₁，d₁₀，d₂₀，d₁₁，d₀₂，d₀₃，d₁₂，d₂₁，d₃₀，d₄₀，d₃₁，d₂₂，d₁₃，d₀₄，d₀₅，d₁₄，d₂₃，d₃₂，d₄₁，d₅₀，d₆₀，d₅₁，d₄₂，d₃₃，d₂₄，d₁₅，d₀₆，d₀₇，d₁₆，d₂₅，d₃₄，d₄₃，d₅₂，d₆₁，d₇₀，d₇₁，d₆₂，d₅₃，d₄₄，d₃₅，d₂₆，d₁₇，d₂₇，d₃₆，d₄₅，d₅₄，d₆₃，d₇₂，d₇₃，d₆₄，d₅₅，d₄₆，d₃₇，d₄₇，d₅₆，d₆₅，d₇₄，d₇₅，d₆₆，d₅₇，d₆₇，d₇₆，d₇₇The first 49 coefficient values were left unchanged: d₀₀，d₀₁，d₁₀，d₂₀，d₁₁，d₀₂，d₀₃，d₁₂，d₂₁，d₃₀，d₄₀，d₃₁，d₂₂，d₁₃，d₀₄，d₀₅，d₁₄，d₂₃，d₃₂，d₄₁，d₅₀，d₆₀，d₅₁，d₄₂，d₃₃，d₂₄，d₁₅，d₀₆，d₀₇，d₁₆，d₂₅，d₃₄，d₄₃，d₅₂，d₆₁，d₇₀，d₇₁，d₆₂，d₅₃，d₄₄，d₃₅，d₂₆，d₁₇，d₂₇，d₃₆，d₄₅，d₅₄，d₆₃，d₇₂0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0; if the scanning party adopted in the reordering is in a field format, the corresponding scanning sequence is as follows: d₀₀，d₁₀，d₂₀，d₃₀，d₀₁，d₁₁，d₀₂，d₁₂，d₂₁，d₃₁，d₄₀，d₅₀，d₆₀，d₇₀，d₇₁，d₆₁，d₅₁，d₄₁，d₃₂，d₂₂，d₀₃，d₁₃，d₀₄，d₁₄，d₂₃，d₃₃，d₄₂，d₅₂，d₆₂，d₇₂，d₇₃，d₆₃，d₅₃，d₄₃，d₃₄，d₂₄，d₀₅，d₁₅，d₀₆，d₁₆，d₂₅，d₃₅，d₄₄，d₅₄，d₆₄，d₇₄，d₇₅，d₆₅，d₅₅，d₄₅，d₃₆，d₂₆，d₁₇，d₀₇，d₂₇，d₃₇，d₄₆，d₅₆，d₁₆，d₇₆，d₇₇，d₆₇，d₅₇，d₄₇Likewise, the first 49 coefficient values are kept unchanged d₀₀，d₁₀，d₂₀，d₃₀，d₀₁，d₁₁，d₀₂，d₁₂，d₂₁，d₃₁，d₄₀，d₅₀，d₆₀，d₇₀，d₇₁，d₆₁，d₅₁，d₄₁，d₃₂，d₂₂，d₀₃，d₁₃，d₀₄，d₁₄，d₂₃，d₃₃，d₄₂，d₅₂，d₆₂，d₇₂，d₇₃，d₆₃，d₅₃，d₄₃，d₃₄，d₂₄，d₀₅，d₁₅，d₀₆，d₁₆，d₂₅，d₃₅，d₄₄，d₅₄，d₆₄，d₇₄，d₇₅，d₆₅，d₅₅，0，0，0，0，0，0，0，0，0，0，0，0，0，0，0。

(124) Processing each 8 x 8DCT coefficient block D' of each macro block after step (123)_stSplit into 4 x 4 blocks of DCT coefficients

The specific operation is as follows:

0≤i≤1，0≤j≤1 (III)

wherein L is₀＝[I 0]And L₁＝[0 I]Is a matrix of 4 x 8 and,

R_{0} = [\begin{matrix} I \\ 0 \end{matrix}]

and

R_{1} = [\begin{matrix} 0 \\ I \end{matrix}]

is a matrix of 8 × 4 matrices, and I is a unit matrix of 4 × 4. Then, each 4 × 4DCT coefficient block obtained by splitting is quantized according to the corresponding quantization strategy of h.264, and the quantized coefficients, their motion vectors and macroblock modes are variable length coded to obtain a recompressed video stream.

(2) And (3) transmitting the video stream recompressed by the H.264 in the step (1) to a user end through a network.

(3) And (3) the user side receives the video stream with the CIF resolution which is recompressed in the step (1). The compressed video stream is decoded and resampled at 11: 10 (width) and 6: 5 (height), wherein the resampling method can adopt a pixel domain method and a compressed domain method. The video stream can then be played on a display terminal that supports a 320 x 240 resolution.

Claims

1. A video spatial resolution conversion method based on code rate type transcoding assistance processes each frame image of a compressed video stream according to the following method:

(1) carrying out code rate type transcoding on the frame image:

(11) decoding a frame image to a pixel domain to obtain video spatial resolution M multiplied by N of the frame image, and pixel values, motion vectors and macroblock coding modes of each macroblock contained in the frame image;

(124) keeping X multiplied by Y DCT coefficient values of the DCT coefficient block of each macro block unchanged, and setting the other coefficient values as 0;

(125) quantizing each macro block according to the DCT coefficient block processed in the step (124), and then performing variable length coding on a quantization result, a motion vector and a macro block mode;

(2) transmitting the encoding result of the step (125) to the user terminal;

(3) and the user end decodes the coding result and resamples the video obtained by decoding according to the size of the display screen of the user end.

2. The method of claim 1, wherein said step (124) assigns a coefficient value C to each DCT coefficient block of each macroblock_ijThe remaining rows i are 1, 2, …, X, and the columns j are 1, 2, …, Y.

3. The method of claim 1, wherein said step (124) leaves unchanged the first X Y DCT coefficient values for the coefficients of each DCT coefficient block of each macroblock in a zig-zag scan order.