CN107295334B - Adaptive reference picture chooses method - Google Patents

Abstract

The present invention relates to adaptive reference pictures to choose method, comprising: A. obtains the reference picture set R of t frame in coded image_Set(t)；B. the image pic for being k by timing_kAs reference picture, the i-th frame is calculated in reference pic_kWhen the bit number and pic that consume_kReference value；C. the reference value of each timing image is obtained, and then obtains over the reference value set of continuous w image；D. the mean value and variance of all w reference values in reference value set are calculated, and the farthest reference distance L of interframe is arranged by the ratio of mean variance；E. the selection of time domain or quality is carried out to image according to the size of the mean value and variance, and new reference picture set is set；F. according to new reference picture set

Description

Adaptive Reference Image Selection Method

technical field

The invention relates to an image processing method, specifically an adaptive reference image selection method.

Background technique

Video coding is essentially a process of continuously removing spatial and temporal redundancy of video signals through various algorithms, strategies, and tools. The elimination of spatial redundancy is usually realized by intra-frame prediction, and the removal of temporal redundancy is realized by inter-frame prediction. In the course of historical development, many new technologies and methods have been proposed and applied to video encoders, such as entropy coding, quadtree block segmentation, multi-directional mode for intra-frame prediction, bi-directional multi-reference mode for inter-frame prediction, decoding end Export vector technology and more. The common goal of these methods is to reduce the bit overhead as much as possible while ensuring the encoding quality. Coding quality is usually measured by peak signal ratio (PSNR), and the number of bits (Bits) is usually converted into bit rate (Bit Rate, BR) according to the frame rate (Frame Rate, FR) of the video sequence. In terms of technical terms, the source video sequence is composed of time-series adjacent frames. When a frame enters a codec, the entire process is called an image.

The international standards for video coding range from H.264/AVC to HEVC/H.265, and the national standards range from AVS1 to AVS+ to AVS2. It has been nearly two decades of development. The formulation of the latest international standard H.266 and the national standard AVS3 has recently started. The great progress of coding technology in the past two decades has made the coding structure more standardized. The introduction of the Hierarchical Reference Structure (HRS) in 2002 made the inter-picture reference structure more stable during the coding process. In today's mainstream coding configurations, both Low Delay (LD) and Random Access (RA) have the spirit of HRS, and are solidified according to the agreed reference structure. There are indications that in the forthcoming H.266 standard, user-defined inter-image reference adjustment may be canceled. These circumstances show that most engineers and researchers are more accepting of the current reference image management methods. HRS is very mature and has excellent performance.

At the same time, the content and picture quality of video sources vary greatly, and it is not appropriate to use a priori reference management method to apply to all situations. For a frame sequence with intense motion, the temporal correlation between adjacent frames is strong, and the temporal correlation drops sharply at a slightly longer time distance. However, the scene is still and the content of the frame is simple, even at a long time distance, its temporal correlation is still preserved.

The temporal redundancy in the process of inter-picture reference is expressed by "vector + compensation" obtained by motion estimation. Even considering the optimal selection of rate-distortion optimization techniques, the encoder always tends to select those modes with smaller "vector + compensation" for the final transform and entropy encoding. At this time, the quality of the reference image itself plays a very important role. This is because the encoder always tends to choose the block with the smallest residual error when "compensating" as a reference. In order to improve the quality of image coding, the increase in the number of bits is unavoidable overhead. Generally, for a certain frame, the smaller the quantization parameter QP value used in encoding, the higher the quality and the more bits will be consumed; the larger the quantization parameter QP value used, the lower the quality and the less bits will be consumed. Its monotonicity has been confirmed by information theory. Relatively more bits are paid in exchange for a higher quality of an image. If the reference frequency of subsequent images is high and its high quality is transmitted in the time domain, the reference value obtained by the number of invested bits is very high. Conversely, if it is referenced by subsequent images infrequently, its high quality will be quickly truncated in the time domain, and the reference value of the number of invested bits is very low.

For video sequences with sharp motion changes, the temporal correlation is limited to very adjacent frames. For video sequences that are stationary or change little, inter-frame temporal correlations can persist over long temporal distances. For the former case, it is most beneficial to the overall coding performance to invest as many bits as possible in each frame to maintain a uniform coded image quality. For the latter case, it is appropriate to invest a large number of bits to encode an image with higher quality and use it for subsequent long-distance reference, which is most beneficial to the overall encoding performance.

In view of this, it is necessary to independently decide which images to set as long-term reference (Long-Term Reference, LTR) in the reference decoding buffer during the encoding process, and invest more encoding bits to ensure its reference quality; The image is set as a short-term reference (Short-Term Reference, STR) and reasonably moved out of the reference decoding buffer to ensure high temporal correlation between images.

Contents of the invention

The invention provides an adaptive reference image selection method, which can independently determine the basis for selecting images, and judge and select the image quality or the temporal correlation between images according to different situations.

The adaptive reference image selection method of the present invention includes:

A. Select a plurality of coded images in one frame of the coded image as a reference, and obtain the reference image set R _Set (t) of the tth frame, which includes its temporally adjacent reconstruction Reference image pic _t-1 and the most recent consecutive three forward key images;

B. According to the parameter setting of the reference image set R _Set (t), the image pic _k whose time sequence is k is used as the reference image, and the number of bits consumed by the i-th frame when referring to the reference image pic _k is calculated, and the number of bits consumed is about It is equal to the sum of the number of bits of all the blocks of the cumulative calculation reference image pic _k , and then obtains the reference value of the reference image pic _k whose time sequence is k;

C. According to the calculation method of the reference value, obtain the reference value of each time-series image, and then obtain the reference value set IP _Set (t) of w consecutive images in the past, and t>w, t mod w=0;

D. Calculate the mean and variance of all w reference values in the reference value set, and set the farthest reference distance L between frames through the ratio of the mean variance;

E. Compare the size of the mean and variance, if the mean is large and the variance is small, select the nearest image in the time domain as a reference; if the mean is small and the variance is large, select an image with good quality instead of the nearest image in the time domain as a reference; The selected reference image resets the new set of reference images for frame t

F. According to the new reference image set The quantization parameter encoded in each frame is offset-set according to its reference situation.

Specifically, the calculation method of the reference image set of the tth frame described in step A is: Where pic is the reference image.

Specifically, the calculation method of the number of bits consumed by the i-th frame in step B when referring to the reference image pic _k is as follows: first calculate the reference value generated by the encoding of the reference image pic _k where |kt| represents the absolute value of the inter-frame reference distance, |Δx _b | and |Δy _b | represent the absolute values of the inter-frame predicted motion vectors in the horizontal and vertical directions of the block b of the i-th frame, respectively, and B represents the i-th The total number of blocks in the frame, SAD _b represents the residual sum after motion compensation of block b, and qstep _k represents the quantization step size used when the time sequence is k reference image pic _k earlier encoding itself; then according to the obtained reference value v(· | ), calculate the influence I( ) of the reference image pic _k : n is the number of pictures to be subsequently encoded depending on the reference picture pic _k .

On this basis, the reference value set IP _Set (t)={I(pic _i )|i∈[tw,t-1]} of the past continuous w images mentioned in step C.

Further, the mean of the w reference values described in step D is: The stated variance is:

Further, the farthest reference distance L between frames is:

Further, the new reference image set of the reset t frame described in step E The method is:

The offset setting is: QP(pic _t )=QP_I+QP_offset-Q(t,4)-Q(t,L), wherein QP_I represents the reference value of the quantization parameter QP input by encoding, and QP_offset represents the quantization parameter QP The maximum offset of , Q( , ) is the quantization parameter QP offset correction, and its formula is:

The self-adaptive reference image selection method of the present invention can adaptively select image quality or time-domain correlation according to different image requirements, and can formulate reference image management rules according to image content change characteristics to significantly improve video sequence coding performance .

The above-mentioned content of the present invention will be further described in detail below in conjunction with the specific implementation manners of the examples. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following examples. Without departing from the above-mentioned technical idea of the present invention, various replacements or changes made according to common technical knowledge and customary means in this field shall be included in the scope of the present invention.

Detailed ways

The number of binary bits (Bits) required to encode an image is called information entropy (H). Specifically, an image is divided into blocks (CU/PU/TU) according to reasonable rules for encoding. The block number defined in the i-th row and j-th column of the t-th frame in the image sequence is denoted as u _t,i,j , and the number of bits consumed by its encoding, that is, the entropy is H(u _t,i,j ). In the inter-frame coding mode, the entropy H(u _t,i,j ) is not generated independently, it is dependent. as shown in formula (1).

In formula (1), Indicates the reference block finally adopted after the optimal mode is selected by the rate-distortion optimization (RDO) from the reference image set R _Set (t) that can be used in the t-th frame, τ is the coding timing POC number where it is located, ξ and Indicates the row and column number of the block. The minimum entropy Encode(·) generated by encoding the entire block consists of three parts, namely E _MV { } represents the entropy coding operation for the motion vector record, and E _QT { } represents the transformation and quantization of the residual Entropy coding operation, E _MODE { } means entropy coding operation for mode information. The normal form ||·|| denotes block u _{t, i, j} and block Residuals, constraints determined reference block Therefore, the formulation and management of the reference image set R _Set (t) will directly affect the coding performance, that is, the information entropy H(u _t,i,j ) generated by coding.

In a relatively limited period of time, the content change rate of the video source tends to be uniform. In inter-frame prediction, that is, in the process of inter-picture reference, one frame can select multiple coded pictures as references, which is called multi-reference. Those images with low distortion and high quality will have a greater probability of being selected for reference, especially those still image contents whose motion vectors tend to zero. Images with high coding distortion and low quality are less likely to be selected as references. Under the multi-reference structure, it is not necessary to encode the quality of multiple coded images that can be referenced by one frame to be very high, as long as there is one high-quality image that belongs to the reference set. This is known as encoder reference picture management. Specific to the present invention, this is adaptive reference image selection.

Without loss of generality, the current mainstream coding standards, such as H.264/AVC, HEVC/H.265, and AVS2, all support multi-reference image management. Generally speaking, the number of reference image management is configured to be four. Taking the LD coding structure as an example, for any inter-frame reference frame timing t, its playback sequence POC is equal to its codec sequence DOC. All the images it can refer to are included in the reference image set R _Set (t).

On the basis of the above, the adaptive reference image selection method of the present invention includes steps:

A. Select a plurality of coded images in one frame of the coded image as a reference, and obtain the reference image set R _Set (t) of the tth frame, which includes its temporally adjacent reconstruction The reference image pic _t-1 and the three most recent consecutive forward key images.

Usually a canonical group of pictures (GOP) contains 4 consecutive images, and their timing subscripts are respectively consecutive {τ+1, τ+2, τ+3, τ+4} where τmod 4=0. Among these 4 frames, the last image frame of the GOP with POC=τ+4 is usually called a key frame, and its subscript is an integer multiple of 4. When the key frame is encoded and decoded into the reference image pic _τ+4 , more bits will be invested in exchange for better decoding quality, so as to benefit subsequent multiple inter-frame references of other non-key images.

Therefore, the expression of the reference image set R _Set (t) of the tth frame is:

Formula (2) can be expressed as the reference image set of the tth frame contains its temporally adjacent reconstructed reference image pic _t-1 , and the latest three consecutive forward key images, which takes into account the strong temporal correlation between frames , taking into account the spatial correlation of high-quality reference images, has long been fixed. Consistent with this, the allocation rule of the quantization parameter QP of the key image in the encoding also conforms to the reference set setting stipulated by formula (2). The value of the quantization parameter QP used in key image coding should be smaller than that of other non-key images to obtain higher image quality.

In the management of the reference image in formula (2), the time sequence distance between the reference image and the current frame will be more than 12, that is, the key image will be referenced 12 times by the subsequent image. For video images with severe motion changes, the content correlation between frames will be attenuated to a minimum at 2 or 3 timing distances; for video images with slow motion changes, the content correlation between frames will be at hundreds of timing distances The distance will also be high. The former invests in high bit rate with a step size of 4 in exchange for reference value. Due to the weak inter-frame correlation, the probability of reference hits is small, such investment is unnecessary and wastes the number of bits; the latter also invests in high bit rate with a step size of 4 , but 3 key images are not needed for reference, only one high-quality image is enough, and the high-quality reference image can be set as long as possible in terms of timing, so the frequent input of high bit rate is also unnecessary, wasting number of bits. Combining the above two situations, adaptively setting the quantity, quality, and distance of key images will directly save bits and improve coding efficiency.

B. According to the parameter setting of the reference image set R _Set (t), the image pic _k with time sequence k is used as the reference image, which will be referenced by several consecutive frames. Calculate the number of bits consumed by the i-th frame when referring to the reference image pic _k , the number of bits consumed is approximately equal to the sum of the bits of all blocks of the cumulative calculation of the reference image pic _k , specifically:

The influence function I(·) of the reference image pic _k is defined as the sum of the reference value v(·|·) generated by the subsequent n images relying on the reference image pic _k , as shown in formula (3). This is a conditional statistic if and only if those frame encoding processes that use the reference image pic _k as reference are computed.

The reference value v(·|·) is:

The reference value v(·|·) contains not only the hit frequency of the block encoding reference, but also the motion vector and temporal distance between the original image block and the reference block. The above three factors are directly provided in the encoding decision of each block under each frame encoding, and are easy to collect.

In formula (4), |kt| represents the absolute value of the reference distance between frames. |Δx _b | and |Δy _b | represent the absolute values of the inter-frame prediction motion vectors in the horizontal and vertical directions of the b-th block in the i-th frame, respectively, and B represents the total number of blocks in the i-th frame. SAD _b represents the residual sum after motion compensation of the bth block. qstep _k represents the quantization step used when the reference image k was coded earlier, and can be calculated from the quantization parameter QP value. log ₂ (·) is a logarithmic operation with base 2, and the operation of adding 1 to the exponent is to prevent the operation from going out of bounds.

In this way, it is obtained that the number of bits consumed by the i-th frame when referring to the reference image pic _k is approximately equal to the sum of the bits of all blocks of the reference image pic _k accumulated and calculated.

Further, the reference value of the reference image pic _k whose timing is k is obtained: combining formulas (3) and (4), the reference value of the reference image pic _k is:

C. The reference value I(·) of each time-series image can be obtained by performing statistical operations on all reference images according to formula (5). Define the farthest reference distance between frames as L, that is, the farthest time-domain distance that can be reached when two images are used as inter-frame reference. For applicability, the definition domain of the farthest reference distance L is generally in the interval , where the minimum value of 4 indicates that the minimum reference distance is the number of reference frame configurations, and FR is the frame rate, that is, the number of frames refreshed in 1 second. is the rounding up operation. Without loss of generality, the time window is taken as the upper bound of the definition domain of the farthest reference distance L Then, when the current encoding frame number is t, the reference value set IP _Set (t) of w consecutive images in the past is shown in formula (6), generally requiring t>w, t mod w=0.

IP _Set (t)＝{I(pic _i )|i∈[tw,t-1]} (6)

D. Take the w reference value I(pic _i ) in the reference value set IP _Set (t) as a sample for mean μ _t and variance registration. The mean of the statistical reference value set IP _Set (t) is:

The mean of the statistical reference value set IP _Set (t) is:

Then set the farthest reference distance L between frames by the ratio of the mean variance:

E. Comparing the mean μ _t and variance , if the mean μ _{t is} large and the variance is small, select the nearest image in the time domain as a reference; if the mean μ _{t is} small and the variance Large, choose a good-quality image instead of the closest image in the time domain as a reference; reset the new reference image set of the t-th frame according to the selected reference image

F. According to the new reference image set The quantization parameters encoded in each frame are offset and set according to their reference conditions: with the reset new reference image set To be consistent, the quantization parameter QP value used in each frame encoding is offset according to its reference situation, as shown in formula (11):

QP(pic _t )=QP_I+QP_offset-Q(t,4)-Q(t,L) (11)

In formula (11), QP_I represents the reference QP value of the encoding input, QP_offset represents the maximum offset of the frame encoding QP value, which is usually fixed at 3, and Q(·,·) is the inter-frame reference QP value offset correction amount, Its calculation process is:

Claims (6)

1. adaptive reference picture chooses method, feature includes:

A. it selects multiple encoded images as reference the frame in coded image, obtains the reference picture set R of t frame_Set (t), the reference picture set of the t frame contains the reference picture pic of the adjacent reconstruct of its timing_t-1With continuous three recently Forward direction key images；

B. according to reference picture set R_Set(t) parameter setting, the image pic for being k by timing_kAs reference picture, i-th is calculated Frame is in reference reference picture pic_kWhen the bit number that consumes, the bit number of the consumption is approximately equal to accumulation and calculates reference picture pic_k All pieces of the sum of bit number, and then obtain timing be k reference picture pic_kReference value；

I-th frame is in reference reference picture pic_kWhen the calculation method of bit number that consumes are as follows: first calculate and rely on reference picture pic_kIt compiles Reference value caused by codeWherein | K-t | indicate inter-reference apart from absolute value, | Δ x_b| and | Δ y_b| respectively indicate the inter-prediction motion vector of the i-th frame b block In the absolute value of horizontal and vertical directions, B indicates the sum of block in the i-th frame, SAD_bIt is residual after indicating b block motion compensation Difference and qstep_kExpression timing is k reference picture pic_kThe quantization step used when encoding itself in the early time, Bit (b) indicate ginseng Examine image pic_kB block bit number；Then according to obtained reference value v (|), reference picture pic is calculated_kShadow It rings power I ():N is subsequent dependence reference picture pic_kThe image number of coding；

C. according to the calculation method of the reference value, the reference value of each timing image is obtained, and then obtains over continuous w The reference value set IP of a image_SetAnd t > w, tmodw=0 (t),；

D. the mean value and variance of all w reference values in reference value set are calculated, and is arranged by the ratio of mean variance The farthest reference distance L of interframe；

E. the size of the mean value and variance, if mean value is big, variance is small, the image for selecting time domain nearest is made reference；Such as Fruit mean value is small, variance is big, selects high-quality image and the nearest image of non-temporal makes reference；According to the reference picture weight of selection The new reference picture set of new setting t frame

F. according to the new reference picture setBy quantization parameter that each frame encodes according to its be referenced situation into Line displacement setting.

2. adaptive reference picture as described in claim 1 chooses method, it is characterized in that: t frame described in step A Reference picture setWherein pic is reference Image.

3. adaptive reference picture as described in claim 1 chooses method, it is characterized in that: connect in the past described in step C The reference value set IP of continuous w image_Set(t)={ I (pic_i)|i∈[t-w,t-1]}。

4. adaptive reference picture as described in claim 1 chooses method, it is characterized in that: w reference price described in step D The mean value of value are as follows:The variance are as follows:

5. adaptive reference picture as claimed in claim 4 chooses method, it is characterized in that: the interframe farthest refer to away from From L are as follows:

6. adaptive reference picture as claimed in claim 5 chooses method, it is characterized in that: it is reset described in step E T frame new reference picture setMethod are as follows:

The offset setting are as follows: QP (pic_t)=QP_I+QP_offset-Q (t, 4)-Q (t, L), wherein QP_I presentation code is defeated The a reference value of the quantization parameter QP entered, QP_offset indicate that the maximum offset of quantization parameter QP, Q () are quantization parameter QP offset correction, formula are as follows: