CN101883284B - Video encoding/decoding method and system based on background modeling and optional differential mode - Google Patents

Abstract

The invention discloses video encoding/decoding method and system based on background modeling and an optional differential mode. The encoding method comprises the following steps of: modeling and generating a background image by using an input video sequence, and encoding to obtain a reconstructed background image; carrying out pixel or sub-pixel accuracy global motion estimation on each input image to obtain a global motion vector; and carrying out encoding on each video block by selectively using an original mode and a differential mode on the basis of the reconstructed background image and the global motion vector. The invention can improve the encoding performance and has the advantages of not increasing the encoding delay and being beneficial to further processing as the code stream contains the background image per se.

Description

Video encoding/decoding method and system based on background modeling and optional differential mode

technical field

The present invention relates to video compression technology in the technical field of digital media processing, and more particularly to a video encoding/decoding method and system based on background modeling and optional differential mode.

Background technique

Video compression (also known as video coding) is one of the key technologies in applications such as digital media storage and transmission. Its purpose is to reduce the amount of data in storage and transmission by eliminating redundant information. All current mainstream video compression standards adopt a block-based predictive transformation hybrid coding framework, which eliminates statistical redundancy (including spatial redundancy, temporal redundancy, and information entropy redundancy) in video images through methods such as prediction, transformation, and entropy coding. surplus), in order to achieve the purpose of reducing the amount of data. Because video data has scene-invariant characteristics in a certain period of time, in recent years, with the development and progress of background modeling technology, background modeling technology has been more and more used in video coding, and rational use of built-in The background generated by the model can further eliminate the information redundancy in the video, so as to obtain better compression performance.

For the use of modeling background, the current main methods are divided into two categories. One is to break away from the block-based predictive transform hybrid coding framework, and use object-based video compression technology to generate background by using modeling algorithms to complete object detection, object Tracking, foreground/background segmentation and other technologies separate each object in the video, and adopt different compression methods for different objects to further mine the information redundancy in the video, thereby improving the compression efficiency. Therefore, the object-based video compression method is an important research direction to solve the video compression problem, but there are two problems: first, object detection and segmentation in video is still an unsolved problem in the field of computer vision and image processing , the existing methods are still not ideal in terms of correctness and accuracy of detection and segmentation, which has become a bottleneck in object-oriented video compression methods; second, the computational complexity of the above-mentioned object detection and segmentation methods is high, which is not conducive to encoder realization.

Therefore, another type of coding method that is still based on the block-based predictive transform hybrid coding framework has become another research direction that has attracted much attention.

Contents of the invention

The invention provides a video encoding/decoding method and system based on background modeling and optional differential mode. Based on the present invention, the coding performance can be better improved.

On the one hand, the present invention discloses a video encoding method based on background modeling and optional differential mode, the method includes the following steps: a background modeling step, using input video sequence modeling to generate a background image, and obtaining a reconstructed image after encoding construct a background image; the global motion estimation step is to perform global motion estimation of pixel or sub-pixel precision on each input image to obtain a global motion vector; the mode selection step is to select based on the reconstructed background image and the global motion vector Each video block is encoded using the original mode and the differential mode selectively.

In the above video encoding method, preferably in the background modeling step, the step of using the input video sequence to model and generate the background image includes: for each pixel, find the pixel set of the pixel in the training set, and then start traversing; For each pixel value, according to the difference between the current pixel value and the next adjacent pixel value in this pixel set, the dynamic threshold value generated at the current moment is used for judgment. If the absolute value of the difference is greater than the threshold value, it is determined that the current data segment is over. , start the next data segment; furthermore, divide the entire pixel set at the current pixel position into several data segments; assign a weight to each segment, and the weight is the size of the segment data set; based on the weight, the calculated The background pixel value of the above pixel.

In the video coding method above, preferably, the background modeling step further includes a step of periodically reselecting a training set to update the background image.

In the above video encoding method, preferably in the background modeling step, the background image is encoded to obtain a reconstructed background image, the encoding method includes: using a lossy or lossless encoding method for image or video encoding modeling Generated background images; or regard all background images as a sequence, and use video encoding method for encoding, the video encoding method is: MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS , JPEG, JPEG2000 or MJPEG.

In the above video encoding method, preferably in the global motion estimation step, the global motion estimation includes: using data blocks as the basic unit, making the current image use the reconstructed background image as a reference image to perform global, integer or sub-pixel motion search , take the median, maximum cluster or average value of the motion vector set as the global motion vector of the current image.

In the above video encoding method, preferably in the mode selection step, the selective use of the original mode and the differential mode to encode each video block is selected by comparing the rate-distortion results of the original mode and the differential mode.

In the above video encoding method, preferably, the original mode encoding is as follows: according to the global motion vector, find the corresponding data in the background image of the prediction reference data of the data to be encoded, and if the prediction reference data has been encoded as a differential mode, then use the decoding of the two The superimposed value is a reference; otherwise, the decoded original value of the prediction reference data is directly used as a reference to directly encode the data to be encoded.

In the above video encoding method, preferably, the differential mode encoding is as follows: according to the global motion vector, it is the data corresponding to the matching of the prediction reference data of the data to be encoded in the background image, and if the prediction reference data has been encoded into the original mode, then the two Otherwise, the decoded original value of the prediction reference pixel is directly used as a reference to encode the difference data between the data to be encoded and the corresponding data in the background image.

In the above video encoding method, it is preferred that the regular reselection of the training set to update the background image specifically includes: updating the background image according to a video segment, and the video segment is an input video encoded using the same reconstructed background image Sequence, the entire input video sequence can be regarded as composed of several end-to-end video segments. When encoding, the training image set is selected from the current video segment for background modeling and a background image is generated for the encoding of the next video segment. , so that when the current video segment is encoded, the background image generated while the previous video segment is encoded is used.

On the other hand, the present invention also discloses a video encoding system based on background modeling and optional differential mode, including: a background modeling module, which is used to model and generate a background image using an input video sequence, and obtain a reconstructed image after encoding. Constructing a background image; a global motion estimation module for performing global motion estimation with pixel or sub-pixel precision on each input image to obtain a global motion vector; a mode selection module for based on the reconstructed background image and the global Motion vector, optionally using raw mode, differential mode to encode each video block.

The above-mentioned video coding system, preferably in the background modeling module, the use of the input video sequence modeling to generate the background image includes: for each pixel, find the pixel set of the pixel in the training set, and then start Traverse; for each pixel value, according to the difference between the current pixel value and the next adjacent pixel value in this pixel set, use the dynamic threshold generated at the current moment to judge, if the absolute value of the difference is greater than the threshold, then judge the current data end of the segment, and start the next data segment; furthermore, the pixel set at the entire current pixel position is divided into several data segments; a weight is assigned to each segment, and the weight is the size of the segment data set; based on the weight, calculate The pixel value of the submodule of the background pixel.

In the above video coding system, preferably, the background modeling module further includes a submodule for periodically reselecting a training set to update the background image.

In the above video coding system, preferably in the background modeling module, the background image is coded to obtain a reconstructed background image, and the coding method includes: using a lossy or lossless, image or video coding method to code and model Generated background images; or regard all background images as a sequence, and use video encoding method for encoding, the video encoding method is: MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS , JPEG, JPEG2000 or MJPEG.

In the above-mentioned video coding system, preferably, the global motion estimation module further includes: using the data block as the basic unit, making the current image use the reconstructed background image as the reference image to perform global, integer or sub-pixel motion search, and obtain the motion vector The median, largest cluster or mean of the set is a submodule of the global motion vector for the current image.

In the above video coding system, preferably in the mode selection module, the selective use of the original mode and the differential mode to encode each video block is selected by comparing the rate-distortion results of the original mode and the differential mode.

In the above video coding system, it is preferable that the original mode coding is as follows: according to the global motion vector, find the corresponding data in the background image of the prediction reference data of the data to be coded, and if the prediction reference data has been coded as a differential mode, then use the decoding of the two The superimposed value is a reference; otherwise, the decoded original value of the prediction reference data is directly used as a reference to directly encode the data to be encoded.

In the above video encoding system, preferably, the differential mode encoding is as follows: according to the global motion vector, it is the data corresponding to the matching of the prediction reference data of the data to be encoded in the background image, and if the prediction reference data has been encoded into the original mode, the two Otherwise, the decoded original value of the prediction reference pixel is directly used as a reference to encode the difference data between the data to be encoded and the corresponding data in the background image.

In the above video coding system, it is preferred that the regular reselection of the training set to update the background image specifically includes: updating the background image according to a video segment, and the video segment is an input video encoded using the same reconstructed background image Sequence, the entire input video sequence can be regarded as composed of several end-to-end video segments. When encoding, the training image set is selected from the current video segment for background modeling and a background image is generated for the encoding of the next video segment. , so that when the current video segment is encoded, the background image generated while the previous video segment is encoded is used.

On the other hand, the present invention also discloses a video decoding method corresponding to the above-mentioned video coding method, including: decoding background images and global motion vectors; performing original mode or differential mode decoding on each video block.

In the above video decoding method, it is preferred that the original mode decoding includes: if the encoding of the data to be decoded is in the original mode, according to the global motion vector, the corresponding data of the prediction reference data in the background image is obtained; if the prediction reference data is coded as a differential mode , the decoded superimposed value of the two is used as a reference, otherwise, the decoded data to be decoded is directly decoded by using the decoded original value of the prediction reference data as a reference.

In the above video decoding method, preferably, the differential mode decoding includes: if the encoding of the data to be decoded is a differential mode, according to the global motion vector, the corresponding data of the prediction reference data in the background image is obtained; if the predicted reference data has been encoded as the original mode , then use the decoded difference value of the two as a reference, otherwise, directly use the decoded original value of the prediction reference data as a reference to decode the current data to be decoded, and then the decoded data is superimposed with the corresponding data in the background image.

On the other hand, the present invention also discloses a video decoding system corresponding to the above-mentioned video coding system, including: a module for decoding background images and global motion vectors; decoded module.

In the above video decoding system, preferably, the original mode decoding includes: if the encoding of the data to be decoded is the original mode, according to the global motion vector, the corresponding data of the prediction reference data in the background image is obtained; , the decoded superimposed value of the two is used as a reference, otherwise, the decoded data to be decoded is directly decoded by using the decoded original value of the prediction reference data as a reference.

In the above video decoding system, preferably, the differential mode decoding includes: if the encoding of the data to be decoded is a differential mode, according to the global motion vector, the corresponding data of the prediction reference data in the background image is obtained; if the predicted reference data has been encoded as the original mode , then use the decoded difference value of the two as a reference, otherwise, directly use the decoded original value of the prediction reference data as a reference to decode the current data to be decoded, and then superimpose the decoded data with the corresponding data in the background image to obtain the final the decoded data.

Compared with the prior art, the present invention has the following characteristics: first, no object or foreground/background segmentation; second, encoding is performed in units of blocks or macroblocks; third, global motion compensation is added; fourth , using the mode selection method to select the best from two types of coding modes to ensure the coding efficiency. The invention can improve the encoding performance, and has the advantages of not increasing the encoding delay, and the code stream itself contains the background image, which is beneficial to further processing.

Description of drawings

Fig. 1 is a flow chart of steps of an embodiment of a video encoding method based on background modeling and optional differential mode in the present invention;

Fig. 2 is a block diagram for implementing the video coding method of the present invention;

Fig. 3 is the corresponding relationship between the data in the current image and the background image under global motion estimation;

Fig. 4 is a schematic diagram of the background modeling process;

Fig. 5 is the selection diagram of training set;

Fig. 6 is an example of decoding the current data to be encoded as a differential mode, and the prediction reference has been encoded as an original mode;

Fig. 7 is an example of encoding where the current data to be encoded will be encoded as a differential mode, and the prediction reference has been encoded as a differential mode;

Fig. 8 is an example of encoding the current data to be encoded into the original mode, and the prediction reference has been encoded into the differential mode;

Figure 9 is an example of decoding where the current data to be decoded has been encoded as a differential mode and the prediction reference has been encoded as an original mode;

FIG. 10 is an example of coding in which the current data to be decoded has been coded in differential mode and the prediction reference has been coded in differential mode;

Fig. 11 is an example of decoding where the current data to be decoded has been encoded as the original mode and the prediction reference has been encoded as the differential mode.

Detailed ways

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

In the present invention, a background modeling method is used for the relatively fixed scene in the video sequence to generate a background image for description by utilizing the feature that the scene in the video sequence is fixed within a certain period of time. In the encoding process, by establishing and updating the background image that describes a relatively fixed scene in the video, a newly introduced differential mode encoding method is selectively used to encode each data block, thereby eliminating the video sequence to a greater extent. Redundancy for better compression performance. Correspondingly, the basic idea of the encoding method is as follows: firstly establish and update the background image to describe the fixed scene information of the video image; then use the optional global motion estimation to obtain the global motion vector of each image; then directly encode An optimal encoding mode is selected between the original video encoding mode of the original data and the differential video encoding mode of encoding the differential result of the original data and the corresponding background data. In the above coding algorithm idea, the background image is obtained by using the original input video image through background modeling, and the generated background image needs to be encoded into the code stream; if global motion estimation is used, the global motion vector also needs to be written into the code stream; the design requirements of the decoding end match the encoding end, that is, when the code stream is decoded, the background image and the optional encoded global motion vector are first directly decoded.

Referring to FIG. 1, FIG. 1 is a flow chart of the steps of an embodiment of a video coding method based on background modeling and optional differential mode in the present invention, including the following steps:

The background modeling step S1 is to use the input video sequence to model and generate a background image, and the background image is encoded and decoded to obtain a reconstructed background image; the global motion estimation step S2 is to perform global motion estimation with pixel or sub-pixel precision on each input image , to obtain the global motion vector; the mode selection step S3, based on the reconstructed background image and the global motion vector, selectively use the original mode and the differential mode to encode each video block.

The above embodiments have the following features: first, no object or foreground/background segmentation; second, encoding is performed in units of blocks or macroblocks; third, global motion compensation is added; fourth, mode selection is adopted , choose the best from the two coding modes to ensure the coding efficiency. The invention can improve the encoding performance, and has the advantages of not increasing the encoding delay, and the code stream itself contains the background image, which is beneficial to further processing. Moreover, the video compression technology based on background modeling is especially suitable for video surveillance, video conferencing, smart rooms, etc. The videos of these application scenarios have the characteristics of long scene maintenance time and extremely low camera switching frequency, which is beneficial to improve Compression efficiency.

Referring to Fig. 2, the present invention can be completed under the codec framework as shown in Fig. 2, which includes background modeling, optional global motion estimation, global motion compensation, background image encoding, background image decoding, differential There are seven functional units of mode encoding and original mode encoding, which respectively complete the background modeling algorithm, the optional global motion estimation algorithm, the data compensation operation with the background image, the background image encoding algorithm, the background image decoding algorithm, and the differential mode. Data encoding method, data encoding algorithm operation in the original mode; the corresponding decoding end is composed of differential mode decoding, original mode decoding, background image decoding, difference and background superposition unit, which respectively realize the decoding of the encoded data in the differential mode, and the decoding in the original mode. Decoding of encoded data, superimposition operation of differential decoding result and corresponding data in the background image.

As shown in FIG. 2 , the coding end of this embodiment includes a background image modeling operation for accepting an input video sequence, a background image coding operation for accepting the output of the background image modeling operation, and a background image for connecting the output of the background image coding operation. A decoding operation, an optional global motion estimation operation that accepts an input video sequence and a background image decoding module output, that accepts an optional global motion estimation vector, a global motion compensation operation that accepts an optional global motion estimation vector, an input sequence and a background image decoding output The input video sequence, the global motion estimation vector output by the global motion estimation module and the differential data output by the global motion compensation operation raw mode encoding operation, accepts the input video sequence and the optional global motion estimation vector output by the global motion estimation module differential mode encoding operation , and finally select to accept the encoded information and the encoded code stream output by the encoding operation in the differential mode or the encoded information and the encoded stream output by the encoding operation in the original mode through the mode selection. At the decoding end, it includes the background image decoding operation that connects and accepts the encoded code stream input by the encoding end; the differential mode decoding operation that selectively accepts the input code stream information according to the mode selection flag bit; selectively accepts the input code according to the mode selection flag bit Raw mode decoding operation for stream information; differential and background overlay operation accepting differential block decoded output and background image decoded output. The following present invention describes in detail the video encoding and decoding method based on the background model described in FIG. 2 and the functions and implementation methods of the various operations involved:

1. Background Image Modeling

1. For the input video sequence, select a training image set, model and generate a background image and send it to the video encoding module 3 for compression. The output of this module is the first frame before the first background image is generated. Taking the brightness component as an example, the modeling method used includes but is not limited to the background modeling algorithm shown in Figure 3, including the following steps:

Step S11, initialize the average value of the current pixel position, the threshold;

Step S12, establishing a data segment, initializing its weight and average value;

Step S13, read a new pixel in the training set;

Step S14, judging: Is the square of the difference between temporally adjacent pixels greater than a threshold? If yes, execute step S15a, if not execute step S15b;

Step S15a, updating the average value of the current data segment;

Step S16a, update the threshold under the current weight;

Step S17a, adding 1 to the weight of the current data segment;

Step S15b, updating the total average value and the threshold value under the weight of the current data segment;

Step S16b, create a new data segment, initialize the weight and average value;

Step S18, after executing step S17a or step S16b, judge whether the training set is over, if so, execute step S19; if not, execute step S13;

Step S19, updating the total average value under the weight of the current data segment, and using the total average value as the average value of the background pixels.

In Figure 3, the total average value, the average value and weight of each new data segment are initialized to 0. The threshold is initialized as the pixel-level average of the squared difference of the previous two frames of data in the training set. Both overall average and threshold updates are based on weights. Let the weight of data segment i be W _i , the sum of pixels be Sum _i , and the sum of the squares of differences between all adjacent pixels in data segment i be T _i , then the total average value AVG and the threshold Th are calculated as formulas (1) and (2) :

$\mathrm{<span\; class="notranslate">\; AVG</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Sum</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; Th</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Corresponding to the sequence structure required for the background image update mentioned above, whenever a training image set is input, the background modeling is performed again to generate a background image to complete the background image update. The modeling process for the chroma components is the same.

2. Background image encoding operation

The background image generated by the background modeling module is encoded and compressed, and the encoding result is written into the code stream and passed to the background image reconstruction module. Encoders used for encoding include but are not limited to MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS, JPEG, JPEG2000, MJPEG. The configuration of the encoder includes, but is not limited to, encoding each background image in an independent frame, treating all background images as a sequence of background images using IPPP structure encoding, and losslessly compressing each background image. In a specific implementation, we use the AVS coding technology extended to 9-bit input bit width to code the background image with QP=0.

3. Background image decoding operation

Decode and reconstruct the background image code stream output by the background image coding module, and transfer the reconstructed background image to the reconstructed background compensation operation in order to make the codec match. In a specific implementation manner, the background image is decoded by using the AVS decoding technology extended to a 9-bit code stream, and the decoded output is also 9 bits.

4. Optional global motion estimation operation

Using global motion estimation on the reconstructed background image output by the background image decoding operation and the input current video image, a global motion vector is obtained. Including but not limited to taking the data block as the basic unit, making the current image use the reconstructed background image as the reference image to perform global integer or sub-pixel motion search to obtain the median, maximum clustering or average value of the motion vector as the global motion of the current image vector. The obtained global motion vector needs to be written into the code stream. If the global motion vector is zero, you can also use the corresponding flag bit instead of the global motion vector to write into the code stream.

5. Global Motion Compensation Operation

The global motion vector generated by the global motion estimation is used to match the background image data corresponding to the current original data to be encoded in the background image, as shown in FIG. 4 . A differential operation is performed on the matched original data currently to be encoded and the background image data, and then the differential result is output to a differential mode encoding operation for encoding.

6. Differential mode encoding operation

Coding and compressing the differential data blocks output by the reconstructed background compensation operation. The selected encoding algorithm should match the format of the differential data block. For example, when the output of the differential image is 9 bits, a block compression algorithm that can be configured to 9 bits should be selected, as described in the reconstruction background compensation operation. In this mode, we perform differential operations on the prediction reference data used to encode the data block and the current data block data with the corresponding data in the reconstructed background image, so that the reference data of the data block are all differential. Take the brightness difference calculation when the global motion vector is 0 as an example, let s(x, y) be the brightness pixel value of the data to be differenced at position (x, y), and b(x, y) finally reconstruct the background image For the brightness pixel value at position (x, y), the calculation method of differential data includes but not limited to the two algorithms described in the following formula:

r(x, y)=Clip1(s(x, y)-b(x, y)+256), (3)

r(x,y)=Clip2((s(x,y)-b(x,y))>>1+128), (4)

The clip1 in the formula means to limit the calculation result within [0, 511], if it is out of bounds, take the pixel value on the nearest boundary; clip2 means to limit the calculation result within [0, 255], if it is out of bounds, Then take the pixel value on the nearest boundary. Combined with the global motion compensation operation, when the prediction reference data of the current data block is encoded in the original mode, the operation of this unit is shown in FIG. 5 . When the prediction reference data of the current data block is coded in differential mode, the operation of this unit is shown in FIG. 6 . During intra-frame prediction, the reference image in Figs. 5 and 6 may be equivalent to the current image.

7. Raw mode encoding operation

Encode and compress the input data block. The encoding algorithm is selected to match the input data block. For example, when the input data block is 9 bits, a block compression algorithm that can be configured to 9 bits should be selected, as described in the differential image calculation module. In this mode, we superimpose the prediction reference data used to encode the data block with the corresponding data in the reconstructed background image, so that the data referenced by the data block are all non-differential. Taking the calculation of brightness difference when the global motion vector is 0 as an example, let s(x, y) be the decoded brightness pixel value of the differentially encoded data at position (x, y), and b(x, y) be the decoded The brightness pixel value of the background image pixel at position (x, y), the calculation method of the final value r(x, y) of the reference pixel includes but is not limited to the two algorithms described in the following formula:

r(x,y)=Clip1(s(x,y)-256+b(x,y)), (5)

r(x,y)=Clip2((s(x,y)<<1-128)+b(x,y))), (6)

When the prediction reference data of the current data block is encoded in the original mode, this unit directly decodes the code stream. When the prediction reference data of the current data block is coded in differential mode, the operation of this unit is shown in FIG. 7 . During intra-frame prediction, the reference image in FIG. 7 may be equivalent to the current image.

In the implementation of the above encoding method, the method for regularly updating the background image can be described and realized through a new video sequence structure——video segment. A video segment is a relatively long input video sequence (hundreds of frames or longer), and the entire input video sequence can be regarded as composed of end-to-end video segments. The same reconstructed background image is used to compute the difference image for each video segment. During encoding, the background modeling module selects a training image set from the current video segment, performs background modeling and generates a background image for the encoding of the next video segment. From another perspective, when the current video segment is encoded, the background image generated while encoding the previous video segment is used. Therefore, the entire encoding method will not bring additional delay due to the generation of the background image. For the first video segment, the first few images can adopt traditional video coding technology (including but not limited to MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS, JPEG, JPEG2000, MJPEG ) to encode. While encoding these images, the background modeling module selects a training image set from these images as shown in Figure 8, generates the first background image and transmits it to the decoding end. For the next image in the first video segment, the reconstructed image of the above-mentioned first background image can be used to generate a difference image for encoding. The above method can ensure that when the entire sequence starts to encode, there will be no additional delay due to background modeling.

Corresponding to the above sequence structure, the first training image set that is directly coded is encoded into the code stream first. This is followed by the encoded bitstream of the first background image. Next, it is the differential image coding code stream corresponding to other parts other than the first training image set of the first video segment. Afterwards, the background image and differential image corresponding to each video segment code are alternately encoded into the final code stream.

The code stream generated by the above encoding method and system can be decoded by four operations at the decoding end as shown in Figure 2:

1. Background image decoding operation

The code stream of the background image is decoded, and the decoded background image is transmitted to the differential image compensation module.

2. Difference and background overlay operation.

The differential data output by the differential mode block decoding operation is superimposed on the background data corresponding to the global motion vector in the background image, and the superposition result is output.

3. Differential mode decoding operation

Decode the differential mode image code stream written by the encoder. If the referenced pixel is encoded in the original mode during decoding, the reference pixel for decoding the current block must first be obtained according to formulas (3) and (4). After decoding the current block data, it is necessary to reconstruct the final decoded pixel. At this time, taking the luminance component with the global motion vector as 0 as an example, let b'(x, y) and r'(x, y) be the decoding The output background pixel and the pixel value of the reference pixel at the position (x, y), then the pixel value d'(x, y) of the output image at the position can be according to the following formula:

d'(x, y)=Clip1(b'(x, y)+r'(x, y)-256). (7)

d'(x,y)=Clip2(b'(x,y)+((r'(x,y)-128)<<1)) (8)

Formulas (7) and (8) match with formulas (3) and (4) at the encoding end respectively. Combining the operation of difference and background superposition, when the prediction reference data of the data block to be decoded is encoded in the original mode, the unit operation is shown in Figure 9 . When the prediction reference data of the current data block is in differential mode, the operation of this unit is shown in FIG. 10 . During intra-frame prediction, the reference image in Figs. 9 and 10 may be equivalent to the current image.

4. Original mode block decoding operation.

Decode the original mode image code stream written by the encoder. During the decoding process, if the referenced pixel is encoded in the differential mode, the reference for decoding the current block must first be obtained according to formulas (5) and (6). pixels, and the decoded current block data is the final reconstructed decoded image. When the prediction reference data of the data to be decoded is encoded in the original mode, and when the prediction reference data of the data to be decoded is in the differential mode, the operation of this unit is shown in FIG. 11 . During intra-frame prediction, the reference picture in FIG. 11 may be equivalent to the current picture.

An example is given below to illustrate a possible implementation method of the method of the present invention. Set the input video to YUV4:2:0 format, and set the video segment length to 990 frames. The input data pixel values need to be extended to 9 bits by adding 256. The background modeling operation adopts the modeling method based on segmentation weight as shown in Fig. 11 and formulas (1) (2).

Specifically, in the training set, 118 images that are evenly distributed in each video segment can be selected as the training image set, and background modeling is performed on the brightness and each chrominance component respectively to generate a background image for the next video segment. In addition, we introduce video segment 0 as the initial video segment, and the first image is the background image of video segment 0. The pixel values of these generated background images are extended to 9 bits by adding 256, and then the AVS-S encoder RM0903 extended to 9 bits is used to directly encode them into I frames under the condition of QP=0. The background image decoding operation adopts the AVS-S encoder RM0903 decoder extended to 9 bits to realize the decoding of the background image. Global motion estimation is not used, and global motion vectors are not encoded. In the differential mode encoding method, the method described in the foregoing formula (3) is used for differential calculation, and the 9-bit extended AVS-S encoding method is used to encode the current block after the differential. In the original mode encoding method, the method described in the aforementioned formula (5) is used for superposition calculation, and the current block to be encoded is encoded by using the 9-bit extended AVS-S encoding method. The contents of the code stream of the background image and the differential image are in sequence: the first 118 images directly encoded, the first background image, the first video segment, the second background image, and the second video segment.

For the above implementation, the following performance tests were carried out. Eight static camera sequences of indoor/outdoor scenes with 3088 frames were selected for testing, and compared with the Shenzhan Profile of AVS reference encoder RM0903 with general configuration. The implementation example of the method of the present invention can realize the performance gain of 0.92～1.53dB on the SD sequence in the code rate range of 1Mbps～4Mbps, corresponding to the code rate saving of 40.1%～74.76%, in the code rate of 128kbps～768kbps In the rate range, a performance gain of 1.27-1.87dB is realized on the CIF sequence, corresponding to a code rate saving of 36.61%-85.77.

A video encoding/decoding method and system based on background modeling and optional differential mode provided by the present invention has been introduced above in detail. In this paper, the principle and implementation of the present invention have been explained by using specific embodiments. The above embodiments The description is only used to help understand the system and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (24)

1. the method for video coding based on background modeling and optional differential mode, is characterized in that, the method comprises the steps:

The background modeling step is used the video sequence modeling generation background image of inputting, and background image is through reconstructed background image after encoding and decoding;

The overall motion estimation step is carried out the overall motion estimation of pixel or sub-pixel precision to every width input picture, obtain global motion vector;

The model selection step based on background image and the described global motion vector of reconstruct, optionally uses raw mode, difference modes that each video block is encoded.

2. method for video coding according to claim 1, is characterized in that, in described background modeling step, the video sequence modeling generation background image of described use input comprises:

For each pixel, find the pixel set of this pixel in training set, then begin traversal;

For each pixel value, according to poor between current pixel value and next adjacent pixel values in this pixel set, the dynamic threshold that utilizes current time to generate judges, if poor absolute value is greater than threshold value, judge that the current data section finishes, and begins next data segment; And then, the pixel set of whole current pixel position is divided into some data segments;

Distribute a weight for each section, described weight is the size of this segment data set; Based on described weight, the background pixel value of the described pixel that calculates.

3. method for video coding according to claim 2, is characterized in that, described background modeling step also comprises regularly chooses training set again to upgrade the step of described background image.

4. method for video coding according to claim 1, is characterized in that, in described background modeling step, background image encoded to obtain the method for rebuilding background image comprise:

The background image that use diminishes or the coding method coding modeling of harmless, image or video generates; Or all background images are regarded as a sequence, use MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS, JPEG, JPEG2000 or MJPEG encode to this sequence.

5. method for video coding according to claim 1, is characterized in that, in described overall motion estimation step, described overall motion estimation comprises:

Take data block as base unit, make present image carry out take the background image of reconstruct as reference picture the overall situation, whole pixel or minute pixel motion search, getting the intermediate value of motion vector set, maximum cluster or mean value is the global motion vector of present image.

6. method for video coding according to claim 1, it is characterized in that, in described model selection step, described raw mode, the difference modes optionally used encoded to each video block, and the method for selection is: by comparing the rate distortion result of raw mode and difference modes.

7. method for video coding according to claim 6, is characterized in that, described raw mode is encoded to:

According to global motion vector, find the corresponding data of predictive reference data in background image of data to be encoded, if predictive reference data has been encoded to difference modes, take the decoding superposition value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to coming the direct coding data to be encoded.

8. method for video coding according to claim 7, is characterized in that, described difference modes is encoded to:

According to global motion vector, find the data of the correspondence of predictive reference data in background image of data to be encoded, if institute's predictive reference data has been encoded to raw mode, the decoding difference value of the corresponding data in this predictive reference data and its background image is as reference, otherwise, directly take the decoding original value of predictive reference data as the differential data with reference to data corresponding in encode data to be encoded and background image.

9. method for video coding according to claim 3, is characterized in that, the described training set of regularly again choosing is specially to upgrade described background image:

Carry out the renewal of background image according to video-frequency band, one section input video sequence that described video-frequency band is encoded for the background image that uses same reconstruct, whole input video sequence can be regarded as by some end to end video-frequency bands and consists of, during coding, choosing the training plan image set from the current video section carries out background modeling and generates a width background image, for next video-frequency band coding, when making current video section coding, use be background image in the generation of previous video section coding.

10. the video coding system based on background modeling and optional differential mode, is characterized in that, comprising:

The background modeling module is used for using the video sequence modeling generation background image of inputting, and background image is through reconstructed background image after encoding and decoding;

The overall motion estimation module for every width input picture being carried out the overall motion estimation of pixel or sub-pixel precision, obtains global motion vector;

Mode selection module is used for background image and described global motion vector based on reconstruct, optionally uses raw mode, difference modes that each video block is encoded.

11. video coding system according to claim 10 is characterized in that, in described background modeling module, the video sequence modeling generation background image of described use input comprises:

Be used for for each pixel, find the pixel set of this pixel in training set, then begin traversal; For each pixel value, according to poor between current pixel value and next adjacent pixel values in this pixel set, the dynamic threshold that utilizes current time to generate judges, if poor absolute value is greater than threshold value, judge that the current data section finishes, and begins next data segment; And then, the pixel set of whole current pixel position is divided into some data segments; Distribute a weight for each section, described weight is the size of this segment data set; Based on described weight, the submodule of the background pixel value of the described pixel that calculates.

12. video coding system according to claim 11 is characterized in that, described background modeling module also comprises regularly chooses training set again to upgrade the submodule of described background image.

13. video coding system according to claim 10 is characterized in that, in described background modeling module, background image is encoded to obtain in the submodule of background image of reconstruct, described coding method comprises:

The background image that use diminishes or the coding method coding modeling of harmless, image or video generates; Or

All background images are regarded as a sequence, use MPEG-1/2/4, H.263, H.264/AVC, VC1, AVS, JPEG, JPEG2000 or MJPEG encode to this sequence.

14. video coding system according to claim 10 is characterized in that, described overall motion estimation module also comprises:

Be used for take data block as base unit, make present image carry out the overall situation take the background image of reconstruct as reference picture, whole pixel or minute pixel motion search, getting the intermediate value of motion vector set, maximum cluster or mean value is the submodule of the global motion vector of present image.

15. video coding system according to claim 10, it is characterized in that, in described mode selection module, described raw mode, the difference modes optionally used encoded to each video block, and the method for selection is: by comparing the rate distortion result of raw mode and difference modes.

16. video coding system according to claim 15 is characterized in that, described raw mode is encoded to:

According to global motion vector, find the corresponding data of predictive reference data in background image of data to be encoded, if predictive reference data has been encoded to difference modes, take the decoding superposition value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to coming the direct coding data to be encoded.

17. video coding system according to claim 16 is characterized in that, described difference modes is encoded to:

According to global motion vector, find the corresponding data of predictive reference data in background image of data to be encoded, if institute's predictive reference data has been encoded to raw mode, take the decoding difference value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as the differential data with reference to data corresponding in encode data to be encoded and background image.

18. video coding system according to claim 12 is characterized in that, the described training set of regularly again choosing is specially to upgrade described background image:

Carry out the renewal of background image according to video-frequency band, one section input video sequence that described video-frequency band is encoded for the background image that uses same reconstruct, whole input video sequence can be regarded as by some end to end video-frequency bands and consists of, during coding, choosing the training plan image set from the current video section carries out background modeling and generates a width background image, for next video-frequency band coding, make current video section when coding, use be the background image that generates in previous video section coding.

19. one kind generates the video encoding/decoding method of code stream with method for video coding claimed in claim 1, it is characterized in that, comprising:

Decoding and rebuilding background image and global motion vector;

Each video block is carried out raw mode or difference modes decoding.

20. video encoding/decoding method according to claim 19 is characterized in that, described raw mode decoding comprises:

If data to be decoded be encoded to raw mode, according to global motion vector, obtain the corresponding data of predictive reference data in background image; If predictive reference data is for being encoded to difference modes, take the decoding superposition value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to the data to be decoded that directly decode.

21. video encoding/decoding method according to claim 19 is characterized in that, described difference modes decoding comprises:

If data to be decoded be encoded to difference modes, according to global motion vector, obtain the corresponding data of predictive reference data in background image; If institute's predictive reference data has been encoded to raw mode, take the decoding difference value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to decoding current data to be decoded, the data that decode pass through again with background image in the superposition of corresponding data.

22. one kind generates the video decoding system of code stream with video coding system claimed in claim 10, it is characterized in that, comprising:

The module that is used for decoding background image and global motion vector;

Be used for each video block is carried out the module of raw mode or difference modes decoding.

23. video decoding system according to claim 22 is characterized in that, described raw mode decoding comprises:

If data to be decoded be encoded to raw mode, according to global motion vector, obtain the corresponding data of predictive reference data in background image; If predictive reference data is for being encoded to difference modes, take the decoding superposition value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to directly decoding, to obtain final decoded data.

24. video encoding/decoding method according to claim 22 is characterized in that, described difference modes decoding comprises:

If data to be decoded be encoded to difference modes, according to global motion vector, obtain the corresponding data of predictive reference data in background image; If institute's predictive reference data has been encoded to raw mode, take the decoding difference value of this predictive reference data and its corresponding data in background image as reference, otherwise, directly take the decoding original value of predictive reference data as with reference to decoding current data to be decoded, the data that decode again through with background image in the superposition of corresponding data obtain final decoded data.

Images