CN102917233A - Stereoscopic video coding optimization method in space teleoperation environment - Google Patents

Abstract

The invention puts forward a stereoscopic video coding optimization method in a space teleoperation environment. Based on the H.264 coding technology, the following stereoscopic video coding scheme is adopted: correlation of left and right channels and residual predicted based on right channel parallax are considered, the left channel adopts a 264 coding mode, the right channel adopts a prediction coding mode based on the parallax of the left channel, and the residual predicted based on the parallax of the right channel also adopts the H.264 coding mode. During the H.264 coding process, in combination of the actual need, partial optimization is carried out on the codes. The method provided by the invention has the advantages that in combination with the characteristics of the novel H.264 efficient coding compression technology, based on the requirements on compression ratio and instantaneity, the H.264 coding algorithm is optimized, thus reducing operand on the premise of ensuring the compression ratio. Test results show that the effect is more obvious after optimization, and the instantaneity of the system is improved.

Description

Stereo scopic video coding optimization method under the distant operating environment in space

Technical field

The present invention relates to a kind of coding method, specifically a kind of optimization method of stereo scopic video coding.

Background technology

Along with reaching its maturity of 3-D technology, developed rapidly by the application of the stereo video signals of many viewpoints camera system collection, video display technology must be shown by simple plane in the developing direction in future and changes to stereo display.Compare with common single channel video, three-dimensional video-frequency has increased the depth information of scenery, strengthens presence and the realism of vision.Stereo video signals will be the important content of Future Multimedia communication, to be widely used in the many aspects such as Digital Television, long-distance education, remote industrial control, 3 D video conference system and virtual reality system, but compare with common single channel video, the data volume of three-dimensional video-frequency transmission and storage doubles above at least; And under the distant operating environment in space, because communication limited bandwidth, for the stereoscopic video data are effectively stored and are transmitted, must take full advantage of the information redundancy between space, time and two passages, adopt motion compensated prediction technology and disparity compensation prediction technology to come the stereoscopic image video data effectively to compress.

Three-dimensional video-frequency is simultaneously a scene to be taken by two parallel optical axis video cameras at a distance of about the 65mm to obtain, thus image between very strong binocular vision correlation is arranged.

The three-dimensional video-frequency overall framework as shown in Figure 1, collecting part adopts multichannel data collecting technology, utilize two cameras, stereoscopic views to the scene is carried out real-time data acquisition, and synchronous transmission is to vedio data encoder, and encoder carries out Real Time Compression to the video data that gathers, and stores or transmit, video data decoder is according to the processing of decoding of the code stream that receives, and decoded image carries out stereo display after being divided into left and right sides two-way again.

Two passages about the video of this system's picked-up comprises, the image of two passages between very strong binocular correlation is arranged, H.264 be a kind of efficient algorithm for single-pass video encoding standard, but for stereo scopic video coding, be not that left and right sides passage is adopted respectively efficient algorithm for single-pass video encoding, just can reach good compression effectiveness.This is owing to be outside one's consideration except the spatial redundancies that will consider each passage I picture and the time redundancy between the inter frame image in the stereo scopic video coding, also will consider the spatial redundancies between the channel image of the left and right sides.

Summary of the invention

The technical problem to be solved in the present invention provide a kind of have the good compression effect the distant operating environment in space under the stereo scopic video coding optimization method.

The present invention solves the problems of the technologies described above by the following technical solutions: in the two-path video signal, with left road video as the reference image, use the method for video coding of classical based on motion compensation prediction to encode, remove spatial coherence and temporal correlation between the image, with the right wing video as target image, adopt disparity estimation/compensation technology removal of images between redundancy.The parallax here refer to stereo-picture between disparity.

Concrete, the present invention is based on H.264 coding techniques, stereo scopic video coding scheme below proposing: the residual error of considering the prediction of left and right sides passage correlation and right passage parallax, left passage adopts H.264 coding, right passage adopts DCP (disparity compensation prediction) coding based on left passage, and H.264 the residual error of right passage parallax prediction also adopts mode to encode, in carrying out H.264 cataloged procedure, in conjunction with actual needs, and this coding carried out partly optimizing.

The present invention further is specially: in the H.264 coding that left passage adopts, intraframe prediction algorithm and motion estimation algorithm are optimized;

Method to intraframe prediction algorithm optimization is: the Given information of macro block around utilizing, possible predictive mode to current coding macro block sorts according to probability, then calculate successively the SATD value of each pattern, if the SATD value is lower than the threshold value of setting, stop so calculating the SATD value of pattern of surplus, color difference components and luminance component all adopt this algorithm;

Method to motion estimation algorithm optimization is: use three step search methods to carry out estimation, the performing step of three step search methods is as follows:

1. centered by " 1 " point, step-length is 4, searches for it ± interior 9 points that are labeled as " 1 " of 4 scopes, according to the evaluation and test criterion function, finds the point with minimum MSE or lowest mean square absolute difference from these 9 pixels, as the optimum Match pixel of this step;

2. be centered around the pixel that step is selected as optimum Match in 1., selecting step-length is 2, search for it ± 2 scopes in selected 8 points that are labeled as " 2 ", with step 1. mutually roughly the same, find an optimum Match pixel according to criterion function;

3. selecting step-length is 1, search in 2. optimal match point of step ± 1 scope, and the optimum Match pixel that obtains will be as the final result of three steps search.

Optimize, in the motion search process of motion estimation algorithm, the rate distortion equation of use is as follows:

$J\left(m\right|{\mathrm{\λ}}_{\mathrm{motion}})=\mathrm{SAD}(s,c\left(m\right))+{\mathrm{\λ}}_{\mathrm{motion}}*R(m-p)$

When interframe and intra-frame encoding mode selection, the rate distortion equation below using:

$\begin{array}{c}J(s,c,\mathrm{pattern}|\mathrm{QP},{\mathrm{\λ}}_{\mathrm{mode}})=\mathrm{SSD}(s,c,\mathrm{pattern})\\ +{\mathrm{\λ}}_{\mathrm{mode}}*R(s,c,\mathrm{mode}|\mathrm{QP})\end{array}$

Wherein, J (m| λ _Motion) be in the criterion of selecting motion vector, so that should be selected J (s, c, pattern|QP, λ by the minimum motion vector of value _Mode) be used for the criterion of model selection, so that should be selected by the minimum coding mode of value, SAD is the absolute value sum of difference between source video data and the target video data, and SSD is the squared absolute value sum of difference between source video data and the target video data, λ _MotionAnd λ _ModeIt is LaGrange parameter.

Optimization method of the present invention can also be optimized the compressed encoding of right wing video is logical, perhaps on the basis that the compressed encoding to left road video is optimized simultaneously the compressed encoding to the right wing video be optimized, step to the compressed encoding of right wing video is: the right wing video is carried out disparity estimation, it is the macro block of 16x16 that coding units can be selected, at first the macro block in the right wing image of present encoding is searched for the image block that mates most in the image of the left road of time synchronized, obtain corresponding difference vector, difference vector is carried out undistorted coding, simultaneously, the match block that current encoding block and search obtain is subtracted each other, obtain the disparity estimation residual error, and then it is carried out transition coding, inspect the fast algorithm of estimating and comprise with next or three:

At first, according to the polarization binding feature of parallax, only carry out along polarized line during search, in parallel camera system, only need along continuous straight runs to search for; Secondly according to the directivity constraint, preferentially to a direction search, consider left channel image as basic layer coding during search, right channel image is preferentially searched for to the right during disparity estimation as enhancement layer coding; Utilize relativity of time domain or spatial correlation, when carrying out disparity estimation, use the corresponding difference vector of former frame image as initial search point.

The invention has the advantages that: in conjunction with up-to-date H.264 high efficient coding compress technique characteristics, according to compression ratio and requirement of real-time, H.264 encryption algorithm is optimized processing, reduced operand on the basis of assurance compression ratio, mainly intraframe prediction algorithm and motion estimation algorithm are optimized, result of the test shows, the successful after the optimization, and the real-time of system improves.

Description of drawings

Fig. 1 is three-dimensional video-frequency overall framework figure.

Fig. 2 is algorithm flow chart H.264.

Embodiment

1, the compressed encoding of left road video

At first the left road video in the stereoscopic video uses up-to-date international video standard H.264 to compress.

H.264 be the up-to-date video encoding standard of being united release by the ITU-T video experts VCEG of group and ISO/IEC Motion Picture Experts Group, compared with former coding standard, H.264 coding efficiency has been obtained unprecedented raising, some new algorithms have been adopted, such as Forecasting Methodology, integer transform, the motion estimation/motion compensation of 4x4 piece, loop filtering, new entropy coding method, but H.264 when performance improves, the complexity of encoder also increases considerably rate-distortion optimization technology RDO(rate distortion optimization) etc.,, be difficult to requirement of real-time in the distant operation of meeting spatial, according to the requirement of compression ratio and real-time, this paper has only used the H.264 basic class of standard (baseline profile) part, and coding H.264 is optimized processing.H.264 algorithm flow chart as shown in Figure 2.

1.1 the optimization process of algorithm

Algorithm optimization is to reduce operand on the basis that guarantees compression ratio, and in Fig. 2, we mainly optimize intraframe prediction algorithm and motion estimation algorithm.The main redundancy that has two kinds of forms in the video data, be the interior spatial redundancy of frame and the time redundancy of interframe, infra-frame prediction is mainly eliminated the spatial redundancy between the pixel in the frame, predict the present encoding piece with the reconstruction pixel value around the encoding block, owing to there is the Multi-encoding pattern optional, in the process of coding, need to carry out exhaustive search, computation complexity is larger.And motion estimation module, elimination be the time redundancy of interframe, in former frame, search for blocks and optimal matching blocks with the macro block in the current encoded frame, then coupling residual sum motion vector is encoded, the complexity of search procedure is equally also very high here.

1.1.1 fast frame intraprediction encoding

In infra-frame prediction, in order to determine the intra prediction mode of a macro block, need to calculate the SATD value that institute might pattern (Sum of Absolute Transform Difference, residual error conversion absolute value with), then selection has the pattern of minimum SATD as final predictive mode.In H.264, be based on the intra prediction mode of 16x16, predictive mode and the luminance component of color difference components are similar.In order to reduce the number of times that calculates the SATD value in the infra-frame prediction, the Given information of macro block around we utilize, possible predictive mode to current coding macro block sorts according to probability, then calculate successively the SATD value of each pattern, if the SATD value is lower than the threshold value that we arrange, stop so calculating the SATD value of pattern of surplus, color difference components and luminance component all adopt this algorithm, thereby effectively reduce computation complexity.

1.1.2 quick inter prediction encoding

For the redundancy of removal of images interframe, adopted the motion estimation algorithm based on the piece coupling in H.264, its basic thought is image to be divided into the sub-block of many non-overlapping copies, and thinks that the displacement of all pixels is all identical in the piece.Full-search algorithm is research and most widely used a kind of technology in the block matching algorithm, but because amount of calculation is quite large.In order to reduce amount of calculation, we use three step searching algorithms to carry out estimation in this project, and the realization schematic diagram of three step searching algorithms as shown in Figure 3.

The performing step of three step searching algorithms is as follows:

1. centered by " 1 " point, step-length is 4, searches for it ± interior 9 points that are labeled as " 1 " of 4 scopes, according to the evaluation and test criterion function, finds the point with minimum MSE or lowest mean square absolute difference from these 9 pixels, as the optimum Match pixel of this step.

2. be centered around the pixel that step is selected as optimum Match in 1., selecting step-length is 2, search for it ± 2 scopes in selected 8 points that are labeled as " 2 ", with step 1. mutually roughly the same, find an optimum Match pixel according to criterion function.

3. selecting step-length is 1, search in 2. optimal match point of step ± 1 scope, and the optimum Match pixel that obtains will be as the final result of three steps search.

1.1.3 rate-distortion optimization algorithm

The rate-distortion optimization algorithm refers under band-limited condition, how to obtain optimum picture quality, this programme uses a simple equation expression formula that code check and distortion are connected, thereby between code check and the distortion factor, obtain one balance point, thereby guarantee under given code check condition, to obtain optimum picture quality.

In the process of motion search, the rate distortion equation of use is as follows:

$J\left(m\right|{\mathrm{\λ}}_{\mathrm{motion}})=\mathrm{SAD}(s,c\left(m\right))+{\mathrm{\λ}}_{\mathrm{motion}}*R(m-p)$

When interframe and intra-frame encoding mode selection, the rate distortion equation below using:

$\begin{array}{c}J(s,c,\mathrm{pattern}|\mathrm{QP},{\mathrm{\λ}}_{\mathrm{mode}})=\mathrm{SSD}(s,c,\mathrm{pattern})\\ +{\mathrm{\λ}}_{\mathrm{mode}}*R(s,c,\mathrm{mode}|\mathrm{QP})\end{array}$

Wherein, J (m| λ _Motion) be for the criterion of selecting motion vector, so that should be selected by the minimum motion vector of value.J (s, c, pattern|QP, λ _Mode) be the criterion for model selection, so that should be selected by the minimum coding mode of value.

SAD(Sum of Absolute Difference, absolute difference and) be the yardstick of weighing the distortion factor, by the absolute value sum of difference between source video data and the target video data.SSD(Sum of Squared Difference, squared difference and) also be another yardstick of weighing distortion, by the squared absolute value sum of difference between source video data and the target video data.

λ _MotionBe LaGrange parameter, its size determines the impact that code check and distortion are selected for motion vector, λ _MotionLarge then code check is larger on the impact of model selection, λ _MotionLittle then model selection is more responsive to distortion.

λ _ModeBe LaGrange parameter, its size decision code check and distortion are on the impact size of model selection, λ _ModeLarge then code check is large on the impact of model selection, λ _ModeLittle then model selection is more responsive to distortion.

Use above-mentioned method can obtain optimal image quality under the nominated bandwidth condition.

The compressed encoding of 2 right wing videos

The right wing video is carried out disparity estimation, and coding units also is the macro block of 16x16.At first the macro block in the right wing image of present encoding is searched for the image block that mates most in the image of the left road of time synchronized, obtain corresponding difference vector, difference vector is carried out undistorted coding, simultaneously, the match block that current encoding block and search obtain is subtracted each other, obtain the disparity estimation residual error, and then it is carried out transition coding.

BMA is the parallax estimation method of commonly using, and its basic thought is the piece that image segmentation is become fixed dimension, and the parallax value supposition of piece interior pixels is identical, seeks best matching blocks according to certain matching criterior in reference picture.But the distribution of difference vector is subject to certain constraint, between the difference vector of front and back image block, all has correlation between the difference vector of consecutive frame image corresponding blocks in the two field picture, and we will carry out fast disparity estimation with these characteristics.

2.1 polarization constraint

For a given scene point, it always appears on the corresponding left and right sides polarized line in the picture point on the plane of delineation of the left and right sides, therefore only need to get final product along the polarized line search when carrying out the parallax search.In parallel camera system, the horizontal scanning line of polarized line and image is parallel, so search needs only the horizontal line along right image place.

2.2 directivity constraint

In parallel camera system, difference vector only has horizontal component, for same scenery, the left image of its perspective projection with respect to right image local move right, when the match block of the right image of search in left image, can preferentially search for to the right.

2.3 the spatial correlation of difference vector and temporal correlation

Parallax is the function of the degree of depth, and the pixel that the degree of depth is identical has identical parallax.The degree of depth is exactly that body surface is to the distance of camera photocentre, although actual scenery surface generally is discontinuous, but might regard continually varying as in regional area, because space scenery always is made of some objects, some body surface can be similar to be regarded as smooth or local smooth.For smooth surface, it be continuous that its parallax value changes, and the difference vector that changes continuously in the optical parallax field has very strong correlation, namely has correlation between the difference vector in the same frame.

For adjacent two two field pictures, only there is a few pixels that motion has occured, the position of most pixels does not change.For the pixel of invariant position, its parallax is substantially constant.So when carrying out disparity estimation, can carry out interior among a small circle search as initial search point with the corresponding difference vector of former frame image, thereby find fast actual difference vector.

2.4 the fast algorithm of disparity estimation

Based on above-mentioned constraints on disparity distribution and correlation analysis, fast Disparity Estimation is described below: at first, polarization binding feature according to parallax, only carry out along polarized line during search, in parallel camera system, polarized line is parallel with scan line, and the match point of left and right sides pixel is positioned on the same level line, therefore, only needing along continuous straight runs to search for gets final product.Secondly retrain according to directivity, preferentially search for to a direction during search. consider that left channel image is as basic layer coding, right channel image is as enhancement layer coding, disparity estimation is the match block of image block in left channel image in the right channel image of search, by the about beam analysis of directivity as can be known, can preferentially search for to the right during disparity estimation.In addition, utilize relativity of time domain or spatial correlation can determine the initial value of disparity estimation.For spatial correlation, because its correlation of zones of different is different, the flat site correlation is eager to excel, the non-flat forms zone is relatively weak, so when search, the hunting zone of flat site can be obtained smaller, but not the hunting zone of flat site can be larger.

In test, use the CIF stereo-picture of Miss to carrying out l-G simulation test, all take left image as reference picture, right image is target image, adopt respectively based on H.264 with quality and the compression ratio based on H.264 improving algorithm and rebuild right image of the present invention, and to upper above-mentioned two kinds of algorithms are compared.

Can draw to draw a conclusion from result of the test figure: in the identical situation of compression parameters, the image of rebuilding by the algorithm after improving is significantly better than former algorithm, and blocking effect is significantly improved.

Compare on our objective quality with Y-PSNR (PSNR) presentation video simultaneously.Comparative result is as shown in Table 1:

The contrast of table one experimental data

Can find out that from simulation result with respect to independent coding, the compression ratio of two kinds of algorithms all is improved largely, but picture quality descends to some extent all, improve algorithm with respect to former algorithm, improved the signal to noise ratio of target image.

The above is only for the preferred embodiment of the invention; not in order to limit the invention; all in the invention spirit and principle within do any modification, be equal to and replace and improvement etc., all should be included within the protection range of the invention.

Claims (10)

1. stereo scopic video coding optimization method under the distant operating environment in space, it is characterized in that: the method is based on H.264 coding techniques, stereo scopic video coding scheme below adopting: the residual error of considering the prediction of left and right sides passage correlation and right passage parallax, left passage adopts H.264 coding, right passage adopts the disparity compensation prediction coding based on left passage, and H.264 the residual error of right passage parallax prediction also adopts mode to encode.

2. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 1, it is characterized in that: in the H.264 coding that left passage adopts, intraframe prediction algorithm is optimized, method to intraframe prediction algorithm optimization is: the Given information of macro block around utilizing, possible predictive mode to current coding macro block sorts according to probability, then calculate successively the SATD value of each pattern, if the SATD value is lower than the threshold value of setting, stop so calculating the SATD value of pattern of surplus, color difference components and luminance component all adopt this algorithm.

3. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 2, it is characterized in that: in the H.264 coding that left passage adopts, motion estimation algorithm is optimized, method to motion estimation algorithm optimization is: use three step search methods to carry out estimation, the performing step of three step search methods is as follows:

1. centered by " 1 " point, step-length is 4, searches for it ± interior 9 points that are labeled as " 1 " of 4 scopes, according to the evaluation and test criterion function, finds the point with minimum MSE or lowest mean square absolute difference from these 9 pixels, as the optimum Match pixel of this step;

2. be centered around the pixel that step is selected as optimum Match in 1., selecting step-length is 2, search for it ± 2 scopes in selected 8 points that are labeled as " 2 ", with step 1. mutually roughly the same, find an optimum Match pixel according to criterion function;

3. selecting step-length is 1, search in 2. optimal match point of step ± 1 scope, and the optimum Match pixel that obtains will be as the final result of three steps search.

4. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 3, it is characterized in that: in the motion search process of motion estimation algorithm, the rate distortion equation of use is as follows:

$J\left(m\right|{\mathrm{\λ}}_{\mathrm{motion}})=\mathrm{SAD}(s,c\left(m\right))+{\mathrm{\λ}}_{\mathrm{motion}}*R(m-p)$

Wherein, J (m| λ _Motion) be in the criterion of selecting motion vector, minimum motion vector is selected, and SAD is the absolute value sum of difference between source video data and the target video data, λ _MotionIt is LaGrange parameter.

5. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 4 is characterized in that: when interframe and intra-frame encoding mode are selected, and the rate distortion equation below using:

$\begin{array}{c}J(s,c,\mathrm{pattern}|\mathrm{QP},{\mathrm{\λ}}_{\mathrm{mode}})=\mathrm{SSD}(s,c,\mathrm{pattern})\\ +{\mathrm{\λ}}_{\mathrm{mode}}*R(s,c,\mathrm{mode}|\mathrm{QP})\end{array}$

Wherein, J (s, c, pattern|QP, λ _Mode) for the criterion of model selection, minimum coding mode is selected, SSD is the squared absolute value sum of difference between source video data and the target video data, λ _ModeIt is LaGrange parameter.

6. such as stereo scopic video coding optimization method under each distant operating environment in described space of claim 1 to 5, it is characterized in that: the step to the compressed encoding of right wing video is: the right wing video is carried out disparity estimation, at first the macro block in the right wing image of present encoding is searched for the image block that mates most in the image of the left road of time synchronized, obtain corresponding difference vector, difference vector is carried out undistorted coding, simultaneously, the match block that current encoding block and search obtain is subtracted each other, obtain the disparity estimation residual error, and then it is carried out transition coding.

7. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 6 is characterized in that: the described fast algorithm that the right wing video is carried out disparity estimation comprises: according to the polarization binding feature of parallax, only carry out along polarized line during search.

8. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 7, it is characterized in that: the described fast algorithm that the right wing video is carried out disparity estimation also comprises: retrain according to directivity, preferentially search for to a direction during search, consider that left channel image is as basic layer coding, right channel image is preferentially searched for to the right during disparity estimation as enhancement layer coding.

9. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 8, it is characterized in that: the described fast algorithm that the right wing video is carried out disparity estimation also comprises: utilize relativity of time domain or spatial correlation, when carrying out disparity estimation, use the corresponding difference vector of former frame image as initial search point.

10. stereo scopic video coding optimization method under the distant operating environment in space as claimed in claim 6, it is characterized in that: the coding units that the right wing video is carried out disparity estimation is the macro block of 16x16.

Images