CN108886598B - Compression method and device for panoramic stereoscopic video system - Google Patents

Abstract

Provided is a method for compressing a stereoscopic video containing a left-view frame and a right-view frame, the method comprising: determining, through intra-frame prediction, a texture saliency value of a first sub-block in the left-view frame (1101); Through motion estimation, determine the motion saliency value of the first sub-block (1102); determine the disparity saliency value between the first sub-block and the corresponding second sub-block in the right-view frame (1103) ; determining a quantization parameter according to the disparity saliency value, the texture saliency value and the motion saliency value (1104); and according to the quantization parameter, quantizing the first partition (1105).

Description

Compression method and device for panoramic stereoscopic video system

CROSS-REFERENCE TO RELATED APPLICATIONS

This application requires an international patent application with an application number of PCT/CN2016/070712, titled "Calibration method and device for a panoramic stereoscopic video system", an application date of January 12, 2016, and an application number of PCT/CN2016/070823, The title is "Splicing Method and Device for Panoramic Stereoscopic Video System", the application date is the rights and priority of the international patent application on January 13, 2016. The entire disclosures of both applications are incorporated herein by reference.

technical field

The present invention relates to a panoramic stereoscopic video system for shooting, processing, compressing and displaying 3D panoramic stereoscopic video, in particular to a method and device for compressing 3D panoramic stereoscopic video in the panoramic stereoscopic video system.

Background technique

The panoramic stereoscopic video system proposed in the above-mentioned documents realizes an immersive 3D experience by displaying the stereoscopic panoramic video on a head-mounted display (HMD). The resolution and persistence of stereoscopic video are the two main characteristics that determine the user experience. The system stitches images from 16 high-resolution (HD) cameras to each other to generate stereoscopic video with a resolution of at least 3840 x 2160 (4K) per field of view. Since the system has a frame rate of 50fps, it greatly reduces motion blur and flickering effects. However, on the other hand, its ultra-high resolution and high refresh rate result in a huge amount of video data, which poses a problem for 3D video services and broadcasting.

The video coding efficiency of existing hybrid video coding methods such as H.264, VC-1 and HEVC has improved significantly over the past decade, and by implementing dense spatiotemporal prediction, the temporal and spatial redundancy in video sequences has been greatly reduced redundancy. Recent advances in 3D technologies such as MV-HEVC and 3D-HEVC have further investigated disparity prediction between different fields of view. However, in order to achieve better compression performance of stereoscopic panoramic video, it is also necessary to improve the subjective video quality by taking into account the characteristics of human vision and the characteristics of panorama.

In general, a 360-degree panoramic image contains an elongated field of view, and most of the field of view is likely to be background only. However, the user may only focus on a small portion of the field of view where color, texture, motion, or depth contrast is significant.

The basic principle of compression methods based on human visual features is to encode only a small number of selected areas of interest with high priority to obtain high subjective video quality, while processing less attention areas with low priority to obtain high subjective video quality. Save bits. To achieve this, attention prediction methods are usually used to predict the areas that users may pay attention to.

Existing 2D image saliency calculations mainly consider the contrast of features such as color, shape, orientation, texture, curvature, etc. In image sequences or videos, region-of-interest detection focuses on motion information that distinguishes foreground from background. However, since existing video compression methods do not consider the stereoscopic contrast in stereoscopic videos, they are not suitable for stereoscopic videos. Furthermore, when the salient objects are not visually distinctive at the spatial level and do not move at the temporal level, this existing method is difficult to detect their regions of interest.

Therefore, there is a need to provide a new stereoscopic video compression method that simultaneously uses texture, motion and stereoscopic contrast for saliency analysis.

SUMMARY OF THE INVENTION

In order to solve the problems in the prior art, embodiments of the present invention provide a new stereoscopic video compression method that simultaneously uses texture, motion and stereoscopic contrast for saliency analysis. Specifically, by employing block-based stereo vision detection, we further provide depth cues that play an important role in human vision.

According to an embodiment of the present invention, a method for compressing a stereoscopic video containing left-view frames and right-view frames is provided, the method comprising: determining, by intra-frame prediction, a Texture saliency value; determine the motion saliency value of the first sub-block through motion estimation; determine the disparity saliency value between the first sub-block and the corresponding second sub-block in the right-view frame; and determining a quantization parameter according to the disparity saliency value, the texture saliency value, and the motion saliency value.

Preferably, the method further comprises: quantizing the first sub-block according to the quantization parameter.

Preferably, the method further comprises: determining a hybrid stereo saliency map of the left-view frame; reducing the size of the hybrid stereo saliency map to match the size of a transform unit (TU); binary quantization parameter; and quantizing the transform unit according to the second quantization parameter.

Advantageously, the method further comprises: determining the texture saliency value from a DC mode intra prediction output of High Efficiency Video Coding (HEVC).

Advantageously, the method further comprises: determining a motion saliency value of the first partition according to a motion estimation output of High Efficiency Video Coding (HEVC).

Preferably, the method further comprises: determining the hybrid stereo saliency value of the first segment by superimposing the parallax saliency value, the texture saliency value and the motion saliency value with a weighting parameter.

Preferably, the left-view frame and the right-view frame are modified in a first direction, and the method further comprises: searching for the parallax saliency value in a second direction perpendicular to the first direction.

Preferably, the disparity saliency value includes a non-integer value.

Advantageously, the method further comprises determining the disparity saliency value from 1/4 pixel samples generated by high efficiency video coding (HEVC) sub-pixel motion estimation.

According to another embodiment of the present invention, there is provided a non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions comprising a method for compressing a stereoscopic video including left-view frames and right-view frames The method includes: determining a texture saliency value of a first sub-block in the left-view frame through intra-frame prediction; determining a motion saliency value of the first sub-block through motion estimation; determining the first sub-block a disparity saliency value between a partition and a corresponding second partition within the right-view frame; and determining a quantization parameter based on the disparity saliency value, texture saliency value, and motion saliency value.

Preferably, the method further comprises: quantizing the first sub-block according to the quantization parameter.

Preferably, the method further comprises: determining a hybrid stereo saliency map of the left-view frame; reducing the size of the hybrid stereo saliency map to match the size of a transform unit (TU); binary quantization parameter; and quantizing the transform unit according to the second quantization parameter.

Advantageously, the method further comprises determining the texture saliency value from an output of DC mode intra prediction of High Efficiency Video Coding (HEVC).

Advantageously, the method further comprises: determining a motion saliency value of the first partition according to an output of high efficiency video coding (HEVC) motion estimation.

Preferably, the method further comprises: determining the hybrid stereo saliency value of the first segment by superimposing the parallax saliency value, the texture saliency value and the motion saliency value with a weighting parameter.

Preferably, the left-view frame and the right-view frame are modified in a first direction, and the method further comprises: searching for the parallax saliency value in a second direction perpendicular to the first direction.

Preferably, the disparity saliency value includes a non-integer value.

Advantageously, the method further comprises determining the disparity saliency value from 1/4 pixel samples generated by high efficiency video coding (HEVC) sub-pixel motion estimation.

According to an embodiment of the present invention, a video coding scheme based on a region of interest is adopted, and the scheme adopts a bit allocation method based on visual attention. Specifically, spatial, temporal and stereo cues are taken into account in video attention prediction. The spatial and temporal contrast features are directly extracted from the existing video coding process without introducing additional computation. In addition, sub-pixel disparity intensity estimation is also employed to improve the visual saliency accuracy of stereo systems. In this way, stereoscopic video can be compressed efficiently without affecting the perceived quality of the end user.

Description of drawings

In order to better illustrate the technical features of the embodiments of the present invention, various embodiments of the present invention are briefly described below with reference to the accompanying drawings.

FIG. 1 is a schematic diagram illustrating a panoramic stereoscopic video system according to an embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating a camera array of a panoramic stereoscopic video system according to an embodiment of the present invention.

FIG. 3 is a schematic diagram illustrating an example of a data processing unit of a panoramic stereoscopic video system according to an embodiment of the present invention.

FIG. 4 is an exemplary flowchart of a panoramic stereoscopic video stitching method according to an embodiment of the present invention.

FIG. 5 is an exemplary flowchart of a panoramic stereoscopic video display method according to an embodiment of the present invention.

FIG. 6 is a schematic diagram illustrating an HEVC spatial prediction mode according to an embodiment of the present invention.

FIG. 7 is a schematic diagram illustrating a block-based motion estimation using motion vector prediction according to an embodiment of the present invention.

FIG. 8 is a schematic diagram illustrating a motion intensity map obtained by motion estimation according to an embodiment of the present invention.

FIG. 9 is a schematic diagram illustrating a block-based disparity estimation employed in stereoscopic video coding according to an embodiment of the present invention.

FIG. 10 is a schematic diagram illustrating a stereoscopic video compression system based on a mixed region of interest according to an embodiment of the present invention.

FIG. 11 is an exemplary flowchart of a method for compressing a stereoscopic video based on a mixed region of interest according to an embodiment of the present invention.

Detailed ways

In order to better illustrate the objectives, technical features and advantages of the embodiments of the present invention, various embodiments of the present invention will be further described below with reference to the accompanying drawings. It is readily understood that the accompanying drawings are only used to illustrate exemplary embodiments of the present invention, and other drawings may be obtained by those skilled in the art without departing from the principles of the present invention.

According to an embodiment of the present invention, a panoramic stereoscopic video system with functions of multi-camera video shooting, data processing, stereoscopic video encoding, transmission and 3D display is provided. The panoramic stereoscopic video system adopts real-time multi-field video shooting, image correction and preprocessing, and region-of-interest (ROI)-based stereoscopic video compression. After the transmission and decoding process, the left and right fields of view are displayed using a head mounted display (HMD) headset.

1. System overview

FIG. 1 is a schematic diagram illustrating a panoramic stereoscopic video system according to an embodiment of the present invention. The panoramic stereoscopic video system uses a camera array to shoot 3D panoramic video, and displays the captured 3D panoramic video on a 3D TV or a head-mounted virtual reality display device. As shown in FIG. 1 , the panoramic stereoscopic video system includes a data acquisition unit 200 , a data processing unit 300 and a data display unit 400 . The data acquisition unit 200 includes a plurality of cameras in the camera array 210 and a camera calibration unit 220 . The data processing unit 300 includes a data preprocessing unit 310 and an advanced stereoscopic video transcoding unit 320 . The data display unit 400 includes a decoding unit 410 and a display earphone 420 .

2. Data acquisition unit

As shown in FIG. 1 , the data acquisition unit 200 includes a plurality of cameras in the camera array 210 , and a camera calibration unit 220 for calibrating the camera array 210 .

2.1 Camera array

2 is a schematic diagram illustrating a camera array in a panoramic stereoscopic video system according to an embodiment of the present invention.

As shown in FIG. 2, the camera array 210 has 16 high-definition cameras c1-c16 mounted on a regular octagon mounting frame, and a pair of cameras is mounted on each side of the octagon. The two cameras on each side, such as c1 and c2, have parallel optical axes and are spaced apart from each other by a distance d. The raw video data captured by the camera array 210 is sent over a cable to a computer for further processing. The parameters of the camera are listed in Table 1 below.

Table 1

It should be noted that although the camera array is shown as a regular octagon in FIG. 2 , in other embodiments of the present invention, the camera array may also be set in other shapes. Specifically, in an embodiment of the present invention, each camera is mounted on a rigid frame, so that the relative positions among the plurality of cameras are substantially constant. In another embodiment of the present invention, the cameras are basically arranged on the same plane, for example, arranged on each side of a polygon.

2.2 Camera Calibration

In order to stitch together the images captured by each camera and generate a 3D effect, both the internal and external parameters of these cameras need to be obtained. The external parameters include rotation and translation between the cameras, so that the images captured by the different cameras can be corrected and aligned in the horizontal direction. In addition, the images captured by each camera may be distorted. In order to obtain a distortion-free image, it is necessary to know the distortion parameters of each camera. These parameters are available during the calibration of the camera.

2.2.1 Internal and Distortion Parameter Calibration

The internal and distortion parameters of each camera can be obtained by various methods, such as the calibration method proposed by Zhengyou Zhang. Furthermore, tools such as MatLab can be used to obtain such parameters.

2.2.2 External parameter calibration

After obtaining the internal parameters of each camera, the rotation and translation between the cameras are obtained by a method based on the motion recovery structure. This method has the following advantages:

Efficiency: No need to calibrate cameras pair by pair. Instead, all cameras capture a scene at the same time during the calibration process, and the extrinsic parameters of all cameras are obtained at the same time.

Accuracy: In the pattern-based calibration method, the pattern needs to be photographed by two adjacent cameras, which often reduces the resolution and calibration accuracy of the pattern. In the method based on the motion recovery structure of the present invention, the motion of each camera is independently estimated to obtain the above parameters, and adjacent cameras do not need to have overlapping fields of view. As a result, each camera can be placed closer to the scene to be captured, allowing for greater accuracy.

Scalability: Since the adjacent cameras of the method of the present invention do not need to overlap fields of view, it can even be applied to cameras placed in a back-to-back posture.

2.3 Data collection method

The data from the 16 cameras are saved by software after being collected and then provided to the data processing unit. The image data of each frame captured by each camera can be collected by software such as FFmpeg and DirectShow (or DShow). Each frame captured by each camera is compressed and saved as a video file. Since there are a plurality of cameras, it is necessary to synchronize the frames captured by the cameras by using a time stamp or the like. For example, each frame captured by each camera can be time stamped and placed in a queue so that it is synchronized with every other frame with the same time stamp. The synchronized frames are encoded as video streams and stored locally, or transmitted simultaneously over the network.

3. Data processing unit

As shown in FIG. 1 , the data processing unit 300 includes a data preprocessing unit 310 and an advanced stereoscopic video transcoding unit 320 .

FIG. 3 is a schematic diagram illustrating an example of a data processing unit in a panoramic stereoscopic video system according to an embodiment of the present invention. As shown in FIG. 3 , the data preprocessing unit 310 includes: a time axis synchronization 311 for synchronizing images captured by each camera; several decoders 312 for decoding the original video stream; several modifiers for modifying the original video 313; an encoder for implementing video processing including noise reduction and editing 314; a stitching unit for stitching each video to generate a panoramic video. The data preprocessing unit 310 outputs the left-eye video and the right-eye video to the advanced stereoscopic video transcoding unit 320 . The advanced stereoscopic video transcoding unit 320 generates a motion map 321 and a texture map 322 of the video. A hybrid region of interest (ROI) generation unit 323 identifies a region of interest in the video based on the motion map 321 and the texture map 322 . The bit allocation unit 324 allocates bits according to the identified region of interest, and the HEVC encoding unit 325 encodes the video. The H.265 packer 326 packs the encoded video for transmission.

FIG. 4 is an exemplary flowchart of a panoramic stereoscopic video stitching method according to an embodiment of the present invention.

3.1 Distortion Correction and Preprocessing

According to the distortion parameters obtained during the calibration process, each frame captured by each camera is curled to obtain a distortion-free frame. To improve image alignment and stitching accuracy, each frame is first filtered to reduce noise.

3.2 Image Alignment

Image alignment is performed on each pair of cameras disposed on each side of the octagon, and the images captured by each pair of cameras are aligned in the horizontal direction. According to an embodiment of the present invention, each frame captured by each pair of cameras is curled to a plane parallel to the optical axis of the pair of cameras.

4. Panoramic video stitching

The camera array has 8 pairs of cameras. After each frame from all the left cameras is projected onto the cylinder, it is stitched into a panoramic image. A panoramic video is obtained by repeating the above steps on all frames captured by each left camera. Another panoramic video can be obtained by processing the frames captured by each right camera in the same way. The two panoramic videos form a panoramic stereoscopic video.

5. Data display unit

As shown in FIG. 1 , the data display unit 400 includes a decoding unit 410 and a display earphone 420 . After passing through the codec system, the panoramic stereoscopic video is played on a display headset 420, which may be a wearable virtual reality (VR) device such as that provided by Oculus VR. The panoramic stereoscopic video is rendered on the left-eye display and the right-eye display of the Oculus device, respectively. The display area of the panoramic stereoscopic video can be adjusted according to the movement of the detection device, so as to simulate the change of the viewing angle in virtual reality.

FIG. 5 is an exemplary flowchart of a panoramic stereoscopic video display method according to an embodiment of the present invention. As shown in FIG. 5, in step 501, the encoded video stream is first decoded into YUV. In step 502, position calculation and field of view selection are performed according to the Oculus sensor data. In step 503, the left-eye and right-eye images are rendered separately. In step 504, the rendered image is displayed on the Oculus display headset.

6. Stereoscopic video compression

In the stereoscopic panoramic video system, the left and right super-resolution videos are spliced by the video processing module. However, the huge amount of video data becomes a difficult problem in video compression and transmission. According to an embodiment of the present invention, a video coding scheme based on a region of interest is provided, and the scheme adopts a bit allocation method based on visual attention. Specifically, spatial, temporal and stereo cues are taken into account in video attention prediction. Spatial and temporal contrast features are directly extracted from the video encoding process without introducing additional computations. In addition, sub-pixel disparity intensity estimation is also employed to improve the visual saliency accuracy of stereo systems. The repeated use of sub-pixel samples and block-based matching ensure that the algorithm of the present invention can perform real-time detection with good performance. Overall, this scheme greatly improves the video compression rate without affecting the end user's perceived quality.

6.1 Region of Interest Detection

6.1.1 Extracting Spatial Features by Intra Prediction

In the HEVC coding standard, intra-frame prediction (or spatial prediction) is used to encode blocks that need to be compressed independently of previously coded frames, and the adjacent blocks of previously coded and reconstructed blocks are coded. Samples obtain pixel-level spatial correlation. After this, the predicted samples are subtracted from the original pixel values to obtain a residual block. This residual, obtained from intra prediction, contains texture contrast information and is used to generate a spatial saliency map.

In HEVC video coding, spatial prediction includes 33 directional prediction modes for prediction unit (PU) selection (there are only 8 such modes in H.264), DC prediction mode (overall average) and planar (surface fitting) prediction model. FIG. 6 is a schematic diagram illustrating an HEVC spatial prediction mode according to an embodiment of the present invention. All 35 prediction modes are shown in FIG. 6 . The size of the HEVC prediction unit is selected from 64×64 to 8×8, and all 35 modes can achieve the best block segmentation and the best residual. To reduce complexity, in one embodiment, the block-based residual mapping reuses the result of implementing DC mode prediction on fixed 8x8 blocks. The residuals for block k are calculated as follows:

Among them, C _ij and R _ij are the (i, j)th elements of the current original block C and reconstructed block R. After that, the texture saliency value S _T can be calculated from the residual of each block and normalized to the range [0, 1]:

Among them, N is the number of blocks in the frame. Since the HEVC intra-frame prediction method is used for the spatial residual detection of the 8×8 block, there is no need to introduce additional computation.

In other embodiments: each frame may be divided into non-overlapping blocks of different sizes of 64x64 or 16x16 pixels, etc.; and may be based on the results of other video coding methods similar or comparable to intra prediction methods Calculates texture saliency maps; and can be compressed according to other encoding standards such as H.264/AVC or AVS. Preferably, the intra prediction or other video processing is based on partitions of the same size from the frame.

6.1.2 Extracting Temporal Features by Motion Estimation

A fast-moving object can attract visual attention. However, since the video sequence is captured by a camera that is in motion, there is global motion in it. Therefore, there is a need to measure local motion saliency by estimating motion vector difference (MVD) in HEVC inter-predictive motion estimation.

Motion estimation techniques in most video coding standards are generally based on block matching, where a motion vector is represented by a 2D translation pattern, and each block is matched to all candidate positions within a predetermined search area. Since the motion vectors of adjacent blocks are usually highly correlated, in the motion vector prediction technique adopted by HEVC, the motion vector of the current block is predicted based on the motion vectors of nearby coded blocks.

7 is a schematic diagram illustrating a block-based motion estimation employing motion vector (MV) prediction according to an embodiment of the present invention. As shown in FIG. 7, for the current block 711 in the current frame 710, the vector mv _pred 712 is predicted according to the motion vector of the adjacent block, and the corresponding block 721 is matched with all the candidate positions in the predetermined search area 725 . Finally, the vector difference between the best vector mv _best 723 and the predicted vector mv _pred 721 is encoded and sent.

In one embodiment, motion vector differences generated by 8x8 block motion estimation are used. Among them, the magnitude of the motion vector difference can be defined as:

MVD _k =||mv _best (k)-mv _pred (k)|| (3)

The motion saliency map S _M can then be computed by normalizing the motion vector differences within the same frame:

The motion saliency map can be calculated from the results of motion estimation, which is the main process of HEVC video coding. Therefore, this method can extract motion features without introducing any additional processing. FIG. 8 is a schematic diagram illustrating a motion intensity map obtained by motion estimation according to an embodiment of the present invention.

In other embodiments, each frame may be divided into non-overlapping blocks of different sizes of 64x64 or 16x16 pixels, etc.; and may be based on other video coding methods similar or comparable to inter-predictive motion estimation. The result calculates a motion saliency map; and can be compressed according to other coding standards such as H.264/AVC or AVS. Preferably, the motion estimation or other video processing is based on equally sized blocks divided from the frame.

6.1.3 Disparity estimation via disparity prediction

We also employ stereo vision in our saliency analysis to further provide depth cues, and this stereo vision plays an important role in stereo panoramic videos. Among them, a block-based disparity estimation method is introduced in the disparity mapping process.

FIG. 9 is a schematic diagram illustrating a block-based disparity estimation used in stereoscopic video coding according to an embodiment of the present invention. As shown in FIG. 9 , in the high-resolution video system, both the left field of view 910 and the right field of view 920 of the stereoscopic image are well corrected. Each field of view is divided into non-overlapping blocks of size 8x8 pixels, and all pixels within a block are assumed to have the same disparity. As such, it is expected that the left field block that matches the right field block can be found on the same scanline, and the disparity 922 becomes a one-dimensional vector (with the vertical component equal to zero). This disparity matching scheme is similar to motion estimation in inter prediction. Specifically, the search area 925 is limited to a range within ±32 in the horizontal direction. The initial search position is set to the position of the corresponding segment 921 of the right field of view 920, and the Sum of Absolute Differences (SAD) is used as the matching criterion.

To achieve better prediction accuracy, we take into account the disparity of non-integer values, and use HEVC 7/8-tap filters to interpolate sub-pixel intensities. Since sub-pixel sample interpolation is one of the most complex operations, the sub-pixel disparity search of the present invention directly uses the 1/4 pixel samples generated by HEVC sub-pixel motion estimation. Computational complexity is greatly reduced by reusing HEVC's 7-tap interpolation. And, based on the block disparity value d _k , a block-wise disparity map is generated:

In other embodiments, each frame may be divided into non-overlapping blocks of different sizes of 64x64 or 16x16 pixels, etc.; disparity may be calculated from the results of other video coding methods similar or comparable to motion estimation Mapping; and can be compressed according to other coding standards such as H.264/AVC or AVS. Preferably, the motion estimation process is based on partitions of the same size from the frame.

6.1.4 Determination of mixed areas of interest

In one embodiment, by combining spatiotemporal features and disparity features, that is, the texture contrast S _T in Equation (2), the motion contrast S _M in Equation (4) and the disparity intensity in Equation (5), way to detect the region of interest. While each feature has its own advantages and disadvantages, the best results are achieved by combining all of them. First, each feature map is normalized to the range [0, 1]. Second, a hybrid stereo saliency map S is formed by superimposing S _M , S _T and S _D :

S(b _i )=λ _T S _T +λ _M S _M +λ _D S _D (6)

Among them, λ _T , λ _M and λ _D are weighting parameters.

FIG. 10 is a schematic diagram illustrating a stereoscopic video compression system based on a mixed region of interest according to an embodiment of the present invention. As shown in FIG. 10 , the stereoscopic video compression system has a spatial prediction module 1001 , a temporal prediction module 1102 and a disparity prediction module 1103 . The results generated by the spatial prediction module 1001, the temporal prediction module 1102, and the disparity prediction module 1103 are input to the mixed region of interest generation module 1004, which performs corresponding bit allocation after identifying the saliency region. Transform and quantization module 1105 performs quantization according to the bit allocation determined by hybrid region of interest generation module 1004 , while entropy encoding module 1106 is used to encode each frame to generate compressed frame 1006 .

6.2 Stereoscopic Video Coding Based on Region of Interest

One of the concepts of region-of-interest based compression is to allocate bits in a way that favors saliency regions. The hybrid region-of-interest detection method can generate high-quality and high-accuracy saliency maps. In addition, in order to improve the video compression performance, HEVC, an advanced video standard with high compression efficiency, is also selected.

Since the ROI detection of the present invention is based on 8×8 blocks, the size of the estimated saliency map needs to be reduced to match the size of the current transform unit, and the size of the existing transform unit can be selected as 32× 32, 16×16 and 8×8. The new QP value can be calculated as:

Q'=max(Q-ψ·(S-ES), 0) (8)

为由x265编码器所选择的原始QP值，in,

is the original QP value chosen by the x265 encoder,

为针对当前帧的尺寸缩减的显著性映射。

is the saliency map for the size reduction of the current frame.

In this way, the QP value of the coding unit containing the saliency region is decreased, and the QP value of the coding unit without the saliency region is increased. The parameter ψ is user-selectable and controls the bitrate distribution between salient and non-salient regions: the higher the value of ψ, the more bits in the saliency region.

FIG. 11 is an exemplary flowchart of a method for compressing a stereoscopic video based on a mixed region of interest according to an embodiment of the present invention. As shown in Figure 11, the compression method includes the following steps:

Step 1101: Determine the texture saliency value of the first sub-block in the left-view frame through intra-frame prediction. Preferably, the texture saliency value is determined from the output of high efficiency video coding (HEVC) DC mode intra prediction.

Step 1102: Determine the motion saliency value of the first sub-block through motion estimation. Preferably, the motion saliency value is determined from the output of high efficiency video coding (HEVC) motion estimation. Step 1103: Determine the disparity between the first sub-block and the corresponding second sub-block in the right-view frame. Preferably, the left-view frame and the right-view frame are first corrected in a first direction, and then a parallax search is performed in a second direction perpendicular to the first direction.

Step 1104: Determine a quantization parameter according to the disparity, texture saliency value and motion saliency value. Preferably, the hybrid stereo saliency value is determined by superimposing the disparity, texture saliency value and motion saliency value with a weighting parameter.

Step 1105: Quantize the first sub-block according to the quantization parameter. Wherein, if the size of the block is different from the size of the current transform unit, the size of the hybrid stereo saliency map is reduced to match the size of the current transform unit, and a new quantization parameter is calculated.

According to an embodiment of the present invention, a video coding scheme based on a region of interest is adopted, and the scheme adopts a bit allocation method based on visual attention. Specifically, spatial, temporal and stereo cues are taken into account in video attention prediction. The spatial and temporal contrast features are directly extracted from the video encoding process without introducing extra computation. In addition, sub-pixel disparity intensity estimation is also employed to improve visual saliency accuracy. In this way, stereoscopic video can be compressed efficiently without affecting the perceived quality of the end user.

The various modules, units, and components described above may be implemented as: application specific integrated circuits (ASICs); electronic circuits; combinational logic circuits; field programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) executing code; or Other suitable hardware components that provide the above functionality. The processor may be a microprocessor of Intel Corporation, or a mainframe computer of IBM Corporation.

It should be noted that one or more of the above functions may be implemented by software or firmware stored in memory and executed by a processor, or stored in program memory and executed by a processor. Furthermore, the software or firmware may be stored and/or transmitted within any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system , a processor-containing system, or other system that can retrieve and execute instructions from the instruction execution system, apparatus, or device. In this context, a "computer-readable medium" can be any medium that can contain or store a program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or devices, portable computer disks (magnetic), random access memory (RAM) (magnetic), read only memory (ROM) (magnetic), Erasable Programmable Read-Only Memory (EPROM) (magnetic), a portable optical disc such as CD, CD-R, CD-RW, DVD, DVD-R or DVD-RW, or a compact flash card, Flash memory such as secure digital cards, USB storage devices, memory sticks, etc.

The various embodiments of the present invention described above are only preferred embodiments and are not intended to limit the scope of the present invention, and the scope of the present invention covers any modifications, equivalents and improvements without departing from the spirit and principles of the present invention.

Claims (18)

1. A method for compressing a stereoscopic video containing a left-view frame and a right-view frame, wherein the method comprises: determining the texture saliency value of the first sub-block in the left-view frame by intra-frame prediction; Determine the motion saliency value of the first sub-block through motion estimation of inter-frame prediction; determining a disparity saliency value between the first partition and a corresponding second partition within the right-view frame; determining a quantization parameter based on the disparity saliency value, the texture saliency value, and the motion saliency value; and quantizing the first sub-block according to the quantization parameter; Wherein, the mixed stereo saliency value of the first block is determined by weighting the parallax saliency value, the texture saliency value and the motion saliency value, and the following formula is used to determine the Quantization parameter Q': Q'=max(Q-ψ·(S-ES), 0) Among them, Q is the original quantization parameter value, S is the mixed stereo saliency value, and ψ is the parameter that controls the bit rate distribution between the saliency area and the non-saliency area. 2. The method of claim 1, further comprising: The texture saliency value is determined from an output of DC mode intra prediction of high efficiency video coding. 3. The method of claim 1, further comprising: A motion saliency value for the first partition is determined from the output of the motion estimation of the high efficiency video coding. 4. The method of claim 1, wherein the left-view frame is partitioned into a plurality of non-overlapping partitions, and the motion estimation is based on the same size as the first partition. 5. The method of claim 4, further comprising: The first sub-block is quantized according to the quantization parameter. 6. The method of claim 4, further comprising: determining a hybrid stereo saliency map for the left-view frame; reducing the size of the hybrid stereo saliency map to match the size of the transform unit; determining a second quantization parameter for the transform unit; and The transform unit is quantized according to the second quantization parameter. 7. The method of claim 1, wherein the left-view frame and the right-view frame are modified in a first direction, and the method further comprises: The disparity saliency value is searched for in a second direction perpendicular to the first direction. 8. The method of claim 7, wherein the disparity saliency value comprises a non-integer value. 9. The method of claim 8, further comprising: The disparity saliency value is determined from 1/4 pixel samples generated by sub-pixel motion estimation for high efficiency video coding. 10. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a processor, can perform the following stereoscopic analysis of a left-view frame and a right-view frame: A method for compressing video, characterized in that the method includes: determining the texture saliency value of the first sub-block in the left-view frame by intra-frame prediction; Determine the motion saliency value of the first sub-block by motion estimation; determining a disparity saliency value between the first partition and a corresponding second partition within the right-view frame; determining a quantization parameter based on the disparity saliency value, the texture saliency value, and the motion saliency value; and quantizing the first sub-block according to the quantization parameter; Wherein, the mixed stereo saliency value of the first sub-block is determined by weighted summation of the parallax saliency value, the texture saliency value and the motion saliency value, and the following formula is used to determine the Quantization parameter Q': Q'=max(Q-ψ·(S-ES), 0) Among them, Q is the original quantization parameter value, S is the mixed stereo saliency value, and ψ is the parameter that controls the bit rate distribution between the saliency area and the non-saliency area. 11. The computer-readable medium of claim 10, wherein the method further comprises: The texture saliency value is determined from an output of DC mode intra prediction of high efficiency video coding. 12. The computer-readable medium of claim 10, wherein the method further comprises: A motion saliency value for the first partition is determined from the output of the motion estimation of the high efficiency video coding. 13. The computer-readable medium of claim 10, wherein the left-view frame is partitioned into a plurality of non-overlapping partitions, and the motion estimation is based on the same partition as the first partition size. 14. The computer-readable medium of claim 13, wherein the method further comprises: The first sub-block is quantized according to the quantization parameter. 15. The computer-readable medium of claim 13, wherein the method further comprises: determining a hybrid stereo saliency map for the left-view frame; reducing the size of the hybrid stereo saliency map to match the size of the transform unit; determining a second quantization parameter for the transform unit; and The transform unit is quantized according to the second quantization parameter. 16. The computer-readable medium of claim 10, wherein the left-view and right-view frames are modified in a first direction, and the method further comprises: The disparity saliency value is searched for in a second direction perpendicular to the first direction. 17. The computer-readable medium of claim 16, wherein the disparity saliency value comprises a non-integer value. 18. The computer-readable medium of claim 17, wherein the method further comprises: The disparity saliency value is determined from 1/4 pixel samples generated by sub-pixel motion estimation for high efficiency video coding.

Images