JP2017532858A - Perceptual optimization for model-based video coding - Google Patents

Abstract

A computer-based method for processing video data that applies an importance map to video compression to improve the quality of video coding. Perceptual statistics are used to calculate an importance map that indicates which regions of a video frame are important to the human visual system. The importance map is applied to the video encoding process so as to improve the quality of the encoded bitstream. A temporal contrast sensitivity function (TCSF) is calculated from the motion vector of the encoder. A motion vector quality measure is used to build a true motion vector map (TMVM). True motion vector maps (TMVM) are used to refine the TCSF. A spatial complexity map (SCM) is calculated. SCM is combined with TCSF to obtain an integrated importance map. The importance map is used to improve encoding. [Selection] Figure 8B

Description

Related applications

  This application claims the benefit of US Provisional Patent Application No. 62 / 158,523 filed May 7, 2015 and US Provisional Patent Application No. 62 / 078,181 filed November 11, 2014. This application is further a continuation-in-part (CIP) of US patent application Ser. No. 14 / 532,947, filed Nov. 4, 2014. This U.S. Patent Application No. 14 / 532,947 claims the benefit of U.S. Provisional Patent Application No. 61 / 950,784, filed March 10, 2014, and U.S. Provisional Patent Application No. 62 / 049,342, filed September 11, 2014. To do. The entire teachings of these referenced patent applications are incorporated herein by reference.

  Video compression is considered to be a process of expressing digital video data in a format using a small number of bits during storage or transmission. Video coding can achieve compression by exploiting the spatial redundancy, temporal redundancy, or color space redundancy of video data. Typically, the video compression process divides video data into parts such as a collection of frames or a collection of pels, identifies redundant parts in the video, and finds the redundant parts in the original video data. It can be expressed with a smaller number of bits than is possible. By taking advantage of this redundancy of data, greater compression can be achieved. An encoder can be used to convert the video data into an encoded format. Then, by using the decoder, the encoded video can be converted into a form comparable to the original video data. What implements an encoder / decoder is called a codec.

  In encoding, a standard encoder (standard encoder) divides a given video frame into a plurality of encoding units, that is, macro blocks (rectangular regions including a plurality of continuous pels) that do not overlap each other. Typically, macroblocks (more generally referred to herein as “input blocks” or “data blocks”) are processed in a left-to-right scan order or top-to-bottom scan order of a video frame. Is done. The compression can be achieved by predicting and encoding the input block using encoded data. The process of encoding an input block using samples spatially adjacent to the input block among previously encoded blocks in the same frame is called intra prediction. Intra prediction seeks to exploit spatial redundancy in the data. Encoding an input block with similar regions from previously encoded frames found using a motion estimation process is referred to as inter prediction. Inter prediction seeks to take advantage of temporal redundancy in the data. The motion estimation process may generate a motion vector. The motion vector specifies, for example, the position of the matching region in the reference frame with respect to the input block being encoded. Most motion estimation processes involve a motion initial estimate (initial motion estimate) that provides an initial coarse estimate (and corresponding temporal prediction) of the motion vector for a given input block, and in the vicinity of this initial estimate. It consists of two main steps: fine motion estimation (fine motion estimation) that determines a more accurate estimation (and corresponding prediction) of the motion vector for the input block by performing a local search.

  The encoder can generate a residual by measuring the difference between the data to be encoded and the prediction (prediction result). This residual may provide the difference between the predicted block and the original input block. These predictions, motion vectors (for inter prediction), residuals and related data are combined with other processes such as spatial transformation, quantization, entropy coding, loop filter, etc., to efficiently code video data ( Encoding) can be generated. The quantized and transformed residuals are processed and recombined into the prediction, assembled into a decoded frame and stored in the frame store. Details of such video encoding techniques are well known to those skilled in the art.

  MPEG-2 (H.262) and H.264 (MPEG-4 Part 10 Advanced Video Coding (AVC)) are two types of codec standards for video compression that achieve high quality video representation at relatively low bit rates. (Hereinafter referred to as MPEG-2 and H.264, respectively). MPEG-2 and H.264 An H.264 encoding basic unit is a 16 × 16 macroblock. H. H.264 is a widespread recent video compression standard and is generally considered to have twice the efficiency of MPEG-2 in compressing video data.

  The basic MPEG standard defines three types of frames (or pictures) based on the encoding method of input blocks in a frame. An I frame (intra-encoded picture) is encoded using only data existing in the frame, and is therefore composed only of intra-predicted blocks. A P frame (predicted picture) is encoded by forward prediction using data from an I frame or P frame (also referred to as a reference frame) decoded in advance. A P frame may include both intra blocks and (forward) predicted blocks. B frames (bi-predictive pictures) are encoded by bi-directional prediction using data from both the previous and subsequent frames. A B frame may include any of an intra block, a (forward) prediction block, and a bi-prediction block.

  A specific set of reference frames is referred to as a Group of Pictures (GOP). The GOP contains only the decoded pels in each reference frame, and information about how the input block or frame was encoded (I frame, B frame or P frame) Not included. Older video compression standards such as MPEG-2 use one reference frame (past frame) to predict P frames and two reference frames (previous frame and 1) to predict B frames. Use the next frame). In contrast, H. Newer compression standards such as H.264, HEVC (High Efficiency Video Coding) make it possible to use multiple reference frames for P and B frame prediction. A typical reference frame is a frame that is temporally adjacent to the current frame, but these standards also allow a frame that is not temporally adjacent to be a reference frame.

  Conventional inter prediction is based on block-based motion estimation and compensation (BBMEC). The BBMEC process searches for the best match between the target block (the current input block being encoded) and a region of the same size in a previously decoded reference frame. If such a match is found, the encoder may send a motion vector that acts as a pointer to the position of this best match in the reference frame. However, the BBMEC search process is not only limited in terms of time in terms of reference frames that can be searched, but also in terms of space in terms of neighboring regions that can be searched, for computational reasons. ing. This means that the “best possible” match is not always found, and is especially true for fast changing data.

  The simplest form of BBMEC process takes a (0,0) motion vector as the initial setting for motion estimation. This means that the initial estimation of the target block is the block at the same position in the reference frame. Next, motion refinement estimation is performed by searching for a region that best matches the target block in the local neighborhood of this region (ie, the error for the target block is minimized). This local search can be done by exhaustively querying the local neighborhood or by using any of several “fast search” methods such as diamond search, hexagonal search, etc. .

  An improved predictive zone search (EPZS) method (Non-Patent Document 1: “Single-multi-frame motion by Tourapis et al.”) Enhanced predictive zonal search for single and multiple frame motion estimation ”). This EPZS method uses a motion vector candidate set based on a motion vector of a neighboring block that has already been encoded and a motion vector of a block (and a neighborhood) of the same position in the previous reference frame for initial estimation of the target block. consider. The EPZS method considers that the motion vector field of the video has some spatial and temporal redundancy, so the initial motion estimation for the target block is either in the motion vector of the neighboring block or in the encoded frame. Assume that it is reasonable to use motion vectors from neighboring blocks. In the EPZS method, when a set of initial estimates is collected, the set is narrowed down by approximate rate distortion analysis. After this, motion precision estimation is performed.

  For any given target block, the encoder may generate multiple inter predictions that are selection candidates. These predictions can arise from multiple prediction processes (eg, BBMEC, EPZS, model-based, etc.). Also, these predictions may differ based on the target block sub-partitioning process. In the sub-partition processing, different motion vectors are associated with different sub-partitions of the target block, and each motion vector indicates an area of the sub-partition size in the reference frame. Also, these predictions may differ based on the reference frame that the motion vector points to. This is because, as described above, recent compression standards allow the use of multiple reference frames. Usually, the selection of the best prediction for a given target block is achieved by rate distortion optimization. In rate distortion optimization, the best prediction is the rate distortion measure D + λR, where distortion D is the error between the target block and the prediction, and rate R quantifies the cost (in bits) to encode the prediction. , Λ is a scalar weighting factor).

Tourapis, A., 2002, "Enhanced predictive zonal search for single and multiple frame motion estimation," Proc. SPIE 4671, Visual Communications and Image Processing, pp. 1069-1078

  In the past, many model-based compression schemes have been proposed in order to avoid the limitations of BBMEC prediction. Such a model-based compression scheme (the MPEG-4 Part 2 standard is perhaps best known as this kind of scheme) is an object or feature (generally defined as “component of interest”) in a video. Detection and tracking as well as a method of encoding these features / objects separately from the rest of the video frame. Since feature / object detection / tracking is done independently of the spatial search in the standard motion estimation process, the feature / object track results in a different set of predictions than those obtained by standard motion estimation. obtain.

  However, such model-based compression schemes based on features / objects face problems due to dividing a video frame into object regions and non-object regions (or feature regions and non-feature regions). First, since the size of an object can vary widely, it is necessary to encode not only the texture (color content) of the object but also the shape of the object. Second, tracking multiple objects with motion can be difficult, and inaccurate tracking causes inaccurate segmentation and usually leads to poor compression performance. The third problem is that not all video content is composed of objects and features, so if no object / feature exists, an alternative encoding scheme is required.

  Co-pending US Provisional Patent Application No. 61 / 950,784, filed November 4, 2014 (referred to herein as the '784 application), presents a model-based compression scheme that avoids the above segmentation problem. ing. The continuous block tracker (CBT) of the '784 application does not detect objects or features, and eliminates the need to split objects and features from non-object / non-feature backgrounds. Rather, CBT tracks all input blocks (“macroblocks”) in a video frame as if they were a region of interest by incorporating frame-to-frame motion estimation into a continuous track. By doing so, the CBT models the motion in the video so as to enjoy the benefits of higher-order modeling (modeling) of data to improve inter prediction while avoiding the segmentation problem.

  Another model-based compression approach is to model the human visual system (HVS) response to video data content as an importance map that indicates which parts of the video frame are most noticeable to human perception. It is done. The importance map takes a numerical value for each input block or data block in the video frame. Also, the importance map value (value of the importance map) for any given block can vary from frame to frame throughout the video. In general, importance maps are defined such that higher numbers indicate more important data blocks.

  As a kind of importance map, temporal contrast sensitivity function (TCSF) (de Lange, H., 1954, "Relationship between critical flicker frequency and a set of low frequency characteristics of the eye" And J. Opt. Soc. Am., 44: 380-389). TCSF measures the response of HVS to periodic stimuli over time, revealing that certain temporal characteristics in the data are easily noticeable to the observer human. With these temporal characteristics associated with movement in the data, the TCSF is a “medium” that does not fall into either the very high or very low temporal frequencies of the type of movement most noticeable in the data. Predict that it is movement.

  An important point to note is that the TCSF requires an accurate measurement of the speed of the content with motion in the video in order to generate an accurate temporal contrast value. Such a speed can be approximated by calculating an optical flow that represents the net (apparent) motion of the video content due to camera motion and / or object motion. However, most standard video encoders employ a motion estimation process that optimizes compression efficiency rather than accurately calculating the optical flow.

  Other types of importance maps include those based on spatial contrast sensitivity, which measures the response of HVS to spatial characteristics such as brightness, edges, spatial frequency, and color. Spatial contrast sensitivity function (SCSF) (eg Barten, P., 1999, Contrast Sensitivity of the Human Eye and Its Effects on Image Quality), SPIE Press, etc. (See also), also known simply as the contrast sensitivity function (CSF), measures the spatial contrast that is significant for HVS. SCSF has been successfully applied in the JPEG2000 image compression standard for the purpose of reducing image compression artifacts. Objects and features are also typically detected with the aid of spatial contrast techniques (eg, the presence of edges indicated by spatial frequency gradients). Spatial contrast sensitivity has been studied and used in image compression (for example, JPEG2000 codec), and many video compression processes based on object / feature detection have been proposed, but the time represented by TCSF. Conventional contrast sensitivity has never been applied to video compression.

  Some inventive embodiments disclosed apply an importance map to video compression so as to improve the quality of video coding. In an exemplary embodiment, the temporal frequency within the standard video encoding process stream is determined by determining the wavelength approximation using structural similarity (SSIM) in the color space domain, and encoder motion. It is calculated by obtaining an approximation of speed using a vector (encoder motion vector). The temporal frequency then serves as an input to the temporal contrast sensitivity function (TCSF). The TCSF may be calculated for all data blocks, thereby generating a temporal importance map that indicates which regions of the video frame are most noticeable to the human being who is the observer.

  In an exemplary further embodiment, information about the relative quality of the motion vectors generated by the encoder can be calculated at various points in the encoding process and used to generate a true motion vector map. The true motion vector map outputs how reliable the motion vector is for each target block. This true motion vector map that takes a value of 0 or 1 refines the TCSF so that TCSF is not applied to target blocks whose motion vectors are not accurate (ie, target blocks whose true motion vector map is 0). Can be used as a mask.

  In further embodiments, measures such as block variance (intra-block variance), block luminance, edge detection, etc., so that the spatial complexity map (SCM) determines the spatial contrast of a given target block to its neighborhood. Can be calculated from In other embodiments, information from the SCM can be combined with the TCSF to obtain a composite integrated importance map. The combination of spatial and temporal contrast information in this integrated importance map effectively balances both sides of the human visual response.

  In one exemplary embodiment, an integrated importance map (an importance map that includes information from both TCSF and SCM) is used to weight the distortion portion of the standard rate distortion measure D + λR. Used. This weights the solution that matches the perceptual relative importance of each target block (a low distortion solution when the importance map is close to its maximum value and a low rate solution when the importance map is close to its minimum value) Modified rate distortion optimization is obtained. In an alternative embodiment, TCSF or SCM can be used independently for the above purposes.

  In another exemplary embodiment, TCSF and SCM (with refinement with true motion vectors) may be used to adjust the block level quantization of the encoder. In the target block in which the importance map has a high numerical value, the quantization parameter is made smaller than the frame quantization parameter, so that high quality can be obtained for these blocks. In the target block having a low importance map, the quantization parameter is set larger than the frame quantization parameter, so that a low quality can be obtained for these blocks. In an alternative embodiment, TCSF or SCM can be used independently for the above purposes.

  The TCSF can be calculated for any encoder that incorporates inter prediction and generates motion vectors (used by the TCSF to approximate the speed of the content in the video), Applying TCSF to video compression is most effective in model-based compression frameworks, such as the '784 application continuous block tracker (CBT), which can accurately determine which motion vectors are true motion vectors. Become. As described above, most standard video encoders calculate motion vectors that optimize compression efficiency rather than reflecting true motion. In contrast, CBT provides both motion vectors suitable for high compression efficiency and modeling information that maximizes the effect of TCSF.

  Some exemplary inventive embodiments are directed to any video compression standard in which the resulting bitstream uses block-based motion estimation followed by residual signal transformation, quantization and entropy coding. Constructed to be compliant. Such video compression standards are MPEG-2, H.264, etc. Including, but not limited to, H.264 and HEVC. The present invention can be applied to a non-standard video encoder that is not block-based as long as it incorporates inter prediction and generates a motion vector.

  Some exemplary embodiments may include a method and system for encoding video data, and any codecs (encoders and decoders) to implement it. A plurality of video frames that have target blocks that are non-overlapping video frames can be processed by the encoder. The plurality of video frames uses an importance map, and the importance map modifies (adjusts) quantization so as to change the encoding quality of each target block to be encoded in each video frame. Can be encoded by the encoder.

The importance map may be configured using at least one of temporal information and spatial information. When both temporal information and spatial information are used, the importance map is considered as an integrated importance map. The importance map may be set so as to indicate (specify or represent) a portion of a certain video frame that is most noticeable to human perception. More specifically, in blocks where the importance map has a high value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that high quality is obtained for these blocks. It is done. Further, in the target block having a low importance map, the block quantization parameter is set larger than the frame quantization parameter QP _frame , so that low quality is obtained for these blocks.

The spatial information may be provided by a rule-based spatial complexity map (SCM), the first step of which is to determine which target block in the _frame is greater than the average block variance var _{frame in the frame} Is to determine whether to have A QP value higher than the frame quantization parameter QP _frame can be assigned to a block having a variance greater than the average block variance var _frame , and the block equivalent QP _block of the block QP is the block variance var _block. Is linearly increased or decreased between the QP _frame and the quantization parameter upper limit QP _max according to how much is larger than the var _frame .

  Preferably, the temporal information includes a temporal contrast sensitivity function (TCSF) indicating which target block is most easily noticed by a human being who is an observer, and which target block corresponds to foreground data. It can be provided by a true motion vector map (TMVM) shown. The TCSF can be valid only for the target block specified as the foreground data.

A high-variance block has a QP _block equivalent QP _{block for} which the TMVM identifies the target block as foreground data and the contrast sensitivity log value for this block in the TCSF is less than 0.5. The QP _block can be further refined by the TCSF and the TMVM to increase by 2.

The SCM may include luminance masking in which the block quantization parameter QP _block of the target block that is very bright (luminance greater than 170) or extremely dark (luminance less than 60) is readjusted to QP _max . The SCM may include a dynamic determination of QP _max based on the quality level of the encoded video, where the dynamic determination of the average structural similarity (SSIM) of the target block within an intra (I) frame. ) Using the calculation result together with the average block variance var _frame of these frames, the quality is measured, and if the measured quality is low, the numerical value of the quantization parameter upper limit QP _max is somewhat closer to the frame quantization parameter QP _frame As reduced.

In order to ensure high quality coding in these regions for very-low-variance blocks, the smaller the block variance, the lower the value of the vibration equivalent QP _block ( In addition, the shaking equivalent QP _block, which is a value of the determined low quantization parameter (QP), can be assigned (so that the quality is high). The shaking equivalent QP _block , which is the value of the low quantization parameter (QP) for a very small variance _block, is first determined for I frames, and then using the i and p brati parameters for P and B frames. Can be determined. A block whose variance is small but whose variance is not considered to be extremely small is the initial equivalent value of the block quantization parameter (QP) to determine whether quality improvement is necessary for the block. The QP _block is examined to be calculated by averaging the quantization parameter (QP) values of the already encoded neighboring blocks at the left, upper left, right and upper right of the current block. The estimated SSIM _{est of the} SSIM of the current block may be calculated from the SSIM values of the already encoded neighboring blocks at the left, upper left, right and upper right of the current block. When the SSIM _est is less than 0.9, the value of the shaking equivalent QP _block may be decreased by 2.

  In some embodiments, the quality enhancement is applied only to blocks identified as foreground data by the TMVM and having a contrast sensitivity logarithm value of the TCSF greater than 0.8. The TMVM can be set to 1 only for foreground data.

  In some embodiments, the temporal frequency of the TCSF is determined using a SSIM in the color space region between the target block and its reference block to approximate the wavelength and the magnitude of the motion vector (motion vector magnitude). And the frame rate are used to calculate an approximation of the speed.

  The TCSF may be calculated over multiple frames so that the TCSF for the current frame is a weighted average of the TCSF maps in the most recent frame and more recent frames receive greater weighting.

  Foreground data can be identified by calculating the difference between the encoder motion vector for a given target block and the global motion vector for that block, and determining that a block with a sufficiently large difference is foreground data. .

  For a data block identified as foreground data, a difference motion vector may be obtained by subtracting the encoder motion vector from the global motion vector, and the magnitude of the difference motion vector is a temporal frequency of the TCSF. Is used to calculate

  Computer based methods for processing video data, codecs (encoders and decoders) for processing video data, and other computer systems and devices for processing video data may embody the aforementioned principles of the present invention.

  The foregoing will become apparent from the following more detailed description of exemplary embodiments of the invention as illustrated in the accompanying drawings. In the drawings, like reference numerals refer to like elements / components throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention.

It is a block diagram which shows the structure of a standard encoder. It is a block diagram which shows the step accompanying the inter prediction in the case of a general encoder. It is a block diagram which shows the step accompanying the motion initial estimation by continuous block tracking. FIG. 6 is a block diagram illustrating integrated motion estimation through a combination of continuous block tracking and extended prediction area search. It is a plot which shows the latest measurement result (2010) of the temporal contrast sensitivity function by Wooten et al. It is a block diagram which shows the mode of calculation of the structural similarity (SSIM) in CIE1976Lab color space in one Embodiment of this invention. FIG. 6 is a block diagram illustrating a general application of perceptual statistics to improve the perceptual quality of video encoding in one embodiment of the present invention. FIG. 6 is a block diagram illustrating the application of perceptual statistics to improve the perceptual quality of video coding by modifying inter prediction with continuous block tracking in one embodiment of the present invention. FIG. 6 is a block diagram illustrating an example of a process for modifying and encoding block quantization using an importance map. It is the schematic of the computer network environment by which each embodiment is deployed. FIG. 9B is a block diagram of computer nodes in the network of FIG. 9A.

  The entire teachings of all patent publications, all patent application publications and all publications cited herein are hereby incorporated by reference. In the following, exemplary embodiments of the invention will be described.

  The present invention is applicable to various standard encodings. In the following, unless otherwise specified, the terms “conventional” and “standard” (often used in conjunction with “compression”, “codec”, “encoding” and “encoder”) are MPEG-2, MPEG- 4. H. H.264 or HEVC. “Input block” refers to the basic encoding unit of the encoder without loss of generality, and can often be referred to as “data block” or “macroblock”. The current input block being encoded is referred to as the “target block”.

<Video coding and inter prediction by continuous block tracking>
The encoding process may convert video data into a compressed or encoded format. Similarly, the decompression or decoding process may convert the compressed video to an uncompressed or raw format. The video compression / decompression process may be implemented as an encoder and decoder pair commonly referred to as a codec.

  FIG. 1 is a block diagram of a standard transform-based and motion compensated encoder. The encoder of FIG. 1 may be implemented in a software environment, a hardware environment, or a combination thereof. The encoder may comprise any combination of components. These components are the motion estimation unit 15 output to the inter prediction unit 20, the intra prediction unit 30, the transform / quantization unit 60, the inverse transform / quantization unit 70, the in-loop filter 80, the frame store 85, and the entropy coding. Including, but not limited to, means 90. The purpose of the above prediction means (both inter prediction and intra prediction) is to provide the best prediction for a given input video block 10 (abbreviated “input block”, or “macroblock” or “data block”). Generating a signal 40. A prediction residual 50 is generated by subtracting the prediction signal 40 from the input block 10, and the prediction residual 50 is subjected to transformation / quantization 60. Thereafter, the quantized coefficient 65 of the residual is passed to the entropy encoding unit 90, and the entropy encoding unit 90 encodes the compressed bit stream. The quantization coefficient 65 is also passed to the inverse transform / quantization means 70, and the resulting signal (approximation of the prediction residual) is recombined into the prediction signal 40, thereby reconstructing the input block 10 A signal 75 is generated. The reconstructed signal 75 can be passed through an in-loop filter 80, such as a deblocking filter, and this (optionally filtered) reconstructed signal becomes part of the frame store 85. The frame store 85 supports prediction of future input blocks. Those skilled in the art are familiar with the function of each component of the encoder shown in FIG.

  FIG. 2 shows various steps in standard inter prediction (reference numeral 30 in FIG. 1). The purpose of inter prediction is to encode new data using pre-decoded data from the previous frame, taking advantage of temporal redundancy in the data. In inter prediction, an input block 10 from a frame that is currently being encoded (also referred to as a target frame) is stored from a region of the same size in a pre-decoded reference frame stored in the frame store 85 of FIG. "is expected. A two-component vector indicating a shift of (x, y) between the position of an input block in a frame being encoded and the position of a region matching this in a reference frame is referred to as a motion vector. Thus, the process of motion estimation involves determining the motion vector that best associates the input block to be encoded with the matching region in the reference frame.

  Most inter prediction processes begin with an initial motion estimate (reference 110 in FIG. 2) that generates one or more coarse estimates of the “good” motion vector 115 for a given input block. This is optionally followed by a motion vector candidate filtering step 120 where multiple motion vector candidates can be reduced to a single candidate using an approximate rate distortion measure. In rate distortion analysis, the best motion vector candidate (prediction) is the rate distortion measure D + λR, where distortion D is the error between the input block and the matching region, and rate R encodes the prediction. The one that quantifies the cost (in bits) and minimizes λ is a scalar weighting factor is selected. The actual rate cost includes two types of components, texture bits and motion vector bits. Texture bits are the number of bits required to encode the quantized transform coefficients of the residual signal (input block minus prediction) and motion vector bits are required to encode the motion vector The number of bits. Usually, a motion vector is differentially encoded with respect to an already encoded motion vector. Since texture bits are not available at an early stage in the encoder, the rate portion of the rate distortion measure is approximated by motion vector bits. On the other hand, the motion vector bit is approximated as a motion vector penalty coefficient depending on the magnitude of the difference motion vector. Thus, in the motion vector candidate filtering step 120, this approximate rate distortion measure is used to select a single “best” initial motion vector or a smaller set of “best” initial motion vectors 125. Such initial motion vector 125 is then refined (refine) by motion refinement estimation 130. Motion refinement estimate 130 performs a local search in the vicinity of each initial estimate to determine a more accurate estimate of the motion vector (and corresponding prediction) for that input block. This local search is usually followed by sub-pixel refinement where integer motion vectors are refined to 1/2 or 1/4 pixel accuracy by interpolation. The motion refinement estimation block 130 generates a refined set of motion vectors 135.

  Next, when the motion vector 135 is given, the mode generation unit 140 generates a set of prediction candidates 145 based on the encoding modes that can be adopted by the encoder. Such a mode differs depending on the codec. Different coding modes mean interlaced vs progressive (field vs frame) motion estimation, reference frame direction (forward prediction, backward prediction, bi-prediction), reference frame index (multiple reference frames allowed H.264, for codecs such as HEVC), inter prediction versus intra prediction (some scenarios that allow returning to intra prediction in the absence of good inter prediction), different quantization parameters, and inputs Different sub-sections of the block (but not limited to). The entire set of prediction candidates 145 is subjected to a “final” rate distortion analysis 150 to determine a single best candidate. In the “final” rate-distortion analysis, an accurate rate-distortion measure D + λR is used to predict the distortion error D (usually calculated as the sum of squared errors (SSE)) and the actual coded bit R for the rate part. (From the entropy encoding 90 of FIG. 1) is calculated. The final prediction 160 (ie, code 40 in FIG. 1) is the prediction with the lowest rate distortion score D + λR among all candidates, and this final prediction along with its motion vector and other coding parameters at the encoder. Passed to subsequent steps.

  FIG. 3 shows how initial motion estimation by continuous block tracking (CBT) can be performed during inter prediction. CBT is useful when there are multiple frame gaps between a target frame and a reference frame from which temporal prediction is derived. In the case of MPEG-2, the typical GOP structure of IBBPBBP (consisting of intra-predicted I-frames, bi-predicted B-frames and forward-predicted P-frames) allows for reference frames that are separated by up to 3 frames from the current frame. (The reason is that in MPEG-2, the B frame cannot function as a reference frame). H. allows multiple reference frames for each frame to be encoded. In H.264 and HEVC, even with the same GOP structure as described above, a reference frame that is more than six frames away from the current frame is enabled. For longer GOP structures (eg, 7 B frames between P frames, etc.), the reference frame can be even further away from the target frame. When there are multiple frame gaps between the current frame and the reference frame, continuous tracking allows the encoder to capture motion in the data that is not captured by standard temporal prediction techniques, CBT makes it possible to generate better temporal predictions.

  The first step in CBT is to perform frame-to-frame tracking (reference 210 in FIG. 3). For each input block 10 in a given frame, motion vectors are calculated both backward to the previous frame in the frame buffer 205 and forward to the next frame in the frame buffer. In one embodiment, frame-to-frame tracking operates on frames from the original source video rather than reconstructed reference frames. This is because the source video frame is not degraded by quantization or other coding artifacts, so tracking based on the source video frame is advantageous because it more accurately represents the true motion field in the video. Frame-to-frame tracking may be performed using conventional block-based motion estimation (BBME) or hierarchical motion estimation (HME).

  The result of the frame-to-frame tracking is the result of the best matching region in the most recent frame in the frame buffer 205 and the most recent frame in the frame buffer 205 for each input block in the frame. A set of frame-to-frame motion vectors 215 representing the best matching region within the current frame for each block. Next, continuous tracking 220 generates a continuous track for each input block across multiple reference frames by aggregating available frame-to-frame tracking information. Details of how to perform continuous tracking are described in the '784 application, which is incorporated herein by reference in its entirety. The output of continuous tracking 220 is a continuous block tracking (CBT) motion vector 225 that tracks all input blocks in the current frame being encoded to regions that match them in past reference frames. In the case of CBT, these CBT motion vectors become the initial motion vector (reference numeral 125 in FIG. 2) and can be refined by motion refinement estimation (reference numeral 130 in FIG. 2) as described above.

  FIG. 4 illustrates how in one embodiment of the invention, CBT can be combined with the EPZS method to create an integrated motion estimation process. In FIG. 4, after the CBT generates motion vectors with frame-to-frame tracking 210 and continuous tracking 220 for initial motion estimation 110, local search and subpixel refinement 250 for motion refinement estimation 130 is continued. . After the EPZS generates the initial motion vector by the candidate generation means 230, the candidate filtering means 240 for performing filtering by the approximate rate distortion analysis as described in detail above is continued. This is further followed by motion refinement estimation 130 with local search and subpixel refinement 260. Both the CBT motion vector 255 and the EPZS motion vector 265 obtained in this way are used for the remaining inter prediction steps (mode generation 140 and final rate in FIG. 2) to determine the overall “best” inter prediction. To the distortion analysis 150).

  In an alternative embodiment, additional candidates may be added to the CBT motion vector candidate 255 and EPZS motion vector candidate 265 of FIG. Such candidates include (but are not limited to) random motion vectors, (0,0) motion vectors, and so-called “median predictors”. The motion refinement estimation 130 may be applied to the random motion vector so as to find the best candidate in the local vicinity. The (0,0) motion vector is one of the first candidates for EPZS, but is not always selected at the time after EPZS candidate filtering (reference numeral 240 in FIG. 4). Even if selected at the time after filtering, there is a possibility that a motion vector other than (0, 0) may be output by the motion refinement estimation 130. Explicit inclusion of (0,0) motion vectors (not subject to fine motion estimation) as candidates for the final rate distortion analysis is considered at least one small and "small motion" candidate Make sure. Similarly, the “median predictor” is one of the first candidates for EPZS, but is not always selected at the time after EPZS candidate filtering (reference numeral 240 in FIG. 4). The median predictor is defined as the median of motion vectors calculated in advance in the left, upper and upper right data blocks of the data block currently being encoded. Explicit inclusion of intermediate predictors (not subject to motion estimation) as candidates for final rate distortion analysis encodes spatially uniform ("flat") regions of the video frame Can be particularly beneficial. That is, in this alternative embodiment, five or more types of motion vector candidates (motion vector derived from CBT, motion vector derived from EPZS, motion vector derived from random motion vector, (0, 0) motion vector, and median prediction) Including (but not limited to) children may be passed to the remaining inter prediction steps (mode generation 140 and final rate distortion analysis 150 of FIG. 2).

<Calculation of importance map for video coding>
Perceptual statistics can be used to calculate an importance map that indicates which regions of the video frame are important to the human visual system (HVS).

  An example of a perceptual statistic is the so-called temporal contrast sensitivity function (TCSF), which models the human visual system (HVS) response to temporally periodic stimuli. As mentioned in the background section, the concept of TCSF has existed since the 1950s (introduced as “time-modulated transfer function”), but has never been applied to video compression. Figure 5 shows recent results of TCSF (Wooten, B. et al. 2010, "A practical method of measuring the temporal contrast sensitivity function", "Biomedical Optical Express, l (l): 47-58) in the form of logarithm of temporal contrast sensitivity as a function of logarithm of frequency (logarithm of frequency on the horizontal axis and logarithm of temporal contrast sensitivity on the vertical axis). Measurement data points (circles in FIG. 5) are fitted using a cubic polynomial (solid line in FIG. 5). This fitting is used for all TCSF calculations described later. TCSF shows that the human visual system (HVS) shows a maximum response to the medium frequency range, while the HVS response slightly decreases in the low frequency range and decreases rapidly in the high frequency range. I expect.

  Application of TCSF to video compression requires a method of calculating a temporal frequency (horizontal axis in FIG. 5) that is an input to TCSF. One method according to an embodiment of the present invention for calculating the frequency will be described below. The frequency f is given by f = v / λ, where v is the velocity and λ is the wavelength. In one embodiment, the content speed v (in pixels / second) of any data block is determined by the motion vector generated by the encoder (eg, reference numeral 135 in FIG. 2, reference numerals 215 and 225 in FIG. 3, reference numerals in FIG. 4). V = | MV | × frame rate / N (where | MV | is the magnitude of the motion vector of the data block, and the frame rate is 1 at which the video was generated). The number of frames per second, where N is the number of frames between the reference frame pointed to by the motion vector and the current frame.

  A suitable approximation of the wavelength λ is the structural similarity (SSIM) calculated in the CIE 1976 Lab color space (www: //en.wikipedia.org/wiki/Lab_color_space) (Wang, Z. et al. 2004, “Image quality assessment: From error visibility to structural similarity, "IEEE Trans, on Image Processing, 13 (4): 600-612). FIG. 6 shows how the SSIM is calculated in the Lab color space. The SSIM is calculated between the target block 300 (the current data block to be encoded) and the reference block 310 pointed to by its motion vector. Usually, video data processed by the encoder is represented in a standard space such as YUV420, so the next step is to generally describe both the target block (reference numeral 320) and the reference block (reference numeral 330) in the literature. It is to convert to CIE 1976 Lab space using an arbitrary method. Next, the error ΔE (reference numeral 340) between these target block and reference block in Lab space is

(In the formula, subscript T means “target block” and subscript R means “reference block”). Finally, the SSIM 360 between the error ΔE and the same dimension zero matrix is calculated as indicating a measure of the color space change of the data. The SSIM determined at the beginning takes a value of −1 to 1, and the value 1 indicates complete similarity (no spatial difference). For the purpose of converting SSIM to wavelength λ, it may be possible to use a spatial dissimilarity DSSIM = (1−SSIM) / 2 which takes a value between 0 and 1, where 0 is the short wavelength (maximum space 1 corresponds to a long wavelength (minimum spatial similarity). To convert SSIM to pixel units, it may be possible to multiply the SSIM value by the number of pixels in the block to be calculated. In one embodiment, the DSSIM value is multiplied by 64 because the block size of the SSIM is 8x8. In this case, the final calculation result of the frequency is
f = | MV | × frame rate / (N × 64 × (1-SSIM) / 2)
Given by.

Once the frequency for a given target block is calculated, the TCSF value for this block can be determined from the curve fit (solid line) of FIG. TCSF takes a numerical value of 0 to 1.08 on a log ₁₀ scale or 1 to 11.97 on an absolute scale. Different blocks in a frame take different TCSF values, so an aggregate set of TCSF values across all blocks in the frame forms an importance map, with higher numbers in terms of temporal contrast Perceptually important blocks are indicated and low numbers indicate perceptually insignificant blocks.

In a further embodiment, TCSF values from recent frames may be averaged for each data block so that the TCSF-based importance map does not fluctuate too much between frames. For example, as one such calculation of the average TCSFTCSF _avg , TCSF _avg = 0.7 × TCSF _cur + 0.3 × TCSF _prev (where TCSF _cur is the TCSF value from the current frame and TCSF _prev is The TCSF value from the most recently encoded past frame). The calculation of TCSF becomes more robust by being averaged in this way.

  In further embodiments, information about the relative quality of the motion vectors generated by the encoder can be calculated at various points in the encoding process and used to generate a true motion vector map (TMVM). A true motion vector map (TMVM) outputs for each data block how reliable the motion vector is. This true motion vector map, which takes a value of 0 or 1, is used as a mask to refine the TCSF so that TCSF is not applied to data blocks whose motion vectors are not accurate (ie, data blocks with a TMVM value of 0). Can be.

  In one embodiment, the accuracy of the motion vector is determined by estimating the global motion model for a given video frame and applying the motion model to each data block in the frame. It can be determined by determining a motion vector and comparing this global motion vector with the encoder motion vector (encoder motion vector) for that data block. Global motion is an aggregated set of encoded motion vectors from that frame, and can be estimated from an aggregated set fitted to a 6-parameter or 8-parameter affine motion model. If the global motion vector and the encoder motion vector are the same (or similar) for a given data block, the encoder motion vector is considered accurate (and TMVN = 1 for that data block). If the two vectors are not identical, their prediction errors (measured by sum of squared errors (SSE) or sum of absolute differences (SAD)) may be compared. If one error is small and the other error is large, the motion vector with the smaller error is used for encoding and considered accurate (TMVM = 1).

  In an alternative embodiment, the magnitude of the difference between the global motion vector and the encoder motion vector for a given data block is such that the data block is foreground data (this means that the content in the data block is Used to identify a different motion than the rest of the frame (background). In this embodiment, TMVM is set to 1 and this is applied only when TCSF is foreground data. In a further embodiment, for a data block identified as foreground data, a differential motion vector is obtained by subtracting the encoder motion vector from the global motion vector, and the magnitude of this differential motion vector (not the encoder motion vector). Is used to calculate the frequency of the TCSF (in the above equation, | MV | is replaced with | DMV | (DMV is a differential motion vector)).

In other embodiments, motion vector symmetry may be used to refine the TMVM. Motion vector symmetry (Bartels, C. and de Haan, 2009 by G., "Temporal symmetry constraints in block matching ( temporal symmetry constraints in block ^{matching)," Proc. IEEE 13 th} Int'l. Symposium on Consumer Electronics , pp. 749-752) is defined as the relative symmetry of a pair of motion vectors that pair with each other when the temporal direction of motion estimation is switched, and is a measure of the quality of the calculated motion vector (symmetric) The higher the degree, the better the quality of the motion vector). The “symmetry error vector” is defined as a difference between a motion vector obtained by forward motion estimation and a motion vector obtained by backward motion estimation. Low motion vector symmetry (large symmetry error vector) often results in occlusion (obtaining or exposing the background object by moving one object in front of another object) It is an indicator that there is a complicated phenomenon such as that the movement of the image moves on or out of the video frame, illumination change, etc. (both make it difficult to derive an accurate motion vector).

  In one embodiment, the magnitude of the symmetry error vector is greater than half of the range of the data block being encoded (eg, for a 16 × 16 macroblock, the magnitude is greater than the (8,8) vector). ) Is determined to have a low degree of symmetry (low degree of symmetry). In other embodiments, the magnitude of the symmetric error vector is based on a motion vector statistic derived during the tracking process (eg, the motion vector magnitude (motion at a given combination of current or recent frames). If the average value of the vector magnitude) is larger than the average value of the motion vector magnitude plus a multiple of the standard deviation of the motion vector magnitude), it is determined that the degree of symmetry is low (the degree of symmetry is low). In one embodiment, TMVM value = 0 is automatically assigned to a data block whose motion vector has low symmetry in the above definition, and the other data block compares the global motion vector with the encoder motion vector. The previous TMVM value derived from is maintained.

  Flat blocks have a high spatial contrast sensitivity but are well known aperture problems when calculating motion vectors

Tends to produce unreliable motion vectors. A flat block is, for example, using an edge detection process (determined to be a flat block if no edge is detected in the data block) or by comparing the variance of the data block with a threshold ( A variance less than this threshold indicates a flat block). In one embodiment, the flatness of the block may be used to change the TMVM calculated as described above. For example, a TMVM value = 0 can be reassigned to a block detected as a flat block.

  In one embodiment, TMVM can be used as a mask to refine TCSF that is affected by whether it has reliable motion vectors. Since the value of TMVM is 0 or 1, multiplying the TMVM value for a block by the TCSF value for that block for each block has the effect of masking the TCSF. In the case of a block having a TMVM value of 0, the motion vector necessary for calculating the TCSF is unreliable, so that the TCSF is “invalid”. In the case of a block having a TMVM value of 1, the TCSF calculation result is considered to be reliable, and any method described so far is used with certainty.

  In other embodiments, a spatial contrast map may be generated instead of or in addition to the temporal contrast map (TCSF described above). In the present invention, a simple measure is used to measure spatial contrast (the opposite is referred to as “spatial complexity”). In one embodiment, block variance measured for both the luminance and chrominance components of the data is used to measure the spatial complexity of a given input block. An input block with a large variance is spatially complex and difficult to notice for HVS, so its spatial contrast is small.

  In other embodiments, block luminance measured for the luminance component of the data is used to refine the spatial complexity variance measurement. An input block that has a small variance (low spatial complexity, high spatial contrast) but is very bright or very dark is automatically considered low spatial contrast and has previously been measured as large Overrides the contrast. The reason is that an extremely dark area or an extremely bright area is difficult for the HVS to notice. The brightness threshold for classifying a given block as very bright or very dark is specific to the occasional application, but typical values for 8-bit video are very bright, More than 170 ", very dark but" less than 60 ".

  In order to form a spatial contrast map (SCM) indicating the regions that are easy to notice for HVS and the regions that are not easily noticed from the viewpoint of spatial contrast, the block dispersion modified by the block luminance as described above. Can be calculated for all input blocks of a video frame.

  In one embodiment, SCM can be combined with TCSF (refined by TMVM) to form an integrated importance map. This integrated map can be formed, for example, by normalizing both SCM and TCSF as appropriate, and then multiplying the SCM value for a given block by the TCSF value for that block on a block-by-block basis. . In other embodiments, SCM can be used as a substitute for TCSF. In other embodiments, SCM can be used to refine the TCSF. For example, for a high complexity block, the SCM value may overwrite the TCSF value for that block, and for a low complexity block, the TCSF value for that block may be used directly.

<Application of importance map to video coding>
The importance map described above can be applied to any video encoding process of a general encoder (FIG. 2) and a CBT encoder (FIG. 3) to improve the quality of the encoded bitstream.

  FIG. 7 shows a general application of the importance map to video coding. Input video frame 5 and frame store 85 are used to generate perceptual statistics 390. Perceptual statistics 390 are then applied to form a TCSF and / or SCM importance map 400 (refined by TMVM) as described above. Perceptual statistics 390 may include (but is not limited to) motion vector magnitude, block variance, block luminance, edge detection, and global motion model parameters. The input video frame 5 and the frame store 85 are further input as usual into the encoding of the video frame at 450. The encoding includes normal encoding steps (motion estimation 15, inter prediction 20, intra prediction 30, transformation / quantization 60 and entropy encoding 90 in FIG. 1). However, in FIG. 7, the encoding 450 is expanded by the importance map 400 in a method described later.

  FIG. 8A shows a specific application of the importance map for improving video coding using CBT. FIG. 8A shows an initial motion estimation (reference numeral 110 in FIG. 2) with a frame-to-frame tracking 210 step and a continuous tracking 220 step from the CBT. Then, the motion refinement estimation 130 is applied to the global CBT motion vector 225 in the same local search and subpixel refinement motion refinement estimation step (reference numeral 250 in FIG. 4) as described above. Again, this is followed by mode generation means 140 that generates a set of prediction candidates 145 based on encoding modes that the encoder may employ. Similar to FIG. 4, EPZS and other non-model based candidates (eg, (0,0) motion vectors, median predictors, etc.) can also be generated in parallel as part of an integrated motion estimation framework. (In FIG. 8A, these other candidates are not shown for the sake of simplicity). Also in FIG. 8A, the entire set of prediction candidates 145, including every coding mode of the CBT candidate and possibly every other coding mode that is not model-based, determines the single best candidate. A “final” rate distortion analysis 155 is received. In the “final” rate distortion analysis, an accurate rate distortion measure D + λR is used to predict the distortion error D for the distortion portion and the actual coded bit R for the rate portion (from entropy coding 90 in FIG. 1). Is calculated. The final prediction 160 (or 40 in FIG. 1) is passed along with its motion vector and other coding parameters to subsequent steps in the encoder.

  In FIG. 8A, perceptual statistics 390 may be calculated from motion vectors derived from frame-to-frame motion tracking 210 and then applied to form importance map 400 as described above. These importance maps 400 are input to the final rate distortion analysis 155. Again, perceptual statistics 390 may include (but is not limited to) motion vector magnitude, block variance, block luminance, edge detection, and global motion model parameters.

  In one embodiment, an importance map is used to modify the rate distortion optimization conditions in response to the importance map. In a standard encoder (see FIG. 2), the “final” rate distortion analysis 150 is performed so that the entire set of prediction candidates 145 for a given input block 10 determines a single best candidate. receive. In the “final” rate distortion analysis, an accurate rate distortion measure D + λR is used to predict the distortion error D for the distortion portion and the actual coded bit R for the rate portion (from entropy coding 90 in FIG. 1). Is calculated. The candidate with the lowest score for the rate distortion measure D + λR becomes the final prediction 160 for a given input block 10. In one embodiment of the invention, for the perceptually optimized encoder of FIG. 7 or FIG. 8, the rate at which the importance map IM is calculated at 400 and the final rate distortion analysis 155 is modified. The distortion scale D × IM + λR is used. In this modified rate distortion measure, the IM value for a given input block is multiplied by the distortion term, the higher the IM value, the more importance is assigned to the less distorted solution. This is because a high IM value indicates that the corresponding input block is perceptually important. The importance map may include TCSF, SCM (possibly refined with TMVM values), or a combination of these.

In a further embodiment, in addition to the above, the distortion D on the rate distortion measure may be calculated as a weighted sum of SSE (square error sum: “standard” technique for calculating distortion) and SSIM calculated in YUV space. . The weight γ is the average SSIM value SSIM _avg in the first few (or most recent) frames of the video is the first few (or the most recent) of the video. Can be calculated adaptively so as to be equal to the average SSE value SSE _avg in the frames (γ × SSIM _avg = SSE _avg ). That is, for each input block, the modified rate distortion measure is (SSE + γ × SSIM) × IM + 2λR, where the multiplication factor 2 in front of the λR term means that there are two distortion terms. ) Inclusion of SSIM in the distortion measurement further increases the proportion of HVS perception in rate distortion optimization since SSIM corresponds to the structural information of the data.

In other embodiments, importance maps (eg, TCSF with TMVM refinement, SCM, etc.), in addition to (or instead of) modifying rate distortion optimization, encoder block level quantization Can be used to modify Quantization controls the relative quality with which a given input block is encoded. That is, highly quantized data is a low quality encoded output, and low quantized data is a high quality encoded output. The amount of quantization is controlled by the quantization parameter QP. The standard encoder allocates different QP values QP _frame to different frame types, I frames are encoded with the lowest QP (highest quality) and B frames are encoded with the highest QP (lowest quality) And the P frame is encoded with an intermediate QP (intermediate quality).

In other words, the above method is to quantize a plurality of video frames (the video frames have target blocks that do not overlap each other) using an importance map, and each target block in each video frame is quantized. It presents a method of encoding by altering (and thereby affecting its encoding quality). Such importance maps can be set using temporal information (TCSF with TMVM refinement), spatial information, or a combination of the two types (ie, an integrated importance map). Since the importance map indicates which part of each video frame is most easily noticed by human perception, the numerical value of the importance map indicates the QP for each target block, and (i) the importance map is high. In the block that takes a numerical value, the block QP is made smaller than the QP _frame , so that these blocks have high quality, and (ii) in the block in which the importance map has a low numerical value, It is desirable that the block QP is changed so as to have a lower quality by making the block QP larger than the frame quantization parameter QP _frame .

  FIG. 8B shows an example of a process for modifying quantization during encoding using the importance map 400. At 400, an importance map may be constructed / formed using temporal and / or spatial information derived from perceptual statistics 390. The temporal information includes, for example, a temporal contrast sensitivity function (TCSF) indicating which target block is most easily noticed in time by a human being who is an observer, and a true value indicating which target block corresponds to the foreground data. The TCSF is valid only for target blocks identified as foreground data. Spatial information may be provided, for example, by a rule-based spatial complexity map (SCM).

The importance map 400 is then used to modify the quantization step 430 in the encoding 450 as described above. In blocks where the importance map takes a high value, the block quantization parameter (QP) is reduced with respect to the frame quantization parameter QP _frame , and high coding quality is obtained for these blocks. In blocks where the importance map has a low value, the block quantization parameter is increased with respect to the frame quantization parameter QP _frame , and low coding quality is obtained for these blocks. By using information from the importance map, the quantization can be modified to improve the encoding quality of each target block to be encoded in each video frame.

In one embodiment, the TCSF map for a given frame may be used to adjust the frame QP on a block-by-block basis. One method of calculating the block QP and QP _block is (2011, “Visual attention guided bit allocation in video compression by Li, Z. et al.), J. of the image and vision computing, 29 (1): 1-14), associating the amount of adjustment with the entire TCSF map in the frame, the resulting expression is QP _block = (TCSF _frame / (TCSF) _block × M)) × QP _frame (where TCSF _frame is the sum of TCSF values for all blocks in the _frame , QP _frame is frame QP, and M is the number of blocks in the frame). given. in a further embodiment, the multiplication factor _{(TCSF frame / (TCSF block ×} M)) is, QP _b The final value of the _ock can be increased or decreased so as not to become too small or too large for QP _frame.

In an alternative embodiment, block-by-block adjustment of a QP with a TCSF map may be achieved without reference to the entire TCSF map for that frame. In this embodiment, the calculation of QP _block is simpler: QP _block = QP _frame / TCSF _block . In one embodiment, the resulting numerical value of the QP _block is range limited (clipped) so as not to exceed a predetermined upper limit QP value or lower limit QP value for the frame: QP _min ≦ QP _block ≦ QP _max .

In other embodiments, the output of the SCM can be used to modify the quantization parameter on a block-by-block basis using a rule-based approach. This embodiment starts by assigning a high QP value (low quality) to a block with a large variance. This is because highly complex areas are difficult for HVS to notice. A low QP value (high quality) is assigned to a block with small variance. This is because low complexity areas are easily noticed by HVS. In one embodiment, the QP swing equivalent for a given block is regulated by the upper limit QP value QP _max and the lower limit QP value QP _min of the frame, and relative to other block variances in the frame It is linearly increased or decreased based on its own block distribution. In an alternative embodiment, only blocks having a variance greater than the average variance of the entire _frame are assigned a QP value between QP _frame and QP _max , which are _frame QPs, and the equivalent weight is It is linearly increased or decreased so that QP _block = ((var _block −var _frame / var _block )) × (QP _max −QP _frame ) + QP _frame . In this alternative embodiment, the QP swing equivalent for highly dispersed blocks may be further refined by TCSF. For example, if the block is considered foreground data in TMVM and the contrast sensitivity logarithm of TCSF (vertical axis in FIG. 5) is less than 0.5 (meaning that the block is not temporally important) The QP _block is increased by 2. In an alternative embodiment, an edge detection process can be applied and the QP of the block containing the edge is adjusted to QP _min to overwrite the QP from the spatial complexity previously allocated. obtain. The reason is that the edge is very noticeable to HVS. In a further embodiment, the QP of a very bright or very dark block can be readjusted to QP _max by overwriting the QP from previously distributed variance and (possibly) edge detection. The reason is that an extremely dark area or an extremely bright area is difficult for the HVS to notice. This process is known as luminance masking.

In a further embodiment, in addition to the above, the value of QP _max for a highly distributed block may be dynamically determined based on the quality level of the encoded video. The idea is that QP _max should be closer to QP _frame because low quality coding cannot tolerate quality degradation in blocks with high variance, whereas high quality coding is about blocks with high variance to save bits. It is acceptable to increase the QP _max . The quality of the encoding can be updated for each I (intra) frame by calculating the average SSIM of blocks with a variance within ± 5% of the average frame variance, and the higher the SSIM value, the higher It corresponds to the higher numerical value of QP _max . In an alternative embodiment, the average SSIM is adjusted by the average variance of that frame so that the quality indicator is calculated as the product of the average SSIM and the average frame variance.

  In a further embodiment, in addition to the above, for very small blocks of variance (corresponding to flat regions that are particularly visible to HVS), a decision was made to ensure high quality coding in these regions. A low QP value can be allocated. For example, in the case of an I (intra) frame, QP = 28 can be allocated to a block having a variance of 0 to 10, and QP = 30 can be allocated to a block having a variance of 10 to 30, A block with 60 variances can be allocated QP = 32. Then, QP equivalents for the blocks in the P and B frames can be derived from the above QP using the ipratio (ip rate) and pbratio (pb rate) parameters, respectively.

In a further embodiment, in addition to the above, a frame QPQP _frame is allocated to a low variance block (eg, a block having a variance of 60 to average frame variance), and then the low variance block is A check is made to determine if further quality improvement is required. In one embodiment, blockiness artifacts are reconstructed from the current (target) block being encoded and the surrounding blocks that have encoded the spatial complexity and intensity of the original pixel ( For example, it can be detected by comparing with the spatial complexity and luminance of the left, upper left, upper, upper right (if they exist) blocks, etc. Although there are significant differences between the spatial complexity measure and luminance measure of the reconstructed pixel of the target block and the corresponding measure of the neighboring block, the original pixel of the target block and the original of the neighboring block A target block is considered to be “blocky” if there is no such difference in spatial complexity and brightness from that pixel. In this case, the QP value of the block is reduced so as to improve the coding quality of the block (for example, reduced by 2). In other embodiments, the estimated quality of the target block averages the SSIM and QP values of the encoded surrounding blocks (eg, the left, upper left, right, upper right blocks, etc.). It is calculated by doing. The average QP value QP _avg is used as the estimated QPQP _block for the target block. If the average SSIM value SSIM _est is less than 0.9, QP _block = QP _avg is reduced by 2 to improve its quality. In a further embodiment, the target block identified as foreground data by TMVM has a TCSF contrast sensitivity logarithm value (vertical axis in FIG. 5) greater than 0.8 (that block is temporally significant). QP _block is decreased by 2 only if

  The methods described so far may use temporal importance maps (TCSF with or without TMVM refinement), spatial importance maps, or both. If both a temporal importance map and a spatial importance map are used, the result is referred to as an integrated importance map.

  The importance map generated from the perceptual statistic as described above can be applied to any video compression frame that generates motion vectors using motion compensation. Both rate distortion analysis and quantization are improved to produce visually better coding with the same coding size. The application of the importance map to the video compression does not require any special application when applied to the continuous block tracker (CBT) already described in detail. Moreover, the importance map is more effective in a CBT-based coding framework because CBT provides the additional ability to accurately determine which motion vector is a true motion vector. Specifically, the CBT frame-to-frame motion vector (from frame-to-frame tracking 210 in FIG. 8A) is generated from the original frame of the video and is reconstructed. It is a point that was not generated. The frame store 85 of FIGS. 2 and 7 for a typical encoder includes the reconstructed frame generated from the encoding process, while the frame store 205 of FIGS. Includes picture frames. Therefore, the CBT frame-to-frame tracking (reference numeral 210 in FIGS. 3, 4 and 8) can better track the true motion of the video, and the frame-to-frame motion vector is Generate a more accurate true motion vector map. In contrast, typical encoder motion vectors have been selected to optimize rate distortion (compression) performance and may not reflect the true motion of the video.

  The generated importance map can also be applied to intra prediction frames by modifying the rate distortion optimization between intra prediction modes or modifying block level quantization according to the method described above. obtain. However, in the case of an all-intra encoder (all-intra-encoder), in calculating the TCSF, separate encoding means (for example, FIG. 8A) for generating a motion vector for each data block in the video frame. Frame-to-frame tracking 210).

<Digital processing environment>
An exemplary implementation of the invention may be feasible in a software environment, a firmware environment, or a hardware environment. FIG. 9A shows one such environment. At least one client computer / device 950 (e.g., cell phone, computing device, etc.) and cloud 960 (or server computer or cluster of server computers) have processing, execution, storage, encoding, and decoding capabilities for executing application programs. Provide the function and I / O device.

  Also, at least one client computer / device 950 may be connectable with other computing devices including other client devices / processes 950 and at least one server computer 960 via a communication network 970. The communication network 970 can be part of a remote access network, part of a global network (eg, the Internet, etc.), part of a worldwide collection of computers, part of a local area network, part of a wide area network, or Currently, it may be part of a gateway that communicates with each other using respective protocols (TCP / IP, Bluetooth (registered trademark), etc.). Other electronic device / computer network architectures can also be used.

  Embodiments of the present invention may include means for encoding, tracking, modeling, filtering, adjusting, decoding or displaying video or data signal information. FIG. 9B illustrates a given computer / computing node in the processing environment of FIG. 9A (eg, client processor / device / cell phone device /) that can be used to facilitate the encoding of such video or data signal information. 2 is a diagram of the internal structure of a tablet 950, a server computer 960, and the like. Each computer 950, 960 includes a system bus 979, which is a set of real or virtual hardware lines used to transfer data between components of the computer or processing system. The bus 979 is like a common pipe that connects different components (for example, a processor, an encoder chip, a decoder chip, a disk storage, a memory, an input / output port, etc.) of the computer system. Exchange of data between them. An input / output device interface 982 for connecting various input / output devices (eg, keyboard, mouse, display, printer, speaker, etc.) to the computers 950 and 960 is attached to the system bus 979. The network interface 986 allows the computer to connect to various other devices attached to the network (eg, the network indicated by reference numeral 970 in FIG. 9A). Memory 990 is a volatile memory that stores computer software instructions 992 and data 994 used to implement the software implementation of the present invention.

  Disk storage 995 is a non-volatile storage that stores computer software instructions 998 (equivalently “OS programs”) and data 994 used to implement one embodiment of the present invention. The disk storage 995 can also be used for long-term storage of video in a compressed format. Also attached to the system bus 979 is a central processing unit 984 that executes computer instructions. Throughout this specification, “computer software instructions” and “OS programs” are equivalent to each other.

  As an example, the encoder may be configured with computer readable instructions 992 for encoding video data using an importance map formed from temporal and spatial information. These importance maps may be configured to provide a feedback loop to the encoder (or encoder components) to optimize the encoding / decoding of the video data.

  In one embodiment, processor routine 992 and data 994 are computer program products that comprise an encoder (generally indicated by reference numeral 992). Such a computer program product includes a computer readable medium that can be stored in the storage device 994 that provides at least a portion of the software instructions for the encoder.

  Computer program product 992 may be installable by any suitable software installation method known in the art. In other embodiments, at least some of the software instructions of the encoder may be downloadable via cable and / or communication and / or wireless connection. In other embodiments, the encoder system software is a computer program propagated signal product 907 (FIG. 9A) embedded in a non-transient computer readable medium that is executed when executed. It can be realized as a propagation signal on a propagation medium (for example, a radio wave, an infrared wave, a laser wave, a sound wave, an electric wave propagated by a global network such as the Internet or at least one other network). Such a carrier medium or carrier signal provides at least part of the software instructions for the routine / program 992 according to the invention.

  In an alternative embodiment, the propagation signal is an analog carrier wave or digital signal carried by a propagation medium. For example, the propagated signal can be a digital signal carried by a global network (eg, the Internet, etc.), a telecommunications network, or other network. In one embodiment, the propagated signal is transmitted by a propagation medium for a given period of time, for example, transmitted in packets by the network for a period of milliseconds, seconds, minutes or longer. And instructions for software applications. In another embodiment, the computer readable medium of computer program product 992 is a propagation medium that can be received and read by computer system 950. For example, the computer system 950 receives a propagation medium and identifies a propagation signal embedded in the propagation medium, as in the computer program propagation signal product described above.

  Although the present invention has been particularly shown and described with reference to exemplary embodiments, those skilled in the art will recognize that the form and details fall within the scope of the invention as encompassed by the appended claims. You will understand that various changes can be made.

Claims (34)

A method of encoding a plurality of video frames,
The video frames have target blocks that do not overlap each other;
The method is
Encoding the plurality of video frames using the importance map such that the importance map affects the encoding quality of each target block to be encoded in each video frame by adjusting quantization ,
The importance map comprises:
Setting up the importance map using temporal and spatial information; and
(I) In blocks where the importance map has a high numerical value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that these blocks have high quality. And (ii) in the target block in which the importance map has a low value, the block quantization parameter is set larger than the frame quantization parameter QP _frame , so that these blocks have low quality. And causing the importance map to indicate which part of the video frame among the plurality of video frames is most easily noticed by human perception by calculation;
Consists of, the method. The method of claim 1, wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which target blocks within the frame are averaged within the frame. Determining whether to have a variance greater than the block variance var _frame ;
A block having a variance larger than the average block variance var _frame is assigned a quantization parameter (QP) value higher than the frame quantization parameter QP _frame , and a block equivalent QP of the block quantization parameter (QP) is assigned. _{The block} is linearly increased or decreased between the frame quantization parameter QP _frame and the quantization parameter upper limit QP _max according to how much the block variance var _block is larger than the average block variance var _frame . The method of claim 1, wherein the temporal information is
A temporal contrast sensitivity function (TCSF) that indicates which target block is most noticeable in time for the observer human, and
True motion vector map (TMVM) showing which target blocks correspond to foreground data
Provided that the TCSF is only valid for target blocks identified as foreground data. 3. The method according to claim 2, wherein a block having a large variance has a block quantization parameter (QP) of the shaking equivalent QP _block , the TMVM identifies a target block as foreground data, and the block of the TCSF. The method is further refined by the TCSF and the TMVM such that the shaking equivalent QP _block is increased by 2 when the contrast sensitivity logarithm of is less than 0.5. 3. The method of claim 2, wherein the SCM further comprises the vibration equivalent QP _block that is a block quantization parameter of a very bright (greater than 170 brightness) or very dark (less than 60 brightness) target block QP _max. A method comprising luminance masking re-adjusted. 3. The method of claim 2, wherein the SCM further includes a dynamic determination of the quantization parameter upper limit QP _max based on a quality level of the encoded video.
In this dynamic decision, the quality is measured using the average structural similarity (SSIM) calculation result of the target blocks in an intra (I) _frame together with the average block variance var _frame of these frames,
The method, wherein if the measured quality is low, the value of the quantization parameter upper limit QP _max is reduced to approach the frame quantization parameter QP _frame . 3. The method according to claim 2, wherein for a block with extremely small variance, in order to ensure high quality coding in these regions, the smaller the block variance, the lower the value of the vibration equivalent QP _block. (And so that the quality is high), the shaking equivalent QP _block, which is a value of a determined low quantization parameter (QP), is allocated. 8. The method of claim 7, wherein the equivalent QP _block , which is the value of the low quantization parameter (QP) for a very small variance _block, is first determined for an I frame and then for P and B frames. Is determined using the ipratio and pbratio parameters. The method according to claim 7, wherein a block whose variance is small but which is not considered to be extremely small is used to determine whether or not quality improvement is necessary for the block.
The shaking equivalent QP _block, which is the initial estimate of the block quantization parameter (QP), is the value of the quantization parameter (QP) of the already-encoded neighboring block on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from the SSIM values of the already encoded neighboring blocks at the left, upper left, right and upper right of the current block; and
When the SSIM _est is less than 0.9, the numerical value of the shaking equivalent QP _block is decreased by 2,
Examine the method.   10. The method of claim 9, wherein the quality enhancement is applied only to blocks identified as foreground data by the TMVM and having a contrast sensitivity logarithm value of the TCSF greater than 0.8.   4. The method according to claim 3, wherein the temporal frequency of the TCSF is obtained by approximating the wavelength using SSIM in a color space region between the target block and its reference block, and the magnitude and frame rate of the motion vector. And a method of calculating an approximation of speed using   4. The method of claim 3, wherein the TCSF is multiple such that the TCSF for a current frame is a weighted average of the TCSF maps in a recent frame and that more recent frames receive a greater weight. Calculated over a number of frames.   4. The method of claim 3, wherein the TMVM is set to 1 only for foreground data.   14. The method according to claim 13, wherein foreground data is calculated by calculating a difference between an encoder motion vector for a given target block and a global motion vector for the block, and a block having a sufficiently large difference is foreground data. The method specified by being judged.   15. The method according to claim 14, wherein a difference motion vector is obtained by subtracting the encoder motion vector from the global motion vector for a data block identified as foreground data, and the magnitude of the difference motion vector is the value of the difference motion vector. A method used to calculate the temporal frequency of a TCSF.   4. The method of claim 3, wherein the TCSF is calculated from a motion vector from an encoder.   The method according to claim 1, wherein when the importance map is set by the temporal information and the spatial information, the importance map is an integrated importance map. A system for encoding video data,
A codec that encodes a plurality of video frames using an importance map, wherein the video frames have target blocks that do not overlap each other;
The importance map is configured to influence the encoding quality of each target block to be encoded in each video frame by adjusting quantization,
The importance map is:
The importance map is set using temporal information and spatial information, and the importance map set by these temporal information and spatial information is an integrated heavy element map. As well as
(I) In blocks where the importance map has a high numerical value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that these blocks have high quality. And (ii) in the target block in which the importance map has a low value, the block quantization parameter is set larger than the frame quantization parameter QP _frame , so that these blocks have low quality. And, by calculating, let the importance level map indicate a part of the video frame that is most easily noticed by human perception of the video frame;
The system that is configured by. 19. The encoder of claim 18, wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which target blocks within the frame are averaged within the frame. Determining whether to have a variance greater than the block variance var _frame ;
A block having a variance larger than the average block variance var _frame is assigned a quantization parameter (QP) value higher than the frame quantization parameter QP _frame , and a block equivalent QP of the block quantization parameter (QP) is assigned. _The encoder is linearly increased or decreased between the frame quantization parameter QP _frame and the quantization parameter upper limit QP _max according to how much the block variance var _block is larger than the average block variance var _frame . The encoder according to claim 18, wherein the temporal information is
A temporal contrast sensitivity function (TCSF) that indicates which target block is most noticeable in time for the observer human, and
True motion vector map (TMVM) showing which target blocks correspond to foreground data
Provided that the TCSF is only valid for target blocks identified as foreground data. 20. The encoder according to claim 19, wherein a block having a large variance has a block quantization parameter (QP) of the shaking equivalent QP _block , the TMVM specifies a target block as foreground data, and the block of the TCSF. The encoder is further refined by the TCSF and the TMVM so that the vibration equivalent QP _block is increased by 2 when the contrast sensitivity logarithm of is less than 0.5. 20. The encoder according to claim 19, wherein the SCM is further the block equivalent parameter QP _{block of} a very bright (greater than 170 brightness) or very dark (less than 60 brightness) target block QP _max Encoder, including brightness masking re-adjusted. The encoder of claim 19, wherein the SCM further comprises a dynamic determination of QP _max based on the quantization parameter upper limit on the quality level of the encoded video.
In this dynamic decision, the quality is measured using the average structural similarity (SSIM) calculation result of the target blocks in an intra (I) _frame together with the average block variance var _frame of these frames,
If the measured quality is low, the numerical value of the quantization parameter upper limit QP _max is reduced so as to approach the frame quantization parameter QP _frame . The encoder according to claim 19, wherein the block equivalent QP _block has a lower numerical value as the block variance is smaller, in order to ensure high quality coding in these regions for blocks with extremely small variance. (And so that the quality is high), the predetermined equivalent quantization parameter QP _block, which is a value of a low quantization parameter (QP), is allocated. 25. The encoder according to claim 24, wherein the vibration equivalent QP _block , which is the value of the low quantization parameter (QP) for a very small variance _block, is first determined for an I frame, and then P frames and B An encoder, which is determined using ipratio and pbratio parameters for a frame. The system according to claim 19, wherein a block whose variance is small but which is not considered to be extremely small is to determine whether a quality improvement is necessary for the block.
The shaking equivalent QP _block, which is the initial estimate of the block quantization parameter (QP), is the value of the quantization parameter (QP) of the already-encoded neighboring block on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from the SSIM values of the already encoded neighboring blocks at the left, upper left, right and upper right of the current block; and
When the SSIM _est is less than 0.9, the numerical value of the shaking equivalent QP _block is decreased by 2,
The system being examined.   27. The system of claim 26, wherein the quality enhancement is applied only to blocks identified as foreground data by the TMVM and having a contrast sensitivity logarithm value of the TCSF greater than 0.8.   21. The system according to claim 20, wherein the temporal frequency of the TCSF is obtained by approximating a wavelength using SSIM in a color space region between the target block and its reference block, and a motion vector magnitude and a frame rate. The system is calculated by calculating the approximation of speed using   21. The system of claim 20, wherein the TCSF is multiple such that the TCSF for a current frame is a weighted average of the TCSF maps in a recent frame and a more recent frame receives a greater weight. The system is calculated over a number of frames.   21. The system of claim 20, wherein the TMVM is set to 1 only for foreground data.   The system according to claim 30, wherein the foreground data is calculated by calculating a difference between an encoder motion vector for a given target block and a global motion vector for the block, and a block having a sufficiently large difference is foreground data. A system identified by being judged.   The system according to claim 20, wherein a difference motion vector is obtained by subtracting the encoder motion vector from the global motion vector for a data block identified as foreground data, and the magnitude of the difference motion vector is the size of the difference motion vector. A system used to calculate the temporal frequency of TCSF.   21. The system of claim 20, wherein the TCSF is calculated from a motion vector from the encoder.   The system according to claim 18, wherein the importance map is an integrated importance map when the importance map is set by the temporal information and the spatial information.