JP6698077B2 - Perceptual optimization for model-based video coding - Google Patents

Description

Related application

  This application claims the benefit of US Provisional Patent Application No. 62/158,523 filed May 7, 2015 and US Provisional Patent Application No. 62/078,181 filed November 11, 2014. This application is also a continuation-in-part (CIP) of US patent application Ser. No. 14/532,947 filed Nov. 4, 2014. This U.S. Patent Application No. 14/532,947 claims the benefit of U.S. Provisional Patent Application No. 61/950,784 filed March 10, 2014 and U.S. Provisional Patent Application No. 62/049,342 filed September 11, 2014. To do. The entire teachings of these referenced patent applications are incorporated herein by reference.

  Video compression is considered to be the process of representing digital video data in a format that uses a small number of bits during storage or transmission. Video encoding can achieve compression by exploiting spatial redundancy, temporal redundancy, or color space redundancy of video data. Typically, the video compression process divides the video data into parts such as a group of frames and a group of pels, identifies a redundant part in the video, and obtains the redundant part from the original video data. It can be expressed with fewer bits than required. Greater compression can be achieved by exploiting this redundancy of data. An encoder may be used to convert the video data into a coding format. Then, by using the decoder, the coded video can be converted into a form comparable to the original video data. What realizes an encoder/decoder is called a codec.

  Upon encoding, a standard encoder (standard encoder) divides a given video frame into a plurality of non-overlapping coding units, that is, macroblocks (rectangular regions composed of a plurality of consecutive pels). Typically, macroblocks (more generically referred to herein as “input blocks” or “data blocks”) are processed in left-to-right or top-to-bottom scan order of a video frame. To be done. The compression can be achieved by predicting/encoding the input block using encoded data. The process of coding an input block using previously coded blocks of the same frame that are spatially adjacent to the input block is called intra prediction. Intra-prediction seeks to exploit spatial redundancy in the data. Encoding the input block with similar regions from the previously encoded frame found using the motion estimation process is called inter prediction. Inter-prediction seeks to take advantage of temporal redundancy in the data. The motion estimation process may generate motion vectors. The motion vector specifies, for example, the position of the matching region in the reference frame with respect to the input block being encoded. Most motion estimation processes use a motion initial estimate (the initial motion estimate) that provides the initial coarse estimate (and corresponding temporal prediction) of the motion vector for a given input block, and in the neighborhood of this initial estimate. It consists of two main steps, a motion fine estimation (fine motion estimation) that determines a more accurate estimation (and corresponding prediction) of the motion vector for that input block by performing a local search.

  The encoder may generate a residual by measuring the difference between the data to be encoded and the prediction (prediction result). This residual may provide the difference between the predicted block and the original input block. These predictions, motion vectors (for inter prediction), residuals and related data are combined with other processes such as spatial transformation, quantization, entropy coding, loop filter, etc. to efficiently code the video data ( Encoding) can be generated. The quantized and transformed residuals are processed and reassembled into the prediction to be assembled into decoded frames and stored in the frame store. The details of such a video coding technique are well known to those skilled in the art.

  MPEG-2 (H.262) and H.264 (MPEG-4 Part 10 Advanced Video Coding (AVC)) are two codec standards for video compression that achieve high quality video representation at relatively low bit rates. (Hereinafter referred to as MPEG-2 and H.264, respectively). MPEG-2 and H.264. The H.264 coding basic unit is a 16x16 macroblock. H. H.264 is a widely used recent video compression standard and is generally considered to have twice the efficiency of MPEG-2 in compressing video data.

  The basic MPEG standard defines three types of frames (or pictures) based on how the input blocks within a frame are encoded. Since an I frame (intra coded picture) is coded using only the data existing in that frame, it is composed of only intra prediction blocks. P frames (predicted pictures) are encoded by forward prediction using data from predecoded I frames or P frames (also referred to as reference frames). P-frames may include both intra blocks and (forward) predicted blocks. B-frames (bi-predictive pictures) are encoded by bi-directional prediction using data from both the previous frame and the subsequent frame. B-frames may include both intra blocks, (forward) predictive blocks and bi-predictive blocks.

  A particular set of reference frames is called a Group of Pictures (GOP). The GOP contains only the decoded pels in each reference frame, and information about how the input block or frame was encoded (I frame, B frame, or P frame). Does not include. Older video compression standards such as MPEG-2 utilize one reference frame (previous frame) to predict a P frame and two reference frames (previous frame and 1 frame) to predict a B frame. Frame after one) is used. In contrast, H. Newer compression standards such as H.264, HEVC (High Efficacy Video Coding) allow the use of multiple reference frames for P-frame and B-frame prediction. A typical reference frame is a frame that is temporally adjacent to the current frame, but these standards also allow a frame that is not temporally adjacent to be a reference frame.

  Conventional inter-prediction is based on block-based motion estimation and compensation (BBMEC). The BBMEC process searches for the best match between the target block (the current input block being encoded) and a region of the same size in the previously decoded reference frame. Upon finding such a match, the encoder may send a motion vector that acts as a pointer to the location of this best match within the reference frame. However, the BBMEC search process is not only temporally limited from the viewpoint of the reference frame that can be the search target, but also spatially limited from the viewpoint of the neighborhood region that can be the search target, for computational reasons. ing. This means that the "best possible" match is not always found, and especially for fast changing data.

  The simplest form of the BBMEC process sets the motion estimation initialization to a (0,0) motion vector. This means that the initial estimate of the target block is the co-located block in the reference frame. Next, motion fine estimation is performed by searching for a region in the local neighborhood of this region that best matches the target block (ie, has the smallest error for the target block). This local search may be done by exhaustively querying its local neighborhood, or by using any of several "fast search" methods such as diamond search, hexagon search, etc. ..

  An enhanced predictive zone search (EPZS) method (EPZS) method as an improvement of the BBMEC process provided in the standard codecs of the later version of MPEG-2 and later (Non-patent document 1: Tourapis et al. "Enhanced predictive zonal search for single and multiple frame motion estimation"). This EPZS method calculates a set of motion vector candidates based on the motion vector of an already coded neighboring block and the motion vector of a block (and a neighboring) at the same position in the previous reference frame for the purpose of initial estimation of the target block. consider. The EPZS method considers the motion vector field of the video to have some spatial and temporal redundancy, so the initialization of the motion estimation for the target block is the motion vector of the neighboring block or in the coded frame. Suppose it is reasonable to use motion vectors from neighboring blocks. The EPZS method narrows down the set by the approximate rate distortion analysis when the set of the first estimations is collected. After this, motion precision estimation is performed.

  For any given target block, the encoder may generate multiple inter predictions that are candidates for selection. These predictions may result from multiple prediction processes (eg, BBMEC, EPZS, model-based, etc.). Also, these predictions may differ based on sub-partitioning of the target block. In the sub-segment processing, different motion vectors are associated with different sub-segments of the target block, and each motion vector points to an area of sub-segment size in the reference frame. Also, these predictions may differ based on the reference frame pointed to by the motion vector. This is because, as mentioned above, modern compression standards allow the use of multiple reference frames. Usually, the best prediction choice for a given target block is achieved by rate distortion optimization. In rate-distortion optimization, the best prediction is the rate-distortion measure D+λR, where the distortion D is the error between the target block and the prediction, and the rate R quantifies the cost (in bits) of coding the prediction. , Λ are scalar weighting factors).

Tourapis, A., 2002, "Enhanced predictive zonal search for single and multiple frame motion estimation," Proc. SPIE 4671, Visual Communications and Image Processing, pp. 1069-1078

  In the past, a number of model-based compression schemes have been proposed with the aim of avoiding the limitations of BBMEC prediction. Such a model-based compression scheme (of which the MPEG-4 Part 2 standard is perhaps best known as a scheme of this kind) is based on the object and feature (generally defined as "component of interest") in the video. It utilizes detection and tracking, as well as a method of encoding these features/objects separately from the rest of the video frame. Since the feature/object detection/tracking is done independently of the spatial search in the standard motion estimation process, the feature/object track results in a different set of predictions than that obtained by standard motion estimation. obtain.

  However, such model-based compression schemes based on features/objects face the problem of dividing the video frame into object regions and non-object regions (or feature regions and non-feature regions). First, since the size of an object can vary widely, it is necessary to encode not only the texture (color content) of the object, but also the shape of the object. Secondly, tracking multiple objects with motion can be difficult, and inaccurate tracking can lead to inaccurate segmentation, usually leading to poor compression performance. The third problem is that not all video content is composed of objects or features, so if there are no objects/features, an alternative encoding scheme is needed.

  Co-pending U.S. Provisional Patent Application No. 61/950,784 filed November 4, 2014 (referred to herein as the "'784 application") presents a model-based compression scheme that avoids the segmentation problem described above. ing. The Continuous Block Tracker (CBT) of the '784 application does not detect objects or features and eliminates the need to split an object or feature from a non-object/non-feature background. Rather, the CBT tracks all input blocks ("macroblocks") in a video frame as if it were a region of interest by incorporating frame-to-frame motion estimation into a continuous track. By doing this, the CBT models motion in the video to enjoy the benefits of higher order modeling of data to improve inter prediction while avoiding segmentation problems.

  Another model-based compression approach is to model the response of the human visual system (HVS) to the content of video data as an importance map that indicates which parts of the video frame are most noticeable to human perception. Be done. The importance map takes a numerical value for each input block or data block in the video frame. Also, the importance map value (value of the importance map) for any given block may change from frame to frame through the video. Generally, importance maps are defined such that higher numbers indicate more important blocks of data.

  As a kind of importance map, temporal contrast sensitivity function (TCSF) (de Lange, H., 1954, "Relationship between critical flicker frequency and a set of low frequency characteristics of the eye Relationship with the set), "J. Opt. Soc. Am., 44:380-389). TCSF temporally measures the response of HVS to periodic stimuli and reveals that certain temporal characteristics in the data are noticeable to human observers. These temporal characteristics are associated with movements in the data, and the TCSF has a "moderate" degree in which the most noticeable type of movement in the data does not fall into either the very high temporal frequencies or the very low temporal frequencies. Predict that it is a movement.

  An important point to note is that TCSF requires an accurate measurement of the speed of moving content in the video to produce an accurate temporal contrast value. Such velocities can be approximated by calculating the optical flow, which represents the net (obvious) movement of the video content due to camera movement and/or object movement. However, most standard video encoders employ a motion estimation process that optimizes compression efficiency rather than accurately calculating optical flow.

  Another type of importance map is based on spatial contrast sensitivity, which measures the response of HVS to spatial characteristics such as brightness, edges, spatial frequencies, and colors. Spatial Contrast Sensitivity Function (SCSF) (eg Barten, P., 1999, Contrast Sensitivity of the Human Eye and Its Effects on Image Quality), SPIE Press, etc. Also known as the Contrast Sensitivity Function (CSF), measures the spatial contrast that is significant to HVS. SCSF has been successfully applied in the JPEG2000 image compression standard for the purpose of reducing image compression artifacts. Objects and features are also typically detected with the aid of spatial contrast techniques (eg, the presence of edges indicated by the spatial frequency gradient). Spatial contrast sensitivity has been studied and used in image compression (eg, JPEG2000 codec, etc.), and although many video compression processes based on object/feature detection have been proposed, the time represented by TCSF. Contrast contrast sensitivity has never been applied to video compression.

  Some disclosed inventive embodiments apply importance maps to video compression so as to improve the quality of video coding. In one exemplary embodiment, the temporal frequency within a standard video encoding process stream is determined by approximating the wavelength using structural similarity (SSIM) in the color space domain, and the motion of the encoder. It is calculated by finding an approximation of velocity using a vector (encoder motion vector). The temporal frequency then serves as an input to the temporal contrast sensitivity function (TCSF). The TCSF may be calculated for all data blocks, which may generate a temporal importance map that indicates which areas of the video frame are most noticeable to the human observer.

  In a further exemplary embodiment, information about the relative quality of motion vectors generated by the encoder may be calculated at various points in the encoding process and used to generate a true motion vector map. The true motion vector map outputs for each target block how reliable that motion vector is. This true motion vector map, which takes a numeric value of 0 or 1, refines the TCSF so that it is not applied to target blocks for which the motion vector is not accurate (ie target blocks where the true motion vector map is 0). It can be used as a mask.

  In a further embodiment, a spatial complexity map (SCM) measures block variance (intra-block variance), block intensity, edge detection, etc. so as to determine the spatial contrast of a given target block with respect to its neighbors. Can be calculated from In other embodiments, the information from the SCM can be combined with the TCSF to obtain a composite integrated importance map. The combination of spatial and temporal contrast information in this integrated importance map effectively balances both aspects of human visual response.

  In one exemplary embodiment, an integrated importance map (an importance map containing information from both TCSF and SCM) is used to weight the distorted portion of the standard rate distortion measure, D+λR. Used. This gives a solution that matches the perceptual relative importance of each target block (low distortion solution when the importance map is close to its maximum value, low rate solution when the importance map is close to its minimum value). And a modified, rate-distortion optimization is obtained. In an alternative embodiment, TCSF or SCM may be used independently for the above purposes.

  In other exemplary embodiments, TCSF and SCM (with refinement by true motion vectors) may be used to adjust the block level quantization of the encoder. In the target block where the importance map has a high numerical value, the quantization parameter is made smaller than the frame quantization parameter, so that high quality can be obtained for these blocks. In the target block where the importance map has a low numerical value, the quantization parameter is made larger than the frame quantization parameter, so that low quality is obtained for these blocks. In an alternative embodiment, TCSF or SCM may be used independently for the above purposes.

  TCSF can be calculated for any encoder that incorporates inter-prediction and produces motion vectors (used by TCSF to approximate the velocity of content in the video). Applying TCSF to video compression is most effective in model-based compression frameworks, such as the Continuous Block Tracker (CBT) of the '784 application, which can accurately determine which motion vector is the true motion vector. Become. As mentioned above, most standard video encoders calculate motion vectors that optimize compression efficiency rather than reflect true motion. In contrast, CBT provides both motion vectors suitable for high compression efficiency and modeling information that maximizes the effect of TCSF.

  Some exemplary inventive embodiments relate to any video compression standard in which the resulting bitstream uses block-based motion estimation and is followed by transformation, quantization and entropy coding of the residual signal. Is built to be compliant with. Such video compression standards include MPEG-2, H.264. H.264 and HEVC, but is not limited to these. The present invention can be applied to a non-standard video encoder that is not block-based as long as it incorporates inter prediction and generates a motion vector.

  Some example embodiments may include a method and system for encoding video data, and any codecs (encoders and decoders) for achieving this. A plurality of video frames may be processed by the encoder, the video frames having target blocks that do not overlap each other. For the plurality of video frames, the importance map is used to change the coding quality of each target block to be coded in each video frame by modifying (adjusting) the quantization. , Can be encoded by the encoder.

The importance map may be configured using at least one of temporal information and spatial information. If both temporal and spatial information are used, the importance map is considered to be an integrated importance map. The importance map may be set to indicate (specify or represent) a part of a certain video frame of the plurality of video frames that is most noticeable to human perception. Specifically, in the block in which the importance map has a high numerical value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that high quality is obtained for these blocks. Be done. Further, in the target block in which the importance map has a low numerical value, the block quantization parameter is made larger than the frame quantization parameter QP _frame , so that low quality is obtained for these blocks.

The spatial information may be provided by a rule-based spatial complexity map (SCM), the first step of which is to say which target block in the _frame has a variance greater than the average block variance var _{frame in the frame.} Is to decide. A QP value higher than the frame quantization parameter QP _frame may be assigned to a block having a variance larger than the average block variance var _{frame, and} the allocation amount QP _block of the block QP is the block variance var _block. Is linearly scaled between the QP _frame and the quantization parameter upper bound QP _max depending on how large is the var _frame .

  Preferably, the temporal information indicates a temporal contrast sensitivity function (TCSF) indicating which target block is most noticeable in time to an observer human, and which target block corresponds to the foreground data. It can be provided by the true motion vector map (TMVM) shown. The TCSF can be valid only for the target block specified as the foreground data.

A high-variance block has a QP allocation amount QP _{block of} that _{block in} which the TMVM specifies the target block as foreground data and the TCSF has a contrast sensitivity logarithmic value of less than 0.5. , The QP _block is increased by 2, and can be further refined by the TCSF and the TMVM.

The SCM may include intensity masking in which the block quantization parameter QP _block of a very bright (greater than 170 intensity) or very dark (less than 60 intensity) target _block is readjusted to QP _max . The SCM may include a dynamic determination of QP _max based on a quality level of the coded video, the dynamic determination including the average structural similarity (SSIM) of target blocks in an intra (I) frame. ) The calculation result is used together with the average block variance var _frame of these frames to measure the quality, and when the measured quality is low, the numerical value of the quantization parameter upper limit QP _max approaches the frame quantization parameter QP _frame somewhat. To be reduced.

For very-low-variance blocks, the smaller the block variance, the lower the value of the allocation QP _block to ensure high quality coding in these regions ( , And the allocation amount QP _block, which is a low value of the quantization parameter (QP) determined so that the quality is high, may be allocated. The allocation amount QP _block , which is the value of the low quantization parameter (QP) for a block with a very small variance, is first determined for I frames, and then using the iratio and pbratio parameters for P and B frames. Can be decided. A block that has a small variance but is not considered to have a very small variance is the first estimated value of the block quantization parameter (QP) in order to determine whether or not quality improvement is necessary for the block. The QP _block is examined as calculated by averaging the values of the quantization parameter (QP) of the already coded neighboring blocks to the left, top left, right and top right of the current block. An estimated SSIM _{est of the} SSIM of the current block may be calculated from SSIM values of previously coded neighboring blocks to the left, top left, right and top right of the current block. When the SSIM _est is less than 0.9, the value of the allocation amount QP _block may be decreased by 2.

  In some embodiments, the quality enhancement is applied only to blocks identified by the TMVM as foreground data and having a TCSF contrast sensitivity log value greater than 0.8. The TMVM can be set to 1 only for foreground data.

  In some embodiments, the temporal frequency of the TCSF is determined by wavelength approximation using SSIM in the color space region between the target block and its reference block and the magnitude of the motion vector (motion vector magnitude). S) and the frame rate to obtain an approximation of the velocity.

  The TCSF may be calculated over multiple frames such that the TCSF for the current frame is a weighted average of the TCSF maps in the most recent frames and the more recent frames are given more weight.

  Foreground data may be identified by calculating the difference between the encoder motion vector for a given target block and the global motion vector for that block, and determining that the block with a sufficiently large difference is the foreground data. ..

  A difference motion vector may be obtained by subtracting the encoder motion vector from the global motion vector for the data block identified as the foreground data, and the magnitude of the difference motion vector is the temporal frequency of the TCSF. Is used to calculate

  Computer-based methods for processing video data, codecs (encoders and decoders) for processing video data, and other computer systems and devices for processing video data may embody the foregoing principles of the invention.

  The foregoing will become apparent from the following more detailed description of exemplary embodiments of the invention that is illustrated in the accompanying drawings. In the drawings, like reference numbers refer to like elements/components throughout the different figures. The drawings are not necessarily drawn to scale, but rather focused on illustrating embodiments of the invention.

It is a block diagram which shows the structure of a standard encoder. FIG. 3 is a block diagram showing steps involved in inter prediction in the case of a general encoder. FIG. 6 is a block diagram showing steps involved in initial motion estimation by continuous block tracking. FIG. 6 is a block diagram showing integrated motion estimation with a combination of continuous block tracking and extended prediction area search. It is a plot which shows the recent measurement result (2010) of the temporal contrast sensitivity function by Wooten et al. FIG. 6 is a block diagram showing how to calculate a structural similarity (SSIM) in the CIE1976 Lab color space in the embodiment of the present invention. FIG. 6 is a block diagram illustrating a general application of perceptual statistics for improving the perceptual quality of video coding in an embodiment of the present invention. FIG. 6 is a block diagram showing how perceptual statistics are applied to modify inter prediction by continuous block tracking to improve the perceptual quality of video coding in an embodiment of the present invention. It is a block diagram which shows an example of the process of modifying and encoding block quantization using an importance map. 1 is a schematic diagram of a computer network environment in which each embodiment is deployed. 9B is a block diagram of a computer node in the network of FIG. 9A. FIG.

  The entire teachings of all patent publications, all patent application publications and all publications cited herein are hereby incorporated by reference. In the following, exemplary embodiments of the invention will be described.

  The present invention is applicable to various standard encodings. In the following, the words "conventional" and "standard" (often used with "compression", "codec", "encoding" and "encoder") are MPEG-2, MPEG-, unless otherwise stated. 4, H.I. H.264 or HEVC. An "input block" refers to a coding basic unit of an encoder without loss of generality, and can be often referred to as a "data block" or a "macroblock". The current input block being encoded is referred to as the "target block".

<Video coding and inter prediction by continuous block tracking>
The encoding process may be the conversion of video data into a compressed or encoded format. Similarly, the decompression or decoding process may transform the compressed video into a pre-compressed or raw format. The video compression/decompression process can be implemented as a pair of encoder and decoder, commonly referred to as a codec.

  FIG. 1 is a block diagram of a standard transform-based and motion-compensated encoder. The encoder of FIG. 1 may be implemented in a software environment, a hardware environment or a combination thereof. The encoder may comprise any combination of components. These constituent elements are the motion estimation means 15, the intra prediction means 30, the transform/quantization means 60, the inverse transform/quantization means 70, the in-loop filter 80, the frame store 85 and the entropy coding which are output to the inter prediction means 20. Means 90, but is not so limited. The purpose of the above prediction means (both inter prediction and intra prediction) is that the best prediction for a given input video block 10 (abbreviated as “input block” or “macroblock” or “data block”). Generating signal 40. The prediction signal 40 is subtracted from the input block 10 to generate a prediction residual 50, and the prediction residual 50 undergoes transformation/quantization 60. Then, the quantized coefficient 65 of the residual is passed to the entropy coding means 90, and the entropy coding means 90 codes it into a compressed bit stream. The quantized coefficient 65 is also passed to the inverse transform/quantization means 70, and the resulting signal (approximation of the prediction residual) is recombined into the prediction signal 40, thereby reconstructing the input block 10. Generate signal 75. The reconstructed signal 75 may be passed through an in-loop filter 80, such as a deblocking filter, which (possibly filtered) reconstructed signal becomes part of the frame store 85. The frame store 85 supports prediction of future input blocks. The function of each component of the encoder shown in FIG. 1 is well known to those skilled in the art.

  FIG. 2 shows various steps in the standard inter prediction (reference numeral 30 in FIG. 1). The purpose of inter-prediction is to encode new data using pre-decoded data from the previous frame, taking advantage of temporal redundancy in that data. In inter-prediction, an input block 10 from the frame currently being encoded (also called the target frame) is taken from a region of the same size in a pre-decoded reference frame stored in the frame store 85 of FIG. "is expected. The two-component vector that indicates the (x,y) shift between the position of the input block in the frame being encoded and the position of the matching region in the reference frame is called the motion vector. Thus, the process of motion estimation involves determining the motion vector that best associates the input block to be encoded with its matching region in the reference frame.

  Most inter-prediction processes begin with a motion initial estimation (reference numeral 110 in FIG. 2) that produces one or more coarse estimates of the “good” motion vector 115 for a given input block. This is optionally followed by a motion vector candidate filtering step 120, where multiple motion vector candidates can be reduced to a single candidate using an approximate rate distortion measure. In rate-distortion analysis, the best motion vector candidate (prediction) is the rate-distortion measure D+λR, where the distortion D is the error between the input block and the matching region, and the rate R encodes the prediction. The one that minimizes the cost (in bits) and λ is a scalar weighting factor is chosen. The actual rate cost includes two types of components, texture bits and motion vector bits. Texture bits are the number of bits required to encode the quantized transform coefficients of the residual signal (input block minus prediction), and motion vector bits are required to encode the motion vector. Is the number of bits. Usually, the motion vector is differentially encoded with respect to the already encoded motion vector. The rate portion of the rate distortion measure is approximated by the motion vector bits, since texture bits are not available early in the encoder. On the other hand, the motion vector bit is approximated as a motion vector penalty coefficient that depends on the size of the differential motion vector. Therefore, in the motion vector candidate filtering step 120, this approximate rate distortion measure is used to single out a single "best" starting motion vector or a smaller set of "best" starting motion vectors 125. Then, such initial motion vector 125 is refined by motion fineness estimation 130. The motion fine estimate 130 determines a more accurate estimate of the motion vector (and corresponding prediction) for that input block by performing a local search in the vicinity of each initial estimate. This local search is usually followed by sub-pixel refinement, where integer-valued motion vectors are refined by interpolation to ½ or ¼ pixel precision. The motion fineness estimation block 130 produces a refined set of motion vectors 135.

  Next, given the motion precision vector 135, the mode generation means 140 generates a set of prediction candidates 145 based on the coding mode that can be adopted by the encoder. Such modes depend on the codec. Different coding modes mean interlace-to-progressive (field-to-frame) motion estimation, reference frame direction (forward prediction, backward prediction, bi-prediction), reference frame index (allows multiple reference frames) H.264, for codecs such as HEVC), inter-prediction vs. intra-prediction (some scenarios that allow reverting to intra-prediction if no good inter-prediction exists), different quantization parameters, and input Different subdivisions of blocks (but not limited to). The entire set of prediction candidates 145 undergo a "final" rate distortion analysis 150 to determine a single best candidate. In the "final" rate-distortion analysis, an accurate rate-distortion measure D+λR is used to predict error for the distortion part D (typically calculated as sum of squared errors (SSE)) and the actual coded bits R for the rate part (From entropy coding 90 of FIG. 1) is calculated. The final prediction 160 (ie 40 in FIG. 1) is the prediction with the lowest rate distortion score D+λR among all the candidates, and this final prediction along with its motion vector and other coding parameters at the encoder. Passed on to subsequent steps.

  FIG. 3 shows how initial motion estimation by continuous block tracking (CBT) can be performed during inter prediction. The CBT is useful when there are multiple frame gaps between the target frame and the reference frame from which temporal prediction is derived. For MPEG-2, the typical GOP structure of IBBPBBP (consisting of intra-predicted I-frames, bi-predicted B-frames and forward-predicted P-frames) allows reference frames up to 3 frames separated from the current frame. (The reason is that in MPEG-2 B frames cannot function as reference frames). H.264 that allows multiple reference frames for each frame to be encoded. In H.264 and HEVC, even the same GOP structure as described above enables a reference frame that is separated from the current frame by 6 frames or more. For longer GOP structures (eg, 7 B frames between P frames, etc.), the reference frame can be even further away from the target frame. When multiple frame gaps exist between the current frame and the reference frame, continuous tracking allows the encoder to capture motion in the data that cannot be captured by standard temporal prediction techniques. CBT makes it possible to generate better temporal predictions.

  The first step in CBT is to perform frame-to-frame tracking (reference numeral 210 in FIG. 3). For each input block 10 in a given frame, motion vectors are calculated both backward to the previous frame in the frame buffer 205 and forward to the next frame in the frame buffer. In one embodiment, frame-to-frame tracking operates on frames from the original source video rather than the reconstructed reference frame. This is because the frame-based tracking of the source video is more accurate in representing the true motion field in the video, since the frames of the source video have not been degraded by quantization or other coding artifacts. Frame-to-frame tracking may be performed using conventional block-based motion estimation (BBME) or hierarchical motion estimation (HME).

  The result of the frame-to-frame tracking is the best matching region in the most recent frame in frame buffer 205 for each input block in the frame and the most recent frame in frame buffer 205. A set of frame-to-frame motion vectors 215 that represent, for each block, the best matching region in the current frame. Next, a continuous track 220 aggregates the available frame-to-frame tracking information to generate a continuous track for each input block over multiple reference frames. Details of how to perform continuous tracking are described in the '784 application, which is incorporated herein by reference in its entirety. The output of the continuous track 220 is a continuous block track (CBT) motion vector 225 that tracks all the input blocks in the current frame being coded into regions that match these in past reference frames. For CBT, these CBT motion vectors become the initial motion vector (reference numeral 125 in FIG. 2) and may be refined by motion fine estimation (reference numeral 130 in FIG. 2) as described above.

  FIG. 4 illustrates how CBT can be combined with the EPZS method to create an integrated motion estimation process in one embodiment of the invention. In FIG. 4, after the CBT has generated motion vectors with frame-to-frame tracking 210 and continuous tracking 220 for initial motion estimation 110, local search and sub-pixel refinement 250 for motion fine estimation 130 is continued. .. After the EPZS has generated the initial motion vector by the candidate generation means 230, the candidate filtering means 240 for filtering by the approximate rate distortion analysis as detailed above is continued. This is then followed by a motion refinement estimation 130 with local search and sub-pixel refinement 260. Both the CBT motion vector 255 and the EPZS motion vector 265 thus obtained are used in the remaining inter prediction steps (mode generation 140 and final rate of FIG. 2) to determine the overall “best” inter prediction. Strain analysis 150).

  In an alternative embodiment, further candidates may be added to CBT motion vector candidate 255 and EPZS motion vector candidate 265 of FIG. Such candidates include (but are not limited to) random motion vectors, (0,0) motion vectors, and so-called "median predictors". Motion refinement estimation 130 may be applied to the random motion vector to find the best candidate in its local neighborhood. The (0,0) motion vector is one of the first candidates of EPZS, but it is not always selected at a time point after EPZS candidate filtering (reference numeral 240 in FIG. 4), and the candidate is tentative. Even if the motion vector is selected at the time point after the filtering, there is a possibility that the motion precision estimation 130 outputs a motion vector other than (0,0). Explicit inclusion of the (0,0) motion vector (which is not subject to motion fine estimation) as a candidate for final rate distortion analysis considers at least one small and “small motion” candidate. Make sure that. Similarly, the “median predictor” is also one of the initial candidates for EPZS, but it is not always selected at a time point after EPZS candidate filtering (reference numeral 240 in FIG. 4). The median predictor is defined as the median of the motion vectors precomputed in the data blocks to the left, above and to the right of the data block currently being encoded. Explicit inclusion of a median predictor (which does not undergo motion estimation) as a candidate for final rate distortion analysis encodes a spatially homogeneous (“flat”) region of the video frame. Can be especially beneficial to That is, in this alternative embodiment, there are five or more motion vector candidates (CBT-derived motion vector, EPZS-derived motion vector, random motion vector-derived motion vector, (0,0) motion vector, and median prediction. Children, including but not limited to children, may be passed to the remaining inter prediction steps (mode generation 140 and final rate distortion analysis 150 of FIG. 2).

<Calculation of importance map for video coding>
Perceptual statistics can be used to calculate an importance map that indicates which regions of the video frame are important to the human visual system (HVS).

  One example of perceptual statistics is the so-called temporal contrast sensitivity function (TCSF), which models the response of the human visual system (HVS) to temporally periodic stimuli. As mentioned in the background section, the concept of TCSF has existed since the 1950's (introduced as "time-modulated transfer function"), but has never been applied to video compression. Figure 5 shows the recent measurement results of TCSF (Wooten, B. et al. 2010, "A practical method of measuring the temporal contrast sensitivity function", "Biomedical Optical Express," l(l):47-58) in the form of the logarithm of temporal contrast sensitivity as a function of the logarithm of frequency (logarithmic frequency on the horizontal axis and logarithmic temporal sensitivity on the vertical axis). The measured data points (circles in FIG. 5) are fitted using a cubic polynomial (solid line in FIG. 5). Note that this fitting is used for all TCSF calculations described below. While the human visual system (HVS) has a maximum response in the medium frequency range, the TCSF has a slight decrease in the HVS response in the low frequency range and a sharp decrease in the high frequency range. Anticipate.

  Application of TCSF to video compression requires a method of calculating the temporal frequency (horizontal axis in FIG. 5) that is an input to TCSF. One of the methods according to an embodiment of the present invention for calculating the frequency is described below. The frequency f is given by f=v/λ, where v is the velocity and λ is the wavelength. In one embodiment, the velocity v (in pixels/second) of the content of any data block is determined by the motion vector generated by the encoder (eg, 135 in FIG. 2, 215, 225 in FIG. 3, 225 in FIG. 3). 255, 265, etc., v=|MV|×frame rate/N (where |MV| is the magnitude of the motion vector of the data block, and the frame rate is 1 when the video was generated. Frames per second, N is the number of frames between the reference frame pointed to by the motion vector and the current frame).

  A suitable approximation of the wavelength λ is the Structural Similarity (SSIM) calculated in the CIE 1976 Lab color space (www://en.wikipedia.org/wiki/Lab_color_space) (Wang, Z. et al. 2004, "Image. Quality assessment: From error visibility to structural similarity, "IEEE Trans, on Image Processing, 13(4):600-612". FIG. 6 shows how SSIM is calculated in the Lab color space. The SSIM is calculated between the target block 300 (current data block to be encoded) and the reference block 310 pointed to by its motion vector. Since the video data processed by the encoder is usually represented in a standard space such as YUV420, the next step is to describe both the target block (reference numeral 320) and the reference block (reference numeral 330) generally in the literature. Conversion to the CIE 1976 Lab space using any of the techniques described. Then, the error ΔE (reference numeral 340) between these target blocks and reference blocks in Lab space is

(Where the subscript T means “target block” and the subscript R means “reference block”). Finally, the SSIM 360 between the error ΔE and the zero matrix of the same dimension is calculated as an indication of the color space change of the data. The SSIM determined first takes a numerical value of -1 to 1, and a numerical value of 1 indicates perfect similarity (no spatial difference). It may be possible to use the spatial dissimilarity DSSIM=(1-SSIM)/2, which takes a value from 0 to 1 for the purpose of converting the SSIM into the wavelength λ, where 0 is the short wavelength (maximum space 1) corresponds to a long wavelength (minimum spatial similarity). To convert the SSIM into pixels, it may be possible to multiply the SSIM number by the number of pixels in the block to be calculated. In one embodiment, the SSIM block size is 8x8, so the DSSIM value is multiplied by 64. In this case, the final calculation result of the frequency is
f=|MV|×frame rate/(N×64×(1-SSIM)/2)
Given by.

Once the frequency for a given target block is calculated, the TCSF value for this block may be determinable from the curve fit (solid line) of FIG. TCSF takes values from 0 to 1.08 on the log ₁₀ scale or 1 to 11.97 on the absolute scale. Since different blocks in a frame take different TCSF values, an aggregate set of TCSF values across all blocks in a frame forms an importance map, with high numbers being viewed in terms of temporal contrast. A perceptually important block is indicated and a low number indicates a perceptually unimportant block.

In a further embodiment, the TCSF numbers from recent frames may be averaged for each data block so that the TCSF-based importance map does not fluctuate too much between frames. For example, as one such calculation of the average TCSF TCSF _avg , TCSF _avg =0.7×TCSF _cur +0.3×TCSF _prev , where TCSF _cur is the TCSF value from the current frame, and TCSF _prev is Which is the TCSF value from the most recently encoded past frame). The TCSF calculation becomes more robust by being averaged in this way.

  In a further embodiment, information about the relative quality of the motion vectors generated by the encoder can be calculated at various points in the encoding process and used to generate a true motion vector map (TMVM). A true motion vector map (TMVM) outputs for each data block how reliable that motion vector is. This true motion vector map, which takes a numeric value of 0 or 1, is used as a mask to refine the TCSF so that it will not be applied to data blocks with inaccurate motion vectors (ie, data blocks with a TMVM value of 0). Can be done.

  In one embodiment, the motion vector accuracy is determined by estimating the global motion model for a given video frame and applying the motion model to each data block in the frame to determine the global motion model for each data block. It can be determined by determining a motion vector and then comparing this global motion vector to the encoder motion vector for that data block (encoder motion vector). Global motion can be estimated from an aggregate set of coded motion vectors from that frame, fitted to a 6-parameter or 8-parameter affine motion model. If the global motion vector and the encoder motion vector are the same (or similar) for a given data block, then the encoder motion vector is considered correct (and TMVN=1 for that data block). If the two vectors are not the same, their prediction errors (measured as sum of squared errors (SSE) or sum of absolute differences (SAD)) may be compared. If one error is small and the other is large, the motion vector with the smaller error is used for encoding and is considered accurate (TMVM=1).

  In an alternative embodiment, the magnitude of the difference between the global motion vector and the encoder motion vector for a given data block is that the data block is foreground data, which means that the content within that data block is It is used to identify the rest of the frame (meaning it has different movement than the background). In this embodiment, TMVM is set to 1 and only applies if TCSF is foreground data. In a further embodiment, for a data block identified as foreground data, the encoder motion vector is subtracted from the global motion vector to obtain the difference motion vector, and the magnitude of the difference motion vector (rather than the encoder motion vector) is obtained. Is used to calculate the frequency of the TCSF (replace |MV| with |DMV| (where DMV is the differential motion vector) in the above equation).

In other embodiments, motion vector symmetry may be used to refine the TMVM. Motion vector symmetry (Bartels, C. and de Haan, G. 2009, "Temporal symmetry constraints in block matching," Proc. IEEE 13 ^th Int'l. Symposium on Consumer Electronics , pp. 749-752) is defined as the relative symmetry of a pair of motion vectors that form a pair when the temporal direction of motion estimation is switched, and is a measure of the quality of the calculated motion vector (symmetry). The higher the degree, the better the motion vector quality). The “symmetry error vector” is defined as the difference between the motion vector obtained by the forward motion estimation and the motion vector obtained by the backward motion estimation. Low motion vector symmetry (large symmetry error vector) often means occlusion (hiding or revealing a background object by moving one object in front of another). Is an indicator that there is a complex phenomenon such as the movement of the image on or outside the image frame, the change of illumination, etc. (all of which make it difficult to derive an accurate motion vector).

  In one embodiment, if the magnitude of the symmetry error vector is greater than half the extent of the data block being encoded (eg, for a 16×16 macroblock, the magnitude is greater than the (8,8) vector). ), the degree of symmetry is low (the degree of symmetry is low). In another embodiment, a threshold value based on motion vector statistics in which the magnitude of the symmetric error vector is derived during the tracking process (eg, motion vector magnitude (motion at a given combination of the current frame or recent frames). Vector magnitude) is greater than a standard deviation of motion vector magnitudes), and the degree of symmetry is low (the degree of symmetry is low). In one embodiment, a TMVM value=0 is automatically assigned to a data block whose motion vector has a low symmetry according to the above definition, and the other data blocks are compared with the global motion vector and the encoder motion vector. The TMVM value up to that point derived from is maintained.

  Flat blocks have high spatial contrast sensitivity, but are a well-known aperture problem in calculating motion vectors.

Tend to cause unreliable motion vectors. A flat block is, for example, using an edge detection process (which is determined to be a flat block if no edge is detected in the data block) or by comparing the variance of the data block with a threshold ( Variances below this threshold indicate flat blocks). In one embodiment, block flats may be used to modify the TMVM calculated as described above. For example, a block detected as a flat block may be reassigned TMVM value=0.

  In one embodiment, TMVM may be used as a mask to refine TCSF that is affected by having reliable motion vectors. Since the value of TMVM is 0 or 1, multiplying the TMVM value for a block to the TCSF value for that block on a block-by-block basis has the effect of masking the TCSF. In the case of a block having a TMVM value of 0, the motion vector required to calculate the TCSF becomes unreliable, and therefore the TCSF is made "invalid". For a block with a TMVM value of 1, the TCSF calculation result is considered reliable and any of the techniques described thus far are confidently used.

  In other embodiments, a spatial contrast map may be generated instead of, or in addition to, the temporal contrast map (TCSF described above). In the present invention, a simple measure is used to measure spatial contrast (the opposite is called "spatial complexity"). In one embodiment, block variances measured for both the luma and chrominance components of the data are used to measure the spatial complexity of a given input block. An input block with a large variance is spatially complex and hard to notice for HVS, so its spatial contrast is small.

  In another embodiment, block intensities measured on the intensity component of the data are used to refine the spatial complexity variance measurement. Input blocks with low variance (low spatial complexity, high spatial contrast) but very bright or very dark are automatically considered to have low spatial contrast and were previously measured as high space. To override the dynamic contrast. The reason is that extremely dark areas and extremely bright areas are difficult for HVS to notice. The intensity threshold for classifying a given block as either very bright or very dark is made specific to the occasional application, but typical numbers for 8-bit video are very bright but "Over 170" is extremely dark, but "less than 60".

  In order to form the spatial contrast map (SCM) in which the block variance modified by the block brightness as described above shows the regions where the HVS is easily noticed and the regions where the HVS is not noticeable from the viewpoint of spatial contrast. , Can be calculated for all input blocks of a video frame.

  In one embodiment, the SCM may be combined with TCSF (refined by TMVM) to form a consolidated importance map. This integrated map may be formed, for example, by normalizing both SCM and TCSF accordingly and then multiplying the SCM value for a given block by the TCSF value for that block, block by block. .. In other embodiments, SCM may be used as a replacement for TCSF. In other embodiments, SCM may be used to refine TCSF. For example, in a high complexity block, the SCM value may overwrite the TCSF value for that block, and in a low complexity block, the TCSF value for that block may be used directly.

<Application of importance map to video coding>
The importance map described above can be applied to both the general encoder (FIG. 2) and the CBT encoder (FIG. 3) video encoding process to improve the quality of the encoded bitstream.

  FIG. 7 shows a general application of the importance map to video coding. Input video frame 5 and frame store 85 are used to generate perceptual statistics 390. The perceptual statistics 390 are then applied to form the TCSF and/or SCM importance map 400 (refined by TMVM) as described above. Perceptual statistics 390 may include (but are not limited to) motion vector magnitude, block variance, block intensity, edge detection, and global motion model parameters. The input video frame 5 and the frame store 85 are further input to the coding of the video frame at 450 as usual. The coding includes the usual coding steps (motion estimation 15, inter prediction 20, intra prediction 30, transform/quantization 60 and entropy coding 90 of FIG. 1). However, in FIG. 7, the coding 450 is expanded in function by the importance map 400 in a method described later.

  FIG. 8A shows a concrete application of the importance map for improving the video coding using the CBT. FIG. 8A shows initial motion estimation (reference numeral 110 in FIG. 2) with 210 frame-to-frame tracking and 220 continuous tracking from the CBT. The motion refinement estimation 130 is then applied to the global CBT motion vector 225 in the same motion refinement estimation steps of local search and sub-pixel refinement as previously described (reference numeral 250 in FIG. 4). Again, this is followed by mode generation means 140, which generates a set of prediction candidates 145 based on the coding modes that the encoder can adopt. Similar to FIG. 4, EPZS and other non-model-based candidates (eg, (0,0) motion vectors, median predictors, etc.) can also be generated in parallel as part of the integrated motion estimation framework. (In FIG. 8A, these other candidates are not shown in order to simplify the illustration). Again in FIG. 8A, the entire set of prediction candidates 145, including any coding mode for CBT candidates and possibly any other non-model-based coding mode, determines the single best candidate. Receive a "final" rate distortion analysis 155. In the "final" rate-distortion analysis, an accurate rate-distortion measure D+λR is used to predict error D for the distortion part and the actual coded bits R for the rate part (from the entropy coding 90 of FIG. 1). To calculate. The final prediction 160 (or code 40 in FIG. 1) is passed along with its motion vector and other coding parameters to subsequent steps in the encoder.

  In FIG. 8A, perceptual statistics 390 can be calculated from the motion vectors derived from frame-to-frame motion tracking 210 and then applied to form importance map 400 as described above. Then, these importance maps 400 are input to the final rate distortion analysis 155. Again, the perceptual statistics 390 may include (but are not limited to) motion vector magnitude, block variance, block intensity, edge detection, and global motion model parameters.

  In one embodiment, an importance map is used to modify the rate distortion optimization conditions in response to the importance map. In a standard encoder (see FIG. 2), the entire set of prediction candidates 145 for a given input block 10 undergo a “final” rate distortion analysis 150 to determine the single best candidate. receive. In the "final" rate-distortion analysis, an accurate rate-distortion measure D+λR is used to predict error D for the distortion part and the actual coded bits R for the rate part (from the entropy coding 90 of FIG. 1). To calculate. The candidate with the lowest score on the rate distortion measure D+λR becomes the final prediction 160 for a given input block 10. In one embodiment of the present invention, for the perceptually optimized encoder of FIG. 7 or 8, the significance map IM is calculated at 400 and the final rate distortion analysis 155 is the modified rate. Use the distortion measure D×IM+λR. In this modified rate-distortion measure, the IM value for a given input block is multiplied by the distortion term, and the higher the IM value, the more important the less-distorted solution. This is because a high IM value indicates that the corresponding input block is perceptually important. The importance map may include TCSF (possibly refined by TMVM values), SCM, or a combination thereof.

In a further embodiment, in addition to the above, the distortion D on the rate distortion measure may be calculated as a weighted sum of SSE (sum of squared errors: the “standard” method of calculating distortion) and SSIM calculated in YUV space. . The weighting γ is such that the average SSIM value SSIM _avg in the first some (or most recent some) frames of the video is the first some (or the most recent some) of the video. Can be adaptively calculated to be equal to the average SSE value SSE _avg in the frame of (γ×SSIM _avg =SSE _avg ). That is, for each input block, the modified rate distortion measure is (SSE+γ×SSIM)×IM+2λR, where the multiplication factor 2 before the λR term means that there are two distortion terms. ). The inclusion of SSIM in the distortion measurement further increases the proportion of HVS perception in rate distortion optimization, since SSIM corresponds to the structural information in the data.

In other embodiments, the importance map (eg, TCSF with TMVM refinement, SCM, etc.) may be used in addition to (or instead of) modifying the rate-distortion optimization, in addition to block-level quantization of the encoder. Can be used to modify Quantization controls the relative quality with which a given input block is coded. That is, highly quantized data results in a low quality coded output, and low quantized data results in a high quality coded output. The amount of quantization is controlled by the quantization parameter QP. Standard encoders assign different QP values QP _frame to different frame types, I frames are coded with the lowest QP (highest quality) and B frames are coded with the highest QP (lowest quality). And the P frame is encoded with an intermediate QP (intermediate quality).

That is, the above method uses a significance map to quantize a plurality of video frames (the video frames have target blocks that do not overlap each other) in each video frame. It presents a method of encoding by modifying (and thereby affecting its encoding quality). Such importance maps may be set using temporal information (TCSF with TMVM refinement), spatial information, or a combination of the two (ie, an integrated importance map). Since the importance map indicates which part of each video frame is most noticeable to human perception, the numerical value of the importance map indicates the QP for each target block, and (i) the importance map is high. In the block that takes a numerical value, the block QP is made smaller than the QP _frame so that the quality of these blocks becomes high, and (ii) in the block that the numerical value of the importance map is low, It is preferable to change the block QP so that the block QP has a large quality as compared with the frame quantization parameter QP _frame , so that the quality of these blocks is low.

  FIG. 8B shows an example of a process of modifying the quantization at the time of encoding using the importance map 400. At 400, a significance map may be constructed/formed using temporal and/or spatial information derived from the perceptual statistics 390. The temporal information is, for example, a temporal contrast sensitivity function (TCSF) indicating which target block is most noticeable in time for an observer human being, and a true information indicating which target block corresponds to foreground data. Motion vector map (TMVM), the TCSF is only valid for the target block identified as foreground data. The spatial information may be provided by, for example, a rule-based spatial complexity map (SCM).

The importance map 400 is then used to modify the quantization step 430 in the encoding 450 as described above. In the blocks where the importance map has a high numerical value, the block quantization parameter (QP) is reduced with respect to the frame quantization parameter QP _frame , and high coding quality is obtained for these blocks. In the blocks where the importance map has a low numerical value, the block quantization parameter is increased with respect to the frame quantization parameter QP _frame , and low coding quality is obtained for these blocks. By using the information from the importance map, the quantization can be modified to improve the coding quality of each target block to be coded in each video frame.

In one embodiment, the TCSF map for a given frame may be used to adjust the frame QP block by block. One of the methods for calculating the block QP and QP _block is (Li, Z. T. 2011, "Visual attention guided bit allocation in video compression", J. of image and vision computing, 29(1): 1-14) and associate that adjustment with the entire TCSF map in the frame, the resulting equation being QP _block =(TCSF _frame /(TCSF _block *M))*QP _frame (where TCSF _frame is the sum of TCSF values for all blocks in the _frame , QP _frame is the frame QP, and M is the number of blocks in the frame). given. in a further embodiment, the multiplication factor _{(TCSF frame / (TCSF block ×} M)) _is, so as not to final value of _{QP block} is too smaller or too large for _{QP frame} It can be increased or decreased.

In an alternative embodiment, block-by-block adjustment of QP with a TCSF map may be accomplished without reference to the entire TCSF map for that frame. In this embodiment, the calculation of QP _block is simpler: QP _block =QP _frame /TCSF _block . In one embodiment, the resulting number of QP _blocks is ranged (clip) such that it does not exceed a predetermined upper bound QP value or fall below a lower bound QP value for that frame: QP _min ≤ QP _block ≤. QP _max .

In other embodiments, the output of the SCM may be used to modify the quantization parameters block by block using a rules-based approach. This embodiment begins by first assigning high QP values (low quality) to blocks with large variance. Because highly complex areas are hard to notice for HVS. A low QP value (high quality) is assigned to a block with a small variance. This is because low complexity areas are easy to notice for HVS. In one embodiment, the QP allocation for a given block is constrained by a frame's upper QP value, QP _max, and its lower QP value, QP _min , and relative to other block variances within that frame. It scales linearly based on its block variance. In an alternative embodiment, only blocks having a variance greater than the average variance of the entire _frame are assigned the QP value between the _frame QPs QP _frame and QP _max , and the allocation amount is _{QP block = ((var block -var} frame / var block)) × (QP max -QP frame) + is linearly increased or decreased such that _{QP frame.} In this alternative embodiment, the QP allocation for blocks with high variance may be further refined by TCSF. For example, if the block is considered as foreground data in TMVM and the contrast sensitivity logarithmic value of TCSF (vertical axis in FIG. 5) is less than 0.5 (meaning that the block is not temporally important). QP _block is increased by 2. In an alternative embodiment, an edge detection process may be applied, where the QP of the block containing the edge is adjusted to QP _min to overwrite the QP from the spatial complexity previously allocated. obtain. The reason is that edges are extremely noticeable to HVS. In a further embodiment, the QPs of very bright or extremely dark blocks can be readjusted to QP _max by overwriting the previously allocated variance and (possibly) QP from edge detection. The reason is that extremely dark areas and extremely bright areas are difficult for HVS to notice. This process is known as luminance masking.

In a further embodiment, in addition to the above, the numerical value of QP _max for high variance blocks may be dynamically determined based on the quality level of the coded video. The idea is that it is desirable to bring QP _max closer to QP _frame because lower quality coding cannot tolerate a deterioration of quality in a block with a large variance, while high quality coding allows a block with a large variance to save bits. It is permissible to increase the QP _max of The coding quality can be updated for each I (intra) frame by calculating the average SSIM of blocks having a variance within ±5% of the average frame variance, and the higher the SSIM value, the higher the SSIM value. It is made to correspond to the higher value of QP _max . In an alternative embodiment, the average SSIM is adjusted by the average variance of that frame so that the quality metric is calculated as the product of the average SSIM and the average frame variance.

  In a further embodiment, in addition to the above, for very low variance blocks (corresponding to the flat regions that are particularly visible to HVS), it was decided to ensure high quality coding in these regions. Low QP values can be assigned. For example, for an I (intra) frame, QP=28 may be assigned to a block having a variance of 0-10, QP=30 may be assigned to a block having a variance of 10-30, and 30-. QP=32 may be assigned to blocks with a variance of 60. Then, the QP allocations for the blocks in the P and B frames can be derived from the above QPs using the ipratio (ip rate) and pbratio (pb rate) parameters, respectively.

In a further embodiment, in addition to the above, a frame QPQP _frame is assigned to a block with a small variance (eg, a block having a variance of 60 to the average frame variance), and then the block with a small variance is assigned. It is checked to determine if further quality improvement is needed. In one embodiment, blockiness artifacts are reconstructed from the current (target) block being encoded and the spatial complexity and intensity of the original pixel encoded surrounding block ( For example, it can be detected by comparing the spatial complexity and brightness of the left, upper left, upper, upper right blocks (if they exist, etc.). Even though there is a large difference between the spatial complexity measure and the intensity measure of the reconstructed pixel of the target block and the corresponding measure of the neighboring block, the original pixel of the target block and the original of the neighboring block are A target block is considered to be “blocky” if there is no such difference in spatial complexity and intensity from the pixels of the. In this case, the QP value of the block is reduced (eg, reduced by 2) to improve the coding quality of the block. In another embodiment, the estimated quality of the target block averages the SSIM and QP values of the coded surrounding blocks (eg, left, upper left, right, upper right (if they exist), etc.). It is calculated by The average QP value QP _avg is taken as the estimated QPQP _block for the target block. If the average SSIM value SSIM _est is less than 0.9, QP _block =QP _avg is reduced by 2 to improve its quality. In a further embodiment, the target block identified as foreground data by TMVM has a TCSF contrast sensitivity logarithmic value (vertical axis in FIG. 5) of greater than 0.8 (noting that the block is temporally important). (Meaning)), the QP _block is reduced by 2.

  The methods described so far may use temporal importance maps (TCSF with or without TMVM refinement), spatial importance maps, or both. If both temporal and spatial importance maps are used, the result is called the integrated importance map.

  The importance map generated from the perceptual statistics as described above may be applicable to any video compression frame as long as it is a video compression frame that generates a motion vector using motion compensation. , Both rate-distortion analysis and quantization are improved to produce a visually superior encoding with the same encoding size. The application of the importance map to video compression does not require any special application to apply to the continuous block tracker (CBT) already detailed above. Moreover, the importance map is more effective in a CBT-based coding framework, as the CBT provides the additional ability to accurately determine which motion vector is the true motion vector. The specific reason for this is that the CBT frame-to-frame motion vector (from the frame-to-frame tracking 210 of FIG. 8A) was generated from the original frame of the video and reconstructed from the reconstructed frame. The point is that it was not generated. The frame store 85 of FIGS. 2 and 7 for a typical encoder contains the reconstructed frames generated from the encoding process, whereas the frame store 205 of FIGS. 3, 4 and 8A is the original. Contains video frames. Therefore, the CBT frame-frame tracking (reference numeral 210 in FIGS. 3, 4 and 8) can better track the true motion of the image, and the frame-frame motion vector is Generate a more accurate true motion vector map. In contrast, typical encoder motion vectors are chosen to optimize rate distortion (compression) performance and may not reflect the true motion of the video.

  Note that the generated importance map can also be applied to intra prediction frames by modifying the rate distortion optimization between intra prediction modes or by modifying the block level quantization according to the method described so far. obtain. However, in the case of an all-intra encoder (all-intra encoder), a separate encoding means (for example, as shown in FIG. 8A) for generating the motion vector for each data block in the video frame in calculating the TCSF is used. Frame-to-frame tracking 210, etc.) is required.

<Digital processing environment>
An exemplary implementation of the invention may be feasible in a software, firmware or hardware environment. FIG. 9A shows one such environment. At least one client computer/device 950 (eg, cell phone, computing device, etc.) and cloud 960 (or server computer or cluster of server computers) includes processing, storage, encoding, and decoding functionality for executing application programs. Computerization function and I/O device.

  Also, at least one client computer/device 950 may be connectable via a communication network 970 to other client devices/processes 950 and other computing devices including at least one server computer 960. The communications network 970 may be part of a remote access network, part of a global network (eg, the Internet, etc.), part of a worldwide collection of computers, part of a local area network, part of a wide area network, or It may be part of a gateway that currently communicates with each other using respective protocols (TCP/IP, Bluetooth®, etc.). Other electronic device/computer network architectures can be used.

  Embodiments of the invention may include means for encoding, tracking, modeling, filtering, adjusting, decoding or displaying video or data signal information. FIG. 9B illustrates a given computer/computing node (eg, client processor/device/cellphone device/cell phone device/in the processing environment of FIG. 9A that may be used to facilitate the encoding of such video or data signal information. It is a figure of an internal structure of tablet 950, server computer 960, etc. Each computer 950, 960 comprises a system bus 979, which is a set of real or virtual hardware lines used to transfer data between the components of a computer or processing system. The bus 979 is like a shared plumbing connecting different components of a computer system (eg, processor, encoder chip, decoder chip, disk storage, memory, input/output ports, etc.). Allows the exchange of data between. An input/output device interface 982 for connecting various input/output devices (for example, a keyboard, a mouse, a display, a printer, a speaker, etc.) to the computers 950 and 960 is attached to the system bus 979. The network interface 986 allows the computer to connect to a variety of other devices attached to a network (eg, the network shown at 970 in FIG. 9A). Memory 990 is a volatile memory that stores computer software instructions 992 and data 994 used to implement the software implementation of the present invention.

  Disk storage 995 is a non-volatile storage that stores computer software instructions 998 (equivalently “OS programs”) and data 994 used to implement one embodiment of the present invention. The disk storage 995 can also be used for long term storage of video in a compressed format. A central processing unit 984 that executes computer instructions is also attached to the system bus 979. Note that the “computer software instruction” and the “OS program” are equivalent to each other throughout the present specification.

  As an example, the encoder can be configured with computer readable instructions 992 for encoding video data using a significance map formed from temporal and spatial information. These importance maps may be configured to provide a feedback loop to the encoder (or components of the encoder) to optimize the encoding/decoding of video data.

  In one embodiment, processor routines 992 and data 994 are computer program products that include an encoder (generally indicated at 992). Such computer program product includes a computer-readable medium readable by storage device 994 that provides at least some of the software instructions for the encoder.

  Computer program product 992 may be installable by any suitable software installation method known in the art. Also, in other embodiments, at least some of the software instructions of the encoder may be downloadable via a cable and/or communication and/or wireless connection. In another embodiment, the encoder system software is a computer program propagated signal product 907 (FIG. 9A) embedded in a non-transitory computer readable medium, the computer program propagated signal product 907 being executed. Can be realized as a propagation signal on a propagation medium (for example, electric wave, infrared wave, laser wave, sound wave, electric wave propagated by a global network such as the Internet or at least one other network). Such carrier media or signals provide at least some of the software instructions for routines/programs 992 of the present invention.

  In an alternative embodiment, the propagating signal is an analog carrier or digital signal carried by a propagating medium. For example, the propagated signal may be a digital signal carried by a global network (eg, the Internet, etc.), a telecommunication network, or other network. In one embodiment, the propagating signal is sent by a propagating medium for a given period of time, eg, packets by the network for a period of milliseconds, seconds, minutes or more. , Instructions for software applications, etc. In another embodiment, the computer readable medium of computer program product 992 is a propagation medium readable and readable by computer system 950. For example, computer system 950 receives a propagation medium and identifies the propagated signal embedded within the propagation medium, as is the case for the computer program propagated signal product described above.

Although the present invention has been specifically shown and described with reference to exemplary embodiments, those skilled in the art can make forms and details without departing from the scope of the present invention included in the appended claims. It will be appreciated that various changes can be made.
The present invention includes the following contents as embodiments.
[Aspect 1]
A method of encoding a plurality of video frames, the method comprising:
The video frame has target blocks that do not overlap with each other,
The method is
Encoding the plurality of video frames using the importance map so that the importance map affects the encoding quality of each target block to be encoded in each video frame by adjusting the quantization. ,
And the importance map is:
Establishing the importance map using temporal and spatial information; and
(I) In a block in which the importance map has a high numerical value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that high quality is obtained for these blocks. In addition, (ii) in the target block in which the importance map has a low numerical value, the block quantization parameter is made larger than the frame quantization parameter QP _frame , so that the quality of these blocks becomes low. Causing the importance map to indicate, by calculation, which part of a video frame of the plurality of video frames is most noticeable to human perception;
Composed by the method.
[Aspect 2]
The method of aspect 1, wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which is to determine which target block in the frame is an average block in the frame. _Is to have a variance that is greater than the variance var _frame ,
A quantization parameter (QP) value higher than the frame quantization parameter QP _frame is assigned to a block having a variance larger than the average block variance var _frame , and an allocation amount QP of the block quantization parameter (QP) is assigned. _{The block} is linearly increased or decreased between the frame quantization parameter QP _frame and a quantization parameter upper limit QP _max according to how large the block variance var _block is than the average block variance var _frame .
[Aspect 3]
In the method according to aspect 1, the temporal information is
A temporal contrast sensitivity function (TCSF) indicating which target block is most noticeable in time for an observer human, and
True motion vector map (TMVM) showing which target block corresponds to foreground data
And the TCSF is only valid for target blocks identified as foreground data.
[Mode 4]
In the method according to aspect 2, a block with a large variance has a block quantization parameter (QP), the allocation amount QP _{block, in} which the TMVM specifies a target block as foreground data, and The method further refined by the TCSF and the TMVM such that the entrainment amount QP _block is increased by 2 when the contrast sensitivity logarithmic value is less than 0.5 .
[Aspect 5]
In the method according to aspect 2, the SCM is further configured such that the allocation amount QP _block, which is a block quantization parameter of an extremely bright (brightness of 170 or more) or extremely dark (brightness of less than 60) target block, is QP _max . A method comprising brightness masking being readjusted.
[Aspect 6]
A method according to aspect 2, wherein the SCM further comprises a dynamic determination of the quantization parameter upper limit QP _max based on a quality level of the encoded video ,
In this dynamic decision , the quality is measured using the average structural similarity (SSIM) calculation result of the target blocks in the intra (I) _frame together with the average block variance var _frame of these frames ,
The method, wherein when the measured quality is low, the value of the quantization parameter upper limit QP _max is reduced to approach the frame quantization parameter QP _frame .
[Aspect 7]
In the method according to aspect 2, for blocks with extremely small variance, in order to ensure high quality coding in these regions, the smaller the block variance is, the lower the value of the allocation amount QP _block is. A method in which the allocation amount QP _block, which is a determined low value of the quantization parameter (QP), is allocated (and high quality) .
[Aspect 8]
In the method according to aspect 7, the allocation amount QP _block , which is the value of the low quantization parameter (QP) for a block with extremely small variance, is first determined for I frames and then for P and B frames. Is determined using the ipratio and pbratio parameters.
[Aspect 9]
In the method according to Aspect 7, a block whose variance is small but whose variance is not considered to be extremely small is used to determine whether quality improvement is necessary for the block.
The allocation amount QP _block, which is the first estimated value of the block quantization parameter (QP), represents the values of the quantization parameter (QP) of the already coded neighboring blocks on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from SSIM values of previously coded neighboring blocks to the left, top left, right and top right of the current block, and
If the SSIM _est is less than 0.9, the value of the allocation amount QP _block is decreased by 2,
How to be examined.
[Aspect 10]
A method according to aspect 9, wherein the quality improvement is applied only to blocks identified as foreground data by the TMVM and having a contrast sensitivity logarithmic value of the TCSF greater than 0.8.
[Aspect 11]
In the method according to Aspect 3, the temporal frequency of the TCSF is determined by using SSIM in the color space region between the target block and its reference block to approximate the wavelength, and the magnitude of the motion vector and the frame rate. Calculated by finding an approximation of the velocity using.
[Aspect 12]
The method of aspect 3, wherein the TCSF is a plurality of TCSFs such that the TCSF for the current frame is a weighted average of the TCSF maps in the most recent frames and the more recent frames are given more weight. Method calculated over the frame.
[Aspect 13]
A method according to aspect 3, wherein the TMVM is set to 1 only for foreground data.
[Aspect 14]
In the method described in Aspect 13, in the foreground data, a difference between an encoder motion vector for a given target block and a global motion vector for the block is calculated, and a block having a sufficiently large difference is the foreground data. A method identified by being judged.
[Aspect 15]
In the method according to aspect 14, a difference motion vector is obtained by subtracting the encoder motion vector from the global motion vector for a data block specified as foreground data, and the magnitude of the difference motion vector is the TCSF. The method used to calculate the temporal frequency of the.
[Aspect 16]
The method of aspect 3, wherein the TCSF is calculated from motion vectors from an encoder.
[Aspect 17]
The method according to aspect 1, wherein when the importance map is set with the temporal information and the spatial information, the importance map is an integrated importance map.
[Aspect 18]
A system for encoding video data,
A codec for encoding a plurality of video frames using an importance map, the video frames having target blocks that do not overlap each other, the codec,
Wherein the importance map is configured to affect the coding quality of each target block to be coded in each video frame by adjusting the quantization,
The importance map is:
The importance map is set by using temporal information and spatial information, and the importance map set by these temporal information and spatial information is an integrated heavy element map. ; And
(I) In a block in which the importance map has a high numerical value, the block quantization parameter (QP) is made smaller than the frame quantization parameter QP _frame , so that high quality is obtained for these blocks. In addition, (ii) in the target block in which the importance map has a low numerical value, the block quantization parameter is made larger than the frame quantization parameter QP _frame , so that the quality of these blocks becomes low. Causing the calculation to show, in the importance map, a part of a video frame of the plurality of video frames that is most noticeable to human perception.
The system that is composed by.
[Aspect 19]
The encoder according to aspect 18, wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which is to determine which target block in the frame is an average block in the frame. _Is to have a variance that is greater than the variance var _frame ,
A quantization parameter (QP) value higher than the frame quantization parameter QP _frame is assigned to a block having a variance larger than the average block variance var _frame , and an allocation amount QP of the block quantization parameter (QP) is assigned. _An encoder in which a _block is linearly increased or decreased between the frame quantization parameter QP _frame and a quantization parameter upper limit QP _max according to how large the block variance var _block is than the average block variance var _frame .
[Aspect 20]
In the encoder according to aspect 18, the temporal information is
A temporal contrast sensitivity function (TCSF) indicating which target block is most noticeable in time for an observer human, and
True motion vector map (TMVM) showing which target block corresponds to foreground data
An encoder provided by the above, wherein the TCSF is valid only for target blocks identified as foreground data.
[Aspect 21]
21. In the encoder according to aspect 19, a block with a large variance has a block quantization parameter (QP), the allocation amount QP _{block, in} which the TMVM specifies a target block as foreground data, and the block of the TCSF is An encoder further refined by the TCSF and the TMVM such that the allocation amount QP _block is increased by 2 when the contrast sensitivity logarithmic value is less than 0.5 .
[Aspect 22]
20. In the encoder according to aspect 19, the SCM is further configured such that the allocation amount QP _block, which is a block quantization parameter of an extremely bright (brightness above 170) or extremely dark (brightness below 60) target block, is QP _max . An encoder that includes readjusted intensity masking.
[Aspect 23]
20. The encoder according to aspect 19, wherein the SCM further comprises a dynamic determination of QP _max based on the quantization parameter upper bound on a quality level of encoded video ,
In this dynamic decision , the quality is measured using the average structural similarity (SSIM) calculation result of the target blocks in the intra (I) _frame together with the average block variance var _frame of these frames ,
The encoder , wherein when the measured quality is low, the numerical value of the quantization parameter upper limit QP _max is reduced to approach the frame quantization parameter QP _frame .
[Aspect 24]
In the encoder according to aspect 19, for blocks with extremely small variance, in order to ensure high quality coding in these regions, the smaller the block variance is, the lower the value of the allocation amount QP _block is. An encoder , to which the assignment amount QP _block, which is a determined low value of the quantization parameter (QP), is assigned (and in order to improve the quality) .
[Aspect 25]
25. In the encoder according to aspect 24, the allocation amount QP _block , which is the value of the low quantization parameter (QP) for a block having an extremely small variance, is first determined for an I frame, and then a P frame and a B frame. Is determined using the ipratio and pbratio parameters.
[Aspect 26]
In the system according to Aspect 19, a block whose variance is small but whose variance is not considered to be extremely small is used to determine whether or not quality improvement is required for the block.
The allocation amount QP _block, which is the first estimated value of the block quantization parameter (QP), represents the values of the quantization parameter (QP) of the already encoded neighboring blocks on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from SSIM values of previously coded neighboring blocks to the left, top left, right and top right of the current block, and
When the SSIM _est is less than 0.9, the value of the allocation amount QP _block is decreased by 2,
System examined.
[Mode 27]
27. The system of aspect 26, wherein the quality enhancement is applied only to blocks identified by the TMVM as foreground data and having a TCSF contrast sensitivity logarithmic value greater than 0.8.
[Aspect 28]
Aspect 20. In a system as set forth in aspect 20, the temporal frequency of the TCSF is determined using SSIM in the color space region between the target block and its reference block to approximate the wavelength and the magnitude of the motion vector and the frame rate. The system, calculated by finding an approximation of the velocity using.
[Aspect 29]
A system according to aspect 20, wherein the TCSF comprises a plurality of TCSFs such that the TCSF for the current frame is a weighted average of the TCSF maps in the most recent frames and the more recent frames are given more weight. A system calculated over a frame.
[Aspect 30]
21. The system according to aspect 20, wherein the TMVM is set to 1 only for foreground data.
[Mode 31]
In the system according to Aspect 30, the foreground data calculates a difference between an encoder motion vector for a given target block and a global motion vector for the block, and the block having a sufficiently large difference is the foreground data. A system identified by being judged.
[Aspect 32]
Aspect 20. In the system according to aspect 20, a difference motion vector is obtained by subtracting the encoder motion vector from the global motion vector for a data block specified as foreground data, and the magnitude of the difference motion vector is the TCSF. The system used to calculate the temporal frequency of the.
[Aspect 33]
21. The system according to aspect 20, wherein the TCSF is calculated from the motion vector from the encoder.
[Aspect 34]
A system according to aspect 18, wherein when the importance map is set by the temporal information and the spatial information, the importance map is an integrated importance map.

Claims (30)

A method of encoding a plurality of video frames, the method comprising:
The video frame has target blocks that do not overlap with each other,
The method is
Encoding the plurality of video frames using the importance map so that the importance map affects the encoding quality of each target block to be encoded in each video frame by adjusting the quantization. ,
And the importance map is:
Establishing the importance map using temporal and spatial information; and
(I) as the importance map is a block taking a high numerical value in the block quantization parameter (QP) is smaller than the frame quantization parameter QP _frame, a higher quality for these blocks as such, and as (ii) the importance map is a target block to take low value, the by block quantization parameter is larger than the frame quantization parameter QP _frame, for these blocks as a lower quality, by calculation, what portion of the video frame certain of said plurality of video frames that tends most noticed for human perception be indicated in the importance map;
And the temporal information is
A temporal contrast sensitivity function (TCSF) that measures the response of the human visual system to a temporally periodic stimulus to indicate which target block is most temporally noticeable to an observer human; and
True motion vector map (TMVM) showing which target block corresponds to foreground data
And the TCSF is valid only for target blocks identified as foreground data,
The temporal frequency of the TCSF is an approximation of the velocity obtained using the magnitude of the motion vector and the frame rate, and is the average structural similarity (SSIM) in the color space region between the target block and its reference block. The method is calculated by dividing by the approximation of the wavelength determined using . The method of claim 1, wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which is to determine which target block in the frame is an average in the frame. To determine if it has a variance greater than the block variance var _frame ,
A quantization parameter (QP) value higher than the frame quantization parameter QP _frame is assigned to a block having a variance larger than the average block variance var _frame , and an allocation amount QP of the block quantization parameter (QP) is assigned. _block is the block variance _{var block} of the block, the average block variance _{var frame,} the quantization parameter upper _{QP max} and said frame quantization parameter _{QP frame using, QP block = ((var block} -var frame) / var block )×(QP _max −QP _frame )+QP _frame , which is linearly increased or decreased. The method according to claim 2, wherein a block having a variance larger than an average block variance var _{frame in the frame} has the allocation amount QP _block , which is a block quantization parameter (QP) thereof, and the TMVM has a foreground target block. Further refined by the TCSF and the TMVM such that the QP _block is increased by 2 if specified as data and the contrast sensitivity logarithmic value for this block of the TCSF is less than 0.5, Method. The method according to claim 2, wherein the SCM further adjusts the allocation amount QP _block, which is a block quantization parameter of a target block having a block average brightness of more than 170 or less than 60, to QP _max. A method comprising brightness masking being repaired. The method according to claim 2, wherein the SCM further comprises a dynamic determination of the quantization parameter upper bound QP _max based on a quality level of the coded video,
In this dynamic decision, the quality is measured using the average structural similarity (SSIM) calculation result of the target block in the intra (I) _frame together with the average block variance var _{frame of the} I frame,
The measured lower the quality value of the quantization parameter limit _{QP max} is reduced so as to approach more to the frame quantization parameter _{QP frame,} method. The method according to claim 2, wherein for blocks having a variance of less than 60, the smaller the block variance, the lower the value of the allocation QP _block to ensure high quality coding in these regions. The allocation amount QP _block, which is a determined low quantization parameter (QP) value, is allocated such that The method of claim 6, wherein the appropriation amount _{QP block} is a value before Symbol amount Coca parameter (QP) for the dispersion is less than 60 blocks, initially determined for the block of I frame, then , A P frame and a block corresponding to the block in the B frame are expressed as a ratio of the frame QP value of the I frame to the frame QP value of the P frame and the ratio of the QP value of the P frame to the QP value of the B frame. A method that is determined using certain pbratio parameters. 7. The method according to claim 6 , wherein a block having a variance of 60 or more and an average block variance or less is :
The allocation amount QP _block, which is the first estimated value of the block quantization parameter (QP), represents the values of the quantization parameter (QP) of the already coded neighboring blocks on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from SSIM values of previously coded neighboring blocks to the left, top left, right and top right of the current block, and
When the SSIM _est is less than 0.9, it is determined that the quality improvement is necessary in the determination of whether the quality improvement is necessary, and the quality improvement process is performed so that the calculated value of the allocation amount QP _block is decreased by 2. The way it is done . 9. The method of claim 8 , wherein the quality enhancement is applied only to blocks identified by the TMVM as foreground data and having a TCSF contrast sensitivity log value greater than 0.8. The method of claim 1 , wherein the TCSF is encoded such that the TCSF for the current frame is a weighted average of the TCSF maps in the past frames in which the TCSF was encoded, and the past is closer in time. The calculated past frames are calculated over multiple frames such that they are given greater weighting. The method of claim 1 , wherein the TMVM is set to 1 only for foreground data. The method according to claim 11 , wherein the foreground data calculates a difference between an encoder motion vector for a given target block and a global motion vector for the block, and a block having a difference larger than a specific value is the foreground data. A method identified by being determined to be. Calculation The method according to claim 12, for a particular data block as a foreground data, the obtaining a differential motion vector by the encoder motion vector from the global motion vector is subtracted, the time frequency before Symbol TCSF The velocity v used to do is v , using the magnitude |DMV| of this difference motion vector, the number N of frames between the reference frame pointed to by this difference motion vector and the current frame, and the frame rate. =|DMV|×frame rate/N . The method according to claim 1 , wherein the TCSF uses a motion vector MV from an encoder to calculate a speed v for calculating a temporal frequency that is an input to the TCSF, v=|MV|×frame rate/N (Where the frame rate is the number of frames per second in which the video was generated, and N is the number of frames between the reference frame pointed to by the motion vector and the current frame). the, way. The method according to claim 2 , wherein when the importance map is set by the temporal information and the spatial information, the importance map is a combined importance map (both TCSF and SCM). Method, which is an importance map containing information from . A system for encoding video data,
A codec for encoding a plurality of video frames using an importance map, the video frames having target blocks that do not overlap each other, the codec,
Wherein the importance map is configured to affect the coding quality of each target block to be coded in each video frame by adjusting the quantization,
The importance map is:
Establishing the importance map using temporal and spatial information; and
(I) as the importance map is a block taking a high numerical value in the block quantization parameter (QP) is smaller than the frame quantization parameter QP _frame, a higher quality for these blocks as such, and as (ii) the importance map is a target block to take low value, the by block quantization parameter is larger than the frame quantization parameter QP _frame, for these blocks as a lower quality, by calculation, picture frames certain of said plurality of video frames, thereby it indicated the most noticed easily parts for human perception to the importance map;
And the temporal information is
A temporal contrast sensitivity function (TCSF) that measures the response of the human visual system to a temporally periodic stimulus to show which target block is most noticeable in time to an observer human; and
True motion vector map (TMVM) showing which target block corresponds to foreground data;
And the TCSF is only valid for target blocks identified as foreground data,
The temporal frequency of the TCSF is an approximation of the velocity obtained using the magnitude of the motion vector and the frame rate, and is the average structural similarity (SSIM) in the color space region between the target block and its reference block. The system , calculated by dividing by the approximation of the wavelength used . 17. The system of claim 16 , wherein the spatial information is provided by a rule-based spatial complexity map (SCM), the first step of which is to determine which target block in the frame is an average within the frame. To determine if it has a variance greater than the block variance var _frame ,
A quantization parameter (QP) value higher than the frame quantization parameter QP _frame is assigned to a block having a variance larger than the average block variance var _frame , and an allocation amount QP of the block quantization parameter (QP) is assigned. _block is the block variance _{var block} of the block, the average block variance _{var frame,} the quantization parameter upper _{QP max} and said frame quantization parameter _{QP frame using, QP block = ((var block} -var frame) / var block _{) × (QP max -QP frame)} + QP frame and linearly is increased or decreased so that the system. 18. The system according to claim 17 , wherein a block having a variance larger than an average block variance var _{frame in the frame} has the allocation amount QP _block , which is its block quantization parameter (QP), and the TMVM has a target block in the foreground. Further refined by the TCSF and the TMVM such that the QP _block is increased by 2 if specified as data and the contrast sensitivity logarithmic value for this block of the TCSF is less than 0.5, system. The system according to claim 17 , wherein the SCM further adjusts the allocation amount QP _{block to} be QP _max , which is a block quantization parameter of a target block having a block average brightness of more than 170 or less than 60. A system including brightness masking that is corrected. The system of claim 17, wherein the SCM further comprises a dynamic determination of the upper the quantization parameter based on the quality level of the encoded video limit Q P _max,
In this dynamic decision, the quality is measured using the average structural similarity (SSIM) calculation result of the target block in the intra (I) _frame together with the average block variance var _{frame of the} I frame,
As measured quality is low, the numerical value of the quantization parameter limit _{QP max} is reduced as closer to the frame quantization parameter _{QP frame,} system. 18. The system of claim 17 , for blocks with variance less than 60, the smaller the block variance, the lower the value of the allocation QP _block , to ensure high quality coding in these regions. The allocation amount QP _block, which is a value of a determined quantization parameter (QP), is allocated so that. 22. The system of claim 21 , wherein the allocation QP _block , which is the value of the low quantization parameter (QP) for blocks with variance less than 60, is first determined for I frames and then for P frames. And for a block corresponding to the block in the B frame , an ipratio parameter that is the ratio of the frame QP value of the I frame to the frame QP value of the P frame and a pbraio parameter that is the ratio of the P frame QP value to the B frame QP value. A system that can be determined using. The system according to claim 17 , wherein a block having a variance of 60 or more and an average block variance or less is :
The allocation amount QP _block, which is the first estimated value of the block quantization parameter (QP), represents the values of the quantization parameter (QP) of the already coded neighboring blocks on the left, upper left, right and upper right of the current block. Calculated by averaging, and
An estimated SSIM _{est of the} SSIM of the current block is calculated from SSIM values of previously coded neighboring blocks to the left, top left, right and top right of the current block, and
When the SSIM _est is less than 0.9, it is determined that the quality improvement is necessary in the determination of whether the quality improvement is necessary, and the quality improvement process is performed so that the calculated value of the allocation amount QP _block is decreased by 2. The system to be done . 24. The system of claim 23 , wherein the quality enhancement is applied only to blocks identified by the TMVM as foreground data and having a TCSF contrast sensitivity log value greater than 0.8. 17. The system of claim 16 , wherein the TCSF is encoded such that the TCSF for the current frame is a weighted average of the TCSF maps in the past frames in which the TCSF was encoded , and the temporally closer past. The system is calculated over multiple frames such that the past frames that have been processed receive a greater weighting. 17. The system according to claim 16 , wherein the TMVM is set to 1 only for foreground data. 27. The system according to claim 26 , wherein the foreground data calculates a difference between an encoder motion vector for a given target block and a global motion vector for the block, and a block having a difference larger than a specific value is the foreground data. A system identified by being judged to be. The system of claim 16, for a particular data block as a foreground data, obtaining a differential motion vector by encoder motion vectors for the data blocks from the global motion vector for the data block is subtracted, the The velocity v used to calculate the temporal frequency of the TCSF is the magnitude of this difference motion vector |DMV|, the number N of frames between the reference frame pointed to by this difference motion vector and the current frame, and A system where v=|DMV|×frame rate/N is calculated using the frame rate . The system according to claim 16 , wherein the TCSF uses a motion vector MV from the encoder to calculate a speed v for calculating a temporal frequency that is an input to the TCSF, v=|MV|×frame rate/ N, where frame rate is the number of frames per second in which the video was generated, and N is the number of frames between the reference frame pointed to by the motion vector and the current frame. The system to calculate . 18. The system according to claim 17 , wherein when the importance map is set with the temporal information and the spatial information, the importance map is an integrated importance map ( both TCSF and SCM). System, which is an importance map containing information from ).