US20200236390A1 - Decoupled mode inference and prediction - Google Patents
Decoupled mode inference and prediction Download PDFInfo
- Publication number
- US20200236390A1 US20200236390A1 US16/652,813 US201816652813A US2020236390A1 US 20200236390 A1 US20200236390 A1 US 20200236390A1 US 201816652813 A US201816652813 A US 201816652813A US 2020236390 A1 US2020236390 A1 US 2020236390A1
- Authority
- US
- United States
- Prior art keywords
- block
- information
- prediction
- motion
- video data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 74
- 239000013598 vector Substances 0.000 claims abstract description 45
- 230000002123 temporal effect Effects 0.000 claims description 31
- 230000008569 process Effects 0.000 claims description 28
- 238000005286 illumination Methods 0.000 claims description 21
- 230000006870 function Effects 0.000 claims description 16
- 238000009795 derivation Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000004590 computer program Methods 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 15
- 230000006835 compression Effects 0.000 description 10
- 238000007906 compression Methods 0.000 description 10
- 230000000875 corresponding effect Effects 0.000 description 8
- 230000009466 transformation Effects 0.000 description 8
- 230000008901 benefit Effects 0.000 description 3
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- GXFZCDMWGMFGFL-KKXMJGKMSA-N (+)-Tubocurarine chloride hydrochloride Chemical compound [Cl-].[Cl-].C([C@H]1[N+](C)(C)CCC=2C=C(C(=C(OC3=CC=C(C=C3)C[C@H]3C=4C=C(C(=CC=4CC[NH+]3C)OC)O3)C=21)O)OC)C1=CC=C(O)C3=C1 GXFZCDMWGMFGFL-KKXMJGKMSA-N 0.000 description 1
- 239000008709 Curare Substances 0.000 description 1
- 241000297530 Holoparamecus integer Species 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/439—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using cascaded computational arrangements for performing a single operation, e.g. filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
- H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/517—Processing of motion vectors by encoding
- H04N19/52—Processing of motion vectors by encoding by predictive encoding
Definitions
- the following described aspects relate generally to the field of video compression, and more particularly to prediction and parsing.
- CTU Coding Tree Unit
- Each CTU is represented by a Coding Tree in the compressed domain. This is a quad-tree division of the CTU, where each leaf is called a Coding Unit (CU.
- CU Coding Unit
- Motion compensated temporal prediction is employed to exploit the redundancy that exists between successive pictures of a video. To do so, a motion vector is associated with each prediction unit (PU). Each CU is then given some Intra or Inter prediction parameters (Prediction Info) and is spatially partitioned into one or more Prediction Units (PUs), each PU being assigned some prediction information. The Intra or Inter coding mode is assigned on the CU level.
- Prediction Info Intra or Inter prediction parameters
- PUs Prediction Units
- the Intra or Inter coding mode is assigned on the CU level.
- the following description is aimed at ideas solving prediction and parsing issues in digital video coding.
- the present embodiments aim to improve the compression efficiency in video encoders by decoupling mode inference and prediction as further explained herein.
- a method for coding a block of video data comprises steps for motion compensating a block of video data, the motion compensating comprising: determining whether a current motion prediction mode for the block is a temporal motion prediction mode; deriving a coding parameter using information from a spatial neighbor when a temporal prediction mode is determined; and, encoding a block of video data, using said coding parameter.
- a method for decoding a block of video data comprises steps for decoding a piece of information within a coded video bitstream; predicting the piece of information based on contextual information; performing an inverse combining function of said decoded piece of information and said prediction of said piece of information to generate a piece of information for a current block of video data; and, decoding the motion compensated block of video data.
- a method comprising determining whether the coding mode is ATMVP, STMVP or FRUC/unilateral; predicting an illumination compensation flag for the current block of video data from spatial neighboring illumination compensation flags when the determined coding mode is ATMVP, STMVP or FRUC/unilateral; using an illumination compensation flag of a predictor block if the determined coding mode is not ATMVP, STMVP or FRUC/unilateral; and, using the illumination compensation flag for coding the current block of video data.
- a method comprising determining whether the coding mode is TMVP, ATMVP or FRUC; predicting an illumination compensation flag for the current block of video data from spatial neighboring illumination compensation flags when the determined coding mode is TMVP, ATMVP or FRUC; using an illumination compensation flag of a predictor block if the determined coding mode is not TMVP, ATMVP or FRUC; and, using the illumination compensation flag for coding the current block of video data.
- apparatus for coding or decoding a block of video data comprising: a memory, and a processor, configured to perform any of the aforementioned method embodiments with variations.
- a nontransitory computer readable storage medium containing data content generated according to the method of any one of the aforementioned method embodiments, or by the apparatus of any one of the aforementioned apparatus embodiments for playback using a processor.
- a signal comprising video data generated according to the method of any one of the aforementioned method embodiments for coding a block of video data, or by the apparatus of any one of the aforementioned apparatus embodiments for coding a block of video data, for playback using a processor.
- a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of the aforementioned method embodiments.
- FIG. 1 a shows an example of a coding tree unit and coding tree concepts to represent a compressed HEVC picture.
- FIG. 1 b shows an example of division of a coding tree unit into coding units, prediction units, and transform units.
- FIG. 1 c shows one example of a coded at a tree for a motion compensated block.
- FIG. 2 shows an example of spatial motion vector prediction.
- FIG. 3 shows an example of temporal motion vector prediction.
- FIG. 4 shows an example of a generic video compression scheme to which the present embodiments can be applied.
- FIG. 5 shows one example of a generic video decompression scheme to which the present embodiments can be applied.
- FIG. 6 shows an example block diagram of information prediction/inferring from spatial predictors.
- FIG. 7 shows an example block diagram for illumination compensation (IC) flag prediction/inference.
- FIG. 8 shows an example illustration of temporal motion prediction and spatial prediction of an IC flag.
- FIG. 9 shows another example of a block diagram for IC flag prediction/inference.
- FIG. 10 shows an example block diagram of another IC flag prediction/inference.
- FIG. 11 shows an example of sub-PU derivation process in STMVP (Spatial Temporal Motion Vector Prediction).
- FIG. 12 shows a block diagram of decoupled predictor and information coding.
- FIG. 13 shows a block diagram of decoupled predictor and information parsing/decoding.
- FIG. 14 shows a block diagram of decoupled predictor and flag coding.
- FIG. 15 shows one embodiment of an encoding method using the aspects described.
- FIG. 16 shows one embodiment of a decoding method using the aspects described.
- FIG. 17 shows one embodiment of an apparatus for encoding or decoding using the aspects described.
- CTU Coding Tree Unit
- Each CTU is represented by a Coding Tree in the compressed domain. This is a quad-tree division of the CTU, where each leaf is called a Coding Unit (CU), as shown in the example in FIG. 1 a.
- CU Coding Unit
- Each CU is then given some Intra or Inter prediction parameters (Prediction Info). To do so, it is spatially partitioned into one or more Prediction Units (PUs), each PU being assigned some prediction information.
- the Intra or Inter coding mode is assigned on the CU level, as shown in the example of FIG. 1 b.
- Coding Units are also divided into so-called transform units, in a recursive way, following a “transform tree”.
- a transform tree is a quad-tree division of a coding unit, and transform units are the leaf of the transform tree.
- a transform unit encapsulates the square transform blocks of each picture component corresponding to a considered square spatial area.
- a transform block is a square block of samples in a single component, where the same transform is applied.
- New emerging video compression tools include a Coding Tree Unit representation in the compressed domain to represent picture data in a more flexible way in the compressed domain.
- the advantage of this flexible representation of the coding tree is that it provides increased compression efficiency compared to the CU/PU/TU arrangement of the HEVC standard.
- the domain of the described embodiments is video compression.
- prediction of a given block is obtained by a motion compensated block from a reference picture.
- Such prediction is built from a motion vector which can undergo motion transformation used to motion compensate the block.
- the prediction can be processed by a sample transformation, for example using the IC (Illumination Compensation) transformation.
- Such transformations may be activated or de-activated using a flag, either coded or inferred.
- current solutions generally predict/infer the flag from the motion predictor block (the block from which the motion is predicted).
- the predictor block can be in the spatial neighbor of the current block (spatial motion predictor block), or it can be in a reference picture (temporal motion predictor block).
- temporal motion predictor like the ones in the ATMVP (Advanced Temporal Motion Vector Prediction), STMVP (Spatial Temporal Motion Vector Prediction) or FRUC (Frame Rate Up Conversion) modes of the JEM (Joint Exploration Model)
- ATMVP Advanced Temporal Motion Vector Prediction
- STMVP Spatial Temporal Motion Vector Prediction
- FRUC Full Conversion
- the encoding/decoding process described here is the one used in another application for an equi-rectangular projection predictor for omnidirectional video, but is applicable to other flags or syntax elements (like IC, or OBMC (overlapped block motion compensation) and IMV (initial motion vector)).
- OBMC overlapped block motion compensation
- IMV initial motion vector
- the following description is aimed at ideas solving prediction and parsing issues in digital video coding.
- the issues are that for predicting or parsing an information, contextual data are required, but these contextual data may not always be available. Solutions are proposed to improve the prediction and parsing processes for such cases.
- the inference and predictor block for inter modes TMVP, ATMVP, STMVP, FRUC/unilateral can be decoupled. This proposed idea aims mostly at improving the prediction process of the information.
- CABAC context can be removed.
- decouple the CABAC context and a predictor block for inter coding to avoid the non-availability of the piece of information for the parsing.
- a given piece of information for example the IC flag, or symbol is coded or decoded using CABAC
- contextual information from spatial neighbors may need to be available in order to select the CABAC context; however, in the decoder, this contextual information may be not directly available when the bitstream is parsed, but needs to be first decoded (or reconstructed) from the decoded data; therefore the parsing process becomes dependent on the decoding process, which is generally not desired nor recommended.
- a concrete implementation of this idea is to use neighboring blocks (e.g. Above and Left blocks) to predict the symbol. What is coded is the combination, for example by an “XOR” operator, of the actual symbol value and of the prediction value. Similarly, at a decoder, the decoded value is inverse combined with the prediction value, for example using XOR.
- the parsing of the decoded value may use CABAC contexts, but these contexts do not depend on neighboring information not yet available during the parsing. Hence, the coded/decoded value is not directly linked to the contextual (spatial/temporal) neighboring information related to the symbol.
- v being the value of INF
- p being the prediction
- c the result of the combining function
- the inverse combining function g is such that:
- c is encoded.
- c is decoded (parsed), and v is reconstructed by computing g(c, p).
- f is chosen as the XOR operator, g is simply also XOR. This works when INF is a flag.
- At least one embodiment consists in decoupling the inference/prediction of a flag from the predictor block in inter motion compensation.
- the domain of the described embodiments is the video compression.
- prediction of a given block (or coding unit, also noted CU) is given by a motion compensated block.
- Such prediction is built from a motion vector which can undergo transformation used to motion compensate the block.
- the prediction can be processed by a transformation, for example the IC (llumination Compensation) transformation.
- the motion compensation tools in existing codecs such as H.265 (ITU-T, 2013) are based on the description of the motion of a current block between the current frame and a reference frame, which has already been decoded. This motion is classically represented with a 2D vector and a reference frame index, which informs the decoder on which reference frame to select among a set of previously decoded frames stored in the reference pictures buffer.
- FIG. 1 c depicts an example of data coding tree for an inter block:
- FIG. 4 and FIG. 5 depict classical video encoders and decoders, respectively.
- the current embodiments concern two points.
- the embodiments change the way of predicting or inferring syntax elements or parameters of a motion compensated block (i.e. a non intra block), i.e. mainly impacting 170 in FIGS. 4 and 275 in FIG. 5 . They also relate to the parsing of a video stream, and therefore impacts block 145 of FIG. 4 and 230 of FIG. 5 .
- This problematic case can, for example, happen when a syntax element (SE) being parsed depends on contexts depending on neighboring coding units (CUs) of the current block, that are coded with Merge mode; when a CU is coded with Merge mode, all its parameters are deduced but not explicitly coded in the stream; therefore, during the parsing of the stream, these parameters are not yet known; and they cannot be used for parsing the syntax element SE.
- SE syntax element
- INF which can be, for example, the value of the IC flag
- a given piece of information which can be referred to as INF (which can be, for example, the value of the IC flag)
- INF which can be, for example, the value of the IC flag
- predictor blocks either located in the spatial neighbor of the current block, co-located in reference pictures or in a FRUC/unilateral (frame rate up conversion) motion field.
- the described embodiments cover at least two separate aspects.
- the first aspect is forcing the prediction of the information INF to be based on spatial neighbors—this point is covered at least in the first two embodiments.
- the second aspect is parsing the information INF using decoupled predictors—this point is covered at least by the third embodiment.
- the information INF inference/prediction for motion predicted blocks can be decoupled.
- the embodiment proposes to predict/infer the information INF from spatial neighbors even when the motion predictor of the current block is built using a temporal prediction, for instance using ATMVP, STMVP or FRUC/unilateral modes.
- the concept is shown in the block diagram of FIG. 6 .
- the input of the prediction/inferring process of INF is the motion prediction mode M of the current block.
- a checking step is applied to evaluate if the mode M corresponds to a temporal prediction or a spatial prediction (step 201 ). If M corresponds to a spatial prediction, the prediction/inferring of INF is made using the corresponding information from the spatial predictor blocks used to prediction M (step 202 ). If M corresponds to a temporal prediction, the prediction/inferring of INF is made using the corresponding information from spatially neighboring predictor blocks, not used to prediction M and therefore different from the blocks used to predict M (step 203 ).
- a first alternative to the first embodiment involves the application to ATMVP, STMVP or FRUC/unilateral modes.
- the block diagram of FIG. 7 shows the implementation of this process for INF corresponding to the flag IC.
- the mode M is compared to the ATMVP, STMVP or FRUC/unilateral modes. If M is one of these modes, the IC flag of the current block is spatially predicted/inferred from the TOP and LEFT block IC flags when the current motion predictor block is coming from ATMVP, STMVP or FRUC/unilateral candidates. This corresponds to step 203 . If M is none of the ATMVP, STMVP or FRUC/unilateral modes, the IC value of the predictor block is used. This corresponds to step 202 .
- FIG. 8 illustrates how the process of FIG. 7 applies for deriving the motion vector and the IC flag of a block.
- the motion vector dP and reference index rP of block P in the current picture are temporally predicted by the motion vector dW and reference index rW of block W in the reference picture.
- the IC flag IC_P of block P is predicted from the IC flag IC_V of a spatially neighboring block V.
- a second alternative to the first embodiment involves the application to ATMVP, STMVP or FRUC modes.
- a third alternative of the first general embodiment involves the use of spatial predictors for IC flag.
- the motion vectors of the sub-PUs are derived recursively, following raster scan order. This is illustrated in FIG. 11 , where a CU contains four sub-Pus (or sub-CUs), noted A, B, C, D. The CU has fours neighboring blocks a, b, c, d.
- the motion derivation for sub-PU A uses the motion from the two spatial neighbors, b and c, plus the temporal motion vector predictor (TMVP) of sub-block A. Finally, a merging of the (up to three) motion vectors is achieved (e.g. by averaging them). The same process then applies to block B, with spatial neighbors A, d and the temporal motion vector predictor (TMVP) of sub-block B. And so on for the two remaining blocks of the CU.
- TMVP temporal motion vector predictor
- the information INF is deduced from the neighboring PU at block level (not sub-PU) using above methods.
- One single prediction of INF is used for all sub-PUs of the current block.
- the derivation of INF therefore differs from the derivation of the motion vectors, for which the recursive STMVP process is maintained.
- the same information INF is then available for all sub-PUs.
- a third general embodiment involves flag coding using decoupled predictor.
- FIG. 12 shows a simplified block diagram of the encoder side, for coding the information INF_cur of a given block.
- contextual information 505
- it consists in a list of candidate motion vectors from the neighborhood of the block of from blocks in reference pictures.
- a first prediction 500
- a predictor of INF_cur is generated ( 501 ).
- the encoding process also performs the choice of the actual value of the information INF, typically based on rate-distortion optimization ( 502 ).
- INF_pred and INF_cur are then combined to generate the value of the information to be coded, INF_cod.
- INF_cod is then coded by the entropy coder ( 504 ) into the output bitstream.
- contexts can be used to code the INF_cod. But these contexts shall only depend on data directly available from the parsing process, but not from data directly available after the decoding/reconstruction process. Hence, the parsing can be decoupled from the decoding process.
- FIG. 13 shows a simplified block diagram of the decoder side, for decoding the information INF_cur of a given block.
- the information INF_dec is decoded ( 602 ). This is the same as INF_cod of the encoder side in FIG. 12 .
- the parsing of this information is decoupled from the reconstruction/decoding process, by using contexts that only depend on data directly available from the parsing process, but not from data directly available after the decoding/reconstruction process.
- a first prediction ( 600 ) is performed, for example, the prediction of the motion vector to be used by the block by the FRUC mode.
- This prediction uses the decoded (or reconstructed) contextual information ( 605 ), for instance made of a list of candidate motion vectors from the neighborhood of the block of from blocks in reference pictures.
- a prediction of the information, INF_pred is then generated ( 601 ) from this first prediction.
- INF_pred and INF_cod are then re-combined (using the inverse of the combine operation performed at the encoder side) to generate the value of the information of the current block, INF_rec, that shall be the same as INF_cur of the encoder side.
- FIG. 14 shows the encoder (left) and decoder (right) for the IC flag, according to one embodiment.
- the first prediction is the prediction of the motion vector to be used for temporally predicting the block signal.
- the “Combine” and “inverse combine” operations simply consist of applying an “XOR” operator:
- FIG. 8 the example of the IC flag where the contextual CABAC coding is replaced by a regular coding of the predictor of the flag is shown.
- the predictor is then allowed to come from either of the spatial neighbors (solving issues A and C) or from the predictor block flag.
- FIG. 15 shows one embodiment of a method 1500 under the aspects described.
- the method commences at Start block 1501 and control proceeds to block 1510 for motion compensating a block of video data.
- the motion compensation process comprises block 1520 , 1530 and 1540 .
- the method proceeds to determine whether a current motion prediction mode is a temporal prediction mode. If not, derivation of information is made from motion predictors of the current block. If, however, the current motion prediction mode is temporal, control proceeds from block 1520 to block 1530 for deriving a coding parameter from a spatial neighbor. Control then proceeds from block 1530 to block 1540 for encoding a motion compensated block of video data.
- FIG. 16 shows one embodiment of a method 1600 under the aspects described.
- the method commences at Start block 1601 and control proceeds to block 1610 for decoding a piece of bitstream information and to block 1620 for predicting a piece of bitstream information based on contextual information.
- Control proceeds from blocks 1610 and 1620 to block 1630 for performing an inverse combining operation.
- Control proceeds from block 1630 to block 1640 for decoding a motion compensated block of video data using the piece of information.
- FIG. 17 shows one embodiment of an apparatus 1300 for coding or decoding a block of video data.
- the apparatus comprises Processor 1310 which has input and output ports and is in signal connectivity with Memory 1320 , also having input and output ports.
- the apparatus can execute any of the aforementioned method embodiments, or variations.
- processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.
- DSP digital signal processor
- ROM read-only memory
- RAM random access memory
- any switches shown in the figures are conceptual only. Their function can be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
- any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function.
- a method of predicting or inferring a piece of information for encoding or decoding is provided.
- a predictor from a spatial neighbor can be used, or from a reference picture.
- This procedure can be used in several ways, or used in various motion vector prediction modes, such as FRUC or AMP.
- Other embodiments provide other methods and combinations to derive or infer motion information to increase coding efficiency.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
- The following described aspects relate generally to the field of video compression, and more particularly to prediction and parsing.
- In the HEVC (High Efficiency Video Coding of International Telecommunication Union, ITU-T H.265) video compression standard, a picture is divided into so-called Coding Tree Units (CTU), whose size is typically 64×64, 128×128, or 256×256 pixels.
- Each CTU is represented by a Coding Tree in the compressed domain. This is a quad-tree division of the CTU, where each leaf is called a Coding Unit (CU.
- Motion compensated temporal prediction is employed to exploit the redundancy that exists between successive pictures of a video. To do so, a motion vector is associated with each prediction unit (PU). Each CU is then given some Intra or Inter prediction parameters (Prediction Info) and is spatially partitioned into one or more Prediction Units (PUs), each PU being assigned some prediction information. The Intra or Inter coding mode is assigned on the CU level.
- The following description is aimed at ideas solving prediction and parsing issues in digital video coding.
- The present embodiments aim to improve the compression efficiency in video encoders by decoupling mode inference and prediction as further explained herein.
- These and other drawbacks and disadvantages of the prior art are addressed by at least one of the described embodiments, which are directed to a method and apparatus for improving the compression efficiency of video encoders and decoders through decoupling mode inference and prediction.
- According to at least one general embodiment described herein, there is provided a method for coding a block of video data. The method comprises steps for motion compensating a block of video data, the motion compensating comprising: determining whether a current motion prediction mode for the block is a temporal motion prediction mode; deriving a coding parameter using information from a spatial neighbor when a temporal prediction mode is determined; and, encoding a block of video data, using said coding parameter.
- According to at least one general embodiment described herein, there is provided a method for decoding a block of video data. The method comprises steps for decoding a piece of information within a coded video bitstream; predicting the piece of information based on contextual information; performing an inverse combining function of said decoded piece of information and said prediction of said piece of information to generate a piece of information for a current block of video data; and, decoding the motion compensated block of video data.
- According to another embodiment, there is provided a method comprising determining whether the coding mode is ATMVP, STMVP or FRUC/unilateral; predicting an illumination compensation flag for the current block of video data from spatial neighboring illumination compensation flags when the determined coding mode is ATMVP, STMVP or FRUC/unilateral; using an illumination compensation flag of a predictor block if the determined coding mode is not ATMVP, STMVP or FRUC/unilateral; and, using the illumination compensation flag for coding the current block of video data.
- According to another embodiment, there is provided a method comprising determining whether the coding mode is TMVP, ATMVP or FRUC; predicting an illumination compensation flag for the current block of video data from spatial neighboring illumination compensation flags when the determined coding mode is TMVP, ATMVP or FRUC; using an illumination compensation flag of a predictor block if the determined coding mode is not TMVP, ATMVP or FRUC; and, using the illumination compensation flag for coding the current block of video data.
- According to another aspect described herein, there is provided apparatus for coding or decoding a block of video data, comprising: a memory, and a processor, configured to perform any of the aforementioned method embodiments with variations.
- According to another aspect described herein, there is provided a nontransitory computer readable storage medium containing data content generated according to the method of any one of the aforementioned method embodiments, or by the apparatus of any one of the aforementioned apparatus embodiments for playback using a processor.
- According to another aspect described herein, there is provided a signal comprising video data generated according to the method of any one of the aforementioned method embodiments for coding a block of video data, or by the apparatus of any one of the aforementioned apparatus embodiments for coding a block of video data, for playback using a processor.
- According to another aspect described herein, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of the aforementioned method embodiments.
- These and other aspects, features and advantages of the present principles will become apparent from the following detailed description of exemplary embodiments, which is to be read in connection with the accompanying drawings.
-
FIG. 1a shows an example of a coding tree unit and coding tree concepts to represent a compressed HEVC picture. -
FIG. 1b shows an example of division of a coding tree unit into coding units, prediction units, and transform units. -
FIG. 1c shows one example of a coded at a tree for a motion compensated block. -
FIG. 2 shows an example of spatial motion vector prediction. -
FIG. 3 shows an example of temporal motion vector prediction. -
FIG. 4 shows an example of a generic video compression scheme to which the present embodiments can be applied. -
FIG. 5 shows one example of a generic video decompression scheme to which the present embodiments can be applied. -
FIG. 6 shows an example block diagram of information prediction/inferring from spatial predictors. -
FIG. 7 shows an example block diagram for illumination compensation (IC) flag prediction/inference. -
FIG. 8 shows an example illustration of temporal motion prediction and spatial prediction of an IC flag. -
FIG. 9 shows another example of a block diagram for IC flag prediction/inference. -
FIG. 10 shows an example block diagram of another IC flag prediction/inference. -
FIG. 11 shows an example of sub-PU derivation process in STMVP (Spatial Temporal Motion Vector Prediction). -
FIG. 12 shows a block diagram of decoupled predictor and information coding. -
FIG. 13 shows a block diagram of decoupled predictor and information parsing/decoding. -
FIG. 14 shows a block diagram of decoupled predictor and flag coding. -
FIG. 15 shows one embodiment of an encoding method using the aspects described. -
FIG. 16 shows one embodiment of a decoding method using the aspects described. -
FIG. 17 shows one embodiment of an apparatus for encoding or decoding using the aspects described. - In the HEVC video compression standard, a picture is divided into so-called Coding Tree Units (CTU), with typical sizes of 64×64, 128×128, or 256×256 pixels.
- Each CTU is represented by a Coding Tree in the compressed domain. This is a quad-tree division of the CTU, where each leaf is called a Coding Unit (CU), as shown in the example in
FIG. 1 a. - Each CU is then given some Intra or Inter prediction parameters (Prediction Info). To do so, it is spatially partitioned into one or more Prediction Units (PUs), each PU being assigned some prediction information. The Intra or Inter coding mode is assigned on the CU level, as shown in the example of
FIG. 1 b. - According to the HEVC standard, Coding Units are also divided into so-called transform units, in a recursive way, following a “transform tree”. Thus, a transform tree is a quad-tree division of a coding unit, and transform units are the leaf of the transform tree. A transform unit encapsulates the square transform blocks of each picture component corresponding to a considered square spatial area. A transform block is a square block of samples in a single component, where the same transform is applied.
- New emerging video compression tools include a Coding Tree Unit representation in the compressed domain to represent picture data in a more flexible way in the compressed domain. The advantage of this flexible representation of the coding tree is that it provides increased compression efficiency compared to the CU/PU/TU arrangement of the HEVC standard.
- The domain of the described embodiments is video compression. For inter coding, prediction of a given block is obtained by a motion compensated block from a reference picture. Such prediction is built from a motion vector which can undergo motion transformation used to motion compensate the block. Optionally the prediction can be processed by a sample transformation, for example using the IC (Illumination Compensation) transformation. Such transformations may be activated or de-activated using a flag, either coded or inferred. When predicting/inferring the flag corresponding to such transformations for the current block, current solutions generally predict/infer the flag from the motion predictor block (the block from which the motion is predicted). The predictor block can be in the spatial neighbor of the current block (spatial motion predictor block), or it can be in a reference picture (temporal motion predictor block). However, for temporal motion predictor (like the ones in the ATMVP (Advanced Temporal Motion Vector Prediction), STMVP (Spatial Temporal Motion Vector Prediction) or FRUC (Frame Rate Up Conversion) modes of the JEM (Joint Exploration Model)), such flags might not be available.
- The encoding/decoding process described here is the one used in another application for an equi-rectangular projection predictor for omnidirectional video, but is applicable to other flags or syntax elements (like IC, or OBMC (overlapped block motion compensation) and IMV (initial motion vector)).
- The following description is aimed at ideas solving prediction and parsing issues in digital video coding. The issues are that for predicting or parsing an information, contextual data are required, but these contextual data may not always be available. Solutions are proposed to improve the prediction and parsing processes for such cases.
- The following are two of the solutions considered:
- First, the inference and predictor block for inter modes TMVP, ATMVP, STMVP, FRUC/unilateral can be decoupled. This proposed idea aims mostly at improving the prediction process of the information.
- Second, CABAC context can be removed. In this embodiment, decouple the CABAC context and a predictor block for inter coding to avoid the non-availability of the piece of information for the parsing. When a given piece of information, for example the IC flag, or symbol is coded or decoded using CABAC, contextual information from spatial neighbors may need to be available in order to select the CABAC context; however, in the decoder, this contextual information may be not directly available when the bitstream is parsed, but needs to be first decoded (or reconstructed) from the decoded data; therefore the parsing process becomes dependent on the decoding process, which is generally not desired nor recommended. To solve this issue, it is proposed to decouple the contextual CABAC encoding of the symbol from the decoding process of the contextual (spatial/temporal) neighboring information. A concrete implementation of this idea is to use neighboring blocks (e.g. Above and Left blocks) to predict the symbol. What is coded is the combination, for example by an “XOR” operator, of the actual symbol value and of the prediction value. Similarly, at a decoder, the decoded value is inverse combined with the prediction value, for example using XOR. The parsing of the decoded value may use CABAC contexts, but these contexts do not depend on neighboring information not yet available during the parsing. Hence, the coded/decoded value is not directly linked to the contextual (spatial/temporal) neighboring information related to the symbol.
- If the combining function is
-
c=f(v,p) - v being the value of INF, p being the prediction, c the result of the combining function, the inverse combining function g is such that:
-
g(c,p)=v - At an encoder, c is encoded. At a decoder c is decoded (parsed), and v is reconstructed by computing g(c, p). When f is chosen as the XOR operator, g is simply also XOR. This works when INF is a flag.
- Another idea aims at addressing the parsing issue when decoding the information. At least one embodiment consists in decoupling the inference/prediction of a flag from the predictor block in inter motion compensation. Some advantages in this approach are that it increases the number of cases where a predictor for the flag is available and/or improves the accuracy of the prediction, and therefore coding, for the given flag.
- The domain of the described embodiments is the video compression. For inter coding, prediction of a given block (or coding unit, also noted CU) is given by a motion compensated block. Such prediction is built from a motion vector which can undergo transformation used to motion compensate the block. Optionally the prediction can be processed by a transformation, for example the IC (llumination Compensation) transformation.
- The motion compensation tools in existing codecs such as H.265 (ITU-T, 2013) are based on the description of the motion of a current block between the current frame and a reference frame, which has already been decoded. This motion is classically represented with a 2D vector and a reference frame index, which informs the decoder on which reference frame to select among a set of previously decoded frames stored in the reference pictures buffer.
- In order to predict the block, several pieces of information can be coded or inferred for the current block. The following list refers to the notations of
FIG. 1c , which depicts an example of data coding tree for an inter block: -
- A. Motion vector prediction index (IDX or MV INFO)
- B. Refinement of the motion vector predictor (MERGE, FRUC)
- C. Refinement of the prediction (FRUC, BIO)
- D. Direction of motion compensation (uni- or bi-directional, indicated in DIR)
- E. Type of motion compensation (AFFINE)
- F. Sub-block motion compensation (OBMC)
- G. Illumination Compensation of the block (IC)
- H. Integer Motion Vector (IMV)
- I. When coding or inferring some of these information, a common method is to predict/infer the information from the block used to predict the motion vector.
- J. For example, in
FIG. 2 , the motion vector is spatially predicted from neighboring blocks. The current block P uses the spatial neighbor block V for the motion prediction: the motion vector dP of block P is predicted from the motion vector dV of block V. Similarly, other information (like IC in MERGE mode) from the block V can be used to predict or infer the corresponding information for the block P. - K. In another example, in
FIG. 3 , the motion vector is temporally predicted from blocks from a reference picture. The current block P uses the block W, located in a reference picture, for the motion prediction: the motion vector dP of block P is predicted from the motion vector dW of block W. Similarly, other information (like IC or parameters values) from the block W can be used to predict or infer the corresponding information for the block P. For instance, in the JEM, different modes are temporal motion predictors: TMVP or ATMVP or STMVP or FRUC/unilateral. - L. Of course, this requires that the information is properly stored in the blocks used to predict the information of the block P.
-
FIG. 4 andFIG. 5 depict classical video encoders and decoders, respectively. The current embodiments concern two points. The embodiments change the way of predicting or inferring syntax elements or parameters of a motion compensated block (i.e. a non intra block), i.e. mainly impacting 170 inFIGS. 4 and 275 inFIG. 5 . They also relate to the parsing of a video stream, and therefore impacts block 145 ofFIG. 4 and 230 ofFIG. 5 . - With the aforementioned prior-art prediction or inferring method described above, four issues can arise.
-
- A. Non-availability of the Information for the prediction—For some prediction modes of the motion vector, like the temporal prediction in TMVP or ATMVP or STMVP or FRUC/unilateral, the information might be not available anymore in the blocks of the decoded reference picture. Typically, in order to reduce the memory needs, only the information related to the motion vector(s) and reference frame are stored in reference pictures buffer in HEVC or in the JEM. The other information, such as the IC flag and parameters, the Affine flag and parameters, are not stored. For instance, in
FIG. 3 , only dW is stored for block W, but the IC flag is not stored and cannot be used to predict the IC flag of the current block P. - B. Non-availability of the Information for the parsing—When a given information (for example the IC flag) or syntax element is coded/decoded using CABAC, contextual information from spatial neighbors may need to be available in order to select the CABAC context; however, in the decoder, this contextual information may be not directly available when the bitstream is parsed, but needs to be first decoded (or reconstructed) from the decoded data; therefore the parsing process becomes dependent on the decoding process, which is generally not desired nor recommended; usually, it is widely preferred to be perform the parsing process independently from the decoding/reconstruction process.
- A. Non-availability of the Information for the prediction—For some prediction modes of the motion vector, like the temporal prediction in TMVP or ATMVP or STMVP or FRUC/unilateral, the information might be not available anymore in the blocks of the decoded reference picture. Typically, in order to reduce the memory needs, only the information related to the motion vector(s) and reference frame are stored in reference pictures buffer in HEVC or in the JEM. The other information, such as the IC flag and parameters, the Affine flag and parameters, are not stored. For instance, in
- This problematic case can, for example, happen when a syntax element (SE) being parsed depends on contexts depending on neighboring coding units (CUs) of the current block, that are coded with Merge mode; when a CU is coded with Merge mode, all its parameters are deduced but not explicitly coded in the stream; therefore, during the parsing of the stream, these parameters are not yet known; and they cannot be used for parsing the syntax element SE.
-
- C. Low correlation of the Information—In case of FRUC predictor, the information of the chosen candidate (the predictor block) might not be completely correlated with the information of the current block; this is notably because the block is selected based on a SAD (Sum of Absolute Differences) criteria.
- D. Non-unicity of the Information inside a coded block/CU—In case of ATMVP and STMVP, several flags (or syntax elements) from neighboring blocks might be used to predict or infer the flag (or syntax element) of the current block. In ATMVP, an initial “temporal” motion vector predictor is computed from neighboring blocks and then refined for all sub-PU blocks where another motion vector predictor (with different flag value) might be finally used. Moreover, in STMVP, several motion vector predictors are averaged from several blocks, leading to several (potentially different) flag inference.
In current implementation of the JEM (Joint Exploration Model), two methods are used to avoid these issues: - Case A: When using temporal inference (for deriving the IC flag for example), additional memory is used to store the IC flag in the reference picture buffer.
- Case B: The flag never uses spatial context for encoding (for example IC flag). It increases the coding cost of the flag since the entropic coding does not use the spatial context. Case B: The flag is never inferred (like for IMV flag). It might decrease the performance by adding information to encode.
- Case C: for IC, the flag is not inferred but explicitly coded. This might decrease the performance by adding information to encode.
- Case D: for IC, the flag is inferred from the initial temporal motion vector predictor in ATMVP and propagates to all the sub-PU blocks. In STMVP, the flag is copied from the ATMVP block. This might decrease the performance by inferring a flag badly correlated with the current block mode.
- In the following, it is considered that a given piece of information, which can be referred to as INF (which can be, for example, the value of the IC flag), of the current block has to be predicted or inferred from predictor blocks, either located in the spatial neighbor of the current block, co-located in reference pictures or in a FRUC/unilateral (frame rate up conversion) motion field.
- The described embodiments cover at least two separate aspects. The first aspect is forcing the prediction of the information INF to be based on spatial neighbors—this point is covered at least in the first two embodiments. The second aspect is parsing the information INF using decoupled predictors—this point is covered at least by the third embodiment.
- In a first general embodiment, the information INF inference/prediction for motion predicted blocks can be decoupled.
- In order to solve issue A and D, the embodiment proposes to predict/infer the information INF from spatial neighbors even when the motion predictor of the current block is built using a temporal prediction, for instance using ATMVP, STMVP or FRUC/unilateral modes. The concept is shown in the block diagram of
FIG. 6 . The input of the prediction/inferring process of INF is the motion prediction mode M of the current block. First, a checking step is applied to evaluate if the mode M corresponds to a temporal prediction or a spatial prediction (step 201). If M corresponds to a spatial prediction, the prediction/inferring of INF is made using the corresponding information from the spatial predictor blocks used to prediction M (step 202). If M corresponds to a temporal prediction, the prediction/inferring of INF is made using the corresponding information from spatially neighboring predictor blocks, not used to prediction M and therefore different from the blocks used to predict M (step 203). - A first alternative to the first embodiment involves the application to ATMVP, STMVP or FRUC/unilateral modes.
- The block diagram of
FIG. 7 shows the implementation of this process for INF corresponding to the flag IC. Instep 201, the mode M is compared to the ATMVP, STMVP or FRUC/unilateral modes. If M is one of these modes, the IC flag of the current block is spatially predicted/inferred from the TOP and LEFT block IC flags when the current motion predictor block is coming from ATMVP, STMVP or FRUC/unilateral candidates. This corresponds to step 203. If M is none of the ATMVP, STMVP or FRUC/unilateral modes, the IC value of the predictor block is used. This corresponds to step 202. - The concept of this embodiment, corresponding to the block diagram of
FIG. 7 , is also explained inFIG. 8 .FIG. 8 illustrates how the process ofFIG. 7 applies for deriving the motion vector and the IC flag of a block. The motion vector dP and reference index rP of block P in the current picture are temporally predicted by the motion vector dW and reference index rW of block W in the reference picture. However, the IC flag IC_P of block P is predicted from the IC flag IC_V of a spatially neighboring block V. - A second alternative to the first embodiment involves the application to ATMVP, STMVP or FRUC modes.
- In order to solve issue A and C, we propose to predict/infer the information INF from spatial neighbors when the motion predictor of the current block is ATMVP, STMVP or FRUC (
FIG. 9 ). - A third alternative of the first general embodiment involves the use of spatial predictors for IC flag.
- In order to solve issue A, B and C, we propose to always predict/infer the information INF from spatial neighbors.
- In a second general embodiment, there is a single INF prediction for all sub-PUs in STMVP mode.
- In STMVP, the motion vectors of the sub-PUs are derived recursively, following raster scan order. This is illustrated in
FIG. 11 , where a CU contains four sub-Pus (or sub-CUs), noted A, B, C, D. The CU has fours neighboring blocks a, b, c, d. - The motion derivation for sub-PU A uses the motion from the two spatial neighbors, b and c, plus the temporal motion vector predictor (TMVP) of sub-block A. Finally, a merging of the (up to three) motion vectors is achieved (e.g. by averaging them). The same process then applies to block B, with spatial neighbors A, d and the temporal motion vector predictor (TMVP) of sub-block B. And so on for the two remaining blocks of the CU.
- In one embodiment, in order to solve D, the information INF is deduced from the neighboring PU at block level (not sub-PU) using above methods. One single prediction of INF is used for all sub-PUs of the current block. The derivation of INF therefore differs from the derivation of the motion vectors, for which the recursive STMVP process is maintained. The same information INF is then available for all sub-PUs.
- A third general embodiment involves flag coding using decoupled predictor.
- In order to solve issue B, but also to improve the coding efficiency of an information INF, linked to the current block, coded in a mode that involves cascading of several prediction processes (as for instance the prediction of the motion vector by the FRUC mode), we propose to decouple the contextual CABAC encoding of INF from the decoding process of the contextual (spatial/temporal) neighboring information. The point here is to use a predictor for INF which depends on another predictor (for example in case of IC flag for a current block being coded in FRUC mode, the IC predictor depends on the IC flag of the predictor selected by FRUC). In the prior-art, in such cases of prediction cascading, no predictor is used for coding INF.
-
FIG. 12 shows a simplified block diagram of the encoder side, for coding the information INF_cur of a given block. When coding this information, contextual information (505) is available, from previously encoded blocks in the neighborhood of the current block or in reference pictures. For instance, it consists in a list of candidate motion vectors from the neighborhood of the block of from blocks in reference pictures. From this contextual information, a first prediction (500) is performed, for instance the prediction of the motion vector to be used by the block by the FRUC mode. From this first prediction, a predictor of INF_cur, noted INF_pred, is generated (501). The encoding process also performs the choice of the actual value of the information INF, typically based on rate-distortion optimization (502). INF_pred and INF_cur are then combined to generate the value of the information to be coded, INF_cod. INF_cod is then coded by the entropy coder (504) into the output bitstream. Instep 504 of entropy coding, contexts can be used to code the INF_cod. But these contexts shall only depend on data directly available from the parsing process, but not from data directly available after the decoding/reconstruction process. Hence, the parsing can be decoupled from the decoding process. -
FIG. 13 shows a simplified block diagram of the decoder side, for decoding the information INF_cur of a given block. From the bitstream, the information INF_dec is decoded (602). This is the same as INF_cod of the encoder side inFIG. 12 . The parsing of this information is decoupled from the reconstruction/decoding process, by using contexts that only depend on data directly available from the parsing process, but not from data directly available after the decoding/reconstruction process. - A first prediction (600) is performed, for example, the prediction of the motion vector to be used by the block by the FRUC mode. This prediction uses the decoded (or reconstructed) contextual information (605), for instance made of a list of candidate motion vectors from the neighborhood of the block of from blocks in reference pictures. A prediction of the information, INF_pred, is then generated (601) from this first prediction. INF_pred and INF_cod are then re-combined (using the inverse of the combine operation performed at the encoder side) to generate the value of the information of the current block, INF_rec, that shall be the same as INF_cur of the encoder side.
-
FIG. 14 shows the encoder (left) and decoder (right) for the IC flag, according to one embodiment. In this embodiment, the first prediction is the prediction of the motion vector to be used for temporally predicting the block signal. And the “Combine” and “inverse combine” operations simply consist of applying an “XOR” operator: -
IC_cod = IC_cur XOR IC_pred IC_rec = IC_dec XOR IC_pred - In
FIG. 8 , the example of the IC flag where the contextual CABAC coding is replaced by a regular coding of the predictor of the flag is shown. The predictor is then allowed to come from either of the spatial neighbors (solving issues A and C) or from the predictor block flag. -
FIG. 15 shows one embodiment of amethod 1500 under the aspects described. The method commences atStart block 1501 and control proceeds to block 1510 for motion compensating a block of video data. The motion compensation process comprisesblock Block 1520, the method proceeds to determine whether a current motion prediction mode is a temporal prediction mode. If not, derivation of information is made from motion predictors of the current block. If, however, the current motion prediction mode is temporal, control proceeds fromblock 1520 to block 1530 for deriving a coding parameter from a spatial neighbor. Control then proceeds fromblock 1530 to block 1540 for encoding a motion compensated block of video data. -
FIG. 16 shows one embodiment of amethod 1600 under the aspects described. The method commences atStart block 1601 and control proceeds to block 1610 for decoding a piece of bitstream information and to block 1620 for predicting a piece of bitstream information based on contextual information. Control proceeds fromblocks block 1630 to block 1640 for decoding a motion compensated block of video data using the piece of information. -
FIG. 17 shows one embodiment of an apparatus 1300 for coding or decoding a block of video data. The apparatus comprises Processor 1310 which has input and output ports and is in signal connectivity with Memory 1320, also having input and output ports. The apparatus can execute any of the aforementioned method embodiments, or variations. - The functions of the various elements shown in the figures can be provided using dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.
- Other hardware, conventional and/or custom, can also be included. Similarly, any switches shown in the figures are conceptual only. Their function can be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
- The present description illustrates the present ideas. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present ideas and are included within its scope.
- All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the present principles and the concepts contributed by the inventor(s) to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
- Moreover, all statements herein reciting principles, aspects, and embodiments of the present principles, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
- Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the present principles. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which can be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
- In the claims herein, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function.
- The present principles as defined by such claims reside in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.
- Reference in the specification to “one embodiment” or “an embodiment” of the present principles, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.
- In conclusion, a method of predicting or inferring a piece of information for encoding or decoding is provided. Instead of predicting or inferring the information, such as a flag for a current block, a predictor from a spatial neighbor can be used, or from a reference picture. This procedure can be used in several ways, or used in various motion vector prediction modes, such as FRUC or AMP. Other embodiments provide other methods and combinations to derive or infer motion information to increase coding efficiency.
Claims (15)
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP17306332.2A EP3468194A1 (en) | 2017-10-05 | 2017-10-05 | Decoupled mode inference and prediction |
EP17306332.2 | 2017-10-05 | ||
EP17306332 | 2017-10-05 | ||
PCT/US2018/054315 WO2019070941A1 (en) | 2017-10-05 | 2018-10-04 | Decoupled mode inference and prediction |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2018/054315 A-371-Of-International WO2019070941A1 (en) | 2017-10-05 | 2018-10-04 | Decoupled mode inference and prediction |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/368,247 Division US20240007665A1 (en) | 2017-10-05 | 2023-09-14 | Decoupled mode inference and prediction |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200236390A1 true US20200236390A1 (en) | 2020-07-23 |
US11805271B2 US11805271B2 (en) | 2023-10-31 |
Family
ID=60119969
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/652,813 Active US11805271B2 (en) | 2017-10-05 | 2018-10-04 | Decoupled mode inference and prediction |
US18/368,247 Pending US20240007665A1 (en) | 2017-10-05 | 2023-09-14 | Decoupled mode inference and prediction |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/368,247 Pending US20240007665A1 (en) | 2017-10-05 | 2023-09-14 | Decoupled mode inference and prediction |
Country Status (4)
Country | Link |
---|---|
US (2) | US11805271B2 (en) |
EP (2) | EP3468194A1 (en) |
CN (1) | CN111386706B (en) |
WO (1) | WO2019070941A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210360275A1 (en) * | 2019-02-01 | 2021-11-18 | Huawei Technologies Co., Ltd. | Inter prediction method and apparatus |
US20220303526A1 (en) * | 2019-06-12 | 2022-09-22 | lnterDigital VC Holdings, lnc. | Illumination compensation for video encoding and decoding |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI722467B (en) * | 2018-06-29 | 2021-03-21 | 大陸商北京字節跳動網絡技術有限公司 | Video processing method, device of video system and product of computer program |
WO2020116242A1 (en) * | 2018-12-07 | 2020-06-11 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | Encoding device, decoding device, encoding method, and decoding method |
Citations (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070177672A1 (en) * | 2006-01-12 | 2007-08-02 | Lg Electronics Inc. | Processing multiview video |
US20070177671A1 (en) * | 2006-01-12 | 2007-08-02 | Lg Electronics Inc. | Processing multiview video |
US20090003455A1 (en) * | 2007-06-26 | 2009-01-01 | Do-Young Joung | Method and apparatus for illumination compensation in multi-view video coding |
US20090010340A1 (en) * | 2007-06-25 | 2009-01-08 | Do-Young Joung | Method and apparatus for illumination compensation in multi-view video coding |
US20100098157A1 (en) * | 2007-03-23 | 2010-04-22 | Jeong Hyu Yang | method and an apparatus for processing a video signal |
US20150326881A1 (en) * | 2012-12-26 | 2015-11-12 | Sharp Kabushiki Kaisha | Image decoding device |
US20160021393A1 (en) * | 2013-04-12 | 2016-01-21 | Mediatek Singapore Pte. Ltd. | Method of Error-Resilient Illumination Compensation for Three- Dimensional Video Coding |
US20160366416A1 (en) * | 2015-06-09 | 2016-12-15 | Qualcomm Incorporated | Systems and methods of determining illumination compensation status for video coding |
US20160366415A1 (en) * | 2015-06-09 | 2016-12-15 | Qualcomm Incorporated | Systems and methods of determining illumination compensation parameters for video coding |
US20170339404A1 (en) * | 2016-05-17 | 2017-11-23 | Arris Enterprises Llc | Template matching for jvet intra prediction |
US20170339405A1 (en) * | 2016-05-20 | 2017-11-23 | Arris Enterprises Llc | System and method for intra coding |
US20170347128A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Binary ternary quad tree partitioning for jvet |
US20170347095A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Jvet quadtree plus binary tree (qtbt) structure with multiple asymmetrical partitioning |
US20170347102A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Weighted angular prediction for intra coding |
US20170347096A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | General block partitioning method |
US20170347103A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Weighted angular prediction coding for intra coding |
US20170347123A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Jvet coding block structure with asymmetrical partitioning |
US20170347093A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Coding Weighted Angular Prediction for Intra Coding |
US20180063531A1 (en) * | 2016-08-26 | 2018-03-01 | Qualcomm Incorporated | Unification of parameters derivation procedures for local illumination compensation and cross-component linear model prediction |
US20180098072A1 (en) * | 2016-09-30 | 2018-04-05 | Qualcomm Incorporated | Memory and bandwidth reduction of stored data in image/video coding |
US20180098086A1 (en) * | 2016-10-05 | 2018-04-05 | Qualcomm Incorporated | Systems and methods of performing improved local illumination compensation |
US20180098066A1 (en) * | 2016-10-05 | 2018-04-05 | Qualcomm Incorporated | Systems and methods of switching interpolation filters |
US20180176587A1 (en) * | 2016-12-21 | 2018-06-21 | Arris Enterprises Llc | Constrained position dependent intra prediction combination (pdpc) |
US20180310017A1 (en) * | 2017-04-21 | 2018-10-25 | Mediatek Inc. | Sub-prediction unit temporal motion vector prediction (sub-pu tmvp) for video coding |
US20180352223A1 (en) * | 2017-05-31 | 2018-12-06 | Mediatek Inc. | Split Based Motion Vector Operation Reduction |
US20180359483A1 (en) * | 2017-06-13 | 2018-12-13 | Qualcomm Incorporated | Motion vector prediction |
US20190200021A1 (en) * | 2016-09-22 | 2019-06-27 | Lg Electronics Inc. | Illumination compensation-based inter-prediction method and apparatus in image coding system |
US20190230376A1 (en) * | 2018-01-25 | 2019-07-25 | Qualcomm Incorporated | Advanced motion vector prediction speedups for video coding |
US20190238882A1 (en) * | 2016-09-22 | 2019-08-01 | Lg Electronics Inc. | Inter-prediction method and device in image coding system |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7086026B2 (en) | 2003-05-12 | 2006-08-01 | International Business Machines Corporation | Decoupling capacitance analysis method |
US8345070B2 (en) | 2009-06-10 | 2013-01-01 | Himax Media Solutions, Inc. | Apparatus and method for frame rate up conversion |
US9083983B2 (en) | 2011-10-04 | 2015-07-14 | Qualcomm Incorporated | Motion vector predictor candidate clipping removal for video coding |
US9860529B2 (en) | 2013-07-16 | 2018-01-02 | Qualcomm Incorporated | Processing illumination compensation for video coding |
CN108632629B9 (en) | 2014-03-19 | 2021-06-15 | 株式会社Kt | Method of generating merge candidate list for multi-view video signal and decoding apparatus |
CN106454378B (en) * | 2016-09-07 | 2019-01-29 | 中山大学 | Converting video coding method and system in a kind of frame per second based on amoeboid movement model |
-
2017
- 2017-10-05 EP EP17306332.2A patent/EP3468194A1/en not_active Withdrawn
-
2018
- 2018-10-04 CN CN201880075057.4A patent/CN111386706B/en active Active
- 2018-10-04 WO PCT/US2018/054315 patent/WO2019070941A1/en unknown
- 2018-10-04 EP EP18786930.0A patent/EP3692718A1/en active Pending
- 2018-10-04 US US16/652,813 patent/US11805271B2/en active Active
-
2023
- 2023-09-14 US US18/368,247 patent/US20240007665A1/en active Pending
Patent Citations (30)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070177672A1 (en) * | 2006-01-12 | 2007-08-02 | Lg Electronics Inc. | Processing multiview video |
US20070177671A1 (en) * | 2006-01-12 | 2007-08-02 | Lg Electronics Inc. | Processing multiview video |
US20100098157A1 (en) * | 2007-03-23 | 2010-04-22 | Jeong Hyu Yang | method and an apparatus for processing a video signal |
US20090010340A1 (en) * | 2007-06-25 | 2009-01-08 | Do-Young Joung | Method and apparatus for illumination compensation in multi-view video coding |
US20090003455A1 (en) * | 2007-06-26 | 2009-01-01 | Do-Young Joung | Method and apparatus for illumination compensation in multi-view video coding |
US20150326881A1 (en) * | 2012-12-26 | 2015-11-12 | Sharp Kabushiki Kaisha | Image decoding device |
US20160021393A1 (en) * | 2013-04-12 | 2016-01-21 | Mediatek Singapore Pte. Ltd. | Method of Error-Resilient Illumination Compensation for Three- Dimensional Video Coding |
US10110922B2 (en) * | 2013-04-12 | 2018-10-23 | Hfi Innovation Inc. | Method of error-resilient illumination compensation for three-dimensional video coding |
US20160366416A1 (en) * | 2015-06-09 | 2016-12-15 | Qualcomm Incorporated | Systems and methods of determining illumination compensation status for video coding |
US20160366415A1 (en) * | 2015-06-09 | 2016-12-15 | Qualcomm Incorporated | Systems and methods of determining illumination compensation parameters for video coding |
US20170339404A1 (en) * | 2016-05-17 | 2017-11-23 | Arris Enterprises Llc | Template matching for jvet intra prediction |
US20170339405A1 (en) * | 2016-05-20 | 2017-11-23 | Arris Enterprises Llc | System and method for intra coding |
US20170347096A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | General block partitioning method |
US20170347128A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Binary ternary quad tree partitioning for jvet |
US20170347095A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Jvet quadtree plus binary tree (qtbt) structure with multiple asymmetrical partitioning |
US20170347103A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Weighted angular prediction coding for intra coding |
US20170347123A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Jvet coding block structure with asymmetrical partitioning |
US20170347093A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Coding Weighted Angular Prediction for Intra Coding |
US20170347102A1 (en) * | 2016-05-25 | 2017-11-30 | Arris Enterprises Llc | Weighted angular prediction for intra coding |
US20180063531A1 (en) * | 2016-08-26 | 2018-03-01 | Qualcomm Incorporated | Unification of parameters derivation procedures for local illumination compensation and cross-component linear model prediction |
US20190238882A1 (en) * | 2016-09-22 | 2019-08-01 | Lg Electronics Inc. | Inter-prediction method and device in image coding system |
US20190200021A1 (en) * | 2016-09-22 | 2019-06-27 | Lg Electronics Inc. | Illumination compensation-based inter-prediction method and apparatus in image coding system |
US20180098072A1 (en) * | 2016-09-30 | 2018-04-05 | Qualcomm Incorporated | Memory and bandwidth reduction of stored data in image/video coding |
US20180098066A1 (en) * | 2016-10-05 | 2018-04-05 | Qualcomm Incorporated | Systems and methods of switching interpolation filters |
US20180098086A1 (en) * | 2016-10-05 | 2018-04-05 | Qualcomm Incorporated | Systems and methods of performing improved local illumination compensation |
US20180176587A1 (en) * | 2016-12-21 | 2018-06-21 | Arris Enterprises Llc | Constrained position dependent intra prediction combination (pdpc) |
US20180310017A1 (en) * | 2017-04-21 | 2018-10-25 | Mediatek Inc. | Sub-prediction unit temporal motion vector prediction (sub-pu tmvp) for video coding |
US20180352223A1 (en) * | 2017-05-31 | 2018-12-06 | Mediatek Inc. | Split Based Motion Vector Operation Reduction |
US20180359483A1 (en) * | 2017-06-13 | 2018-12-13 | Qualcomm Incorporated | Motion vector prediction |
US20190230376A1 (en) * | 2018-01-25 | 2019-07-25 | Qualcomm Incorporated | Advanced motion vector prediction speedups for video coding |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210360275A1 (en) * | 2019-02-01 | 2021-11-18 | Huawei Technologies Co., Ltd. | Inter prediction method and apparatus |
US20220303526A1 (en) * | 2019-06-12 | 2022-09-22 | lnterDigital VC Holdings, lnc. | Illumination compensation for video encoding and decoding |
Also Published As
Publication number | Publication date |
---|---|
US11805271B2 (en) | 2023-10-31 |
CN111386706A (en) | 2020-07-07 |
WO2019070941A1 (en) | 2019-04-11 |
CN111386706B (en) | 2024-04-19 |
EP3692718A1 (en) | 2020-08-12 |
US20240007665A1 (en) | 2024-01-04 |
EP3468194A1 (en) | 2019-04-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20240073417A1 (en) | Blending method for inter prediction with geometry partition | |
US11310526B2 (en) | Hardware friendly constrained motion vector refinement | |
US10715827B2 (en) | Multi-hypotheses merge mode | |
US11115653B2 (en) | Intra block copy merge list simplification | |
US11671586B2 (en) | Modified history based motion prediction | |
US11553173B2 (en) | Merge candidates with multiple hypothesis | |
JP7478864B2 (en) | History-Based Motion Candidate List Construction for Intra Block Copy | |
US20240007665A1 (en) | Decoupled mode inference and prediction | |
KR101422422B1 (en) | System and method for enhanced dmvd processing | |
US11240524B2 (en) | Selective switch for parallel processing | |
US20240080490A1 (en) | Signaling for multi-reference line prediction and multi-hypothesis prediction | |
WO2020156464A1 (en) | Method and apparatus of combined inter and intraprediction for video coding | |
US20200137413A1 (en) | Error resilience and parallel processing for decoder side motion vector derivation | |
US20210044821A1 (en) | Illumination compensation flag in frame rate up-conversion with template matching | |
US20230412793A1 (en) | Inter-prediction on non-dyadic blocks |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: INTERDIGITAL VC HOLDINGS. INC., DELAWARE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GALPIN, FRANCK;ROBERT, ANTOINE;URBAN, FABRICE;SIGNING DATES FROM 20190214 TO 20190226;REEL/FRAME:052933/0464 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP, ISSUE FEE PAYMENT VERIFIED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |