GB2585017A - Video coding and decoding - Google Patents

Video coding and decoding Download PDF

Info

Publication number
GB2585017A
GB2585017A GB1909041.4A GB201909041A GB2585017A GB 2585017 A GB2585017 A GB 2585017A GB 201909041 A GB201909041 A GB 201909041A GB 2585017 A GB2585017 A GB 2585017A
Authority
GB
United Kingdom
Prior art keywords
flag
mode
merge
prediction mode
skip
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB1909041.4A
Other versions
GB201909041D0 (en
Inventor
Laroche Guillaume
Onno Patrice
Gisquet Christophe
Taquet Jonathan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Priority to GB1909041.4A priority Critical patent/GB2585017A/en
Publication of GB201909041D0 publication Critical patent/GB201909041D0/en
Priority to GB1912356.1A priority patent/GB2591721A/en
Publication of GB2585017A publication Critical patent/GB2585017A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51Motion estimation or motion compensation
    • H04N19/513Processing of motion vectors
    • H04N19/517Processing of motion vectors by encoding
    • H04N19/52Processing of motion vectors by encoding by predictive encoding
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103Selection of coding mode or of prediction mode
    • H04N19/109Selection of coding mode or of prediction mode among a plurality of temporal predictive coding modes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103Selection of coding mode or of prediction mode
    • H04N19/11Selection of coding mode or of prediction mode among a plurality of spatial predictive coding modes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/157Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/157Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
    • H04N19/159Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/96Tree coding, e.g. quad-tree coding

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

A method of decoding an image or image portion from a bit stream is disclosed; the method comprising; decoding a Skip flag from the bitstream; if said flag is enabled, setting the mode to Skip Regular; if said flag is disabled, determining a prediction mode and decoding corresponding data. Preferably a motion predictor merge mode is indicated for the prediction mode as well as a zero residual. Examples of suitable merge modes being SubBlock, Motion Vector Difference (MMVD), triangle or Combined inter/intra Prediction (CIIP). Preferably the zero residual is not indicated for Regular merge mode. If the skip flag is enabled, an Intra Block Copy (IBC) flag may be decoded if the Luma size of the current coding unit (CU) is not 128x128. There is an additional invention which comprises coding/decoding an image where a plurality of flags are used to indicate respective prediction modes.

Description

VIDEO CODING AND DECODING
Field of invention
The present invention relates to video coding and decoding. Background Recently, the Joint Video Experts Team (IVET), a collaborative team formed by MPEG and ITU-T Study Group 16's VCEG, commenced work on a new video coding standard referred to as Versatile Video Coding (VVC). The goal of VVC is to provide significant improvements in compression performance over the existing HEVC standard (i.e., typically twice as much as before) and to be completed in 2020. The main target applications and services include -but not limited to -360-degree and high-dynamic-range (HDR) videos. In total, JVET evaluated responses from 32 organizations using formal subjective tests conducted by independent test labs. Some proposals demonstrated compression efficiency gains of typically 40% or more when compared to using HEVC. Particular effectiveness was shown on ultra-high definition (UED) video test material. Thus, we may expect compression efficiency gains well-beyond the targeted 50% for the final standard.
The JVET exploration model (JEM) uses all the HEVC tools, as well as various new motion prediction tools which can be enabled or disabled by flags. This generates additional flags signalling when one or more Merge modes are disabled at Sequence Parameter Sets (SPS) level or Slice level. These signalled flags unnecessarily increase the bitrate.
Accordingly, a solution to at least one of the aforementioned problems is desirable.
According to one aspect of the present invention, there is provided a method of decoding an image or image portion from a bit stream; the method comprising: decoding a Skip flag from the bitstream; if said flag is enabled, setting the mode to Skip Regular; if said flag is disabled, determining a prediction mode and decoding corresponding data. One advantage of 25 this method is that only one bit is needed to signal the Regular Merge Skip which is the most probable mode.
Optionally, the method further comprises indicating a motion predictor merge mode for said determined prediction mode.
Optionally, the method further comprises indicating a zero residual for said determined motion predictor mode.
Optionally, the method further comprises not indicating the zero residual for Regular merge mode among said determined motion merge predictor modes.
Optionally, the determined motion predictor mode comprises one of SubBlock, Motion Vector Difference (MMVD), Triangle or Combined Inter/Intra Prediction (CIIP).
Optionally, the method further comprises decoding an Intra Block Copy (IBC) flag after determining said prediction mode.
Optionally, if said skip flag is enabled, an Infra Block Copy (IBC) flag is decoded from the bitstream if the Luma size of the current coding unit (CU) is not 128x128.
Optionally, decoding the data comprises determining whether the prediction mode is Regular and setting a coding block flag in dependence on said determining.
Optionally, decoding the data comprises determining whether the prediction mode is Regular or Infra Block Copy (IBC) and setting a coding block flag in dependence on said determining.
Optionally, decoding the data comprises determining whether the prediction mode is Regular, Combined Inter/Intra Prediction (CIIP) or IBC (2018) and setting a coding block flag in dependence on said determining.
Optionally, in the case that the prediction mode is one of said modes inferring the coding block flag.
Optionally, in the case that the prediction mode is not one of said modes decoding the coding block flag (2008) from the bitstream.
Optionally, the method further comprises determining a merge flag, and said coding block flag has a context depending on said merge flag.
Optionally, if said skip flag is disabled, the method further comprises setting a transform unit coding block flag to zero so as to indicate zero residual.
Optionally, if said skip flag is disabled, the method further comprises setting sig coeff so as to indicate zero residual for said determined merge modes.
Optionally, if said skip flag is disabled, the method further comprises setting a second skip flag so as to indicate zero residual for said determined merge mode.
Optionally, the method further comprises determining a merge flag before said determining said prediction mode.
Optionally, determining said prediction mode comprises determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.
According to another aspect of the present invention there is provided a method of decoding an image or image portion from a bit stream, the bit stream comprising a plurality of flags, each flag indicating a respective motion prediction mode; the method comprising: determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying Ting a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.
Optionally, the first mode is regular mode.
Optionally, said determining comprises verifying a condition relating to at least one of all of the subsequent motion prediction modes.
Optionally, the regular mode is set at the last position of the list of Merge modes when a coding unit is Merge. This enables the condition to decode a flag to depend only on the availability of the related mode and not on the other Merge modes.
Encoder According to another aspect of the present invention there is provided a method of encoding an image or image portion into a bit stream; the method comprising: encoding a Skip flag into the bitstream; if said flag is enabled, setting the mode to Skip Regular; if said flag is disabled, determining a prediction mode and encoding corresponding data.
According to another aspect of the present invention there is provided a method of encoding an image or image portion into a bit stream, the bit stream comprising a plurality of flags, each flag indicating a respective motion prediction mode; the method comprising: determining whether to encode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.
Optionally, determining a motion prediction mode comprises determining a mode with the lowest rate distortion.
Optionally, determining a mode with the lowest rate distortion comprises evaluating a coding block flag corresponding to said motion prediction mode.
Optionally, the method further comprises setting the coding block flag value equal to the coding block flag value of a current BestCu.
Optionally, said determined motion prediction mode is Regular and the rate distortion is estimated based on a Merge idx corresponding to the Regular mode.
Optionally, said determined motion prediction mode is IBC and the rate distortion is estimated based on a Merge idx corresponding to the IBC mode Device According to another aspect of the present invention there is provided a device for decoding an image or image portion from a bit stream, the device comprising: means for decoding a Skip flag from the bitstream; means for setting the mode to Skip Regular if said flag is enabled; means for determining a prediction mode if said flag is disabled; and means for decoding corresponding data.
According to another aspect of the present invention there is provided a device for encoding an image or image portion from a bit stream, the device comprising: means for encoding a Skip flag from the bitstream; means for setting the mode to Skip Regular if said flag is enabled; means for determining a prediction mode if said flag is disabled; and means for encoding corresponding data.
According to another aspect of the present invention there is provided a device for decoding an image or image portion from a bit stream, the device comprising: means for determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said means for determining comprising: means for verifying a condition relating to said motion prediction mode; and means for verifying a condition relating to at least one of the subsequent motion prediction modes.
According to another aspect of the present invention there is provided a device for encoding an image or image portion from a bit stream, the device comprising: means for determining whether to encode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said means for determining comprising: means for verifying a condition relating to said motion prediction mode; and means for verifying a condition relating to at least one of the subsequent motion prediction modes.
Yet further aspects of the present invention relate to a program as defined by claim 34. The program may be provided on its own or may be carried on, by or in a carrier medium. The carrier medium may be non-transitory, for example a storage medium, in particular a computer-readable storage medium. The carrier medium may also be transitory, for example a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet.
Further features of the invention are characterised by the other independent and dependent claims Any feature in one aspect of the invention may be applied to other aspects of the invention, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa.
Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly Any apparatus feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory.
It should also be appreciated that particular combinations of the various features described and defined in any aspects of the invention can be implemented and/or supplied and/or used independently.
Reference will now be made, by way of example, to the accompanying drawings, in which: Figure 1 is a diagram for use in explaining a coding structure used in HEVC; Figure 2 is a block diagram schematically illustrating a data communication system in which one or more embodiments of the invention may be implemented; Figure 3 is a block diagram illustrating components of a processing device in which one or more embodiments of the invention may be implemented; Figure 4 is a flow chart illustrating steps of an encoding method according to embodiments of the invention; Figure 5 is a flow chart illustrating steps of a decoding method according to embodiments of the invention; Figures 6 and 7 show the labelling scheme used to describe blocks situated relative to a current block; Figures 8(a) and (b) illustrate the Affine (SubBlock) mode; Figures 9(a) and (b) illustrate the Triangle mode; Figure 10 illustrates The Combined Inter MERGE / Intra prediction (CLIP) mode; Figure 11 illustrates an example decoding process of various coding modes; Figure 12 illustrates an example first stage of residual signalling; Figure 13 illustrates an example decoding process of various Merge modes; Figure 14 illustrates sub-block transform (SBT) partitioning; Figure 15 illustrates Intra Sub-Partitions; Figure 16 illustrates an evaluation of the Regular CIIP and MMVD MERGE and Skip modes; Figure 17 illustrates a rate estimation process; Figure 18 shows an example decoding process of various Merge modes; Figures 19a and 19b illustrate signalling of a Skip mode; Figure 20 illustrates signalling of the MMVD, SubBlock, Triangle Merge modes without residual; Figure 21 illustrates an example decoding process of various Merge modes; Figure 22 shows a variation of signalling of the MMVD, SubBlock, Triangle Merge modes without residual illustrated in Figure 20; Figure 23a and 23b show variations of signalling of a Skip mode illustrated in Figures 19a and 19b; Figure 24 shows the coding of the IBC inside the Merge data; Figure 25 illustrates a rate estimation process; Figure 26 shows a variation of Figure 16 which sets the variable BestlsSkip; Figure 27 illustrates an example decoding process of various Merge modes; Figure 28 illustrates signalling zero residual using a second Skip flag; Figure 29 is a diagram showing a system comprising an encoder or a decoder and a communication network according to embodiments of the present invention.
Figure 30 is a schematic block diagram of a computing device for implementation of one or more embodiments of the invention; Figure 31 is a diagram illustrating a network camera system * and Figure 32 is a diagram illustrating a smart phone.
Detailed description
Figure 1 relates to a coding structure used in the High Efficiency Video Coding (HEVC) video standard. A video sequence 1 is made up of a succession of digital images i. Each such digital image is represented by one or more matrices. The matrix coefficients represent pixels.
An image 2 of the sequence may be divided into slices 3. A slice may in some instances constitute an entire image. These slices are divided into non-overlapping Coding Tree Units (CTUs). A Coding Tree Unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and conceptually corresponds in structure to macroblock units that were used in several previous video standards. A CTU is also sometimes referred to as a Largest Coding Unit (LCU). A CTU has luma and chroma component parts, each of which component parts is called a Coding Tree Block (CTB). These different color components are not shown in Figure 1.
A CTU is generally of size 64 pixels x 64 pixels for HEVC, yet for VVC this size can be 128 pixels x 128 pixels. Each CTU may in turn be iteratively divided into smaller variable-size Coding Units (CUs) 5 using a quadtree decomposition.
Coding units are the elementary coding elements and are constituted by two kinds of sub-unit called a Prediction Unit (PU) and a Transform Unit (TU). The maximum size of a PU or TU is equal to the CU size. A Prediction Unit corresponds to the partition of the CU for prediction of pixels values. Various different partitions of a CU into PUs are possible as shown by 606 including a partition into 4 square PUs and two different partitions into 2 rectangular PUs. A Transform Unit is an elementary unit that is subjected to spatial transformation using DCT. A CU can be partitioned into TUs based on a quadtree representation 607.
Each slice is embedded in one Network Abstraction Layer (NAL) unit. In addition, the coding parameters of the video sequence are stored in dedicated NAL units called parameter sets. In HEVC and H.264/AVC two kinds of parameter sets NAL units are employed: first, a Sequence Parameter Set (SP S) NAL unit that gathers all parameters that are unchanged during the whole video sequence. Typically, it handles the coding profile, the size of the video frames and other parameters. Secondly, a Picture Parameter Set (PPS) NAL unit includes parameters that may change from one image (or frame) to another of a sequence. HEVC also includes a Video Parameter Set (VPS) NAL unit which contains parameters describing the overall structure of the bitstream. The VPS is a new type of parameter set defined in HEVC, and applies to all of the layers of a bitstream. A layer may contain multiple temporal sub-layers, and all version 1 bitstreams are restricted to a single layer. HEVC has certain layered extensions for scalability and multiview and these will enable multiple layers, with a backwards compatible version 1 base layer.
Figure 2 illustrates a data communication system in which one or more embodiments of the invention may be implemented. The data communication system comprises a transmission device, in this case a server 201, which is operable to transmit data packets of a data stream to a receiving device, in this case a client terminal 202, via a data communication network 200. The data communication network 200 may be a Wide Area Network (WAN) or a Local Area Network (LAN). Such a network may be for example a wireless network (Wifi / 802.11a or b or g), an Ethernet network, an Internet network or a mixed network composed of several different networks. In a particular embodiment of the invention the data communication system may be a digital television broadcast system in which the server 201 sends the same data content to multiple clients.
The data stream 204 provided by the server 201 may be composed of multimedia data representing video and audio data. Audio and video data streams may, in some embodiments of the invention, be captured by the server 201 using a microphone and a camera respectively. In some embodiments data streams may be stored on the server 201 or received by the server 201 from another data provider, or generated at the server 201. The server 201 is provided with an encoder for encoding video and audio streams in particular to provide a compressed bitstream for transmission that is a more compact representation of the data presented as input to the encoder.
In order to obtain a better ratio of the quality of transmitted data to quantity of transmitted data, the compression of the video data may be for example in accordance with the HEVC format or H.264/AVC format or VVC format.
The client 202 receives the transmitted bitstream and decodes the reconstructed bitstream to reproduce video images on a display device and the audio data by a loud speaker.
Although a streaming scenario is considered in the example of Figure 2, it will be appreciated that in some embodiments of the invention the data communication between an encoder and a decoder may be performed using for example a media storage device such as an optical disc.
In one or more embodiments of the invention a video image is transmitted with data representative of compensation offsets for application to reconstructed pixels of the image to provide filtered pixels in a final image.
Figure 3 schematically illustrates a processing device 300 configured to implement at least one embodiment of the present invention. The processing device 300 may be a device such as a micro-computer, a workstation or a light portable device. The device 300 comprises a communication bus 313 connected to: -a central processing unit 311, such as a microprocessor, denoted CPU; -a read only memory 306, denoted ROM, for storing computer programs for implementing the invention; -a random access memory 312, denoted RAM, for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method of encoding a sequence of digital images and/or the method of decoding a bitstream according to embodiments of the invention; and -a communication interface 302 connected to a communication network 303 over which digital data to be processed are transmitted or received Optionally, the apparatus 300 may also include the following components: -a data storage means 304 such as a hard disk, for storing computer programs for implementing methods of one or more embodiments of the invention and data used or produced during the implementation of one or more embodiments of the invention; -a disk drive 305 for a disk 306, the disk drive being adapted to read data from the disk 306 or to write data onto said disk; -a screen 309 for displaying data and/or serving as a graphical interface with the user, 10 by means of a keyboard 310 or any other pointing means.
The apparatus 300 can be connected to various peripherals, such as for example a digital camera 320 or a microphone 308, each being connected to an input/output card (not shown) so as to supply multimedia data to the apparatus 300.
The communication bus provides communication and interoperability between the various elements included in the apparatus 300 or connected to it. The representation of the bus is not limiting and in particular the central processing unit is operable to communicate instructions to any element of the apparatus 300 directly or by means of another element of the apparatus 300.
The disk 306 can be replaced by any information medium such as for example a compact disk (CD-ROM), rewritable or not, a ZIP disk or a memory card and, in general terms, by an information storage means that can be read by a microcomputer or by a microprocessor, integrated or not into the apparatus, possibly removable and adapted to store one or more programs whose execution enables the method of encoding a sequence of digital images and/or the method of decoding a bitstream according to the invention to be implemented.
The executable code may be stored either in read only memory 306, on the hard disk 304 or on a removable digital medium such as for example a disk 306 as described previously. According to a variant, the executable code of the programs can be received by means of the communication network 303, via the interface 302, in order to be stored in one of the storage means of the apparatus 300 before being executed, such as the hard disk 304.
The central processing unit 311 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, instructions that are stored in one of the aforementioned storage means. On powering up, the program or programs that are stored in a non-volatile memory, for example on the hard disk 304 or in the read only memory 306, are transferred into the random access memory 312, which then contains the executable code of the program or programs, as well as registers for storing the variables and parameters necessary for implementing the invention.
In this embodiment, the apparatus is a programmable apparatus which uses software to implement the invention. However, alternatively, the present invention may be implemented in hardware (for example, in the form of an Application Specific Integrated Circuit or AS IC).
Figure 4 illustrates a block diagram of an encoder according to at least one embodiment of the invention. The encoder is represented by connected modules, each module being adapted to implement, for example in the form of programming instructions to be executed by the CPU 311 of device 300, at least one corresponding step of a method implementing at least one 10 embodiment of encoding an image of a sequence of images according to one or more embodiments of the invention.
An original sequence of digital images i0 to in 401 is received as an input by the encoder 400. Each digital image is represented by a set of samples, sometimes also referred to as pixels (hereinafter, they are referred to as pixels).
A bitstream 410 is output by the encoder 400 after implementation of the encoding process. The bitstream 410 comprises a plurality of encoding units or slices, each slice comprising a slice header for transmitting encoding values of encoding parameters used to encode the slice and a slice body, comprising encoded video data.
The input digital images i0 to in 401 are divided into blocks of pixels by module 402.
The blocks correspond to image portions and may be of variable sizes (e.g. 4x4, 8x8, 16x16, 32x32, 64x64, 128x128 pixels and several rectangular block sizes can be also considered). A coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial prediction coding (Intra prediction), and coding modes based on temporal prediction (Inter coding, Merge, SKIP). The possible coding modes are tested.
Module 403 implements an Intra prediction process, in which the given block to be encoded is predicted by a predictor computed from pixels of the neighbourhood of said block to be encoded. An indication of the selected Intra predictor and the difference between the given block and its predictor is encoded to provide a residual if the Intra coding is selected.
Temporal prediction is implemented by motion estimation module 404 and motion compensation module 405. Firstly a reference image from among a set of reference images 416 is selected, and a portion of the reference image, also called reference area or image portion, which is the closest area (closest in terms of pixel value similarity) to the given block to be encoded, is selected by the motion estimation module 404. Motion compensation module 405 then predicts the block to be encoded using the selected area. The difference between the selected reference area and the given block, also called a residual block, is computed by the motion compensation module 405. The selected reference area is indicated using a motion vector.
Thus, in both cases (spatial and temporal prediction), a residual is computed by subtracting the predictor from the original block.
In the INTRA prediction implemented by module 403, a prediction direction is encoded. In the Inter prediction implemented by modules 404, 405, 416, 418, 417, at least one motion vector or data for identifying such motion vector is encoded for the temporal prediction.
Information relevant to the motion vector and the residual block is encoded if the Inter prediction is selected. To further reduce the bitrate, assuming that motion is homogeneous, the motion vector is encoded by difference with respect to a motion vector predictor. Motion vector predictors from a set of motion information predictor candidates is obtained from the motion vectors field 418 by a motion vector prediction and coding module 417.
The encoder 400 further comprises a selection module 406 for selection of the coding mode by applying an encoding cost criterion, such as a rate-distortion criterion. In order to further reduce redundancies a transform (such as DCT) is applied by transform module 407 to the residual block, the transformed data obtained is then quantized by quantization module 408 and entropy encoded by entropy encoding module 409. Finally, the encoded residual block of the current block being encoded is inserted into the bitstream 410.
The encoder 400 also performs decoding of the encoded image in order to produce a reference image (e.g. those in Reference images/pictures 416) for the motion estimation of the subsequent images. This enables the encoder and the decoder receiving the bitstream to have the same reference frames (reconstructed images or image portions are used). The inverse quantization ("dequantization") module 411 performs inverse quantization ("dequantization") of the quantized data, followed by an inverse transform by inverse transform module 412. The intra prediction module 413 uses the prediction information to determine which predictor to use for a given block and the motion compensation module 414 actually adds the residual obtained by module 412 to the reference area obtained from the set of reference images 416.
Post filtering is then applied by module 415 to filter the reconstructed frame (image or image portions) of pixels. In the embodiments of the invention an SAO loop filter is used in which compensation offsets are added to the pixel values of the reconstructed pixels of the reconstructed image. It is understood that post filtering does not always have to performed. Also, any other type of post filtering may also be performed in addition to, or instead of, the SAO loop filtering.
Figure 5 illustrates a block diagram of a decoder 60 which may be used to receive data from an encoder according an embodiment of the invention. The decoder is represented by connected modules, each module being adapted to implement, for example in the form of programming instructions to be executed by the CPU 311 of device 300, a corresponding step of a method implemented by the decoder 60.
The decoder 60 receives a bitstream 61 comprising encoded units (e.g. data corresponding to a block or a coding unit), each one being composed of a header containing information on encoding parameters and a body containing the encoded video data. As explained with respect to Figure 4, the encoded video data is entropy encoded, and the motion vector predictors' indexes are encoded, for a given block, on a predetermined number of bits.
The received encoded video data is entropy decoded by module 62. The residual data are then dequantized by module 63 and then an inverse transform is applied by module 64 to obtain pixel values.
The mode data indicating the coding mode are also entropy decoded and based on the 15 mode, an INTRA type decoding or an INTER type decoding is performed on the encoded blocks (units/sets/groups) of image data.
In the case of INTRA mode, an INTRA predictor is determined by intra prediction module 65 based on the intra prediction mode specified in the bitstream.
If the mode is INTER, the motion prediction information is extracted from the bitstream so as to find (identify) the reference area used by the encoder. The motion prediction information comprises the reference frame index and the motion vector residual. The motion vector predictor is added to the motion vector residual by motion vector decoding module 70 in order to obtain the motion vector. The various motion predictor tools used in VVC are discussed in more detail below with reference to Figures 6-10.
Motion vector decoding module 70 applies motion vector decoding for each current block encoded by motion prediction. Once an index of the motion vector predictor for the current block has been obtained, the actual value of the motion vector associated with the current block can be decoded and used to apply motion compensation by module 66. The reference image portion indicated by the decoded motion vector is extracted from a reference image 68 to apply the motion compensation 66. The motion vector field data 71 is updated with the decoded motion vector in order to be used for the prediction of subsequent decoded motion vectors.
Finally, a decoded block is obtained. Where appropriate, post filtering is applied by post filtering module 67. A decoded video signal 69 is finally obtained and provided by the decoder 60.
Motion prediction (INTER) modes HEVC uses 3 different INTER modes: the Inter mode (Advanced Motion Vector Prediction (AMVP)), the "classical" Merge mode (i.e. the "non-Affine Merge mode" or also known as "regular" Merge mode) and the "classical" Merge Skip mode (i.e. the "non-Affine Merge Skip" mode or also known as "regular" Merge Skip mode). The main difference between these modes is the data signalling in the bitstream. For the Motion vector coding, the current HEVC standard includes a competitive based scheme for Motion vector prediction which was not present in earlier versions of the standard. It means that several candidates are competing with the rate distortion criterion at encoder side in order to find the best motion vector predictor or the best motion information for respectively the Inter or the Merge modes (i.e. the "classical/regular" Merge mode or the "classical/regular" Merge Skip mode). An index corresponding to the best predictors or the best candidate of the motion information is then inserted in the bitstream, together with a 'residual' which represents the difference between the predicted value and the actual value. The decoder can derive the same set of predictors or candidates and uses the best one according to the decoded index. Using the residual, the decoder can then recreate the original value.
In the Screen Content Extension of HEVC, the new coding tool called Intra Block Copy (IBC) is signalled as any of those three INTER modes, the difference between IBC and the equivalent INTER mode being made by checking whether the reference frame is the current one. This can be implemented e.g. by checking the reference index of the list LO, and deducing this is Intra Block Copy if this is the last frame in that list. Another way to do is comparing the Picture Order Count of current and reference frames: if equal, this is Intra Block Copy.
The design of the derivation of predictors and candidates is important in achieving the best coding efficiency without a disproportionate impact on complexity. In HEVC two motion vector derivations are used: one for Inter mode (Advanced Motion Vector Prediction (AMVP)) and one for Merge modes (Merge derivation process -for the classical Merge mode and the classical Merge Skip mode). The following describes the various motion predictor modes used in VVC.
Figures 6 and 7 show the labelling scheme used herein to describe blocks situated relative to a current block (i.e. the block currently being en/decoded) between frames (Fig. 6) and within the same frame (Fig. 7).
Affine mode (SubBlock mode) In HIEVC, only translation motion model is applied for motion compensation prediction (MCP). While in the real world, there are many kinds of motion, e.g. zoom in/out, rotation, perspective motions and other irregular motions.
In the JEM, a simplified affine transform motion compensation prediction is applied 10 and the general principle of Affine mode is described below based on an extract of document JVET-G1001 presented at a JVET meeting in Torino at 13-21 July 2017. This entire document is hereby incorporated by reference insofar as it describes other algorithms used in JEM. As shown in Figure 8(a), the affine motion field of the block is described by two control point motion vectors.
The affine mode is a motion compensation mode like the Inter modes (AMVP, "classical" Merge, or "classical" Merge Skip). Its principle is to generate one motion information per pixel according to 2 or 3 neighbouring motion information. In the JEM, the affine mode derives one motion information for each 4x4 block as depicted in Figure 8(a) (each square is a 4x4 block, and the whole block in Figure 8(a) is a 16x16 block which is divided into 16 blocks of such square of 4x4 size -each 4x4 square block having a motion vector associated therewith). The Affine mode is available for the AMVP mode and the Merge modes (i.e. the classical Merge mode which is also referred to as "non-Affine Merge mode" and the classical Merge Skip mode which is also referred to as "non-Affine Merge Skip mode"), by enabling the affine mode with a flag. This flag is Context Adaptive Binary Arithmetic Coding (CABAC) coded.
CABAC is an extension of the arithmetic coding used in H.265/HEVC which separates the probabilities of a syntax element depending on a 'context' defined by a context variable. This corresponds to a conditional probability. The context variable may be derived from the value of the current syntax for the top left block (A2 in Figure 6b as described in more detail below) and the above left block (B3 in Figure 6b), which are already decoded. It should be appreciated that other coding modes may be used -for example Context-Adaptive Variable-Length Coding (CAVLC), Golomb-rice Code, or simple binary representation called Fixed Length Coding -however CABAC has been found to provide the greatest bit-rate savings (at a cost of increased complexity).
In the VVC specification the Affine Mode is also known as SubBlock mode; these terms are used interchangeably in this specification.
Other modes In addition to the regular MERGE mode and afTine MERGE mode of the JEM software, the current VVC standard under definition contains 3 other MERGE modes. One mode is the Combined Inter MERGE / Intra prediction (CILP) also known as Multi-Hypothesis Intra Inter (MI-III) MERGE mode. Another one is the TRIANGLE MERGE mode and the last one is the Motion Vector Difference (MMVD) MERGE mode.
Triangle Mode The TRIANGLE MERGE mode is a particular bi-prediction mode. Figure 9 illustrates this particular block predictor generation. The block predictor contains one triangle from a first block predictor (901 or 911) and a second triangle from a second block predictor (902 or 912). There are two types of block generations. For the first one the splitting between the 2 block predictors is from top the left corner to the bottom right corner as depicted in Figure 9(a). For the second one, the splitting between the two block predictors is from the top right corner to the bottom left corner as depicted in Figure 9(b). In addition, the samples near the frontier between both triangle sample predictors are filtered with a weighted average where the weight depends on the sample position.
The TRIANGLE MERGE should be interpreted in this specification as being a mode which combines features of two Inter non square predictors, and not necessarily as a label given to one specific mode.
MMVD
The MMVD MERGE mode is a specific regular MERGE mode candidate derivation.
It can be considered as an independent NIERGE candidates list. The selected MMVD MERGE candidate, for the current CU, is obtained by adding an offset value to one motion vector component (mvx or mvy) to an initial regular MERGE candidate. The offset value is added to the motion vector of the first list LO or to the motion vector of the second list LI depending on the configuration of these reference frames (both backward, both forward or forward and backward). The initial Merge candidate is signalled thanks to an index. The offset value is signalled thanks to a distance index between the 8 possible distances (1/4-pel, 1/2-pel,l-pel, 2-pel, 4-pel, 8-pel, 16-pel, 32-pel) and a direction index giving the x or the y axis and the sign of the offset.
CIIP
The Combined Inter MERGE / Intra prediction (CIIP) MERGE can be considered as a combination of the regular MERGE mode and the Intra mode and is described below with reference to Figure 10. The block predictor for the current block (1001) of this mode is an average between a MERGE predictor block and an Intra predictor block as depicted in Figure 10. The MERGE predictor block is obtained with exactly the same process of the MERGE mode so it is a temporal block (1002) or bi-predictor of 2 temporal blocks. As such, a MERGE index is signalled for this mode in the same manner as the regular MERGE mode. The Intra predictor block is obtained based on the neighbouring sample (1003) of the current block (1001). The amount of available Intra modes for the current block is however limited compared to an Intra block. Moreover, there is no Chroma Intra predictor block signalled for a CIIP block. The Chroma predictor is equal to the Luma predictor. As a consequence, 1, 2 or 3 bits are used to signal the Intra predictor for a CIIP block.
The CIIP block predictor is obtained by a weighted average of the MERGE block predictor and the Intra block predictor. The weighting of the weighted average depends on the block size and/or the Intra predictor block selected.
The obtained CIIP predictor is then added to the residual of the current block to obtain the reconstructed block. It should be noted that the CIIP mode is enabled only for non-Skipped blocks. Indeed, use of the CUP Skip typically results in losses in compression performance and an increase in encoder complexity. This is because the CIIP mode has often a block residual in opposite to the other Skip mode. Consequently its signalling for the Skip mode increase the bitrate. -when the current CU is Skip, the CIIP is avoided. A consequence of this restriction is that the CIIP block can't have a residual containing only 0 value as it is not possible to encode a VVC block residual equal to 0. Indeed, in VVC the only way to signal a block residual equal to 0 for a Merge mode is to use the Skip mode, this is because the CU CBF flag is inferred to be equal to true for Merge modes. And when this CBF flag is true, the block residual can't be equal to 0.
In such a way, CIIP should be interpreted in this specification as being a mode which combines features of Inter and Intra prediction, and not necessarily as a label given to one specific mode.
Signalling of motion prediction (INTER) modes The coding modes are signalled inside the bitstream with their related syntax elements. Figure Ila illustrates partially the decoding process of the coding modes for the current CU inside an Inter frame. First the CU Skip flag is extracted from the bitstream (1101). If the CU is not Skip the pred mode flag (H 03) and/or Merge flag (1106) are decoded to identify if the current CU is a Merge CU. If the current CU is a Skip Merge (1102) or a Merge CU (1107), the other Merge data (1108) are extracted from the bitstream as illustrated in Figure 13.
When the IBC mode is enabled at SPS level, this process is modified as depicted in Figure 11b. When the current CU is Skip, the IBC flag is extracted from the bitstream (1119) if the Luma size of the current CU is not 128x128. Then, the Merge data is extracted from the bitstream (1118). In the same way when the Current CU is not a Skip CU and when the current CU is Inter, the IBC flag is extracted (1121) from the bitstream if the Luma size of the current CU is not 128x128. In a particular way, the IBC mode is allowed for 4x4 block size which is not the case for Inter CU. Consequently the Skip flag or pred mode or Merge flag are decoded when IBC is enabled at SPS level.
Figure 13 illustrates the Merge flags coding (1108 and 1118) as described in Figure lla and Figure 11b. First, if the current mode is an IBC MERGE or Skip (1321), the Merge index is decoded (1304) otherwise it is verified if the REGULAR MERGE flag is decoded (1301). In the current VVC specification the condition -regular cond" is the following: ( sps mmvd enabled flag I cbWidth * cbHeight!= 32) In this formula and others formula in this invention, the symbol "!=" means different to, the symbol means OR, the symbol "&&" means AND, the symbol "*-means multiply, the symbol "==" means equal to.
This formula means that the Regular flag (1302) is decoded only when the MMVD MERGE mode is enabled at SPS level or when the Luma width multiplies by the Luma height is different to 32. Otherwise the Regular flag is not extracted from the bitstream and it is set equal to 1, if the sps mmvd enabled flag is false and if the Luma width multiplies by the Luma height is equal to 32. Otherwise it is set equal to false.
If the regular flag is set equal to 1 (1303), the Merge index is decoded (1304) and other data if the current CU is not a Skip CU (1320). If the regular Merge flag is false (1303), the MMVD condition "MMVD cone is verified. This condition is currently the following: (sps mmvd enabled flag && cbWidth * cbHeight!= 32) It means that the MMVD flag is decoded (1306) if the MMVD Merge mode is enabled at SPS level and if the Luma width multiplies by the Luma height is different to 32. If it is not extracted from the bitstream, the MIVIVD flag is set equal to true if the sps mmvd enabled flag is true and if the Luma width multiplies by the Luma height is equal to 32. Otherwise it is set equal to false.
If the current MMVD flag is equal to true, the MMVD data are decoded (1308).
If the CU is not MAIVD (1307) the SubBlock condition "SubBlock _fond" is checked (1309). This condition is the following: ( MaxNumSubblocicMergeCand > 0 && cbWidth >= 8 && cbHeight >= 8) It means that the SubBlock flag is decoded (1310) if the maximum Number of SubBlock 5 MERGE candidates is superior to 1. This maximum number of SubBlock MERGE candidates can change for each slice and it is of course set equal to 0 when the SubBlock MERGE mode is disabled at SPS level.
Moreover the Luma width and the Luma height shall be superior or equal to 8 otherwise the SubBlock MERGE mode is not allowed. If the SubBlock MERGE flag is not decoded it is set equal to false. If the current SubBlock flag is equal to true (1311), the SubBlock data are decoded (1312).
If the CU is not SubBlock CU (1311) the CEP condition "CIIP _fond" is checked (1313). This condition is the following: ( sps ciip enabled flag && cu skip flag[ x0][ y0] == 0 && ( cbWidth * cbHeight)>= 64 && cbWidth < 128 && cbHeight < 128) It means that the CIIP flag is decoded (1314) if CIIP is enabled at SPS level and if the Current CU is not Skip and if the CU Luma height and width are strictly inferior to 128 and their multiplication is superior or equal to 64. When it is not extracted from the bitstream the CIIP flag is set equal to false.
If the current CIIP flag is equal to true (1315), the CIIP data are decoded (1316).
If the CU is not CIIP CU (1315) the Triangle condition "Triangle Cond' is checked (1317). This condition is the following: (sps_triangle_enabled_flag && slice_type == B_Slice && cbWidth * cbHeight >= 64 && 25 MaxNumTriangleMergeCand>2).
It means that the current CU is Triangle if the TRIANGLE MERGE mode is enabled at SPS level and if the current slice is not a B slice and if the CU Luma height multiplies by the Luma width is superior to 64. Moreover, the maximum number of TRIANGLE MERGE candidates shall be superior to 2. This number is transmitted for each slice when the TRIANGLE MERGE mode is enabled at SPS level.
In the normal way, the last modules 1317, 1318 are not needed if the encoder did correctly its work because when the CU is a Skip or a Merge CU and when it is not one of each other MERGE modes, it is sure that the current CU is TRIANGLE MERGE (or Skip) mode.
Please note that in the current VVC specification, all the Merge modes can be disabled at SPS level except the REGULAR MERGE mode. In the Same way the REGULAR MERGE or Skip mode can't be disabled at slice level thanks to the maximum number of Merge candidates which is at least set equal to 1 for the REGULAR MERGE mode. But the REGULAR MERGE mode, as all other Merge modes except IBC is not allowed for 4x4 CU size.
Residual signalling As discussed above, a residual is encoded into the bitstream to indicate the difference 10 between the value being predicted and the actual value being encoded. The following description outlines the manner Coding Block Flag (CBF) Figure 12 illustrates a first step of the residual signalling for a current CU (from the perspective of a decoder). In this figure the CBF value of the current CU is determined. The Coding Block Flag (CBF) flag specifies that the current block contains a residual. First, if the current CU is a SKIP CU (1202), the cu cbf flag is set equal to 0 (1203) and no residual is decoded for the current block. Indeed by definition, a SKIP CU has no residual. If the current CU is Intra (1204), the cu cbf is set equal to 1 (1205). If the Intra block has no residual it will be signalled thanks to TU cbf flags. If the current CU is a MERGE CU (1206), the cu_cbf flag is set equal to 1 (1207). Indeed, if the current MERGE block has no residual it will be signalled thanks to the SKIP mode. If the current CU is not a MERGE (1206), the current CU is coded with another Inter mode different to the SKIP or MERGE mode, then the cu cbf flag is decoded (1208). If this cbf flag is equal to 1 (1210) or if the current CU is a MERGE CU but not CLIP (1209), the cu_sbt_flag (1212) is decoded if the current CU has a size allowed for SBT method (1211). In the same way, the other data related to SBT are decoded (1213) (1214) (1215). Please note that when the cu cbf is equal to 0 (1210), the process is ended (1217) so no residual is decoded.
If the cbf flag is equal to 1, one or several residuals are decoded (1216)..
However this process means that it is not possible to signal a zero residual for a non-skip motion prediction mode. When a CU CBF is equal to 1, the block residual can't be a block with only 0 residuals.
SBT
The Sub-block transform for Inter blocks (SBT) is a particular Transform Unit (TU) splitting in addition to some particular transform selection based on the splitting parameters.
Figure 14 illustrates the splitting which can be used when cu_sbt_flag is set equal to 1. In this Figure 14, only the dashed blocks contain a residual. For example of CU 1410, only the partition 1412 contains a residual and not the partition 1411. The same applies to the other example splitting where the numerals 14m0, 14m1 and 14m2 (where m=1...7) indicate the CU, partition not containing a residual, and partition containing a residual respectively. These different partitions are signalled by the syntax elements cu sbt flag, cu sbt horizontal flag and cu sbt pos flag. These TU partitioning is allowed only for some TU size (1211).
ISP
For Intra blocks there are also some possible sub-partitionings. The Intra Sub-Partitions Coding Mode (ISP) is a particular partitioning scheme. In contrast to the SBT method discussed above, each sub partition can contain a residual for both Luma and Chroma blocks. Figure EE illustrates the possible partitioning of a block (1510). The block can be split into 4 vertical blocks (1520) or four horizontal blocks (1530). For small block size 8x4, 4x8, only 2 sub-partitions are considered. The last sub partition has its tu cbf luma always equal to 1 in the current VVC design.
Encoder Merge selection in VTM (encoding choice) Figure 16 illustrates the VTM software implementation for the evaluation of the 20 Regular CIIP and MMVD MERGE and Skip modes. They are evaluated in the same process. The SubBlock (Affine), IBC and Triangle modes are evaluated in separate processes as all other Inter and Intra modes.
First the variable BestlsSkip is set equal to true if the current variable BestCU is a Skip CU (1601). The variable BestCU represents the best set of coding parameters among all already evaluated coding parameters for the current CU (eg. Current block size). If the variable BestlsSkip is false or if the CIIP mode is enabled for the current CU (1602), a first "fast" RD loop is computed (1605). The CIIP is enabled for the current CU if the following condition is true: CUwidth * CUheight < 64 or CUwidth >= 128 or CUheight >= MAX 128 SIZE) The "fast" RD loop (1605) consists in evaluating the SAD and an estimated rate (1606) of each possible Merge candidates among all possible candidates between the regular, MMVD and CLIP Merge modes. This loop orders these possible candidates from the best candidate (in term of RD cost) to the worst in a list Cand[] (1607). It should be noted that in this loop the best Intra predictor for each possible CIIP merge candidate is also selected. At the end of this fast loop, the ordered Merge candidates have been set (1608), and the Maximum number for the Full RD evaluation is set to 4, or 5 if CIIP is allowed for the current CU size (1609).
If CIIP is disabled and if the best parameters set for the current CU is the Skip mode, the maximum number of candidates for the Full RD loop is set equal to 70 (1604) (6 for Regular MRG + 64 MMVD) if the best mode is MMVD or to 6 if not.
There is 2 passes for the Full RD loop (1610). In the first one, the candidates are evaluated with their residuals and in the second one without their residuals. Each candidate index idx from 0 to MAX (1611) in the list Cand[] is evaluated. If the current pass is the no residual pass and if the Cand[Idx] is a CIIP candidate (1612), the candidate Cand[idx] is replaced by its corresponding Regular Merge candidate (1613). (So it is the CIIP Merge candidate but not averaged with the Intra predictor). If the pass is the residual pass and if the variable BestlsSkip is true (1614), the next candidate will be proceed (1611). (Please note that all candidates will be not processed in that case so it corresponds to a switch to the next pass without residual). Otherwise the Full RD evaluation process (1615) gives the best parameters set BestCU (1616). If the BestCU is not CLIP, the variable BestlsSkip is set equal to true if the cu cbf of the best parameters set is equal to 0 (1618).
In the current VTM-5.0, the rates of the Regular, MMVD and CIIP modes are estimated with a real CABAC encoding in order to improve the coding efficiency. The current implementation of this rate estimation is depicted in Figure 17. The amount of bits that are obtained in this evaluation is the same as those encoded for the Merge data process as depicted in Figure 13 and the Merge flags.
As discussed above, the current Merge and Skip Modes flag coding generates additional flags signalling when one or more Merge modes are disabled at SPS level or Slice level. These signalled flags unnecessarily increase the bitrate.
The check of several conditions for the decoding of the flag is not desirable for both hardware and software implementation.
For example, with reference to Figure 13, when the Triangle Mode is disabled, if the CU is a Skip SubBlock CU, the SubBlock Flag (1310) doesn't need to be encoded and decoded, because when the CU is not IBC, not Regular or not MMVD Merge, it is sure that the CU is a SubBlock CU.
Another problem, for a design point of view, is that it is preferable that the most selected mode is the mode set at the first position of syntax in order to reduce the number of bins and bits that should be read for the decoding in average. It has been identified that the Regular Skip mode is the most selected mode for natural sequences so it is desirable that it is set at the first position of the Merge types which is the case when IBC is disabled as IBC is considered as a screen content tools and will may be disabled for natural sequences.
The proposed invention improves the coding efficiency when some Modes are disabled and/or improves the current design.
EMBODIMENTS
In one embodiment of the invention, determining whether or not to decode a flag corresponding to a particular motion predictor mode is based on conditions of subsequent flags.
The condition checked for the first flag (in this case, Regular mode) is whether Regular cond is satisfied and at least one of the conditions relating to the subsequent flags is satisfied. This allows for the process to terminate as early as possible by checking for disabled modes at the earliest opportunity. As such, the decoding process may be faster.
Figure 18 illustrates this embodiment. In this embodiment, the regular Merge flag decoding condition (1801) is not limited to the Regular cond as currently defined in the VVC specification yet the condition to decode the regular flag is: if Regular_cond is true and at least 20 one of the Triangle_Cond or SubBlock_Cond or CIIP_Cond or NINIVD_cond condition is true. The NIIVIVD Merge flag decoding condition (1805) is not limited to the MNIVD cond as currently defined in the VVC specification yet the condition to decode the MMVD flag (1806) is if MMVD cond is true and at least one of the Triangle Cond or SubBlock Cond or CIIP Cond or Regular_cond condition is true. Please note that, in this formula the 25 Regular cond can be removed because it have been already evaluated (1801).
The SubBlock Merge flag decoding condition (1809) is not limited to the SubBlock cond as currently defined in the VVC specification yet the condition to decode the SubBlock flag (1810) is if SubBlock cond is true and at least one of the Triangle Cond or CIIP Cond is true.
The CIIP Merge flag decoding condition (1813) is not limited to the CIIP cond as currently defined in the VVC specification yet the condition to decode the CIIP flag (1814) is: if CIIP cond is true and if the Triangle Cond condition is true.
As described for Figure 13, the condition Triangle_Cond (1817) and module 1818 are not needed because when the CU is not IBC, not Regular, not MMVD, not SubBlock and not CI1P it is sure that the Merge is Triangle.
Please note that the IBC condition has not been changed because IBC flag doesn't create an issue with the current order of Merge flags as it is set before the Regular flag in this
example.
The advantage of this embodiment is that when one or more Merge mode is not allowed a correct number of flags is encoded and decoded. For example, when the Triangle Mode is disabled, if the CU is a Skip SubBlock CU, the SubBlock Flag (1810) is not encoded and decoded, because when the CU is not IBC, not Regular or not MMVD Merge, it is sure that the CU is a SubBlock CU with the current embodiment 1 Even though this solution increases the number of conditions for Regular, MMVD, SubBlock, CIIP flags -it doesn't increase the worst case of the total amount of conditions. An advantage of this solution is that it has the same worst case complexity as the current VVC specification. The fact that this solution does not lead to significant losses at the worst case is surprising given that the solution could be seen as increasing the complexity -especially for regular mode. However, when some modes are disabled at SPS or Slice level, significant coding efficiency gains can be achieved.
In another, complimentary, embodiment, only the Regular and IBC mode are signalled as a Skip mode and MMVD, SubBlock, Triangle Merge mode without residual are signalled as a Merge mode. Table 1 shows an example Skip Merge flag signalling for this embodiment: Signaled Modes IBC flag IBC 1 Regular 0 Table 1 -Skip Merge flags signaling In addition, the Regular Merge mode type may be set at the last position of the list of Merge when a CU is a Merge -as shown in Figure 21 and described below.
Figure 19 illustrates the signalling of the Skip Regular (Figure 19a) only and the Skip IBC and Skip Regular (Figure 19b). Figure 19 is based on Figure 11. In Figure 19a, IBC is not allowed. When the CU is Skip (1902), the Merge index is decoded for the Skip Regular. For all other Merge modes and for Regular Merge with residual the Merge data are decoded. Figure 19b shows a modification where IBC is allowed. When the current decoded CU is Skip (1912), the IBC flag is extracted from the bitstream if the Luma size of the current CU is not 128x128, and the Merge index is decoded (1919) for both Regular and IBC Skip mode.
If the current CU is one among all other Merge types or IBC or Regular Merge modes with residual, the Merge data will be decoded (1918).
A particular advantage of this embodiment/feature, is that the Regular Skip mode remains as the first decoded mode -this reduces the global decoding complexity average as this is the most probable mode.
Another advantage is that the conditions needed to decode the regular Skip mode are reduced to only check if the current CU has a size different to 4x4 in order to decode the Skip flag.
Moreover, the requirement to check the condition for the Regular Merge flag has been 15 removed (compared to the current VVC specification) for the Skip Regular mode. This improves coding efficiency and provides a cleaner design leading to simpler implementation in software and/or hardware.
Figure 20 illustrates the signalling of the MMVD, SubBlock, Triangle Merge modes without residual by using a CBF equal to 0. This figure is based on Figure 12. Compared to Figure 12, a condition (2018) has been added to set the cu_cbf equal to 1 (2007), when the CU is a Merge CU (2006). The related condition, is that the current CU is a Regular, CIIP or IBC Merge mode (2018), otherwise (MMVD, SubBlock, Triangle), the cu cbf is extracted from the bitstream (2008).
The advantage of this feature/embodiment, is a clean design, and a small increase of coding efficiency. The cleaner design for the en/decoder may lead to simpler implementation in software and/or hardware. Furthermore, as the Regular Skip is the most selected (most probable) mode, the signalling of the other modes outside the Skip mode improves the coding efficiency.
Figure 22 shows a variation of Figure 20 where the zero residual is signalled for the Merge modes without using Skip mode. In this variation, the MMVD, SubBlock (Affine), Triangle and CIIP Merge are signalled in the same way when they have no residual. In this embodiment, the CIIP without residual is enabled in opposite to the current VVC specification.
For this embodiment, zero residual can be signalled by setting the cu_cbf flag equal to 0 as depicted in Figure 22. In this figure, the cu cbf is set equal to 1 only when the CU is Regular or IBC (2218). Otherwise, the cu cbf is extracted from the bitstream (2208).
The other zero residual signalling methods can be used also for this variation.
The advantages of this variation include a cleaner design by using zero residual signalling for all Merge types, a coding efficiency increase and an encoding time complexity reduction as the usage of zero residual for CBI) reduces the encoding time.
It should be noted that IBC may not be possible/permitted, in which case the condition may be simplified to the current CU is Regular.
The cu cbf described above may have its own context variable for its CABAC coding.
Advantageously, the context is the value of the current Merge flag.
The advantage of the cu cbf having its own context is a coding efficiency increase compared to the previous embodiment. Indeed the Merge mode without residual (SubBlcok, NIVIVD, Triangle) are more frequent than the other Inter modes without residual. So having separate context states leads to a coding efficiency improvement.
In one alternative feature/embodiment to the zero residual signalling of Merge modes the TU CBF of each component is set equal to 0 to signal of a residual equal to zero for these particular Merge types (MMVD, SubBlock, Triangle). This can lead more flexibility especially when the Luma and Chroma components have a high QP difference.
In one alternative feature/embodiment to the sig_coeff is signalled when the last coefficient is the first coefficient of a block residual in order to signal a residual equal to zero for these particular Merge types (MMVD, SubBlock, Triangle). This can lead to a cleaner design for the en/decoder and as such simpler implementation in software and/or hardware.
In one embodiment the zero residual signalling of some Merge modes is signalled by a second Skip flag. Figure 28 illustrates this embodiment. This Figure is based on Figure 19.
Figure 28a shows a variation of the mode signalling when IBC is disabled. If the Skip flag (2801) is true (2802), the Merge index is decoded and the current CU is Regular Skip (2809). Otherwise, the Skip Non Regular flag is decoded (2801b). If this flag is true (2802b), the current CU has a zero residual and the Merge data with the other Merge flags will be decoded (2808).
This embodiment can be combined with other embodiments as those where the IBC mode is coded with the other Merge flags or with the embodiment where the Fred mode is set before the Merge flag.
The advantages of this embodiment include coding efficiency increase in average (some cases improvement, and negligible losses in others); and a cleaner design for the en/decoder and which leads to simpler implementation in software and/or hardware.
In one feature/embodiment, the Regular Merge mode type is set at the last position of the list of Merge modes when a CU is Merge (Merge flag equal to 1). Figure 21 illustrates this embodiment. Table 2 shows example Merge flag signalling for this embodiment: Signaled Modes IBC flag MMVD Flag Subblock Flag CIIP Flag Triangular Flag IBC 1 MMVD 0 1 Subblock 0 0 1 CUP 0 0 0 1 Triangle 0 0 0 0 1 Regular 0 0 0 0 0 Table 2 -Merge flags signaling Moving Regular Merge mode to the end of the list reduces the number of conditions to decode a flag as the conditions only depend on the availability of the related mode, and not in the other Merge modes.
Figure 21 is based on Figure 13. When the current mode is IBC (2121) the Merge index is decoded (2104) otherwise the MMVD flag can be decoded (2106) if the MMVD Cond2 is true. Compared to the current VVC specification the MMVD Cond2 is simplified as it depends only on the SPS signalling of MMVD. In this example, MMVD cond2 is defined as the following: sps mmvd enabled flag To be exhaustive -a condition prohibiting the size of CU being equal to 4x4 should be added; however, this condition would never happen in practice because the Merge mode is allowed for 4x4 CU only with IBC.
In the same ways the SubBlock flag (2110) is decoded if the SubBlock cond2 (2109) is true and it is given by the following formula: (MaxNum SubblockMergeCand > 0 && cbWi dth >= 8 && cb Hei ght >= 8) In the same ways the CUP flag (2114) is decoded if the CIEP_cond2 (2114) is true. This condition is the same as CIIP cond.
Eventually, the Triangle flag can be decoded (2122), if the Triangle cond2 is true (2117) and it is the same as the previous Triangle cond.
The advantage of this solution is that there is no issue when one or more mode is not allowed because each flag (M,MVD, SubBlock, CIIP, Triangle, (IBC)) depends only to its own SPS enabled flag and on its own block size restriction. This is possible because, the regular Merge is the only one which can't be disabled at SPS or slice level.
As such, the most selected mode is signalled with the shortest code word available.
Moreover, this method solves the issue for different corner cases without introducing additional conditions for the decoding of Merge flags as the decoding of a Merge mode flag depends only on the availability of its related Merge mode This also simplifies the design as the conditions of each flag are simpler that the usage of long conditions as defined in the embodiment described above with reference to Figure 18.
However, it is also possible to use the same conditions as shown in Figure 18. In that case the REGULAR MERGE mode stays the first Merge mode as this is the most selected; this solution provides coding efficiency gains when decoding the Merge type flags when some of them are disabled at the SPS level. As described above, this method may in some circumstances increase the amount of checking for the decoding of each flag.
In one additional embodiment, the Pred mode is set after the Merge flag. Figure 23 illustrates this embodiment. This Figure is based on Figure 19. Figure 23a shows a variation where the IBC mode is not allowed. When the CU is not a Skip CU (2302), the Merge Flag is decoded (2306), If the CU is not a Merge Cu (2307), the Pred mode is decoded (2303).
When the IBC mode is allowed, the IBC flag (2319) is encoded after the Skip whatever 25 the value of the Skip flag. Please note that the Pred mode decoded after the Merge flag is decoded only when the CU is not an IBC CU.
The advantages of this embodiment are a clean design (and as such a simpler implementation in software and/or hardware) and a coding efficiency increase. One reason for a coding efficiency gain is because the Pred mode is less selected than the Merge mode (especially when the zero residual Merge modes are used) so it is more efficient to set the Pred mode after the Merge flag.
In one additional embodiment, the IBC mode is signalled as a Merge mode. Figure 23a can be used for this case, Yet, when the CU is an AMVP IBC the IBC flag is coded after the Pred mode. Figure 24 shows the coding of the IBC inside the Merge data. In this Figure the IBC flag (4123) is set after the Triangle Flag (2422). First it is checked if the IBC Flag is signalled thanks to the IBC Cond condition (2411). The condition is the following: (sps IBC enabled flag && (cbWidth < 128 cbHeight < 128) && !(cbWidth =4 && cbHeight =4)) So the IBC flag is not decoded if it is not enabled as SPS level or if the CU block Size is not 128x128 as IBC is not allowed for 128x128 CU size, Or if the CU size is 4x4, in that case IBC flag is set to 1 as the Regular is not allowed for this size.
The advantages of this embodiment are a better coding efficiency, a clean design as all Merge modes are coded in the same way except the most selected one (Regular mode). And it 10 reduces in average, the number of bins or bits decoded when IBC is enabled.
It should be appreciated that, the IBC merge without residual may be signaled as the same way as the other Merge types without residual described above -for example by using cu_cbf, tu_cbf, sig_coeff or by using a further skip flag.
It should be appreciated that combinations of features described are possible and may give rise to synergistic results. One particular example is to use, the embodiment described with reference to Figure 18 is used for the Skip mode for a set of Merge types allowed for the Skip mode, and the Merge flag setting described with reference to Figure 19 for the other type of Merge modes. In this embodiment, for example, the Regular, IBC and Affine are allowed for Skip modes and MMVD, Triangle with zero residual are signalled thanks to a CBF equal to 0 with the Merge mode. But only the condition for the Regular and Affine need to be updated for the Skip mode. In that case the condition for the regular flag for Skip mode is: ( MaxNumSubblockMergeCand > 0 && cbWidth >= 8 && cbHeight >= 8) Which is the condition of the current SubBlock flag and no condition on SubBlock is needed for SubBlock Skip flag as it don't need be decoded. Indeed when the CU is not Regular it is sure that it is a SubBlock CU.
These embodiments result in each code word of the Merge modes flags corresponding to an available Merge type. This solves issues relating to unnecessary signalling of additional flags when some Merge modes are disabled. As such, coding efficiency improvement is achieved -notably for the Low Delay P (LDP) configuration with the current OTCs, and also for some "corner cases". Moreover many of the above embodiments reduce the amount conditions to decode a flag which further improves the coding efficiency. Eventually only one bit is needed to decode the most probable mode for a CU instead of 2 with the previous design.
Encoder: The above description has focussed on the method from a decoder side, but an encoder performs essentially the same process when encoding an image or image portion into a bitstream. However, there are particular features which relate to processes undertaken at the encoder which improve the coding efficiency of the encoder.
In one embodiment, the cu cbf flag is estimated in the rate distortion estimation (1606) when the zero residual signalling of SubBlock MMVD Triangle is used. Figure 25 illustrates this embodiment. This Figure is based on Figure 17, but has been adapted to match the removal of the Regular flag signalling and the Triangle Flag signalling (2501, 2502, 2524, 2525). In this Figure when the current Merge mode is MMVD, SubBlock or Triangle the rate of the cu cbf flag is evaluated to determine the number of bits (2523) to be used when calculating the rate distortion.
The advantage of this embodiment is a coding efficiency improvement.
This encoder embodiment can be adapted to match the usage of a cu cbf equals to zero to signal a zero residual. For example the when the CIIP has a zero residual with a cu cbf equal to zero, the cu cbf flag is evaluated.
Figure 26 shows a variation of Figure 16 which sets the variable BestlsSkip. 'The cu cbf value is not already determined when the rate distortion is evaluated in the first loop (1605, 1606, 1607) because this value is evaluated in the full RD loop (2610-2618).
In an additional embodiment, the cu_cbf value is set equal to the cu_cbf value of the current BestCu.
The advantage of this embodiment is a coding efficiency improvement. Indeed, when the bestCu as no residual there is a lot of chance that the selected best parameters set for the current CU has no residual.
In one embodiment, the REGULAR MERGE mode is evaluated with only the rate cost of its Merge idx in the rate distortion estimation (1606). Figure 27 illustrates this embodiment. This Figure is based on Figure 25. But can be adapted depending on the configuration used. In this Figure when the current evaluated mode is the REGULAR MERGE mode (2726), only the rate of the Merge idx (2704) is evaluated.
The advantage of this embodiment is a coding efficiency improvement.
In a variation, when IBC is used the IBC MERGE mode is evaluated with only the rate cost of its Merge idx in the rate distortion estimation (1606). This can be implemented by changing the order of modules 2722 and 2721 in Figure 27. The advantage of this embodiment is a coding efficiency improvement.
Implementation of the invention Figure 29 shows a system 191 195 comprising at least one of an encoder 150 or a decoder 100 and a communication network 199 according to embodiments of the present invention. According to an embodiment, the system 195 is for processing and providing a content (for example, a video and audio content for displaying/outputting or streaming video/audio content) to a user, who has access to the decoder 100, for example through a user interface of a user terminal comprising the decoder 100 or a user terminal that is communicable with the decoder 100. Such a user terminal may be a computer, a mobile phone, a tablet or any other type of a device capable of providing/displaying the (provided/streamed) content to the user. The system 195 obtains/receives a bitstream 101 (in the form of a continuous stream or a signal -e.g. while earlier video/audio are being displayed/output) via the communication network 199. According to an embodiment, the system 191 is for processing a content and storing the processed content, for example a video and audio content processed for displaying/outputting/streaming at a later time. The system 191 obtains/receives a content comprising an original sequence of images 151, which is received and processed (including filtering with a deblocking filter according to the present invention) by the encoder 150, and the encoder 150 generates a bitstream 101 that is to be communicated to the decoder 100 via a communication network 191. The bitstream 101 is then communicated to the decoder 100 in a number of ways, for example it may be generated in advance by the encoder 150 and stored as data in a storage apparatus in the communication network 199 (e.g. on a server or a cloud storage) until a user requests the content (i.e. the bitstream data) from the storage apparatus, at which point the data is communicated/streamed to the decoder 100 from the storage apparatus. The system 191 may also comprise a content providing apparatus for providing/streaming, to the user (e.g. by communicating data for a user interface to be displayed on a user terminal), content information for the content stored in the storage apparatus (e.g. the title of the content and other meta/storage location data for identifying, selecting and requesting the content), and for receiving and processing a user request for a content so that the requested content can be delivered/streamed from the storage apparatus to the user terminal. Alternatively, the encoder 150 generates the bitstream 101 and communicates/streams it directly to the decoder 100 as and when the user requests the content. The decoder 100 then receives the bitstream 101 (or a signal) and performs filtering with a deblocking filter according to the invention to obtain/generate a video signal 109 and/or audio signal, which is then used by a user terminal to provide the requested content to the user.
Any step of the method/process according to the invention or functions described herein may be implemented in hardware, software, firmware, or any combination thereof If implemented in software, the steps/functions may be stored on or transmitted over, as one or more instructions or code or program, or a computer-readable medium, and executed by one or more hardware-based processing unit such as a programmable computing machine, which may be a PC ("Personal Computer"), a DSP ("Digital Signal Processor"), a circuit, a circuitry, a processor and a memory, a general purpose microprocessor or a central processing unit, a microcontroller, an ASIC ("Application-Specific Integrated Circuit"), a field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques describe herein.
Embodiments of the present invention can also be realized by wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of JCs (e.g. a chip set). Various components, modules, or units are described herein to illustrate functional aspects of devices/apparatuses configured to perform those embodiments, but do not necessarily require realization by different hardware units. Rather, various modules/units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors in conjunction with suitable software/firmware.
Embodiments of the present invention can be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium to perform the modules/units/functions of one or more of the above-described embodiments and/or that includes one or more processing unit or circuits for performing the functions of one or more of the above-described embodiments, and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiments and/or controlling the one or more processing unit or circuits to perform the functions of one or more of the above-described embodiments. The computer may include a network of separate computers or separate processing units to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a computer-readable medium such as a communication medium via a network or a tangible storage medium. The communication medium may be a signaFbitstream/carrier wave. The tangible storage medium is a "non-transitory computer-readable storage medium" which may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like. At least some of the steps/functions may also be implemented in hardware by a machine or a dedicated component, such as an FPGA ("Field-Programmable Gate Array") or an ASIC ("Application-Specific Integrated Circuit").
Figure 30 is a schematic block diagram of a computing device 3600 for implementation of one or more embodiments of the invention. The computing device 3600 may be a device such as a micro-computer, a workstation or a light portable device. The computing device 3600 comprises a communication bus connected to: -a central processing unit (CPU) 3601, such as a microprocessor; -a random access memory (RAM) 3602 for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method for encoding or decoding at least part of an image according to embodiments of the invention, the memory capacity thereof can be expanded by an optional RAM connected to an expansion port for example; -a read only memory (ROM) 3603 for storing computer programs for implementing embodiments of the invention; -a network interface (NET) 3604 is typically connected to a communication network over which digital data to be processed are transmitted or received. The network interface (NET) 3604 can be a single network interface, or composed of a set of different network interfaces (for instance wired and wireless interfaces, or different kinds of wired or wireless interfaces). Data packets are written to the network interface for transmission or are read from the network interface for reception under the control of the software application running in the CPU 3601; -a user interface (UI) 3605 may be used for receiving inputs from a user or to display information to a user; -a hard disk (HD) 3606 may be provided as a mass storage device; -an Input/Output module (JO) 3607 may be used for receiving/sending data from/to external devices such as a video source or display. The executable code may be stored either in the ROM 3603, on the HD 3606 or on a removable digital medium such as, for example a disk. According to a variant, the executable code of the programs can be received by means of a communication network, via the NET 3604, in order to be stored in one of the storage means of the communication device 3600, such as the HD 3606, before being executed.
The CPU 3601 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to embodiments of the invention, which instructions are stored in one of the aforementioned storage means. After powering on, the CPU 3601 is capable of executing instructions from main RAM memory 3602 relating to a software application after those instructions have been loaded from the program ROM 3603 or the FM 3606, for example. Such a software application, when executed by the CPU 3601, causes the steps of the method according to the invention to be performed.
It is also understood that according to another embodiment of the present invention, a decoder according to an aforementioned embodiment is provided in a user terminal such as a computer, a mobile phone (a cellular phone), a table or any other type of a device (e.g. a display apparatus) capable of providing/displaying a content to a user. According to yet another embodiment, an encoder according to an aforementioned embodiment is provided in an image capturing apparatus which also comprises a camera, a video camera or a network camera (e.g. a closed-circuit television or video surveillance camera) which captures and provides the content for the encoder to encode. Two such examples are provided below with reference to Figures 37 and 38.
Figure 31 is a diagram illustrating a network camera system 3700 including a network camera 3702 and a client apparatus 202.
The network camera 3702 includes an imaging unit 3706, an encoding unit 3708, a communication unit 3710, and a control unit 3712.
The network camera 3702 and the client apparatus 202 are mutually connected to be able to communicate with each other via the network 200.
The imaging unit 3706 includes a lens and an image sensor (e.g., a charge coupled 20 device (CCD) or a complementary metal oxide semiconductor (CMOS)), and captures an image of an object and generates image data based on the image. This image can be a still image or a video image.
The encoding unit 3708 encodes the image data by using said encoding methods explained above. or a combination of encoding methods described above.
The communication unit 3710 of the network camera 3702 transmits the encoded image data encoded by the encoding unit 3708 to the client apparatus 202.
Further, the communication unit 3710 receives commands from client apparatus 202. The commands include commands to set parameters for the encoding of the encoding unit 3708. The control unit 3712 controls other units in the network camera 3702 in accordance 30 with the commands received by the communication unit 3712.
The client apparatus 202 includes a communication unit 3714, a decoding unit 3716, and a control unit 3718.
The communication unit 3714 of the client apparatus 202 transmits the commands to the network camera 3702.
Further, the communication unit 3714 of the client apparatus 202 receives the encoded image data from the network camera 3712.
The decoding unit 3716 decodes the encoded image data by using said decoding methods explained above, or a combination of the decoding methods explained above.
The control unit 3718 of the client apparatus 202 controls other units in the client apparatus 202 in accordance with the user operation or commands received by the communication unit 3714.
The control unit 3718 of the client apparatus 202 controls a display apparatus 2120 so as to display an image decoded by the decoding unit 3716.
The control unit 3718 of the client apparatus 202 also controls a display apparatus 2120 so as to display GUI (Graphical User Interface) to designate values of the parameters for the network camera 3702 includes the parameters for the encoding of the encoding unit 3708. The control unit 3718 of the client apparatus 202 also controls other units in the client apparatus 202 in accordance with user operation input to the GUI displayed by the display apparatus 2120.
The control unit 3718 of the client apparatus 202 controls the communication unit 3714 of the client apparatus 202 so as to transmit the commands to the network camera 3702 which designate values of the parameters for the network camera 3702, in accordance with the user operation input to the GUI displayed by the display apparatus 2120.
Figure 32 is a diagram illustrating a smart phone 3800.
The smart phone 3800 includes a communication unit 3802, a decoding unit 3804, a control unit 3806 and a display unit 3808.
the communication unit 3802 receives the encoded image data via network 200.
The decoding unit 3804 decodes the encoded image data received by the communication unit 3802.
The decoding / encoding unit 3804 decodes / encodes the encoded image data by using said decoding methods explained above above..
The control unit 3806 controls other units in the smart phone 3800 in accordance with a user operation or commands received by the communication unit 3806.
For example, the control unit 3806 controls a display unit 3808 so as to display an image decoded by the decoding unit 3804. The smart phone 3800 may also comprise sensors 3812 and an image recording device 3810. In such a way, the smart phone 3800 may record images, encode the images (using a method described above).
The smart phone 3800 may subsequently decode the encoded images (using a method described above) and display them via the display unit 3808 -or transmit the encoded images to another device via the communication unit 3802 and network 200.
Alternatives and modifications While the present invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. It will be appreciated by those skilled in the art that various changes and modification might be made without departing from the scope of the invention, as defined in the appended claims. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.
It is also understood that any result of comparison, determination, assessment, selection, execution, performing, or consideration described above, for example a selection made during an encoding or filtering process, may be indicated in or determinable/inferable from data in a bitstream, for example a flag or data indicative of the result, so that the indicated or determined/inferred result can be used in the processing instead of actually performing the comparison, determination, assessment, selection, execution, performing, or consideration, for example during a decoding process.
In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used.
Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

Claims (1)

  1. CLAIMS1 A method of decoding an image or image portion from a bit stream; the method comprising: decoding a Skip flag from the bitstream; if said flag is enabled, setting the mode to Skip Regular; if said flag is disabled, determining a prediction mode and decoding corresponding data 2. A method according to claim I further comprising indicating a motion predictor merge mode for said determined prediction mode.3. A method according to claim 2 further comprising indicating a zero residual for said determined motion merge predictor mode.4. A method according to claim 3 further comprising not indicating the zero residual for Regular merge mode among said determined motion merge predictor modes.A method according to claim 3 or 4 wherein said determined motion predictor merge mode comprises one of SubBlock, Motion Vector Difference (MMVD), Triangle or Combined Inter/Intra Prediction (CLIP).6. A method according to any preceding claim further comprising decoding an Intra Block Copy (IBC) flag after determining said prediction mode.7. A method according to any preceding claim wherein if said skip flag is enabled, an Intra Block Copy (IBC) flag is decoded from the bitstream if the Luma size of the current coding unit (CU) is not 128x128.8. A method according to any preceding claim wherein decoding the data comprises determining whether the prediction mode is Regular (2018) and setting a coding block flag in dependence on said determining.9. A method according to any preceding claim wherein decoding the data comprises determining whether the prediction mode is Regular or Intra Block Copy (IBC) (2018) and setting a coding block flag in dependence on said determining.10. A method according to any preceding claim wherein decoding the data comprises determining whether the prediction mode is Regular, Combined Inter/Intra Prediction (CUP) or IBC (2018) and setting a coding block flag in dependence on said determining.11. A method according to any of claims 8 to 10 wherein in the case that the prediction mode is one of said modes inferring the coding block flag (2007).12. A method according to claim 8 to 10 wherein in the case that the prediction mode is not one of said modes decoding the coding block flag (2008) from the bitstream.13. A method according to any preceding claim further comprising determining a merge flag, and said coding block flag has a context depending on said merge flag.14. A method according to claim 3 to 13 wherein if said skip flag is disabled, setting a transform unit coding block flag to zero so as to indicate zero residual.15. A method according to claim 3 to 13 wherein if said skip flag is disabled, setting sig coeff so as to indicate zero residual for said determined merge modes.16. A method according to any of claims 3 to 13 wherein if said skip flag is disabled, setting a second skip flag so as to indicate zero residual for said determined merge mode.17. A method according to any preceding claim comprising determining a merge flag before said determining said prediction mode.18. A method according to any preceding claim wherein determining said prediction mode comprises determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.19. A method of decoding an image or image portion from a bit stream, the bit stream comprising a plurality of flags, each flag indicating a respective motion prediction mode; the method comprising: determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.20. The method according to claim 19 where n the first mode is regular mode.21. The method according to any of claims 18 to 20 wherein said determining comprises verifying a condition relating to at least one of all of the subsequent motion prediction modes.22. A method according to any of claims 1 to 16 wherein the regular mode is set at the last position of the list of Merge modes when a coding unit is Merge.23. A method of encoding an image or image portion into a bit stream; the method comprising: encoding a Skip flag into the bitstream; if said flag is enabled, setting the mode to Skip Regular; if said flag is disabled, determining a prediction mode and encoding corresponding data.24. A method of encoding an image or image portion into a bit stream, the bit stream comprising a plurality of flags, each flag indicating a respective motion prediction mode; the method comprising: determining whether to encode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said determining comprising: verifying a condition relating to said motion prediction mode; and verifying a condition relating to at least one of the subsequent motion prediction modes.25. The method according to claim 23 or 24 wherein determining a motion prediction mode comprises determining a mode with the lowest rate distortion.26. The method according to claim 25 wherein determining a mode with the lowest rate distortion comprises evaluating a coding block flag corresponding to said motion prediction mode.27. The method according to claim 26 comprising setting the coding block flag value equal to the coding block flag value of a current BestCu 28. The method according to claim 25 wherein said determined motion prediction mode is Regular and the rate distortion is estimated based on a Merge idx corresponding to the Regular mode.29. The method according to claim 25 wherein said determined motion prediction mode is IBC and the rate distortion is estimated based on a Merge idx corresponding to the IBC mode 30. A device for decoding an image or image portion from a bit stream, the device comprising: means for decoding a Skip flag from the bitstream; means for setting the mode to Skip Regular if said flag is enabled; means for determining a prediction mode if said flag is disabled; and means for decoding corresponding data.31. A device for encoding an image or image portion from a bit stream, the device comprising: means for encoding a Skip flag from the bitstream; means for setting the mode to Skip Regular if said flag is enabled; means for determining a prediction mode if said flag is disabled; and means for encoding corresponding data.32. A device for decoding an image or image portion from a bit stream, the device comprising: means for determining whether to decode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said means for determining comprising: means for verifying a condition relating to said motion prediction mode; and means for verifying a condition relating to at least one of the subsequent motion prediction modes.33 A device for encoding an image or image portion from a bit stream, the device comprising: means for determining whether to encode a flag indicating a motion prediction mode, said flag being determined before at least one subsequent flag; said means for determining comprising: means for verifying a condition relating to said motion prediction mode; and means for verifying a condition relating to at least one of the subsequent motion prediction modes.34. A program which, when executed by a computer or processor, causes the computer or processor to carry out the method of any one of claims t to 29
GB1909041.4A 2019-06-24 2019-06-24 Video coding and decoding Withdrawn GB2585017A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
GB1909041.4A GB2585017A (en) 2019-06-24 2019-06-24 Video coding and decoding
GB1912356.1A GB2591721A (en) 2019-06-24 2019-08-28 Video coding and decoding

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB1909041.4A GB2585017A (en) 2019-06-24 2019-06-24 Video coding and decoding

Publications (2)

Publication Number Publication Date
GB201909041D0 GB201909041D0 (en) 2019-08-07
GB2585017A true GB2585017A (en) 2020-12-30

Family

ID=67511728

Family Applications (2)

Application Number Title Priority Date Filing Date
GB1909041.4A Withdrawn GB2585017A (en) 2019-06-24 2019-06-24 Video coding and decoding
GB1912356.1A Withdrawn GB2591721A (en) 2019-06-24 2019-08-28 Video coding and decoding

Family Applications After (1)

Application Number Title Priority Date Filing Date
GB1912356.1A Withdrawn GB2591721A (en) 2019-06-24 2019-08-28 Video coding and decoding

Country Status (1)

Country Link
GB (2) GB2585017A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12047598B2 (en) 2020-03-16 2024-07-23 Beijing Dajia Internet Information Technology Co., Ltd. Merge mode with motion vector differences

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3979645A4 (en) * 2019-06-19 2022-08-10 LG Electronics Inc. Motion prediction-based image coding method and device
EP4029253A4 (en) * 2019-12-30 2022-10-19 Huawei Technologies Co., Ltd. Method and apparatus of harmonizing weighted prediction with non-rectangular merge modes
CN113163212B (en) * 2020-01-07 2024-08-13 腾讯科技(深圳)有限公司 Video decoding method and device, video encoding method and device, medium and equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018121506A1 (en) * 2016-12-27 2018-07-05 Mediatek Inc. Method and apparatus of bilateral template mv refinement for video coding
GB2563936A (en) * 2017-06-30 2019-01-02 Canon Kk Method and apparatus for encoding or decoding a flag during video data encoding
GB2564133A (en) * 2017-07-04 2019-01-09 Canon Kk Method and apparatus for encoding or decoding video data with sub-pixel motion vector refinement
WO2019016287A1 (en) * 2017-07-19 2019-01-24 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method of coding of pictures

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018121506A1 (en) * 2016-12-27 2018-07-05 Mediatek Inc. Method and apparatus of bilateral template mv refinement for video coding
GB2563936A (en) * 2017-06-30 2019-01-02 Canon Kk Method and apparatus for encoding or decoding a flag during video data encoding
GB2564133A (en) * 2017-07-04 2019-01-09 Canon Kk Method and apparatus for encoding or decoding video data with sub-pixel motion vector refinement
WO2019016287A1 (en) * 2017-07-19 2019-01-24 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Apparatus and method of coding of pictures

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12047598B2 (en) 2020-03-16 2024-07-23 Beijing Dajia Internet Information Technology Co., Ltd. Merge mode with motion vector differences

Also Published As

Publication number Publication date
GB201912356D0 (en) 2019-10-09
GB201909041D0 (en) 2019-08-07
GB2591721A (en) 2021-08-11

Similar Documents

Publication Publication Date Title
JP7530465B2 (en) Video Encoding and Decoding
JP7514236B2 (en) Video Encoding and Decoding
GB2582929A (en) Residual signalling
JP7514345B2 (en) Motion Vector Predictor Index Coding in Video Coding
GB2585017A (en) Video coding and decoding
GB2585019A (en) Residual signalling
JP7321345B2 (en) video encoding and decoding
WO2023052489A1 (en) Video coding and decoding
GB2611367A (en) Video coding and decoding
GB2585018A (en) Residual signalling
JP7413576B2 (en) Video encoding and decoding
WO2023202956A1 (en) Video coding and decoding
GB2617569A (en) Data coding and decoding
GB2617568A (en) Video coding and decoding
GB2628209A (en) Image and video coding and decoding
GB2589735A (en) Video coding and decoding
GB2597616A (en) Video coding and decoding

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)