Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/174—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a slice, e.g. a line of blocks or a group of blocks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
  • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/146—Data rate or code amount at the encoder output
  • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/188—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/513—Processing of motion vectors
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards

Definitions

• the present invention relates to video coding and decoding.
• VVC Versatile Video Coding
• JVET Since the end of the standardisation of VVC vl, JVET has launched an exploration phase by establishing an exploration software (ECM). It gathers additional tools and improvements of existing tools on top of the VVC standard to target better coding efficiency.
• ECM exploration software
• VVC has a modified set of ‘merge modes’ for motion vector prediction which achieves greater coding efficiency at a cost of greater complexity.
• Motion vector prediction is enabled by deriving a list of ‘motion vector predictor candidates’ with the index of the selected candidate being signalled in the bitstream.
• the merge candidate list is generated for each coding unit (CU). But CUs may be split into smaller blocks for Decoder-side Motion Vector Refinement (DMVR) or other methods.
• DMVR Decoder-side Motion Vector Refinement
• the make-up and order of this list can have significant impact on coding efficiency as an accurate motion vector predictor reduces the size of the residual or the distortion of the block predictor, and having such a candidate at the top of the list reduces the number of bits required to signal the selected candidate.
• the present invention aims to improve at least one of these aspects.
• Modifications incorporated into VVC vl and ECM mean there can be up to 10 motion vector predictor candidates; this enables a diversity of candidates, but the bitrate can increase if candidates lower down the list are selected.
• the present invention broadly relates to how to transmit / receive (or derive) a value which enables a decoder to make decoding decisions. For example, how to reorder the list of candidates.
• a method of decoding a portion of an image comprising, determining a threshold from a header associated with the image portion, wherein a decoding criterion is based on said threshold; and decoding said image portion based on said criterion.
• the method may comprise decoding a syntax element in the header which is capable of indicating the presence of the threshold in that header.
• the syntax element is also capable of indicating that the threshold has a default value.
• the syntax element is also capable of indicating that no threshold is transmitted and that the decoding process uses an alternative method.
• the method may further comprise skipping decoding of the corresponding syntax elements in hierarchically lower headers in the case where the syntax element indicates the presence of the threshold.
• the syntax element is a flag
• the present invention is applicable to a number of decoding tools, for example the decoding comprises a decision to perform motion vector refinement based on the criterion. This improves the performance of this decoding tool.
• the decoding comprises a decision to perform motion vector ordering or removing based on the criterion. Appropriate selection of a decoding criterion leads to a more accurately ordered list, and thus a rate reduction.
• the method further comprises determining a coding mode and determining the threshold is based on the determined mode.
• the criterion relates to a cost.
• Threshold values signalling comprises decoding a syntax element indicating a maximum number of bits of said threshold,
• determining the threshold comprises decoding a syntax element indicating the number of bits of said threshold.
• the method comprises transforming the decoded threshold prior to using it in the decoding process.
• the transforming comprises dequantizing, performing a left bit shift and/or squaring.
• the threshold value may be predicted.
• the method comprises decoding a flag indicating whether decoding the threshold comprises predicting the threshold or directly decoding the threshold from the header.
• determining the threshold may comprise predicting the threshold or directly decoding the threshold from the header when not predictable.
• the method comprises decoding a residual and modifying the predicted threshold value based on said residual.
• the predicted threshold value is predicted based on a previously decoded threshold.
• the previously decoded threshold is from a hierarchically higher level than said header.
• the predicted threshold value is predicted based on a syntax element relating to the same header.
• the syntax element is one which is indicative of the reconstructed quality of the image portion.
• the syntax element is a quantization parameter or quantization parameter offset. This element is already known by the decoder and is likely to be correlated to a threshold.
• the method comprises either: (a) decoding a flag indicating that the threshold is correctly predicted; or (b) verifying if the threshold is associated with a syntax element relating to the same header.
• the method may comprise decoding a flag indicating whether to use option (a) or (b).
• an association between the syntax element relating to the same header and a threshold value may be determined from a table.
• the table is transmitted in a hierarchically higher level than the header.
• each syntax element may be transmitted along with its associated threshold so as to generate the table.
• the table is predictively coded.
• the table is generated without duplicates.
• the table is used to determine a block level threshold when the syntax element is defined at the block level.
• Threshold represented by an index
• determining the threshold comprises decoding an index indicating a threshold value from among a plurality of threshold values known by the decoder. This allows for signalling with a reduced number of bits.
• the method comprises receiving the plurality of threshold values.
• the plurality of threshold values are transmitted in a hierarchically higher level than the header.
• the method may comprise decoding a variable indicating the number of received threshold values.
• the plurality of threshold values may be transmitted in a header at a level relating to a plurality of image portions.
• the header at a level relating to a plurality of image portions is a sequence parameter set header.
• one value of the index may indicate that a threshold value not in the plurality of threshold values is to be decoded from the header.
• the value of the index is a maximum value.
• the method further comprises adding the threshold value decoded from the header to the plurality of threshold values and associating this with the index.
• a method of encoding a header associated with a plurality of image portions comprising: determining a threshold for each image portion, said threshold to be used to determine a decoding criterion for said image portion; generating an index of threshold values; wherein said index is ordered based on a frequency of occurrence of the threshold values; and encoding the index into the header.
• the header is a sequence parameter set (SPS).
• SPS sequence parameter set
• a method of encoding a portion of an image comprising, determining a decoding criterion for the image portion determining a threshold used to determine the criterion; signalling the threshold in a header associated with the image portion; and encoding said image portion based on said criterion.
• an apparatus for encoding image data into a bitstream said apparatus being configured to perform any aspect or embodiment recited above.
• an apparatus for decoding image data from a bitstream said apparatus being configured perform any aspect or embodiment recited above.
• a (computer) program which when upon execution causes a programmable apparatus to perform a method according to any aspect or embodiment recited above.
• the program may be store on a computer readable storage medium.
• the program may be provided on its own or may be carried on, by or in a carrier medium.
• the carrier medium may be non-transitory, for example a storage medium, in particular a computer-readable storage medium.
• the carrier medium may also be transitory, for example a signal or other transmission medium.
• the signal may be transmitted via any suitable network, including the Internet.
• Figure 1 is a diagram for use in explaining a coding structure used in HEVC
• Figure 2 is a block diagram schematically illustrating a data communication system in which one or more embodiments of the invention may be implemented;
• FIG. 3 is a block diagram illustrating components of a processing device in which one or more embodiments of the invention may be implemented;
• Figure 4 is a flow chart illustrating steps of an encoding method according to embodiments of the invention.
• Figure 5 is a flow chart illustrating steps of a decoding method according to embodiments of the invention.
• Figures 6 and 7 show the labelling scheme used to describe blocks situated relative to a current block
• Figures 8(a) and (b) illustrate the Affine (SubBlock) mode
• Figures 9(a), (b), (c), (d) illustrate the geometric mode
• Figure 10 illustrates the first steps of the Merge candidates list derivation of VVC
• FIG. 11 illustrates further steps of the Merge candidates list derivation of VVC
• Figure 12 illustrates the derivation of a pairwise candidate
• Figure 13 illustrates the template matching method based on neighbouring samples
• Figure 14 illustrates a modification of the first steps of the Merge candidates list derivation shown in Figure 10;
• Figure 15 illustrates a modification of the further steps of the Merge candidates list derivation shown in Figure 11;
• Figure 16 illustrates a modification of the derivation of a pairwise candidate shown in Figure 12;
• Figure 17 illustrates the costs determination of a list candidates
• Figure 18 illustrates the reordering process of the list of Merge mode candidates
• Figure 19 illustrates the pairwise candidate derivation during the reordering process of the list of Merge mode candidates
• Figure 20 illustrates the ordering process of predictors based on distortion values.
• Figure 21 illustrates the structure of the bitstream in the exemplary coding system VVC
• Figure 22 is a diagram showing a system comprising an encoder or a decoder and a communication network according to embodiments of the present invention.
• Figure 23 is a schematic block diagram of a computing device for implementation of one or more embodiments of the invention.
• Figure 24 is a diagram illustrating a network camera system
• Figure 25 is a diagram illustrating a smart phone
• Figure 1 relates to a coding structure used in the High Efficiency Video Coding (HEVC) video and Versatile Video Coding (VVC) standards.
• a video sequence 1 is made up of a succession of digital images i. Each such digital image is represented by one or more matrices. The matrix coefficients represent pixels.
• An image 2 of the sequence may be divided into slices 3.
• a slice may in some instances constitute an entire image.
• These slices are divided into non-overlapping Coding Tree Units (CTUs).
• a Coding Tree Unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and conceptually corresponds in structure to macroblock units that were used in several previous video standards.
• a CTU is also sometimes referred to as a Largest Coding Unit (LCU).
• LCU Largest Coding Unit
• a CTU has luma and chroma component parts, each of which component parts is called a Coding Tree Block (CTB). These different color components are not shown in Figure 1.
• CTB Coding Tree Block
• a CTU is generally of size 64 pixels x 64 pixels for HEVC, yet for VVC this size can be 128 pixels x 128 pixels.
• Each CTU may in turn be iteratively divided into smaller variablesize Coding Units (CUs) 5 using a quadtree decomposition.
• Coding units are the elementary coding elements and are constituted by two kinds of sub-unit called a Prediction Unit (PU) and a Transform Unit (TU).
• the maximum size of a PU or TU is equal to the CU size.
• a Prediction Unit corresponds to the partition of the CU for prediction of pixels values.
• Various different partitions of a CU into PUs are possible as shown by 606 including a partition into 4 square PUs and two different partitions into 2 rectangular PUs.
• a Transform Unit is an elementary unit that is subjected to spatial transformation using DCT.
• a CU can be partitioned into TUs based on a quadtree representation 607.
• NAL Network Abstraction Layer
• coding parameters of the video sequence are stored in dedicated NAL units called parameter sets.
• SPS Sequence Parameter Set
• PPS Picture Parameter Set
• HEVC also includes a Video Parameter Set (VPS) NAL unit which contains parameters describing the overall structure of the bitstream.
• the VPS is a type of parameter set defined in HEVC, and applies to all of the layers of a bitstream.
• a layer may contain multiple temporal sub-layers, and all version 1 bitstreams are restricted to a single layer.
• HEVC has certain layered extensions for scalability and multiview and these will enable multiple layers, with a backwards compatible version 1 base layer.
• VVC Video Codtures
• subpictures which are independently coded groups of one or more slices.
• FIG. 2 illustrates a data communication system in which one or more embodiments of the invention may be implemented.
• the data communication system comprises a transmission device, in this case a server 201, which is operable to transmit data packets of a data stream to a receiving device, in this case a client terminal 202, via a data communication network 200.
• the data communication network 200 may be a Wide Area Network (WAN) or a Local Area Network (LAN).
• WAN Wide Area Network
• LAN Local Area Network
• Such a network may be for example a wireless network (Wifi / 802.1 la or b or g), an Ethernet network, an Internet network or a mixed network composed of several different networks.
• the data communication system may be a digital television broadcast system in which the server 201 sends the same data content to multiple clients.
• the data stream 204 provided by the server 201 may be composed of multimedia data representing video and audio data. Audio and video data streams may, in some embodiments of the invention, be captured by the server 201 using a microphone and a camera respectively. In some embodiments data streams may be stored on the server 201 or received by the server 201 from another data provider, or generated at the server 201.
• the server 201 is provided with an encoder for encoding video and audio streams in particular to provide a compressed bitstream for transmission that is a more compact representation of the data presented as input to the encoder.
• the compression of the video data may be for example in accordance with the HEVC format or H.264/AVC format or VVC format.
• the client 202 receives the transmitted bitstream and decodes the reconstructed bitstream to reproduce video images on a display device and the audio data by a loud speaker.
• the data communication between an encoder and a decoder may be performed using for example a media storage device such as an optical disc.
• a video image is transmitted with data representative of compensation offsets for application to reconstructed pixels of the image to provide filtered pixels in a final image.
• FIG. 3 schematically illustrates a processing device 300 configured to implement at least one embodiment of the present invention.
• the processing device 300 may be a device such as a micro-computer, a workstation or a light portable device.
• the device 300 comprises a communication bus 313 connected to:
• central processing unit 311 such as a microprocessor, denoted CPU;
• ROM read only memory
• RAM random access memory 312, denoted RAM, for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method of encoding a sequence of digital images and/or the method of decoding a bitstream according to embodiments of the invention;
• the apparatus 300 may also include the following components:
• -a data storage means 304 such as a hard disk, for storing computer programs for implementing methods of one or more embodiments of the invention and data used or produced during the implementation of one or more embodiments of the invention;
• -a screen 309 for displaying data and/or serving as a graphical interface with the user, by means of a keyboard 310 or any other pointing means.
• the apparatus 300 can be connected to various peripherals, such as for example a digital camera 320 or a microphone 308, each being connected to an input/output card (not shown) so as to supply multimedia data to the apparatus 300.
• peripherals such as for example a digital camera 320 or a microphone 308, each being connected to an input/output card (not shown) so as to supply multimedia data to the apparatus 300.
• the communication bus provides communication and interoperability between the various elements included in the apparatus 300 or connected to it.
• the representation of the bus is not limiting and in particular the central processing unit is operable to communicate instructions to any element of the apparatus 300 directly or by means of another element of the apparatus 300.
• the disk 306 can be replaced by any information medium such as for example a compact disk (CD-ROM), rewritable or not, a ZIP disk or a memory card and, in general terms, by an information storage means that can be read by a microcomputer or by a microprocessor, integrated or not into the apparatus, possibly removable and adapted to store one or more programs whose execution enables the method of encoding a sequence of digital images and/or the method of decoding a bitstream according to the invention to be implemented.
• CD-ROM compact disk
• ZIP disk or a memory card
• the executable code may be stored either in read only memory 306, on the hard disk 304 or on a removable digital medium such as for example a disk 306 as described previously.
• the executable code of the programs can be received by means of the communication network 303, via the interface 302, in order to be stored in one of the storage means of the apparatus 300 before being executed, such as the hard disk 304.
• the central processing unit 311 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, instructions that are stored in one of the aforementioned storage means.
• the program or programs that are stored in a non-volatile memory for example on the hard disk 304 or in the read only memory 306, are transferred into the random access memory 312, which then contains the executable code of the program or programs, as well as registers for storing the variables and parameters necessary for implementing the invention.
• the apparatus is a programmable apparatus which uses software to implement the invention.
• the present invention may be implemented in hardware (for example, in the form of an Application Specific Integrated Circuit or ASIC).
• Figure 4 illustrates a block diagram of an encoder according to at least one embodiment of the invention.
• the encoder is represented by connected modules, each module being adapted to implement, for example in the form of programming instructions to be executed by the CPU 311 of device 300, at least one corresponding step of a method implementing at least one embodiment of encoding an image of a sequence of images according to one or more embodiments of the invention.
• An original sequence of digital images iO to in 401 is received as an input by the encoder 400.
• Each digital image is represented by a set of samples, sometimes also referred to as pixels (hereinafter, they are referred to as pixels).
• a bitstream 410 is output by the encoder 400 after implementation of the encoding process.
• the bitstream 410 comprises a plurality of encoding units or slices, each slice comprising a slice header for transmitting encoding values of encoding parameters used to encode the slice and a slice body, comprising encoded video data.
• the input digital images iO to in 401 are divided into blocks of pixels by module 402.
• the blocks correspond to image portions and may be of variable sizes (e.g. 4x4, 8x8, 16x16, 32x32, 64x64, 128x128 pixels and several rectangular block sizes can be also considered).
• a coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial prediction coding (Intra prediction), and coding modes based on temporal prediction (Inter coding, Merge, SKIP). The possible coding modes are tested.
• Module 403 implements an Intra prediction process, in which the given block to be encoded is predicted by a predictor computed from pixels of the neighbourhood of said block to be encoded. An indication of the selected Intra predictor and the difference between the given block and its predictor is encoded to provide a residual if the Intra coding is selected.
• Temporal prediction is implemented by motion estimation module 404 and motion compensation module 405.
• a reference image from among a set of reference images 416 is selected, and a portion of the reference image, also called reference area or image portion, which is the closest area (closest in terms of pixel value similarity) to the given block to be encoded, is selected by the motion estimation module 404.
• Motion compensation module 405 then predicts the block to be encoded using the selected area.
• the difference between the selected reference area and the given block, also called a residual block is computed by the motion compensation module 405.
• the selected reference area is indicated using a motion vector.
• a residual is computed by subtracting the predictor from the original block.
• a prediction direction is encoded.
• the Inter prediction implemented by modules 404, 405, 416, 418, 417 at least one motion vector or data for identifying such motion vector is encoded for the temporal prediction.
• Motion vector predictors from a set of motion information predictor candidates is obtained from the motion vectors field 418 by a motion vector prediction and coding module 417.
• the encoder 400 further comprises a selection module 406 for selection of the coding mode by applying an encoding cost criterion, such as a rate-distortion criterion.
• an encoding cost criterion such as a rate-distortion criterion.
• a transform such as DCT
• the transformed data obtained is then quantized by quantization module 408 and entropy encoded by entropy encoding module 409.
• the encoded residual block of the current block being encoded is inserted into the bitstream 410.
• the encoder 400 also performs decoding of the encoded image in order to produce a reference image (e.g. those in Reference images/pictures 416) for the motion estimation of the subsequent images. This enables the encoder and the decoder receiving the bitstream to have the same reference frames (reconstructed images or image portions are used).
• the inverse quantization (“dequantization”) module 411 performs inverse quantization (“dequantization”) of the quantized data, followed by an inverse transform by inverse transform module 412.
• the intra prediction module 413 uses the prediction information to determine which predictor to use for a given block and the motion compensation module 414 actually adds the residual obtained by module 412 to the reference area obtained from the set of reference images 416.
• Post filtering is then applied by module 415 to filter the reconstructed frame (image or image portions) of pixels.
• an SAO loop filter is used in which compensation offsets are added to the pixel values of the reconstructed pixels of the reconstructed image. It is understood that post filtering does not always have to performed. Also, any other type of post filtering may also be performed in addition to, or instead of, the SAO loop filtering.
• Figure 5 illustrates a block diagram of a decoder 60 which may be used to receive data from an encoder according an embodiment of the invention.
• the decoder is represented by connected modules, each module being adapted to implement, for example in the form of programming instructions to be executed by the CPU 311 of device 300, a corresponding step of a method implemented by the decoder 60.
• the decoder 60 receives a bitstream 61 comprising encoded units (e.g. data corresponding to a block or a coding unit), each one being composed of a header containing information on encoding parameters and a body containing the encoded video data.
• encoded units e.g. data corresponding to a block or a coding unit
• the encoded video data is entropy encoded, and the motion vector predictors’ indexes are encoded, for a given block, on a predetermined number of bits.
• the received encoded video data is entropy decoded by module 62.
• the residual data are then dequantized by module 63 and then an inverse transform is applied by module 64 to obtain pixel values.
• the mode data indicating the coding mode are also entropy decoded and based on the mode, an INTRA type decoding or an INTER type decoding is performed on the encoded blocks (units/sets/groups) of image data.
• an INTRA predictor is determined by intra prediction module 65 based on the intra prediction mode specified in the bitstream.
• the motion prediction information is extracted from the bitstream so as to find (identify) the reference area used by the encoder.
• the motion prediction information comprises the reference frame index and the motion vector residual.
• the motion vector predictor is added to the motion vector residual by motion vector decoding module 70 in order to obtain the motion vector.
• Motion vector decoding module 70 applies motion vector decoding for each current block encoded by motion prediction. Once an index of the motion vector predictor for the current block has been obtained, the actual value of the motion vector associated with the current block can be decoded and used to apply motion compensation by module 66. The reference image portion indicated by the decoded motion vector is extracted from a reference image 68 to apply the motion compensation 66. The motion vector field data 71 is updated with the decoded motion vector in order to be used for the prediction of subsequent decoded motion vectors. Finally, a decoded block is obtained. Where appropriate, post filtering is applied by post filtering module 67. A decoded video signal 69 is finally obtained and provided by the decoder 60.
• Figure 21 illustrates the organisation of the bitstream in the exemplary coding system VVC as describe in JVET-Q2001-vD.
• a bitstream 2100 according to the VVC coding system is composed of an ordered sequence of syntax elements and coded data.
• the syntax elements and coded data are placed into Network Abstraction Layer (NAL) units 2101-2108.
• NAL Network Abstraction Layer
• the network abstraction layer provides the ability to encapsulate the bitstream into different protocols, like RTP/IP, standing for Real Time Protocol / Internet Protocol, ISO Base Media File Format, etc.
• the network abstraction layer also provides a framework for packet loss resilience.
• NAL units are divided into Video Coding Layer (VCL) NAL units and non-VCL NAL units.
• VCL NAL units contain the actual encoded video data.
• the non-VCL NAL units contain additional information. This additional information may be parameters needed for the decoding of the encoded video data or supplemental data that may enhance usability of the decoded video data.
• NAL units 2106 correspond to slices and constitute the VCL NAL units of the bitstream.
• the Decoder Parameter Set (DPS) NAL unit 301 contains parameters that are constant for a given decoding process.
• the Video Parameter Set (VPS) NAL unit 2102 contains parameters defined for the whole video, and thus the whole bitstream.
• the DPS NAL unit may define parameters more static than the parameters in the VPS. In other words, the parameters of DPS change less frequently than the parameter of the VPS.
• the Sequence Parameter Set (SPS) NAL unit 2103 contains parameters defined for a video sequence.
• the SPS NAL unit may define the sub pictures layout and associated parameters of the video sequences.
• the parameters associated to each subpicture specifies the coding constraints applied to the subpicture. In particular, it comprises a flag indicating that the temporal prediction between subpictures is restricted to the data coming from the same subpicture. Another flag may enable or disable the loop filters across the subpicture boundaries.
• the Picture Parameter Set (PPS) NAL unit 2104 contains parameters defined for a picture or a group of pictures.
• the Adaptation Parameter Set (APS) NAL unit 2105 contains parameters for loop filters typically the Adaptive Loop Filter (ALF) or the reshaper model (or luma mapping with chroma scaling (LMCS) model) or the scaling matrices that are used at the slice level.
• ALF Adaptive Loop Filter
• LMCS luma mapping with chroma scaling
• the syntax of the PPS as proposed in the current version of VVC comprises syntax elements that specifies the size of the picture in luma samples and also the partitioning of each picture in tiles and slices.
• the PPS contains syntax elements that make it possible to determine the slices location in a frame. Since a subpicture forms a rectangular region in the frame, it is possible to determine the set of slices, the parts of tiles or the tiles that belong to a subpicture from the Parameter Sets NAL units.
• the PPS as with the APS have an ID mechanism to limit the amount of same PPS’s transmitted.
• the PPS is generally transmitted for a group of pictures compared to the PH which is systematically transmitted for each Picture. Accordingly, the PPS compared to the PH contains parameters which can be constant for several picture.
• the bitstream may also contain Supplemental Enhancement Information (SEI) NAL units (not represented in Figure 21).
• SEI Supplemental Enhancement Information
• a VPS that is defined for the whole bitstream may occur only once in the bitstream.
• an APS that is defined for a slice may occur once for each slice in each picture.
• different slices may rely on the same APS and thus there are generally fewer APS than slices in each picture.
• the APS is defined in the picture header.
• the ALF APS can be refined in the slice header.
• the Access Unit Delimiter (AUD) NAL unit 2107 separates two access units.
• An access unit is a set of NAL units which may comprise one or more coded pictures with the same decoding timestamp.
• This optional NAL unit contains only one syntax element in current VVC specification: pic type, this syntax element, indicates that the slice type values for all slices of the coded pictures in the AU. If pic type is set equal to 0, the AU contain only Intra slice. If equal to 1, it contains P and I slices. If equal to 2 it contains B, P or Intra slice This NAL unit contains only one syntax element the pic-type.
• VVC there is the possibilities to transmit a set of reference picture lists in the SPS or in Picture header or slice header.
• the principle is to identified the POC difference between a current picture and its reference frames. In that case, as generally several frames of GOP share the same POC difference between reference frame, only one set need to be transmitted. When the set has been already transmitted it can be transmitted with an index which save rate dedicated to this reference picture signalling.
• HE VC uses 3 different INTER modes: the Inter mode (Advanced Motion Vector Prediction (AMVP)), the “classical” Merge mode (i.e. the “non-Affine Merge mode” or also known as “n egular” Merge mode) and the “classical” Merge Skip mode (i.e. the “non-Affine Merge Skip” mode or also known as “regular” Merge Skip mode).
• AMVP Advanced Motion Vector Prediction
• classical Merge mode i.e. the “non-Affine Merge mode” or also known as “n egular” Merge mode
• the “classical” Merge Skip mode i.e. the “non-Affine Merge Skip” mode or also known as “regular” Merge Skip mode.
• the main difference between these modes is the data signalling in the bitstream.
• the current HEVC standard includes a competitive based scheme for Motion vector prediction which was not present in earlier versions of the standard.
• Intra Block Copy In the Screen Content Extension of HEVC, the new coding tool called Intra Block Copy (IBC) is signalled as any of those three INTER modes, the difference between IBC and the equivalent INTER mode being made by checking whether the reference frame is the current one. This can be implemented e.g. by checking the reference index of the list L0, and deducing this is Intra Block Copy if this is the last frame in that list. Another way to do is comparing the Picture Order Count of current and reference frames: if equal, this is Intra Block Copy.
• IBC Intra Block Copy
• Figures 6 show the labelling scheme used herein to describe blocks situated relative to a current block (i.e. the block currently being en/decoded) between frames (Fig. 6).
• VVC Merge modes In VVC several inter modes have been added compared to HEVC. In particular, new Merge modes have been added to the regular Merge mode of HEVC.
• MCP motion compensation prediction
• the affine motion field of the block is described by two control point motion vectors.
• the affine mode is a motion compensation mode like the Inter modes (AMVP, “classical” Merge, or “classical” Merge Skip). Its principle is to generate one motion information per pixel according to 2 or 3 neighbouring motion information. In the JEM, the affine mode derives one motion information for each 4x4 block as depicted in Figure 8(a) (each square is a 4x4 block, and the whole block in Figure 8(a) is a 16x16 block which is divided into 16 blocks of such square of 4x4 size - each 4x4 square block having a motion vector associated therewith).
• the Affine mode is available for the AMVP mode and the Merge modes (i.e. the classical Merge mode which is also referred to as “non-Affine Merge mode” and the classical Merge Skip mode which is also referred to as “non-Affine Merge Skip mode”), by enabling the affine mode with a flag.
• the Affine Mode is also known as SubBlock mode; these terms are used interchangeably in this specification.
• the subblock Merge mode of VVC contains a subblock-based temporal merging candidates, which inherit the motion vector field of a block in a previous frame pointed by a spatial motion vector candidate. This subblock candidate is followed by inherited affine motion candidate if the neighboring blocks have been coded with an inter affine mode of subblock merge and then some as constructed affine candidates are derived before some zero
• the VVC standard contains also the Combined Inter Merge / Intra prediction (CIIP) also known as MultiHypothesis Intra Inter (MHII) Merge mode.
• CIIP Combined Inter Merge / Intra prediction
• MHII MultiHypothesis Intra Inter
• the Combined Inter Merge / Intra prediction (CIIP) Merge can be considered as a combination of the regular Merge mode and the Intra mode and is described below with reference to Figure 10.
• the block predictor for the current block (1001) of this mode is an average between a Merge predictor block and an Intra predictor block as depicted in Figure 10.
• the Merge predictor block is obtained with exactly the same process of the Merge mode so it is a temporal block (1002) or bi-predictor of 2 temporal blocks. As such, a Merge index is signalled for this mode in the same manner as the regular Merge mode.
• the Intra predictor block is obtained based on the neighbouring sample (1003) of the current block (1001).
• the amount of available Intra modes for the current block is however limited compared to an Intra block.
• the Chroma predictor is equal to the Luma predictor.
• 1, 2 or 3 bits are used to signal the Intra predictor for a CIIP block.
• the CIIP block predictor is obtained by a weighted average of the Merge block predictor and the Intra block predictor.
• the weighting of the weighted average depends on the block size and/or the Intra predictor block selected.
• the obtained CIIP predictor is then added to the residual of the current block to obtain the reconstructed block.
• the CIIP mode is enabled only for non-Skipped blocks. Indeed, use of the CIIP Skip typically results in losses in compression performance and an increase in encoder complexity. This is because the CIIP mode has often a block residual in opposite to the other Skip mode. Consequently its signalling for the Skip mode increases the bitrate. - when the current CU is Skip, the CIIP is avoided. A consequence of this restriction is that the CIIP block can’t have a residual containing only 0 value as it is not possible to encode a VVC block residual equal to 0.
• CIIP should be interpreted in this specification as being a mode which combines features of Inter and Intra prediction, and not necessarily as a label given to one specific mode.
• the CIIP used the same Motion vector candidates list as the regular Merge mode.
• the MMVD MERGE mode is a specific regular Merge mode candidate derivation. It can be considered as an independent Merge candidates list.
• the selected MMVD Merge candidate, for the current CU is obtained by adding an offset value to one motion vector component (mvx or mvy) to an initial regular Merge candidate.
• the offset value is added to the motion vector of the first list L0 or to the motion vector of the second list LI depending on the configuration of these reference frames (both backward, both forward or forward and backward).
• the initial Merge candidate is signalled thanks to an index.
• the offset value is signalled thanks to a distance index between the 8 possible distances (1/4-pel, 1/2-pel, 1 -pel, 2 -pel, 4-pel, 8-pel, 16-pel, 32-pel) and a direction index giving the x or the y axis and the sign of the offset.
• the Geometric (GEO) MERGE mode is a particular bi-prediction mode.
• Figure 9 illustrates this particular block predictor generation.
• the block predictor contains one triangle from a first block predictor (901 or 911) and a second triangle from a second block predictor (902 or 912). But several other possible splits of the block are possible as depicted in Figure 9(c) and Figure 9(d).
• the Geometric Merge should be interpreted in this specification as being a mode which combines features of two Inter non square predictors, and not necessarily as a label given to one specific mode.
• Each partition (901 or 902) in the example of Figure 9(a), has a motion vector candidate which is a unidirectional candidate. And for each partition an index is signalling to obtain at decoder the corresponding motion vector candidate in a list of unidirectional candidates. And the first and the second can’t use the same candidate.
• This list of candidates comes from the regular Merge candidates list where for each candidate, one of the 2 components (L0 or LI) have been removed.
• IBC Intra block Copy
• the decoder side motion vector derivation (DMVR), in VVC, increases the accuracy of the MVs of the Merge mode.
• a bilateral-matching (BM) based decoder side motion vector refinement is applied.
• BM bilateral-matching
• a refined MV is searched around the initial MVs in the reference pictures list LO and reference pictures list LI.
• the BM method calculates the distortion between the two candidate blocks in the reference pictures list LO and list LI.
• the DMVR method has several steps for motion refinement regarding several sub pixel steps or subblock partitioning in ECM.
• One particularity of the VVC DMVR is that that a first cost is computed between the two initial blocks from LO and LI. When, the Sum of Absolute Different (SAD) is less than the number of samples of the current block, the refinements steps are not applied. This criterion is fixed and not adaptable.
• VVC integrates also a bi-directional optical flow (BDOF) tool.
• BDOF previously referred to as BIO
• BIO is used to refine the bi-prediction sig/nal of a CU at the 4/4 subblock level.
• BDOF is applied to a CU if it satisfies several conditions, especially if the distances (i.e. Picture Order Count (POC) difference) from two reference pictures to the current picture are the same.
• POC Picture Order Count
• the BDOF mode is based on the optical flow concept, which assumes that the motion of an object is smooth.
• a motion refinement (v_x, v_y ) is calculated by minimizing the difference between the LO and LI prediction samples. The motion refinement is then used to adjust the bi-predicted sample values in the 4x4 subblock.
• Prediction refinement with optical flow is used for affine mode.
• VVC also includes Adaptive Motion Vector Resolution (AMVR).
• AMVR allows the motion vector difference of the CU to be coded in different precision. For example for AMVP mode: quarter-luma-sample, half-luma-sample, integer-luma-sample or four-luma-sample are considered.
• the following table of the VVC specification gives the AMVR shift based on different syntax elements.
• AMVR can have an impact on the coding of the other modes than those using motion vector differences coding as the different Merge mode. Indeed, for some candidates the parameter hpellfldx, which represent an index on the luma interpolation filter for half pel precision, is propagated for some Merge candidate.
• the bi-prediction mode with CU-level weight is extended beyond simple averaging (as performed in HEVC) to allow weighted averaging of the two prediction signals P o and P ⁇ according to the following formula.
• the weight index, bcwlndex is signalled after the motion vector difference.
• the weight index is inferred from neighbouring blocks based on the merge candidate index.
• BCW is used only for CUs with 256 or more luma samples. Moreover, for low-delay pictures, all 5 weights are used. And for non-low-delay pictures, only 3 weights (wG ⁇ 3,4,5 ⁇ ) are used.
• the regular Merge list is derived as in Figure 10 and Figure 11.
• Bl (1002), Al (1006), BO (1010), AO (1014) are added if they exist.
• partial redundancy checks are performed, between the motion information of Al and Bl (1007) to add Al (1008), between the motion information of BO and Bl (1011) to add BO (1012) and between the motion information of AO and Al (1015) to add AO (1016).
• variable ent is incremented (1015, 1009, 1013, 1017, 1023, 1027, 1115, 1108).
• the candidate B2 (1019) is added (1022) if it has not the same motion information as Al and Bl (1021).
• the temporal candidate is added.
• the bottom right candidate (1024), if it is available (1025) is added (1026), otherwise the center temporal candidate (1028) is added (1026) if it exists (1029).
• the history based (HMVP) are added (1101), if they have not the same motion information as Al and Bl (1103).
• the number of history based candidates can’t exceed the maximum number of candidates minus 1 of the Merge candidates list (1102). So after the history based candidates there is at least one position missing in the merge candidates list.
• the pairwise candidate is built (1106) and added in the Merge candidates list (1107).
• the parameters the parameters BCWidx and useAltHpellf are set equal to the related parameters of the candidates.
• BCWidx is set equal to 0 and hpellfldxp is set equal to the hpellfldxp of the first candidate if it is equal to the hpellfldxp of the second candidate, and to 0 otherwise.
• the pairwise candidate is built (1106) according to the algorithm of Figure 12.
• the hpellfldxp is derived as mentioned previously (1204, 1202, 1203).
• the inter direction (interDir) is set equal to 0 (1205).
• L0 and LI if at least one reference frame is valid (different to -1) (1207), the parameters will be set. If both are valid (1208), the mv information for this candidate is derived (1209) and set equal to the reference frame of the first candidate and the motion information is the average between the 2 motion vectors for this list and the variable interDir is incremented. If only one of the candidates has motion information for this list (1210), the motion information for the pairwise candidate is set equal to this candidate (1212, 1211) and the inter direction variable interDir is incremented.
• JVET Since the end of the standardization of VVC vl, JVET has launched an exploration phase by establishing an exploration software (ECM). It gathers additional tools and improvements of existing tools on top of the VVC standard to target better coding efficiency. The different additional tools compared to VVC are described in JVET-X2025.
• the regular template matching is based on the template matching estimation as depicted in Figure 13.
• a motion estimation based on the neighboring samples of the current block (1301) and based on the neighboring samples of the multiple corresponding block positions, a cost is computed and the motion information which minimized the cost is selected.
• the motion estimation is limited by a search range and several restrictions on this search range are also used to reduce the complexity.
• the regular template matching candidates list is based on the regular Merge list but some additional steps and parameters have been added which means different Merge candidates lists for a same block may be generated. Moreover, only 4 candidates are available for the template matching regular Merge candidates list compared to the 10 candidates for the regular Merge candidates list in the ECM with common tests conditions defined by JVET.
• This pseudo code can be summarized as: for each reference frame index (unidirection), or pair of reference indexes (bi-prediction), a zero candidate is added. When all are added, only zero candidates with reference frames indexes 0 are added until that the number of candidates reaches its maximum value. In such a way, the Merge list can include multiple zero candidates. Indeed, it has been surprisingly found that this occurs frequently in real video sequences, particularly at the beginnings of slices or frames and sequences. In the recent modification of the derivation of the merge candidates, the number of the candidates in the list can be greater than the maximum number of candidates in the final list Maxcand. Yet this number of candidates in the initial list, MaxCandlnitialList, is used for the derivation. Consequently, the Zero candidates are added until the number of candidates is MaxCandlnitialList and not until it is Maxcand.
• the BM Merge is Merge mode dedicated to the Adaptive decoder side motion vector refinement method which is an extension of multi-pass DMVR of the ECM. As described in JVET-X2025, this mode is equivalent to 2 Merge modes to refine the MV only in one direction. So, one Merge mode for L0 and one Merge mode for LI. So, the BM Merge is enabled only when the DMVR conditions can be enabled. For these two Merge modes only one list of Merge candidates is derived and all candidates respect the DMVR conditions.
• the Merge candidates for the BM Merge mode are derived from spatial neighbouring coded blocks, TMVPs, non-adjacent blocks, HMVPs, pair-wise candidate, in a similar manner as for the regular Merge mode. A difference is that only those meet DMVR conditions are added into the candidate. Merge index is coded in a similar manner as for regular Merge mode.
• the AVMP Merge mode also known as the bi-directional predictor, is defined as the following in JVET-X2025: It is composed of an AMVP predictor in one direction and a Merge predictor in the other direction.
• the mode can be enabled to a coding block when the selected Merge predictor and the AMVP predictor satisfy DMVR condition, where there is at least one reference picture from the past and one reference picture from the future relatively to the current picture and the distances from two reference pictures to the current picture are the same, the bilateral matching MV refinement is applied for the Merge MV candidate and AMVP MVP as a starting point. Otherwise, if template matching functionality is enabled, template matching MV refinement is applied to the Merge predictor or the AMVP predictor which has a higher template matching cost.
• AMVP part of the mode is signalled as a regular uni-directional AMVP, i.e. reference index and MVD are signalled, and it has a derived MVP index if template matching is used or MVP index is signalled when template matching is disabled.
• the Merge part in the other direction (1 - LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP predictor and a Merge predictor, i.e. for a pair of the AMVP and a Merge motion vectors.
• the bilateral matching cost is calculated using the Merge candidate MV and the AMVP MV.
• the Merge candidate with the smallest cost is selected.
• the bilateral matching refinement is applied to the coding block with the selected Merge candidate MV and the AMVP MV as a starting point.
• the third pass of multi pass DMVR which is 8x8 sub-PU BDOF refinement of the multi-pass DMVR is enabled to AMVP -merge mode coded block.
• the mode is indicated by a flag, if the mode is enabled AMVP direction LX is further indicated by a flag.
• the sign prediction method is described in JVET-X0132.
• the motion vector difference sign prediction can be applied in regular inter modes if the motion vector difference contains non-zero component. In the current ECM version, it is applied for AMVP, Affine MVD and SMVD modes. Possible MVD sign combinations are sorted according to template matching cost and index corresponding to the true MVD sign is derived and coded with context model. At decoder side, the MVD signs are derived as following: 1/Parse the magnitude of MVD components. 2/Parse context-coded MVD sign prediction index. 3/Build MV candidates by creating combination between possible signs and absolute MVD value and add it to the MV predictor.
• the intra prediction, fusion for template-based intra mode derivation is described as the following in JVET-X2025: For each intra prediction mode in MPMs, The SATD between the prediction and reconstruction samples of the template is calculated. First two intra prediction modes with the minimum SATD are selected as the TIMD modes. These two TIMD modes are fused with the weights after applying PDPC process, and such weighted intra prediction is used to code the current CU. Position dependent intra prediction combination (PDPC) is included in the derivation of the TIMD modes.
• PDPC Position dependent intra prediction combination
• a duplicate check for each candidate was added (1440, 1441, 1442, 1443, 1444, 1445, and 1530). But, the duplicate is also for the Non-adjacent candidates (1540) and for the history based candidates (1501). It consists in comparing the motion 1 information of the current candidate of the index ent to the motion information of each other previous candidates. When this motion information is equal, it is considered as duplicate and the variable ent is not incremented.
• the motion information means inter direction, reference frame indexes and motion vectors for each list (LO, LI). Note that zero candidates corresponding to different reference frames are not considered duplicates.
• MvThBDMVRMvdThreshold used for GEO Merge derivation and also for the duplicate check of the non-adjacent candidates as described below.
• AR.MC In the ECM, in order to reduce the number of bits for the Merge index, an Adaptive Reordering of Merge Candidates with Template Matching (ARMC) was added. According to the template matching cost computed as in Figure 13, the candidates are reordered based on the cost of each candidate. In this method only one cost is computed per candidate. This method is applied after that this list has been derived and only on the 5 first candidates of the regular Merge candidates list. It should be appreciated that the number 5 was chosen to balance the complexity of the reordering process with the potential gains, and as such a greater number (e.g. all of the candidates) may be reordered.
• Figure 18 gives an example of this method applied to a regular Merge candidate list containing 10 candidates as in the CTC.
• This method is also applied to the subblock merge mode except for the temporal candidate and for the regular TM mode for all of the 4 candidates.
• this method was also extended to reorder and select a candidate to be included in the final list of Merge mode candidates.
• JVET-X0087 all possible non-adjacent candidates (1540) and History based candidates (1501) are considered with temporal non-adjacent candidates in order to obtain a list of candidates.
• This list of candidates is built without considering the maximum number of candidates.
• This list candidates is then reordered. Only a correct number of candidates from this list are added to the final list of Merge candidates.
• the non-adjacent candidates and History based candidates are processed separately from the adjacent spatial and temporal candidates.
• the processed list is used to supplement the adjacent spatial and temporal Merge candidates already present in the Merge candidate list to generate a final Merge candidate list.
• ARMC is used to select the temporal candidate from among 3 temporal candidates bi-dir, L0 or LI.
• the selected candidate is added to the Merge candidate list.
• the Merge temporal candidate is selected from among several temporal candidates which are reordered using ARMC. In the same way, all possible Adjacent candidates are subject to ARMC and up to 9 of these candidates can be added to the list of Merge candidates.
• JVET X0087 re-uses the cost computed during the reordering of the non-adjacent and History based candidates, to avoid additional computation costs.
• JVET- X0133 applies a systematic reordering on all candidates on the final list of merge candidates.
• ARMC Since the first implementation of ARMC, the method was added to several other modes.
• the ARMC is applied additionally to the Regular, and Template matching Merge mode, Subblock Merge modes and also for IBC, MMVD, Affine MMVD, CUP, CUP with template matching and BM Merge mode.
• the principle of ARMC to reorder the list of candidates based on template matching cost is also applied to the AMVP Merge candidates derivation as well as for the Intra method TIMD to select the most probable predictors.
• the ECM4.0 derivation includes a cascading ARMC process for the candidates derivation of the regular, TM and BM merge modes as depicted for the regular and TM Merge mode in Figure 19.
• non- Adjacent spatial candidates are derived among 59 positions.
• the first list of non-duplicate candidates which can reach 18 candidates is derived.
• the Motion Threshold is different to those used in for temporal derivation and for the rest of the list and doesn’t depends on the regular or template Merge modes. It is set equal to the mv the of BDMVR. Up to 18 non-adjacent candidates are reordered and only the 9 first non- Adjacent candidate are kept and added in the Merge candidates list. Then the other candidates are added except if the list contain already the maximum number of candidates. In addition, for the TM Merge mode the maximum number of candidates in the list MaxCandlnitialList is higher than the maximum number of candidates that the final list can contain Maxcand.
• Figure 17 illustrates the template cost computation of the ARMC method.
• the number of candidates considered in this process, NumMergeCandlnList is greater than or equal to the maximum number that the list can contain Maxcand (1712).
• the cost is set equal to 0 (1703).
• a cost non computed associated to a candidate “i”, mergeList[i].cost was set equal to the maximum value MAXVAL. If the top template for the current block is available (1704), the distortion compared to the current block template is computed (1705) and added to the current cost (1706). Then or otherwise, if the left template for the current block is available (1707), the distortion compared the current block template is computed (1708) and added to the current cost (1709).
• the cost of the current Merge candidate, mergeList[i].cost is set equal to the computed cost (1710) and the list is updated (1711).
• the current candidate i is set to a position regarding its cost compared to the cost of the other candidates.
• NumMergeCandlnList is set equal to the maximum number of possible candidates in the list Maxcand.
• Figure 20 shows an illustration of the update of candidate list (1710) of Figure 17.
• a variable Shift is set equal to 0.
• the Shift is less than to the current candidate i of the Figure 17 and that the related cost of the current candidate is less than the cost of the previous candidate number i-l-shift, (2002) the variable shift is incremented (2003).
• this loop is finished and if the variable shift is different to 0 (2007), a candidate number i is inserted at the position i-shift (2010).
• MHP Multiple Hypothesis Prediction
• the Local Illumination Compensation (LIC) have been added. It is based on a linear model for illumination changes. The linear model is computed thanks to neighboring samples of the current block and the neighboring samples of the previous blocks.
• LIC is enabled only for unidirectional prediction. LIC is signaled by way of a flag. For the Merge modes no LIC flag is transmitted but instead the LIC flag is inherited from the merge candidates in the following manner.
• Non-adjacent merge candidates and history-based Merge candidates the value of the LIC flag is inherited.
• the LIC flag is set equal to 0.
• the value of the LIC flag it is set as depicted in Figure 16. This figure is based on Figure 12 and modules 1620 and 1621 have been added and modules 1609, 1612 and 1611 have been updated. A variable average is set equal to false (1620), if for the current list the average for the pairwise have been computed the LIC flag for the pairwise LICFlagfcnt] is set equal to false and the variable averageUsed equals to true (1609). If only candidate have a motion information for the list (1612 1611) the LIC flag is updated if the average wasn’t used. And it is set equal to a OR operation with its current value and the value of the LICflag of the candidate.
• the algorithm as shown in Figure 16 only allows the LICflag to be equal to something different to true if the 2 candidates have motion information for one list and each candidate has its own list.
• the candidate 0 has motion information for L0 only and Candidate 1 has motion information for LI only.
• the LIC flag can be equal to something else different to 0 but as LIC is only for uni-direction it will never happen. So the LIC flag for the pairwise is always equal to false. Consequently the pairwise candidate can’t use LIC when it is potentially needed. So this reduces the efficiency of the candidate and avoid the propagation of LIC for the following coded blocks and consequently decreases the coding efficiency.
• each candidate is added in the list and the duplicate check (1440, 1441, 1442, 1443, 1444, 1445, and 1530) has an impact only on the increment of the variable ent (1405, 1409, 1413, 1417, 1423, 1427, 1508).
• the variable BCWidx is not initialized for the pairwise candidate. Consequently, if the last candidate added in the list was a duplicate candidate, the value BCWidx for the pairwise candidate is the value of this previous duplicate candidate. This was not the case in VVC as candidates are not added when they are considered as duplicate.
• One perspective to increase the coding efficiency of image or video compression is to use decoder side decisions to reduce the rate related to the signaling.
• Some recent video coding standards contain already such technology as DMVR.
• an encoder uses several parameters which may influence this decision and takes the best one according to these parameters.
• the decision should be also adapted based on several parameters.
• One solution is to define the same rule for all decoders and parameters. But this is not efficient in term of coding efficiency and is not adaptable for all encoder implementations.
• signaling of all the parameters needed to take a decision results in an increased bitrate which also impact the coding efficiency (particularly for short sequences or when these parameters regularly change).
• This invention aims to solve or ameliorate at least some of these problems by an efficient signaling of some parameters needed to take a decision at decoder side.
• an information representing a threshold needed to take a decoder-side decision is transmitted in the bitstream in at least one header.
• the threshold can be transmitted at sequence level inside an SPS or for some pictures inside a PPS or for a picture inside a picture header or for a slice or a tile inside a slice header.
• the bitstream comprises at least one header comprising syntax elements relating to a threshold value to be used when making decoder side decisions.
• the threshold value could be explicitly signaled, or derived, as will be discussed in detail below.
• a header comprises a syntax element which enables a threshold to be derived, and this threshold is used to determine a criterion used in a decoding process (such as a rate-distortion criterion)
• the advantage of this example is an increased flexibility to take decisions at decoder side which increases the coding efficiency thanks to an adaptable threshold.
• a syntax element indicates the transmission of the related threshold inside one header.
• this syntax element is a flag sps Th tooll in sps transmitted in the SPS.
• the syntax element can be pps_Th_tooll_in_pps transmitted in the PPS.
• the syntax element can be ph_Th_tooll_in_ph transmitted in the picture header.
• the syntax element can be sh Th tooll in sh transmitted in the slice header.
• the advantage of this is a more flexibility. Indeed, for some video sequences for some video coding applications, the structure of the GOP structure is relatively simple, as only one reference frame in the past with the same QP for example, which implies that the threshold not need to be to much adapted and can be adapted only for the sequence parameters in SPS of for some pictures in PPS. In opposite for some applications the GOP structures is complex, as QP hierarchical or several temporal ID or several slices. To increase the coding efficiency and consequently the threshold needs more adaptations at picture or slice level for example.
• the transmission of the related threshold inside a header before in the hierarchy of the headers when a syntax element indicates, the transmission of the related threshold inside a header before in the hierarchy of the headers, the corresponding syntax element for the current header is not transmitted.
• the hierarchy of the headers refers to the dependencies of the header and the order that these headers appear in the bitstream. Commonly, the hierarchy is the SPS, PPS, picture header, Slice header.
• pps_Th_tooll_in_pps when sps Th tooll in sps is equal to 1, the pps_Th_tooll_in_pps, ph_Th_tooll_in_ph, sh Th tooll in sh are not transmitted and set equal to 0.
• the flag pps_Th_tooll_in_pps is decoded, and the decoded of each ph_Th_tooll_in_ph or sh Th tooll in sh depends on this pps_Th_tooll_in_pps value.
• a syntax element indicates that the threshold is signaled (for example as described above / below) or set equal to a default value. For example, a flag at slice level sh signal Th tooll, when set equal to 1 indicates that another syntax element is decoded to signal the threshold and when set equal to 0, the threshold is not transmitted and set to the default value.
• This default value may be a value transmitted in a higher level.
• This syntax element can be signaled at other levels so that all lower levels inherit this feature.
• the syntax element indicates that the decision is taken with a criterion which doesn’t depend on any threshold.
• the advantage of these examples is a coding efficiency improvement for some applications. Indeed, sometimes, the adaptively of the threshold is not efficient as the adaptation needs too many bits which impact the bitrate. For example, when a picture contains several slices and the threshold should ideally be adapted for each slice, the signaling of the threshold impacts the global bitrate more than its adaptation.
• the following description relates to the types of decoding decisions which a threshold may influence. It should be noted that several thresholds may be signalled/derived, and/or one threshold could be reused for multiple decoding decisions.
• the threshold is a threshold used in the DMVR method
• the threshold is used in a decoder side method to avoid an additional search.
• the VVC DVMR Decoder Side Motion Vector Refinement
• the algorithm does not apply the refinement when the sum of absolute difference between luma samples of these blocks is inferior to the number sample of the current luma block. So, this criterion is not adapted to the quality.
• the advantage of transmitting the threshold for the DMVR method (which is fixed in VVC), is that it offers a possible complexity versus coding efficiency compromise at encoder side, without impacting the worst-case complexity. So, without impacting the availability of the decoder to decode such bitstream.
• the encoder can adapt the complexity of DMVR to the quality of the block, slice, frame, or sequence.
• the threshold is used for cost comparison in a decoder side reordering method or decoder side selection method of a predictors list.
• the threshold is used in the cost comparison of candidates for the purpose of selection or reordering of candidates in a merge candidates list.
• the threshold can be a Lagrangian parameter (also referred to as ‘lambda’ A). In the same way, this threshold can be used for selection or reordering candidates in an intra predictors list.
• the advantage is a coding efficiency improvement of the method where the threshold depends on several parameters and determined by the encoder.
• thresholds value are transmitted for a same method, these thresholds value corresponds to several modes of set of modes where this same decision method at decoder side is applied.
• the different modes produce different types of block predictors, and the decision method can be adapted in order to produce the best global coding efficiency. So, there is an advantage of setting different threshold values for the same method for a mode or a set of modes.
• Threshold values signalling The following description relates to the manner in which the threshold(s) is/are signaled in the header.
• the maximum number of bits needed to signal the threshold value is transmitted and the value of the threshold is decoded according to the number of bits.
• a syntax element sps_log2_bits_Th_tooll is transmitted in the SPS, and then threshold(s) are decoded according to the value of sps_log2_bits_Th_tooll . For example, for each threshold, N bits are decoded and N is the value of sps_log2_bits_Th_tooll.
• This maximum number of bits may be transmitted at a higher level. For example, when the syntax element sps_log2_bits_Th_tooll, the thresholds decoded, at PPS, picture header or slice are decoded based on this number of bits.
• the advantage of these signaling methods is a coding efficiency improvement. Indeed, the threshold value depends on encoder parameters and for certain configurations or sequences the encoder knows the maximum value of the threshold which should be transmitted. This method has an advantage compared to a method which encodes the threshold based on unary code (described below) when the threshold is not close to zero. Indeed, the thresholds considered have high values and unary code produce higher rate.
• the number of bits is signaled for each threshold transmitted.
• the advantage of this embodiment is coding efficiency improvement when the threshold value is highly variable. In that case some bits can be saved.
• Threshold is transformed or quantized.
• the threshold may be transformed. For example, it is quantized. For example, it is quantized at encoder side and when received at decoder side it is dequantized.
• the advantage is a reduction of the number of bits needed to code the threshold. And it is also efficient whatever if the threshold is coded with a number of bits or with a unary code.
• the quantization is efficient as the threshold values used have large values but a high precision doesn’t offer coding efficiency improvement.
• the threshold is shifted with a right shift at encoder side and then transmitted. Then the decoder applies the corresponding left shift to the decoded value to obtained the threshold to be used for the decoding process.
• the encoder shall apply the left shift also to use the correct threshold value at encoder side.
• it is to applied a square root operation to the threshold value at encoder side and the square operation to the decode the threshold at decoder side. In that case we consider that the obtained threshold after the square root operation is truncated to an integer value.
• This operation offers a higher quantization which is needed for some thresholds, such as when the threshold values are high.
• Another way of reducing the bitrate is to derive the threshold via prediction instead of directly signaling it.
• the threshold is predicted. At encoder side a difference between the threshold value is its predictor value is encoded, this difference can be called a ‘residual’. And at decoder side the decoded residual is added to the predictor value to obtain the threshold.
• the advantage is a bitrate reduction for the encoded threshold.
• the predictor is a threshold which has previously been decoded (or encoded at encoder side).
• the advantage is a bitrate reduction as the previously encoded threshold should be close to the current threshold values.
• the threshold is predicted by a threshold transmitted at a higher level (if available).
• this predictor threshold can be transmitted for the prediction only.
• the threshold value at slice header is predicted by the threshold value transmitted in the picture header. So, when a picture contains several slices, the thresholds at slice level should be not so different and correctly predicted by the predictor value transmitted in the picture header as the values of threshold for difference slices should be close to this picture header value.
• the higher level predictor could be a mean or mode of the predictors used at lower levels, or specifically chosen at encoder side to minimize the total residual transmitted.
• This method offers a bitrate reduction, especially if the threshold is coded thanks to a unary code.
• the threshold is represented by an index in the bitstream.
• the index corresponds to a threshold value that both encoder and decoder both know.
• the set of values corresponding to the index may be transmitted (for example at a high level).
• variables may be transmitted by first transmitting a variable indicating the number thresholds. Then the thresholds are decoded and the indexes corresponds to the decoded order of these decoded thresholds.
• the set of thresholds is transmitted in a higher level.
• the set of thresholds can be transmitted in the SPS and used at picture header or slice header.
• the following syntax table of an SPS illustrates this embodiment. So, the set of thresholds is transmitted in the SPS.
• a flag, sps Th tooll enabled flag indicated if the threshold for the related tool 1 is enabled. If this flag is set equal to 1, the syntax element sps_num_Th_in_set_tooll, representing the number of thresholds in the set, is decoded. Then the number of bits for each threshold, sps_log2_bits_Th_tooll, is decoded. In this example, this syntax element is coded with 4 bits. So, by considering that the 0 bit is not possible, it means that the number of bits for the related threshold is between 1 to 16 bits. So, a threshold, in this example, is between 0 to 2 power of 16, so 65536.
• the following syntax table illustrates a corresponding slice header.
• the threshold is signaled for tooll in the slice header.
• the current method is the method based on threshold index (sps_Th_tooll_based_th_set flag equal 1)
• sh Th tooll idx is decoded.
• the decoder has this value in memory thanks to the table sps Th tooll _predictors[]. So the current decoded threshold for the current slice is set equal to sps Th tooll predi ctors [sh Th tooll idx],
• sh Th tooll idx is coded with a unary code as the aim of this example is to reduce the rate of the thresholds signaling.
• the advantage of this is that the number of bits needed to transmit the threshold value is limited thanks to the usage of the index as the threshold generally depends on the frame hierarchy of the GOP. So many frames share the same threshold. Consequently, it is preferable to transmit the threshold value at high level and indicate an index at low level. Please note that the impact of the method is positive only if the same threshold is used for several frames or slices. If at least two thresholds are used only once the method may increase the rate.
• one index represents the possibility to directly transmit a threshold value.
• sh Th tooll idx when the decoded index, sh Th tooll idx, is set equal to the number of thresholds, sps_num_Th_in_set_tool, decoded in the set in the SPS, the threshold value, sh current Th tooll, is decoded for the current slice. And for example, the number of bits needed to decode the threshold is equal to the sps_log2_bits_Th_tool 1.
• the advantage is a coding efficiency improvement. Indeed, when a threshold is signaled a few numbers of time, it is not needed to transmit them in the SPS, especially if there are several thresholds in that case the index signaling is not efficient and it is preferable to signal it thanks to this available index.
• the encoder knows the GOP structure and when the encoder sets the SPS, and when it is able to determine the threshold to be transmitted, it can estimate based on the maximum number of bits needed for each threshold, the number of thresholds and their occurrences, which is the best compromise signaling between index or direct coding.
• the threshold value when the threshold value is directly transmitted at low level, it is added to the set of thresholds. For example, when the sh current Th tool 1 is decoded in the slice header, this value is added in the set of thresholds at the end, so, sps Th tooll _predictors[sps_num_Th_in_set_tool l ] is set equal to sh current Th tool 1. And the number of thresholds, sps_num_Th_in_set_tooll is incremented.
• the thresholds set doesn’t need to be transmitted in the SPS or any thresholds as all thresholds can be added ‘on the fly’.
• This embodiment is advantageous for the encoder which change the parameters frame by frame to be adapted to a target rate for example.
• the encoder cannot know in advance the number of thresholds, their values etc. . .
• the encoder may order the thresholds according to their occurrences. For example, the encoder knows all possible thresholds needed to be transmitted and the number of times that each of these thresholds need to be signaled when it defines the SPS. This can be for example performed by analyzing the reference pictures list when the threshold depends on the QP or distance to reference frames. In that case the encoder orders the thresholds according to their number of times used. The order is from the maximum occurrence to the minimum one. These thresholds are coded in the SPS thanks to this order. Consequently, at slice level the number of bits to signal the threshold index is minimized as the most frequent threshold has the minimum number of bits.
• the advantage is a rate reduction.
• the threshold values are used to take decoding decisions.
• the decisions are taken based on the Rate distortion compromise. Consequently, the threshold values are statistically correlated to parameters which influence this compromise.
• the following examples improve the prediction of the threshold values according to the values of such other syntax elements.
• the predictor of the threshold is obtained based on the value of another syntax element or based on a value obtained from at least one syntax element.
• the decoder uses at least one syntax element or its representation and obtains the value of the related threshold according to a table which associates both.
• a table associates different QP offset values to threshold values.
• the QP offset can be computed, and the related threshold determined.
• the QP of a slice or a picture is signaled in slice header or picture header according to a QP offset (sh qp delta or ph qp delta). This offset is then added to other values to obtain the QP of the current Slice.
• the advantage is increased coding efficiency as fewer bits are needed to determine the threshold value.
• a flag signals that the threshold is correctly predicted.
• a flag in a header signals that the threshold is correctly predicted.
• this flag is sh Th tooll jorediction flag, and when it is equal to 1 the decoder set the threshold value equals to the threshold value of the corresponding the syntax element or its representation obtained previously during the decoding of the slice header.
• the threshold is represented at slice level with only one bit when the encoder was able to determine the threshold based on the syntax element.
• Decoder verifies if a threshold is associated to the syntax element or on a value obtained from at least one syntax element
• the decoder verifies if a threshold is associated to the syntax element or to a value obtained from at least one syntax element. If it is the case, the threshold value associated is set as the decoded threshold value.
• a flag from a higher level signals that the prediction is correct thanks to a flag or if the prediction imposed.
• a flag sps Th tooll inferred flag indicates if a flag (sh Th tooll _prediction_flag) signals that the threshold is correctly predicted by the associated threshold in the table or if the threshold is inferred based on associated threshold in the table.
• sps Th tooll inferred flag has been transmitted in the SPS for example.
• a flag indicates which of the above two method to use (signalling that the threshold is correctly predicted, or verifying if a threshold is associated to the syntax element or on a value obtained from at least one syntax element).
• the advantage is that the encoder can select the best way to signal the threshold. This decision may be based on the number of slices per frame for example.
• sh Th tooll _prediction_flag when the flag, sh Th tooll _prediction_flag, is set equal to 0 or when there is no associated threshold in the table, the threshold value is directly decoded.
• sh Th tooll _prediction_flag when sh Th tooll _prediction_flag is equal to 0, the syntax element sh current Th tool 1 is decoded and the threshold for the current slice is set equal to sh current Th tool 1.
• sh Th tooll _prediction_flag is set equal to 1 and when it is not possible to infer it sh Th tooll _prediction_flag is set equal to 0.
• the advantage is that the threshold can be set equal to another value than the value presents in the table or when there is no value available.
• the number of bits can be a maximum default possible value.
• Threshold represented by an index
• an index can be transmitted to set another threshold value in the table.
• the value of another syntax element or a value obtained from at least one syntax element is added in the table with the decoded threshold.
• the advantage is for the encoders which are not able to determine the list of syntax element values or values obtained from at least one syntax element previously. Thanks to this embodiment the table can be updated ‘on the fly’.
• the set of thresholds is transmitted in a higher level and corresponds to the possible values identified in the set of possible values related to the value of another syntax element or to a value obtained from at least one syntax element.
• the set of thresholds is transmitted inside the SPS and the thresholds are determined picture per picture thanks information in the picture header or slice by slice thanks to the slice header information.
• the following syntax element table illustrates this embodiment when considering the QP offset. Thanks to the syntax element of the reference pictures list decoded, the decoder identified the number of QP or QP offsets which should be used in the sequence. In the same way the list of the related QP or QP offsets QPOffsetListf] has been also extracted from the reference pictures list for example.
• the thresholds predictors sps Th tooll _predi ctors [i] are decoded form 0 to the number of QP offsets. In one example, the number of elements in the table is transmitted in the bitstream.
• the table of syntax elements is similar as the table syntax elements of embodiments related to thresholds prediction based on index where the number sps_num_Th_in_set_tooll is transmitted.
• the advantage of this embodiment is the flexibility as the encoder doesn’t need to transmit the full table.
• a set of the related values of another syntax element or a set of values obtained from at least one syntax element is transmitted with their related Threshold values in higher level.
• the associated values to the thresholds are explicitly signaled.
• each QP or QP offset (sps_qp_offset_for_tooll[i]) is transmitted with the related threshold (sps Th tool 1 _predictors[i]) in order to create the table which associates both.
• the QP offsets are coded thanks to a signed unary max code.
• a threshold is determined from a header by predicting the threshold based on a syntax element which relates to the same header.
• An association between the syntax element and a threshold value is determined from a table. Each syntax element is transmitted along with its associated threshold so as to generate the table.
• An advantage of this is that the decoder doesn’t need to parse the reference pictures list to identify the different possible QP or QP offsets which are used the following frames or slices. As the impact of signaling at SPS level can be considered as small compared to the complexity of for the parsing to determine the list of QP or QP offsets used.
• the set of related syntax element values or a set of values obtained from at least one syntax element is predictively coded.
• the encoder orders the QP offsets from the minimum one to the maximum one. And this list contains only unique value of QP offsets.
• the following table of syntax elements for an SPS illustrates the decoding process.
• the QP offset difference qp offset is extracted by decoding it with a sign unary code (se(v)). This offset qp offset is added to the previous QP offset sps qp offset for tooll fill to obtain the current one sps qp offset for toollfi].
• the QP offset difference qp offset is extracted by decoding it with a unary code (u(v)). So, without sign as it is sure that the qp offset is positive in that case.
• This offset qp offset is added to the previous QP offset sps_qp_offset_for_tooll[i-l] to obtain the current one sps qp offset for tool 1 [i] .
• the table is generated so that there are no duplicates.
• One method of achieving this is for the encoder to identify the unique value of QP or QP offsets and orders the list form the minimum one to the maximum one. Threshold prediction based on a parameter influencing the quality of the block
• the syntax element or a set of values obtained from at least one syntax element used to determine the predictor for the threshold is a parameter which influences the reconstructed quality.
• the threshold relates to decision which depends on the rate distortion compromise.
• the parameter which significantly influences the quality is the QP or the QP offset.
• this parameter is taken into account to determine the Lagrangian parameter of the rate distortion criterion (also referred to as lambda, ).
• the previously described examples can also be based on this parameter.
• the advantage is that many threshold values for decoder side decisions depend on this parameter value or are highly correlated to this parameter. So, the threshold prediction is consequently better.
• the syntax element to be considered is the QP offset value.
• the syntax element or a set of values obtained from at least one syntax element, used to determine the threshold predictor relates to the reference frames.
• the temporal distance between the current frame and the reference frame can be considered.
• the reference frame, the quality of the reference frame, and distance influence the quality of the reconstructed frame. So, for some threshold values of some decoder decision methods these parameters are correlated.
• a similar table of QP or QP offsets with the related thresholds is transmitted in at least one header and the threshold is adapted at block level thanks to this table of thresholds.
• Figure 22 shows a system 191 195 comprising at least one of an encoder 150 or a decoder 100 and a communication network 199 according to embodiments of the present invention.
• the system 195 is for processing and providing a content (for example, a video and audio content for displaying/outputting or streaming video/audio content) to a user, who has access to the decoder 100, for example through a user interface of a user terminal comprising the decoder 100 or a user terminal that is communicable with the decoder 100.
• a user terminal may be a computer, a mobile phone, a tablet or any other type of a device capable of providing/displaying the (provided/streamed) content to the user.
• the system 195 obtains/receives a bitstream 101 (in the form of a continuous stream or a signal - e.g. while earlier video/audio are being di splay ed/output) via the communication network 199.
• the system 191 is for processing a content and storing the processed content, for example a video and audio content processed for displaying/outputting/streaming at a later time.
• the system 191 obtains/receives a content comprising an original sequence of images 151, which is received and processed (including filtering with a deblocking filter according to the present invention) by the encoder 150, and the encoder 150 generates a bitstream 101 that is to be communicated to the decoder 100 via a communication network 191.
• the bitstream 101 is then communicated to the decoder 100 in a number of ways, for example it may be generated in advance by the encoder 150 and stored as data in a storage apparatus in the communication network 199 (e.g. on a server or a cloud storage) until a user requests the content (i.e. the bitstream data) from the storage apparatus, at which point the data is communicated/streamed to the decoder 100 from the storage apparatus.
• the system 191 may also comprise a content providing apparatus for providing/streaming, to the user (e.g. by communicating data for a user interface to be displayed on a user terminal), content information for the content stored in the storage apparatus (e.g.
• the encoder 150 generates the bitstream 101 and communicates/streams it directly to the decoder 100 as and when the user requests the content.
• the decoder 100 then receives the bitstream 101 (or a signal) and performs filtering with a deblocking filter according to the invention to obtain/generate a video signal 109 and/or audio signal, which is then used by a user terminal to provide the requested content to the user.
• any step of the method/process according to the invention or functions described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the steps/functions may be stored on or transmitted over, as one or more instructions or code or program, or a computer-readable medium, and executed by one or more hardware-based processing unit such as a programmable computing machine, which may be a PC (“Personal Computer”), a DSP (“Digital Signal Processor”), a circuit, a circuitry, a processor and a memory, a general purpose microprocessor or a central processing unit, a microcontroller, an ASIC (“Application-Specific Integrated Circuit”), a field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
• a programmable computing machine which may be a PC (“Personal Computer”), a DSP (“Digital Signal Processor”), a circuit, a circuitry, a processor and a memory, a general purpose microprocessor or a central processing unit,
• Embodiments of the present invention can also be realized by wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of JCs (e.g. a chip set).
• IC integrated circuit
• JCs e.g. a chip set
• Various components, modules, or units are described herein to illustrate functional aspects of devices/apparatuses configured to perform those embodiments, but do not necessarily require realization by different hardware units. Rather, various modules/units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors in conjunction with suitable software/firmware.
• Embodiments of the present invention can be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium to perform the modules/units/functions of one or more of the above-described embodiments and/or that includes one or more processing unit or circuits for performing the functions of one or more of the above-described embodiments, and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiments and/or controlling the one or more processing unit or circuits to perform the functions of one or more of the above-described embodiments.
• computer executable instructions e.g., one or more programs
• the computer may include a network of separate computers or separate processing units to read out and execute the computer executable instructions.
• the computer executable instructions may be provided to the computer, for example, from a computer-readable medium such as a communication medium via a network or a tangible storage medium.
• the communication medium may be a signal/bitstream/carrier wave.
• the tangible storage medium is a “non-transitory computer-readable storage medium” which may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like.
• At least some of the steps/functions may also be implemented in hardware by a machine or a dedicated component, such as an FPGA (“Field- Programmable Gate Array”) or an ASIC (“Application-Specific Integrated Circuit”).
• FIG. 23 is a schematic block diagram of a computing device 3600 for implementation of one or more embodiments of the invention.
• the computing device 3600 may be a device such as a micro-computer, a workstation or a light portable device.
• the computing device 3600 comprises a communication bus connected to: - a central processing unit (CPU) 3601, such as a microprocessor; - a random access memory (RAM) 3602 for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method for encoding or decoding at least part of an image according to embodiments of the invention, the memory capacity thereof can be expanded by an optional RAM connected to an expansion port for example; - a read only memory (ROM) 3603 for storing computer programs for implementing embodiments of the invention; - a network interface (NET) 3604 is typically connected to a communication network over which digital data to be processed are transmitted or received.
• CPU central processing unit
• RAM random access memory
• the network interface (NET) 3604 can be a single network interface, or composed of a set of different network interfaces (for instance wired and wireless interfaces, or different kinds of wired or wireless interfaces). Data packets are written to the network interface for transmission or are read from the network interface for reception under the control of the software application running in the CPU 3601; - a user interface (UI) 3605 may be used for receiving inputs from a user or to display information to a user; - a hard disk (HD) 3606 may be provided as a mass storage device; - an Input/Output module (IO) 3607 may be used for receiving/sending data from/to external devices such as a video source or display.
• UI user interface
• HD hard disk
• IO Input/Output module
• the executable code may be stored either in the ROM 3603, on the HD 3606 or on a removable digital medium such as, for example a disk.
• the executable code of the programs can be received by means of a communication network, via the NET 3604, in order to be stored in one of the storage means of the communication device 3600, such as the HD 3606, before being executed.
• the CPU 3601 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to embodiments of the invention, which instructions are stored in one of the aforementioned storage means. After powering on, the CPU 3601 is capable of executing instructions from main RAM memory 3602 relating to a software application after those instructions have been loaded from the program ROM 3603 or the HD 3606, for example.
• a software application when executed by the CPU 3601, causes the steps of the method according to the invention to be performed.
• a decoder according to an aforementioned embodiment is provided in a user terminal such as a computer, a mobile phone (a cellular phone), a table or any other type of a device (e.g. a display apparatus) capable of providing/displaying a content to a user.
• an encoder according to an aforementioned embodiment is provided in an image capturing apparatus which also comprises a camera, a video camera or a network camera (e.g. a closed-circuit television or video surveillance camera) which captures and provides the content for the encoder to encode. Two such examples are provided below with reference to Figures 37 and 38.
• Figure 24 is a diagram illustrating a network camera system 3700 including a network camera 3702 and a client apparatus 202.
• the network camera 3702 includes an imaging unit 3706, an encoding unit 3708, a communication unit 3710, and a control unit 3712.
• the network camera 3702 and the client apparatus 202 are mutually connected to be able to communicate with each other via the network 200.
• the imaging unit 3706 includes a lens and an image sensor (e.g., a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS)), and captures an image of an object and generates image data based on the image.
• This image can be a still image or a video image.
• the encoding unit 3708 encodes the image data by using said encoding methods explained above, or a combination of encoding methods described above.
• the communication unit 3710 of the network camera 3702 transmits the encoded image data encoded by the encoding unit 3708 to the client apparatus 202.
• the communication unit 3710 receives commands from client apparatus 202.
• the commands include commands to set parameters for the encoding of the encoding unit 3708.
• the control unit 3712 controls other units in the network camera 3702 in accordance with the commands received by the communication unit 3712.
• the client apparatus 202 includes a communication unit 3714, a decoding unit 3716, and a control unit 3718.
• the communication unit 3714 of the client apparatus 202 transmits the commands to the network camera 3702.
• the communication unit 3714 of the client apparatus 202 receives the encoded image data from the network camera 3712.
• the decoding unit 3716 decodes the encoded image data by using said decoding methods explained above, or a combination of the decoding methods explained above.
• the control unit 3718 of the client apparatus 202 controls other units in the client apparatus 202 in accordance with the user operation or commands received by the communication unit 3714.
• the control unit 3718 of the client apparatus 202 controls a display apparatus 2120 so as to display an image decoded by the decoding unit 3716.
• the control unit 3718 of the client apparatus 202 also controls a display apparatus 2120 so as to display GUI (Graphical User Interface) to designate values of the parameters for the network camera 3702 includes the parameters for the encoding of the encoding unit 3708.
• GUI Graphic User Interface
• the control unit 3718 of the client apparatus 202 also controls other units in the client apparatus 202 in accordance with user operation input to the GUI displayed by the display apparatus 2120.
• the control unit 3718 of the client apparatus 202 controls the communication unit 3714 of the client apparatus 202 so as to transmit the commands to the network camera 3702 which designate values of the parameters for the network camera 3702, in accordance with the user operation input to the GUI displayed by the display apparatus 2120.
• Figure 25 is a diagram illustrating a smart phone 3800.
• the smart phone 3800 includes a communication unit 3802, a decoding unit 3804, a control unit 3806 and a display unit 3808.
• the communication unit 3802 receives the encoded image data via network 200.
• the decoding unit 3804 decodes the encoded image data received by the communication unit 3802.
• the decoding / encoding unit 3804 decodes / encodes the encoded image data by using said decoding methods explained above.
• the control unit 3806 controls other units in the smart phone 3800 in accordance with a user operation or commands received by the communication unit 3806.
• control unit 3806 controls a display unit 3808 so as to display an image decoded by the decoding unit 3804.
• the smart phone 3800 may also comprise sensors 3812 and an image recording device 3810. In such a way, the smart phone 3800 may record images, encode the images (using a method described above).
• the smart phone 3800 may subsequently decode the encoded images (using a method described above) and display them via the display unit 3808 - or transmit the encoded images to another device via the communication unit 3802 and network 200.
• any result of comparison, determination, assessment, selection, execution, performing, or consideration described above may be indicated in or determinable/inferable from data in a bitstream, for example a flag or data indicative of the result, so that the indicated or determined/inferred result can be used in the processing instead of actually performing the comparison, determination, assessment, selection, execution, performing, or consideration, for example during a decoding process.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=81753212

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (1)

Citations (2)

Family Cites Families (16)

• 2022
  • 2022-04-19 GB GB2205721.0A patent/GB2617839B/en active Active
• 2023
  • 2023-04-14 WO PCT/EP2023/059833 patent/WO2023202956A1/en not_active Ceased
  • 2023-04-14 US US18/857,141 patent/US12676984B2/en active Active
  • 2023-04-14 EP EP23719753.8A patent/EP4512079A1/en active Pending
  • 2023-04-14 JP JP2024554713A patent/JP2025513175A/ja active Pending
  • 2023-04-14 CN CN202380034507.6A patent/CN119032563A/zh active Pending

Patent Citations (2)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events