BR112017020632B1 - METHODS AND DEVICE FOR PROCESSING VIDEO DATA AND COMPUTER READABLE MEMORY - Google Patents

Abstract

DERIVATION MOVEMENT INFORMATION FOR SUB-BLOCKS IN VIDEO TO CODE CONVERSION. It is a method for processing video data that includes dividing a current block of video data into a plurality of sub-blocks to derive motion information of the current block, wherein the motion information indicates motion of the current block with relative to the reference video data. The method also includes deriving, separately for each respective sub-block of the plurality of sub-blocks, motion information comprising performing a motion search for a first set of reference data corresponding to a second set of reference data outside of each respective sub-block. The method also includes decoding the plurality of sub-blocks based on the derived motion information and without decoding syntax elements representative of the motion information.

Description

[001] This application claims the benefit of Provisional Application under U.S. 62/139,572 filed on March 27, 2015 and Provisional Application under U.S. 62/182,367 filed on June 19, 2015, the entire contents of which are incorporated herein , for reference.

FIELD OF TECHNIQUE

[002] This disclosure relates to video coding.

BACKGROUND

[003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, electronic gaming devices, electronic game consoles, satellite or cellular radio phones, so-called "smart phones", video teleconferencing, video streaming devices and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Video Coding Advanced Video Coding (AVC), the High Efficiency Video Coding (HEVC) standard, and extensions of such standards. Video devices can transmit, receive, encode, decode and/or store digital video information more efficiently by implementing such video coding techniques.

[004] Video coding techniques include spatial prediction (intra-image) and/or temporal prediction (inter-image) to reduce or remove inherent redundancy in video sequences. For block-based video coding, a video slice (e.g., a video frame or a portion of a video frame) can be partitioned into video blocks, which for some techniques can also be referred to as blocks. tree, coding units (CUs) and/or coding nodes. Video blocks in an intracoded (I) slice of an image are encoded using spatial prediction with respect to reference samples in nearby blocks in the same image. Video blocks in an intercoded slice (P or B) of an image can use spatial prediction with respect to reference samples in nearby blocks in the same image or temporal prediction with respect to reference samples in other reference images. Images can be termed as frames and reference images can be termed as reference frames.

[005] Spatial or temporal prediction results in a predictive block for a block to be encoded. Residual data represents pixel differences between the original block to be encoded and the predictive block. An intercoded block is coded according to a motion vector that points to a block of reference samples that forms the predictive block, and wherein the residual data indicates the difference between the encoded block and the predictive block. An intracode block is coded according to an intracoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to a transform domain, resulting in residual transform coefficients that can then be quantized. The quantized transform coefficients initially arranged in a two-dimensional matrix can be scanned to produce a one-dimensional transform coefficient vector, and entropy coding can be applied to achieve further compression.

SUMMARY

[006] The techniques of this disclosure relate to deriving motion information in video coding. For example, a video encoder (a video encoder or video decoder) may generate motion information that is not included in the bitstream to encode video data that is included in the bitstream. By deriving motion information in the encoding or decoding cycle, bit savings can be achieved over techniques that include motion information in the bit stream such as traditional interprediction techniques.

[007] In one example, a method for decoding video data includes selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each Mode of deriving motion information from the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, determine the motion information for the current block using the selected motion information derivation mode, and decode the current block using the determined motion information and without decoding the syntax elements representative of movement information.

[008] In another example, a method for encoding video data includes selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, wherein each Mode of deriving motion information from the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, and determine the motion information for the current block using the selected motion information derivation mode, and encode the current block using the determined motion information and without encoding the video elements. syntax representing movement information.

[009] In another example, a device for encoding video data includes a memory configured to store a current block of video data, and one or more processors configured to select a mode of deriving motion information from a plurality of motion information deriving modes for determining motion information for the current block, wherein each motion information deriving mode of the plurality comprises performing a motion search for a first set of reference data corresponding to a second set of reference data outside the current tile, and wherein the motion information indicates motion of the current tile relative to the reference video data, determine the motion information for the current tile using the selected motion information derivation mode and encoding the current block using the determined motion information and without coding syntax elements representative of the motion information.

[0010] In another example, an apparatus for encoding video data includes means for selecting a motion information derivation mode from a plurality of motion information derivation modes to determine motion information for a current block, in that each mode of deriving motion information of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block, and wherein the motion information indicates motion of the current block with respect to the reference video data, means for determining the motion information for the current block using the selected motion information derivation mode, and means for encoding the current block using the determined motion information and without decoding the syntax elements representing the motion information.

[0011] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause one or more processors to select a motion information derivation mode from a plurality of motion information derivation modes. motion information to determine motion information for a current block, wherein each mode of deriving motion information of the plurality comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside of the current block, and wherein the motion information indicates motion of the current block relative to the reference video data, determine the motion information for the current block using the selected motion information derivation mode, and encode the current block using the determined motion information and without decoding syntax elements representative of the motion information.

[0012] In another example, a method for processing video data includes determining a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates the motion of the current block relative to the reference video data and determining a derived motion vector for the current block based on the determined candidate motion vector, wherein determining the derived motion vector comprises performing a motion search for a first set of reference data corresponding to a second set of reference data outside the current block.

[0013] In another example, a device for processing video data includes a memory configured to store a current block of video data, and one or more processors configured to determine a candidate motion vector for deriving motion information from a current block. of video data, wherein the motion information indicates motion of the current block relative to the reference video data, and determining a derived motion vector for the current block based on the determined candidate motion vector, wherein determining the vector Derived motion search comprises performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block.

[0014] In another example, an apparatus for processing video data includes means for determining a candidate motion vector for deriving motion information from a current block of video data, wherein the motion information indicates motion of the current block relative to to the reference video data, and means for determining a derived motion vector for the current block based on the determined candidate motion vector, wherein the means for determining the derived motion vector comprises means for performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block.

[0015] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause one or more processors to determine a candidate motion vector for deriving motion information from a current block of video data. , wherein the motion information indicates motion of the current tile relative to the reference video data, and determining a derived motion vector for the current tile based on the determined candidate motion vector, wherein, to determine the motion vector Derived instructions cause the one or more processors to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block.

[0016] In another example, a method for processing video data includes dividing a current block of video data into a plurality of sub-blocks to derive motion information from the current block, wherein the motion information indicates motion from the current block. with respect to the reference video data, derive, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data corresponding to a second set of reference data outside each respective sub-block and decoding the plurality of sub-blocks based on the derived motion information and without decoding the syntax elements representative of the motion information.

[0017] In another example, a method for processing video data includes dividing a current block of video data into a plurality of sub-blocks to derive motion information from the current block, wherein the motion information indicates motion from the current block. with respect to the reference video data, derive, separately for each respective sub-block among the plurality of sub-blocks, wherein the motion information comprises performing a motion search for a first set of reference data corresponding to a second set of reference data outside each respective sub-block and encoding the plurality of sub-blocks based on the derived motion information and without encoding the syntax elements representative of the motion information.

[0018] In another example, a device for processing video data includes a memory configured to store a current block of video data, and one or more processors configured to divide a current block of video data into a plurality of sub-blocks. to derive motion information of the current block, wherein the motion information indicates motion of the current block relative to the reference video data, derive, separately for each respective sub-block among the plurality of sub-blocks, wherein the information of motion comprise performing a motion search for a first set of reference data corresponding to a second set of reference data outside each respective sub-block and encoding the plurality of sub-blocks based on the derived motion information and without encode the syntax elements representing movement information.

[0019] In another example, a non-transitory computer-readable medium has instructions stored therein that, when executed, cause the one or more processors to divide a current block of video data into a plurality of sub-blocks to derive information of motion of the current block, wherein the motion information indicates motion of the current block relative to the reference video data, derive, separately for each respective sub-block from among the plurality of sub-blocks, wherein the motion information comprises perform a motion search for a first set of reference data corresponding to a second set of reference data outside each respective sub-block and decode the plurality of sub-blocks based on the derived motion information and without decoding the elements of syntax representing movement information.

[0020] In another example, a device for processing video data includes a memory configured to store a current image, and one or more processors configured to obtain an encoded bit stream containing a plurality of encoded images, interpolate one or more images reference images that are not included in the encoded bitstream and decode video data from a current image of the encoded bitstream based on the one or more interpolated reference images.

[0021] Details of one or more examples of the disclosure are presented in the attached drawings and in the description below. Other features, objectives and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram illustrating an exemplary video decoding and encoding system that can implement techniques of this disclosure.

[0023] Figure 2 is a block diagram illustrating an example of a video encoder that can implement techniques of this disclosure.

[0024] Figure 3 is a block diagram illustrating an example of a video decoder that can implement techniques of this disclosure.

[0025] Figures 4A and 4B are conceptual diagrams illustrating exemplary spatial neighboring motion vector candidates for a blend mode and an advanced motion vector prediction (AMVP) mode.

[0026] Figures 5A and 5B are conceptual diagrams illustrating an exemplary temporal motion vector predictor (TMVP) candidate and motion vector scaling.

[0027] Figure 6 is a conceptual diagram illustrating an example of one-way motion estimation (ME) in frame rate up-conversion (FRUC).

[0028] Figure 7 is a conceptual diagram illustrating an example of bilateral motion estimation (ME) in FRUC.

[0029] Figure 8 is a conceptual diagram illustrating an example of decoder side motion vector derivation (DMVD) based on template matching.

[0030] Figure 9 is a conceptual diagram illustrating an example of mirror-based bidirectional motion vector derivation in DMVD.

[0031] Figure 10 is a conceptual diagram illustrating motion vector derivation based on extended bilateral correspondence.

[0032] Figure 11 is a flowchart that illustrates an example of decoding a prediction unit (PU) using DMVD.

[0033] Figure 12 is a flowchart illustrating an exemplary process for determining a mode of deriving motion information for encoding a block of video data.

[0034] Figure 13 is a flowchart illustrating an exemplary process for deriving a motion vector for encoding a block of video data.

[0035] Figure 14 is a flowchart illustrating an exemplary process for deriving motion information for sub-blocks of a video data block.

DETAILED DESCRIPTION

[0036] The techniques of this disclosure relate to deriving decoder side motion information, block partitioning, and/or data interpolation in block-based video coding. The techniques can be applied to any of the existing video codecs such as High Efficiency Video Coding (HEVC) or be an effective encoding tool for any future video coding standards.

[0037] Video encoding devices deploy video compression techniques to encode and decode video data effectively. Video compression techniques may include applying spatial prediction (e.g., intra-frame prediction), temporal prediction (e.g., inter-frame prediction), and/or other prediction techniques to reduce or remove inherent redundancy in video sequences. A video encoder typically partitions each image of an original video sequence into rectangular regions referred to as video blocks or coding units (described in greater detail below). These video blocks can be encoded using a particular prediction mode.

[0038] For interprediction modes, a video encoder typically searches for a block similar to that being encoded in a frame at another temporal location, referred to as a reference frame. The video encoder can restrict the search to a certain spatial displacement of the block to be encoded. A better match can be found using a two-dimensional (2D) motion vector that includes a horizontal displacement component and a vertical displacement component. For an intraprediction mode, a video encoder may form the predicted block using spatial prediction techniques based on data from previously encoded neighboring blocks within the same image.

[0039] The video encoder may determine a prediction error, that is, the difference between the pixel values in the block being encoded and the predicted block (also referred to as residual). The video encoder can also apply a transform to the prediction error, such as a distinct cosine transform (DCT), to generate transform coefficients. After the transform, the video encoder can quantize the transform coefficients. Quantized transform coefficients and motion vectors can be represented using syntax elements and, together with control information, form a coded representation of a video sequence. In some cases, the video encoder may entropy encode syntax elements, thereby further reducing the number of bits required for their representation.

[0040] A video decoder can, using the syntax elements and control information discussed above, construct predictive data (e.g., a predictive block) to decode a current frame. For example, the video decoder can add the predicted block and the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis functions using the quantized coefficients. The difference between the reconstructed frame and the original frame is called reconstruction error.

[0041] In some cases, a video decoder or post-processing device may interpolate images based on one or more reference images. These interpolated images are not included in an encoded bitstream. The video decoder or post-processing device may interpolate images to up-convert an original frame rate of an encoded bit stream. This process can be called frame rate upconversion (FRUC). Alternatively, the video decoder or post-processing device may interpolate images to insert one or more images that have been skipped by a video encoder to encode a video sequence at a reduced frame rate. In either case, the video decoder or post-processing device interpolates frames that are not included in an encoded bit stream that was received by the video decoder. The video decoder or post-processing device may interpolate the images using any of several interpolation techniques, for example, using motion-compensated frame interpolation, frame repetition, or frame averaging. .

[0042] Although certain techniques for interpolating images have been used for upconversion purposes, these techniques have not been used widely during video coding, for example, to encode video data that is included in an encoded bitstream. For example, techniques for interpolating images may be relatively time-consuming and/or require a relatively large amount of processing power. Consequently, these techniques have typically not been performed in a cycle when decoding video data.

[0043] According to aspects of this disclosure, a video encoder (e.g., a video encoder or a video decoder) can derive motion information when encoding a bit stream. For example, the video encoder may generate motion information that is not included in the bitstream to encode video data that is included in the bitstream. By deriving motion information in the encoding or decoding cycle, bit savings can be achieved relative to techniques that include motion information in the bit stream (such as the aforementioned interprediction techniques).

[0044] According to some aspects of this disclosure, a video encoder may utilize a plurality of techniques for deriving motion information during encoding. In these examples, the video encoder may determine a motion information derivation mode to determine which motion information derivation techniques to use when determining motion information for a current block. In general, using a motion information deriving mode to derive motion information may include performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block. For example, using the motion information derivation mode (e.g., a bilateral matching technique, a template matching technique, or another technique as described in greater detail below), the video encoder may select a vector of candidate movement in a movement vector candidate list. The video encoder may select the candidate motion vector based on the candidate motion vector that identifies reference data in a reference image that relatively closely matches current image data (which may be referred to as determining a “best match”). reference data).

[0045] In some cases, the video encoder may use the selected motion vector candidate to identify a search window in a reference image. The video encoder can refine the candidate motion vector based on reference data in the search window that relatively closely matches corresponding data in the current image. That is, the video encoder can derive new motion information for the current block based on motion between reference data in the search window that roughly corresponds to data in the current image. The video encoder can then perform motion compensation for the current block using the derived motion information. In this way, the video encoder can derive motion information for a current block without motion information being signaled in an encoded bitstream.

[0046] In accordance with aspects of this disclosure, in some examples, the video encoder may divide a block of video data into sub-blocks when deriving motion information. For example, the video encoder may separately derive motion information for each sub-block of a larger block. In some cases, the video encoder may initially determine motion information for the block and use the derived motion information as candidate motion information for each of the sub-blocks. The video encoder can then further refine the derived motion information for each of the sub-blocks, for example, using a motion information derivation mode (e.g., a bilateral matching technique, a template matching, or other technique, as described in more detail below).

[0047] Techniques of this disclosure may also include techniques for interpolating images. In some cases, any combination of the above techniques can be used to interpolate an image that is not included in the bitstream, for example, similar to frame rate upconversion. However, rather than simply adding the interpolated image to the video sequence, a video decoder can use the interpolated frame during encoding. For example, the video decoder may decode data from a current image based on at least a portion of the interpolated image. In some cases, the video decoder may make the interpolated image the same as the current image. For example, the video decoder may decode syntax data for the current image that is included in the bitstream (e.g., slice header data and the like), interpolate the image, and establish the interpolated image as the current image. In other cases, the video decoder may interpolate the image and decode data for the current image relative to the interpolated image. In this case, the video decoder may add the interpolated image to a reference image memory for prediction purposes.

[0048] Therefore, certain techniques described herein referring to FRUC may, in some examples, be used to determine motion information (e.g., in a process of deriving decoder-side motion information). In other examples, the techniques described herein referring to FRUC can be used to interpolate video data, for example, for reference to encode video data, or to output.

[0049] Figure 1 is a block diagram illustrating an exemplary video decoding and encoding system 10 that may utilize techniques to derive motion information, perform block partitioning, and/or interpolate video data. As shown in Figure 1, system 10 includes a source device 12 that provides encoded video data to be decoded at a later time by a destination device 14. In particular, the source device 12 provides the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 may comprise any of a wide range of devices, including desktop-type computers, notebook-type computers (i.e., laptop-type computers), tablet-type computers, set-top boxes, telephone devices such as so-called "smart" phones, so-called "smart" pad-type devices, televisions, cameras, display devices, players digital media, video game consoles, video streaming device, or the like. In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication.

[0050] The destination device 14 may receive the encoded video data to be decoded via the computer-readable media 16. The computer-readable media 16 may comprise any type of media or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may comprise a communication medium to enable a source device to transmit encoded video data directly to a destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication media may comprise any wireless or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication media can be part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication media may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 12 to the destination device 14.

[0051] In some examples, the encoded data may be output from the output interface 22 to a storage device. Similarly, the encoded data can be accessed from the storage device via the input interface. The storage device may include any of a variety of locally accessed or distributed data storage media such as a hard drive, Blu-ray Discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other storage media. digital storage suitable for storing encoded video data. In a further example, the storage device may correspond to a file server or other intermediate storage device that may store encoded video generated by the source device 12. The destination device 14 may access video data stored from the source device. storage via streaming or download transfer. The file server can be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14. Exemplary file servers include a web server (e.g., for a website). web), an FTP server, network attached storage (NAS) devices, or a local disk drive. The target device 14 can access the encoded video data over any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for accessing stored encoded video data on a file server. The transmission of encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0052] The techniques of this disclosure are not necessarily limited to wireless applications or definitions. The techniques can be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television broadcasts, satellite television broadcasts, Internet video streaming, such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

[0053] In the example of Figure 1, the source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoder 30 and a display device 32. In accordance with this disclosure, the video encoder 20 of the source device 12 may be configured to apply techniques for deriving motion information, performing block partitioning, and/or interpolating video data. In other examples, a source device and a destination device may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18, such as an external camera. Likewise, the target device 14 may interface with an external display device, rather than including an integrated display device.

[0054] The illustrated system 10 of Figure 1 is merely an example. Techniques for deriving motion information, performing block partitioning, and/or interpolating video data can be performed by any digital video encoding and/or decoding device. Although generally the techniques of this disclosure are performed by a video encoder device, the techniques may also be performed by a video encoder/decoder typically referred to as a "CODEC". Furthermore, the techniques of this revelation can also be performed by a video preprocessor. The source device 12 and the destination device 14 are merely examples of such encoding devices where the source device 12 generates encoded video data for transmission to the destination device 14. In some examples, the devices 12, 14 may operate substantially symmetrically such that each of the devices 12, 14 includes video encoding and decoding components. Therefore, system 10 may support unidirectional or bidirectional video transmission between video devices 12, 14, for example, for video streaming, video playback, video broadcasting or video telephony.

[0055] The video source 18 of the source device 12 may include a video capture device such as a video camera, a video file containing previously captured video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, the video source 18 may generate computer graphics-based data as the source video or a combination of live video, archived video, and computer-generated video. In some cases, if the video source 18 is a video camera, the source device 12 and the destination device 14 may form so-called camera phones or videophones. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding in general and may be applied to wired and/or wireless applications. In each case, the captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output by output interface 22 onto computer-readable media 16.

[0056] Computer-readable media may include transient media, such as a wired network transmission or wireless broadcast, or storage media (i.e., non-transient storage media), such as a hard disk, flash drive, compact disk, digital video disc, Blu-ray disc, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from the source device 12 and provide the encoded video data to the destination device 14, for example, via network transmission. Similarly, a computing device of a media production facility, such as a disk labeling facility, may receive encoded video data from the source device 12 and produce a disk containing the encoded video data. Therefore, computer-readable media may be understood to include one or more computer-readable media of various forms, in various examples.

[0057] The network interface 28 of the target device 14 receives information from the computer-readable media 16. The computer-readable media information 16 may include syntax information defined by the video encoder 20, which is also used by the decoder video 30, which includes syntax elements that describe characteristics and/or processing of blocks and other coded units, e.g., GOPs. The display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display , an organic light-emitting diode (OLED) display, or other type of display device.

[0058] Although not shown in Figure 1, in some aspects, the video encoder 20 and the video decoder 30 may each be integrated with an audio encoder and decoder and may include appropriate MUX-DEMUX units, or other hardware and software, to handle the encoding of both audio and video into a common data stream or separate data streams. If applicable, MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP).

[0059] Video encoder 20 and video decoder 30 may be implemented as any of a variety of suitable encoder circuit sets, such as one or more microprocessors, digital signal processors (DSPs), specific integrated circuits for applications (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combinations thereof. When the techniques have been partially implemented in software, a device can store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to carry out the techniques of this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, one of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device.

[0060] This disclosure may generally refer to video encoder 20 "signaling" certain information to another device, such as video decoder 30. The term "signaling" may generally refer to communication of syntax elements and/or other data used to decode the compressed video data. Such communication can occur in real or near real time. Alternatively, this communication may occur over a period of time, as may occur when storing syntax elements on a computer-readable storage medium in a stream of bits encoded at the time of encoding, which can then be retrieved by a device. of decoding at any time after being stored on that media.

[0061] The video encoder 20 and video decoder 30 may operate in accordance with a video coding standard. Exemplary video coding standards developed by the Joint Video Coding Collaboration Team (JCT-VC) as well as the Joint 3D Video Extension Development Collaboration Team (JCT-3V) of the ITU Video Coding Expert Group -T (VCEG) and ISO/IEC Moving Image Specialist Group (MPEG) include High Efficiency Video Coding (HEVC) or ITU-T H.265, including its range extension, multiview extension (MV- HEVC) and scalable extension (SHVC). The finalized HEVC standard document is published as “ITU-T H.265, SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS Infrastructure of audiovisual services - Coding of moving video - High efficiency video coding”, Telecommunication Standardization Sector of International Telecommunication Union (ITU ), April 2013. Alternatively, video encoder 20 and video decoder 30 may operate in accordance with other industry or proprietary standards, such as ISO/IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO /IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) extension and Multiview Video Coding (MVC) extension. However, the techniques of this disclosure are not limited to any particular coding standard. For example, the techniques of this disclosure can be used with a variety of other non-proprietary and proprietary video coding techniques or subsequent standards, such as ITU-T H.266.

[0062] As noted above, for interprediction modes, the video encoder 20 may search for a block similar to the one being encoded (a “current block”) in an image from another temporal location, referred to as a reference image. The information used to identify the reference image can be referred to as motion information. For example, for each block, a set of motion information may be available. A motion information set contains motion information for forward and backward prediction directions. In the present context, the forward and backward prediction directions are two prediction directions of a bidirectional prediction mode and the terms “forward” and “backward” do not necessarily have a geometry meaning, rather they correspond to to the reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1) of a current image. When only one reference image list is available for an image or slice, only RefPicList0 is available and the motion information for each block of a slice is always forward.

[0063] In some cases, a motion vector together with its reference index is used in decoding processes, this motion vector with the associated reference index is denoted as a uni-predictive set of motion information.

[0064] For each prediction direction, the motion information must contain a reference index and a motion vector. In some cases, for simplicity, a motion vector itself may be referred to in a way that is assumed to have an associated reference index. A reference index is used to identify a reference image in the current reference image list (RefPicList0 or RefPicList1). A motion vector has a horizontal and a vertical component.

[0065] Picture order counting (POC) is widely used in video coding standards to identify a display order of an image. Although there are cases where two images within an encoded video sequence may have the same POC value, the same typically does not occur within an encoded video sequence. When multiple encoded video sequences are present in a bitstream, images with the same POC value may be close to each other in terms of decoding order. The POC values of images are typically used for reference image list construction, reference image set derivation as in HEVC, and motion vector scaling.

[0066] In H.264/AVC, each intermacroblock (MB) can be partitioned into four different ways that include: a 16 x 16 MB partition; two 16 x 8 MB partitions; two 8 x 16 MB partitions; and four 8 x 8 MB partitions. Different partitions in a MB can have different reference index values for each direction (RefPicList0 or RefPicList1). When a MB is not partitioned into four 8 x 8 MB partitions, the has only one motion vector for each MB partition in each direction.

[0067] When a MB is partitioned into four 8 x 8 MB partitions, each 8 x 8 MB partition may be further partitioned into sub-blocks, each of which may have a different motion vector in each direction. There are four different ways to obtain subblocks from an 8 x 8 MB partition which include: an 8 x 8 subblock; two 8 x 4 sub-blocks; two 4 x 8 sub-blocks; and four 4 x 4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector is present at a level equal to greater than sub-block.

[0068] In AVC, temporal direct mode could be enabled at MB or MB partition level for skip or direct mode at B slices. For each MB partition, the motion vectors of the block co-located with the MB partition current block in RefPicList1 [0] are used to derive the motion vectors. Each motion vector in the co-located block is scaled based on POC distances. In AVC, a direct mode can also predict motion information from spatial neighbors, which can be referred to as spatial direct mode.

[0069] In HEVC, to generate a coded representation of an image, the video encoder 20 may generate a set of coding tree units (CTUs). Each of the CTUs may comprise a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples, and syntax structures used to encode the samples of the CTBs. In monochrome images or images that have three separate color planes, a CTU may comprise a single CTB block and syntax structures used to encode the coding tree block samples.

[0070] A coding tree block can be an NxN block of samples. The size of a CTB can be in the ranges of 16x16 to 64x64 in the HEVC main profile (although technically 8x8 CBT sizes can be supported). A coding unit (CU) could be the same size as a CTB and as small as 8 x 8. Each coding unit is coded with a mode. A CTU may also be referred to as a "tree block" or a "largest coding unit" (LCU). HEVC CTUs can be broadly analogous to the macroblocks of other standards, such as H.264/AVC. However, the CTU is not necessarily limited to a particular size and may include one or more coding units (CUs). A slice can include an integer number of CTUs ordered consecutively in a raster scan order.

[0071] In order to generate an encoded CTU, the video encoder 20 may recursively perform quadratic tree partitioning on the encoding tree blocks of a CTU to divide the encoding tree blocks into encoding blocks, thereby , the name "coding tree units". A coding tree block can be an NxN block of samples. A CU may comprise a luma sample coding block and two corresponding chroma sample coding blocks of an image having a luma sample matrix, a Cb sample matrix and a Cr sample matrix and syntax used to encode the coding block samples. In monochrome images or images that have three separate color planes, a CU may comprise a single coding block and syntax structures used to encode the samples of the coding block.

[0072] Video encoder 20 may partition a coding block of a CU into one or more predictive blocks. A prediction block is a rectangular (i.e., square or non-square) block of samples to which the same prediction is applied. A prediction unit (PU) of a CU may comprise a luma sample prediction block, two corresponding chroma sample prediction blocks, and syntax structures used to predict the prediction blocks. In monochrome images or in images that have three separate color planes, a PU may comprise a single predictive block and syntax structures used to predict the predictive block. Video encoder 20 may generate luma, Cb, and Cr predictive blocks for luma, Cb, and Cr predictive blocks of each PU of the CU.

[0073] Video encoder 20 may use intraprediction or interprediction to generate the predictive blocks for a PU. If the video encoder 20 uses intraprediction to generate the predictive blocks of a PU, the video encoder 20 may generate predictive blocks of the PU based on the decoded samples of the image associated with the PU. If the video encoder 20 uses interprediction to generate the predictive blocks of a PU, the video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more images other than the image associated with the PU. When a CU is intercoded, the CU can be further partitioned into two or four PUs. When two PUs are present in a CU, the PUs can, in some cases, be the size of half a rectangle or the size of two rectangles with a size of one-quarter or three-quarters of the CU.

[0074] After video encoder 20 generates predictive luma, Cb, and Cr blocks for one or more PUs of a CU, video encoder 20 may generate a residual luma block for the CU. Each sample in the CU luma residual block indicates a difference between a luma sample in one of the CU's luma predictive blocks and a corresponding sample in the CU's original luma encoding block. Additionally, video encoder 20 may generate a residual block of Cb for the CU. Each sample in the CU Cb residual block may indicate a difference between a Cb sample in one of the CU's Cb predictive blocks and a corresponding sample in the CU's original Cb coding block. The video encoder 20 may also generate a residual block of Cr for the CU. Each sample in the CU residual Cr block may indicate a difference between a Cr sample in one of the CU's predictive Cr blocks and a corresponding sample in the CU's original Cr coding block.

[0075] Additionally, video encoder 20 may use quadratic tree partitioning to decompose the luma, Cb, and Cr residual blocks of a CU into one or more luma, Cb, and Cr transform blocks. A transform block is a rectangular (e.g., square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may comprise a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. Thus, each TU of a CU can be associated with a luma transform block, a Cb transform block and a Cr transform block. The luma transform block associated with the TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a subblock of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In monochrome images or images that have three separate color planes, a TU may comprise a single transform block and syntax structures used to transform samples from the transform block.

[0076] Video encoder 20 may apply one or more transforms to a luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block can be a two-dimensional array of transform coefficients. A transform coefficient can be a scalar quantity. Video encoder 20 may apply one or more transforms to a Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of a TU to generate a Cr coefficient block for the TU.

[0077] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. In general, quantization refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, which provides additional compression. After video encoder 20 quantizes a coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform Context Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements indicating the quantized transform coefficients.

[0078] Video encoder 20 may output a bit stream that includes a sequence of bits that forms a representation of encoded images and associated data. The bit stream may comprise a sequence of network abstraction layer (NAL) units. An NAL unit is a syntax structure that contains an indication of the type of data in the NAL unit and bytes and that contains this data in the form of a raw byte sequence payload (RBSP) interleaved as necessary with emulation prevention bits . Each of the NAL units includes a NAL unit header and encapsulates an RBSP.

[0079] Different types of NAL units can encapsulate different types of RBSPs. For example, a first type of NAL unit may encapsulate an RBSP for a picture parameter setting (PPS), a second type of NAL unit may encapsulate an RBSP for a coded slice, a third type of NAL unit may encapsulate an RBSP for the SEI and so on. The NAL units that encapsulate the RBSPs for video coding data (as opposed to the RBSPs for SEI message and parameter sets) can be referred to as video coding layer (VCL) NAL units.

[0080] The video decoder 30 may receive a bitstream generated by the video encoder 20. Additionally, the video decoder 30 may perform parsing of the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct images of the video data based, at least in part, on syntax elements obtained from the bit stream. The process for reconstructing the video data may be generally reciprocal to the process performed by the video encoder 20. Additionally, the video decoder 30 may inversely quantize the block coefficients associated with the TUs of a current CU. The video decoder 30 may perform inverse transforms on the coefficient blocks to reconstruct transform blocks associated with the TUs of the current CU. The video decoder 30 may reconstruct the coding blocks of the current CU by adding samples from the predictive blocks for the PUs of the current CU to the corresponding samples from the transform blocks of the TUs of the current CU. By rebuilding the coding blocks for each CU of an image, the video decoder 30 can reconstruct the image.

[0081] When a CU is intercoded, a set of motion information is present for each PU. Furthermore, each PU is encoded with a unique interprediction mode to derive a set of motion information. In the HEVC standard, there are two interprediction modes, called blend (skip is considered a special case of blend) and advanced motion vector prediction (AMVP) prediction modes respectively for one prediction unit (PU).

[0082] In AMVP or blend mode, a list of motion vector (MV) candidates is maintained for multiple motion vector predictors. The motion vector (or vectors), as well as reference indices in blend mode, of the current PU are generated by taking a candidate from the MV candidate list.

[0083] The MV candidate list contains up to five candidates for blend mode and two candidates for AMVP mode. A merge candidate can have a set of motion information, for example, motion vectors corresponding to both the reference image lists (list 0 and list 1) and the reference indices. If a merge candidate is identified by a merge index, the reference images are used for the prediction of the current blocks as well as the associated motion vectors are determined. However, under AMVP mode for each potential prediction direction from list 0 or list 1, a reference index needs to be explicitly signaled, along with an MVP index for the list of MV candidates since the AMVP candidate contains only one motion vector. In AMVP mode, the predicted motion vectors can be further refined.

[0084] As seen above, a merge candidate corresponds to a complete set of motion information while an AMVP candidate contains only a motion vector for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks, as described with respect to Figures 4 and 5 below.

[0085] In accordance with aspects of this disclosure, as described in greater detail below, the video encoder 20 and/or the video decoder 30 may be configured to perform any combination of the techniques described herein to derive motion information, perform block partitioning and/or interpolate video data. In connection with deriving motion information, video encoder 20 and/or video decoder 30 may be configured to derive motion information by performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block. Matching may be determined based on an amount of similarity between reference data, and may be referred to herein as determining a “match” or “best match”.

[0086] In some examples, the video encoder 20 and/or video decoder 30 may initially encode one or more syntax elements that indicate whether the motion derivation process is enabled. In some cases, the one or more syntax elements may be merged with another mode, such as the blend mode described above. For example, as described in greater detail with respect to the example of Figure 10, the video encoder 20 and/or video decoder 30 encode one or more syntax elements when performing blend mode (e.g., a flag, a index in a list of merge candidates, or similar) that indicates whether to perform move derivation.

[0087] In cases where motion derivation is enabled, in accordance with aspects of this disclosure, the video encoder 20 and/or video decoder 30 may perform pattern-matched motion vector derivation. For example, video encoder 20 and/or video decoder 30 may encode one or more syntax elements that indicate which motion information derivation mode to apply from a plurality of motion information derivation modes. The video encoder 20 and/or video decoder 30 may encode a flag for distinguishing between two modes of motion information derivation or an index for distinguishing between more than two modes of motion information derivation. As described herein, exemplary modes of deriving pattern-matched motion information include bilateral matching or template matching.

[0088] During the motion derivation process, the video encoder 20 and/or video decoder 30 may derive an initial motion vector for an entire block (e.g., an entire PU) based on the motion derivation process. selected. For example, video encoder 20 and/or video decoder 30 may use motion vectors from a candidate list associated with the blend mode and determine which motion vector from the candidate list results in a best match. That is, the video encoder 20 and/or video decoder 30 can determine which motion vector from the candidate list, when used in the selected motion derivation process, results in a first set of reference data corresponding to a second set of reference data outside the current block, for example, reference data that closely matches the data in the current image or another reference image. In general, “best match” can refer to video data that is most similar in terms of pixel differences.

[0089] As an example for illustration purposes, as described in greater detail with respect to Figure 8, the video encoder 20 and/or video decoder 30 may select a template matching motion information derivation mode. In this example, the video encoder 20 and/or video decoder 30 may select the candidate motion vector from the blend mode based on the vector candidate that results in a model that most closely matches a model in an image. of reference. For example, video encoder 20 and/or video decoder 30 may be configured to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block, wherein the first dataset comprises the model in the current image and the second reference dataset comprises the model in the reference image. In some cases, as described in greater detail below, the video encoder 20 and/or video decoder 30 may additionally or alternatively select a candidate motion vector based on a minimum matching cost.

[0090] In accordance with aspects of this disclosure, after determining the candidate motion vector, the video encoder 20 and/or video decoder 30 may further refine the candidate motion vector to derive motion information for the current block. . For example, the video encoder 20 and/or video decoder 30 may perform a search (e.g., a search using the motion information derivation mode) in a predefined area of the reference image indicated by the motion vector. candidate. The video encoder 20 and/or video decoder 30 may derive the motion information for the current block based on the motion vector that indicates reference data in the predefined area that has a best match and/or a minimum matching cost to video data of the current image.

[0091] In some cases, in accordance with aspects of this disclosure, the video encoder 20 and/or video decoder 30 may divide the video data block into subblocks when deriving motion information. For example, video encoder 20 and/or video decoder 30 may separately derive motion information for each sub-block of a larger block. In some cases, video encoder 20 and/or video decoder 30 may use motion information derived for a block (e.g., using the process described above) as candidate motion information for each of the sub-blocks. blocks. The video encoder 20 and/or video decoder 30 may then further refine the derived motion information for each of the sub-blocks, for example, with the use of a particular motion information derivation mode (e.g. , a two-sided matching technique, a template matching technique, or another technique as described in greater detail below).

[0092] Figure 2 is a block diagram illustrating an example video encoder 20 that can implement techniques to derive motion information, perform block partitioning, and/or interpolate video data. The video encoder 20 can perform intracoding and intercoding of video blocks within video slices. Intracoding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame image. Intercoding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or images of a video sequence. Intramode (mode I) can refer to any of several space-based encoding modes. Intermodes, such as monodirectional prediction (P-mode) or bidirectional prediction (B-mode), can refer to any of several time-based coding modes.

[0093] As shown in Figure 2, video encoder 20 receives a current video block within a video frame to be encoded. In the example of Figure 2, video encoder 20 includes video data memory 38, mode selection unit 40, reference image memory 64, adder 50, transform processing unit 52, quantization unit 54, and entropy coding 56. The mode selection unit 40 in turn includes the motion compensation unit 44, motion estimation unit 42, intraprediction unit 46, and partition unit 48. For reconstruction of the video block , the video encoder 20 also includes the inverse quantization unit 58, the inverse transform unit 60, and the adder 62. A block removal filter (not shown in Figure 2) may also be included at the filter block boundaries to remove blocking artifacts from the reconstructed video. If desired, the block removal filter would typically filter the output of adder 62. Additional filters (loop or post-loop) may also be used in addition to the block removal filter. These filters are not shown for brevity purposes, but if desired, they can filter the output of adder 50 (like a loop filter).

[0094] During the encoding process, video encoder 20 receives a video frame or slice to be encoded. The frame or slice can be divided into multiple video blocks. The video data memory 38 can store the video data to be encoded by the video encoder components 20. The video data stored in the video data memory 38 can be obtained, for example, from the video source 18 Reference picture memory 64 may be referred to as a DPB that stores reference video data for use in encoding video data by video encoder 20, for example, in intra- or inter-coding modes. The video data memory 38 and a reference image memory 64 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 38 and reference image memory 64 may be provided by the same memory device or separate memory devices. In various examples, video data memory 38 may be on-chip with other video encoder components 20, or off-chip with respect to those components.

[0095] The motion estimation unit 42 and motion compensation unit 44 perform interpredictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. The intra-prediction unit 46 may alternatively perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be encoded to provide spatial prediction. The video encoder 20 may perform multiple encoding steps, for example, to select an appropriate encoding mode for each block of video data.

[0096] Furthermore, the partition unit 48 may partition blocks of video data into sub-blocks, based on evaluation of previous partitioning schemes in previous coding steps. For example, the partition unit 48 may initially partition a frame or slice into LCUs, and partition each of the LCUs into sub-CUs based on rate distortion analysis (e.g., rate distortion optimization). The mode selection unit 40 may additionally produce a quadratic tree data structure that indicates the partitioning of an LCU into sub-CUs. Quadratic tree leaf node CUs may include one or more PUs and one or more TUs.

[0097] The mode selection unit 40 may select one of the coding modes, intra or inter, for example, based on error results, and provides the resulting intra or inter coded block to the adder 50 to generate residual block data and to adder 62 to reconstruct the encoded block for use as a reference frame. The mode selection unit 40 also provides syntax elements, such as motion vectors, intramode indicators, partition information, and other such syntax information, to the entropy coding unit 56.

[0098] The motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation, performed by motion estimation unit 42, is the process of generating motion vectors, which estimate motion for video blocks. A motion vector, for example, may indicate the displacement of a PU of a video block within a current video or image frame relative to a predictive block within a reference frame (or other coded unit) relative to the current block being encoded within the current frame (or other encoded unit). A predictive block is a block that is in very close correspondence to the block to be encoded in terms of pixel difference, which can be determined by summation of absolute difference (SAD), summation of squared difference (SSD), or other difference measures .

[0099] In some examples, video encoder 20 may perform a fast motion search to determine a motion vector of a block. There are many fast motion search methods proposed in the literature, such as Block-based Gradient Descent Method Search (BBGDS) as described, for example, in Lurng-Kuo Liu, Ephraim Feig, “A block-based gradient descent search algorithm for block motion estimation in video coding,” IEEE Trans. Circuits Syst. Video Technol. , vol. 6, pages 419 to 422, August 1996, Unrestricted Center-Biased Diamond Search (UCBDS), as described, for example, in Jo Yew Tham, Surendra Ranganath, Maitreya Ranganath, and Ashraf Ali Kassim, “A novel unrestricted center-biased diamond search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol. , vol. 8, pages 369 to 377, August 1998, and HEXagon-Based Search (HEBS) as described, for example, in Ce Zhu, Xiao Lin, and Lap-Pui Chau, “Hexagon-Based Search Pattern for Fast Block Motion Estimation,” IEEE Trans. Circuits Syst. Video Technol. , vol. 12, pages 349 to 355, major 2002. Basically, these techniques include searching only a certain number of positions within a search window based on predefined search patterns. These techniques typically work well when the movement is small and moderate.

[00100] In some examples, video encoder 20 may calculate values for sub-integer pixel positions of reference images stored in reference image memory 64. For example, video encoder 20 may interpolate values from quarter-pixel positions. pixel, one-eighth pixel positions, or other fractional pixel positions of the reference image. Therefore, the motion estimation unit 42 can perform a motion search with respect to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel accuracy.

[00101] The motion estimation unit 42 calculates a motion vector for a PU of a video block in an intercoded slice by comparing the position of the PU to the position of a predictive block of a reference image. The reference image may be selected from a first list of reference images (List 0) or a second list of reference images (List 1), each of which identifies one or more reference images stored in image memory reference frame 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[00102] Motion compensation, performed by the motion compensation unit 44, may involve obtaining or generating the predictive block based on the motion vector determined by the motion estimation unit 42. Again, the motion estimation unit 42 and The motion compensation unit 44 may be functionally integrated, in some examples. Upon receipt of the motion vector for the PU of the current video block, the motion compensation unit 44 can locate the predictive block to which the motion vector points in one of the reference image lists. Adder 50 forms a residual video block by subtracting pixel values from the predictive block from pixel values from the current video block being encoded, forming pixel difference values, as discussed below. In general, the motion estimation unit 42 performs motion estimation relative to luma components, and the motion compensation unit 44 uses motion vectors calculated based on luma components for both chroma components and luma components. . The mode selection unit 40 may also generate syntax elements associated with the video blocks and the video slice for use by the video decoder 30 in decoding the video blocks of the video slice.

[00103] The intraprediction unit 46 may intrapredict a current block, as an alternative to the intraprediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. In particular, the intraprediction unit 46 may determine an intraprediction mode to be used to encode a current block. In some examples, the intraprediction unit 46 may encode a current block using multiple intraprediction modes, e.g., during separate encoding steps, and the intraprediction unit 46 (or mode selection unit 40, in some examples ) can select a suitable intraprediction mode to use from the tested modes.

[00104] For example, the intraprediction unit 46 may calculate rate distortion values using a rate distortion analysis for the various intraprediction modes tested and select the intraprediction mode that has the best rate distortion characteristics. among the tested modes. Rate distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original uncoded block that was encoded to produce the encoded block, as well as a bitrate (i.e., a number of bits) used to produce the coded block. The intraprediction unit 46 may calculate ratios from distortions and rates for the various coded blocks to determine which intraprediction mode exhibits the best rate distortion value for the block.

[00105] After selecting an intraprediction mode for a block, the intraprediction unit 46 may provide information indicative of the selected intraprediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 may encode the information which indicate the selected intraprediction mode. Video encoder 20 may include in the transmitted bit stream configuration data, which may include a plurality of intraprediction mode index tables and a plurality of modified intraprediction mode index tables (also referred to as word mapping tables). -code), definitions of coding contexts for various blocks and indications of a most likely intraprediction mode, a table of intraprediction mode indexes, and a table of modified intraprediction mode indexes to use for each of the contexts.

[00106] In accordance with aspects of this disclosure, as described herein, the video encoder 20 may be configured to perform any combination of the techniques described herein to derive motion information, perform block partitioning, and/or interpolate data of video. In particular, certain techniques of this disclosure may be performed by branch unit 49. For example, branch unit 49 may be configured to determine motion information for a current block and without including data indicating motion information in the bit stream.

[00107] In some cases, the derivation unit 49 (and/or mode selection unit 40) may determine whether to perform motion derivation for a particular block (e.g., versus traditional intraprediction or interprediction) based on a rate distortion analysis. For example, derivation unit 49 may determine whether to perform motion derivation in a manner similar to a rate distortion cost selection as is performed for merge candidates in merge mode. In this example, the derivation unit 49 may check each motion information derivation mode of a plurality of motion information derivation modes (e.g., a bilateral matching mode, template matching mode, or the like) with the use of a rate distortion cost selection. The derivation unit 49 may further compare the motion information derivation mode that has the minimum cost to other PU modes (e.g., intraprediction modes, a traditional interprediction mode, a pallet coding mode, or the like). In cases where the motion derivation mode is the most effective mode with respect to coding efficiency, the video encoder 20 may encode one or more syntax elements that indicate that the motion information is derived (rather than flagged) for the current block. The video encoder 20 may also encode one or more syntax elements to indicate the motion derivation mode from a plurality of motion information derivation modes.

[00108] In other examples, in accordance with aspects of this disclosure, the derivation unit 49 may interpolate video data that is not included in the encoded bit stream for a video sequence. For example, the derivation unit 49 may perform any motion derivation combination techniques to interpolate an image that is not included in the bit stream, e.g., similar to frame rate upconversion. In some cases, video encoder 20 may use the interpolated image during encoding. For example, the derivation unit 49 may interpolate an image and the video encoder 20 may encode data for a current image relative to the interpolated image. In this example, video encoder 20 may add the interpolated image to reference image memory 64 and encode data from other images based on at least a portion of the interpolated image.

[00109] In other examples, the derivation unit 49 may interpolate an image and the video encoder 20 may set the interpolated image equal to the current image. For example, the branch unit 49 may interpolate the current image and the video encoder 20 may encode syntax data for the current image to be included in the bit stream (e.g., slice header data and the like), but may skip encoding the video data to the current image.

[00110] Although derivation unit 49 may be configured to perform certain derivation and/or interpolation techniques as described herein, it should be understood that one or more other video encoder units 20 may also or alternatively be configured to interpolate data. For example, video encoder 20 may include a variety of other interpolators or filters, for example, to interpolate a pixel at a sub-pixel (sub-pel) location during motion compensation.

[00111] The video encoder 20 forms a residual video block by subtracting the prediction data from the mode selection unit 40 from the original video block being encoded. Adder 50 represents the component or components that perform this subtraction operation. The transform processing unit 52 applies a transform, such as a distinct cosine transform (DCT) or one conceptually similar, to the residual block, producing a video block comprising residual transform coefficient values. The transform processing unit 52 can perform other transforms that are conceptually similar to DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms could also be used.

[00112] In any case, the transform processing unit 52 applies the transform to the residual block, producing a block of residual transform coefficients. The transform can convert the residual information from a pixel value domain into a transform domain, such as a frequency domain. The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan of the matrix that includes the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform scanning.

[00113] After quantization, the entropy coding unit 56 entropy codes the quantized transform coefficients. For example, the entropy coding unit 56 may perform context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) or other entropy coding technique. In the case of context-based entropy coding, the context can be based on neighboring blocks. After entropy coding by entropy coding unit 56, the encoded bit stream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[00114] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for later use as a reference block. The motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference image memory frames 64. The motion compensation unit 44 may also apply one or more motion compensation filters. interpolation to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion-compensated predictive block produced by motion compensation unit 44 to produce a reconstructed video block for storage in reference image memory 64. The reconstructed video block can be used by the motion estimation 42 and by the motion compensation unit 44 as a reference block for intercoding a block in a subsequent video frame.

[00115] Figure 3 is a block diagram illustrating an example video decoder 30 that can implement techniques to derive motion information, perform block partitioning, and/or interpolate video data. In the example of Figure 3, the video decoder 30 includes video data memory 68, an entropy decoding unit 70, motion compensation unit 72, intraprediction unit 74, inverse quantization unit 76, inverse transform unit 78 , reference image memory 82 and adder 80. Video decoder 30 may, in some examples, perform a decoding step generally reciprocal to the encoding step described with respect to video encoder 20 (Figure 2). The motion compensation unit 72 may generate prediction data based on motion vectors received from the entropy decoding unit 70, while the intraprediction unit 74 may generate prediction data based on intraprediction mode indicators received from from the entropy decoding unit 70.

[00116] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from video encoder 20. The stored video data in video data memory 68 may be obtained, for example, from computer-readable media, for example, from a local video source such as a camera, via wireless or wired network communication. video data or by accessing physical data storage media. The video data memory 68 may form a coded picture buffer (CPB) that stores coded video data from a coded video bit stream.

[00117] Reference picture memory 82 may be referred to as a DPB that stores reference video data for use in decoding video data by video decoder 30, for example, in intra- or inter-coding modes. The video data memory 68 and a reference image memory 82 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 68 and reference image memory 82 may be provided by the same memory device or separate memory devices. In various examples, video data memory 68 may be on-chip with other video decoder components 30 or off-chip with respect to those components.

[00118] The entropy decoding unit 70 of the video decoder 30 entropy decodes the bit stream to generate quantized coefficients, motion vectors or intraprediction mode indicators, and other syntax elements. The entropy decoding unit 70 forwards the motion vectors and other syntax elements to the motion compensation unit 72. The video decoder 30 may receive the syntax elements at the video slice level and/or at the block level. of video.

[00119] When the video slice is encoded as an intracoded (I) slice, the intraprediction unit 74 may generate prediction data for a video block of the current video slice based on a signaled intraprediction mode and data from previously decoded blocks of the current frame or image. When the video frame is encoded as an intercoded slice (i.e., B, P, or GPB), the motion compensation unit 72 produces predictive blocks for a video block of the current video slice based on the motion vectors and other syntax elements received from the entropy decoding unit 70. Predictive blocks may be produced from one of the reference images within one of the reference image lists. The video decoder 30 may construct the reference frame lists, List 0 and List 1, using standard construction techniques based on reference images stored in reference image memory 82.

[00120] The motion compensation unit 72 determines prediction information for a video block of the current video slice by analyzing the motion vectors and other syntax elements, and uses the prediction information to produce the predictive blocks for the current video block being decoded. For example, the motion compensation unit 72 may use a portion of the received syntax elements to determine a prediction mode (e.g., intra- or inter-prediction) used to encode the video blocks of the video chunk, a type of video slice. interprediction (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more of the reference image lists for the slice, motion vectors for each intercoded video block of the slice, interprediction status for each intercoded video block of the slice and other information to decode the video blocks in the current video slice.

[00121] The motion compensation unit 72 can also perform interpolation based on interpolation filters. The motion compensation unit 72 may use interpolation filters as used by the video encoder 20 during encoding of the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In this case, the mode compensation unit 72 may determine the interpolation filters used by the video encoder 20 from the received syntax elements and use the interpolation filters to produce predictive blocks.

[00122] In accordance with aspects of this disclosure the video decoder 30 may be configured to perform any combination of the techniques described herein to derive motion information, perform block partitioning and/or interpolate video data. In particular, certain techniques of this disclosure may be performed by branch unit 75. For example, in accordance with aspects of this disclosure, branch unit 75 may be configured to determine motion information for a current block and without decoding the motion information. movement of an encoded bit stream.

[00123] In some cases, the derivation unit 75 may determine whether to perform motion derivation for a particular block (e.g., versus traditional intraprediction or interprediction). For example, video decoder 30 may decode one or more syntax elements that indicate that motion information is derived (rather than signaled) for a block being decoded. The video decoder 30 may also decode one or more syntax elements that indicate a mode of deriving motion information from a plurality of modes of deriving motion information to be used to decode a block. Derivation unit 75 may determine whether to perform motion derivation and the manner of deriving motion information for a block based on the decoded syntax. In some examples, as described herein, the syntax may be associated with one or more other modes, such as blend mode, AMVP, or other decoding functions.

[00124] According to other aspects of this disclosure, the branch unit 75 can interpolate video data that is not included in the encoded bit stream for a video sequence. For example, the derivation unit 75 may perform any motion derivation combination techniques to interpolate an image that is not included in the parsed bit stream, e.g., similar to frame rate upconversion. In some cases, the video decoder 30 may use the interpolated image during encoding. For example, the derivation unit 75 may interpolate an image and the video decoder 30 may decode data for a current image relative to the interpolated image. In this example, video decoder 30 may add the interpolated image to reference image memory 82 and decode data from other images based on at least a portion of the interpolated image.

[00125] In other examples, the derivation unit 75 may interpolate an image and the video decoder 30 may set the interpolated image equal to the current image. For example, the derivation unit 75 may interpolate the current image and the video decoder 30 may decode syntax elements for the current image from the encoded bit stream (e.g., slice header data and the like), but may skip decoding the video data for the current image and instead interpolate the current image.

[00126] Although the derivation unit 75 may be configured to perform certain interpolation techniques as described herein, it should be understood that one or more other units of the video decoder 30 may also or alternatively be configured to interpolate data. For example, video decoder 30 may include a variety of other interpolators or filters, for example, to interpolate a pixel at a sub-pixel (sub-pel) location during motion compensation.

[00127] The inverse quantization unit 76 inversely quantizes, e.g., dequantizes, the quantized transform coefficient provided in the bit stream and decoded by the entropy decoding unit 70. The inverse quantization process may include the use of a quantization calculated by video decoder 30 for each video block in the video slice to determine a degree of quantization and, similarly, a degree of inverse quantization that should be applied.

[00128] The inverse transform unit 78 applies an inverse transform 78, e.g., a DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to produce residual blocks in the pixel domain .

[00129] After the mode compensation unit 72 generates the predictive block for the current video block based on the motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual blocks coming from the inverse transform unit 78 with the corresponding predictive blocks generated by the mode compensation unit 72. The adder 80 represents the component or components that perform this summation operation. If desired, a block removal filter can also be applied to filter the decoded blocks to remove blocking characteristic artifacts. Other loop filters (either in the encoding loop or after the encoding loop) can also be used to smooth pixel transitions, or otherwise improve video quality. The video blocks decoded into a given reference frame or image are then stored in image memory 82, which stores reference images used for subsequent motion compensation. Reference image memory 82 also stores the decoded video for later presentation on a display device, such as display device 32 of Figure 1.

[00130] Figures 4A and 4B are conceptual diagrams illustrating spatial neighbor candidates in HEVC. In some examples, video encoder 20 and/or video decoder 30 may derive spatial motion vector (MV) candidates from neighboring block 0, neighboring block 1, neighboring block 2, neighboring block 3, or neighboring block 4 to PU0.

[00131] In some cases, the techniques for generating the MV candidates from the blocks differ for the merge and AMVP modes. Figure 4A illustrates an example for the blend mode. For example, in HEVC, a video encoder (e.g., such as video encoder 20 and/or video decoder 30 of Figures 1 to 3) can derive up to four spatial MV candidates. Candidates may be included in a candidate list that has a particular order. In an example, the order for the example in Figure 4A might be neighboring block 0 (A1), neighboring block 1 (B1), neighboring block 2 (B0), neighboring block 3 (A0), and neighboring block 4 (B2).

[00132] Figure 4B illustrates an example for the AMVP mode. For example, in HEVC, the video encoder can divide neighboring blocks into two groups: where the left group includes neighboring block 0 and neighboring block 1 and the above group includes neighboring block 2, neighboring block 3 and neighboring block 4. For each group, the potential motion vector candidate associated with a neighboring block that refers to the same reference image as that indicated by the signed reference index (for the block currently being encoded) may have the highest priority to be chosen to form a final candidate from the group. It is possible that none of the neighboring blocks contain a motion vector that points to the same reference image. Therefore, if such a candidate cannot be found, the video encoder can scale the first available candidate to form the final candidate, thus temporal distance differences can be compensated.

[00133] In accordance with aspects of this disclosure, motion vector candidates such as the motion vectors associated with neighboring blocks shown in Figures 4A and 4B can be used to derive a motion vector for a block. For example, the video encoder may generate a candidate list that includes motion vector candidates from the neighboring blocks shown in Figures 4A and 4B. In this example, the video encoder may use one or more of the candidates from the candidate list as an initial motion vector in a process of deriving motion information (e.g., bilateral matching, template matching, or the like). The video encoder may apply one or more of the motion vector candidates in a motion search of a motion derivation process vector to identify reference data. The video encoder may select the candidate from the list that identifies close match reference data (e.g., as described in relation to Figures 8 to 9 below). For example, the video encoder may perform motion search for a first set of reference data that corresponds to a second set of reference data outside the current block. The video encoder may, in some cases, further refine the candidate, for example, by additionally performing a motion search in an area indicated by the selected candidate, to determine a derived motion vector using the information derivation process. of movement.

[00134] Figures 5A and 5B are conceptual diagrams that illustrate temporal prediction motion vector in HEVC. A temporal motion vector predictor (TMVP) candidate, if enabled and available, is added to a list of MV candidates after spatial motion vector candidates. In HEVC, the motion vector derivation process for a TMVP candidate is the same for both blend mode and AMVP, however, the target reference index for the TMVP candidate in blend mode is typically set to zero.

[00135] Figure 5A illustrates a primary block location (shown as block “T”) for a TMVP candidate, which is the lower right block outside of the co-located PU. Localization can compensate for bias to the top and left blocks used to generate spatial neighbor candidates. However, if the T block is located outside the current CTB row or motion information is not available, the block is replaced with a central PU block, as illustrated by the T block dashed arrows in Figure 5A.

[00136] Figure 5B illustrates the derivation of a TMVP candidate 84 for a current block 86 of a current image 88 from a co-located PU 90 of a co-located image 92, as indicated at the slice level (e.g., in a slice header). Similar to the temporal direct mode in AVC, a TMVP candidate motion vector can be subjected to motion vector scaling, which is performed to compensate for distance differences, e.g., temporal distances between images. In connection with motion vector scaling, a video encoder (such as video encoder 20 and/or video decoder 30) may be configured to initially determine that the value of the motion vectors is proportional to the distance of the images at the time of presentation. A motion vector associates two images, the reference image and the image containing the motion vector (namely, the containment image). When one motion vector is used to predict the other motion vector, the distance of the containment image and the reference image is calculated based on the Image Order Count (POC) values.

[00137] For a motion vector to be predicted, both the associated containment image for the motion vector and a reference image of the motion vector may be different. Therefore, the video encoder can calculate a new distance based on POC values, and the video encoder can scale the motion vector based on these two POC distances. For a spatial neighbor candidate, the containment images for the two motion vectors are the same, while the reference images are different. In HEVC, motion vector scaling applies both TMVP and AMVP to spatial and temporal neighboring candidates.

[00138] In some examples, a video encoder may be configured to determine one or more artificial motion vector candidates. For example, if a list of motion vector candidates is not complete, the video encoder may generate artificial motion vector candidates and insert the artificial motion vector candidates at the end of the list until the list includes a predetermined number of entries. . In merge mode, there are two types of artificial MV candidates that include a combined candidate derived only for B-slices and a zero candidate. In some cases, candidate zero is only used for AMVP if the combined type does not provide enough artificial candidates.

[00139] For each pair of candidates that are already in the candidate list and have required motion information, the bidirectional combined motion vector candidates are derived by a combination of the motion vector of the first candidate that refers to an image in the list 0 and the motion vector of a second candidate that refers to an image in list 1.

[00140] In accordance with aspects of this disclosure, motion vector candidates, such as the TMVP shown in Figures 5A and 5B, can be used to derive a motion vector for a block. For example, the video encoder may generate a candidate list that includes a TMVP determined in accordance with the process described above. In this example, the video encoder may use the TMVP as an initial motion vector in a process of deriving motion information (e.g., bilateral matching, template matching, or the like). The video encoder can apply TMVP to a motion derivation process vector to identify reference data. The video encoder may select the TMVP in cases where the TMVP identifies closely matched reference data (e.g., as described with respect to Figures 8 through 9 below). The video encoder may, in some cases, further refine the TMVP to determine a derived motion vector using the motion information derivation process.

[00141] In some examples, the video encoder may remove a candidate list that includes motion vector candidates (such as those described with respect to Figures 4A to 5B). For example, in some cases, candidates from different blocks may be the same, which decreases the efficiency of a merge/AMVP candidate list. Video code may apply a removal process to resolve this issue. The video encoder can compare a candidate against others in the current candidate list to avoid inserting an identical candidate. To reduce complexity, the video encoder can only apply limited numbers of removal processes rather than comparing each potential to all existing ones.

[00142] Figure 6 is a conceptual diagram illustrating an example of one-way motion estimation (ME) in frame rate upconversion (FRUC). In particular, Figure 6 illustrates a current frame 100, a reference frame 102, and an interpolated frame 104. In some cases, a video decoder or post-processing device may interpolate images based on one or more reference images. The video decoder or post-processing device may interpolate images to upconvert an original frame rate from an encoded bit stream. Alternatively, the video decoder or post-processing device may interpolate images to insert one or more images that have been skipped by a video encoder to encode a video sequence at a reduced frame rate. In any case, the video decoder or post-processing device interpolates frames (such as interpolated frame 104) that are not included in an encoded bit stream that was received by the video decoder using images that have been decoded (such as frame current 100 and reference frame 102). The video decoder or post-processing device may interpolate the images using any of several interpolation techniques, for example, using motion-compensated frame interpolation, frame repetition, or frame averaging. .

[00143] The frame interpolation techniques noted above are typically implemented post-cycle. For example, a video decoder typically receives and decodes an encoded bitstream to generate a reconstructed representation of a video sequence that includes current frame 100 and reference frame 102. After the decoding cycle, the video decoder or Another post-processing device may interpolate the images to be included with the reconstructed representation that includes the interpolated frame 104. In some cases, the process of interpolating the image may be referred to as frame rate upconversion (FRUC), due to the fact that the resulting sequence of images includes (interpolated) additional images that were not included in the encoded bit stream.

[00144] Consequently, FRUC technology can be used to generate high frame rate videos based on low frame rate videos. FRUC was used in the display industry. Examples include H. Liu, R. Xiong, D. Zhao, S. Ma, W. Gao, “Multiple Hypotheses Bayesian Frame Rate Up-Conversion by Adaptive Fusion of Motion-Compensated Interpolations”, IEEE transactions on circuits and systems for video technology, vol. 22, No. 8, August 2012; W. H. Lee, K. Choi, J. B. Ra, “Frame rate up conversion based on variational image fusion,” IEEE transactions on image processing, vol. 23, No. 1, January 2014; and U. S. Kim, M. H. Sunwoo, “New frame rate up-conversion algorithms with low computational complexity,” IEEE transactions on circuits and systems for video technology, vol. 24, no 3, March 2014.

[00145] FRUC algorithms can be divided into two types. One type of methods interpolates intermediate frames by averaging or simple frame repetition. However, this method provides inadequate results on an image that contains a lot of motion. The other type of method, called motion-compensated FRUC (MC-FRUC), considers object motion when it generates intermediate frames and consists of two steps: motion estimation (ME) and motion-compensated interpolation (MCI). ME generates motion vectors (MVs), which represent object motion using vectors, whereas MCI uses MVs to generate intermediate frames.

[00146] The block matching algorithm (BMA) is widely used for ME in MC-FRUC as it is simple to deploy. BMA divides an image into blocks and detects the movement of those blocks, for example, to determine whether the blocks match. Two types of ME are primarily used for BMA: unilateral ME and bilateral ME.

[00147] As shown in Figure 6, the unilateral ME obtains MVs by seeking the best block match of the reference frame 102 of the current frame 100. Then, the block in the motion path in the interpolated frame can be located so that the MV is achieved. As shown in Figure 6, three blocks including 106A, 106B and 106C of the current frame 100, the reference frame 102 and the interpolated frame 104, respectively, are involved following the motion trajectory. Although the block 106A in the current frame 100 belongs to a coded block, the best matching block 106B in the reference frame 102 may not completely belong to a coded block, and neither does the block 106C in the interpolated frame 104. Consequently, the overlapping regions of blocks and unfilled regions (holes) may occur in the interpolated frame.

[00148] To handle overlaps, simple FRUC algorithms merely involve averaging and overwriting the overlapping pixels. Furthermore, the holes are covered by the pixel values of a reference or a current frame. However, these algorithms result in blocking and blurring artifacts. Therefore, motion field segmentation, consecutive extrapolation using the distinct Hartley transform, and image retouching are proposed to manipulate holes and overlaps without increasing blocking and blurring artifacts.

[00149] In accordance with aspects of this disclosure, a video encoder (such as video encoder 20 and/or video decoder 30) may generate interpolated frame 104 in the encoding or decoding cycle using the one-way matching technique shown in Figure 6. For example, the video encoder may use image level FRUC to interpolate interpolated frame 104 as a predictor of the current image, using the reconstructed pixel matrix. In some examples, this interpolated image may be considered as a reference image or reconstruction of the current frame 100. In other examples, the video encoder may set the current image equal to the interpolated image. This image may be marked as a throwaway image and/or non-reference image by syntax elements or decoding processes.

[00150] In some examples, the video encoder may interpolate a current image such that a FRUC mode is the only allowed mode, wherein the FRUC mode indicates the one-way matching technique shown in Figure 6 or any other derivation of motion information or interpolation techniques described in this document. Therefore, instead of a quadratic tree-based CU structure signaling, all blocks can have the same predefined or signaled size. In some cases, only a subset of valid Cu encoding modes, such as regular skip, regular merge, FRUC mode, and intra mode may be allowed. Alternatively or additionally, a hole fill mode for FRUC may be enabled on this image or slice.

[00151] In accordance with aspects of this disclosure, an SEI message may be introduced to indicate which images or subsequence of images are encoded by the image level FRUC. These images can be discarded without impacting the quality of other images. This SEI message may indicate (or additionally indicate) which temporal level (or levels) contains FRUC encoded images, or contains only FRUC encoded images. This information in the SEI message may also be present in other high-level syntax locations, such as PPS, SPS, and VPS.

[00152] In some examples, a video encoder may encode a portion of an image and interpolate the remaining video data. For example, the video encoder may encode a so-called “hint” for decoder-side frame rate upconversion, which may allow feature-rich or intelligent decoders to optionally generate the FRUC frames. For example, several key regions (such as rectangle regions) can be flagged as a hint for these FRUC frames. When the hint is received and optionally processed, the FRUC method specified as part of the decoder may be used primarily by regions that are not key regions, while the key regions must be further processed by means that may not be specified by the decoder, such as hole filling methods.

[00153] With respect to hole filling, in accordance with aspects of this disclosure, a video encoder can implement block-based hole filling techniques. For example, one hole filling technique is to use spatially neighboring interblocks to predict the current intrablock in the same CTU. For example, the video encoder may encode/decode a CTU twice. The first encoding/decoding is as normal. In the second round, only intra blocks are encoded/decoded and overwritten. For an intra block, all its spatially neighboring interblocks in the same CTU, which include those to the lower right of the current block, are marked as available for intra prediction. The hole filling method can also be slice, cut, image or any other level. Another hole filling method may use an image retouching technique. Other hole filling techniques may also be applicable.

[00154] Figure 7 is a conceptual diagram illustrating an example of bilateral motion estimation (ME) in FRUC. In particular, Figure 7 illustrates an interpolated block 108 of an interpolated frame 110 that is interpolated from a current block 112 of a current frame 114 and a reference block 116 of a reference frame 118. As shown in Figure 7, bilateral ME is another solution (in MC-FRUC) that can be used to avoid the problems caused by overlaps and holes shown in Figure 6. Bilateral ME obtains MVs passing through the interpolated block 108 using temporal symmetry between blocks 112 and 116 of current frame 114 and reference frame 118, respectively. As a result, it does not generate overlaps and holes. Since the current block is assumed to be a block being processed, in a certain order, for example, as in the case of video encoding, a sequence of these blocks would cover the total intermediate image without overlap. For example, in the case of video encoding, blocks can be processed in decoding order.

[00155] In accordance with aspects of this disclosure, the bilateral motion estimation shown in the example of Figure 7 can be leveraged to derive motion information. For example, as described in greater detail with respect to Figure 9 below, a video encoder (such as video encoder 20 or video decoder 30) may apply bilateral correspondence as a mode of deriving motion information to derive motion information. during coding. In bilateral matching, the video encoder may perform a motion search for a first set of reference data in a first reference image that corresponds to a second set of reference data in a second reference image.

[00156] According to other aspects of this disclosure, a video encoder (such as video encoder 20 and/or video decoder 30) may generate the interpolated frame in the encoding or decoding cycle using the bilateral matching technique shown in Figure 7. For example, the video encoder can use image-level FRUC to interpolate the interpolated image as a predictor of the current image, using the reconstructed pixel matrix. In some examples, this interpolated image can be considered as a reference image or a reconstruction of the current image. In other examples, the video encoder may set the current image equal to the interpolated image. This image may be marked as a throwaway image and/or non-reference image by syntax elements or decoding processes.

[00157] Figure 8 is a conceptual diagram illustrating an example of decoder side motion vector derivation (DMVD) based on template matching. With advanced video codecs, the bit percentage of bitstream motion information becomes bigger and bigger. In some cases, DMVD can reduce the bit cost of motion information. Template matching-based DMVD can exhibit a coding efficiency improvement, as described, for example, in S. Kamp, M. Wien, “Decoder-side motion vector derivation for block-based video coding”, IEEE transactions on circuits and systems for video technology, vol. 22, no 12, December 2012.

[00158] In the example of Figure 8, a current image 120 includes a prediction target 122 (e.g., a block currently being encoded) and a template 124. Reference images 126 include a co-located template 128, a best match 130, and a displacement vector 132. A video encoder (such as video encoder 20 and/or video decoder 30) may use model 124 to search for a best match for prediction target 122 (e.g., instead of using the target itself prediction code 122, which is still being coded). For example, the video encoder may perform a motion search to identify a first set of reference data (e.g., best match 130) that corresponds to a second set of reference data outside the prediction target 122 (e.g., model 124). As noted above, match may be determined based on an amount of similarity between reference data, and may be referred to herein as determining a “match” or “best match”.

[00159] In the example shown, the video encoder can identify the co-located model 128 in the reference images 126. The video encoder can then search for the best match 130, which includes pixel values that are similar to the model 124. video encoder may determine the displacement vector 132 based on the displacement of the co-located model 128 and the best match 130 in reference images 126.

[00160] Assuming that the model 124 and the prediction target 122 come from the same object, the motion vector of the model can be used as the motion vector of the prediction target. Therefore, in the example of Figure 8, the video encoder may apply the displacement vector 132 to the prediction target 122. Since template matching is conducted in both a video encoder and a video decoder, the displacement vector Motion can be derived on the decoder side to avoid signaling cost.

[00161] In accordance with aspects of this disclosure, the video encoder may apply template matching as a mode of deriving motion information. For example, the video encoder may apply template matching to derive motion information from a current block by finding a best match between the template 124 of the current image and corresponding reference data in reference images 126. Although the example of Figure 8 illustrates model 124 as an L-shaped block of video data, it should be understood that other models can be used. For example, the video encoder may use multiple blocks as a template, e.g., one or more blocks positioned to the left of the prediction target 122 and one or more blocks positioned above the prediction target 122.

[00162] In accordance with aspects of this disclosure, the video encoder may apply the template matching techniques shown in Figure 8 with the use of one or more motion vectors from a list of motion vector candidates. For example, the video encoder may be configured to determine one or more candidate motion vectors using any combination of the techniques described herein (e.g., blend mode candidates, AMVP candidates, a TMVP, or the like). . The video encoder can then be configured to apply one or more of the candidate motion vectors to model 124 to locate the co-located model 128 (in this example, the location of the co-located model 128 is dictated by the one or more candidate motion vectors and not necessarily strictly co-located). The video encoder may be configured to determine which of the candidate motion vectors result in a better match between the model 124 and the co-located model 128.

[00163] In accordance with aspects of this disclosure, the video encoder may then be configured to refine the candidate motion vector to derive motion information for the prediction target 122. For example, the video encoder may search for a best match for the model 124 in a region of the reference images 126 identified by the candidate motion vector. Upon determining a best match, the video encoder may determine an offset between the template 124 and the determination-based match. The video encoder may assign the displacement as a derived motion vector to the prediction target 122.

[00164] Figure 9 is a conceptual diagram illustrating an example of bidirectional motion vector derivation in DMVD. Another category of DMVD is mirror-based bidirectional MV derivation, as described, for example, in Y.-J. Chiu, L. Xu, W. Zhang, H. Jiang, “Decoder-side Motion Estimation and Wiener filter for F£EVC”, Visual communications and Image Processing (VCIP), 2013. The concept of bidirectional motion vector derivation in DMVD may be similar to bilateral ME in FRUC. For example, mirror-based MV derivation can be applied by estimating centrosymmetric motion around search centers at fractional sample precision.

[00165] The example of Figure 9 includes the current image 140 that has the current block 142 (in which the block is currently encoded), a first candidate motion vector PMV0 that identifies a first model block 144 of a first reference image 146 (ref L0), and a second candidate motion vector PMV1 that identifies a second template block 148 of a second reference image 150. The video encoder may apply dMV as a bias to locate a first reference block 152 in the window search window 154 of the first reference image 146 and to locate a second reference block 156 in the search window 158 of the second reference image 150.

[00166] For example, the video encoder may add dMV to PMV0 and subtract dMV from PMV1 to generate a pair of MV, MV0 and MV1. The video encoder may check all dMV values within the search window 154 and 158 to determine which dMV value results in the best match between the first reference block 152 (e.g., a first reference data set) of ref. L0 and the second reference block 156 (e.g., a second set of reference data) of ref L1. In some examples, the video encoder may determine the best match based on the Sum of Absolute Difference (SAD). In other examples, the video encoder may use another metric to determine the best match. The size and location of search windows 154 and 158 may be predefined or may be signaled in a bit stream.

[00167] The video encoder may select the MV pair with the minimum SAD as the output of the centrosymmetric motion estimation. Since this technique uses a future reference (reference at a later temporal position relative to the current frame) and a back reference (reference at an earlier temporal position relative to the current frame) for SAD matching, it cannot be applied to the current frame. Low delay P or B frames where only the last reference is available.

[00168] In accordance with aspects of this disclosure, the video encoder may apply bidirectional motion vector derivation techniques as a mode of deriving motion information. In some examples, the video encoder may apply the techniques shown in Figure 9 using one or more motion vectors from a list of motion vector candidates. For example, the video encoder may be configured to determine one or more candidate motion vectors using any combination of the techniques described herein (e.g., blend mode candidates, AMVP candidates, a TMVP, or the like). . The video encoder may then be configured to apply one or more of the candidate motion vectors as PMV0 and/or PMVl to locate the first model block 144 and second model block 148. The video encoder may be configured to determine which of the candidate motion vectors results in a better match between the first model block 144 and the second model block 148.

[00169] In accordance with aspects of this disclosure, the video encoder may then be configured to refine the candidate motion vector to derive motion information for the current block 142. For example, the video encoder may search for a best match by applying a range of dMV values as described above. In this way, the video encoder can derive the MV pair MV0 and MV1.

[00170] Figure 11 is a conceptual diagram illustrating motion vector derivation based on extended bilateral correspondence. A potential disadvantage of mirror-based bidirectional MV derivation (e.g., as shown in Figure 10) is that it does not work when two references of the current image are both earlier or both later than the current image. The extended bilateral matching techniques described in this document can, in some cases, overcome the disadvantage that all reference images of the current image are on the same side (in display order) as the current image.

[00171] The example of Figure 11 includes a current image 160 that includes a current block 162, a first reference image (Ref0) 164 that includes a first reference block 166, and a second reference image (Ref1) 168 that includes a second reference block 170. As shown in Figure 11, the first reference image (Ref0) 164 and the second reference image (Ref1) 168 are both located before the current image in the temporal direction. Assuming that the first reference block 166, the second reference block 170 and the current block 162 are along the same movement path, the ratio between MV0 and MV1 should be equal to the ratio between the temporal distance TDO and TD1. In other words, given the two references Ref0 and Ref1 with the temporal distance TDO and TD1 for the current image, for any MV0 in Ref0, MV1 in Ref1 can be determined by scaling MV0.

[00172] The video encoder may select the final pair of MV0 and MV1 as the pair that minimizes the cost of matching between the even block pointed to by MV0 and MV1 (e.g., as described above in relation to Figure 10). Theoretically, the current block 162 can be considered as an extrapolated block based on first reference block 166 and second reference block 170. It should be noted that extended bilateral matching also works in the normal bidirectional case where the current image is temporarily between both references. In this case, the current block 162 can be considered as an interpolated block based on first reference block 166 and second reference block 170. Furthermore, the bilateral matching techniques described herein do not require a “mirror relationship” between MV0 and MV1, even in the bidirectional case. The bilateral correspondence assumption is that the ratio between MV0 and MV1 is in proportion to the ratio of the temporal distance from Ref0 to the current image and that from Ref1 to the current image.

[00173] Clearly, for reference blocks other than the first reference block 166 and second reference block 170, the video encoder can derive a different MV pair. In one example, the video decoder may select reference images to perform bilateral matching according to an order in which the reference images appear in a reference image list. For example, the video encoder may select the first reference in reference list 0 as Ref0 and the first reference in reference list 1 as Ref1. The video encoder can then search for the MV pair (MV0, MV1). In another example, the video encoder selects Ref0 based on an entry in an initial list (e.g., a list of initial motion vector candidates). The video encoder can then establish Ref1 as a reference image in the other reference image list that is temporally closer to the current image. Consequently, the video encoder can search for the MV pair (MV0, MV1) in Ref0 and Ref1.

[00174] Therefore, in accordance with aspects of this disclosure, the video encoder can apply the extended bidirectional motion derivation techniques illustrated in Figure 11 as a mode of deriving motion information. For example, the video encoder may use bilateral correspondence to derive motion information from current block 162 by finding the best match between two blocks (e.g., as first reference block 166 and second reference block 170) along the motion trajectory of the current block in two different reference images. Under the assumption of continuous motion path, the motion vectors MV0 and MV1 pointing to the two reference blocks, the first reference block 166 and the second reference block 170 should be proportional to the temporal distances, i.e., TD0 and TD1, between the current image and the two reference images. As a special case, when the current image 160 is temporarily between two reference images (as shown in the example of Figure 10) and the temporal distance from the current image to the two reference images is the same, the bilateral correspondence becomes derivation of Mirror-based bidirectional MV.

[00175] Figure 11 is a flowchart illustrating an example of decoding a prediction unit (PU) using DMVD. In Y.-J. Chiu, L. Xu, W. Zhang, H. Jiang, “Decoder-side Motion Estimation and Wiener filter for HEVC”, Visual communications and Image Processing (VCIP), 2013, it was further proposed to combine bidirectional MV derivation based on mirror with blend mode in HEVC. In the proposed technique, a flag called pu_dmvd_flag is added to a B-slice PU to indicate whether a DMVD mode is applied to the current PU. Because DMVD mode does not explicitly convey any MV information in the bitstream, the pu_dmvd_flag syntax element is integrated with the blend mode syntax in HEVC (which uses an index for the data representative of a vector motion rather than the motion vector itself).

[00176] In the example of Figure 11, a video decoder (such as video decoder 30) may begin decoding a PU (180). The video decoder 30 may determine whether the mode used to decode the PU is blend mode (182), for example, based on the syntax included in a bitstream that includes the PU. If the blend mode is not used (the “no” branch of step 182), the video decoder 30 may use a regular process for an unmerged PU to decode the PU (184) and terminate the process (186).

[00177] If blend mode is used (the “yes” branch of step 182), the video decoder 30 may determine whether DMVD is used to determine motion information for the PU based on the value of the syntax element pu_dmvd_flag ( 188). If DMVD is not used (the “no” branch of step 188), the video decoder 30 may use a regular blending mode to decode the PU (190) and terminate the process (186). If DMVD is used (the “yes” branch of step 188), video decoder 30 may apply a DMVD process to determine motion information for the PU (192) and terminate the process (186).

[00178] In some cases, current video coding techniques may have certain limitations. For example, certain DMVD techniques can be considered as a subset of FRUC technology. Although DMVD has been applied in video coding, other effective FRUC techniques have not been deployed in video coding, for example, in the video coding cycle by a video encoder or video decoder. Furthermore, although different DMVD techniques have been proposed, the interaction and overlap of these techniques are not explored in the decoder. That is, only one DMVD mode has been used in other video coding systems. How to use a plurality of DMVD techniques to further improve coding efficiency has not been studied.

[00179] As another potential limitation, DMVD may only apply to relatively large blocks of video data and therefore may not be very effective. Applying these methods to smaller blocks can lead to significant overhead due to signaling cost. In some cases, the search range for traditional DMVD techniques may be relatively small, and only several points are searched, for example, 16 points. Furthermore, as noted above, mirror-based bidirectional MV derivation cannot be applied in the case of low-delay B due to the fact that two reference images with display order before and after a current image need to be identified and this is not possible in the case of low delay.

[00180] Another potential limitation may be that, in the decoder, the matching cost of traditional DMVD techniques may only consider distortion. However, the magnitude of motion vector has not yet been considered in the matching cost, which may lead to local optimization or inaccurate matching result, for example, due to noise in the pixel domain. Furthermore, the complexity of traditional DMVD techniques can be relatively high, with respect to both memory bandwidth and computational complexity, especially due to the fact that interpolation is required for fractional pel motion vectors during search in the decoder.

[00181] The techniques of this disclosure may address one or more of the potential limitations described above. In some examples, techniques for deriving motion information can be applied individually. Alternatively, any combination of the techniques described herein can be applied together. As described herein, benchmark information can generally be considered as part of movement information. In some cases, the motion vector information and reference index information may be referred to as a motion information set.

[00182] Certain techniques of this disclosure include selecting a motion derivation mode from a plurality of potential motion derivation modes. For example, in accordance with aspects of this disclosure, a video encoder (e.g., video encoder 20 or video decoder 30) may use two or more motion derivation techniques to provide better prediction of motion information in a video decoder to avoid signaling motion information in the bitstream. The two or more motion derivation techniques may include, but are not limited to, bilateral matching, template matching, and any other matching method. These techniques can generally be referred to as motion information derivation modes or FRUC modes. Therefore, it should be understood that in some cases a technique referred to as a FRUC mode may instead be used to interpolate motion information to a current block (e.g., instead of interpolating a new block of video data).

[00183] In some examples, when multiple motion derivation methods were used, rather than having different optimization methods for different derivation methods, the process for finding the best motion for two or more of the motion derivation methods may be aligned, with regard to selecting the starting points to search and how to search around the starting points. For example, the video encoder may construct a list of motion vector candidates, select an initial candidate from the list, and refine the candidate using the same matching and searching techniques. In this example, bidirectional motion derivation and template matching-based motion derivation can be used adaptively on the decoder side.

[00184] In accordance with aspects of this disclosure, additional block-level signaling is introduced to identify which motion derivation method is used to encode the current block. For example, the video encoder may encode one or more syntax elements to indicate whether derivation of motion information is enabled. The video encoder may also encode one or more syntax elements to indicate a mode of deriving particular motion information from a plurality of potential modes. In other examples, the motion information derivation technique to be used may not be signaled, but derived in the video decoder, for example, based on a prediction mode or other information available before the current block is decoded. In still other examples, the video encoder may perform multiple modes of deriving motion information and determine a weighted average of the predictions of the two or more deriving techniques to encode the current block.

[00185] In an example for purposes of illustration, a video decoder, such as video decoder 30, may first identify whether to apply a motion information interpolation mode (e.g., based on syntax in the bit stream). The video decoder 30 can then identify which motion information derivation mode should be applied (e.g., based on syntax in the bitstream). In an example, when multiple motion derivation methods are supported simultaneously, e.g., both extended bilateral matching and template matching modes, an index value may be signaled in the bitstream to indicate which motion derivation method is actually supported. in use for a current block. The index can have three values which include, two-sided matching and template matching.

[00186] When using CABAC encoding, two receptacles can be used to represent the index. The two receptacles may both use spatial neighbors as contexts or only one of them may use spatial neighbor contexts. Alternatively, one or both receptacles may use other encoded information, such as CU depth, depending on the contexts. Index binarization can be predefined, such as “0” for off, “10” for extended two-sided matching, and “11” for template matching. Alternatively, index binarization can be signaled in a slice header or derived from encoded information such as slice type, temporal level, or QP information.

[00187] According to aspects of this disclosure, the syntax indicating a mode of deriving motion information may be included with another mode of encoding. In an example, a mode of deriving motion information may be considered a particular blend candidate, as indicated by a blend index. In this case, the size of the merge candidate list can be increased to accommodate the additional index. The merge candidate index for the motion information derivation mode can be predefined or signaled in the bitstream.

[00188] In some examples, extended bilateral matching and template matching are both supported with blend mode. In these examples, when the merge flag equals 1, a new derivation of motion information flag is flagged to indicate whether the motion information is derived for the current PU. The flag may use the same flag as its spatial neighbor, such as top and left blocks as CABAC encoding contexts. When this flag is on, a second flag is flagged to indicate which motion information derivation mode (for example, extended bilateral matching or template matching) is used to derive the block's motion information. When motion derivation is enabled, even when the current mode is blend mode, no blend index is signaled. Alternatively or additionally, a derivation of particular motion information (e.g., as template matching) is not permitted if the PU is not the first PU of a CU in the decoding order. In this case, only the flag needs to be flagged to indicate whether the two-sided correspondence extended by used for the PU or not.

[00189] In another example, the motion information derivation mode can be combined with the AMVP mode, for example, at the PU level. In an example, additional syntax elements (e.g., an additional flag) may be flagged to indicate the mode of deriving motion information. When this mode is enabled, no AMVP indexes can be signaled in the bitstream. Otherwise, a regular AMVP index can be signaled in the bitstream. In another example, the mode of deriving motion information may be considered as a particular AMVP candidate, as the first AMVP candidate, in a list of AMVP candidates. In some cases, when combined with AMVP mode, a motion vector may not be derived at a sub-PU level.

[00190] According to other aspects, a video encoder may initially conditionally encode a CU level flag (e.g., for an intercoded CU) to indicate whether all PUs within the current CU use the derivation of motion information. In one example, the PU level flag is not flagged. In another example, when the CU flag is equal to 0 (that is, not all PUs are encoded with the mode), the PU level flag of the first PU is additionally signaled when the second PU does not include the level flag of PU.

[00191] In some examples, the motion information derivation mode may be disabled for specific slice types, temporal levels, block types, or block sizes. In one example, derivation of motion information is not permitted when the current slice includes only reference images whose temporal positions are all before or after that of the current image. In another example, deriving motion information is not allowed for PUs other than 2Nx2N. When disabling motion information derivation, no block-level signaling related to motion information derivation is required.

[00192] In accordance with aspects of this disclosure, enabling or disabling the motion information derivation techniques described herein can be controlled by high-level syntax to provide a trade-off of better complexity versus coding efficiency and/or flexibility of encoder/decoder. In an example, a flag may be flagged in an SPS, PPS, slice header, or any other high-level syntax header to indicate the use of a motion information derivation mode. When this flag indicates that this encoding mode is not enabled, the CU/PU level flags may not be encoded into the bitstream.

[00193] In some cases, high-level syntax may additionally or alternatively be used to indicate other motion information derivation parameters. For example, a search algorithm index that should be used to search for the PU level motion vector may be encoded in a bit stream within an SPS, PPS, or slice header. In some cases, a search algorithm index that is used to search the sub-block level motion vector may be encoded into the bitstream within an SPS, PPS, or slice header. In some examples, to keep computational complexity low on the decoder side, the maximum block/partition match numbers at the PU level, the maximum block/partition match numbers at the PU sublevel, and/or the total match number of both PU and sub-PU levels can be restricted. In some cases, these maximum numbers may be predefined or signaled in the bitstream.

[00194] The techniques of this disclosure include a variety of techniques for deriving motion information. In some examples, the video encoder may determine a starting list (initial candidate list) of inputs that contain motion vectors, and a best starting point is identified as an entry from the initial list. Inputs containing motion vectors may be motion vectors of spatial neighbors, temporal neighbors, and/or motion vectors derived by other means. Alternatively, the best starting point (or index of the best starting point) can be signaled to the decoder.

[00195] In some examples, the initial list may contain the motion vectors of spatial and/or temporal neighbors. Each entry in the initial list can be a set of uni-predictive motion information, including a motion vector and its reference index. In an example, the initial list may be generated in the same way as the motion prediction candidate list used in another encoding mode, e.g., the same as the merge candidate list. In this case, up to two motion vectors from each merge candidate can be used to generate up to two entries in the initial list. In some cases, entries in the initial list may be generated from a subset of motion prediction candidates in a list used in another encoding mode, for example, a subset of the merge candidate list.

[00196] In another example, additional motion vectors may be added to the initial list in addition to those in the motion prediction candidate list used in another encoding mode, for example, in addition to those in the merge candidate list. Alternatively or additionally, the motion vectors of the spatial neighbors of the current block, such as a top block, a left block, a top right block, or another block can be added to the initial list. In some cases, zero motion vectors with different reference image indices can also be added to the list.

[00197] Alternatively or additionally, the motion vectors of the blocks of the current block temporally co-located (e.g., a TMVP for the current block) and/or motion vectors of the blocks temporally co-located to the upper right of the current block in the reference images may be added to the initial list. Before adding a particular candidate motion vector to the list, the motion vectors can be scaled or unscaled based on temporal distance.

[00198] In some aspects, a motion vector may be interpolated and/or extrapolated from a reference image and may be added to the initial list. For example, before encoding an inter-image, an interpolated and/or extrapolated motion field of the image can be generated based on its reference images with a technique similar to unilateral EM. The interpolated and/or extrapolated motion field of a block can be used for MV prediction or used as additional initial candidates in a MV search of a motion information derivation mode. Note that the interpolated and/or extrapolated motion field is saved in the unit of 4 x 4 or 8 x 8 or any other predefined/flagged block level, and a PU can contain multiples of these blocks so that the multiple interpolated MVs and/or or extrapolated can be used.

[00199] In one example, the motion field of each of the reference images in both reference lists is crossed by the NxN block, where N can be predefined as 4, 8 or 16, or flagged. For each block, if the movement associated with the block passes through an NxN block in the current image and the block has not been assigned any interpolated movement, the movement of the reference block is scaled to the current image in the same way as that of the reference block scaling. MV in TMVP and staggered motion is assigned to the block in the current frame. If no scaled MV is assigned to an NxN block, the block's motion is marked as unavailable in the interpolated motion field. In another example, an NxN block in the current image can be assigned multiple motion vectors in the interpolated motion field.

[00200] In some cases, a video encoder may remove one or more candidates from a candidate list. Pruning can be applied to remove identical entries from an initial list before the best starting point selection process, for example, before calculating the matching cost for each candidate on the list.

[00201] In some cases, the first image in each reference list may be used as the reference image, and motion vector candidates are scaled accordingly if necessary. In this case, the reference index of each entry in the initial list can be modified after the motion vector is scaled based on, for example, the distance from POC, similar to TMVP. In some cases, the reference index of each input can be fixed to one or two images and the associated motion vectors can be scaled to those images.

[00202] In an example, for bilateral matching, a motion vector pair, which is a complete motion information set that contains both motion vectors and their reference indices associated with reference list 0 and reference list 1, respectively, can be obtained based on each entry from an initial candidate list. A video encoder can then perform two-sided matching for all MV pairs generated from all entries in the initial list and select the candidate that leads to the minimum matching cost.

[00203] To generate the MV pair, the video encoder may select an entry from the initial list as the first motion vector and generate the second motion vector. For example, suppose an input contains the first current motion vector MVA and is associated with a reference index (with POC value POCA) to a first reference image in the reference image list A (with A being equal to 0 or 1). Assuming that the POC value of the current image is POCC, the video encoder can be configured to find a second reference image from the reference image list B (with B being equal to 1-A) so that its value of POC POCB is equal to (2 X POCC - POCA). If no reference picture in reference picture list B has a POC value equal to (2 X POCC-POC0), the video encoder can select the second reference picture by checking all reference pictures in list B so that the POCB is not a POCA and the absolute value of POCC-POCB is the minimum. In short, the video encoder can select an image located on the other side of the current image (in display order) that has the same POC distance. If not available, the video encoder may select an image on the other side that has the shortest distance to the current image. If all reference images are on the same temporal side as the first reference with POCA compared to the current image, the video encoder can select the reference that is temporally closest to the current image and has a POC other than POCA. aforementioned, the video encoder can scale the first MVA motion vector to generate the second motion vector associated with the second reference image, for example, based on the POC difference as in TMVP. Alternatively, any techniques based on the bilateral MV assumption can be used to generate the second reference image and its motion vector.

[00204] In some aspects, two initial motion vectors may be selected from the initial list. Each of these two motion vectors is selected (with respect to a minimum matching cost) from the entries in two subsets of the initial list. Each subset contains motion vectors associated with the reference index only reference image list 0 or only reference image list 1.

[00205] In accordance with aspects of this disclosure, a video encoder may be configured to select a candidate from a candidate based on a matching cost associated with the motion vector. Furthermore, after selecting a candidate from a candidate list, the video encoder can refine the candidate motion vector to derive motion information for a block. For example, the candidate motion vector can be used to indicate a starting point of a reference image, which can then be searched to find a best match based on a matching cost.

[00206] In accordance with aspects of this disclosure, a variety of matching costs may be used, for example, when determining a best match for an initial motion vector candidate and/or when refining the motion vector candidate initial. In an example, when searching for the movement of a block, an MV cost may be included in the matching cost to avoid negative impact due to noise. For example, the refinement portion of the motion vector (e.g., difference between the current MV and the search center), denoted as MVR can be used to calculate the cost. In this example, the cost might be w*(|MVR[0]|+|MVR[1]|), where w is a weighting factor that can be signed or predefined and MVR[0] and MVR[1] are the two MVR components. Alternatively, the refined motion vector MV can be used to calculate the cost, for example as w*(|MV[0]|+|MV[1]|).

[00207] In some examples, when the block is relatively small, e.g., 4 x 4 or smaller, a larger block that covers the block (e.g., the block with extended boundaries) may be used in the matching cost calculation in order to to suppress noise. For example, when searching for the best match for a 4 x 4 block, the matching cost can be calculated based on the 8 x 8 block with a center block being the block.

[00208] In some examples, the matching cost can be any type of distance/cost, such as sum of absolute differences (SAD), sum of squared prediction errors (SSE), or sum of absolute transform differences (SATD). . To reduce computational complexity the SAD, SSE, or other cost can be calculated with reduced resolution in the horizontal, vertical, or both directions. For example, for an 8 x 8 block, the SAD can be calculated based on odd rows only. In another example, the matching cost can be calculated based on a selected subset of a block, for example, only a central region of a can be used.

[00209] In accordance with aspects of this disclosure, the refinement process for the best match (e.g., candidate selected based on matching cost) may be performed within a predefined or flagged search window rather than always with the use of a small window, for example, within a 2 x 2 window (in the present context the unit is pixel and fractional motion vectors can be searched within a window) to achieve a low complexity but effective search. In this example, the range of a search window (e.g., with a size of 16 x 16) may be predefined or signaled in the bitstream.

[00210] The search algorithm for searching can be predefined, such as a full search, a three-step search, a diamond search, a block-based gradient descent search (BBGDS) algorithm as described, for example, in Lurng -Kuo Liu, Ephraim Feig, “A block-based gradient descent search algorithm for block motion estimation in video coding,” IEEE Trans. Circuits Syst. Video Technol. , vol. 6, pages 419 to 422, August 1996, or an unconstrained near-center diamond search (UCBDS) algorithm as described, for example, in Jo Yew Tham, Surendra Ranganath, Maitreya Ranganath, and Ashraf Ali Kassim, “A novel unrestricted center-biased diamond search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol. , vol. 8, pages 369 to 377, August 1998.

[00211] In some cases, different search techniques may be used in different cases based on predefined flagging or criteria. For example, to search an entire PU, a diamond search can be used. To search for a sub-PU, a small diamond search can be used. Additionally or alternatively, early stopping can be applied during search, for example, when the matching cost is below adaptive or predefined thresholds. When using template matching, a constraint can be additionally applied that the two motion vectors of the two reference lists after refinement will not point to the same block in the same reference image.

[00212] As noted above, this disclosure describes certain optimization techniques for existing DMVD processes. For example, as described above in relation to the example in Figure 10, techniques include extending two-way matching from two-way prediction to one-way prediction. Bilateral matching can also be applied when the temporal positions of two reference images are before or after the temporal position of the current slice (i.e., the POC values are smaller or larger than that of the current image). These techniques may be collectively referred to herein as extended bilateral matching.

[00213] Other aspects of this disclosure relate to interpolation techniques. For example, in accordance with aspects of this disclosure, interpolation techniques can be simplified to reduce complexity. As noted above in relation to Figure 2, motion search is typically performed using subpixel precision. Consequently, interpolation is required for non-integer pixel positions. To reduce computational complexity, in accordance with aspects of this disclosure, a video encoder may use an interpolation filter with shorter taps compared to normal motion compensation interpolation. In one example, the video encoder may use a bilinear interpolation filter during motion search, for example, when applying initial candidate motion vectors or refining such motion vectors. In another example, the video encoder may be configured to initially perform motion search using integer-pixel positions, followed by performing motion search on fractional pixel positions with the best integer-pixel position match. as the starting point.

[00214] Other techniques of this disclosure relate to deriving motion information for subblocks. For example, according to aspects of this disclosure, a video encoder may divide a block/PU into (non-overlapping) sub-PUs/sub-blocks based on any motion derivation technique. When a PU is divided into multiple sub-PUs of smaller size, the video encoder can derive a set of motion information unique to each sub-PU.

[00215] In an example for illustration purposes, a 32x32 PU can be divided into 16 8 x 8 sub-PUs. In this example, the video encoder can determine different reference indices and/or motion vectors for each of the sub-PUs. -8 x 8 PUs. In other examples, sub-PUs can have other sizes, for example, 4 x 4, 2 x 2, or 1 x 1.

[00216] In some cases, the sub-block/sub-PU size may be predefined and fixed independently of the block/PU size. In other examples, a split depth D for PUs can be defined, which controls the number of times a PU can be split according to a quadratic tree structure. In some examples, a minimum sub-PU/sub-block size may be predefined or signaled to indicate the target sub-block/sub-PU size to which the current block/PU should be split. The target size can be the greater of the minimum sub-PU/sub-block size and the size obtained by dividing the current block D times according to a quadratic tree structure.

[00217] According to aspects of this disclosure, a video encoder can take advantage of motion information derived for a PU when deriving motion information for sub-PUs of the PU. For example, the video encoder may, for each sub-PU, search for respective unique motion information by setting the motion information of the entire block as the search center (initial search point). The video encoder can then refine the motion for each sub-PU. Alternatively, the search center of each sub-PU can be derived from a list of starting point candidates.

[00218] In another example, a list of motion vector candidates can be generated for the sub-PU using any of the techniques described herein. The video encoder can then check each candidate in the list after checking the search center (e.g., the initial search point derived from the PU). According to aspects of this disclosure, sub-PUs can be refined using any of the techniques described herein as being applicable to regular blocks. In other examples, refinement may always or conditionally be skipped for sub-PUs after checking the list of motion vector candidates for a better match in the manner described above. One possible condition may be whether the best match remains the candidate indicated by the PU movement information (e.g., the initial search point) after checking the candidate list. If true, the video encoder can skip refinement. Otherwise, the video encoder can perform the refinement.

[00219] In some examples, motion vector filtering may be performed for sub-PU in order to correct an isolated erroneous motion vector. For example, a video encoder may use a median filter with the motion vector of the current sub-PU and motion vectors of up to four neighboring sub-PUs. According to aspects of this disclosure, when applying transforms, the video encoder may consider the entire PU as an entire block so that a transform may cross sub-PU boundaries. In other examples, the video encoder may apply transforms to each sub-PU such that the transform size is not greater than the sub-PU size. In some cases, when template matching is used, a complete PU can also be further divided into smaller sub-PUs. For sub-PUs whose spatial neighbors are all in the current PU (their models are not available), the video encoder can set their motion vectors to the derived motion vectors for the entire PU.

[00220] In some examples, a video encoder may encode separate syntax elements that indicate whether a particular block is split. In another example, all 2Nx2N blocks with extended bilateral matching mode are additionally divided into small partitions and no additional flags are signaled. The subblock size can be predefined or signaled. Alternatively, the sub-block size can be based on the current block size. For example, the subblock size can be derived as the larger of two values. The first value is a predefined or minimum flagged subblock size (for example, such as 4x4 or 8x8). The second value is a size relative to the current block, such as (S»d)x(S»d) where SxS is the current block size, while d is a predefined or signed value to indicate the quadratic tree depth with the which the current block is divided into sub-blocks. In the above examples, it should be understood that the term PU is used interchangeably with the term block and the term sub-PU is used interchangeably with the term sub-block.

[00221] As noted above, the techniques described in this document can be used independently or in combination. As an example, a motion information derivation process associated with a motion information derivation mode (e.g., an extended bilateral matching mode, a template matching mode, or any other mode) may include three steps, although The third step can be conditionally performed based on slice type, time level, block type, block size, or syntax defined in the bitstream.

[00222] In a first step, a video encoder may form an initial candidate list of unipredicted sets of motion information for a current block. The video encoder can select the candidate who has the best starting point from the list, for example, based on a matching cost. In a second step, the video encoder can refine the selected candidate to derive the motion information for the current block. Refinement may also be based on a matching cost, as described herein. The video encoder may then optionally further divide the current block into sub-blocks as described herein, and refine each motion vector for each sub-block. Finally, the video encoder can apply motion compensation to the current block using the derived motion information. If division has been performed, the video encoder can apply motion compensation on a sub-block by sub-block basis.

[00223] In an example, the following pseudocode can be used to reconstruct a block using a motion information derivation mode, wherein the motion information is derived by bilateral matching or template matching. When two-sided matching is used, the block is further divided into sub-blocks (note that the MV in the present context includes reference information): RECONSTRUCT BLOCK B (WITH SIZE WXH) IN FRUC Build initial list if B is two-sided matching { Find the best match (e.g. two-sided match) in the starting list as the starting point with two-sided matching measurement Refine the MV based on the starting point with two-sided matching to obtain the MVB motion vector for block B for each sub-block in block B { use MVB as the starting point, refine MV for each sub-block do motion compensation for the sub-block with the derived MV information } } else // template match { Find the best match (e.g. template match) in the starting list as the starting point with template matching measurement Refine the MV based on the starting point with template matching }

[00224] Therefore, in accordance with aspects of this disclosure, the video encoder 20 or video decoder 30 may determine a motion information interpolation mode to determine motion information for a current block (a block currently being encoded or decoded). Using motion information interpolation mode (e.g., bilateral matching, template matching, or other technique), video encoder 20 or video decoder 30 can determine the best candidate motion vector from a list of candidate motion vectors. motion vector, for example, the motion vector that identifies a reference block that closely corresponds to the current block. The video encoder 20 or video decoder 30 may use the candidate motion vector to identify a search window in a reference image.

[00225] Video encoder 20 or video decoder 30 may refine the candidate motion vector based on a reference block in the search window that entirely matches the current block. That is, the video encoder 20 or video decoder 30 may determine a new interpolated motion vector for the current block based on the movement between the reference block in the search window that closely corresponds to the current block and the current block. The video encoder 20 or video decoder 30 may then perform motion compensation for the current block using the interpolated motion vector.

[00226] In some cases, video encoder 20 or video decoder 30 may divide the current block into more than one sub-block for prediction purposes. Furthermore, in other examples, video encoder 20 or video decoder 30 may perform more, less, or a different arrangement of techniques for interpolating motion information.

[00227] Therefore, certain techniques from this disclosure can be generalized as a block-level coding tool that takes advantage of certain FRUC concepts, given the assumption that a current block of a current image can be considered as predicted by the reference images similarly to how a current image can be considered as interpolated by reference images in FRUC. In one example, only motion-based processes are used for the block-level encoding tool. In another example, only pixel-based processes are used for the block-level encoding tool. In another example, motion-based processes or pixel-based processes are used for a given block. In another example, both pixel-based processes and motion-based processes are used for the block-level encoding tool. In another example, other syntax can be reused or predicted from the other time frames and can be used for the encoding tool, such as encoding tree information, SAO, ALF, RQT.

[00228] Figure 12 is a flowchart illustrating an exemplary process for determining a mode of deriving motion information for encoding a block of video data. The exemplary process of Figure 12 is described with respect to a video encoder, which may be configured as a video encoder 20, video decoder 30, or other processor.

[00229] In the example of Figure 12, the video encoder may select a motion derivation mode from a plurality of motion derivation modes (200). In general, each of the motion derivation modes may include performing a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block. For example, in connection with template matching, the video encoder may perform a motion search to identify a template in a current image (e.g., a first set of reference data) that matches the template in a reference image ( for example, a second set of reference data). In another example, in connection with bilateral motion, the video encoder may perform a motion search to identify a reference block in a first reference image (e.g., a first reference data set) that corresponds to a second block reference on a second reference image (e.g., a second reference dataset). Exemplary motion derivation modes may include, a unilateral motion estimation mode, a bilateral matching mode, a template matching mode, or a mirror-based mode.

[00230] In some aspects, the video encoder may select the mode of deriving motion information according to one or more syntax elements included in a bit stream. For example, a video decoder may parse and decode the one or more syntax elements from the bit stream and determine how to derive motion information based on the syntax. A video encoder may test the plurality of motion information derivation modes, select the mode that has the best RD cost, and encode the one or more syntax elements in the bit stream indicating the selected mode.

[00231] The video encoder may determine motion information for the block using the selected motion derivation mode, which may include determining initial motion information using the selected mode (202) and using an optimization process to refine initial motion information (204). For example, the video encoder may construct a motion vector candidate list with candidates to determine initial motion information. The initial motion information may provide a starting point for refining the motion information as described herein.

[00232] The video encoder can then encode the block using the determined motion information and without encoding the syntax representing the motion information (206). For example, in cases where the video encoder comprises a video decoder, the video decoder may determine a reference block in a reference image based on the determined motion information, decode residual data for each sub-block from of an encoded bitstream and combine the given reference block and decoded residual to reconstruct the current block. In cases where the video encoder comprises a video encoder, the video encoder may encode residual data for the current block into an encoded bit stream without coding reference indices or motion vectors for the residual data.

[00233] Figure 13 is a flowchart illustrating an exemplary process for deriving a motion vector for encoding a block of video data. The exemplary process of Figure 13 is described with respect to a video encoder, which may be configured as a video encoder 20, video decoder 30, or other processor.

[00234] The video encoder may generate a list of motion vector candidates (210). The list of candidate motion vectors may include one or more motion vector candidates that may be used to derive motion information for the current block. In some examples, motion vector candidates may be determined from spatially neighboring blocks, temporal blocks, or from other locations.

[00235] The video encoder may determine a candidate from the list to derive motion information (212). In some examples, the video encoder may perform one or more cost calculations to determine the candidate using a particular motion derivation mode. For example, the video encoder may determine a matching cost for a first set of reference data and a second set of reference data, which may include a cost associated with respective motion vectors, as described herein.

[00236] The video encoder can then determine the derived motion vector based on the determined candidate (214). For example, the video encoder may refine the given candidate to determine the derived motion vector using the motion derivation mode. The video encoder can then encode the block using the derived motion vector (216). For example, in cases where the video encoder comprises a video decoder, the video decoder may determine a reference block in a reference image based on the derived motion vector, decode residual data for each sub-block from of an encoded bitstream and combine the given reference block and decoded residual to reconstruct the current block. In cases where the video encoder comprises a video encoder, the video encoder may encode residual data for the current block into an encoded bitstream without encoding the derived motion vector.

[00237] Figure 14 is a flowchart illustrating an exemplary process for deriving motion information for sub-blocks of a video data block. The exemplary process of Figure 14 is described with respect to a video encoder, which may be configured as a video encoder 20, video decoder 30, or other processor.

[00238] The video encoder can derive motion information for a current block (220). In some examples, the video encoder may use any combination of motion information derivation techniques described herein to derive motion information. For example, the video encoder may use any of the techniques described herein to perform a motion search for a first set of reference data that corresponds to a second set of reference data outside the current block (e.g., model, bilateral correspondence or similar).

[00239] In accordance with aspects of this disclosure, the video encoder may also divide the block into a plurality of sub-blocks (222). The video encoder may separately derive motion information for respective sub-blocks comprising a motion search for a first set of reference data corresponding to a second set of reference data outside each respective sub-block (224). Again, the video encoder may use any techniques described herein to derive the motion information such as, for example, template matching, bilateral matching, or the like. For example, the video encoder may use the derived motion vector as a starting point to derive motion information for each of the sub-blocks, and may further refine the derived motion information using any combination of the derivation techniques. movement information described in this document.

[00240] The video encoder may then encode each of the sub-blocks based on motion information derived without coding syntax elements representative of the motion information (226). For example, in cases where the video encoder comprises a video decoder, the video decoder may determine a reference block in a reference image for each sub-block based on the determined motion information, decode residual data for each sub-block from an encoded bit stream and combine the given reference block and decoded residual to reconstruct each sub-block. In cases where the video encoder comprises a video encoder, the video encoder may encode residual data for each sub-block into an encoded bit stream without coding reference indices or motion vectors for the residual data.

[00241] It should be recognized that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left out entirely (e.g., not all actions and events described are necessary for the practice of the techniques). Furthermore, in certain examples, actions or events may be performed concurrently, for example, through multiple thread processing, interruptive processing or on multiple processors, rather than sequentially.

[00242] In one or more examples, the described functions can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, functions can be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, which include any media that facilitate the transfer of a computer program from one place to another , for example, according to a communication protocol. In this way, computer-readable media can generally correspond to (1) tangible computer-readable storage media that are non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, data structures and/or code for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

[00243] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, memory flash, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is appropriately termed as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscription line (DSL), or wireless technologies such as infrared , radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. . Magnetic disc and optical disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where magnetic discs generally reproduce data in a magnetic, while optical discs reproduce data optically with lasers. Combinations of the above shall also fall within the scope of computer-readable media.

[00244] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or another set of integrated or distinct equivalent logic circuits. Consequently, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined codec. Furthermore, the techniques can be fully implemented in one or more circuits or logic elements.

[00245] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including a handset, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require performance by different hardware units. Instead, as described above, multiple units may be combined into a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above, in combination with suitable software and/or firmware. .

[00246] Several examples have been described. These and other examples are within the scope of the following claims.

Claims (12)

1. Method for processing video data, the method characterized by the fact that it comprises: dividing a current block of video data of a current image into a plurality of sub-blocks to derive motion information from the current block, wherein the information of motion indicate movement of the current block with respect to reference video data, and where splitting the current block comprises i) splitting the current block to form sub-blocks of a fixed size independent of the size of the current block, or ii) splitting the current block based on a minimum subblock size determined based on a split depth parameter D that indicates a maximum split depth for the current block; derive motion information for the entire current block by performing a motion search for a respective first set of reference video data from a reference image based on a respective second set of reference video data, the respective second set of reference video data being outside the current block, wherein the second set of reference video data is from the current image or a second reference image; deriving, separately for each respective sub-block among the plurality of sub-blocks, motion information, the derivation comprising performing a motion search for a respective third set of reference video data from a reference image based on a respective fourth reference video data set, the respective fourth reference video data set being outside the respective sub-block, and wherein the search is performed by setting the motion information for the entire current block as the search center and then refine the movement for each sub-block; apply a transform to the current block so that the transform crosses sub-block boundaries; and decoding the plurality of sub-blocks based on the derived motion information and without decoding syntax elements representative of the motion information. 2. Method according to claim 1, characterized by the fact that the movement information for each of the sub-blocks comprises at least one movement vector and at least one reference index of a reference image in a list of reference image. 3. Method according to claim 1, characterized by the fact that the size of the sub-blocks comprises at least one of 1x1, 2x2, 4x4, 8x8, 16x16 or 32x32. 4. Method according to claim 1, characterized by the fact that the current block comprises a prediction unit, PU, and the sub-blocks comprise subPUs. 5. Method, according to claim 1, characterized by the fact that deriving the movement information for the respective sub-blocks comprises: searching for a respective best match for the respective sub-blocks within a search window; and derive the movement information for the respective sub-blocks based on the respective best match. 6. The method of claim 5, further comprising: generating a candidate list comprising a plurality of candidate motion vectors, the candidate list including the derived motion information for the block as a candidate of the plurality of candidate motion vectors; determining respective candidate motion vectors from the list of candidates for respective sub-blocks; and wherein searching for the respective best match comprises searching based on the respective candidate motion vectors. 7. Method according to claim 1, characterized by the fact that it further comprises applying a transform to the current block so that the transform is applied to all sub-blocks. 8. The method of claim 1, wherein splitting the current block comprises splitting the current block based on one or more syntax elements obtained from an encoded bit stream, and decoding the plurality of sub-blocks comprises: determining a reference block in a reference image for the respective sub-blocks based on the derived motion information; decoding residual data to respective subblocks from an encoded bit stream; and combining the decoded residue for the respective sub-blocks and the reference block for the respective sub-blocks to reconstruct the respective sub-blocks. 9. Method for processing video data, the method characterized by the fact that it comprises: dividing a current block of video data of a current image into a plurality of sub-blocks to derive motion information from the current block, wherein the information of motion indicate movement of the current block with respect to reference video data, and where splitting the current block comprises i) splitting the current block to form sub-blocks of a fixed size independent of the size of the current block, or ii) splitting the current block based on a minimum subblock size determined based on a split depth parameter D that indicates a maximum split depth for the current block; derive motion information for the entire current block by performing a motion search for a respective first set of reference video data from a reference image based on a respective second set of reference video data, the respective second set of reference video data being outside the current block, wherein the second set of reference video data is from the current image or a second reference image; deriving, separately for each respective sub-block among the plurality of sub-blocks, motion information, the derivation comprising performing a motion search for a respective third set of reference video data from a reference image based on a respective fourth reference video data set, the respective fourth reference video data set being outside the respective sub-block, and wherein the search is performed by setting the motion information for the entire current block as the search center and then refine the movement for each sub-block; apply a transform to the current block so that the transform crosses sub-block boundaries; and encoding the plurality of sub-blocks based on the derived motion information and without encoding syntax elements representative of the motion information. 10. Device for processing video data, the device characterized in that it comprises: a memory configured to store a current block of video data; and one or more processors configured to: divide a current block of video data into a plurality of sub-blocks to derive motion information from the current block, wherein the motion information indicates motion of the current block with respect to video data of reference, and wherein splitting the current block comprises i) splitting the current block to form sub-blocks of a fixed size independently of the current block size, or ii) splitting the current block based on a determined minimum sub-block size based on a split depth parameter D that indicates a maximum split depth for the current block; derive motion information for the entire current block by performing a motion search for a respective first set of reference video data from a reference image based on a respective second set of reference video data, the respective second set of reference video data being outside the current block, wherein the second set of reference video data is from the current image or a second reference image; deriving, separately for each respective sub-block among the plurality of sub-blocks, motion information, wherein to derive the motion information the one or more processors are configured to perform a motion search for a respective third set of motion data. reference video based on a respective fourth set of reference video data outside each respective sub-block, and in which the search is performed by setting the motion information for the entire current block as the search center and then refining the movement for each sub-block; apply a transform to the current block so that the transform crosses sub-block boundaries; and encoding the plurality of sub-blocks based on the derived motion information and without encoding syntax elements representative of the motion information. 11. Device according to claim 10, characterized by the fact that the device comprises at least one of: an integrated circuit; a microprocessor; or a wireless communication device. 12. Computer-readable memory characterized by the fact that it comprises instructions stored therein that, when executed, cause a computer to perform the method as defined in any one of claims 1 to 9.