KR20210109049A - Optical flow estimation for motion compensated prediction in video coding - Google Patents

Abstract

An optical flow reference frame portion (eg, a block or an entire frame) is generated that can be used for inter prediction of blocks of the current frame in the video sequence. A forward reference frame and a backward reference frame are used for light flow estimation to generate respective motion fields for pixels in the current frame. Motion fields are used to warp some or all pixels of reference frames to pixels of the current frame. The warped reference frame pixels are blended to form a lightflow reference frame portion. Inter prediction may be performed as part of encoding or decoding portions of the current frame.

Description

OPTICAL FLOW ESTIMATION FOR MOTION COMPENSATED PREDICTION IN VIDEO CODING

[0001] Digital video streams may represent video using a series of frames or still images. Digital video may be used in a variety of applications including, for example, video conferencing, high-definition video entertainment, video advertisements, or sharing of user-generated videos. A digital video stream may contain large amounts of data and may consume significant amounts of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including compression and other encoding techniques.

[0002] One technique for compression uses a reference frame to generate a predictive block corresponding to the current block to be encoded. Differences between the prediction block and the current block may be encoded instead of the values of the current block itself to reduce the amount of encoded data.

[0003] FIELD OF THE INVENTION The present invention relates generally to encoding and decoding video data, and more particularly to the use of block-based optical flow estimation for motion compensation prediction in video compression. Frame-level based lightflow estimation, which can interpolate collocated reference frames for motion compensation prediction in video compression, is also described.

[0004] The present invention describes encoding and decoding methods and apparatus. A method according to an embodiment of the present invention comprises the steps of determining a first frame portion of a first frame to be predicted, wherein the first frame is in a video sequence, the video for forward inter prediction of the first frame. Determining a first reference frame from a sequence, determining a second reference frame from a video sequence for backward inter prediction of the first frame, light flow using the first reference frame and the second reference frame generating an optical flow reference frame portion for inter prediction of the first frame portion by performing optical flow estimation, and performing a prediction process on the first frame portion using the optical flow reference frame portion includes The first frame portion and the light flow reference frame portion may be, for example, a block or an entire frame.

[0005] An apparatus according to an embodiment of the present invention includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform the method, the method comprising: determining a first frame to be predicted in a video sequence; and availability of a first reference frame for forward inter prediction of the first frame; and determining availability of a second reference frame for backward inter prediction of the first frame. The method also includes, in response to determining availability of both the first reference frame and the second reference frame, using the first reference frame and the second reference frame as inputs to an optical flow estimation process to: generating a respective motion field for pixels of , warping a first reference frame portion into a first frame portion using the motion fields to form a first warped reference frame portion—a first reference frame the portion comprises pixels of a first reference frame juxtaposed with pixels of the first frame portion, wherein the second reference frame portion is converted to the first frame portion using the motion fields to form a second warped reference frame portion. warping with , wherein the second reference frame portion includes pixels of the second reference frame juxtaposed with pixels of the first frame portion, and a lightflow reference frame for inter prediction of at least one block of the first frame. and blending the first warped reference frame portion and the second warped reference frame portion to form the portion.

[0006] Another device according to an embodiment of the present invention also includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform the method, the method comprising: initializing motion fields for pixels of a first frame portion at a first processing level for lightflow estimation, the first processing level comprising: representing downscaled motion within the first frame portion and including one of a plurality of levels, wherein the first reference frame portion of the video sequence is used using a first reference frame portion from the video sequence and a second reference frame portion of the video sequence. generating a lightflow reference frame portion for inter prediction of a block of frame, and for each of the plurality of levels, a first reference frame using the motion fields to form a first warped reference frame portion warping a portion into a first frame portion; warping a second reference frame portion into a first frame portion using motion fields to form a second warped reference frame portion; estimating motion fields between the warped reference frame portion and the second warped reference frame portion, and using the motion fields between the first warped reference frame portion and the second warped reference frame portion updating the motion fields for the pixels. The method also includes, for a last level of the plurality of levels: warping the first reference frame portion into the first frame portion using the updated motion fields to form a final first warped reference frame portion, a final warping the second reference frame portion into the first frame portion using the updated motion fields to form a second warped reference frame portion of blending the reference frame portion and the second warped reference frame portion.

[0007] Another device according to an embodiment of the present invention also includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform the method, the method comprising: determining a first frame portion of a first frame to be predicted, wherein the first frame is within a video sequence, in a forward direction of the first frame determining a first reference frame from the video sequence for inter prediction, determining a second reference frame from the video sequence for backward inter prediction of the first frame, optical using the first reference frame and the second reference frame generating an optical flow reference frame portion for inter prediction of the first frame portion by performing flow estimation, and performing a prediction process on the first frame portion using the optical flow reference frame portion.

[0008] These and other aspects of the invention are set forth in the following detailed description of the embodiments, the appended claims, and the accompanying drawings.

DETAILED DESCRIPTION [0009] The description of the present specification refers to the accompanying drawings set forth below, wherein like reference numerals refer to like parts throughout, unless otherwise stated.
1 is a schematic diagram of a video encoding and decoding system;
2 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station.
3 is a diagram of an exemplary video stream to be encoded and subsequently decoded;
4 is a block diagram of an encoder according to implementations of the present invention;
5 is a block diagram of a decoder according to implementations of the present invention;
6 is a block diagram of an example of a reference frame buffer.
7 is a diagram of a group of frames in a display order of a video sequence.
FIG. 8 is a diagram of an example of a coding order for the group of frames of FIG. 7 .
9 is a diagram used to illustrate a linear projection of a motion field in accordance with the teachings herein.
10 is a flow diagram of a process for motion compensated prediction of a video frame using at least a portion of a reference frame generated using optical flow estimation.
11 is a flowchart of a process for generating a lightflow reference frame portion;
12 is a flowchart of another process for generating a lightflow reference frame portion;
13 is a diagram illustrating the processes of FIGS. 11 and 12 .
14 is a diagram illustrating object occlusion.
15 is a diagram illustrating a technique for optimizing a decoder;

[0025] A video stream may be compressed by various techniques to reduce the bandwidth required to transmit or store the video stream. The video stream can be encoded into a bitstream with compression, which is then sent to a decoder, which can decode or decompress the video stream and prepare it for viewing or further processing. Compression of a video stream often utilizes spatial and temporal correlation of video signals through spatial and/or motion compensated prediction. Inter-prediction uses one or more motion vectors to generate a block (also called a prediction block) similar to the current block to be encoded, for example using previously encoded and decoded pixels. By encoding the motion vector(s) and the difference between the two blocks, a decoder receiving the encoded signal can regenerate the current block. Inter prediction may also be referred to as motion compensated prediction.

[0026] Each motion vector used to generate a prediction block in the inter prediction process may refer to a frame other than the current frame, that is, a reference frame. The reference frames may be located before or after the current frame in the sequence of the video stream, and may be frames reconstructed before being used as the reference frame. In some cases, there may be three reference frames used to encode or decode blocks of a current frame of a video sequence. One is a frame that may be referred to as a golden frame. The other is the most recently encoded or decoded frame. The last one is an alternate reference frame that is encoded or decoded before one or more frames in the sequence but displayed after these frames in the output display order. In this way, the replacement reference frame is a reference frame usable for backward prediction. One or more forward and/or backward reference frames may be used to encode or decode a block. The validity of a reference frame when used to encode or decode a block within the current frame may be measured based on the resulting signal-to-noise ratio or other measures of rate-distortion.

[0027] In this technique, the pixels forming the predictive blocks are obtained directly from one or more of the available reference frames. Reference pixel blocks or their linear combinations are used for prediction of a given coding block in the current frame. This direct block-based prediction does not capture the true motion activity available from reference frames. For this reason, motion compensation prediction accuracy may be degraded.

[0028] To more fully utilize motion information from available bidirectional reference frames (eg, one or more forward and one or more backward reference frames), implementations of the teachings herein estimate true motion activities in a video signal. We describe the reference frame portions juxtaposed with the current coding frame portions using the per-pixel motion field calculated by the light flow to Reference frame portions are interpolated, allowing tracking of complex non-translational motion activity beyond the capabilities of conventional block-based motion compensated prediction determined directly from reference frames. The use of these reference frame portions can improve the prediction quality. As used herein, a frame portion refers to some or all of a frame, such as a block, slice, or entire frame. A frame portion of one frame is juxtaposed with a frame portion of another frame if this frame portion and the frame portion of another frame have the same dimensions and are at the same pixel positions within the dimensions of each frame.

[0029] Additional details of using optical flow estimation to interpolate reference frame portions for use in video compression and reconstruction are set forth herein with initial reference to a system in which the teachings herein may be implemented.

[0030] 1 is a schematic diagram of a video encoding and decoding system 100 . The transmitting station 102 may be, for example, a computer having an internal configuration of hardware as described in FIG. 2 . However, other suitable implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 may be distributed among multiple devices.

[0031] The network 104 may connect the transmitting station 102 and the receiving station 106 for encoding and decoding of video streams. Specifically, the video stream may be encoded at the transmitting station 102 and the encoded video stream may be decoded at the receiving station 106 . Network 104 may be, for example, the Internet. The network 104 may also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or in this example from a transmitting station 102 . It may be any other means of transmitting the video stream to the receiving station 106 .

[0032] Receiving station 106 may be, in one example, a computer having an internal configuration of hardware as described in FIG. 2 . However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 may be distributed among multiple devices.

[0033] Other implementations of the video encoding and decoding system 100 are possible. For example, one implementation may omit the network 104 . In another implementation, the video stream may be encoded and then stored for later transmission to the receiving station 106 or any other device having a non-transitory storage medium or memory. In one implementation, the receiving station 106 receives the encoded video stream (eg, over the network 104, computer bus, and/or some communication path) and stores the video stream for later decoding. . In an example implementation, Real Time Transport Protocol (RTP) is used for transmission of encoded video over network 104 . In other implementations, a transport protocol other than RTP may be used, for example, a video streaming protocol based on Hypertext Transfer Protocol (HTTP).

[0034] When used in a videoconferencing system, for example, the transmitting station 102 and/or the receiving station 106 may include the ability to encode and decode a video stream as described below. For example, receiving station 106 receives, decodes and views an encoded video bitstream from a video conferencing server (eg, transmitting station 102) and also encodes its own video bitstream to allow other participants to participate. may be a video conference participant who transmits to a video conference server for decoding and viewing by

[0035] 2 is a block diagram of an example of a computing device 200 that may implement a transmitting station or a receiving station. For example, computing device 200 may implement either or both of transmit station 102 and receive station 106 of FIG. 1 . Computing device 200 may be in the form of a computing system including multiple computing devices, or in the form of a single computing device, eg, a mobile phone, tablet computer, laptop computer, notebook computer, desktop computer, and the like.

[0036] The CPU 202 in the computing device 200 may be a central processing unit. Alternatively, the CPU 202 may be any other type of device or multiple devices capable of manipulating or processing information that currently exists or is developed in the future. Although the disclosed implementations may be implemented with one processor, eg, CPU 202 as shown, advantages in speed and efficiency may be achieved using two or more processors.

[0037] Memory 204 of computing device 200 may be a read only memory (ROM) device or a random access memory (RAM) device in implementations. Any other suitable type of storage device or non-transitory storage medium may be used as the memory 204 . Memory 204 may contain code and data 206 that is accessed by CPU 202 using bus 212 . The memory 204 may further include an operating system (OS) 208 and application programs 210 that enable the CPU 202 to perform the methods described herein. at least one program that For example, application programs 210 may include applications 1 to N, wherein applications 1 to N further include a video coding application for performing the methods described herein. Computing device 200 may also include secondary storage 214 , which may be, for example, a memory card used with a mobile computing device. Because video communication sessions may contain a significant amount of information, they may be stored in whole or in part in secondary storage 214 and loaded into memory 204 as needed for processing.

[0038] Computing device 200 may also include one or more output devices, such as display 218 . Display 218 may, in one example, be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch inputs. Display 218 may be coupled to CPU 202 via bus 212 . Other output devices that allow a user to program or otherwise use computing device 200 may be provided in addition to or in place of display 218 . Where the output device is or comprises a display, the display may be displayed in a variety of ways, including by a light emitting diode (LED) display, such as a liquid crystal display (LCD), a cathode ray tube (CRT) display, or an organic LED (OLED) display. can be implemented as

[0039] Computing device 200 may detect image sensing device 220 , eg, a camera, or any other image sensing currently in existence or developing in the future capable of sensing an image, such as an image of a user operating computing device 200 . It may also include or communicate with device 220 . The image sensing device 220 may be positioned to be directed towards a user manipulating the computing device 200 . In one example, the position and optical axis of the image sensing device 220 may be configured such that the field of view is immediately adjacent to the display 218 and includes an area in which the display 218 is visible.

[0040] Computing device 200 also includes or communicates with sound sensing device 222 , for example a microphone, or any other sound sensing device currently existing or developed in the future capable of sensing sound in the vicinity of computing device 200 . can do. The sound sensing device 222 may be positioned to be directed towards a user who is operating the computing device 200 , and sounds such as a voice or other utterances made by the user while the user is operating the computing device 200 . utterances).

[0041] 2 shows the CPU 202 and the memory 204 of the computing device 200 as integrated into one unit, other configurations may be used. The operations of CPU 202 may be distributed directly or across multiple machines (individual machines may have one or more processors), which may be coupled across a local area or other network. Memory 204 may be distributed across multiple machines, such as a network-based memory or memory of multiple machines performing operations of computing device 200 . Although shown here as one bus, the bus 212 of the computing device 200 may be comprised of multiple buses. In addition, secondary storage 214 may be coupled directly to other components of computing device 200 or may be accessed over a network and may include an integrated unit, such as a memory card, or multiple units, such as multiple memory cards. can do. Computing device 200 may thus be implemented in a variety of configurations.

[0042] 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302 . At the next level, the video sequence 302 includes a number of contiguous frames 304 . Although three frames are shown as contiguous frames 304 , the video sequence 302 may include any number of contiguous frames 304 . The adjacent frames 304 may then be further subdivided into individual frames, eg, frame 306 . At the next level, the frame 306 may be divided into a series of planes or segments 308 . Segments 308 may be, for example, a subset (subset) of frames that allow for parallel processing. Segments 308 may also be a subset of frames that may separate video data into distinct colors. For example, a frame 306 of color video data may include a luminance plane and two chrominance planes. Segments 308 may be sampled at different resolutions.

[0043] Irrespective of whether the frame 306 is divided into segments 308 , the frame 306 is divided into blocks 310 , which may contain data corresponding to, for example, 16×16 pixels of the frame 306 . can be further subdivided. Blocks 310 may also be arranged to include data from one or more segments 308 of pixel data. Blocks 310 may also be of any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or more. Unless stated otherwise, the terms block and macroblock are used interchangeably herein.

[0044] 4 is a block diagram of an encoder 400 in accordance with implementations of the present invention. The encoder 400 may be implemented at the transmitting station 102 , as described above, for example, by providing a computer software program stored in a memory, eg, the memory 204 . The computer software program may include machine instructions that, when executed by a processor, such as CPU 202 , cause transmitting station 102 to encode video data in the manner described in FIG. 4 . The encoder 400 may also be implemented as special hardware included, for example, in the transmitting station 102 . In one particularly preferred implementation, the encoder 400 is a hardware encoder.

[0045] The encoder 400 performs the following steps to perform various functions in a forward path (shown by solid connecting lines) to generate an encoded or compressed bitstream 420 using the video stream 300 as input. It has: an intra/inter prediction step 402 , a transform step 404 , a quantization step 406 , and an entropy encoding step 408 . The encoder 400 may also include a reconstruction path (shown by dashed connecting lines) to reconstruct a frame for encoding of future blocks. In FIG. 4 , the encoder 400 has the following steps to perform various functions in the reconstruction path: inverse quantization step 410 , inverse transform step 412 , reconstruction step 414 and loop filtering step 416 . Other structural variants of the encoder 400 may be used to encode the video stream 300 .

[0046] When the video stream 300 is presented for encoding, each frame 304 , such as frame 306 , may be treated in units of blocks. In the intra/inter prediction step 402 , each of the blocks may be encoded using either intra-frame prediction (also called intra prediction) or inter-frame prediction (also called inter prediction). In any case, a prediction block can be formed. In the case of intra prediction, a prediction block may be formed from samples of a previously encoded and reconstructed current frame. For inter prediction, a prediction block may be formed from samples of one or more previously constructed reference frames. The assignment of reference frames to groups of blocks is discussed in more detail below.

[0047] Next, still referring to FIG. 4 , the predictive block may be subtracted from the current block in an intra/inter prediction step 402 to generate a residual block (also referred to as a residual). Transform step 404 transforms the residual into transform coefficients in the frequency domain using, for example, block-based transforms. The quantization step 406 transforms the transform coefficients into discrete quantum values referred to as quantized transform coefficients using a quantizer value or quantization level. For example, the transform coefficients may be divided or truncated by the quantizer value. The quantized transform coefficients are then entropy encoded by an entropy encoding step 408 . The entropy encoded coefficients are then stored in the compressed bitstream ( 420) is output. The compressed bitstream 420 may be formatted using various techniques such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 may also be referred to as an encoded video stream or an encoded video bitstream, so these terms will be used interchangeably herein.

[0048] The reconstruction path of FIG. 4 (shown by dashed connecting lines) is used to ensure that encoder 400 and decoder 500 (described below) use the same reference frames to decode the compressed bitstream 420 . can be used The reconstruction path inversely quantizes the quantized transform coefficients in an inverse quantization step 410 to produce a derivative residual block (also called a derivative residual) and inverse transforms the inverse quantized transform coefficients in an inverse transform step 412 . performs functions similar to those occurring during the decoding process discussed in more detail below. In the reconstruction step 414, the prediction block predicted in the intra/inter prediction step 402 may be added to the derivative residual to generate a reconstructed block. A loop filtering step 416 may be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0049] Other variants of the encoder 400 may be used to encode the compressed bitstream 420 . For example, a non-transform-based encoder may quantize the residual signal directly without a transform step 404 for specific blocks or frames. In another implementation, the encoder may combine the quantization step 406 and the inverse quantization step 410 into a common step.

[0050] 5 is a block diagram of a decoder 500 in accordance with implementations of the present invention. The decoder 500 may be implemented at the receiving station 106 by, for example, providing a computer software program stored in the memory 204 . The computer software program may include machine instructions that, when executed by a processor, such as CPU 202 , cause receiving station 106 to decode video data in the manner described in FIG. 5 . The decoder 500 may also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106 .

[0051] The decoder 500 includes the following steps to perform various functions for generating an output video stream 516 from the compressed bitstream 420 in one example, similar to the reconstruction path of the encoder 400 discussed above. Do: entropy decoding step 502, inverse quantization step 504, inverse transform step 506, intra/inter prediction step 508, reconstruction step 510, loop filtering step 512, and deblocking filtering step ( 514). Other structural variants of the decoder 500 may be used to decode the compressed bitstream 420 .

[0052] When the compressed bitstream 420 is presented for decoding, the data elements in the compressed bitstream 420 may be decoded by an entropy decoding step 502 to produce a set of quantized transform coefficients. Inverse quantization step 504 inverse quantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantizer value), and inverse transform step 506 inverse transforms the inverse quantized transform coefficients to an encoder ( produce a derived residual, which may be identical to that produced by the inverse transform step 412 at 400). Using the header information decoded from the compressed bitstream 420 , the decoder 500 uses an intra/inter prediction step 508 in the encoder 400 , for example in an intra/inter prediction step 402 . It is possible to generate the same prediction block as generated. In the reconstruction step 510 , the prediction block may be added to the derivative residual to produce a reconstructed block. A loop filtering step 512 may be applied to the reconstructed block to reduce blocking artifacts.

[0053] Other filtering may be applied to the reconstructed block. In this example, a deblocking filtering step 514 is applied to the reconstructed blocks to reduce blocking distortion, and the result is output as an output video stream 516 . Output video stream 516 may also be referred to as a decoded video stream, so these terms will be used interchangeably herein. Other variants of the decoder 500 may be used to decode the compressed bitstream 420 . For example, the decoder 500 may generate the output video stream 516 without the deblocking filtering step 514 .

[0054] 6 is a block diagram of an example of a reference frame buffer 600 . The reference frame buffer 600 stores reference frames used to encode or decode blocks of frames of a video sequence. In this example, reference frame buffer 600 contains reference frames identified as last frame LAST_FRAME 602 , golden frame GOLDEN_FRAME 604 , and replacement reference frame ALTREF_FRAME 606 . The frame header of the reference frame may include a virtual index for a location in the reference frame buffer in which the reference frame is stored. The reference frame mapping may map a virtual index of the reference frame to a physical index of a memory in which the reference frame is stored. When two reference frames are the same frame, these reference frames have the same physical index even though they have different virtual indexes. The number, types, and names of reference locations in the reference frame buffer 600 used are merely examples.

[0055] Reference frames stored in the reference frame buffer 600 may be used to identify motion vectors for predicting blocks of frames to be encoded or decoded. Different reference frames may be used according to the type of prediction used to predict the current block of the current frame. For example, in bi-prediction, blocks of the current frame can be forward predicted using a frame stored as LAST_FRAME 602 or GOLDEN_FRAME 604 and backward using a frame stored as ALTREF_FRAME 606 . can be predicted.

[0056] There may be a finite number of reference frames that may be stored in the reference frame buffer 600 . As shown in FIG. 6 , the reference frame buffer 600 may store up to 8 reference frames, and each stored reference frame may be associated with a different virtual index of the reference frame buffer. 3 of 8 spaces in reference frame buffer 600 are used by frames designated as LAST_FRAME 602 , GOLDEN_FRAME 604 , and ALTREF_FRAME 606 , while 5 spaces are used to store other reference frames remain possible. For example, one or more available spaces in reference frame buffer 600 may be used to store some or all of additional reference frames, particularly an interpolated reference frame described herein. Although the reference frame buffer 600 is shown as capable of storing up to eight reference frames, other implementations of the reference frame buffer 600 may store additional or fewer reference frames.

[0057] In some implementations, the alternate reference frame designated ALTREF_FRAME 606 may be a frame of a video sequence that is encoded or decoded earlier than it is displayed, but far from the current frame in display order. For example, the replacement reference frame may be 10, 12, or more (or fewer) frames after the current frame in display order. The additional alternate reference frames may be frames located closer to the current frame in display order.

[0058] The replacement reference frame may not directly correspond to a frame in the sequence. Alternatively, the replacement reference frame may be generated using one or more of the filtered frames as well as to which filtering is applied, combined together, or combined together. The replacement reference frame may not be displayed. Instead, the replacement reference frame may be a frame or part of a frame that is generated and transmitted for use only in the prediction process (ie, the decoded sequence is omitted when displayed).

[0059] 7 is a diagram of a group of frames in a display order of a video sequence. In this example, the group of frames includes eight frames 702-716, preceded by frame 700, which may be referred to as a key frame or, in some cases, an overlay frame. No block within frame 700 is inter-predicted using the reference frames of the group of frames. Frame 700 is a key (also called intra-predicted frame) in this example, which refers to a state in which the predicted blocks within the frame are predicted only using intra prediction. However, frame 700 may be an overlay frame, which is an inter-predicted frame, which may be a reconstructed frame of a previous group of frames. In an inter-predicted frame, at least some of the predicted blocks are predicted using inter prediction. The number of frames forming each group of frames may vary depending on the spatial/temporal characteristics of the video and other encoded configurations, such as the key frame interval chosen for, for example, random access or error resilience. have.

[0060] The coding order of each group of frames may be different from the display order. This allows a frame positioned after the current frame in the video sequence to be used as a reference frame for encoding the current frame. A decoder such as decoder 500 may share a common group coding structure with an encoder such as encoder 400 . The group coding structure assigns different roles that each frame in the group can play in a reference buff (eg, last frame, alternate reference frame, etc.) and defines or indicates the coding order for the frames in the group.

[0061] FIG. 8 is a diagram of an example of a coding order for the group of frames of FIG. 7 . The coding order of FIG. 8 relates to a first group coding structure in which a single reverse reference frame is available for each frame of the group. Since the encoding and decoding order are the same, the order shown in FIG. 8 is generally referred to herein as a coding order. A key or overlay frame 700 is designated as a golden frame in the reference frame buffer, such as GOLDEN_FRAME 604 in the reference frame buffer 600 . Frame 700 is intra predicted in this example, so it does not require a reference frame, but as frame 700 , which is a frame reconstructed from a previous group, the overlay frame also does not use the reference frame of the group of current frames. The last frame 716 in the group is designated as an alternate reference frame in the reference frame buffer, such as ALTREF_FRAME 606 in the reference frame buffer 600 . In this coding order, frame 716 is coded out of display order after frame 700 to provide a reverse reference frame for each of the remaining frames 702-714. In coding the blocks of frame 716 , frame 700 serves as an available reference frame for the blocks of frame 716 . 8 is only an example of a coding order for a group of frames. Other group coding structures may designate one or more different or additional frames for forward and/or backward prediction.

[0062] As briefly mentioned above, the available reference frame portion may be the reference frame portion interpolated using lightflow estimation. The reference frame portion may be, for example, a block, a slice, or an entire frame. When frame-level light flow estimation is performed as described herein, the resulting reference frame is referred to herein as a co-located reference frame because its dimensions are the same as the current frame. This interpolated reference frame may also be referred to herein as a light flow reference frame.

[0063] 9 is a diagram used to illustrate a linear projection of a motion field in accordance with the teachings herein. Within the hierarchical coding framework, the light flow (also called motion field) of the current frame can be estimated using the closest available reconstructed (eg, reference) frame before and after the current frame. In FIG. 9 , reference frame 1 is a reference frame that can be used for forward prediction of the current frame 900 , while reference frame 2 is a reference frame that can be used for backward prediction of the current frame 900 . Using the example of FIGS. 6 to 8 for illustration, if the current frame 900 is the frame 706, the previous or last frame 704 (eg, LAST_FRAME 602) is stored in the reference frame buffer 600. The stored reconstructed frame) may be used as reference frame 1 , while frame 716 (eg, the reconstructed frame stored in reference frame buffer 600 as ALTREF_FRAME 606 ) may be used as reference frame 2.

[0064] Knowing the display indices of the current and reference frames, motion vectors can be projected into the pixels of the current frame 900 between the pixels of reference frame 1 and reference frame 2, assuming the motion field is temporally linear. In the simple example described with reference to FIGS. 6 to 8 , the index for the current frame 900 is 3, the index for the reference frame 1 is 0, and the index for the reference frame 2 is 8. In FIG. 9 , a projected motion vector 904 for a pixel 902 of a current frame 900 is shown. Using the previous example in the description, the display indices of the group of frames of FIG. 7 would show that frame 704 is closer in time to frame 706 than frame 716 . Thus, the single motion vector 904 shown in FIG. 9 represents the amount of motion between reference frame 1 and the current frame 900 that is different from between reference frame 2 and the current frame 900 . Nevertheless, the projected motion field 906 is linear between reference frame 1 , current frame 900 , and reference frame 2 .

[0065] Selecting the closest available reconstructed forward and backward reference frames and assuming a motion field for each pixel of the current frame that are temporally linear is an interpolated reference frame using optical flow estimation without transmitting additional information. Allows the generation of α to be performed at both the encoder and decoder (eg, in the intra/inter prediction step 402 and the intra/inter prediction step 508 ). It is also possible that, instead of the closest available reconstructed reference frames, different frames may be used as specified a priori between the encoder and the decoder. In some implementations, an identification of frames used for lightflow estimation may be transmitted. The generation of interpolated frames is discussed in more detail below.

[0066] 10 is a flow diagram of a method or process 1000 for motion compensated prediction of a frame of a video sequence using at least a portion of a reference frame generated using optical flow estimation. The reference frame portion may be, for example, a block, a slice, or an entire reference frame. The lightflow reference frame portion may also be referred to herein as a collocated reference frame portion. Process 1000 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106 . For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1000 when executed by a processor such as CPU 202 . may include machine readable instructions. Process 1000 may be implemented using specialized hardware or firmware. Some computing devices may have multiple memories or processors, and the operations described in process 1000 may be distributed using multiple processors, memories, or both.

[0067] At 1002 , a current frame to be predicted is determined. Frames may be coded and thus predicted in any order, such as the coding order shown in FIG. 8 . Frames to be predicted may also be referred to as first, second, third, etc. frames. The first, second, etc. labels do not necessarily indicate the order of the frames. Instead, unless stated otherwise, labels are used herein to distinguish one current frame from another. In the encoder, a frame may be processed into units of blocks in a block coding order, such as a raster scan order. At the decoder, a frame may also be processed into units of blocks upon receipt of its encoded residuals within the encoded bitstream.

[0068] At 1004, forward and backward reference frames are determined. In the examples described herein, the forward and backward reference frames are the closest reconstructed frames before and after the current frame (eg, in display order), such as the current frame 900 . Although not explicitly shown in FIG. 10 , if a forward or backward reference frame does not exist, the process 1000 ends. Then, the current frame is processed without considering the light flow.

[0069] If there are forward and backward reference frames at 1004 , a lightflow reference frame portion may be generated using the reference frames at 1006 . The generation of the light flow reference frame portion is described in more detail with reference to FIGS. 11 to 14 . The lightflow reference frame portion may, in some implementations, be stored at a defined location within the reference frame buffer 600 . Initially, light flow estimation according to the teachings herein is described.

[0070] Lightflow estimation can be performed for each pixel of the current frame portion by minimizing the following Lagrangian function (1):

(1)[0071]

(One)

는 명도 항등성 가정(즉, 이미지의 작은 부분의 강도 값은 위치 변화에도 불구하고 시간 경과에 따라 변화하지 않고 유지된다는 가정)에 기초한 데이터 페널티이다.

은 모션 필드의 평활도(즉, 인접한 픽셀들은 이미지에서 동일한 객체 항목에 속할 가능성이 높고 그래서 실질적으로 동일한 이미지 모션을 초래한다는 특성)에 기초한 공간 페널티이다. 라그랑지 파라미터 λ는 모션 필드의 평활도의 중요성을 제어한다. 파라미터 λ의 큰 값은 모션 필드를 보다 평활하게 하고, 보다 큰 스케일의 모션을 더 잘 감안할 수 있다. 대조적으로, 파라미터 λ의 더 작은 값은 객체의 에지들 및 작은 객체들의 이동에 보다 효과적으로 적응할 수 있다.[0072] In function 1,

is a data penalty based on the brightness constancy assumption (ie, the assumption that the intensity value of a small portion of an image remains unchanged over time despite changes in position).

is a spatial penalty based on the smoothness of the motion field (ie, the property that adjacent pixels are likely to belong to the same object item in the image and thus result in substantially the same image motion). The Lagrangian parameter λ controls the importance of smoothness of the motion field. A large value of the parameter λ makes the motion field smoother and can better account for larger-scale motion. In contrast, a smaller value of the parameter λ can more effectively adapt to the movement of edges of objects and small objects.

[0073] According to an implementation of the teachings herein, a data penalty may be expressed as a data penalty function:

(2)[0074]

(2)

[0075] The horizontal component of the motion field for the current pixel is represented by u, while the vertical component of the motion field is represented by v. Roughly speaking, E _x , E _y , and E _t are the derivatives of pixel values of the reference frame portions with respect to the horizontal axis x (eg, as indicated by the frame indices), the vertical axis y, and the time t. admit. A horizontal axis and a vertical axis are defined for an array of pixels forming a current frame, such as current frame 900 , and reference frames, such as reference frames 1 and 2.

[0076] In the data penalty function, the derivatives E _x , E _y , and E _t can be calculated according to the following functions (3), (4), and (5):

[0077]

(3)[0078]

(3)

[0079]

(4)[0080]

(4)

(5)[0081]

(5)

은 인코딩되는 현재 프레임 내의 현재 픽셀 위치의 모션 필드에 기초한 레퍼런스 프레임 1 내의 투영된 위치에서의 픽셀 값이다. 유사하게, 변수

는 인코딩되는 현재 프레임 내의 현재 픽셀 위치의 모션 필드에 기초한 레퍼런스 프레임 2 내의 투영된 위치에서의 픽셀 값이다.[0082] variable

is the pixel value at the projected position in reference frame 1 based on the motion field of the current pixel position in the current frame being encoded. Similarly, the variable

is the pixel value at the projected position in reference frame 2 based on the motion field of the current pixel position in the current frame being encoded.

은 레퍼런스 프레임 1의 디스플레이 인덱스이며, 여기서 프레임의 디스플레이 인덱스는 비디오 시퀀스의 디스플레이 순서에 있어서의 그 인덱스이다. 유사하게, 변수

는 레퍼런스 프레임 2의 디스플레이 인덱스이고, 변수

은 현재 프레임(900)의 디스플레이 인덱스이다.[0083] variable

is the display index of reference frame 1, where the display index of the frame is its index in the display order of the video sequence. Similarly, the variable

is the display index of reference frame 2, and

is the display index of the current frame 900 .

은 선형 필터를 사용하여 레퍼런스 프레임 1에서 계산된 수평 도함수(horizontal derivative)이다. 변수

는 선형 필터를 사용하여 레퍼런스 프레임 2에서 계산된 수평 도함수이다. 변수

은 선형 필터를 사용하여 레퍼런스 프레임 1에서 계산된 수직 도함수이다. 변수

는 선형 필터를 사용하여 레퍼런스 프레임 2에서 계산된 수직 도함수이다.[0084] variable

is the horizontal derivative computed in reference frame 1 using a linear filter. variable

is the horizontal derivative calculated in reference frame 2 using a linear filter. variable

is the vertical derivative calculated in reference frame 1 using a linear filter. variable

is the vertical derivative calculated in reference frame 2 using a linear filter.

[0085] In an implementation of the teachings herein, the linear filter used to calculate the horizontal derivative is the filter coefficient [-1/60, 9/60, -45/60, 0, 45/60, -9/60, 1/60 ] is a 7-tap filter. A filter may have a different frequency profile, a different number of taps, or both. The linear filter used to calculate the vertical derivatives may be the same or different from the linear filter used to calculate the horizontal derivatives.

[0086] A spatial penalty can be expressed as a spatial penalty function:

(6)[0087]

(6)

[0088] In the spatial penalty function (6), Δu is the Laplacian of the horizontal component u of the motion field, and Δv is the Laplacian of the vertical component v of the motion field.

[0089] 11 is a flow diagram of a method or process 1100 for generating an opticalflow reference frame portion. In this example, the light flow reference frame portion is the entire reference frame. Process 1100 may implement step 1006 of process 1000 . Process 1100 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106 . For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1100 when executed by a processor such as CPU 202 . may include machine readable instructions. Process 1100 may be implemented using specialized hardware or firmware. As noted above, multiple processors, memories, or both may be used.

[0090] Because the forward and backward reference frames may be relatively far from each other, there may be dramatic motion between them, thus reducing the accuracy of the brightness constancy assumption. To reduce the potential error in the motion of the pixel resulting from this problem, the estimated motion vectors from the current frame to the reference frames can be used to initialize the lightflow estimate for the current frame. At 1102 , all pixels within the current frame may be assigned an initialized motion vector. These define initial motion fields that can be used to warp reference frames to the current frame for a first level of processing to shorten motion lengths between reference frames.

은 다음의 함수에 따라, 현재 픽셀로부터 역방향 레퍼런스 프레임, 이 예에서는 레퍼런스 프레임 2를 가리키는 추정된 모션 벡터

와 현재 픽셀로부터 순방향 레퍼런스 프레임, 이 예에서는 레퍼런스 프레임 1을 가리키는 추정된 모션 벡터

사이의 차이를 나타내는 모션 벡터를 사용하여 초기화될 수 있다:[0091] Motion field of the current pixel

is the estimated motion vector pointing backwards from the current pixel to the reference frame, in this example reference frame 2, according to the function

and the estimated motion vector pointing forward from the current pixel to the reference frame, in this example reference frame 1.

It can be initialized using a motion vector representing the difference between:

[0092]

[0093] If any of the motion vectors are not available, it is possible to extrapolate the initial motion using the available motion vector according to any one of the following functions:

, 또는[0094]

, or

.[0095]

.

[0096] If the current pixel does not have a motion vector reference available, one or more spatial neighbors with an initialized motion vector may be used. For example, an average of available adjacent initialized motion vectors may be used.

In the example of initializing the motion field for the first processing level at 1102 , reference frame 2 may be used to predict the pixels of reference frame 1 , where reference frame 1 is the last frame before the current frame is coded. . That motion vector, projected onto the current frame using linear projection in a manner similar to that shown in FIG. 9 , generates a _{motion field mv cur at intersecting pixel locations, such as motion field 906 at pixel location 902 .} make it

[0098] 11 refers to initializing a motion field for a first level of processing since process 1100 preferably has multiple processing levels. This can be seen with reference to FIG. 13 , which is a diagram illustrating process 1100 of FIG. 11 (and process 1200 of FIG. 12 discussed below). The following description uses the phrase motion field. This phrase is intended to collectively refer to the motion fields for each pixel, unless otherwise clear from the context. Accordingly, the phrases “motion fields” or “motion field” may be used interchangeably when referring to two or more motion fields. Also, when referring to the movement of pixels, the phrase light flow may be used interchangeably with the phrase motion field.

[0099] To estimate the motion field/lightflow for the pixels of a frame, a pyramid or multi-layer structure may be used. For example, in one pyramid structure, reference frames are scaled down to one or more different scales. Then, the light flow is estimated to first obtain the motion field at the highest level of the pyramid (first processing level), ie using the most scaled reference frames. After that, the motion field is upscaled and used to initialize the lightflow estimation in the next level. This process of upscaling the motion field, using it to initialize the next level of lightflow estimation, and obtaining the motion field, continues until the lowest level of the pyramid is reached (i.e., the optical until flow estimation is complete).

[00100] The rationale for this process is that it is easier to capture large motions when the image is scaled down. However, using simple rescale filters to scale the reference frames themselves may degrade the reference frame quality. To avoid loss of detail due to rescaling, a pyramid structure that scales derivatives instead of pixels of reference frames is used to estimate the light flow. This pyramid scheme represents a regression analysis for light flow estimation. This scheme is shown in FIG. 13 and is implemented by process 1100 of FIG. 11 and process 1200 of FIG. 12 .

[00101] After initialization, at 1104, the Lagrangian parameter λ is set to solve the Lagrangian function (1). Preferably, process 1100 uses multiple values for the Lagrangian parameter λ. In 1104, the first value to which the Lagrangian parameter λ is set may be a relatively large value such as 100. Although it is preferred for process 1100 to use multiple values for the Lagrangian parameter λ in the Lagrangian function 1, it is also possible for only one value to be used, as described below in the process 1200 described below. .

는 워핑을 수행하기 전에 풀(full) 해상도 값으로부터 그 레벨의 해상도로 다운스케일링된다는 점에 유의할 필요가 있다. 모션 필드의 다운스케일링은 아래에서 보다 상세히 논의된다.At 1106 , the reference frames are warped into the current frame according to the motion field for the current processing level. Warping the reference frames to the current frame may be performed using subpixel location rounding. Motion field used in the first processing level

It is worth noting that is downscaled from a full resolution value to that level of resolution before warping is performed. Downscaling of the motion field is discussed in more detail below.

을 알면, 레퍼런스 프레임 1을 워핑하기 위한 모션 필드는 다음과 같이 (예를 들면, 모션은 시간 경과에 따라 선형으로 투영된다는) 선형 투영 가정에 의해 추론된다:[00103] light flow

Knowing , the motion field for warping reference frame 1 is inferred by the linear projection assumption (e.g., motion is projected linearly over time) as follows:

[00104]

의 수평 성분

및 수직 성분

은 Y 성분에 대해서는 1/8 픽셀 정밀도로 및 U 및 V 성분에 대해서는 1/16 픽셀 정밀도로 라운딩될 수 있다. 서브픽셀 위치 라운딩에 대한 다른 값들도 사용될 수 있다. 라운딩 후, 워핑된 이미지의 각각의 픽셀

은 모션 벡터

에 의해 주어진 참조된 픽셀로 계산된다. 종래의 서브픽셀 보간 필터를 사용하여 서브픽셀 보간이 수행될 수 있다.[00105] To perform warping, a motion field

horizontal component of

and vertical components

can be rounded to 1/8 pixel precision for the Y component and 1/16 pixel precision for the U and V components. Other values for subpixel position rounding may also be used. After rounding, each pixel of the warped image

silver motion vector

It is computed with the referenced pixel given by . Subpixel interpolation may be performed using a conventional subpixel interpolation filter.

가 얻어지는데, 여기서 모션 필드는 다음에 의해 계산된다:[00106] The same warping approach is performed for reference frame 2, so that the warped image

is obtained, where the motion field is calculated by:

[00107]

[00108] At the end of the calculation at 1106, there are two warped reference frames. The two warped reference frames are used at 1108 to estimate the motion field between them. Estimating the motion field at 1108 may include multiple steps.

및/또는

)이 얻어질 수 있다. 그 다음에, 다수의 층들이 있으면, 도함수들은 현재 레벨로 다운스케일링된다. 도 13에 도시된 바와 같이, 레퍼런스 프레임들은 세부 사항들을 캡쳐하기 위해 원래 스케일에서 도함수를 계산하는 데 사용된다. 각각의 레벨 l에서 도함수들을 다운스케일링하는 것은 2¹ x 2¹ 블록 내에서 평균화함으로써 계산될 수 있다. 도함수들을 계산하는 것뿐만 아니라 도함수들을 평균화하여 다운스케일링하는 것은 모두 선형 연산들이기 때문에, 2 개의 연산들은 각각의 레벨 l에서 도함수들 계산하기 위해 단일의 선형 필터로 결합될 수 있다는 것이 주목된다. 이는 계산들의 복잡도를 낮출 수 있다.[00109] First, derivatives E _x , E _y , and E _t are computed using functions (3), (4), and (5). When calculating the derivatives, the frame boundaries of the warped reference frame can be extended by copying the nearest available pixel. In this way, the pixel values (i.e., the pixel values when the projected positions are outside the warped reference frame)

and/or

) can be obtained. Then, if there are multiple layers, the derivatives are downscaled to the current level. As shown in FIG. 13 , reference frames are used to compute the derivative at the original scale to capture the details. Downscaling the derivatives at each level 1 can be calculated by averaging within a ^{2 1} x 2 ^{1 block.} It is noted that since calculating the derivatives as well as averaging and downscaling the derivatives are all linear operations, the two operations can be combined into a single linear filter to compute the derivatives at each level l. This can lower the complexity of calculations.

및

),

개의 선형 방정식들로 프레임의 모든 N 픽셀들에 대해 성분 u 및 v를 풀 수 있다. 이것은 라플라시안(Laplacians)이 2 차원(2D) 필터에 의해 근사된다는 사실에 기인한다. 정확하지만 매우 복잡한 선형 방정식들을 직접 푸는 대신에, 라그랑지 함수(1)를 최소화하기 위해 반복 접근법들이 사용되어 더 빠르지만 덜 정확한 결과를 얻을 수 있다.[00110] Once the derivatives are downscaled to the current processing level, light flow estimation may be performed according to the Lagrangian function (1), if applicable. More specifically, by setting the derivatives of the Lagrangian function (1) with respect to the horizontal component u of the motion field and the vertical component v of the motion field to zero (i.e.,

and

),

We can solve for the components u and v for all N pixels of the frame with the linear equations. This is due to the fact that Laplacians are approximated by a two-dimensional (2D) filter. Instead of solving the exact but very complex linear equations directly, iterative approaches are used to minimize the Lagrangian function (1), resulting in faster but less accurate results.

[00111] At 1108 , the motion field for pixels of the current frame is updated or improved using the estimated motion field between the warped reference frames. For example, the current motion field for a pixel may be updated by adding an estimated motion field for each pixel on a per-pixel basis.

[00112] Once the motion field is estimated at 1108, a query is made at 1110 to determine whether there are additional values available for the Lagrangian parameter λ. Smaller values of the Lagrangian parameter λ can cope with a smaller scale of motion. If additional values exist, the process 1100 may return to 1104 to set the next value for the Lagrangian parameter λ. For example, process 1100 may iterate with halving the Lagrangian parameter λ at each iteration. The motion field updated at 1108 is the current motion field for warping the reference frames at 1106 in this next iteration. The motion field is then estimated again at 1108 . Processing at 1104, 1106, and 1108 continues until all possible Lagrangian parameters at 1110 have been processed. In one example, as shown in FIG. 13 , there are three levels in the pyramid, so the minimum value of the Lagrangian parameter λ in the example is 25. This iterative process performed while changing the Lagrangian parameter may be referred to as annealing the Lagrangian parameter.

[00113] Once there is no remaining value for the Lagrangian parameter λ at 1110, the process 1100 proceeds to 1112 to determine whether there are more processing levels to process. If there are additional processing levels at 1112 , process 1100 proceeds to 1114 , where, starting at 1104 , the motion field is upscaled before processing the next layer using each of the available values for the Lagrangian parameter λ. Upscaling of the motion field may be performed using any known technique including, but not limited to, the reverse of the downscaling calculations described above.

[00114] In general, the light flow is estimated first to obtain the motion field at the highest level of the pyramid. After that, the motion field is upscaled and used to initialize the lightflow estimation in the next level. This process of upscaling the motion field, using it to initialize the next level of lightflow estimation, and obtaining the motion field, continues until the lowest level of the pyramid is reached at 1112 (i.e. to the full scale calculated derivatives). until the light flow estimation is completed).

를 형성한다. 1116에서 블렌딩된 워핑된 레퍼런스 프레임들은 1108에서 추정된 모션 필드를 사용하여 1106에 기술된 프로세스에 따라 다시 워핑되는 풀 스케일 레퍼런스 프레임들일 수 있음에 유의하자. 다시 말하면, 풀 스케일의 레퍼런스 프레임들은 2 회 ― 이전 처리 층으로부터의 최초의 업스케일링된 모션 필드를 사용하여 한 번 및 모션 필드가 풀 스케일 레벨로 개선된 후에 다시 ― 워핑될 수 있다. 블렌딩은 다음과 같이 (예를 들면, 프레임들이 동일한 시간 간격들로 이격되어 있다는) 시간 선형성 가정을 이용하여 수행될 수 있다:[00115] Once the level reaches a level at which the reference frames are not downscaled (ie, the reference frames are at their original resolution), the process 1100 proceeds to 1116 . For example, the number of levels may be three, such as in the example of FIG. 13 . At 1116, the warped reference frames are blended to form a lightflow reference frame.

to form Note that the warped reference frames blended at 1116 may be full scale reference frames that are re-warped according to the process described at 1106 using the estimated motion field at 1108 . In other words, the full scale reference frames can be warped twice - once using the first upscaled motion field from the previous processing layer and again after the motion field is improved to the full scale level. Blending can be performed using the temporal linearity assumption (e.g., frames are spaced at equal time intervals) as follows:

[00116]

로 표시된) 레퍼런스 프레임 1의 레퍼런스 픽셀은 경계들을 벗어하는(예를 들면, 프레임의 치수들의 외부에 있음) 반면 레퍼런스 프레임 2의 레퍼런스 픽셀은 경계들을 벗어나지 않으면, 레퍼런스 프레임 2에서 발생하는 워핑된 이미지의 픽셀만 다음에 따라 사용된다:[00117] In some implementations, it is desirable to prefer a pixel of only one of the warped reference frames over a blended value. For example, (

of the warped image occurring in reference frame 2, if the reference pixel of reference frame 1 (indicated by Only pixels are used according to:

[00118]

[00119] Optional occlusion may be performed as part of the blending. Occlusion of objects and backgrounds is common in video sequences, where parts of an object appear in one reference frame but are hidden in another. In general, the above-described light flow estimation method cannot estimate the motion of an object in such a situation because the assumption of brightness constancy is violated. When the magnitude of the occlusion is relatively small, the smoothness penalty function can estimate the motion fairly accurately. That is, if an undefined motion field in the hidden part is smoothed by adjacent motion vectors, the motion of the entire object may be correct.

[00120] However, even in this case, the simple blending method described above may not provide a satisfactory interpolation result. This can be explained with reference to Fig. 14, which is a diagram illustrating object occlusion. In this example, the occluded portion of object A is displayed in reference frame 1 and hidden by object B in reference frame 2. Since the hidden portion of object A is not visible in reference frame 2, the referenced pixels from reference frame 2 are of object B. In this case, it is preferable to use only the warped pixels from reference frame 1. Therefore, using a technique for detecting occlusions instead of or in addition to the above blending may provide a better blending result, and thus a better reference frame.

및

는 2 개의 상이한 객체들에서 온 것이기 때문에 매우 다르다는 것이다. 두 번째 상황은 객체 B의 픽셀들은 현재 프레임 내의 객체 B 및 현재 프레임 내의 객체 A의 폐색 부분에 대한 다수의 모션 벡터들에 의해 참조된다는 것이다.[00121] Regarding the detection of occlusion, it is observed from FIG. 14 that if occlusion occurs and the motion field is quite accurate, the motion vector of the occluded portion of object A points to object B in reference frame 2. This can lead to situations such as: The first situation is the warped pixel values.

and

is very different because it comes from two different objects. The second situation is that the pixels of object B are referenced by multiple motion vectors for object B in the current frame and the occluded portion of object A in the current frame.

에 대한

만의 폐색 및 사용을 결정하기 위해 다음의 조건들이 확립될 수 있으며,

에 대해

만 사용하는 경우에도 유사한 조건들이 적용된다:[00122] By these observations,

for

The following conditions may be established to determine the occlusion and use of the bay:

About

Similar conditions apply when using only:

이 문턱값

보다 크고; 그리고[00123]

this threshold

greater than; and

가 문턱값

보다 크다.[00124]

is the threshold

bigger than

는 레퍼런스 프레임 1의 참조된 픽셀이 현재의 병치된 프레임 내의 임의의 픽셀에 의해 참조되는 총 횟수이다. 전술한 서브픽셀 보간이 존재하는 경우, 레퍼런스 서브픽셀 위치가 관심있는 픽셀 위치의 1 픽셀 길이 내에 있을 때

가 카운트된다. 또한,

가 서브픽셀 위치를 가리키는 경우, 4 개의 인접한 픽셀들의 가중 평균

는 현재의 서브픽셀 위치에 대한 레퍼런스들의 총 수이다.

도 유사하게 정의할 수 있다.[00125]

is the total number of times the referenced pixel of reference frame 1 is referenced by any pixel in the current collocated frame. When the aforementioned subpixel interpolation, if present, the reference subpixel position is within 1 pixel length of the pixel position of interest.

is counted In addition,

If is a subpixel position, the weighted average of 4 adjacent pixels

is the total number of references to the current subpixel position.

can be defined similarly.

[00126] Accordingly, occlusion may be detected in the first reference frame using the first warped reference frame and the second warped reference frame. The blending of the warped reference frames may then include populating pixel positions of the lightflow reference frame corresponding to the occlusion with pixel values from the second warped reference frame. Similarly, occlusion can be detected in the second reference frame using the first warped reference frame and the second warped reference frame. The blending of the warped reference frames may then include filling pixel positions of the lightflow reference frame corresponding to the occlusion with pixel values from the first warped reference frame.

선형 방정식들을 이용한다. 다시 말해서, 광흐름 추정의 계산 복잡도는 프레임 크기의 다항식 함수이며, 이는 디코더의 복잡도에 부담을 준다. 따라서, 서브프레임 기반(예를 들면, 블록 기반)의 광흐름 추정이 다음에 설명되는데, 이는 도 11과 관련하여 설명된 프레임 기반 광흐름 추정보다 디코더의 복잡도를 저감시킬 수 있다.It has been experimentally shown that process 1100 provides substantial compression performance gains. These performance gains include gains of 2.5% in PSNR and 3.3% in SSIM for the set of low resolution frames, and gains of 3.1% in PSNR and 4.0% in SSIM for the set of medium resolution frames. However, as described above, the light flow estimation performed according to the Lagrangian function (1) is performed to solve for the horizontal component u and the vertical component v of the motion field for all N pixels of the frame.

Use linear equations. In other words, the computational complexity of the optical flow estimation is a polynomial function of the frame size, which places a burden on the complexity of the decoder. Accordingly, subframe-based (eg, block-based) optical flow estimation is described next, which can reduce the complexity of the decoder compared to the frame-based optical flow estimation described with reference to FIG. 11 .

[00128] 12 is a flow diagram of a method or process 1200 for generating an opticalflow reference frame portion. In this example, the lightflow reference frame portion is smaller than the entire reference frame. Although in this example the collocated frame parts are described with reference to blocks, other frame parts may be processed according to FIG. 12 . Process 1200 may implement step 1006 of process 1000 . Process 1200 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106 . For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1200 when executed by a processor such as CPU 202 . may include machine readable instructions. Process 1200 may be implemented using specialized hardware or firmware. As noted above, multiple processors, memories, or both may be used.

[00129] At 1202, all pixels in the current frame are assigned an initialized motion vector. These define initial motion fields that can be used to warp reference frames to the current frame for a first level of processing to shorten motion lengths between reference frames. The initialization at 1202 may be performed using the same processing as described with respect to the initialization at 1102, so the description is not repeated here.

At 1204 , reference frames - eg, reference frame 1 and reference frame 2 - are warped to the current frame according to the motion field initialized at 1202 . Warping at 1204 is preferably the same process as described with respect to warping at 1106, except that _{the motion field mv cur initialized at 1202 is not downscaled from its full resolution value before warping the reference frames.} can be performed using

[00131] At the end of the calculation at 1204, there are two warped reference frames of full resolution. Like process 1100 , process 1200 can estimate a motion field between two reference frames using a multi-level process similar to that described with respect to FIG. 13 . Broadly speaking, process 1200 computes derivatives for a level until all levels are considered, uses the derivatives to perform lightflow estimation, and upscales the resulting motion field for the next level.

이 1206에서 초기화된다. 블록은 현재 프레임의 스캔 순서(예를 들면, 래스터 스캔 순서)에서 선택된 현재 프레임의 블록일 수 있다. 블록에 대한 모션 필드

은 블록의 각각의 픽셀들에 대한 모션 필드를 포함한다. 즉, 1206에서, 현재의 블록 내의 모든 픽셀들에는 초기화된 모션 벡터가 할당된다. 초기화된 모션 벡터들은 레퍼런스 프레임들의 레퍼런스 블록들 사이의 길이들을 단축시키기 위해 레퍼런스 블록들을 현재 블록으로 워핑하는데 사용된다.[00132] More specifically, a motion field for a block at the current (or first) processing level

It is initialized at 1206. The block may be a block of the current frame selected from the scan order (eg, raster scan order) of the current frame. motion field for blocks

contains a motion field for each pixel of the block. That is, at 1206 , all pixels in the current block are assigned an initialized motion vector. The initialized motion vectors are used to warp the reference blocks into the current block to shorten the lengths between the reference blocks of the reference frames.

은 그 풀 해상도 값으로부터 레벨의 해상도로 다운스케일링된다. 다시 말하면, 1206에서의 초기화는 1202에서 초기화된 풀 해상도 값으로부터 블록의 각각의 픽셀들에 대한 모션 필드를 다운스케일링하는 것을 포함할 수 있다. 다운스케일링은 전술한 다운스케일링과 같은, 임의의 기술을 사용하여 수행될 수 있다.[00133] At 1206, the motion field

is downscaled from its full resolution value to the level's resolution. In other words, the initialization at 1206 may include downscaling the motion field for each pixel of the block from the full resolution value initialized at 1202 . Downscaling may be performed using any technique, such as the downscaling described above.

를 알면, 워핑을 위한 모션 필드는 다음과 같이 (예를 들면, 모션은 시간 경과에 따라 선형으로 투영된다는) 선형 투영 가정에 의해 추론된다:At 1208 , collocated reference blocks corresponding to the motion field of each of the warped reference frames are warped to the current block. Warping of the reference blocks is performed similar to process 1100 at 1106 . That is, the light flow of the pixels of the reference block of the reference frame 1

Knowing , the motion field for warping is inferred by the linear projection assumption (e.g., motion is projected linearly over time) as follows:

[00135]

의 수평 성분

및 수직 성분

은 Y 성분에 대해서는 1/8 픽셀 정밀도로 및 U 및 V 성분에 대해서는 1/16 픽셀 정밀도로 라운딩될 수 있다. 다른 값들도 사용될 수 있다. 라운딩 후, 워핑된 블록의 각각의 픽셀, 예를 들면

은 모션 벡터 mv_r1에 의해 주어진 참조된 픽셀로 계산된다. 종래의 서브픽셀 보간 필터를 사용하여 서브픽셀 보간이 수행될 수 있다.[00136] To perform warping, a motion field

horizontal component of

and vertical components

can be rounded to 1/8 pixel precision for the Y component and 1/16 pixel precision for the U and V components. Other values may also be used. After rounding, each pixel of the warped block, e.g.

is computed with the referenced pixel given by the motion vector mv _{r1 .} Subpixel interpolation may be performed using a conventional subpixel interpolation filter.

가 얻어지는데, 여기서 모션 필드는 다음에 의해 계산된다:[00137] The same warping approach is performed for the reference block of the reference frame 2, so that the warped block, for example

is obtained, where the motion field is calculated by:

[00138]

[00139] At the end of the calculation at 1208, there are two warped reference blocks. The two warped reference blocks are used at 1210 to estimate the motion field between them. The processing at 1210 may be similar to that described with respect to the processing at 1108 of FIG. 11 .

및/또는

로 사용될 수 있다. 투영된 픽셀들이 프레임 경계부들의 외부에 있는 경우, 가장 가까운 이용 가능한(즉, 경계들 내의) 픽셀을 복사하는 것이 여전히 사용될 수 있다. 도함수들이 계산된 후, 이들은 현재 레벨로 다운스케일링될 수 있다. 각각의 레벨 l에서 다운스케일링된 도함수들은 앞서 논의된 바와 같이, 2¹ x 2¹ 블록 내에서 평균화함으로써 계산될 수 있다. 도함수들을 계산하고 평균화하는 두 가지 선형 연산들을 단일의 선형 필터로 결합함으로써 계산들의 복잡도를 낮출 수 있지만, 필수는 아니다.More specifically, the two warped reference blocks may be full resolution. According to the pyramid structure of FIG. 13 , derivatives E _x , E _y , and E _t are calculated using functions (3), (4), and (5). When calculating derivatives for frame level estimation, frame boundaries may be expanded by copying the nearest available pixel to obtain out-of-bounds pixel values as described with respect to process 1100 . However, for other frame portions, adjacent pixels in the reference frames warped at 1204 are often available. For example, for block-based estimation, pixels of adjacent blocks are available in warped reference frames unless the block itself is at a frame boundary. Thus, in the case of pixels that are out of bounds with respect to a warped reference frame portion, pixels in adjacent portions of the warped reference frame have pixel values, if applicable.

and/or

can be used as If the projected pixels are outside the frame boundaries, then copying the nearest available (ie within the boundaries) pixel can still be used. After the derivatives are calculated, they can be downscaled to the current level. The downscaled derivatives at each level 1 can be calculated by averaging within a ^{2 1} x 2 ^{1 block, as discussed above.} Combining the two linear operations of calculating and averaging the derivatives into a single linear filter can reduce the complexity of the calculations, but is not required.

및

),

개의 선형 방정식들을 푸는 것에 의해, 부분, 여기서는 블록의 모든 N 픽셀들에 대해 모션 필드의 수평 성분 u 및 수직 성분 v가 결정될 수 있다. 이를 위해, 경계를 벗어난 모션 벡터들을 다루는 두 가지 선택적 방법이 있다. 하나의 방법은 인접한 블록들과 제로 상관(zero correlation)을 가정하고, 경계를 벗어난 모션 벡터가 경계를 벗어난 픽셀 위치에 가장 가까운 경계부 위치에 있는 모션 벡터와 동일하다고 가정하는 것이다. 다른 방법은 현재 픽셀에 대해 초기화된 모션 벡터(즉, 1206에서 초기화된 모션 필드)를 현재 픽셀에 대응하는 경계를 벗어난 픽셀 위치에 대한 모션 벡터로 사용하는 것이다.Continuing the process at 1210 , the downscaled derivatives may be used as inputs of the Lagrangian function 1 to perform lightflow estimation to estimate the motion field between the warped reference portions. Set the derivatives of the Lagrangian function (1) with respect to the horizontal component u and the vertical component v to zero (i.e.,

and

),

By solving the linear equations of , the horizontal component u and the vertical component v of the motion field can be determined for a part, here all N pixels of the block. To this end, there are two alternative ways to deal with out-of-bounds motion vectors. One method is to assume zero correlation with adjacent blocks, and assume that the out-of-bounds motion vector is the same as the motion vector at the boundary position closest to the out-of-boundary pixel position. Another method is to use the motion vector initialized for the current pixel (ie, the motion field initialized at 1206 ) as the motion vector for the out-of-bounds pixel position corresponding to the current pixel.

[00142] After the motion field is estimated, the current motion field for the level is updated or improved using the estimated motion field between the warped reference blocks, completing processing at 1210 . For example, the current motion field for a pixel may be updated by adding an estimated motion field for each pixel on a per-pixel basis.

[00143] In process 1100, an additional loop to set decreasing values for the Lagrangian parameter λ such that at each level the motion field is estimated and improved using increasingly smaller values for the Lagrangian parameter λ. is included In process 1200, this loop is omitted. That is, in the process 1200 as shown, only one value for the Lagrangian parameter λ is used to estimate the motion field at the current processing level. This may be a relatively small value, such as 25. Other values for the Lagrangian parameter λ are possible, for example, depending on the smoothness of the motion, the image resolution, or other variables.

[00144] In other implementations, process 1200 may include an additional loop to change the Lagrangian parameter λ. In an implementation where such a loop is involved, warping the reference blocks at 1208 and estimating and updating the motion field at 1210 as described with respect to processing at 1104 and 1110 in process 1100 depends on the Lagrangian parameter λ. The Lagrangian parameter λ may be set before estimating the motion field at 1210 , such that it is repeated until all values for α are used.

[00145] The process 1200 proceeds to the query at 1212 after estimating and updating the motion field at 1210 . This is done after the first and only motion field estimation and update at the level of 1210 when a single value is used for the Lagrangian parameter λ. When multiple values for the Lagrangian parameter λ are modified at the processing level, the process 1200 estimates and updates the motion field at 1210 using the final value of the Lagrangian parameter λ before proceeding to the query at 1212 .

[00146] If there are additional processing levels in response to the query at 1212 , the process 1200 proceeds to 1214 , where the motion field is upscaled before processing the next layer starting at 1206 . Upscaling may be performed according to any known technique.

[00147] In general, the light flow is first estimated to obtain the motion field at the highest level of the pyramid. The motion field is then upscaled and used to initialize the lightflow estimate at the next level. This process of upscaling the motion field, using it to initialize the next level of lightflow estimation, and obtaining the motion field, continues at 1212 until the lowest level of the pyramid is reached (i.e., to the derivatives calculated at full scale). until the light flow estimation is completed).

)을 형성한다. 1216에서 블렌딩된 워핑된 레퍼런스 블록들은 1210에서 추정된 모션 필드를 사용하여 1208에 기술된 프로세스에 따라 다시 워핑되는 풀 스케일 레퍼런스 블록들일 수 있음에 유의하자. 다시 말하면, 풀 스케일의 레퍼런스 블록들은 2 회 ― 이전 처리 층으로부터의 최초의 업스케일링된 모션 필드를 사용하여 한 번 및 모션 필드가 풀 스케일 레벨로 개선된 후에 다시 ― 워핑될 수 있다. 블렌딩은 1116에서 설명된 처리와 유사하게 시간 선형성 가정을 이용하여 수행될 수 있다. 1116에서 설명되고 도 14에 예로서 도시된 바와 같은 선택적인 폐색 검출은 1216에 블렌딩의 일부로서 통합될 수도 있다.[00148] Once the level reaches a level at which the reference frames are not downscaled (ie, the reference frames are at their original resolution), the process 1200 proceeds to 1216 . For example, the number of levels may be three, such as in the example of FIG. 13 . At 1216, the warped reference blocks are blended to a lightflow reference block (eg, as described above).

) to form Note that the warped reference blocks blended at 1216 may be full scale reference blocks that are warped again according to the process described at 1208 using the estimated motion field at 1210 . In other words, the full scale reference blocks can be warped twice - once using the first upscaled motion field from the previous processing layer and again after the motion field is improved to the full scale level. Blending may be performed using a temporal linearity assumption similar to the process described at 1116 . An optional occlusion detection as described at 1116 and shown by way of example in FIG. 14 may be incorporated as part of the blending at 1216 .

[00149] After the collocated reference block is generated at 1216 , the process 1200 proceeds to 1218 to determine whether there are additional frame portions (here, blocks) for prediction. If so, process 1200 repeats starting at 1206 for the next block. Blocks may be processed in scan order. If there are no more blocks to consider in response to the query at 1218, the process 1200 ends.

[00150] Referring again to FIG. 10 , process 1200 may implement 1006 in process 1000 . At the end of processing at 1006, there are one or more warped reference frame portions, whether performed according to process 1100 , process 1200 , or a variant thereof as described herein.

[00151] At 1008 , a prediction process is performed using the portion of the lightflow reference frame generated at 1006 . Performing the prediction process at the encoder may include generating a prediction block from a lightflow reference frame for a current block of the frame. The optical flow reference frame may be an optical flow reference frame output by process 1100 and stored in a reference frame buffer, such as reference frame buffer 600 . The lightflow reference frame may be a lightflow reference frame created by combining portions of the lightflow reference output by process 1200 . Combining the lightflow reference portions includes arranging the lightflow reference portions (eg, collocated reference blocks) according to the pixel positions of each current frame portion used in the creation of each of the lightflow reference portions. may include The resulting optical flow reference frame may be stored for use in a reference frame buffer of an encoder, such as reference frame buffer 600 of encoder 400 .

[00152] Generating the predictive block at the encoder may include selecting a collocated block in the opticalflow reference frame as the predictive block. Generating the predictive block at the encoder may instead include performing a motion search within the opticalflow reference frame to select a best-matching predictive block for the current block. However, the predictive block is generated at the encoder, and the resulting residual may be further processed using, for example, the lossy encoding process described with respect to the encoder 400 of FIG. 4 .

[00153] At the encoder, process 1000 is a rate for a current block using various prediction modes, including both single and complex inter prediction modes using one or more intra prediction modes and prediction frames available for the current frame. It can form part of a rate distortion loop. The single inter prediction mode uses only a single forward or backward reference frame for inter prediction. The complex inter prediction mode uses both forward and backward reference frames for inter prediction. In the rate distortion loop, the rate (eg, number of bits) used to encode the current block using the respective prediction modes is compared to the distortion due to encoding. Distortion can be calculated as differences between pixel values of a block before encoding and after decoding. Differences may be the sum of absolute differences or another measure that captures the cumulative error for blocks of frames.

[00154] In some implementations, it may be desirable to limit the use of the opticalflow reference frame to a single inter prediction mode. That is, the light flow reference frame may be excluded as a reference frame in all composite reference modes. This can simplify the rate distortion loop, and since the lightflow reference frame already considers both forward and backward reference frames, it is expected that there will be little additional impact on the encoding of the block. According to implementations described herein, a flag may be encoded into the bitstream to indicate whether a lightflow reference frame is available for use in encoding the current frame. The flag may be encoded in one example when any single block within the current frame is encoded using the opticalflow reference frame block. If an opticalflow reference frame is available for the current frame, it may include an additional flag or other indicator (eg, at block level) indicating whether the current block was encoded by inter prediction using the opticalflow reference frame.

[00155] The prediction process at 1008 may be repeated for all blocks of the current frame until the current frame is encoded.

[00156] At the decoder, performing the prediction process using the opticalflow reference frame portion at 1008 may result in a determination that the opticalflow reference frame is available for decoding the current frame. In some implementations, the determination is made by examining a flag indicating that at least one block of the current frame has been encoded using the opticalflow reference frame portion. Performing the prediction process at 1008 at the decoder may include generating a prediction block. Generating the prediction block may include using an inter prediction mode decoded from an encoded bitstream, such as in a block header. A flag or indicator may be decoded to determine the inter prediction mode. When the inter prediction mode is the optical flow reference frame mode (ie, when the block is inter predicted using the optical flow reference frame part), the prediction block for the current block to be decoded is the pixels of the optical flow reference frame part and the motion vector. It is created using modes and/or motion vectors.

[00157] The same processing for generating an opticalflow reference frame for use in the prediction process as part of decoding may be performed in a decoder, such as decoder 500, as performed in the encoder. For example, when the flag indicates that at least one block of the current frame has been encoded using a portion of the opticalflow reference frame, an entire opticalflow reference frame may be generated and stored for use in the prediction process. However, by modifying the process 1200 to limit the performance of the process 1200 in which the coding blocks are identified as using the collocated/optical flow reference frame as the inter prediction reference frame, additional computational power is saved at the decoder. This can be explained with reference to FIG. 15 , which is a diagram illustrating one technique for optimizing a decoder.

[00158] In FIG. 15 , pixels are shown along a grid 1500 , where w denotes the position of the pixel along the first axis of the grid 1500 and y denotes the position of the pixel along the second axis of the grid 1500 . Grid 1500 represents pixel positions of a portion of the current frame. The processing at 1006 and 1008 may be combined to perform the prediction process at the decoder at 1008 . For example, before performing the process at 1006 , the prediction process at 1008 may include finding a reference block used to encode the current block (eg, from header information such as a motion vector). In FIG. 15 , the motion vector for the current coding block 1502 points to the reference block indicated by the inner dashed line 1504 . The current coding block 1502 contains 4x4 pixels. The location of the reference block is shown by dashed line 1504 because the reference block is located in the reference frame and not the current frame.

[00159] Once a reference block is located, all of the reference blocks it spans (ie, overlaps) are identified. This may include extending the size of the reference block by half the filter length at each boundary in order to consider subpixel interpolation filters. In FIG. 15 , the length L of the subpixel interpolation filter is used to extend the reference block to the boundaries indicated by the outer dashed line 1506 . As is relatively common, a motion vector results in a reference block that is not perfectly aligned with full-pel positions. The shaded areas in FIG. 15 represent all pel positions. All pel positions and overlapping reference blocks are identified. Assuming that the block sizes are equal to the current coding block 1502, a first reference block collocated with the current block, a second reference block above the first reference block, and two reference blocks extending from the left of the first reference block , and two reference blocks extending from the left side of the second reference block are identified.

[00160] Once the reference blocks have been identified, the process 1200 may be performed at 1006 only on blocks in the current frame collocated with the identified reference blocks to generate collocated/optical flow estimated reference blocks. In the example of FIG. 15 , this results in 6 lightflow reference frame parts being generated.

[00161] According to this modified process, it is guaranteed that the encoder and decoder have the same predictor while the decoder does not need to compute the entire collocated reference frame. It is noted that the reference block(s) for a subsequent block including any extended boundaries may overlap with one or more reference blocks identified during the decoding process of the current block. In this case, the lightflow estimation need only be performed once for any of the identified blocks to further reduce the computing requirements at the decoder. In other words, the reference block generated at 1216 may be stored for use in decoding other blocks of the current frame.

[00162] However, the predictive block is generated at the decoder, and the decoded residual for the current block from the encoded bitstream is combined with the predictive block to form a reconstruction block, as described by way of example with respect to the decoder 500 of FIG. 5 . can be

[00163] Whether performed after process 1200 or in conjunction with process 1200 , the prediction process at 1008 is to be repeated for all blocks of the current frame encoded using the opticalflow reference frame portion until the current frame is decoded. can When processing blocks in decoding order, blocks not encoded using the opticalflow reference frame portion can be decoded in a conventional manner according to the prediction mode decoded for blocks from the conventionally encoded bitstream.

[00164] For N pixels in a frame or block, the complexity for solving the light flow formula can be expressed as O(N * M), where M is the number of iterations to solve the linear equations. M has nothing to do with the number of levels or the number of values of the Lagrangian parameter λ. Instead, M is related to the computational precision in solving linear equations. The larger the value of M, the better the precision. Given this complexity, moving from frame level to subframe level (eg block based) estimation provides several options for reducing decoder complexity. First, and since the constraint of motion field smoothness is relaxed at block boundaries, it is easier to converge to a solution when solving the linear equations of a block, so that M is smaller for similar precision. Second, solving a motion vector involves its adjacent motion vectors due to the smoothness penalty factor. Motion vectors at block boundaries have fewer adjacent motion vectors, so faster calculations are made. Third, and as discussed above, the light flow need not be calculated for the entire frame, but only for some of the blocks of the collocated reference frame identified by those coding blocks that use the collocated reference frame for inter prediction. have.

[00165] For simplicity of explanation, each of processes 1000 , 1100 , and 1200 is shown and described as a series of steps or operations. However, steps or operations according to the present invention may occur in various orders and/or concurrently. In addition, other steps or operations not shown and described herein may also be used. Moreover, not all illustrated steps or acts are required to implement a method in accordance with the disclosed subject matter.

[00166] Aspects of encoding and decoding described above exemplify some examples of encoding and decoding techniques. It should be understood, however, that encoding and decoding, as these terms are used in the claims, may mean compression, decompression, transformation, or any other processing or alteration of data.

[00167] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “an example” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the word "yes" is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from context, "X comprises A or B" is intended to mean any of the natural inclusive permutations. That is, X comprises A; X comprises B; or if X includes both A and B, then "X includes A or B" is satisfied in any of the preceding instances. Also, as used in this application and the appended claims, the articles "a" and "an" are to be construed generally to mean "one or more" unless otherwise specified to indicate a singular form or clear from the context. Also, the use of the terms “one embodiment” or “an embodiment” throughout is not intended to mean so unless described to mean the same embodiment or embodiment.

[00168] Transmitting station 102 and/or receiving station 106 (and algorithms, methods, instructions, etc. stored thereon and/or executed by, including by, encoder 400 and decoder 500 ) Implementations of may be realized in hardware, software, or any combination thereof. Hardware may include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the foregoing hardware, alone or in combination. The terms "signal" and "data" are used interchangeably. Also, portions of the transmitting station 102 and the receiving station 106 are not necessarily implemented in the same way.

[00169] Further, in an aspect, for example, the transmitting station 102 or the receiving station 106 has a computer program that, when executed, performs any of the respective methods, algorithms, and/or instructions described herein. It may be implemented using a general-purpose computer or general-purpose processor. Additionally or alternatively, a dedicated computer/processor including, for example, other hardware for performing any of the methods, algorithms, or instructions described herein may be used.

[00170] The transmitting station 102 and the receiving station 106 may be implemented on computers of a videoconferencing system, for example. Alternatively, the transmitting station 102 may be implemented on a server and the receiving station 106 may be implemented on a device separate from the server, such as a hand-held communication device. In this case, the transmitting station 102 may encode the content into an encoded video signal using the encoder 400 and transmit the encoded video signal to the communication device. In turn, the communication device can then decode the encoded video signal using the decoder 500 . Alternatively, the communication device may decode content stored locally on the communication device, eg, content not transmitted by the transmitting station 102 . Other suitable transmit and receive implementation schemes are also available. For example, the receiving station 106 may be a generally stationary personal computer rather than a portable communication device and/or the device including the encoder 400 may also include the decoder 500 .

[00171] Further, all or some of the embodiments of the present invention may take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium may be, for example, any device capable of tangibly containing, storing, communicating, or carrying a program for use by or in connection with any processor. . The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[00172] Additional implementations are summarized in the examples below.

[00173] Example 1: determining a first frame to be predicted in a video sequence; determining a first reference frame from the video sequence for forward inter prediction of the first frame; determining a second reference frame from the video sequence for backward inter prediction of the first frame; generating an optical flow reference frame for inter prediction of the first frame by performing optical flow estimation using the first reference frame and the second reference frame; and performing a prediction process on the first frame using the lightflow reference frame.

[00174] Example 2: The method of example 1, wherein generating the light flow reference frame includes: performing light flow estimation by minimizing a Lagrangian function for each pixel of the first frame.

[00175] Example 3: The method of examples 1 or 2, wherein the lightflow estimation generates respective motion fields for pixels of a first frame, and generating the lightflow reference frame comprises: forming a first warped reference frame warping the first reference frame to the first frame using the motion fields to warping the second reference frame into the first frame using the motion fields to form a second warped reference frame; and blending the first warped reference frame and the second warped reference frame to form an optical flow reference frame.

[00176] Example 4: The method of example 3, wherein blending the first warped reference frame and the second warped reference frame comprises: between the first reference frame and the second reference frame and between the current frame and the first reference frame and the second reference and combining the collocated pixel values by scaling the collocated pixel values of the first warped reference frame and the second warped reference frame using the distances between each frame.

[00177] Example 5: The method of Examples 3 or 4, wherein blending the first warped reference frame and the second warped reference frame comprises: combining collocated pixel values of the first warped reference frame and the second warped reference frame. filling the pixel positions of the lightflow reference frame either by combining or using a single pixel value of one of the first warped reference frame or the second warped reference frame.

[00178] Example 6: The method of any of Examples 3-5, the method further comprising: detecting occlusion in the first reference frame using the first warped reference frame and the second warped reference frame; Blending the first warped reference frame and the second warped reference frame includes: filling pixel positions of the lightflow reference frame corresponding to the occlusion with pixel values from the second warped reference frame.

[00179] Example 7: The method of any of Examples 1-6, wherein performing the prediction process includes: using the opticalflow reference frame only for single reference inter prediction of blocks of the first frame.

[00180] Example 8: The method of any of examples 1-7, wherein the first reference frame is the closest reconstructed frame in display order of the video sequence to the first frame available for forward inter prediction of the first frame, The 2 reference frame is the closest reconstructed frame in display order to the first frame available for backward inter prediction of the first frame.

[00181] Example 9: The method of any of Examples 1-8, wherein performing the prediction process comprises: determining a reference block in an opticalflow reference frame collocated with a first block of a first frame; and encoding the residual of the reference block and the first block.

[00182] Example 10: A device includes: a processor; and a non-transitory storage medium comprising instructions executable by a processor to perform the method, the method comprising: determining a first frame to be predicted in a video sequence; determining availability of a first reference frame for forward inter prediction of the first frame and availability of a second reference frame for backward inter prediction of the first frame; In response to determining the availability of both the first reference frame and the second reference frame: respective motions for pixels of the first frame using the first reference frame and the second reference frame using lightflow estimation creating a field; warping the first reference frame into the first frame using the motion fields to form a first warped reference frame; warping the second reference frame into the first frame using the motion fields to form a second warped reference frame; and blending the first warped reference frame and the second warped reference frame to form an optical flow reference frame for inter prediction of blocks of the first frame.

[00183] Example 11: The apparatus of example 10, the method further comprising: performing a prediction process on the first frame using the opticalflow reference frame.

[00184] Example 12: The apparatus of examples 10 or 11, the method further comprising: using the opticalflow reference frame only for single reference inter prediction of blocks of the first frame.

[00185] Example 13: The apparatus of any one of examples 10-12, wherein generating each motion field comprises: a Lagrangian for respective pixels of the first frame using the first reference frame and the second reference frame and calculating the output of the function.

[00186] Example 14: The apparatus of example 13, wherein calculating the output of the Lagrangian function comprises: calculating a first set of motion fields for pixels of the current frame using the first value for the Lagrangian parameter; and using the first set of motion fields as input to a Lagrangian function that uses the second value for the Lagrangian parameter to compute a set of enhanced motion fields for pixels of the current frame, The second value for the Lagrangian parameter is smaller than the first value for the Lagrangian parameter, and the first warped reference frame and the second warped reference are warped using the set of enhanced motion fields.

[00187] Example 15: A device includes: a processor; and a non-transitory storage medium comprising instructions executable by a processor to perform the method, the method comprising: initializing motion fields for pixels of a first frame at a first processing level for lightflow estimation by: wherein the first processing level represents downscaled motion within the first frame and comprises one of a number of levels of the video sequence using a first reference frame from the video sequence and a second reference frame of the video sequence. generating an optical flow reference frame for inter prediction of the first frame; for each one of the plurality of levels: warping the first reference frame into the first frame using the motion fields to form a first warped reference frame; warping the second reference frame into the first frame using the motion fields to form a second warped reference frame; estimating motion fields between the first warped reference frame and the second warped reference frame using optical flow estimation; and updating motion fields for pixels of the first frame using the motion fields between the first warped reference frame and the second warped reference frame; for a last level of the plurality of levels: warping the first reference frame to the first frame using the updated motion fields to form a final first warped reference frame; warping the second reference frame to the first frame using the updated motion fields to form a final second warped reference frame; and blending the final first warped reference frame and the second warped reference frame to form a lightflow reference frame.

[00188] Example 16: The apparatus of example 15, wherein the lightflow estimation uses a Lagrangian function for each pixel of the frame.

[00189] Example 17: The apparatus of example 16, the method, for each one of the plurality of levels: warping the first reference frame, warping the second reference frame, estimating motion fields, and a motion field initializing a Lagrangian parameter of the Lagrangian function to a maximum value for a first iteration of updating s; and using increasingly smaller of the set of possible values for the Lagrangian parameter, warping the first reference frame, warping the second reference frame, estimating motion fields, and updating the motion fields. It further includes the step of performing an additional iteration of the step (additional iteration).

[00190] Example 18: The apparatus of examples 16 or 17, wherein estimating the motion fields comprises: calculating derivatives of pixel values of the first warped reference frame and the second warped reference frame with respect to a horizontal axis, a vertical axis, and time ; downscaling the derivatives in response to the level being different from the final level; and solving linear equations representing the Lagrangian function using the derivatives.

[00191] Example 19: The apparatus of any one of examples 15-18, the method further comprising inter-predicting the first frame using the opticalflow reference frame.

[00192] Example 20: The apparatus of any one of examples 15-19, wherein the processor and the non-transitory storage medium form a decoder.

[00193] The foregoing examples, implementations, and aspects have been described to facilitate understanding of the invention and do not limit the invention. On the contrary, this invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation so as to cover all such modifications and equivalent structures permitted by law. do.

Claims (20)

determining a first frame to be predicted in the video sequence;
determining a first reference frame from the video sequence for forward inter prediction of the first frame;
determining a second reference frame from the video sequence for backward inter prediction of the first frame;
generating an optical flow reference frame for inter prediction of the first frame by performing optical flow estimation using the first frame, the first reference frame, and the second reference frame; performing light flow estimation includes performing regression analysis while maintaining the first reference frame and the second reference frame at original scales;
performing an inter prediction process on the first frame using the optical flow reference frame;
containing,
Way. According to claim 1,
The generating the light flow reference frame includes performing the light flow estimation by minimizing the Lagrangian function for each pixel of the first frame.
Way. According to claim 1,
The light flow estimation generates each motion field for pixels of the first frame, and the step of generating the light flow reference frame comprises:
warping the first reference frame into the first frame using motion fields to form a first warped reference frame;
warping the second reference frame to the first frame using the motion fields to form a second warped reference frame; and
blending the first warped reference frame and the second warped reference frame to form the light flow reference frame;
containing,
Way. 4. The method of claim 3,
The blending of the first warped reference frame and the second warped reference frame may include: between the first reference frame and the second reference frame and between the first frame and the first reference frame and the second reference frame. combining the collocated pixel values by scaling the collocated pixel values of the first warped reference frame and the second warped reference frame using distances between each frame;
Way. 4. The method of claim 3,
The blending of the first warped reference frame and the second warped reference frame may include combining collocated pixel values of the first warped reference frame and the second warped reference frame or the first warped reference frame. populating pixel positions of the lightflow reference frame by using a single pixel value of either a reference frame or the second warped reference frame.
Way. 4. The method of claim 3,
detecting occlusion in the first reference frame using the first warped reference frame and the second warped reference frame;
The blending of the first warped reference frame and the second warped reference frame may include filling pixel positions of the lightflow reference frame corresponding to the occlusion with pixel values from the second warped reference frame. containing,
Way. According to claim 1,
The step of performing the prediction process comprises:
generating a prediction block for the current block of the first frame by performing a motion search within the optical flow reference frame; or
By using a motion vector decoded from an encoded bitstream to generate a predictive block using pixels of the lightflow reference frame,
using the optical flow reference frame only for single reference inter prediction of blocks of the first frame,
Way. According to claim 1,
The first reference frame is a reconstruction frame that is closest to the first frame in a display order of the video sequence and can be used for forward inter prediction of the first frame,
The second reference frame is a reconstruction frame that is closest to the first frame in the display order and can be used for backward inter prediction of the first frame,
Way. According to claim 1,
The step of performing the prediction process comprises:
determining a reference block collocated with a first block of the first frame in the light flow reference frame; and
encoding the residual of the reference block and the first block;
containing,
Way. An apparatus comprising a processor and a non-transitory storage medium, comprising:
The non-transitory storage medium includes instructions executable by the processor to perform a method, the method comprising:
determining a first frame to be predicted in the video sequence;
determining availability of a first reference frame for forward inter prediction of the first frame and a second reference frame for backward prediction of the first frame;
In response to determining that both the first reference frame and the second reference frame are available,
generating respective motion fields for pixels of the first frame using the first reference frame and the second reference frame using a pyramid structure for light flow estimation, wherein the motion field comprises the first scaling derivatives of pixel values of the first reference frame and the second reference frame with respect to a horizontal axis, a vertical axis, and time, instead of pixel values of the reference frame and the second reference frame;
warping the first reference frame into the first frame using motion fields to form a first warped reference frame;
warping the second reference frame into the first frame using the motion fields to form a second warped reference frame; and
forming an optical flow reference frame for inter prediction of blocks of the first frame by blending the first warped reference frame and the second warped reference frame
containing,
Device. 11. The method of claim 10,
The method further comprises performing a prediction process using inter prediction for the first frame using the optical flow reference frame,
The prediction process is
generating a prediction block for a current block of the first frame by performing a motion search within the optical flow reference frame; or
Using a motion vector decoded from an encoded bitstream to generate a predictive block using pixels of the lightflow reference frame.
containing one of
Device. 11. The method of claim 10,
The method further comprises excluding the optical flow reference frame from the composite inter prediction of blocks of the first frame,
Device. 11. The method of claim 10,
generating each motion field comprises calculating an output of a Lagrangian function for each pixel of the first frame using the first reference frame and the second reference frame;
Device. 14. The method of claim 13,
Calculating the output of the Lagrangian function comprises:
calculating a first set of motion fields for pixels of a current frame using the first value for the Lagrangian parameter; and
using the first set of motion fields as input to a Lagrangian function using a second value for the Lagrangian parameter to compute an improved set of motion fields for the pixels of the current frame.
including,
The second value for the Lagrangian parameter is smaller than the first value for the Lagrangian parameter, and the first warped reference frame and the second warped reference frame are warped using the improved set of motion fields. felled,
Device. 11. The method of claim 10,
The method is
performing a synthetic reference inter prediction process on a first block of the first frame using a forward reference frame and a backward reference frame other than the optical flow reference frame; and
performing a single reference inter prediction process for the first block of the first frame using the optical flow reference frame;
further comprising,
Device. A device comprising a processor, comprising:
The processor is
generating an optical flow reference frame for inter prediction of a first frame of the video sequence using a first reference frame from the video sequence and a second reference frame of the video sequence; and
performing an inter prediction process of at least one block of the first frame using the optical flow reference frame;
configured to perform a method comprising:
The step of generating the light flow reference frame includes:
initializing motion fields for pixels of a first frame at a first processing level for lightflow estimation, wherein the first processing level is indicative of downscaled motion within the first frame and of one of a plurality of levels. including level ―;
For each level of the plurality of levels:
warping the first reference frame into the first frame using the motion fields to form a first warped reference frame;
warping the second reference frame into the first frame using the motion fields to form a second warped reference frame;
estimating motion fields between the first warped reference frame and the second warped reference frame using the optical flow estimation; and
updating motion fields for pixels of the first frame using motion fields between the first warped reference frame and the second warped reference frame;
For the last one of the multiple levels:
warping the first reference frame into the first frame using the updated motion fields to form a final first warped reference frame;
warping the second reference frame to the first frame using the updated motion fields to form a final second warped reference frame; and
forming the light flow reference frame by blending the final first warped reference frame and the second warped reference frame;
made by
The light flow estimate is based on the first reference relative to a horizontal axis, a vertical axis, and time, instead of scaling pixel values of the first reference frame and the second reference frame for levels below the final level. a regression analysis performed while the first reference frame and the second reference frame are maintained at their original scale by scaling the derivatives of the pixel values of the frame and the second reference frame;
Device. 17. The method of claim 16,
The light flow estimation uses a Lagrangian function for each pixel of a frame,
Device. 18. The method of claim 17,
The method includes: for each level of the plurality of levels:
Lagrangian parameter of the Lagrangian function for a first iteration of warping the first reference frame, warping the second reference frame, estimating the motion fields, and updating the motion fields initializing to a maximum value; and
warping the first reference frame using increasingly smaller of a set of possible values for the Lagrangian parameter, warping the second reference frame, estimating the motion fields, and performing additional iterations of updating the motion fields.
further comprising,
Device. 19. The method of claim 18,
Estimating the motion fields comprises:
calculating the derivatives;
downscaling the derivatives in response to the level being different from the final level; and
Solving linear equations representing the Lagrangian function using the derivatives
containing,
Device. 17. The method of claim 16,
The processor forms a decoder,
Device.

Images