KR102537087B1 - Motion estimation using 3D auxiliary data - Google Patents

Abstract

A method, system and apparatus comprising a computer program encoded on a computer storage medium for performing motion estimation. In some implementations, the method includes generating a segmentation of the point cloud data based on continuity data of the point cloud data. The segmented representation of point cloud data is projected onto the sides of a 3D bounding box. A patch is created based on the projected representation of the segmented point cloud data. The first frame of the patch is created. The first and second auxiliary information are generated using the first frame and the reference frame. A first patch from the first frame that matches the patch from the reference frame is identified based on the first and second auxiliary information. A motion vector candidate is generated between the first patch and the second patch based on the difference between the first and second auxiliary information. Motion compensation is performed using motion vector candidates.

Description

Motion estimation using 3D auxiliary data

This application is a U.S. Provisional Patent Application No. 2, filed October 2, 2018. 62/740,237 and U.S. Provisional Patent Application No. 62/863,362, both of which are incorporated herein by reference in their entirety.

Point cloud processing has become an integral part of a variety of applications in applications such as the entertainment industry, intelligent car navigation, geospatial inspection, three-dimensional (3-D) modeling of real objects and environment visualization.

In some implementations, the specification describes techniques for performing motion estimation using three-dimensional and two-dimensional auxiliary data. Motion estimation is performed to encode and transmit 3D point cloud data. Three-dimensional point cloud data includes data points that outline or visually represent the outer surface of a three-dimensional object, such as a human or real-world item. The 3D point cloud data may also include attribute information representing color, texture, and depth of the 3D point cloud data. An encoder or decoder may respectively encode or decode the 3D point cloud data using the motion enhancement data.

In some implementations, an encoder or decoder uses a 3-dimensional bounding box to enclose the 3-dimensional point cloud data and then generates patches used for encoding and transmission. The encoder may project images of the 3D point cloud data onto each side of the 3D bounding box. Encoders can group images or patches into frames to be used for encoding. To reduce the amount of bandwidth typically used to transmit typically large three-dimensional point cloud data, an encoder can instead compare a patch of a currently generated frame with a patch of a previously generated frame to generate motion improvement data. . The encoder can match the patches between the two frames and generate motion improvement data based on the data identifying the matching patches. For example, the encoder may use position coordinates and side information defining the size of the patch and include it as motion enhancement data. Instead of encoding and transmitting patch frames, motion enhancement data can be used for encoding and transmission to reduce overall transmission bandwidth and allow messages to be properly decoded and received. Once the motion segmentation data is identified, the motion segmentation data can be added to existing video compression techniques to improve 3D point cloud data transmission.

In one general aspect, the method includes: generating a segmentation of three-dimensional point cloud data of recorded media based on continuity data of the three-dimensional point cloud data; Projecting the segmented representation of 3-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, wherein the segmented representation of 3-dimensional point cloud data is projected onto the projection of the 3-dimensional bounding box. -Different based on aspect; generating one or more patches based on the projected representation of the segmented three-dimensional point cloud data; generating a first frame of the one or more patches; generating first auxiliary information for the first frame; generating second auxiliary information for the reference frame; Between the first patch and the second patch based on the difference between the second patch from the reference frame and the second auxiliary information from the first frame that matches the second patch from the reference frame based on the first auxiliary information and the second auxiliary information. Generating a motion vector candidate of ; and performing motion compensation using the motion vector candidate.

Other embodiments of this disclosure and other aspects of this disclosure include corresponding systems, apparatus and computer programs configured to perform the operations of the methods, encoded on a computer storage device. A system composed of one or more computers may be configured through software, firmware, hardware, or a combination of these installed on the system to enable the system to perform tasks. One or more computer programs may be configured by having instructions that, when executed by a data processing device, cause the device to perform actions.

Each of the foregoing and other embodiments may include one or more of the following features, singly or in combination. For example, one embodiment includes all of the following features in combination.

In some embodiments, the method includes the reference frame being transmitted and corresponding to a previously encoded frame and being decoded to generate the second auxiliary information.

In some embodiments, generating the segmentation of the three-dimensional point cloud data of the recorded media comprises: the one or more processors to subsequently project and encode each segment of the plurality of segmentation the three-dimensional point cloud media Further comprising generating a plurality of partitions for .

In some embodiments, the first auxiliary information includes index data for each of the one or more patches, 2-dimensional data for each of the one or more patches, and 3-dimensional data for each of the one or more patches.

In some embodiments, the index data for each of the one or more patches corresponds to a corresponding side of the three-dimensional bounding box.

In some embodiments, the 2-dimensional data for each of the one or more patches and the 3-dimensional data for each of the one or more patches correspond to a portion of the 3-dimensional point cloud data.

In some embodiments, generating one or more patches of the 3D point cloud data based on continuity data of the 3D point cloud data includes: the one or more processors performing a smoothness of the 3D point cloud data from each direction. determining smoothness criteria; comparing, by the one or more processors, the smoothness criteria from each direction of the 3D point cloud data; and in response to the comparison, the one or more processors selecting a direction of the smoothness criterion of the three-dimensional point cloud data having a larger projection area on the side of the bounding box.

In some embodiments, the generating of the motion vector candidate between the first patch and the second patch includes: the one or more processors between the two-dimensional data of the first auxiliary information and the two-dimensional data of the second auxiliary information. determining the distance of; generating the motion vector candidate based on a distance between the 2D data of the first auxiliary information and the 2D data of the second auxiliary information; and adding the motion vector candidate to a motion vector candidate list.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the subject matter will be apparent from the description, drawings and claims.

1 is a flow diagram of an exemplary method of coding a video signal.
2 is a schematic diagram of an exemplary coding and decoding (codec) system for video coding.
3 is a block diagram illustrating an exemplary video encoder.
4 is a block diagram illustrating an exemplary video decoder.
5 is a schematic diagram illustrating an example of unidirectional inter-prediction.
6 is a schematic diagram illustrating an example of bi-directional inter-prediction.
7 is a schematic diagram illustrating an exemplary intra-prediction mode used for video coding.
8 is a schematic diagram illustrating an example of a directional relationship of blocks in video coding.
9 is a block diagram illustrating an example in-loop filter.
10 shows an exemplary split mode used for block splitting.
11 is a schematic diagram of an exemplary video encoding mechanism.
12 is a schematic diagram of a computing device for video coding.
13 is an example of a system illustrating point cloud media.
14 is an example of a system illustrating a point cloud frame sequence.
15 is an example of a process of converting a 3D patch bounding box into a 2D patch projection.
16 is an example of a system showing 3D to 2D patch projection results.
17 is an example of attribute segmentation for cloud point media.
18 is an example of a system illustrating a packaging patch for point cloud media with attribute information.
19 is an example of a system that performs motion estimation.
20 is an example of a system representing a motion vector candidate between a patch of a current frame and a patch of a reference frame.
21 illustrates a derivation process for merging candidate list construction.
22 illustrates a spatial merge candidate and a location system of candidate pairs that are considered for redundancy check of spatial merge candidates.
23 illustrates a system showing a location for a second PU of Nx2N and 2NxN partitions.
24 illustrates obtaining a scaled motion vector for a temporal merge candidate.
25 is a system showing candidate positions for temporal merging candidates
26 illustrates an example table of combined bi-predictive merging candidates.
Figure 27 is an example of a system that has modified the motion estimation pipeline using auxiliary data.
28 is an example of a packet stream representation of a V-PCC unit payload.
29 is another example of a visual representation of a V-PCC unit payload.
30 is a flow diagram illustrating an example of a process for performing motion estimation using 3D auxiliary data.
Like reference numbers and designations in the various drawings indicate like elements.

Although example implementations of one or more embodiments are provided below, it should be understood from the outset that the disclosed systems and/or methods may be implemented using any number of technologies, whether presently known or existing. This disclosure should not be limited to the example implementations, drawings and techniques illustrated below, including the example designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims, along with the full scope of equivalents. It can be.

1 is a flow diagram of an exemplary method 100 for coding a video signal. Specifically, a video signal is encoded in an encoder. The encoding process compresses the video signal using various mechanisms to reduce the video file size. Smaller file sizes allow you to transmit compressed video files to users while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process so that the decoder can consistently reconstruct the video signal.

In step 101, a video signal is input to an encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, a video file can be captured by a video capture device, such as a video camera, and encoded to support live streaming of video. A video file may include both an audio component and a video component. The video component contains a series of image frames, which when viewed sequentially give the visual impression of motion. A frame includes pixels represented by light, referred to herein as a luma component, and color, referred to as a chroma component. In some examples, a frame may also include a depth value to support a three-dimensional view.

In step 103, the video is partitioned into blocks. Partitioning involves subdividing the pixels of each frame into square and/or rectangular blocks for compression. For example, a coding tree can be used to divide a block and then recursively subdivide it until a configuration that supports further encoding is achieved. As such, a block may be referred to as a coding tree unit in High Efficiency Video Coding (HEVC) (also referred to as H.265 and MPEG-H Part 2). For example, the luma component of a frame can be subdivided until individual blocks contain relatively uniform illumination values. Also, the chroma components of a frame can be subdivided until individual blocks contain relatively uniform color values. Therefore, the segmentation mechanism depends on the content of the video frame.

In step 105, various compression mechanisms are used to compress the image block segmented in step 103. For example, inter-prediction and/or intra-prediction may be used. Inter-prediction is designed to exploit the fact that objects in a common scene tend to appear in successive frames. Thus, a block depicting an object in a reference frame does not need to be repeatedly described in a subsequent frame. Specifically, an object such as a table may remain in a constant position over several frames. Thus, the table only needs to be described once, and later frames can refer back to the reference frame. A pattern matching mechanism can be used to match objects across multiple frames. Also, a moving object may be expressed over several frames due to object movement or camera movement, for example. As a specific example, a video may show a car moving across the screen over several frames. Motion vectors can be used to describe these movements. A motion vector is a two-dimensional vector that provides an offset from the object's coordinates in the frame to the object's coordinates in the reference frame. As such, inter-prediction may encode an image block of a current frame into a set of motion vectors representing an offset from that block of a reference frame.

Intra-prediction encodes blocks of a common frame. Intra-prediction takes advantage of the fact that luma and chroma components tend to cluster in one frame. For example, green patches on parts of trees tend to be located next to similar green patches. Intra-prediction uses a multi-directional prediction mode (eg, 33 in HEVC), a planar mode, and a direct current (DC) mode. The direction mode indicates that the current block is similar/identical to an adjacent block in that direction. Planar mode indicates that a series of blocks (e.g., planes) along a row may be interpolated based on adjacent blocks at the row edges. Effectively flat mode represents a smooth transition of light/color across the rows, using a relatively constant gradient as you change values. The DC mode is used for boundary smoothing and indicates that a block is similar/equal to the average value associated with all adjacent blocks relative to the angular orientation of the direction prediction mode. Accordingly, an intra-prediction block may represent an image block with various relational prediction mode values instead of actual values. Also, the inter-prediction block may represent an image block as a motion vector value instead of an actual value. In both cases, the prediction block may not accurately represent the image block in some cases. All differences are stored in the residual block. A transform can be applied to the remaining blocks to further compress the file.

At step 107, various filtering techniques may be applied. In HEVC, filters are applied according to the in-loop filtering method. The block-based prediction discussed above can result in the creation of block images at the decoder. In addition, the block-based prediction scheme can reconstruct the encoded block for later use as a reference block after encoding the block. The in-loop filtering scheme repeatedly applies a noise suppression filter, a deblocking filter, an adaptive loop filter and a SAO filter to blocks/frames. These filters mitigate these blocking artifacts so that encoded files can be accurately reconstructed. Additionally, these filters mitigate artifacts in the reconstructed reference block to reduce the likelihood that artifacts will create additional artifacts in subsequent blocks that are encoded based on the reconstructed reference block. The in-loop filtering process is detailed below.

Once the video signal is split, compressed and filtered, the resulting data is encoded into a bitstream at step 109. The bitstream contains the data discussed above as well as any signaling data desired to support proper video signal reconstruction at the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission towards the decoder upon request. A bitstream may also be broadcast and/or multicast towards multiple decoders. Generating a bitstream is an iterative process. Thus, steps 101, 103, 105, 107 and 109 may occur sequentially and/or concurrently over many frames and blocks. The order shown in FIG. 1 is presented for clarity and ease of discussion, and is not intended to limit the video coding process to any particular order.

The decoder receives the bitstream and starts the decoding process at step 111 . Specifically, the decoder applies an entropy decoding scheme to convert a bitstream into corresponding syntax and video data. The decoder applies syntax data from the bitstream to determine the partition for the frame at step 111 . Partitioning must match the result of block partitioning in step 103. Entropy encoding/decoding as applied in step 111 is described. The encoder makes many choices during the compression process, such as selecting a block-partitioning scheme from among several possible choices based on spatial positioning values in the input image(s). Many bins can be used to signal correct selection. As used herein, a bin is a binary value that is treated as a variable (eg, a bit value that may vary depending on context). Entropy coding allows the encoder to discard clearly infeasible options for a particular case, leaving only an acceptable set of options. A code word is then assigned to each permitted option. The length of the codeword is based on the number of options allowed (eg, one bin for two options, two bins for three or four options, etc.). The encoder then encodes the codeword for the selected option. This scheme reduces the size of code words because code words are only as large as desired to uniquely represent a choice among a small sub-set of acceptable options, as opposed to uniquely representing a choice among a potentially large set of all possible options. . The decoder then decodes the selection by determining the set of acceptable options in a manner similar to the encoder. By determining the set of acceptable options, the decoder can read the codeword and determine the encoder's choice.

In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding predictive block to reconstruct the image block according to the segmentation. The predictive blocks may include both intra-prediction blocks and inter-prediction blocks as generated at the encoder in step 105 . Then, the reconstructed image block is positioned into a frame of the reconstructed video signal according to the segmentation data determined in step 111. Step 113 may also be signaled in the bitstream via entropy coding as described above.

In step 115, filtering is performed on frames of the reconstructed video signal in a manner similar to step 107 in the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters may be applied to frames to remove blocking artifacts. Once the frames are filtered, the video signal can be output to a display at step 117 for viewing by an end user.

2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functionality to support implementation of the method 100 . Codec system 200 is generalized to describe components used in both encoders and decoders. Codec system 200 receives and divides the video signal as discussed with respect to steps 101 and 103 of method 100 , resulting in partitioned video signal 201 . As discussed with respect to steps 105, 107 and 109 of method 100, codec system 200, acting as an encoder, compresses partitioned video signal 201 into a coded bitstream. As discussed with respect to steps 111, 113, 115 and 117 of method 100 when operating as decoder codec system 200 generates an output video signal from a bitstream. The codec system (200) includes a general coder control component (211), a transform scaling and quantization component (213), an intra-picture estimation component (215), an intra-picture prediction component (217), a motion compensation component (219), motion estimation component 221, scale and inverse transform component 229, filter control analysis component 227, in-loop filter component 225, decoded picture buffer component 223, and header formatting and context adaptive binary arithmetic coding (header formatting and Context adaptive binary arithmetic coding (CABAC) component 231. These components are combined as shown. In FIG. 2 , black lines represent the movement of data to be encoded/decoded, and dotted lines represent the movement of control data that controls the operation of other components. All of the components of the codec system 200 may reside in an encoder. A decoder may include a subset of the components of codec system 200. For example, the decoder includes an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. can do. These components are now described.

A partitioned video signal 201 is a captured video stream that has been divided into blocks of pixels by a coding tree. The coding tree subdivides blocks of pixels into smaller blocks of pixels using various splitting modes. These blocks can then be further subdivided into smaller blocks. A block may be referred to as a node in a coding tree. Larger parent nodes are partitioned into smaller child nodes. The number of times a node is subdivided is called the depth of the node/coding tree. A divided block is also referred to as a coding unit (CU) in some cases. Split mode is a binary tree (BT), a triple tree (TT) and It may include a quad tree (QT). The partitioned video signal 201 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component for compression ( 221) is forwarded.

The generic coder control component 211 is configured to make decisions related to coding images of a video sequence into a bitstream according to application constraints. For example, the generic coder control component 211 manages optimization of bitrate/bitstream size versus reconstruction quality. This decision may be based on storage space/bandwidth availability and image resolution requests. The generic coder control component 211 also manages buffer utilization in terms of transfer rate to mitigate buffer underrun and overrun problems. To manage these issues, the generic coder control component 211 manages partitioning, prediction and filtering by other components. For example, the generic coder control component 211 can dynamically increase compression complexity to increase resolution and increase bandwidth usage or decrease compression complexity to decrease resolution and bandwidth usage. Thus, the generic coder control component 211 controls the other components of the codec system 200 to balance video signal reconstruction quality and bit rate issues. A generic coder control component 211 generates control data that controls the operation of other components. Control data is also forwarded to the header formatting and CABAC component 231 to be encoded in the bitstream to signal parameters for decoding at the decoder.

The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter-prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. Motion estimation component 221 and motion compensation component 219 perform inter-predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. Codec system 200 may perform multiple coding passes to select an appropriate coding mode for each block of video data, for example.

Motion estimation component 221 and motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation, performed by motion estimation component 221, is the process of generating motion vectors that estimate motion for a video block. For example, a motion vector represents the displacement of a predictive unit (PU) of a video block relative to a predictive block within a reference frame (or other coded unit) relative to a current block being coded within the current frame (or other coded unit). can A predictive block is one that closely matches the block to be coded in terms of pixel differences, which can be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metric. block is confirmed. In some examples, the codec system 200 may count values for sub-integer pixel positions of reference pictures stored in the decoded picture buffer 223 . For example, the video codec system 200 may interpolate values of quarter pixel positions, eighth pixel positions, or other fractional pixel positions of a reference picture. Thus, the motion estimation component 221 can perform a motion search for full and sub-pixel locations and output motion vectors with sub-pixel precision. Motion estimation component 221 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The motion estimation component 221 outputs the calculated motion vectors as motion data to a header format and CABAC component 231 for motion and encoding to a motion compensation component 219 .

Motion compensation performed by motion compensation component 219 can include fetching or generating a predictive block based on the motion vector determined by motion estimation component 221 . Again, motion estimation component 221 and motion compensation component 219 may be functionally integrated in some examples. Upon receiving the motion vector for the PU of the current video block, motion compensation component 219 can find the predictive block to which the motion vector points to a reference picture list. The residual video block is formed by subtracting the pixel value of the predictive block from the pixel value of the current video block being coded to form a pixel difference value. In general, motion estimation component 221 performs motion estimation for luma components, and motion compensation component 219 uses motion vectors calculated based on luma components for both chroma and luma components. The predictive block and residual block are forwarded to a transform scaling and quantization component (213).

The partitioned video signal 201 is also sent to an intra-picture estimation component 215 and an intra-picture prediction component 217 . Like motion estimation component 221 and motion compensation component 219, intra-picture estimation component 215 and intra-picture prediction component 217 may be highly integrated, but are described separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217, as an alternative to the inter-prediction performed by the inter-frame motion estimation component 221 and motion compensation component 219 as described above, Intra-predict the current block for blocks in the current frame. In particular, intra-picture estimation component 215 determines the intra-prediction mode to use for encoding the current block. In some examples, intra-picture estimation component 215 selects an appropriate intra-prediction mode to encode the current block from a number of tested intra-prediction modes. The selected intra-prediction mode is forwarded to the header formatting and CABAC component 231 for encoding.

For example, the intra-picture estimation component 215 calculates a rate-distortion value using rate-distortion analysis for the various intra-prediction modes tested, and selects an intra-distortion value with the best rate-distortion characteristics among the tested modes. Choose a prediction mode. Rate distortion analysis is generally performed on the amount of distortion (or error) between the original unencoded block and the encoded block that was encoded to produce the encoded block, as well as the bitrate used to generate the encoded block (e.g., number of bits). Intra-picture estimation component 215 calculates ratios from the rates and distortions for the various encoded blocks to determine the intra-prediction mode that exhibits the best rate-distortion values for the blocks. Additionally, the intra-picture estimation component 215 can be configured to code depth blocks of the depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

Intra-picture prediction component 217 can generate a residual block from the predictive block based on the selected intra-prediction mode determined by intra-picture estimation component 215 . The residual block contains the difference in value between the predicted block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. Intra-picture estimation component 215 and intra-picture prediction component 217 can operate on both luma and chroma components.

The transform scaling and quantization component 213 is configured to further compress the residual block. Transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transform, to the residual block to produce a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, sub-band transforms, or other types of transforms can also be used. A transform may transform the residual information from a pixel value domain to a transform domain such as a frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example based on frequency. This scaling involves applying a scale factor to the residual information so that different frequency information is quantized at different granularity, which can affect the final visual quality of the reconstructed video. Transform scaling and quantization component 213 is also configured to quantize transform coefficients to further reduce bitrate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameter. In some examples, transform scaling and quantization component 213 can perform a scan of the matrix including the quantized transform coefficients. The quantized transform coefficients are forwarded to the header formatting and CABAC component 231 to be encoded into the bitstream.

The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. Scaling and inverse transform component 229 applies inverse scaling, transform and/or quantization to reconstruct the residual block in the pixel domain for later use as a reference block, which can be, for example, a predictive block for another current block. do. Motion estimation component 221 and/or motion compensation component 219 can count the reference block by adding the residual block back to the corresponding prediction block for use in motion estimation of a subsequent block/frame. A filter is applied to the reconstructed reference block to mitigate artifacts created during scaling, quantization and transformation. Otherwise, when subsequent blocks are predicted, these artifacts can lead to inaccurate predictions and create additional artifacts.

Filter control analysis component 227 and in-loop filter component 225 apply filters to residual blocks and/or reconstructed image blocks. For example, the transformed residual block from scaling and inverse transform component 229 is combined with the corresponding prediction block from intra-picture prediction component 217 and/or motion compensation component 219 to reconstruct the original image block. It can be. Filters can then be applied to the reconstructed image blocks. In some examples, a filter may be applied to the residual block instead. Like the other components in FIG. 2 , filter control analysis component 227 and in-loop filter component 225 are highly integrated and can be implemented together, but are shown separately for conceptual purposes. Filters applied to the reconstructed reference block are applied to a specific spatial region and contain several parameters that control how these filters are applied. The filter control analysis component 227 analyzes the reconstructed reference block to determine where such filters should be applied and sets the corresponding parameters. This data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. In-loop filter component 225 applies these filters based on the filter control data. Filters may include deblocking filters, noise suppression filters, SAO filters and adaptive loop filters. Such a filter may be applied in the spatial/pixel domain (eg, to the reconstructed pixel block) or in the frequency domain, depending on the example.

When operating as an encoder, the filtered reconstructed image blocks, residual blocks and/or prediction blocks are stored in the decoded picture buffer 223 for later use in motion estimation as discussed above. Acting as a decoder, the decoded picture buffer 223 stores and forwards the reconstructed and filtered blocks towards the display as part of the output video signal. The decoded picture buffer 223 may be any memory device capable of storing prediction blocks, residual blocks and/or reconstructed image blocks.

Header formatting and CABAC component 231 receives data from the various components of codec system 200 and encodes this data into a coded bitstream for transmission towards a decoder. Specifically, header formatting and CABAC component 231 generates various headers to encode control data such as general control data and filter control data. In addition, intra-prediction and prediction data including motion data as well as residual data in the form of quantized transform coefficient data are all encoded into the bitstream. The final bitstream contains all information required by the decoder to reconstruct the original partitioned video signal 201 . This information may also include intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of encoding contexts for various blocks, indications of most probable intra-prediction modes, indications of partition information, and the like. Such data may be encoded using entropy coding. For example, the information may include context adaptive variable length coding (CAVLC), CABAC and syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy partitioning entropy (PIPE) coding or other entropy coding techniques. Following entropy coding, the coded bitstream may be transmitted to another device (eg, a video decoder) or archived for later transmission or retrieval.

3 is a block diagram illustrating an exemplary video encoder 300 . Video encoder 300 may be used to implement the encoding function of codec system 200 and/or to implement steps 101 , 103 , 105 , 107 and/or 109 of method 100 . The encoder (300) divides the input video signal to produce a partitioned video signal (301), which is substantially similar to the partitioned video signal (201). The partitioned video signal 301 is compressed by components of the encoder 300 and encoded into a bitstream.

Specifically, the partitioned video signal 301 is forwarded to an intra-picture prediction component 317 for intra-prediction. Intra-picture prediction component 317 may be substantially similar to intra-picture estimation component 215 and intra-picture prediction component 217 . The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter-prediction based on the reference block of the decoded picture buffer 323 . Motion compensation component 321 may be substantially similar to motion estimation component 221 and motion compensation component 219 . The prediction block and residual block from intra-picture prediction component 317 and motion compensation component 321 are forwarded to transform and quantization component 313 for transform and quantization of the residual block. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213 . The transformed and quantized residual block and the corresponding predictive block (along with associated control data) are forwarded to the entropy coding component 331 to be coded into a bitstream. Entropy coding component 331 may be substantially similar to header formatting and CABAC component 231 .

The transformed and quantized residual block and/or the corresponding prediction block is also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstructing into a reference block for use by the motion compensation component 321. do. Inverse transform and quantization component 329 may be substantially similar to scaling and inverse transform component 229 . The in-loop filter of the in-loop filter component 325 is also applied to the residual block and/or the reconstructed reference block according to an example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225 . In-loop filter component 325 may include multiple filters including noise suppression filters as discussed below. The filtered block is stored in the decoded picture buffer 323 for use as a reference block by the motion compensation component 321 . The decoded picture buffer 323 may be substantially similar to the decoded picture buffer 223 .

4 is a block diagram illustrating an exemplary video decoder 400 . Video decoder 400 may be used to implement the decoding function of codec system 200 and/or to implement steps 111 , 113 , 115 and/or 117 of method 100 . Decoder 400 generates a reconstructed output video signal based on, for example, a bitstream from encoder 300 for display to an end user.

The bitstream is received by entropy decoding component 433. Entropy decoding component 433 performs the inverse function of entropy encoding component 331 . Entropy decoding component 433 is configured to implement entropy decoding schemes such as CAVLC, CABAC, SBAC and PIPE coding or other entropy coding techniques. For example, entropy decoding component 433 can use the header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, partition information, motion data, prediction data and quantized transform coefficients from the residual block. The quantized transform coefficients are forwarded to an inverse transform and quantization component 429 for reconstruction into a residual block. Inverse transform and quantization component 429 may be substantially similar to inverse transform and quantization component 329 .

The reconstructed residual blocks and/or prediction blocks are forwarded to intra-picture prediction component 417 for reconstruction into image blocks based on intra-prediction operations. Intra-picture prediction component 417 can be substantially similar to intra-picture prediction component 317 but operates in reverse. Specifically, intra-picture prediction component 417 uses a prediction mode to find a reference block in a frame and applies the residual block to the result to reconstruct an intra-predicted image block. The reconstructed intra-predicted image blocks and/or residual blocks and the corresponding inter-prediction data are forwarded through an in-loop filter component 425 to a decoded picture buffer component 423, which is a decoded picture buffer component ( 323) and in-loop filter component 325, respectively. An in-loop filter component 425 filters the reconstructed image blocks, residual blocks and/or predictive blocks, and this information is stored in a decoded picture buffer component 423 . Reconstructed image blocks from the decoded picture buffer component 423 are forwarded to motion compensation. Components for Inter-Prediction 421. Motion compensation component 421 can be substantially similar to motion compensation component 321, but can operate in reverse. Specifically, motion compensation component 421 reconstructs an image block by generating a prediction block using a motion vector from a reference block and applying a residual block to the result. The resulting reconstructed block may also be forwarded through an in-loop filter component 425 to a decoded picture buffer component 423. The decoded picture buffer component 423 continues to store additional reconstructed image blocks that can be reconstructed into frames via segmentation information. These frames may be placed in a sequence. The sequence is output to the display as a reconstructed output video signal.

Inter-prediction.

Many schemes are used together to compress video data during the video coding process. For example, a video sequence is partitioned into image frames. The image frame is then partitioned into image blocks. The image blocks can then be compressed by inter-prediction (correlation between blocks in different frames) or intra-prediction (correlation between blocks in the same frame).

Inter-prediction is when coding objects such as coding tree units (CTUs), coding tree blocks (CTBs), coding units (CUs), sub-CUs, etc. appear in multiple frames of a video sequence. used Instead of coding the same object in each frame, the object is coded in a reference frame and a motion vector (MV) is used to indicate the motion trajectory of the object. The motion trajectory of an object is the movement of the object over time. MV is a vector representing the direction and size of an object whose position changes between frames. Objects and MVs can be coded into a bitstream and decoded by a decoder. To further increase coding efficiency and reduce encoding size, MVs can be omitted from the bitstream and derived at the decoder. For example, a pair of reference frames may be used. A reference frame is a frame of a bitstream that contains data that can be coded as a reference when coding a related frame. Matching algorithms such as bi-directional matching and/or template matching may be used to determine the location of a coded object in both reference frames. A bidirectional matching algorithm matches blocks in the previous frame with blocks in the current frame. A template matching algorithm matches blocks adjacent to a current block with blocks adjacent to the current block in one or more frames of reference. When the position of the object is determined in the two reference frames, it is possible to determine the MV representing the motion of the object between the reference frames. MVs can then be used to place objects in frames between reference frames. As a specific example, initial MVs may be determined for all CUs. You can then refine the initial MV using local search. Also, the MV for the sub-CU component of the object may be determined and improved based on the improved initial MV. This approach represents the exact location of an object as long as the object's motion trajectory is continuous between reference frames.

5 shows, for example, the motion vector (MV) in block compression step 105, block decoding step 113, motion estimation component 221, motion compensation component 219, motion compensation component 321 and/or motion compensation component 421. ) is a schematic diagram illustrating an example of unidirectional inter-prediction 500 performed to determine

Unidirectional inter-prediction 500 uses a reference frame 530 with a reference block 531 to predict a current block 511 in the current frame 510 . The reference frame 530 may be positioned temporally behind the current frame 510 as shown, but may also be positioned temporally before the current frame 510 in some examples. Current frame 510 is an exemplary frame/video being encoded/decoded at a specific time. The current frame 510 includes objects in the current block 511 that match objects in the reference block 531 of the reference frame 530 . The reference frame 530 is a frame used as a reference for encoding the current frame 510, and the reference block 531 includes an object also included in the current block 511 of the current frame 510. ) is a block of A current block 511 is any coding unit that is encoded/decoded at a particular point in the coding process. The current block 511 may be an entire divided block or may be a sub-block in the case of affine inter-prediction. The current frame 510 is separated from the reference frame 530 by a temporal distance (TD) 533 . TD 533 represents the amount of time between a current frame 510 and a reference frame 530 in a video sequence. During the period represented by the TD 533, the object of the current block 511 moves from the location of the current frame 510 to another location of the reference frame 530 (eg, the location of the reference block 531). do. For example, an object may move along a motion trajectory 513 that is a moving direction of the object over time. The motion vector 535 describes the direction and magnitude of the object's motion along the motion trajectory 513 via the TD 533. Thus, the encoded MV 535 and the reference block 531 provide information sufficient to reconstruct the current block 51 to the land position of the current block 511 in the current frame 510 .

6 shows, for example, block compression step 105, block decoding step 113, motion estimation component 221, motion compensation component 219, motion compensation component 321, and/or motion compensation component 421 converting MV to It is a schematic diagram illustrating an example of bi-directional inter-prediction 600 performed to determine. For example, bidirectional inter-prediction 600 can be used to determine motion vectors for blocks in inter-prediction mode and/or motion vectors for sub-blocks in affine inter-prediction mode.

Bidirectional inter-prediction (600) is similar to unidirectional inter-prediction (500), but uses a pair of reference frames to predict the current block (611) in the current frame (610). Accordingly, the current frame 610 and the current block 611 are substantially similar to the current frame 510 and the current block 511, respectively. The current frame 610 is located temporally between a preceding reference frame 620 occurring before the current frame 610 in the video sequence and a subsequent reference frame 630 occurring after the current frame 610 in the video sequence. Prior reference frame 620 and subsequent reference frame 630 are otherwise substantially similar to reference frame 530 .

The current block 611 is matched to the preceding reference block 621 in the preceding reference frame 620 and to the subsequent reference block 631 in the subsequent reference frame 630 . This matching indicates that during the video sequence, the object moves from the position of the preceding reference block 621 to the position of the subsequent reference block 631 through the current block 611 along the motion trajectory 613 . The current frame 610 is separated from the preceding reference frame 620 by some preceding temporal distance (TD0) 623 and from the subsequent reference frame 630 by some subsequent temporal distance (TD1) 633. do. TD0 623 represents the amount of time between the preceding reference frame 620 and the current frame 610 in the video sequence. TD1 633 represents the amount of time between a current frame 610 and a subsequent reference frame 630 in a video sequence. Accordingly, the object moves from the previous reference block 621 to the current block 611 along the motion trajectory 613 during the time indicated by TD0 623 . In addition, the object moves from the current block 611 to the next reference block 631 along the motion trajectory 613 during the time indicated by TD1 633 .

The preceding motion vector (MV0) 625 measures the direction and magnitude of the object's movement along the motion trajectory 613 across TD0 623 (e.g., between the preceding reference frame 620 and the current frame 610). Explain. The subsequent motion vector (MV1) 635 is the direction and magnitude of the object's movement along the motion trajectory 613 over TD1 633 (e.g., between the current frame 610 and the subsequent reference frame 630). explain Thus, in bi-directional inter-prediction 600, current block 611 can be coded and reconstructed by using preceding reference block 621 and/or subsequent reference block 631, MV0 625 and MV1 635. can

Intra prediction.

Many schemes are used together to compress video data during the video coding process. For example, a video sequence is partitioned into image frames. The image frame is then partitioned into image blocks. The image blocks can then be compressed by inter-prediction (correlation between blocks in different frames) or intra-prediction (correlation between blocks in the same frame). In intra-prediction, the current image block is predicted from the baseline of the sample. Baselines include samples from adjacent image blocks, also referred to as adjacent blocks. The samples in the current block match the samples in the baseline with the closest luma (light) or chroma (color) values. The current block is coded in a prediction mode representing matching samples. Prediction modes include angular prediction mode, direct current (DC) mode, and planar mode. The difference between the value predicted in the prediction mode and the actual value is coded as a residual value in the residual block.

7 is a schematic diagram illustrating an exemplary intra-prediction mode 700 used for video coding. For example, intra-prediction mode 700 includes steps 105 and 113 of method 100 , intra-picture estimation component 215 , and intra-picture prediction component 217 of codec system 200 , encoder 300 . ), and/or the intra-picture prediction component 417 of the decoder 400 . Specifically, the intra-prediction mode 700 may be used to compress an image block into a prediction block comprising the selected prediction mode and the remaining residual blocks.

As mentioned above, intra-prediction involves matching a current image block to a corresponding sample or samples of one or more adjacent blocks. The current image block can then be marked with the selected prediction mode index and residual block, which is much smaller than representing all the luma/chroma values contained in the current image block. Intra-prediction can be used when there is no reference frame available or when inter-prediction coding is not used for the current block or frame. Reference samples for intra-prediction may be derived from previously coded (or reconstructed) neighboring blocks in the same frame. Advanced Video Coding (AVC), also known as H.264 and H.265/HEVC, both use baselines of boundary samples of adjacent blocks as reference samples for intra-prediction. Different intraprediction modes are used to handle different textures or structural properties. H.265/HEVC supports 35 intra prediction modes 700 that spatially correlate a current block with one or more reference samples. Specifically, the intra-prediction mode 700 includes 33 directional prediction modes indexed as modes 2 to 34, a DC mode indexed as mode 1, and a planar mode indexed as mode 0.

During encoding, the encoder matches the luma/chroma values of the current block with the luma/chroma values of the corresponding reference sample in the baseline across the edge of the adjacent block. If a best match with one of the baselines is found, the encoder selects one of the directional intra-prediction modes 700 pointing to the best matching baseline. For clarity of discussion, abbreviations are used below to refer to specific directional intra-prediction modes 700 . DirS represents the starting directional intra-prediction mode when counting clockwise from the bottom left (eg, mode 2 of HEVC). DirE represents the end-direction intra-prediction mode when counting clockwise from the bottom left (eg, mode 34 of HEVC). DirD represents a mid-direction intra-coding mode when counting clockwise from the bottom left (eg, mode 18 of HEVC). DirH represents a horizontal intra-prediction mode (eg, mode 10 of HEVC). DirV represents a vertical intra-prediction mode (eg, mode 26 of HEVC).

As described above, the DC mode serves as a smoothing function and derives the predicted value of the current block as the average value of all reference samples of the baseline crossing the neighboring blocks. Also, as discussed above, planar mode returns predictive values that represent a smooth transition (e.g., constant slope of the value) between samples that are below and above the left and top of the reference sample's baseline or between the top left and top right of the sample.

For Planar mode, DC mode and Prediction mode from DirH to DirV, the samples in the top row of the baseline and the column to the left of the baseline are used as reference samples. For prediction modes with prediction directions from DirS to DirH (including DirS and DirH), reference samples of previously coded and reconstructed neighboring blocks in the left column of the baseline are used as reference samples. For prediction modes with prediction directions from DirV to DirE (including DirV and DirE), the reference sample of the previously coded and reconstructed adjacent block in the top row of the baseline is used as the reference sample.

8 is a schematic diagram illustrating an example of the directional relationship of a block 800 in video coding. For example, block 800 may be used when selecting intra-prediction mode 500 . Accordingly, block 800 follows steps 105 and 113 of method 100, intra-picture estimation component 215 and intra-picture prediction component 217 of codec system 200, and intra-picture prediction of encoder 300. component 317 and/or intra-picture prediction component 417 of decoder 400 . In video coding, blocks 800 are partitioned based on video content, and thus may contain many rectangles and squares of various shapes and sizes. Block 800 is shown as a square for explanatory purposes and is therefore simplified from the actual video coding block to support clarity of discussion.

A block 800 includes a current block 801 and an adjacent block 810 . The current block 810 is any block being coded at a designated time. The adjacent block 810 is any block immediately adjacent to the left edge or top edge of the current block 801 . Video coding generally proceeds from top left to bottom right. As such, the adjacent block 810 may be encoded and reconstructed prior to coding of the current block 801 . When coding the current block 801, the encoder matches the luma/chroma values of the current block 801 with a reference sample (or samples) from a baseline traversing the edge of the adjacent block 810. The match is then used to select an intra-prediction mode, such as intra-prediction mode 700 pointing to the matched sample (or sample when DC or planar mode is selected). At this time, the selected intra-prediction mode indicates that the luma/chroma value of the current block 801 is substantially similar to the reference sample corresponding to the selected intra-prediction mode. All differences can be kept in the remaining blocks. The selected intra-prediction mode is encoded into the bitstream. At the decoder, the current block 801 may be reconstructed (with residual information from the residual block) by using the luma/chroma values of the reference samples at the selected baseline of the adjacent block 810 corresponding to the selected intra-prediction mode. .

In-loop filter.

Video coding schemes subdivide a video signal into image frames and then subdivide the image frames into blocks of various types. The image block is then compressed. This approach can create visual artifacts when the compressed video signal is reconstructed and displayed. For example, an image compression process may artificially add a block shape. This is called blocking and usually occurs at block partition boundaries. In addition, non-linear signal dependent rounding error, also known as quantization noise, may be artificially added to the compressed image. Various filters may be used to correct for these artifacts. A filter may be applied to the reconstructed frame in post processing. Post processing occurs after a significant portion of the compressed video signal is reconstructed and immediately before display to the user. Filters can also be applied as part of the compression/decompression process using a mechanism called in-loop filtering. In-loop filtering is a filtering scheme that supports more precise compression between related images by applying filters to reconstructed video images during the encoding and/or decoding process. For example, inter-prediction encodes an image frame based on previous and/or subsequent image frames. At the encoder, the compressed image is reconstructed and filtered through in-loop filtering so that the reconstructed image provides a more accurate image for use in encoding the previous/subsequent image frame(s) via inter-prediction. At the decoder, the compressed image is reconstructed and filtered through in-loop filtering to produce a more accurate image for end-user viewing and to support more accurate inter-prediction. In-loop, filtering uses several filters such as a deblocking filter, a sample adaptive offset (SAO) filter and an adaptive loop filter. In-loop filtering may also include a noise suppression filter.

9 is a block diagram illustrating an exemplary in-loop filter 900. In-loop filter 900 may be used to implement in-loop filters 225, 325, and/or 425. The in-loop filter 900 includes a noise suppression filter 941; A deblocking filter 943, a sample adaptive offset (SAO) filter 945 and an adaptive loop filter 947. The filters of in-loop filter 900 are sequentially applied to the reconstructed image block and/or residual block.

A noise suppression filter 941 is configured to remove quantization noise due to image compression. Specifically, the noise suppression filter 941 is used to remove artifacts occurring at the edges of the image. For example, image compression can produce sharp and inaccurate color/brightness values adjacent to sharp transitions (edges) between different color/bright patches in an image. This is called ringing and is caused by applying a transform to the high-frequency portion of the image data associated with sharp edges. A noise suppression filter 941 is used to mitigate these ringing artifacts. The noise suppression filter 941 operates in both the spatial domain (eg, the spatial orientation of pixels) and the frequency domain (eg, the relationship of transformed coefficient values with respect to pixel data). At the encoder, a noise suppression filter 941 partitions the reconstructed frame into reference macroblocks. These blocks may be subdivided into smaller reference blocks. The noise suppression filter 941 first creates an application map indicating the portion of the frame that should be filtered based on the amount of quantization noise estimated in the block. The noise suppression filter 941 uses a matching component to determine, for each reference block, a set of patches that are similar to the corresponding reference block, as indicated by the application map, where similar have chroma/luma values within a predetermined range. indicates The noise suppression filter 941 can generate frequency domain patches by grouping the patches into clusters and transforming the clusters to the frequency domain using a two-dimensional (2D) transform. The noise suppression filter 941 can also transform the frequency domain patch back to the spatial domain using an inverse 2D transform.

The deblocking filter 943 is configured to remove block-shaped edges produced by block-based inter and intra-prediction. The deblocking filter 943 scans an image portion (eg, an image slice) for discontinuities in chroma and/or luma values that occur at segmentation boundaries. The deblocking filter 943 then applies a smoothing function to the block boundaries to remove such discontinuities. The strength of the deblocking filter 943 may change according to spatial activity (eg, a change in luma/chroma components) occurring in a region adjacent to a block boundary.

SAO filter 945 is configured to remove artifacts related to sample distortion caused by the encoding process. The SAO filter 945 in the encoder classifies the deblocked samples of the reconstructed image into different categories based on relative deblocking edge shape and/or direction. An offset is then determined and added to the samples according to the category. The offset is encoded in the bitstream and used by the SAO filter 945 at the decoder. The SAO filter 945 removes banding artifacts (bands of values instead of smooth transitions) and ringing artifacts (spurious signals near sharp edges).

An adaptive loop filter 947 in the encoder is configured to compare the reconstructed image with the original image. The adaptive loop filter 947 determines coefficients describing the difference between the original image and the reconstructed image via, for example, a Wiener-based adaptive filter. These coefficients are encoded in the bitstream and used in the decoder's adaptive loop filter 947 to remove the difference between the reconstructed image and the original image. Although the adaptive loop filter 947 is effective in correcting artifacts, the larger the difference between the reconstructed image and the original image, the more coefficients can be signaled. This in turn creates a larger bitstream which reduces the efficiency of compression. As such, minimizing the difference caused by the other filters before applying the adaptive loop filter 947 improves compression.

partitioning.

Video coding uses an encoder to compress a media file and a decoder to reconstruct an original media file from the compressed media file. Video coding uses a variety of standardized processes to ensure that any decoder using a standardized process can consistently reproduce a compressed media file by any encoder using a standardized process. For example, both the encoder and decoder may use a coding standard such as High Efficiency Video Coding (HEVC), also known as H.265. At the encoder, the video signal is divided into frames. The frame is then partitioned into image blocks containing groups of pixels. The image blocks are then compressed, filtered and encoded into bitstreams. The bitstream can be sent to a decoder that reconstructs the video signal for display to an end user.

The segmentation system is configured to divide an image block into sub-blocks. For example, a tree structure using various split modes can be used to split a node (eg, block) into child nodes (eg, sub-blocks). Different split modes can be used to obtain different partitions. You can also recursively apply split mode to further subdivide nodes. When this split mode is applied, various partition patterns are created.

10 shows an exemplary split mode 1000 used for block partitioning. Split mode 1000 is a mechanism that splits a parent node (eg, image block) into multiple child nodes (eg, image sub-blocks) during partitioning. The split mode (1000) includes a quad-tree (QT) split mode (1001), a vertical binary tree (BT) split mode (1003), a horizontal BT split mode (1005), and a vertical triple It includes a triple tree (TT) split mode 1007 and a horizontal TT split mode 1009 . The QT split mode 1001 is a tree structure for block partitioning in which a node of size 4Mx4N is split into four child nodes of size MxN, where M represents block width and N represents block height. Vertical BT split mode (1003) and horizontal BT split mode (1005) split a node of size 4Mx4N vertically into two child nodes each of size 2Mx4N or horizontally into two child nodes each of size 4Mx2N. It is a tree structure for partitioning blocks that are divided. In the vertical TT split mode 1007 and the horizontal TT split mode 1009, a node with a size of 4Mx4N is vertically split into three child nodes with sizes of Mx4N, 2Mx4N and Mx4N; Or, it is a tree structure for block partitioning that is horizontally divided into three child nodes of 4MxN, 4Mx2N, and 4MxN sizes, respectively. The largest node among the three child nodes is located in the center.

Split mode 1000 may be applied recursively to further split the block. For example, the quad-tree binary-tree (QT-BT) splits a node with QT split mode 1001, followed by vertical BT split mode (1003) and/or horizontal BT split mode ( 1005) by partitioning each child node (also called a quadruple tree leaf node). Additionally, a quad tree triple tree (QT-TT) can be created by splitting a node with a quad tree split and then splitting the resulting child nodes into a vertical TT split mode (1007) and/or a horizontal TT split mode (1009). there is.

HEVC works on Joint Exploration Model (JEM) applications. In JEM, QT-BT block partitioning is used to partition a coding tree unit (CTU) into multiple blocks. TT block partitioning has also been proposed for inclusion in JEM to further enrich block partition types. In video coding based on the QT, QT-BT, or QT-TT block division split mode, a coded or predicted block of depth K is divided into N smaller coded or It can be divided into prediction blocks, where N is set to 2, 3 or 4 respectively. A partition pattern of the split mode is shown in FIG. 10, and the partition pattern indicates the size and location of two or more child nodes divided from a parent node.

conversion.

Video coding uses an encoder to compress a media file and a decoder to reconstruct an original media file from the compressed media file. Video coding uses a variety of standardized processes to ensure that any decoder using a standardized process can consistently reproduce a compressed media file by any encoder using a standardized process.

For example, both the encoder and decoder may use a coding standard such as High Efficiency Video Coding (HEVC), also known as H.265. H. 265 is based on a prediction and transformation framework. In the encoder, a video file is divided into frames. The frame is then subdivided into image blocks containing groups of pixels. The image block is further decomposed into a prediction block containing prediction information such as prediction mode and motion vector information and a residual block containing residual information such as transform mode, transform coefficients and quantization parameters. Prediction blocks and residual blocks use less storage space than image blocks, but can be used to reconstruct image blocks at the decoder. Prediction blocks and residual blocks are coded into a bitstream and transmitted to a decoder and/or stored for later transmission upon request. Prediction information and residual information are parsed at the decoder. The analyzed prediction information is used to generate prediction samples using either intra-prediction or inter-prediction. Intra-prediction uses a reconstructed image block to predict other image blocks in the same frame. Inter-prediction uses reconstructed image blocks to predict other image blocks between adjacent frames. Also, the residual information is used to generate residual samples by sequentially applying, for example, inverse quantization and inverse transformation. Prediction samples and residual samples are combined to obtain reconstructed samples corresponding to image blocks coded by the encoder (eg, for display to an end user on a monitor).

Spatial varying transform (SVT) is a mechanism used to further improve video coding efficiency. SVT uses a transform block to further compress the residual block. Specifically, the rectangular residual block includes a width and a height h (eg, wxh). A transform block smaller than the residual block is selected. Thus, a transform block is used to transform the corresponding part of the residual block and leave the remaining part of the residual block without further coding/compression. The rationale for SVT is that residual information may not be evenly distributed in residual blocks. Using a smaller transform block with adaptive positioning allows us to capture most of the residual information from the residual block without having to transform the entire residual block. This approach may in some cases achieve better coding efficiency than transforming all residual information of a residual block. Since the transform block is smaller than the residual block, SVT uses a mechanism to signal the transform position for the residual block. This position signaling increases the overall signaling overhead of the coding process and reduces compression efficiency. Also, using the same type of transform block in all cases may not yield beneficial results in some cases.

11 is a schematic diagram of an example video encoding mechanism 1100 . An encoder from one or more frames may obtain an image block 1101 . For example, an image may be divided into a plurality of rectangular image regions. Each area of the image corresponds to a CTU (Coding Tree Unit). A CTU is partitioned into a plurality of blocks, such as coding units of HEVC. The block division information is then encoded in the bitstream 1111. Thus, an image block 1101 is a segmented portion of an image and includes pixels representing luma components and/or chroma components in corresponding portions of the image. During encoding, the image block 1101 is encoded as a prediction block 1103 containing prediction information such as a motion vector for inter-prediction and/or a prediction mode for intra-prediction. Encoding the image block 1101 into the prediction block 1103 may leave a residual block 1105 containing residual information representing the difference between the prediction block 303 and the image block 301 .

It should be noted that the image block 1101 can be partitioned into coding units including one prediction block 1103 and one residual block 1105. The prediction block 1103 may include all prediction samples of the coding unit, and the residual block 1105 may include all of the residual samples of the coding unit. In this case, the prediction block 1103 is the same size as the residual block 1105. In another example, an image block 1101 may be divided into coding units including two prediction blocks 1103 and one residual block 1105 . In this case, each prediction block 1103 includes a portion of the prediction samples of the coding unit, and the residual block 1105 includes all residual samples of the coding unit. In another example, an image block 1101 is partitioned into coding units comprising two prediction blocks 1103 and four residual blocks 1105. The partition pattern of the residual block 1105 of the coding unit may be signaled in the bitstream 1111. This location pattern may include a residual quad-tree (RQT) of HEVC. Also, the image block 1101 may include only a luma component (eg, light) represented by a Y component of an image sample (or pixel). In another case, image block 1101 may include Y, U, and V components of an image sample, where U and V are chrominance components (e.g., color) in the blue luminance and red luminance (UV) color space. indicates

A transform can be used to further compress the information. In particular, the transform block 1107 can be used to further compress the residual block 1105. Transform block 1107 includes a transform such as an inverse discrete cosine transform (DCT) and/or an inverse discrete sine transform (DST). The difference between the prediction block 1103 and the image block 1101 is fit for transformation by using transform coefficients. By indicating the transform mode (eg, inverse DCT and/or inverse DST) of the transform block 1107 and the corresponding transform coefficients, the decoder can reconstruct the residual block 1105 . If exact reproduction is not required, the transform coefficients can be further compressed by rounding certain values to better fit the transform. This process is called quantization and is performed according to a quantization parameter describing acceptable quantization. Accordingly, the transform mode, transform coefficients, and quantization parameters of the transform block 1107 are stored as transformed residual information in the transformed residual block 1109, which in some cases may simply be referred to as a residual block.

Prediction information of the prediction block 1103 and transformed residual information of the transformed residual block 1109 may be encoded in the bitstream 1111 . The bitstream 1111 may be stored and/or transmitted to a decoder. The decoder can then perform the reverse process to recover image block 1101. Specifically, the decoder may determine the transform block 1107 using the transformed residual information. The transform block 1107 can then be used with the transformed residual block 1109 to determine the residual block 1105. Residual block 1105 and prediction block 1103 may be used to reconstruct image block 1101 . Image blocks 1101 can then be positioned relative to other decoded image blocks 1101 to reconstruct frames and position those frames for encoded video recovery.

It should be noted that some prediction blocks 1103 can be encoded without generating residual blocks 1105. However, such a case is not discussed further as it does not result in the use of the transform block 1107. Transform block 1107 can be used for inter-predicted blocks or intra-predicted blocks. In addition, the transform block 1107 can be used for the residual block 1105 generated by a specified inter-prediction mechanism (e.g., transform model based motion compensation), but can be used for other specified inter-prediction mechanisms (e.g., affine It may not be used for the residual block 1105 generated by model-based motion compensation).

12 is a schematic diagram of an example computing device 1200 for video coding in accordance with an embodiment of the present disclosure. Computing device 1200 is suitable for implementing the disclosed embodiments as described herein. Computing device 1200 includes a receive port 1220 and a receiver unit (Rx) 1210 for receiving data; a processor, logic unit or central processing unit (CPU) 1230 for processing data; a transmitter unit (TX) 1240 and egress port 1250 for transmitting data; and a memory 1260 for storing data. Computing device 1200 also includes an optical-to-electrical (OE) device coupled to an inlet port 1220, a receiver unit 1210, a transmitter unit 1240, and an outlet port 1250 for outputting or receiving optical or electrical signals. components and electro-optical (EO) components. Computing device 1200 may also include a wireless transmitter and/or receiver in some examples.

Processor 1230 is implemented by hardware and software. Processor 1230 may include one or more CPU chips, cores (eg, multi-core processors), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signals. It may be implemented as a processor (digital signal processor, DSP). Processor 1230 communicates with inlet port 1220 , receiver unit 1210 , transmitter unit 1240 , egress port 1250 and memory 1260 . Processor 1230 includes coding module 1214 . Coding module 1214 implements the disclosed embodiments described above. For example, coding module 1214 implements, processes, prepares, or provides various coding operations. Thus, inclusion of the coding module 1214 provides substantial improvements to the functionality of the computing device 1200 and affects the transition of the computing device 1200 to other states. Alternatively, coding module 1214 is implemented by being stored in memory 1260 and executed by processor 1230 (eg, as a computer program product stored on non-transitory media).

Memory 1260 includes one or more disks, tape drives, and solid state drives and can be used as overflow data storage, to store programs when such programs are selected for execution, and to be used to store instructions and data. can read during program execution. Memory 1260 may be volatile and/or non-volatile and may include read-only memory (ROM), random access memory (RAM), ternary content-accessible memory addressable memory (TCAM) and/or static random-access memory (SRAM). Computing device 1200 may also be an input/output (I/O) device for interacting with an end user. For example, the computing device 1200 may include a display such as a monitor for visual output, a speaker for audio output, and a keyboard/mouse/trackball for user input.

13 is an example of a system 1300 illustrating point cloud media. In particular, system 1300 illustrates various examples of point cloud media frames. Examples include a male 1302, a front facing female 1304 and a rear facing female 1306. In some implementations, system 1300 may include more or fewer instances of point cloud media. Point cloud media may include virtual representations of objects or people and their corresponding movements over a period of time in three-dimensional (3-D) space.

In general, point cloud media includes a set of data points in 3-D space that outline the outer surface of a 3-D object. For example, objects may include people, manufactured items, and real objects. For example, an object may be recorded in real time by a camera and represented virtually on a display or projection screen in a three-dimensional space. A frame of point cloud media may contain a 3D representation of an object at a particular point in time. Consecutive frames of the point cloud media represent a dynamic representation of the point cloud media.

Acronym definitions are explained below: Group of Frames (GoF) - A set of registered point clouds at a fixed time used for further processing. Group of Pictures (GoP) - A set of projections derived from a point cloud used for video compression encoding. Point cloud compression (PCC) - A technique for compressing point cloud media for transmission.

The video-based PCC codec solution is based on the segmentation of 3-D point cloud data into 2-D projection patches. Video-based PCC is widely used for content such as immersive 6 degrees of freedom, dynamic AR/VR objects, cultural heritage, GIS, CAD, and autonomous navigation.

Point cloud media has become an integral part of a variety of applications, including the entertainment industry, the intelligent car navigation industry, geospatial inspection, and 3D modeling and visualization of real-world objects. In each of these applications, the various client devices can display and describe the point cloud media. Given the non-uniform sampling geometry, it is good to have a concise representation for the storage and transmission of such data.

For example, a client device may use point cloud media to create 3-D computer animation drawing (CAD) models for manufacturing parts, metrology and quality inspection, various visualization, animation, rendering, and mass customization applications. In some instances, point cloud media may be used to represent volumetric data, as is often done in medical imaging, such as when visualizing the amount of space occupied by a bone, arm, or leg, to name a few examples.

Compared to traditional 3D presentations, irregular point cloud surfaces, such as those representing people, are more general and applicable to a wide range of sensors and data collection strategies. For example, for 3D presentations of virtual reality worlds or remote rendering of telepresence environments, renderings of virtual people and real-time instructions are handled as dense point cloud data sets. For example, in the example where a male 1302, a front-facing female 1304, and a rear-facing female 1306 are presented in a virtual reality world via a telepresence environment, each of these single point cloud frames is treated as a dense point cloud data set. do.

System 1300 illustrates different objects represented by frames of point cloud media at a particular point in time. A device, such as a client device, may display successive frames of point cloud media over a period of time. In some examples, the device may include immersive six degrees of freedom (6 DoF) environments, dynamic augmented reality (AR) and virtual reality (VR) environments, heritage environments, geographic information system (GIS) environments, computer-aided design (CAD) environments, and display point cloud media in an autonomous navigation system.

Devices capable of receiving, displaying, and transmitting point cloud media may include client devices such as personal computers, handheld devices, televisions, navigation systems, or other devices capable of displaying point cloud media. In some implementations, these devices may store point cloud media within internal memory or store point cloud media on some external storage device. Because storing point cloud media requires a significant amount of storage space, point cloud media is typically stored and accessed on external storage devices.

For client devices that capture and relay point cloud media, the client device may rely on multisampling techniques and data compression techniques to reduce the bandwidth required to store and/or transmit the point cloud media. In some cases, storing and transmitting dynamic representations of point cloud media is complicated by the size of a single frame of point cloud media. For example, a single frame of point cloud media can contain large amounts of data, such as hundreds of gigabytes. Therefore, storing and transmitting multiple frames of point cloud media requires data compression and encoding techniques to properly transmit large amounts of point cloud media frames over the various media which can cause data loss.

14 is an example of a system 1400 illustrating a point cloud frame sequence. In particular, system 1400 illustrates a dynamic point cloud media sequence representing a sequence of frames of point cloud media at different time stamps. For example, system 1400 provides a first point cloud frame 1402 at a first time stamp, a second point cloud frame 1404 at a second time stamp, and a third point cloud frame 1406 at a third time stamp. ). The first, second and third point cloud frames constitute a dynamic point cloud sequence representing a woman moving her right foot forward 1400 times at incremental time stamps in a three-dimensional environment when viewed in real time. Incremental timestamps can be measured in units of time, such as seconds or microseconds, to name a few.

System 1400 provides bounding boxes 1408 for frames in point cloud media 1402, bounding boxes 1410 for other frames in point cloud media 1404, and bounding boxes 1410 for other frames in point cloud media 1406. Bounding box 1412 is illustrated. Each point cloud media contains a fixed grid of three dimensions: X dimension, Y dimension and Z dimension, denoted by u1, v1 and d1 position coordinates.

A bounding box contains points that can be empty or filled with data. In some examples, an empty point may be a void point. In some examples, an occupancy point may include one or more attributes. A void point may contain the location of a point cloud media frame with no attributes. On the other hand, an occupancy point may include a position within a frame of point cloud media having at least one attribute. For example, attributes may include chroma, luna, reflectance, and color. A point within a bounding box can be identified by a specific coordinate system. For example, points within the bounding box 1408 of the frame of the point cloud media 1402 may be represented by u1, v1 and d1 coordinate systems.

Typically, when a client device transmits one or more frames of point cloud media to another client device over a network, each frame of point cloud media is encoded. Encoding ensures proper transmission and ensures that no data is lost when received at the receiving client device. However, frame encoding of point cloud media can be complicated because the size of a single frame of point cloud media can exceed hundreds of gigabytes. For example, a system such as a client device needs to encode 500 gigabytes of data if it wants to encode the first point cloud frame 1402, wasting time and resources to do so. Thus, another encoding technique that relies on a bounding box, e.g., bounding box 1408, is needed to encode a particular frame of point cloud media. The bounding box representation described below allows the system to encode a single frame of point cloud media into a compressed format.

In some implementations, frames of the point cloud media are encoded using a video-based point cloud compression (V-PCC) coder. The V-PCC coder works by segmenting point cloud media frames into three-dimensional patch sets. A 3D patch is represented by a 3D bounding box and manipulated for transmission. This is explained in detail below.

15 is an example of a process 1500 of converting a 3D patch bounding box into a 2D patch projection. In some implementations, the conversion process 1500 is performed by a V-PCC encoder using a V-PCC encoding solution.

During the V-PCC encoding solution, the V-PCC encoder encodes a frame of point cloud media using a three-dimensional bounding box. Typically, a set of points is iterated and projected onto a bounding box, then patches are partitioned according to the definition of a smooth continuous surface criterion. Each patch corresponds to a specific and unique index and its 3D coordinates. Also, the matching patch list exists so that the order in the list is similar and matching patches in this list have the same index.

First, a frame of 3-D point cloud media is divided into a set of partitions or three-dimensional (3-D) patch bounding boxes, such as one or more 3-D patch-bounding boxes 1504 . Patch bounding box 1504 includes the following parameters: u1, v1, d1, the size of u1, the size of v1, and the size of d1. The parameters of the 3-D patch bounding box 1504 represent the size and placement of the 3-D patch bounding box 1504 within the point cloud 3-D bounding box 1502. In some implementations, the system creates multiple 3-D patch bounding boxes. Let the entire frame of the point cloud media cover the objects within the frame. Each 3D patch bounding box can be placed close to each other. In some implementations, each 3-D patch-bounding box may overlap each other when covering an object within the point cloud 3-D bounding box 1502 .

The V-PCC encoder then defines a projection plane 1508 as one of the sides of the point cloud 3D bounding box 1502 for each 3D patch bounding box, such as the 3D patch bounding box 1504 . A V-PCC encoder defines criteria for selecting a projection plane. For example, the criteria include the definition of a smooth continuous surface criteria on each side of the 3-D patch bounding box 1504. A projection plane is selected when the smooth continuous surface reference contains the area of the projected 3-D patch bounding box 1504, where the area of the projected 3-D patch bounding box 1504 is the area of the three-dimensional bounding box 1502. It is projected on the side and becomes the largest area in each direction for projection. For example, a smooth continuous surface can be determined using one or more smooth continuous algorithms. A smooth continuous surface can be defined as a surface with a minimum number of occluded or occluded data points. The V-PCC encoder then compares each smooth surface from each and every direction to determine which direction will produce a 2D bounding box containing the largest area on the side of the 3D bounding box 1502.

As illustrated in system 1500, projection plane 1508 corresponds to the parallel projected area of 3-D patch bounding box 1504 to have a maximum area. In other words, if the V-PCC encoder looks at the 3-D patch bounding box 1504 from the opposite side of the projection plane 1508 (e.g., side 1505), the V-PCC encoder will see the patch 3-D bounding box 1504 When the attribute points included in are projected onto the projection plane 1508 , that projection will determine that they represent the largest area projected onto the other side of the point cloud 3-D bounding box 1502 . When the frame of the point cloud media included in the point cloud 3-D bounding box 1502 is still and does not move, the V-PCC encoder analyzes the projections from each side of the point cloud 3-D bounding box 1502 and You can decide which projection will contain the maximum area. Thus, the V-PCC encoder determines that the projection plane 1508 will produce the largest area for projection of attributes from the patch 3-D bounding box 1504. The projection is thus displayed within the patch 2-D projection bounding box 1506. The patch 2-D projection bounding box 1506 is defined by the u and v coordinates on the near layer and the far layer of the bounding box. For example, FIG. 17 shows near layer 1704 and far layer 1706 of depth images 1708 and 1712 and attribute images 1710 and 1714 .

When a projection plane, e.g. projection plane 1508, is selected, the normal axis for the patch 3-D bounding box 1504 is drawn so that the V-PCC encoder knows the direction of the projection. For example, the V-PCC encoder draws an axis perpendicular or orthogonal to the projection plane 1508 denoted by "n" within the point cloud 3-D bounding box 1502. Additionally, the V-PCC encoder generates tangential and positive axes for the patch 3D bounding box 1504 denoted by “bt” and “t” within the point cloud 3D bounding box 1502. In response to the V-PCC encoder drawing the normal, tangential, and double-sided axes within the point 3-D bounding box 1502 cloud, the right-hand 3-D coordinate system is drawn through the patch 3-D bounding box 1504 for the embossed and orthogonal axes. created between

At that point, the V-PCC encoder projects the patch 3-D bounding box 1504 onto the projection plane 1508 of the point cloud 3-D bounding box 1502 along the normal axis “n”. As shown in process 1500, the result of the projection is a two-dimensional (2-D) projection 1506 on a particular side of the point cloud (3-D bounding box 1502).

In some implementations, a 3-D projection of the point cloud media frame is projected onto each side of the point cloud 3-D bounding box 1502 . A set of coordinates for the 2-D bounding box can be obtained from the bounding box 1502 when the frame is projected to the side of the point cloud 3-D. For example, the set of coordinates of a 2D bounding box includes u0, v0, size_u0, and size_v0. As shown in the bounding box 1510 , the V-PCC encoder projects the surface of the point cloud within the bounding box 1510 onto a projection plane 1512 .

16 is an example system 1600 illustrating 3-D to 2-D patch projection results. System 1600 illustrates bounding box 1602 with 2-D patch projections on each side of bounding box 1602 . The projections on each side of the bounding box are based on the orthogonal orientation of the patch as described for system 1500. Each 2D patch projection contains patch indices u0, v0, size_u0 and size_v0 describing the 2D coordinates of the 2D projection of the patch. The patch index identifies a particular patch associated with a particular face of the corresponding bounding box 1602 . U0 defines the X coordinate of the patch in the projection plane. V0 defines the Y coordinate of the patch in the projection plane. Size_u0 and size_v0 describe sizes corresponding to respective coordinates of patches u0 and v0, respectively.

The patches projected into the bounding box 1602 form a "patch tile group". Each element of the patch tile group corresponds to a specific patch, which contains a specific unique index and corresponds to a unique 3D bounding box within the 3D point cloud frame.

In addition to generating projection maps in the bounding box 1602, the system stores the patch 2-D and 3-D data in memory. For example, the code described below provides one example solution for generating patches and storing 2D and 3D patch data as supplemental data for video coding:

Algorithms. PCC coder additional data export 1. while frame k in GoF frames do generate patches in frame k .
for each patch in frame k
generate projection map from patches
store patch2D data: U0, sizeU0, V0, sizeV0
store patch3D data: U1, V1, D1, axis
end for
end while

For k-th frame: double patch3DCoor[MAX_NUM_PATCHES][4];

size_t patch2DCoor[MAX_NUM_PATCHES][4];

patch2DCoor[patchIndex][0] = patch.getU0() * patch.getOccupancyResolution();

patch2DCoor[patchIndex][1] =(patch.getU0() + patch.getSizeU0()) * patch.getOccupancyResolution();

patch2DCoor[patchIndex][2] = patch.getV0() * patch.getOccupancyResolution();

patch2DCoor[patchIndex][3] =(patch.getV0() + patch.getSizeV0()) * patch.getOccupancyResolution();

PCCVector3D pointStart, pointEnd;

const double lodScale = params.ignoreLod_ ? 1.0 : double(1u << patch.getLod());

int x = patch.getU0() * patch.getOccupancyResolution(), y = patch.getV0() * patch.getOccupancyResolution();

pointStart[patch.getNormalAxis()] = double(frame0.getValue(0, x, y) + patch.getD1()) * lodScale;

pointStart[patch.getTangentAxis()] = patch.getU1() * lodScale;

pointStart[patch.getBitangentAxis()] = patch.getV1() * lodScale;

x = (patch.getU0() + patch.getSizeU0()) * patch.getOccupancyResolution() - 1;

y = (patch.getV0() + patch.getSizeV0()) * patch.getOccupancyResolution() - 1;

pointEnd[patch.getNormalAxis()] = double(frame0.getValue(0, x, y) + patch.getD1()) * lodScale;

pointEnd[patch.getTangentAxis()] =(patch.getSizeU0() * patch.getOccupancyResolution() - 1 + patch.getU1()) * lodScale;

pointEnd[patch.getBitangentAxis()] =(patch.getSizeV0() * patch.getOccupancyResolution() - 1 + patch.getV1()) * lodScale;

patch3DCoor[patchIndex][0] = pointStart[patch.getTangentAxis()];

patch3DCoor[patchIndex][1] = pointStart[patch.getBitangentAxis()];

patch3DCoor[patchIndex][2] = pointEnd[patch.getTangentAxis()];

patch3DCoor[patchIndex][3] = pointEnd[patch.getBitangentAxis()];

Generate auxiliary information for video coding

The code described above shows the definition of a "patch". how patch 3D information is transmitted as 2D patch information; projection process; and reconstitution process. Specifically, the way these frames are interpreted in terms of 2D projections.

Additionally, for each patch in a frame, the V-PCC encoder creates a projection map. The projection map contains an additional text file provided to the video encoder. The 2D data for a particular patch, eg u0, v0, corresponds to the top left corner of the X and Y coordinates of the projection. Size_u0 and size_v0 correspond to the height and width of the patch. For 3D patch projection data, the projection map contains u1, v1 and d1 coordinates corresponding to the X, Y and Z axes. In particular, u1, v1 and d1 correspond to the projection plane corresponding to the index or normal axis of the projection plane. For example, as illustrated in system 1600, each side of bounding box 1602 includes its own index. Each index therefore represents a specific projection axis. In this regard, u1, v1, and d1 are reconstructions of the X, Y, and Z coordinates shown in bounding box 1602.

In some implementations, the u1, v1 and d1 coordinates correspond to a local coordinate system. X, Y and Z coordinates correspond to the global coordinate system. For example, once the vertical axis is determined (corresponding to the projection axis), the V-PCC encoder can rotate the projection axis to align with the Z axis in 3D space. At this point, the V-PCC encoder can transform between the local coordinate system and the global coordinate system.

The proposed solution also provides additional motion vector candidates for motion compensation based on the auxiliary information used as an additional input to the video compression solution in the video compression technique:

17 is an example of a system 1700 for attribute segmentation for cloud point media. Once a 2D patch is created for each 3D point cloud in the projection, a set of images is created for each 2D patch within the patch tile group. System 1700 creates a set of images for each patch of a 2D patch within a group of patch tiles since depth and attribute information is lost during projection from 3D to 2D. A set of images is created to preserve depth and attribute information corresponding to the point cloud media before projection takes place.

In some implementations, the patch tile group includes patch 1702. The system creates two sets of images from a particular patch 1702. The first set of images includes a near layer 1704 . The second set of images includes a far layer 1712 . The near layer 1704 includes an image 1708 of depth data and an image 1710 of attribute data. Additionally, the far layer 1706 includes an image 1712 of depth data and an image 1712 of attribute data. Thus, each and every point of the 2-D projection has depth information and attribute information such as color, texture, luna, etc. Certain layers, eg, near layer 1704 and far layer 1706 show both sides of a 2-D projection of a 3D point cloud object.

18 is an example system 1800 illustrating packaging patches for point cloud media with attribute information. In some implementations, attribute information can include texture information, depth information, color information, lunar, reflectance, and chroma information. Packaged patch group 1802 illustrates one or more projected patches, 2-D frames, from frames of point cloud media, such as frames of point cloud media 1402 . A collection of patches creates patch tile groups and patch tile groups. Combined into patch data groups for a given point cloud media frame. Each element of the patch data group, referred to as a "patch", contains a specific unique index and corresponds to a unique 3D bounding box within the 3D point cloud frame. The packaged patch 1802 may be encoded and then transmitted. In particular, if a patch of one point cloud frame has a reference point cloud frame, eg, a reference patch corresponding to a previous frame, the index of the reference patch of the reference patch tile group is transmitted as a bitstream. This is explained in detail below.

19 is an example of a system 1900 for performing motion estimation. In previous video compression solutions, the data of a 2D projection image, eg a patch, is estimated using a motion estimation process. System 1900 illustrates this motion estimation process. The encoding and decoding block shown at block 1902 generates a motion estimate using an encoder, transmits the motion estimate over a data conversion channel or storage, and decodes the transmitted motion estimate at a decoder. The result is a newly reconstructed frame.

Block 1904 depicts an example of a motion estimation process. For example, an encoder generates predictors of similar patch images in a reference frame. For example, the reference frame may be a previous frame. In some implementations, an encoder determines similarity by comparing sample values between a reference frame and a current frame. For example, sample values may vary by attribute and location. This process is called motion estimation, with the goal of finding that image in a reference frame that is similar to the image in the current frame. The encoder searches for a pixel within neighboring pixels of a reference frame. When a matching image is found, the encoder encodes the similarity, which requires a small amount of information, without specifically encoding each image in the current and reference frames. Overall, this reduces transmission bandwidth. Thus, the encoder analyzes the current and previous frames for similar images. Determine specific data related to attributes and location information for each image between the current frame and the reference (previous) frame. The encoder then encodes the enhancement data for the current frame and future predicted frames. It is commonly used to transmit 2D motion video or 2D images.

Due to the nature of the patch packing method, the location of patches within a projection frame for each point cloud media can be quite different. Thus, between different frames, patches can move from one location to another, such as across different sides of a 3D bounding box. As such, predictive motion estimation using patch positions in inter-predictive coding algorithms commonly used for 2-D motion video or 2-D images is unsatisfactory and is not properly encoded and transmitted. Thus, blocks 1902 and 1904 are useful but require enhancements to transmit motion estimation data using patch data groups. Thus, the technique described below enhances the use of prediction using motion estimation for two-dimensional patch data by adding auxiliary information to the motion vector candidate list.

For temporal prediction between 2-D patches to be transmitted, valid motion vector candidates are provided for similar patches to ensure maximum compression efficiency during transmission. Existing motion vector candidate list construction methods can be improved by replacing existing candidates or introducing additional candidates generated from patch auxiliary information.

In some implementations, specific patches of a frame have specific 2-D coordinates linked to 3-D coordinates within the point cloud media by patch metadata. Patch data (also called "auxiliary information") is based on the patch's corresponding 3D position and its associated projection position on a 2D plane. In some implementations, the auxiliary information or patch meta data may include patch indices, u0, v0, u1, v1 and d1. u0 and v0 correspond to the upper left corner of the patch in the 2D projection. u1, v1 and d1 represent the X, Y and Z coordinates of the three-dimensional domain. Thus, u0 and v0 coordinates are linked to u1, v1 and d1 coordinates. The connection is based on the mathematical relationship between u1, v1 and d1 coordinates and u0 and v0 coordinates. These connections between 2D and 3D coordinates can be used to generate motion vector candidates, update the vector candidate list, and update the motion vector search process.

20 is an example of a system representing a motion vector candidate between a patch of a current frame and a patch of a reference frame. An encoder of the client device, eg, a V-PCC encoder, parses the current patch frame into a previously encoded reference frame. The encoder must first decode the previously encoded reference frame for comparison with the current frame. The encoder then feeds the current patch frame into one or more small blocks called prediction units (PUs). Since the amount of information stored in each PU is important, the encoder uses a differential coding scheme to reduce the amount of information during transmission. The encoder creates residual frames. For example, a prediction unit PU (belonging to a patch of a reference frame) is subtracted from a current PU (belonging to a current frame), eg, a current PU-prediction unit PU to generate a residual frame. The encoder encodes only the remaining frames to be used for transmission later. For example, the following code illustrates this process:

1. Split current frame into N PU blocks

2. Generate PU_pred candidate list from the current auxiliary frame

3. for i = 0 to N-1

get the(x,y) coordinates for top left corner of the PU_cur[i]

perform search for predictor PU_pred in the reference image

cost[k] = max

while cost[k] > minCost[k-1] do

cost[k] = (PU_cur[i] - PU_pred[k]) + lambda * BitSize

encode in bitstream:

MV = PU_cur[i](x,y) - PU_pred[k](x,y)

ResidualUnit = Transform(PU_cur[i]-PU_pred[k])

The code above describes the coding mode selection process based on minimizing the speed distortion problem. In some implementations, the encoder searches for a predictor unit PU in a previously encoded reference frame and performs a search to match the value for the current PU for the amount of bits needed to encode the residual. The process of selecting a prediction unit (PU) involves a rate distortion minimization problem, where the rate corresponds to the bit size of the coded residual and the distortion corresponds to the L2 norm of the residual and origin. The encoder then encodes the residual information and the corresponding displacement information to become a motion vector. However, this implementation needs improvements when encoding and transmitting point cloud media.

The encoder is enhanced by examining the additional auxiliary information contained from the patch data. For the current frame, auxiliary information is generated from the current 2D patch and the corresponding 3D point cloud. For a reference frame, auxiliary information is generated from the previously encoded reference 2D patch and the corresponding 3D point cloud. The encoder compares the side information of the patch of the current frame, e.g. u0, v0, u1, v1, d1, with the side information of the patch of the reference frame, e.g. u0, v0, u1, v1, and d1, to locate the block. can predict As a result of the comparison, based on the auxiliary information, the distance m between the patches appears.

In some implementations, the encoder first determines the 3-D difference. The 3-dimensional difference corresponds to the 3-dimensional difference between the current auxiliary information frame and the reference auxiliary information frame since the 3-dimensional position of the starting pixel of the CU may be different. Expressions include:

Encoding then computes the two-dimensional difference. The two-dimensional difference corresponds to the two-dimensional difference between the current auxiliary information frame and the reference auxiliary information frame when considering the occupied block resolution (OBR). The OBR corresponds to the coordinate scale of the patch data group versus the patch projection size. For example, patch width equals patch projection size multiplied by OBR. The encoder then generates motion vectors in the 2D domain. Expressions include:

The final derived motion vector is a combination of these two motion vector components. For example, the final derived motion vector corresponds to:

In the non-normative solution, the motion vector (MV) is estimated as a candidate for the center of the search range. After adding the derived estimated MVs as candidates, the encoder will use RDO to select among the predicted MVs, 0 MVs, estimated MVs and MVs of partition 2Nx2N to determine the center of the search range. If the estimated MV has a minimum velocity distortion (R-D) cost, it will be used for video compression.

21 illustrates a derivation process 2100 for merging candidate list construction. In some implementations, when a prediction unit is predicted using merge mode, an index pointing to an entry in the merge candidate list is parsed from the bitstream and used to retrieve motion information. The organization of this list is specified in the HEVC standard and can be summarized in the following sequence of steps:

Step 1: Derivation of initial candidates

Step 1.1: Deriving spatial candidates

Step 1.2: Redundancy Check for Spatial Candidates

Step 1.3: Deriving Temporal Candidates

Step 2: Insert additional candidates

Step 2.1: Generate bi-prediction candidates

Step 2.2: Insert zero motion candidates

These steps are also shown schematically in FIG. 21 . For spatial merging candidate derivation, up to 4 merging candidates are selected among 5 different positional candidates. In case of temporal merging candidate derivation, at most one merging candidate among two candidates is selected. Since the decoder assumes a certain number of candidates for each prediction unit, additional candidates are generated if the number of candidates does not reach the maximum number of merge candidates (MaxNumMergeCand) signaled in the slice header. Since the number of candidates is constant, the index of the best merging candidate is encoded using truncated unary binarization (TU). If the size of the CU is 8, all PUs of the current CU share a single merge candidate list identical to that of the 2Nx2N prediction unit.

In the following, operations related to the foregoing steps are described in detail.

22 illustrates a system 2200 of positions of spatial merge candidates and candidate pairs considered for redundancy checking of spatial merge candidates. In the derivation of spatial merging candidates, up to four merging candidates are selected from candidates located at positions shown in FIG. 22 . The derivation order is A _{1 ,} B _{1 ,} B _{0 ,} A _{0 ,} and B ₂ . Position B ₂ is considered only when any PU at positions A ₁ , B ₁ , B ₀ , A ₀ is unavailable (eg, because it belongs to another slice or tile) or is intra-coded. After the candidate for the position A ₁ is added, the addition of the other candidates is redundantly checked, so that a candidate having the same motion information is excluded from the list, improving coding efficiency. In order to reduce computational complexity, not all possible pairs of candidates are considered in the mentioned redundancy check. Instead, only pairs connected by arrows in FIG. 22 are considered, and candidates are added to the list only when the corresponding candidates used for redundancy check do not have the same motion information. Another source of redundant motion information is a "second PU" associated with a partition other than 2Nx2N. For example, FIG. 23 shows the second PU for cases of Nx2N and 2NxN, respectively. If the current PU is divided into Nx2N, the candidate for position A1 is not considered for list construction. In fact, adding this candidate will lead to two prediction units with the same motion information, which is redundant to having only one PU in a coding unit. Likewise, position B1 is not considered when the current PU is divided into 2NxN.

23 illustrates a system 2300 showing a location for a second PU of Nx2N and 2NxN partitions. In particular, system 2300 describes a temporal candidate derivation process. At this stage, only one candidate is added to the list. In particular, in deriving this temporal merging candidate, a scaled motion vector is derived based on a coexisting PU belonging to a picture having the smallest POC difference from a current picture within a given reference picture list. The reference picture list to be used for deriving the coexisting PU is explicitly signaled in the slice header. The scaled motion vector for the temporal merge candidate is obtained as indicated by the dotted line in FIG. 24 (system 2400), and is scaled from the motion vector of the coexisting PU using the POC distances tb and td, where tb is the reference of the current picture It is defined as the POC difference between the picture and the current picture, and td is defined as the POC difference between the reference picture of the coexisting picture and the coexisting picture. The reference picture index of the temporal merging candidate is set to 0. For a B-slice, two motion vectors, one for reference picture list 0 and the other for reference picture list 1, are obtained and combined to become a bi-predictive merge candidate.

25 is a system 2500 illustrating candidate positions for time merge candidates. At a co-located PU(Y) belonging to a reference frame, a position for a temporal candidate is selected between candidates C ₀ and C ₁ as depicted in system 2500 . If the PU at position C ₀ is not available, intra-coded or outside the current CTU, position C ₁ is used. Otherwise, location C ₀ is used for derivation of temporal merging candidates.

Besides spatiotemporal merge candidates, there are two additional types of merge candidates: combined bi-predictive merge candidates and zero merge candidates. A combined bi-predictive merging candidate is created utilizing the spatiotemporal merging candidate. The combined bi-predictive merge candidate is used only for B-slices. Combined bi-prediction candidates are generated by combining the first reference picture list motion parameter of an initial candidate with the second reference picture list motion parameter of another candidate. If these two tuples provide different motion hypotheses, they will form a new bi-prediction candidate. For example, the system 2500 can use mvL0 and refIdxL0 or mvL1 and refIdxL1 to generate a combined bi-predictive merge candidate where the two candidates from the original list (on the left) are added to the final list (on the right). indicate the case. There are numerous rules regarding combinations that are considered to generate these additional merging candidates.

26 shows an example table 2600 of combined bi-predictive merging candidates. The MaxNumMergeCand capacity is reached as zero motion candidates are inserted to fill the remaining entries in the merge candidate list. These candidates have zero spatial displacement and reference picture indices, starting at 0 and increasing each time a new zero motion candidate is added to the list. The number of reference frames used by these candidates is 1 and 2 for unidirectional and bidirectional prediction, respectively. Finally, no redundant checks are performed on these candidates.

27 is an example of a system 2700 that includes modifications to the motion estimation pipeline using auxiliary data. System 2700 illustrates additional components to the encoder portion of the motion estimation pipeline illustrated in component 1902 of system 1900. For example, system 2700 includes auxiliary frames 2702 (also referred to as patch frame data units or patch frame data groups). System 2700 also includes new frame 2704, motion estimation component 2706, motion entropy coder 2708, motion compensated interpolation 2710, subtraction module 2712, residual encoder 2714, residual decoder 2716 ) and an adder 2718.

Motion estimation component 2706 receives an auxiliary frame 2702, a new frame 2704, and previously encoded (now decoded) residual frames from residual decoder 2716 and adder 2718.

In some implementations, when motion estimation component 2706 receives a new frame 2704 that includes patch data for the frame, motion estimation component 2706 compares the 3D coordinates associated with the patch in auxiliary frame 2702. . This is because each auxiliary frame 2702 contains its own patch metadata (or auxiliary data) that will describe the transfer between the 2-D coordinates of the patch data to the 3-D coordinates of the point cloud media within the bounding box. In particular, motion estimation 2706 uses the patch metadata of auxiliary frame 2702 to create a reference frame. The current 3D information from the current frame, eg the new frame 2704, and the 3D information from the previous frame, eg the reference frame, can then be used to determine the location of the patch. For example, motion estimation 2706 can determine that this patch in the new frame 2704 is from the same location in 3D coordinates as another patch in the reference frame, where the patch in the reference frame is another patch in the new frame 2704. It is in a 2-D position.

The motion estimation component 2706 then runs this process for each patch in the new frame 2704 and retrieves the 2-D information to obtain the corresponding 3-D information. A motion estimation component 2706 obtains 2-D information for the patch in auxiliary frame 2702 . Using the 2-D information of the patch in the new frame 2704 and the 2-D information of the patch in the auxiliary frame 2702, the motion estimation component 2706 can, for example, use the 2-D information of the patch from the new frame 2704. The 2-D distance between these two patches is determined by comparing the u0 and v0 position coordinates to the u0 and v0 position coordinates of the patch in auxiliary frame 2702, and then a new frame 2704 from the position of auxiliary frame 2702. The position of determines how the patch is moved in 2-D. This difference between u0, v0 of the patch in the auxiliary frame 2702 and u0, v0 of the patch in the new frame corresponds to the motion vector candidate. Motion estimation component 2706 inserts this generated motion vector candidate into a motion vector candidate list. The motion estimation component 2706 performs this process comparing each patch in the new frame 2704 to each patch in the auxiliary frame 2702. When a match occurs, the difference between the 2D coordinates between these matched patches is added to the motion vector candidate list.

In some implementations, when a motion vector candidate is added to the motion vector candidate list, a 3D motion search is performed on the patch in the 3D coordinate list.

In some implementations, an encoder includes a candidate list of field neighbors. A candidate list of field neighbors may include possible locations for reviewing motion within a 2D image. Motion estimation component 2706 can specify a position in the candidate list of field neighbors for each patch frame.

The process of using the auxiliary information to determine the matching patch in the reference frame is shown in the code example below. For example, using the existing patch 2D and 3D information in the auxiliary 3D information file created in the point cloud compression solution buffer for the reference frame. For example the code contains:

Algorithms. Find matched patches in reference frames while frame k in GoP frames dofor each patch in patches list of reference frame [refIdx]
for each patch in patches list of frame [k]
estimate cost in RDO based on prediction
if( patchIdx[k] = patchIdx[refIdx])
do motion refinement
end if
if( dist < bestDist)
update pcMvFieldNeighbours
end if
end for
end for end while

For k-th frame:

for(Int yCoorRef = 0; yCoorRef < picHeight; yCoorRef++) {

for(Int xCoorRef = 0; xCoorRef < picWidth; xCoorRef++) {

Double* refDepthX = m_pcPic->getDepthX(Int(eRefPicList) * MAX_NUM_REF + refIdx);

Double* refDepthY = m_pcPic->getDepthY(Int(eRefPicList) * MAX_NUM_REF + refIdx);

Double* refDepthZ = m_pcPic->getDepthZ(Int(eRefPicList) * MAX_NUM_REF + refIdx);

Double xCoorRef3D = refDepthX[yCoorRef * picWidth + xCoorRef];

Double yCoorRef3D = refDepthY[yCoorRef * picWidth + xCoorRef];

Double zCoorRef3D = refDepthZ[yCoorRef * picWidth + xCoorRef];

Double dist =(xCoor3D - xCoorRef3D) *(xCoor3D - xCoorRef3D) +(yCoor3D - yCoorRef3D) *

(yCoor3D - yCoorRef3D) + (zCoor3D - zCoorRef3D) *(zCoor3D - zCoorRef3D);

if(dist < bestDist) {

bestDist = dist;

bestXCoorRef = xCoorRef;

bestYCoorRef = yCoorRef;

}

}

}

TComMv depthMVP((bestXCoorRef - xCoor) << 2, (bestYCoorRef - yCoor) << 2);

pcMvFieldNeighbours[(iCount << 1) + 1].setMvField(depthMVP, refIdx);

In the code above, the V-PCC encoder finds a matching patch in the reference frame to the video encoder. The V-PCC encoder uses the same 2-D and 3-D information corresponding to each frame's patch and the projection information described with respect to FIG. 16 . In this step, the V-PCC encoder generates new candidates for motion improvement. The algorithm runs in the following way: The V-PCC encoder iterates through each frame of the frame group. The V-PCC encoder searches for a patch in the patch list of a specific reference frame. After that, the V-PCC encoder searches for a specific patch in the current frame.

The V-PCC encoder estimates the cost in rate distortion optimization based on the motion vector candidates. A cost or distance, for example a metric returned by the estimated cost of the rate distortion optimization function or a metric distance in a vector space, corresponds to the minimum cost from the motion vector candidate list. In particular, the minimum cost corresponds to the sum of the squares of the X difference, the Y difference, and the Z difference. For example, the X coordinate of the patch in the reference frame is subtracted from the X coordinate of the patch in the current frame to create an X difference ("Xdif"); The Y coordinate of the patch in the reference frame is subtracted from the Y coordinate of the patch in the current frame to create a Y difference (“Ydif”); Then, the Z coordinate of the patch in the reference frame is subtracted from the Z coordinate of the patch in the current frame to create a Z difference ("Zdif"). The PCC encoder then sums the squares of Xdif, Ydif and Zdif to produce a specific distance between the patch of the reference frame and the patch of the current frame.

This process of estimating cost in rate distortion optimization is repeated until a minimum cost or distance is found. If the minimum cost or distance is found, the pcMvFieldNeighbors variable is updated with the minimized cost value corresponding to the X coordinate, Y coordinate, and correct patch associated with the patch in the reference frame.

Then, the V-PCC encoder performs motion compensation based on the refined motion vector candidate and merge candidate. In some implementations, new entities are created in the merge candidate list. This entity is used as a predictor to perform motion estimation to find a better prediction for motion estimation. For example, the motion vector candidate list is updated based on 3D information about a patch and the location of the corresponding 2D patch in the projected image. Also, the merge candidate list can be updated based on the 3D information about the patch and the corresponding 2D location of the projected image. So this information is added to video compression and decompression. This information is generated for the video encoder based on the auxiliary information and the same information is generated for the video decoder. The motion vector candidate list is not transmitted or computed, but generated by the V-PCC encoder upon receiving a 2D patch along with the patch metadata. The motion vector candidate list is generated according to the “Generate Additional Information for Video Coding” code set. Both the V-PCC encoder and the V-PCC decoder perform "generate additional information for video coding" to generate a motion vector candidate list. Therefore, no additional motion vector candidate list or other information transmitted between the V-PCC encoder and the V-PCC decoder is required.

In some implementations, the auxiliary information file is generated from a point cloud coded bitstream. A point cloud coded bitstream will be described and illustrated further below. Side information is used to generate new motion vector candidates in both the V-PCC encoder and V-PCC decoder, both using the same method to ensure that the V-PCC encoder and V-PCC decoder produce the same result. Using the side information, the V-PCC encoder and V-PCC decoder generated new motion vector candidate and merge candidate list inputs. Then, new motion vector candidate and merge candidate lists are provided to the existing video encoding/decoding technology.

28 is an example of a packet stream 2800 representation of a V-PCC unit payload. Packet stream 2800 includes V-PCC bitstream 2802. The V-PCC bitstream 2802 includes one or more V-PCC units. For example, the V-PCC unit 2804 includes a V-PCC unit header 2806 and a V-PCC unit payload 2808. The V-PCC unit payload 2808 includes a sequence parameter set 2810, patch data group 2812, occupancy video data 2814, geometry video data 2818, and attribute video data 2816. Occupying video data 2814 includes 2-D frame data. Attribute video data 2816 includes two sets of 2-D frames, e.g., a near layer and a far layer. Geometry video data 2818 also includes two sets of 2-D frames, e.g., a near layer and a far layer.

The patch data group unit type includes a plurality of data sets. For example, patch data group unit type 2812 includes sequence parameter set 2820, frame geometry parameter set 2822, geometry patch parameter set 2824, frame parameter set 2826, frame attribute parameter set, and attribute patch. parameter set 2830. Additionally, the patch data group unit type 2812 includes a plurality of patch tile groups 2832. As described above, a patch tile group includes a plurality of patches. For example, the patch data group includes one set of patch tile groups of T(i, 0) to T(i, m). This information is needed to reconstruct the point cloud frame from the occupancy, attribute and geometry components of the V-PCC unit payload. "I" is the patch data group index corresponding to the 3-D PCC frame "I". M+1 is the number of 3D patches generated for 3D point cloud frame “I”; In this document, T(i, j) is referred to as a patch.

29 is another example of a visual representation 2900 of a V-PCC unit payload. As illustrated in the visual representation 2900 of the V-PCC unit payload, arrows indicate the prediction flow from the reference frame/data unit to the current data unit. Prediction between the near layer and the far layer is allowed only within the same V-PCC data frame. For example, visual representation 2900 illustrates a 3-D point cloud, patch data group, geometry video data of near and far layers, attribute video data and occupancy video data of near and far layers. d

30 is a flow diagram illustrating an example of a process 3000 for performing motion estimation using 3D auxiliary data. Process 3000 may be performed by a V-PCC encoder or a V-PCC decoder that includes the components shown in FIG. 27 .

The V-PCC encoder generates segmentation of the 3D point cloud data of the recording medium based on the continuity data of the 3D point cloud data (3002). The V-PCC encoder partitions the three-dimensional (3-D) point cloud data into a set of partitions, such as a 3-dimensional patch bounding box or a 3-dimensional patch bounding box 1504. Multiple segmentations may be created for the entirety of the 3D point cloud data covering objects within a frame of the 3D point cloud data. Each segmentation or 3D patch bounding box may be placed close to each other or overlap each other when covering objects with 3D point cloud data.

The V-PCC encoder projects a representation of the segmented 3D point cloud data onto one or more sides of a 3D bounding box, and the representation of the segmented 3D point cloud data is based on the projected side of the 3D bounding box. (3004). V-PCC encloses 3D point cloud data with a 3-D bounding box to project images of the 3D point cloud data to each side of the 3D bounding box, as shown in FIG. 16 . In some implementations, the V-PCC encoder defines criteria for selecting projection planes on the sides of the 3-D bounding box. For example, the criteria may include smooth continuous surface criteria. Thus, a specific projection plane is selected if the smooth continuous surface reference contains the projection region of the division on the side of the bounding box such that it is the largest region in each projection direction. A smooth contiguous surface can be determined using one or more smooth contiguous algorithms. A smooth continuous surface can be defined as a surface with a minimum number of occluded or occluded data points. The V-PCC encoder then compares each smooth surface in each direction and in all directions to determine which direction produces a projection containing the largest area of the side of the 3D bounding box. When a specific projection plane is selected, the V-PCC encoder projects a specific surface of the 3D point cloud data onto a specific projection plane of the 3D bounding box. In some implementations, the V-PCC encoder projects various surfaces of the 3-D point cloud data onto each side of the 3-D bounding box.

The V-PCC encoder generates (3006) one or more patches based on the projected representation of the segmented 3D point cloud data. When an image is projected onto a side of a 3D bounding box, a V-PCC encoder generates a patch from each projection. Each patch corresponding to the 2-dimensional bounding box on the side of the 3-dimensional bounding box contains a set of coordinates. A patch corresponds to a projected area on the side of a 3D bounding box. Coordinates of the patch include, for example, patch indices u0, v0, size_u0 and size_v0. The patch index identifies a specific patch associated with a specific face of the bounding box. U0 defines the X coordinate of the patch in the projection plane. V0 defines the Y coordinate of the patch in the projection plane. Size_u0 and size_v0 describe sizes corresponding to respective coordinates of patches u0 and v0, respectively.

In some implementations, the V-PCC encoder combines the patches from each side of the 3-D bounding box with their corresponding coordinate information to form a “patch tile group”. Each element of the patch tile group corresponds to a specific patch, which contains a specific unique index and corresponds to a unique 3D bounding box within the 3D point cloud frame. For example, packaged patch 1802 includes a group of patch tile groups known as patch data groups.

Coordinate information from each patch in the patch data group is included in the projection map. A projection map is an additional file that is later provided to the video encoding solution by the V-PCC encoder. The projection map contains 2D and 3D location information for each patch. The 2D data for a particular patch, eg u0, v0, corresponds to the top left corner of the X and Y coordinates of the projection. Size_u0 and size_v0 correspond to the height and width of the patch. For 3D patch projection data, the projection map contains u1, v1 and d1 coordinates corresponding to the X, Y and Z axes. In particular, u1, v1 and d1 correspond to projection planes corresponding to indexes or normal axes of projection planes.

The V-PCC encoder generates a first frame of one or more patches (3008). The V-PCC encoder generates a frame of the patch data group generated in (3006). The frame may be encoded and provided to a video compression solution for transmission.

The V-PCC encoder generates first auxiliary information for the first frame (3010). The first auxiliary information corresponds to patch metadata for a specific patch of the first frame. For example, the first auxiliary information may include patch indices u0, v0, u1, v1, and d1 for a specific patch.

The V-PCC encoder generates second auxiliary information for the reference frame (3012). The V-PCC encoder searches for a reference frame, which is a previously encoded frame containing one or more patches. After decoding the reference frame, the V-PCC encoder retrieves a patch from the reference frame and retrieves auxiliary information related to the patch from the reference frame. Auxiliary information related to patches in the reference frame includes patch indices u0, v0, u1, v1, and d1 for a specific patch.

The V-PCC encoder identifies a first patch in the first frame that matches the second patch in the reference frame based on the first auxiliary information and the second auxiliary information (3014). The V-PCC compares the auxiliary information associated with the first frame with the auxiliary information associated with the patch of reference. For example, V-PCC compares the patch indices u0, v0, v1 and d1 associated with each patch to determine the distance between these pieces of information. The V-PCC encoder compares the patch in the first frame with each patch in the reference frame to find the matching patch with the minimum side information distance. When the distance is minimized, the V-PCC encoder represents a matching patch between the two patches.

The V-PCC encoder generates a motion vector candidate between the first patch and the second patch based on the difference between the first side information and the second side information (3016). A motion vector candidate is generated between the first patch and the second patch when the difference between the corresponding side information compared to other patches is minimal. Motion vector candidates are added to the list of motion vector candidates used for transmission. The difference may be based on the estimated cost of velocity distortion optimization. A cost or distance, for example a metric returned by the estimated cost of the rate distortion optimization function or a metric distance in a vector space, corresponds to the minimum cost from the motion vector candidate list. If the minimum cost or distance is found, the pcMvFieldNeighbors variable is updated with the X coordinate, Y coordinate and minimized cost value corresponding to the correct patch associated with the patch in the reference frame.

The V-PCC encoder performs motion compensation using the motion vector candidate (3018). Then, the V-PCC encoder performs motion compensation based on the refined motion vector candidate and merge candidate. In some implementations, new entities are created in the merge candidate list. This entity is used as a predictor to perform motion estimation to find a better prediction for motion estimation. For example, the motion vector candidate list is updated based on 3D information about a patch and the location of the corresponding 2D patch in the projected image. Also, the merge candidate list can be updated based on the 3D information about the patch and the corresponding 2D location of the projected image. So this information is added to video compression and decompression. This information is generated for the video encoder based on the auxiliary information and the same information is generated for the video decoder. The motion vector candidate list is not transmitted or computed, but generated by the V-PCC encoder upon receiving a 2D patch along with the patch metadata. The motion vector candidate list is generated according to the “Generate Additional Information for Video Coding” code set. Thereafter, new motion vector candidate and merge candidate lists are provided to the existing video encoding/decoding technology.

A first component is directly connected to a second component when there is no intervening component between the first component and the second component other than a line, trace or other media. A first component is indirectly coupled to a second component when there is an intervening component other than a line, trace, or other media between the first component and the second component. The term "linkage" and variations thereof include both direct and indirect linkages. Use of the term "about" means a range inclusive of ±10% of the following number unless otherwise specified.

Although several embodiments have been provided in this disclosure, it is to be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. The present examples are to be regarded as illustrative rather than restrictive, and the intent is not to be limited to the details provided herein. For example, various elements or components may be combined or integrated in other systems, or certain functions may be omitted or not implemented.

Additionally, the techniques, systems, subsystems, and methods described and illustrated in the various embodiments, individually or separately, may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of alterations, substitutions and alterations may be identified by those skilled in the art and may be made without departing from the spirit and scope of the disclosure herein.

The embodiments and all functional operations of the invention described herein may be implemented in digital electronic circuitry, or computer software, firmware or hardware, or a combination of one or more of these, including structures disclosed in this specification and their structural equivalents. Embodiments of the invention may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on computer readable media to be executed by or to control the operation of a data processing device. . A computer-readable medium may be a non-transitory computer-readable storage medium, a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagated signal, or a combination of one or more of these. . The term “data processing device” includes all devices, devices and machines for processing data, including, for example, programmable processors, computers or multiple processors or computers. The device may include, in addition to hardware, code that creates an execution environment for the computer program in question, such as code that makes up a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. A propagated signal is an artificially generated signal, for example a machine generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device.

A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and is a stand-alone program or module suitable for use in a computing environment. may be distributed in any form, including as a component, subroutine, or other device. Computer programs do not necessarily correspond to files in a file system. A program is a part of a file holding another program or data (for example, one or more scripts stored in a markup language document), a single file dedicated to that program, or several coordinated files (for example, one or more modules, subprograms). or a file that stores part of the code). A computer program may be distributed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed, and the apparatus may be implemented with special purpose logic circuits, such as, for example, field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). can

Processors suitable for the execution of computer programs include, for example, general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, processors receive instructions and data from either read-only memory or random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes or is operably coupled to receive data from or transmit data from one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or both. However, computers do not need these devices. Moreover, the computer may be embedded in other devices such as tablet computers, mobile phones, personal digital assistants (PDAs), mobile audio players, and Global Positioning System (GPS) receivers. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices. magnetic disks such as internal hard disks or removable disks; magneto-optical disk; and CD ROM and DVD-ROM discs. The processor and memory may be complemented or integrated by special purpose logic circuitry.

To provide interaction with a user, an embodiment of the present invention provides a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to a user and a user It may be implemented in a computer that includes a keyboard and a pointing device capable of providing input to the computer, such as a mouse or trackball. Other types of devices may be used to provide interaction with the user; For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; Also, input from the user may be received in any form including sound, voice, or tactile input.

Embodiments of the present invention may include a backend component, eg as a data server, or may include a middleware component, eg an application server, or may include a web or graphical user interface, eg, through which a user may interact with an implementation of the present invention. It may be implemented in a computing system that includes a front-end component as a client computer with a browser, or a combination of one or more of such back-ends, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, for example a communication network. Examples of communication networks include local area networks ("LAN") and wide area networks ("WAN"), such as the Internet.

A computing system may include a client and a server. Clients and servers are usually remote from each other and usually interact through a communication network. The relationship of client and server arises due to computer programs running on each computer and having a client-server relationship with each other.

Several implementations have been detailed above, but other modifications are possible. For example, although a client application is described as accessing the delegate(s), in other implementations the delegate(s) may be used by other applications implemented by one or more processors, such as applications running on one or more servers. Further, the logic flows depicted in the figures do not require any specific order or sequential order shown to achieve desired results. In addition, other operations may be provided or removed from the described flow, and other components may be added to or removed from the described system. Accordingly, other implementations are within the scope of the following claims.

Although this specification contains many specific implementation details, they should not be construed as limitations on the scope of any invention or what may be claimed, but rather as a description of features that may be specific to a particular embodiment of a particular invention. . Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Moreover, while features may be described above as acting in particular combinations, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be a subcombination or subcombination. Variations of the combination may be oriented.

Similarly, while actions are depicted in the drawings in a particular order, this should not be construed as requiring that such actions be performed in the specific order shown or in sequential order, or that all illustrated actions be performed to achieve a desired result. do. Multitasking and parallel processing can be advantageous in certain situations. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and that the described program components and systems may generally be integrated into a single software product. You have to understand. It is packaged into several software products.

Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still obtain desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (20)

As a computer implemented method,
generating, by an encoder, a segmentation of the 3D point cloud data of the recorded media based on the continuity data of the 3D point cloud data;
Projecting, by the encoder, a representation of the segmented 3-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, the representation of the segmented 3-dimensional point cloud data being the 3-dimensional bounding box; different based on the projected side of -;
In response to projecting the segmented representation of 3D point cloud data onto one or more sides of a 3D bounding box, the encoder performs one or more patches based on the projected representation of the segmented 3D point cloud data. generating them;
generating, by the encoder, a first frame of the one or more patches;
generating, by the encoder, first auxiliary information for the first frame;
generating, by the encoder, second auxiliary information for a reference frame, the reference frame including one or more second patches, the reference frame being a previously encoded and transmitted frame;
identifying, by the encoder, a first patch from a first frame that matches a second patch from one or more second patches of the reference frame based on the first and second auxiliary information;
generating, by the encoder, a motion vector candidate between the first patch and the second patch based on a difference between the first side information and the second side information; and
The encoder performing motion compensation using the motion vector candidate.
A computer implemented method comprising a. According to claim 1,
wherein the reference frame is decoded to generate the second auxiliary information. According to claim 1 or 2,
The step of generating a segmentation of the 3D point cloud data of the recorded media is to:
generating a plurality of partitions for the three-dimensional point cloud media for the encoder to subsequently project and encode each partition of the plurality of partitions;
Further comprising, a computer implemented method. According to claim 1 or 2,
The first auxiliary information includes index data for each of the one or more patches, two-dimensional data for each of the one or more patches, and three-dimensional data for each of the one or more patches. According to claim 4,
wherein the index data for each of the one or more patches corresponds to a corresponding side of the three-dimensional bounding box. According to claim 4,
wherein the two-dimensional data for each of the one or more patches and the three-dimensional data for each of the one or more patches correspond to and are concatenated with a portion of the three-dimensional point cloud data. According to claim 1 or 2,
Generating one or more patches of the 3D point cloud data based on continuity data of the 3D point cloud data includes:
determining, by the encoder, smoothness criteria of the 3D point cloud data from each direction;
comparing, by the encoder, the smoothness criterion from each direction of the 3D point cloud data; and
In response to the comparison, the encoder selecting a direction of a smoothness criterion of the 3-dimensional point cloud data having a larger projection area on the side of the bounding box.
Further comprising, a computer implemented method. According to claim 1 or 2,
Generating a motion vector candidate between the first patch and the second patch:
determining, by the encoder, a distance between the two-dimensional data of the first auxiliary information and the two-dimensional data of the second auxiliary information;
generating, by the encoder, the motion vector candidate based on the distance between the 2D data of the first side information and the 2D data of the second side information; and
adding, by the encoder, the motion vector candidate to a motion vector candidate list;
Further comprising, a computer implemented method. As a system,
comprising one or more computers and one or more storage devices storing operable instructions, which when executed by the one or more computers cause the one or more computers to:
generating a segmentation of the 3D point cloud data of the recorded media based on the continuity data of the 3D point cloud data;
Projecting the segmented representation of 3-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, wherein the segmented representation of 3-dimensional point cloud data is a projection of the 3-dimensional bounding box. different based on aspect -;
in response to projecting the segmented representation of 3-dimensional point cloud data onto one or more sides of a 3-dimensional bounding box, generating one or more patches based on the projected representation of the segmented 3-dimensional point cloud data;
generating a first frame of the one or more patches;
generating first auxiliary information for the first frame;
generating second auxiliary information for a reference frame, the reference frame including one or more second patches, the reference frame being a previously encoded and transmitted frame;
identifying a first patch from a first frame that matches a second patch from one or more second patches of the reference frame based on the first auxiliary information and the second auxiliary information;
generating a motion vector candidate between the first patch and the second patch based on a difference between the first auxiliary information and the second auxiliary information; and
performing motion compensation using the motion vector candidate;
A system that allows to perform an operation that includes. According to claim 9,
wherein the reference frame is decoded to generate the second auxiliary information. The method of claim 9 or 10,
The step of generating a segmentation of the 3D point cloud data of the recorded media is to:
generating a plurality of partitions of the three-dimensional point cloud media for subsequently projecting and encoding each partition of the plurality of partitions.
Further comprising a system. The method of claim 9 or 10,
The first auxiliary information includes index data for each of the one or more patches, 2-dimensional data for each of the one or more patches, and 3-dimensional data for each of the one or more patches. According to claim 12,
wherein the index data for each of the one or more patches corresponds to a corresponding side of the three-dimensional bounding box. According to claim 12,
wherein the two-dimensional data for each of the one or more patches and the three-dimensional data for each of the one or more patches correspond to and are concatenated with a portion of the three-dimensional point cloud data. The method of claim 9 or 10,
Generating one or more patches of the 3D point cloud data based on continuity data of the 3D point cloud data includes:
determining smoothness criteria of the 3D point cloud data from each direction;
comparing the smoothness criterion from each direction of the 3D point cloud data; and
Responsive to the comparison, selecting a direction of a smoothness criterion of the three-dimensional point cloud data having a larger projection area on the side of the bounding box.
Further comprising a system. The method of claim 9 or 10,
Generating a motion vector candidate between the first patch and the second patch:
determining a distance between the two-dimensional data of the first auxiliary information and the two-dimensional data of the second auxiliary information;
generating the motion vector candidate based on a distance between the 2D data of the first auxiliary information and the 2D data of the second auxiliary information; and
adding the motion vector candidate to a motion vector candidate list;
Further comprising a system. As a non-transitory computer-readable storage medium,
Stored are instructions executable by one or more processing devices, which upon execution cause the one or more processing devices to:
generating a segmentation of the 3D point cloud data of the recorded media based on the continuity data of the 3D point cloud data;
Projecting the segmented representation of 3-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, wherein the segmented representation of 3-dimensional point cloud data is a projection of the 3-dimensional bounding box. different based on aspect -;
in response to projecting the segmented representation of 3-dimensional point cloud data onto one or more sides of a 3-dimensional bounding box, generating one or more patches based on the projected representation of the segmented 3-dimensional point cloud data;
generating a first frame of the one or more patches;
generating first auxiliary information for the first frame;
generating second auxiliary information for a reference frame, the reference frame including one or more second patches, the reference frame being a previously encoded and transmitted frame;
identifying a first patch from a first frame that matches a second patch from one or more second patches of the reference frame based on the first auxiliary information and the second auxiliary information;
generating a motion vector candidate between the first patch and the second patch based on a difference between the first auxiliary information and the second auxiliary information; and
performing motion compensation using the motion vector candidate;
One or more non-transitory computer-readable storage media that cause the performance of operations comprising: According to claim 17,
wherein the reference frame is decoded to generate the second auxiliary information. According to any one of claims 17 to 18,
The step of generating a segmentation of the 3D point cloud data of the recorded media is to:
generating a plurality of partitions of the three-dimensional point cloud media for subsequently projecting and encoding each partition of the plurality of partitions.
Further comprising, non-transitory computer readable storage medium. The method of claim 1 , wherein the first auxiliary information for the first frame corresponds to patch metadata for a specific patch of the first frame.

Images