KR20210132997A - Apparatus and method for providing 3d video based on v-pcc bitstream - Google Patents

Abstract

The present invention relates to a device and method for providing a 3D video based on video based point cloud compression (V-PCC) bitstream to support 3D point cloud contents through a transmission protocol adopted in a next-generation broadcasting standard. A method for providing a 3D video comprises: a step of receiving a 3D point cloud content; a step of encoding the 3D point cloud content based on a video based point cloud compression (V-PCC) bitstream; and a step of generating dynamic adaptive streaming over HTTP (DASH) segment based on the V-PCC bitstream.

Description

V-PCC bitstream-based 3D image providing apparatus and method {APPARATUS AND METHOD FOR PROVIDING 3D VIDEO BASED ON V-PCC BITSTREAM}

The present invention relates to a V-PCC (Video based Point Cloud Compression) bitstream-based 3D video providing apparatus and method, and more particularly, to a 3D point cloud content through a transmission protocol adopted in a next-generation broadcasting standard. It relates to an apparatus and method for providing a 3D image for supporting.

Media content has evolved from black-and-white to color images and from low-resolution to high-resolution images. Recently, user-centered 360-degree VR (Virtual Realtiy) content for content similar to reality and UWV (Ultra Wide Vision) content, which is a immersive media with a wide viewing angle, have appeared. The use of HMD (Head Mount Display), etc. has also increased.

As such, media technology has been repeatedly developed to provide users with a realistic experience, and further, they have begun to turn their attention to three-dimensional media in order to provide users with a free viewing angle and a three-dimensional effect. Among them, 3D point cloud contents are attracting attention as next-generation media in the AR/VR and autonomous vehicle fields.

However, since the 3D point cloud content occupies a significant amount of memory and processor resources, a large storage space is required to transmit the 3D point cloud content compared to the existing 2D image.

Accordingly, the demand for a 3D image providing apparatus and method for supporting 3D point cloud content through a transmission protocol adopted in the next generation broadcasting standard is gradually increasing in the industry.

Accordingly, the present invention has been devised to solve the above problems, and an object of the present invention is to provide an apparatus and method for providing a video based point cloud compression (V-PCC) bitstream-based 3D image.

According to a feature of the present invention for achieving the above object, there is provided a method for providing a three-dimensional image, the method comprising: receiving three-dimensional point cloud content; encoding the three-dimensional point cloud content based on a video based point cloud compression (V-PCC) bitstream; and generating a Dynamic Adaptive Streaming over HTTP (DASH) segment based on the V-PCC bitstream.

In addition, the DASH segment includes at least one of an initial segment based on the V-PCC bitstream and a media segment.

In addition, the initial segment is characterized in that it includes information encoded for each V-PCC bitstream, respectively.

In addition, the media segment is characterized in that it includes information about the reproduction of the 3D point cloud content.

In addition, the media segment includes at least one of metadata related to the 3D point cloud content, geometric image information, texture image information, color image information, and occupancy map information.

The method further includes generating a Media Processing Unit (MPU) based on the V-PCC bitstream.

In addition, the MPU is characterized in that it includes conversion information and identification information to a Media Fragment Unit (MFU).

In addition, in the computer-readable recording medium according to another embodiment of the present invention for solving the above technical problem, a program for performing the above method may be recorded.

In addition, a three-dimensional image providing apparatus according to another embodiment of the present invention for solving the above technical problem, the input unit for receiving the three-dimensional point cloud content; a data processing unit for encoding the three-dimensional point cloud content based on a video based point cloud compression (V-PCC) bitstream; and a data generator for generating a Dynamic Adaptive Streaming over HTTP (DASH) segment based on the V-PCC bitstream, wherein the data generator generates the V-PCC bitstream based Media Processing (MPU) through extension of MPEG-ISOBMFF. It is characterized by creating a unit).

In addition, the DASH segment includes at least one of an initial segment based on the V-PCC bitstream and a media segment.

In addition, the initial segment is characterized in that it includes information encoded for each V-PCC bitstream, respectively.

In addition, the media segment is characterized in that it includes information about the reproduction of the 3D point cloud content.

In addition, the media segment includes at least one of metadata related to the 3D point cloud content, geometric image information, texture image information, color image information, and occupancy map information.

In addition, the MPU is characterized in that it includes conversion information and identification information to a Media Fragment Unit (MFU).

The present invention can support 3D point cloud content through the transport protocol adopted in the next-generation broadcasting standard.

In addition, the present invention may configure one or more media tracks in order to independently restore 3D point cloud content by a Dynamic Adaptive Streaming over HTTP (DASH) segment and a Media Processing Unit (MPU).

In addition, the present invention can reduce storage space, transmission time, and hardware cost for 3D point cloud content.

1 shows a V-PCC bitstream structure for explaining the present invention.
2 is a flowchart S200 for explaining a method for providing a 3D image according to an embodiment of the present invention.
3 is a functional block diagram of an apparatus 300 for providing a 3D image according to an embodiment of the present invention.
4 illustrates a structure of an initial segment 331 used to transmit 3D point cloud data through a DASH protocol according to an embodiment of the present invention.
5 illustrates a structure of a media segment 332 used to transmit 3D point cloud data through a DASH protocol according to an embodiment of the present invention.
Figure 6 shows the structure of the V-PCC bitstream-based MPU (333) according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted. In addition, embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto and may be variously implemented by those skilled in the art.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "indirectly connected" with another element interposed therebetween. . Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms.

1 shows a V-PCC bitstream structure 100 for explaining the present invention.

The 3D image providing apparatus 300 according to an embodiment of the present invention may receive 3D cloud content through a V-PCC (Video based Point Cloud Compression) bitstream structure and perform encoding/decoding. Embodiments to be described below reconstruct an image based on 3D point cloud content in augmented reality (AR), virtual reality (VR), or mixed reality in which virtual reality and augmented reality are mixed, or It may be used to model an object, but is not limited thereto.

The 3D image providing apparatus 300 according to an embodiment of the present invention projects the dynamic point cloud data into a 2D patch, and converts the video sequence composed of the 2D patch into a 2D patch. Encoding/decoding may be performed using an existing video codec such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or the like. A V-PCC bitstream structure for implementing this will be described in more detail below.

1 , a V-PCC bitstream unit according to an embodiment of the present invention may include a V-PCC unit header and a V-PCC unit payload. can

For example, V-PCC unit payload data may be identified through V-PCC unit header information, and V-PCC unit payload data may include decoder initialization information and point cloud data. The decoder initialization information may be composed of patch sequence data (PSD) indicating overall encoding information of the bitstream, and the point cloud data may be composed of a geometric image, a color image, an occupancy map, and the like.

As shown in FIG. 1 , the V-PCC unit payload data according to an embodiment is largely a V-PCC Parameter Set (VPS) including decoder initialization information, and Atlas Data (AD) including each 2D image information. , may be composed of Occupancy Video Data (OVD), Geometry Video Data (GVD), Attribute Video Data (AVD), etc. including the actual compressed 3D point cloud data. In this case, the V-PCC Parameter Set (VPS) may indicate overall encoding information of the generated bitstream.

Specifically, OVD (Occupancy Video Data), GVD (Geometry Video Data), and AVD (Attribute Video Data) included in the V-PCC unit payload data is a codec ( codec) type, a 2D bit depth used to represent an image value, or a 3D bit depth used to represent a 3D coordinate value when reconstructing, and the like. In addition, the AD (Atlas Data) included in the V-PCC unit payload data may include parameters according to the patch generation method, for example, the width, height, and projection of the patch ( projection), the number of directions, a patch generation method, a rotation direction of the patch, and the like.

That is, the 3D image providing apparatus 300 is characterized in that it receives 3D cloud content through a V-PCC (Video based Point Cloud Compression) bitstream structure and performs encoding/decoding, which will be described in more detail below. to be described as

2 is a flowchart S200 for explaining a method for providing a 3D image according to an embodiment of the present invention, and FIG. 3 is a functional block of the apparatus 300 for providing a 3D image according to an embodiment of the present invention. shows the diagram

The 3D image providing apparatus 300 may be configured to execute each step constituting the 3D image providing method according to an embodiment of the present invention. For example, as exemplarily shown in FIG. 2 , The 3D image providing apparatus 300 may be configured to execute each step of the 3D image providing method S200 .

Meanwhile, each operation in FIG. 2 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

2 and 3 , an apparatus 300 for providing a 3D image according to an embodiment of the present invention includes an input unit 310 , a data processing unit 320 , a data generation unit 330 , a control unit 340 , and It may be composed of a storage unit 350, and the like.

The input unit 310 may receive 3D point cloud content (S210). For example, the input unit 310 may receive a 3D image (or content) from an external device or server, or may acquire a 3D image (or content) from an internal memory. The 3D point cloud content may include information on points in a 3D space, and may consist of a plurality of 3D frames.

In an embodiment, the input unit 310 is provided to receive 3D point cloud content through a direct connection with an external device or server or a network connection, and may be implemented as a wired and/or wireless communication unit.

For reference, the communication unit is LAN, WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), WiBro (Wireless Broadband Internet), RF (Radio Frequency) communication, wireless LAN, Wi-Fi (Wireless) Fidelity), NFC (Near Field Communication), Bluetooth, infrared communication, etc. However, this is an example, and according to an embodiment to which the present invention is applied, various wired and wireless communication technologies applicable in the art may be used.

The data processing unit 320 may encode the 3D point cloud content based on the V-PCC bitstream (S220). For example, the data processing unit 320 converts the received 3D point cloud content data into a 2D patch through projection, and a video composed of a 2D patch using an existing video codec. Encoding/decoding may be performed on a sequence.

The data generator 330 may configure a V-PCC bitstream-based Dynamic Adaptive Streaming over HTTP (DASH) segment and a Media Processing Unit (MPU) ( S230 ). For example, the data generator 330 may provide an initial segment ( 331 in FIG. 4 ) and a media segment defined in the DASH standard in order to provide a 3D point cloud content to a user as a transport protocol MPEG-DASH. , 332 of FIG. 5 ) may be generated.

The initial segment 331 is configured to include information about decoder initialization information and media metadata of 3D point cloud content, and the media segment 332 is an existing file format standard of MPEG-ISOBMFF. It may be configured in consideration of V-PCC bitstream data through extension. The structure of the DASH segment including the initial segment and the media segment will be described in more detail below.

In an embodiment, the data generator 330 may configure a Media Processing Unit (MPU, 333 of FIG. 6 ) in consideration of the V-PCC bitstream encoding parameters and data. For example, the data generation unit 330 adds a depth information parameter to a signaling message defined in the MPEG-DASH standard through the MPU structure to be described in detail below to provide the user with three-dimensional coordinates of the three-dimensional point cloud content. Information may be provided together (S240).

The controller 340 may control the 3D image providing apparatus 300 to provide 3D point cloud content to the user. For example, the controller 340 is implemented as a controller, a processor, a micro-processor, a micro-controller, etc. of the 3D image providing apparatus 300 . It is possible to collectively control operations, functions, and the like.

The storage 350 may store various data related to 3D point cloud content. For reference, the storage unit 350 includes a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), an electrically erasable and programmable read only memory (EEPROM), and a flash, as known to those skilled in the art. Various types of storage devices that allow input and output of information, such as flash memory, CF (Compact Flash) card, SD (Secure Digital) card, SM (Smart Media) card, MMC (Multimedia) card, or Memory Stick may be implemented, and may be provided inside the 3D image providing apparatus 300 or may be provided in a separate external device.

In addition, according to an additional embodiment of the present invention, an additional memory for data backup may be further provided in the storage unit 350 or separately from the storage unit 350 , and the control unit 340 may include the storage unit By backing up various data related to the 3D point cloud content stored in 350 and storing it in the additional memory, it is possible to actively cope with data loss or loss.

For reference, each element 310 , 320 , 330 , 340 , and 350 of the 3D image providing apparatus 300 shown in FIG. 3 is an exemplary element for explaining the operation and function of the 3D image providing apparatus 300 . It will be apparent that only corresponds to, but is not limited to, an additional element (eg, a display for outputting three-dimensional content, etc.) may be further implemented. Hereinafter, the structure of each of the initial segment 331 , the media segment 332 , and the MPU 333 will be described in detail.

4 illustrates a structure of an initial segment 331 used to transmit 3D point cloud data through a DASH protocol according to an embodiment of the present invention.

Referring to FIG. 4 , the initial segment 331 is composed of a 'ftyp' box and a 'moov' box in consideration of V-PCC bitstream encoding parameters through extension of MPEG-ISOBMFF, which is an existing file format standard. can

The 'ftyp' box may include file compatibility information so that the major brand is set to 'msdh' and the compatible brand is set to 'volm'. The 'moov' box may include a 'track' box including encoding information for each V-PCC bitstream.

For example, the 'moov' box may include a PCC metadata track box, a geometric track box, a texture track box, an occupancy map track box, and the like. In this case, the PCC metadata track box may include a 'tref' box indicating relationship information of the entire V-PCC bitstream and a 'stbl' box including auxiliary patch information. In addition, the geometric track box, the texture track box, the occupancy map track box, and the like may include video sequence encoding information and patch encoding information composed of patches, respectively.

5 illustrates a structure of a media segment 332 used to transmit 3D point cloud data through a DASH protocol according to an embodiment of the present invention.

5, the media segment 332 considers V-PCC bitstream data through the extension of MPEG-ISOBMFF, which is an existing file format standard, a 'styp' box, a 'moof' box, an 'mdat' box, etc. can be composed of

The 'styp' box may include file compatibility information so that the major brand is set to 'msdh' and the compatible brand is set to 'volm'. The 'moof' box may include a PCC metadata track fragment box including reproduction information, a geometric track fragment box, a texture track fragment box, an occupancy map track fragment box, and the like. The 'mdat' box may include geometric image data, color image data, occupancy map data, and the like.

Figure 6 shows the structure of the V-PCC bitstream-based MPU (333) according to an embodiment of the present invention.

Referring to FIG. 6 , the V-PCC bitstream-based MPU 333 according to an embodiment of the present invention considers V-PCC bitstream encoding parameters and data through extension of MPEG-ISOBMFF, which is an existing file format standard. Thus, it may be composed of a 'ftyp' box, an 'mmpu' box, a 'moov' box, a 'moof' box, an 'mdat' box, and the like.

The 'ftyp' box may include file compatibility information so that the major brand is set to 'msdh' and the compatible brand is set to 'volm'. The 'mmpu' box can be given the independence of the MPU by specifying the access user account, sequence number (number), and the like. The 'moov' box contains V-PCC bitstream encoding information and information on conversion from MPU to MFU (Media Fragment Unit), PCC metadata track box, geometric track box, texture track box, occupancy map track box, hint It may be composed of a track box or the like.

Specifically, the PCC metadata track box may include a 'tref' box indicating relationship information of the entire V-PCC bitstream and a 'stbl' box including auxiliary patch information. Also, each of the geometric track box, the texture track box, and the occupancy map track box may include video sequence encoding information and patch encoding information composed of patches.

In addition, the hint track box may include information about conversion to the MPU, and the 'mmth' box in the hint track box may provide information about a variable that distinguishes whether the data of the corresponding track is timed or non-timed. have. The 'moof' box may consist of a PCC metadata hint traf box including reproduction information, a geometric hint traf box, a texture hint traf box, an occupancy map hint traf box, and the like. The 'mdat' box may include geometric image hint data, color image hint data, occupancy map hint data, and the like.

That is, the 3D image providing apparatus 300 processes the encoding information and data of the V-PCC bitstream through the extension of ISOBMFF, which is the existing file format standard, and thus 3D point cloud content through the transmission protocol adopted in the next-generation broadcasting standard. can support

Furthermore, in one embodiment, the 3D image providing apparatus 300 may provide the user with information on the resolution of the 3D point cloud content and coordinate information as a reference for divided transmission of the 3D point cloud content together. have.

As described above, the present invention can support 3D point cloud content through the transport protocol adopted in the next-generation broadcasting standard.

In addition, the present invention may configure one or more media tracks in order to independently restore 3D point cloud content by a Dynamic Adaptive Streaming over HTTP (DASH) segment and a Media Processing Unit (MPU).

In addition, the present invention can reduce storage space, transmission time, and hardware cost for 3D point cloud content.

Meanwhile, various embodiments described herein may be implemented by hardware, middleware, microcode, software, and/or a combination thereof. For example, various embodiments may include one or more application specific semiconductors (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). ), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or a combination thereof.

Also, for example, the various embodiments may be embodied in or encoded on a computer-readable medium comprising instructions. The instructions embodied or encoded on the computer-readable medium may cause a programmable processor or other processor to perform a method, eg, when the instructions are executed. A storage medium may be any available medium that can be accessed by a computer. For example, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage medium, magnetic disk storage medium or other magnetic storage device, or desired program code, containing instructions or may include any other medium that can be used for storage in the form of data structures.

Such hardware, software, firmware, etc. may be implemented in the same device or in separate devices to support the various operations and functions described herein. Additionally, components, units, modules, components, etc. described as “parts” in the present invention may be implemented together or individually as separate but interoperable logic devices. Depictions of different features of modules, units, etc. are intended to emphasize different functional embodiments, and do not necessarily imply that they must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

Although acts are shown in the figures in a particular order, it should not be understood that these acts need to be performed in the particular order shown, or sequential order, or all shown acts need to be performed to achieve a desired result. . In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the division of various components in the above-described embodiments should not be construed as requiring such division in all embodiments, and the described components will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there can be

As described above, the best embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention and are not used to limit the meaning or scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: V-PCC bitstream structure 300: 3D image providing device
310: input unit 320: data processing unit
330: data generator 331: initial segment
332: media segment 333: MPU
340: control unit 350: storage unit

Claims (14)

Receiving a three-dimensional point cloud content;
encoding the three-dimensional point cloud content based on a video based point cloud compression (V-PCC) bitstream; and
A method of providing a 3D image, comprising generating a Dynamic Adaptive Streaming over HTTP (DASH) segment based on the V-PCC bitstream.
The method of claim 1,
The DASH segment includes at least one of an initial segment based on the V-PCC bitstream and a media segment.
3. The method of claim 2,
The initial segment, characterized in that each includes information encoded for each V-PCC bitstream, a 3D image providing method.
3. The method of claim 2,
The media segment, characterized in that it includes information on the reproduction of the 3D point cloud content, a 3D image providing method.
5. The method of claim 4,
The media segment is
and at least one of metadata related to the 3D point cloud content, geometric image information, texture image information, color image information, and occupancy map information.
The method of claim 1,
Further comprising the step of generating the V-PCC bitstream-based MPU (Media Processing Unit), the three-dimensional image providing method.
7. The method of claim 6,
The MPU is
A method for providing a three-dimensional image, characterized in that it includes conversion information and identification information into a Media Fragment Unit (MFU).
A computer-readable recording medium in which a program for performing the method according to any one of claims 1 to 7 is recorded.
In the 3D image providing device,
an input unit for receiving 3D point cloud content;
a data processing unit for encoding the three-dimensional point cloud content based on a video based point cloud compression (V-PCC) bitstream; and
A data generator for generating a DASH (Dynamic Adaptive Streaming over HTTP) segment based on the V-PCC bitstream,
3D image providing apparatus, characterized in that the data generating unit generates the V-PCC bitstream-based MPU (Media Processing Unit) through the extension of MPEG-ISOBMFF.
10. The method of claim 9,
The DASH segment includes at least one of an initial segment based on the V-PCC bitstream and a media segment.
11. The method of claim 10,
The initial segment, the 3D image providing apparatus, characterized in that each includes information encoded for each V-PCC bitstream.
11. The method of claim 10,
The media segment, characterized in that it includes information on the reproduction of the 3D point cloud content, a 3D image providing apparatus.
11. The method of claim 10,
The media segment is
and at least one of metadata related to the 3D point cloud content, geometric image information, texture image information, color image information, and occupancy map information.
10. The method of claim 9,
The MPU is
3D image providing apparatus, characterized in that it includes conversion information and identification information into a Media Fragment Unit (MFU).

Images