KR102417959B1 - Apparatus and method for providing three dimensional volumetric contents - Google Patents

Abstract

According to an embodiment, a method for providing 3D stereoscopic content includes extracting a 3D point cloud based on 2D object images captured by a plurality of cameras; and outputting an encoded bitstream by decomposing the three-dimensional point cloud into a plurality of subframes, differentially compressing the plurality of subframes, and converting the plurality of subframes into a two-dimensional frame representing a geometric image and a texture image; can do.

Description

Apparatus and method for providing 3D stereoscopic content {APPARATUS AND METHOD FOR PROVIDING THREE DIMENSIONAL VOLUMETRIC CONTENTS}

The present invention relates to an apparatus for providing three-dimensional stereoscopic content, and more particularly, by decomposing and outputting a three-dimensional point cloud into two-dimensional frames through a differential compression technology according to a user's experience quality, it is possible to implement an efficient streaming service in a mobile communication network. The present invention relates to an apparatus for providing 3D stereoscopic content and a method therefor.

Numerous media contents that people encounter have developed from black-and-white to color images and from low-resolution to high-resolution images. In addition, user-centered 360-degree VR (Virtual Reality) contents and UWV (Ultra-Wide Vision) contents, which are immersive media with wide viewing angles, have appeared to provide content similar to reality. They started to use curved displays and HMDs (Head Mount Displays).

As such, media technology has evolved to provide a realistic experience, and has begun to turn its attention to three-dimensional media that provides users with a free viewing angle and three-dimensional effect. Among them, 3D point cloud is attracting attention as the next-generation media in AR/VR and autonomous vehicle fields.

However, in order to express a 3D point cloud, tens of thousands to hundreds of thousands of point data are required, and a large amount of storage space is required compared to the existing 2D image. Accordingly, the development of various technologies is required to efficiently service the massive data of the 3D point cloud.

The embodiment decomposes a large-capacity three-dimensional point cloud into two-dimensional frames, but differentially compresses it according to a pre-allocated importance factor in consideration of the user's experience quality, thereby implementing an efficient streaming service in a mobile communication network. to provide that method.

The technical problem to be solved in the embodiment is not limited to the technical problem mentioned above, and another technical problem not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

According to an embodiment, extracting a 3D point cloud based on 2D object images captured by a plurality of cameras; and outputting an encoded bitstream by decomposing the three-dimensional point cloud into a plurality of subframes, differentially compressing the plurality of subframes, and converting the plurality of subframes into a two-dimensional frame representing a geometric image and a texture image; It is possible to provide a method for providing 3D stereoscopic content.

The extracting may include obtaining a parameter related to a transformation relationship between 3D spatial coordinates and 2D image coordinates based on a matching point between the object images.

generating a bounding box corresponding to the three-dimensional point cloud; and dividing the bounding box into a voxel grid of a predetermined size.

The bounding box may be generated for each frame.

The outputting may include orthogonally projecting the points of the 3D point cloud existing inside the voxel grid onto each surface of the bounding box, wherein each of the plurality of subframes includes the bounding box. It may include a plurality of patches consisting of each face of and a cluster of points projected on each face.

The outputting may include adjusting compression ratios of the plurality of subframes based on a pre-allocated importance factor, and differentially compressing the plurality of subframes according to the adjusted compression ratios.

The bounding box is configured in the form of a polyhedron, and the plurality of subframes may include: a first subframe corresponding to a front surface of the bounding box; a second frame corresponding to the rear and left and right sides of the bounding box; and a third frame corresponding to the upper and lower surfaces of the bounding box.

In this case, the compressing may include adjusting the first subframe to a first compression ratio and adjusting the second subframe to a second compression ratio, wherein the first compression ratio is smaller than the second compression ratio.

In addition, the compressing may include adjusting the third subframe to a third compression ratio greater than the second compression ratio.

The geometric structure image may include position information on points of the 3D point cloud, and the texture image may include color information on points of the 3D point cloud.

The method may further include a step of receiving the output bitstream and performing decoding and rendering on the 2D frame to generate 3D stereoscopic content.

According to at least one embodiment of the present invention, a large-capacity three-dimensional point cloud is decomposed into a two-dimensional frame, but by differentially compressing the two-dimensional frame in consideration of user quality of experience (QoE) according to a view, Not only can the bandwidth of transmitted data be significantly reduced, but the quality of rendered 3D stereoscopic content can also be maintained at a certain level. Accordingly, the user can be provided with an efficient streaming service in a mobile communication network with an appropriate level of data communication quality.

Effects obtainable in this embodiment are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention pertains from the description below. .

1 is a schematic block diagram of a system for providing 3D stereoscopic content according to an embodiment of the present invention.
FIG. 2 is a view for explaining the multi-viewpoint image acquisition apparatus shown in FIG. 1 .
3 is a schematic configuration diagram of a server according to an embodiment of the present invention.
4 shows a 3D point cloud according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a bounding box and a voxel grid according to an embodiment of the present invention.
6 shows a configuration example of an encoder according to an embodiment of the present invention.
7 is an exemplary diagram for explaining an operation of projecting a 3D point cloud on each side of a bounding box by an encoder according to an embodiment of the present invention.
8 is an exemplary diagram for explaining an operation of decomposing the patch shown in FIG. 7 into a plurality of subframes.
9 is an exemplary diagram for explaining an operation of differentially compressing a plurality of subframes by an encoder according to an embodiment of the present invention.
10 illustrates 2D frames representing a 3D point cloud according to an embodiment of the present invention.
11 is a schematic configuration diagram of a client according to an embodiment of the present invention.
12 is a schematic flowchart of a method of providing 3D stereoscopic content according to an embodiment of the present invention.
13 is a flowchart for explaining in more detail a procedure for performing step S40 shown in FIG. 12 .

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings. Since the embodiment may have various changes and may have various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the embodiment to the specific disclosed form, and it should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the embodiment.

Terms such as “first” and “second” may be used to describe various elements, but these elements should not be limited by the terms. These terms are used for the purpose of distinguishing one component from another. In addition, terms specifically defined in consideration of the configuration and operation of the embodiment are only for describing the embodiment, and do not limit the scope of the embodiment.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary may be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, it is interpreted in an ideal or excessively formal meaning. doesn't happen

1 is a schematic block diagram of a system for providing 3D stereoscopic content according to an embodiment of the present invention.

As shown in FIG. 1 , the system may include a multi-viewpoint image acquisition apparatus 100 , a server 200 , and a client 300 .

The multi-viewpoint image acquisition apparatus 100 acquires two-dimensional multiview images (hereinafter, referred to as “object images”) of an object through a plurality of cameras. For example, the multi-viewpoint image acquisition apparatus 100 performs a synchronization operation between a plurality of cameras, acquires multiple images by simultaneously photographing objects in different directions, and then processes the multiple images in real-time frame by frame to generate the object images. can be obtained

The server 200 extracts a three-dimensional point cloud (3D Point Cloud) based on the object images received from the multi-viewpoint image acquisition device 100, and uses the 3D point cloud as a two-dimensional frame (2D) representing geometrical structure and texture properties. frame) to output an encoded bitstream.

The client 300 receives the encoded bitstream from the server 200 and performs decoding and rendering processes on the 2D frame to generate 3D volumetric contents, and reproduce it. .

The client 300 is a terminal registered and subscribed to a specific mobile communication network service, and may communicate with the server 200 through a network NW operated by a communication network operator. Such a network (NW) is, for example, Wi-Fi (Wi-Fi), Bluetooth (Bluetooth), cellular (cellular), LTE (Long-Term Evolution), LTE-A (LTE Advanced), 5G (5-Generation), WiMAX (Wimax), or some other type of wireless network.

In addition, the client 300 is, for example, a smartphone, a notebook computer, a digital broadcasting terminal, personal digital assistants (PDA), a portable multimedia player (PMP), a navigation system, a slate PC, a tablet PC, an ultrabook, a wearable device (eg, a mobile terminal such as a smart watch, smart glasses, etc.), a head mounted display (HMD), and/or a fixed terminal such as a digital TV, a desktop computer, and a digital signage.

FIG. 2 is a view for explaining the multi-viewpoint image acquisition apparatus shown in FIG. 1 .

Referring to FIG. 2 , the multi-viewpoint image acquisition apparatus 100 includes a plurality of cameras 110 arranged at regular intervals in a chroma keyed studio space.

For example, the plurality of cameras 110 are arranged at regular intervals along the circumferential direction to face the object 120 , and the multi-viewpoint image acquisition apparatus 100 determines the output viewpoints between the plurality of cameras 110 . Calibration may be performed in advance to be synchronized. However, the arrangement between the cameras 110 is not necessarily limited to the illustrated form, and includes various forms depending on the dynamic characteristics of the object 110 - for example, parallel or divergent arrangement forms. It may be changed to ham-.

After performing calibration between a plurality of cameras 110, the multi-viewpoint image acquisition device 100 collects multiple images by simultaneously photographing the object 120 in all directions, and processes multiple images for each frame in real time. Two-dimensional object images may be obtained.

Also, the multi-viewpoint image acquisition apparatus 100 may extract 3D topographical coordinates and colors of the object 120 based on the acquired 2D object images and transmit the extracted 3D topographical coordinates and colors to the server 200 . In this case, each of the plurality of cameras 110 may be configured as a camera having characteristics of HD level and 24-bit color or higher, but the scope of the present invention is not limited thereto.

Hereinafter, the server 200 that extracts a 3D point cloud based on object images and compresses and decomposes it into a 2D frame will be described in detail with reference to FIG. 3 .

3 is a schematic configuration diagram of a server according to an embodiment of the present invention.

Referring to FIG. 3 , the server 200 may include a point cloud extractor 210 , a voxel grid generator 220 , and an encoder 230 . However, this is an example, and the server 200 may additionally include a memory for storing commands and data in addition to the above-described components, and a network interface that helps communication through a network.

The point cloud extraction unit 210 extracts a 3D point cloud based on object images input from the multi-viewpoint image acquisition apparatus 100 . For example, the point cloud extractor 210 obtains a parameter related to a transformation relationship between 3D spatial coordinates and 2D image coordinates based on a matching point between object images, and applies the parameter to the object 120 based on the parameter. A corresponding 3D point cloud can be extracted. Here, the 3D point cloud is defined as digitized data that visually defines the object 120 in a three-dimensional space, and may be exemplified in more detail in FIG. 4 .

4 shows a 3D point cloud according to an embodiment of the present invention.

Referring to FIG. 4 , the 3D point cloud 40 is composed of a plurality of points 42 , and each point 42 may include one or more properties. The one or more attributes may include geometry, such as the topographical location of each point 42 , and color, brightness, motion, material, reflection, intensity, and the like. Properties other than geometry may be referred to as texture, which represents various aspects and properties associated with each point 42 of the 3D point cloud 40 .

That is, the 3D point cloud 40 is defined as a set of data points defined by a coordinate system, and each point 42 represents the outer surface of the object. For example, in a Cartesian coordinate system, each point 42 of a 3D point cloud 40 is identified by three coordinates X, Y, and Z, and the color of one point is red (R), green (G). ), and a combination of blue (B).

Such a 3D point cloud 40 requires significant bandwidth for transmission due to a large number of points 42 constituting it and properties related thereto. Therefore, in order to efficiently transmit the 3D point cloud 40 from the server 200 to the client 300, decomposition and compression into 2D frames are required, and for projecting the 3D point cloud 40 onto the 2D frame The creation of a bounding box is preceded.

The voxel grid generator 220 generates a bounding box for decomposing the 3D point cloud 40 into a 2D frame, and divides the bounding box into voxel grids of a predetermined size. This will be described below with reference to FIG. 5 .

5 is an exemplary diagram illustrating a bounding box and a voxel grid according to an embodiment of the present invention.

Referring to FIG. 5A , the voxel grid generator 220 generates a bounding box 50 corresponding to the 3D point cloud 40 ( FIG. 4 ). The bounding box 50 may be configured in the form of a cube that completely surrounds the periphery of the 3D point cloud 40 . The bounding box 50 is formed in a three-dimensional structure such as a hexahedron to prevent distortion or noise occurring at the boundary when the 3D point cloud 40 is projected on a 2D plane. However, the scope of the present invention is not necessarily limited thereto, and the shape of the bounding box 50 may be extended and applied to a polyhedron such as an octahedron or an icosahedron according to an embodiment.

The bounding box 50 is a front, a back, a right, a left, a top, and a bottom according to the position of the 3D point cloud 40 accommodated therein. bottom), and in this specification, each face is referred to as a 'subframe' as a kind of 2D plane on which the three-dimensional point cloud 31 is projected. do it with

Referring to FIG. 3B , the voxel grid generator 220 voxelizes the bounding box 50 to divide it into a plurality of voxels 52 , and a voxel grid of a predetermined size; 54) is created.

The voxel 52 means a pixel having a volume, and the size of each voxel 52 may be preset by a user in the system. As the size of the voxel 52 decreases, the rendering result becomes more accurate, but the memory for texture increases, and each point 42 of the 3D point cloud 40 belongs to one voxel 52 .

The encoder 230 projects the 3D point cloud 40 on each side of the bounding box 50 to decompose it into a plurality of subframes, and after differentially compressing the plurality of subframes, 2 representing the geometrical and texture properties It is possible to output an encoded bitstream by converting it into a dimensional frame (2D frame). This will be described in more detail below with reference to FIG. 6 .

6 shows a configuration example of an encoder according to an embodiment of the present invention.

Referring to FIG. 6 , the encoder 230 may include a patch generating unit 231 , a differential compression unit 233 , a frame packing unit 235 , an encoding engine 237 , and a multiplexer 239 .

The patch generator 231 generates a plurality of patches 231a by projecting the 3D point cloud 40 onto each surface of the bounding box 50 . A plurality of patches 231a are generated by segmenting the points 42 of the 3D point cloud 40 . In particular, the 3D point cloud 40 is divided by clustering each point 42 based on the normal vector. The clustered points are projected into a 2D subframe in 3D space. Each projected cluster is defined as a patch 231a. The patches 231a are arranged and collected in individual subframes S1 to S6.

Also, the patch generator 231 generates metadata related to the plurality of generated patches 231a. The metadata is associated with auxiliary information such as the location of the patch 231a in each of the subframes S1 to S6, the size of the patch 231a, and the type of projection surface of the patch 231a.

The differential compression unit 233 differentially compresses the plurality of subframes 231b representing the 3D point cloud 40 according to a pre-allocated importance factor. Here, the importance factor represents the user's quality of experience (QoE) related to the characteristics of the 3D point cloud 40, and the importance value of the user's quality of experience (QoE) and the compression rate have an inversely proportional relationship.

The frame packing unit 235 packs a plurality of differentially compressed subframes 231b into one 2D frame, and decomposes patches included in the 2D frame into a geometric structure image 235a and a texture image 235b. create

The geometric structure image 235a includes geographic coordinate information for each point 42 of the 3D point cloud 40 based on the X, Y, and Z positions in the 3D space. Each pixel in the geometry image 235a has a corresponding pixel in the texture image 235b, and the texture image 235b represents the color of each corresponding point in the geometry image 235a.

The texture image 235b includes texture information for each point 42 of the 3D point cloud 40 . The texture information of one point may include color information consisting of a combination of a red (R) value, a green (G) value, and a blue (B) value. In addition, the additional texture information may include brightness, motion, material, reflection, intensity, and the like.

Also, the frame packing unit 235 generates an occupied area image 235c. The occupancy image refers to pixel positions in the geometry image 235a and the texture image 235b including valid points of the 3D point cloud 40 that are projected or mapped to the 2D frame. For example, the occupied area image 235c indicates whether each pixel on the geometric structure image 235a and the texture image 235b is a valid pixel (refer to FIG. 7B ). An effective pixel on the occupied area image 235c represents a pixel in which each point 42 of the 3D point cloud 40 is projected on the 2D frame.

Occupancy image 235c includes a plurality of occupancy images, where each occupancy image corresponds to one image, such as respective geometric structure image 235a and texture image 235b.

The geometric structure image 235a , the texture image 235b , and the occupied area image 235c generated by the frame packing unit 235 are encoded through the encoding engine 237 , respectively.

The encoding engine 237 recompresses the geometric structure image 235a, the texture image 235b, and the occupation area image 235c to generate an encoded bitstream. Here, the encoding engine 237 reduces the transmission bandwidth of the 3D point cloud 40 . For example, when the 3D point cloud 40 is processed to fit a 2D frame, the geometry image 235a, the texture image 235b, and recompressing the occupied area image 235c. Thereafter, the metadata, the encoded geometric structure image 235a, the texture image 235b, and the occupied area image 235c are multiplexed through the multiplexer 239 .

The multiplexer 239 generates a single bitstream by combining the metadata, the encoded geometric structure image 235a, the texture image 235b, and the occupied area image 235c, and outputs it to the client 300 .

Hereinafter, the patch generating unit 231 illustrated in FIG. 6 will be described in more detail with reference to FIGS. 7 to 8 .

7 is an exemplary diagram for explaining an operation of projecting a 3D point cloud on each side of a bounding box by an encoder according to an embodiment of the present invention.

The patch generator 231 classifies and processes the 3D point cloud 40 into 2D frames to increase coding efficiency.

Referring to FIGS. 7A to 7B together, the patch generator 231 determines a bounding box 50 corresponding to the 3D point cloud 40 for every frame. Then, the patch 231a is generated by projecting the points closest to each side of the generated bounding box 50 in the form of an orthogonal projection.

For example, each point of the 3D point cloud 40 is projected along one axis (eg, the Y axis) of the bounding box 50, and is mapped onto a plane (eg, the XY plane) perpendicular to the one axis (eg, the XY plane) ( mapping). In this case, the patch generator 231 may classify points including spatial correlation data adjacent to each other to be disposed in one patch 231a.

8 is an exemplary diagram for explaining an operation of decomposing the patch shown in FIG. 7 into a plurality of subframes.

Referring to FIGS. 8A to 8B together, the patch generating unit 231 combines each side of the bounding box 50 and the patches 231a projected on each side to form a plurality of subframes 231b. ) constitutes The plurality of two subframes 231b includes first to sixth subframes S1 to S6 corresponding to each side of the bounding box 50, and the number of each subframe S1 to S6 is the bounding box ( 50) corresponds to the number of each side.

The first subframe S1 is a first patch P1 mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the positive Y-axis onto the XZ plane (or the front of the bounding box 50). ) is included.

The second subframe S2 is a second patch P2 that is mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the negative Y axis onto the XZ plane (or the back side of the bounding box 50). ) is included.

The third subframe S3 is a third patch P3 that is mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the positive X axis onto the YZ plane (or the right side of the bounding box 50). ) is included.

The fourth subframe S4 is a fourth patch mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the negative X-axis onto the YZ plane (or the left side of the bounding box 50) ( P4).

The fifth subframe S5 is a fifth patch P5 mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the positive Z axis onto the XY plane (or the upper surface of the bounding box 50). ) is included.

The sixth subframe (S6) is a sixth patch (P6) mapped by orthogonally projecting the points 42 of the 3D point cloud 40 along the negative Z axis on the XY plane (or the bottom surface of the bounding box 50) ) is included.

Hereinafter, an operation of differentially compressing the plurality of subframes 231b according to a pre-allocated importance factor will be described with reference to FIG. 9 .

9 is an exemplary diagram for explaining an operation of differentially compressing a plurality of subframes by an encoder according to an embodiment of the present invention.

Referring to FIG. 9 , the differential compression unit 233 adjusts compression rates of a plurality of subframes S1 to S6 based on a pre-allocated importance factor, and each subframe S1 to S6 according to the adjusted compression ratio. can be differentially compressed. Here, the importance factor represents a user quality of experience (QoE) related to the characteristics of the 3D point cloud 40 .

The plurality of subframes 231b may have different user quality of experience (QoE) required for each view according to the characteristics of the 3D point cloud 40 to be projected.

For example, assuming that the 3D point cloud 40 is a dynamic subject related to a human, an animal, a plant, or a character, the front image of the 3D point cloud 40 is different from images of different views. In comparison, the importance value of the user's quality of experience (QoE) may be assigned relatively higher. The reason is that the front image of the 3D point cloud 40 includes a face shape, body motion, and appearance having unique characteristics for each subject of the same category.

On the other hand, the top and/or bottom image of the 3D point cloud 40 has a relatively lower importance value of the user's quality of experience (QoE) compared to images of other views. have. The reason is that the crown or sole included in the top and/or bottom image of the 3D point cloud 40 has a high degree of similarity between subjects of the same category, and thus there is a limit in deriving distinctive features.

Therefore, the differential compression unit 233 considers the communication quality of the network (NW) to ensure an appropriate level of user quality of experience (QoE) for each view between the plurality of subframes 231b Table 1 below. Each compression ratio can be differentially adjusted as shown.

Referring to Table 1, the differential compression unit 233 adjusts the first subframe S1 corresponding to the front of the bounding box 50 to the first compression ratio, and the back of the bounding box 50 ), the second to fourth subframes S2 to S4 corresponding to the right, and left may be adjusted to a second compression ratio. In addition, the fifth and sixth subframes S5 and S6 corresponding to the top and bottom of the bounding box 50 may be adjusted to a third compression ratio.

Here, the first to third compression ratios may be different from each other. For example, the first compression ratio may be smaller than the second compression ratio, and the third compression ratio may be set to be larger than the second compression ratio. That is, the importance value of the user's quality of experience (QoE) and the compression rate have an inversely proportional relationship.

As described above, when the plurality of subframes 231b are differentially compressed in consideration of the user quality of experience (QoE) for each view, the bandwidth of data transmitted to the client 300 may be significantly reduced. In addition, since the quality of rendered 3D stereoscopic content also maintains a certain level, the user can be provided with an appropriate level of data communication quality.

10 illustrates 2D frames representing a 3D point cloud according to an embodiment of the present invention.

Referring to (a) of FIG. 10 , the geometric structure image 235a of the 2D frame includes a plurality of patches obtained by decomposing each point 42 of the 3D point cloud 40 into a 2D image, and the plurality of patches are described above. It is encoded with a different compression rate for each view of each subframe S1 to S6. For example, the patch 1010 in which the face of the 3D point cloud 40 is depicted is compressed to a lower point density than the patch 1020 in which the crown is depicted.

In addition, the geometric structure image 235a includes geographic coordinate information (x, y, z) for each point 42 of the 3D point cloud 40 based on the X, Y, and Z positions in the three-dimensional space. include

Referring to FIG. 10B , the texture image 235b of the 2D frame includes a plurality of patches obtained by decomposing each point 42 of the 3D point cloud 40 into a 2D image, and the plurality of patches are described above. It is encoded with a different compression rate for each view of each subframe (S1 to S6). For example, the patch 1010 in which the face of the 3D point cloud 40 is depicted is compressed to a lower point density than the patch 1020 in which the crown is depicted.

Also, the texture image 235b includes texture information for each point 42 of the 3D point cloud 40 . The texture information of one point may include color information consisting of a combination of a red (R) value, a green (G) value, and a blue (B) value. In addition, the additional texture information may include brightness, motion, material, reflection, intensity, and the like.

10A to 10B together, each pixel in the geometric structure image 235a has a corresponding pixel in the texture image 235b, and the texture image 235b is the geometric structure image 235a. I express the color of each corresponding point within me.

Referring to FIG. 10C , pixel positions in the geometric structure image 235a and the texture image 235b including effective points of the 3D point cloud 40 are shown in the occupied area image 235c of the 2D frame. . For example, the occupied area image 235c expresses whether each pixel on the geometric structure image 235a and the texture image 235b is a valid pixel in the form of a binary map. An effective pixel on the occupied area image 235c represents a pixel in which each point 42 of the 3D point cloud 40 is projected on the 2D frame.

Occupancy image 235c includes a plurality of occupancy images, where each occupancy image corresponds to one image, such as respective geometric structure image 235a and texture image 235b.

11 is a schematic configuration diagram of a client according to an embodiment of the present invention.

Referring to FIG. 11 , the client 300 receives the bitstream from the encoder 230 and performs decoding and rendering processes on the 2D frame to generate 3D volumetric contents. 310 and a display unit 320 that reproduces the 3D stereoscopic content according to a user's command. However, this is an example, and the client 300 may omit at least one of the above-described components or may additionally include other components.

The decoder 310 includes a demultiplexer 311 , a decoding engine 313 , and a rendering unit 315 .

The demultiplexer 311 receives the encoded bitstream generated from the encoder 230 . The demultiplexer 311 demultiplexes the encoded bitstream into compressed bitstreams for the geometry, texture, occupation area, and metadata of the 3D point cloud 40 , respectively.

That is, the demultiplexer 311 separates various bitstreams of data from the encoded bitstream. For example, the demultiplexer 311 separates various bitstreams of data such as a geometric structure image 235a, a texture image 235b, an occupied area image 235c, and metadata.

The decoding engine 313 decodes the bitstream for the geometry and the bitstream for the texture to generate 2D frames for the geometry and texture. The decoding engine 313 decompresses the bitstream representing the geometry and texture to generate 2D frames related to the geometry image 235a and the texture image 235b corresponding to FIGS. 10 (a) to (b). create

Also, the decoding engine 313 identifies valid points in each 2D frame, such as the geometric structure image 235a and the texture image 235b, using the occupied area image 235c. The occupied area image 235c represents an effective pixel position in 2D frames for reconstructing the 3D point cloud 40 .

The rendering unit 315 generates 2D frames (decompressed geometry image 235a, texture image 235b, and occupied area image 235c) based on the data received from the demultiplexer 311 and the decoding engine 313 . including) to create 3D stereoscopic content.

The display 320 may output 3D stereoscopic content generated by the decoder 310 according to a user's command. The display unit 320 may be configured as a touch screen by forming a layer structure with a touch pad that receives a touch input by a user, and may output an interface related to system settings of the client 300 .

In addition, the display unit 320 is preferably formed of a transparent material for visibility, for example, a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic To be implemented as at least one of an organic light-emitting diode (OLED), a flexible display, a three-dimensional display (3D display), a transparent display, a head-up display (HUD), and a touch screen can

12 is a schematic flowchart of a method of providing 3D stereoscopic content according to an embodiment of the present invention.

Referring to FIG. 12 , the method obtains 2D object images captured by a plurality of cameras (S10), and extracts a 3D point cloud by matching the object images with each other (S20).

Thereafter, a bounding box corresponding to the 3D point cloud is generated, and the bounding box is divided into voxel grids of a predetermined size ( S30 ).

Then, a 3D point cloud is projected on each side of the bounding box, converted into a 2D frame representing geometrical structure and texture properties, and encoded to transmit a compressed bitstream (S40).

Next, decoding of the 2D frame is performed through decompression of the received bitstream, and 3D volumetric contents are generated by reconstructing and rendering (S50).

Thereafter, the 3D stereoscopic content generated in response to the user's command is reproduced (S60).

13 is a flowchart for explaining in more detail a procedure for performing step S40 shown in FIG. 12 .

Referring to FIG. 13 , the step S40 includes decomposing the 3D point cloud into a plurality of 2D subframes (S41); Differential compression of a plurality of subframes in consideration of user quality of experience (QoE) (S43); Packing a plurality of differentially compressed subframes into one 2D frame (S45); decomposing the 2D frame into a geometric structure image, a texture image, and an occupied area image (S47); and outputting a bitstream through encoding and multiplexing (S49).

Step S41 is a 3D point cloud 40 on each side of the bounding box 50, the step of projecting (projection) (S411); and dividing the projected points to generate a plurality of patches configured as a cluster ( 413 ). At this time, the 3D point cloud is divided by clustering each point based on the normal vector. Clustered points are projected into 2D subframes in 3D space to create patches, which are organized and collected in individual subframes.

In step S43, a plurality of subframes representing the 3D point cloud are differentially compressed according to a pre-allocated importance factor. Here, the importance factor represents the user's quality of experience (QoE) related to the characteristics of the 3D point cloud 40, and the importance value of the user's quality of experience (QoE) and the compression rate have an inversely proportional relationship.

In steps S45 and S47, a plurality of differentially compressed subframes are packed into one 2D frame, and patches included in the 2D frame are decomposed into a geometry image and a texture image. The geometric structure image includes geographic coordinate information for each point of the 3D point cloud based on the X, Y, and Z positions in the 3D space. The texture image includes texture information for each point of the 3D point cloud. The texture information of one point may include color information consisting of a combination of a red (R) value, a green (G) value, and a blue (B) value. In addition, the additional texture information may include brightness, motion, material, reflection, intensity, and the like. Each pixel in the geometry image has a corresponding pixel in the texture image, and the texture image represents the color of each corresponding point in the geometry image.

Also, in step S47, an occupied area image is generated. Occupancy images refer to pixel positions in geometry images and texture images that contain valid points in a 3D point cloud projected or mapped to a 2D frame. The occupied area image is used to identify and reconstruct effective pixels in the 2D frame in step S50.

In step S49, the geometric structure image, the texture image, and the occupied area image are each encoded through a predetermined codec, and combined with each other through multiplexing to generate a single bitstream. The generated bitstream is replaced with the input of step S50.

The method for providing 3D stereoscopic content according to the above-described embodiment may be produced as a program to be executed by a computer and stored in a computer-readable recording medium, and examples of the computer-readable recording medium include ROM, RAM , CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The computer-readable recording medium is distributed in a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above-described method can be easily inferred by programmers in the technical field to which the embodiment belongs.

Although only a few have been described as described above in relation to the embodiments, various other forms of implementation are possible. The technical contents of the above-described embodiments may be combined in various forms unless they are incompatible with each other, and may be implemented in a new embodiment through this.

On the other hand, the apparatus and method for providing 3D stereoscopic content according to the above-described embodiment may be utilized in an augmented reality (AR) service, a virtual reality (VR) service, and the like.

It is apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit and essential characteristics of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (23)

extracting a 3D point cloud based on 2D object images captured by a plurality of cameras; and
The three-dimensional point cloud is decomposed into a plurality of subframes, the plurality of subframes are differentially compressed with a compression ratio adjusted based on a pre-allocated importance factor, and then converted into a two-dimensional frame representing a geometric image and a texture image. A method of providing 3D stereoscopic content, comprising: outputting an encoded bitstream. According to claim 1,
The extracting step is
and obtaining a parameter related to a transformation relationship between 3D spatial coordinates and 2D image coordinates based on the matching points between the object images. According to claim 1,
generating a bounding box corresponding to the three-dimensional point cloud; and
and dividing the bounding box into a voxel grid of a predetermined size. 4. The method of claim 3,
Wherein the bounding box is generated for each frame, 3D stereoscopic content providing method. 4. The method of claim 3,
The output step is
orthogonally projecting the points of the 3D point cloud existing inside the voxel grid to each side of the bounding box;
Each of the plurality of subframes,
Each side of the bounding box and a plurality of patches composed of a cluster of points projected on each side, the 3D stereoscopic content providing method. 6. The method of claim 5,
The pre-allocated importance factor is a user experience quality related to the 3D point cloud characteristic. 7. The method of claim 6,
The bounding box is configured in the form of a polyhedron,
The plurality of subframes,
a first subframe corresponding to the front of the bounding box;
a second sub-frame corresponding to the rear and left and right sides of the bounding box; and
A method of providing three-dimensional stereoscopic content, including; a third sub-frame corresponding to an upper surface and a lower surface of the bounding box. 8. The method of claim 7,
The compressing step is
Adjusting the first subframe to a first compression ratio, and adjusting the second subframe to a second compression ratio,
The first compression ratio is smaller than the second compression ratio, 3D stereoscopic content providing method. 9. The method of claim 8,
The compressing step is
and adjusting the third subframe to a third compression ratio greater than the second compression ratio. According to claim 1,
The geometric image includes location information for points of the three-dimensional point cloud,
The method for providing three-dimensional stereoscopic content, wherein the texture image includes color information on points of the three-dimensional point cloud. According to claim 1,
The method further comprising: receiving the output bitstream and performing decoding and rendering processes on the 2D frame to generate 3D stereoscopic content; A computer-readable recording medium in which an application program for realizing the method for providing three-dimensional stereoscopic content according to any one of claims 1 to 11 is recorded through being executed by a processor. a point cloud extractor for extracting a three-dimensional point cloud based on two-dimensional object images captured by a plurality of cameras; and
The three-dimensional point cloud is decomposed into a plurality of subframes, the plurality of subframes are differentially compressed with a compression ratio adjusted based on a pre-allocated importance factor, and then converted into a two-dimensional frame representing a geometric image and a texture image. An encoder for outputting an encoded bitstream; including, an apparatus for providing 3D stereoscopic content. 14. The method of claim 13,
The point cloud extraction unit,
and obtaining a parameter related to a transformation relationship between three-dimensional spatial coordinates and two-dimensional image coordinates based on a matching point between the object images. 14. The method of claim 13,
and a voxel grid generator that generates a bounding box corresponding to the 3D point cloud and divides the bounding box into voxel grids of a predetermined size. 16. The method of claim 15,
The bounding box is generated for each frame, a 3D stereoscopic content providing apparatus. 16. The method of claim 15,
The encoder is
a patch generating unit generating a plurality of patches by orthogonally projecting points of the 3D point cloud existing inside the voxel grid onto each surface of the bounding box; and
A differential compression unit that adjusts the compression ratios of the plurality of subframes based on a pre-allocated importance factor, and differentially compresses the plurality of subframes according to the adjusted compression ratio;
Each of the plurality of subframes,
3D stereoscopic content providing apparatus comprising the plurality of patches consisting of each side of the bounding box and a cluster of points projected on each side. 18. The method of claim 17,
The pre-allocated importance factor is a three-dimensional stereoscopic content providing apparatus that is a user-sensible quality related to the three-dimensional point cloud characteristic. 19. The method of claim 18,
The bounding box is configured in the form of a polyhedron,
The plurality of subframes,
a first subframe corresponding to the front of the bounding box;
a second subframe corresponding to the rear and left and right sides of the bounding box; and
3D stereoscopic content providing apparatus including; a third sub-frame corresponding to the top and bottom surfaces of the bounding box. 20. The method of claim 19,
The differential compression unit,
Adjusting the first subframe to a first compression ratio, and adjusting the second subframe to a second compression ratio,
The first compression ratio is smaller than the second compression ratio, 3D stereoscopic content providing apparatus. 21. The method of claim 20,
The differential compression unit,
and adjusting the third subframe to a third compression ratio greater than the second compression ratio. 14. The method of claim 13,
The geometric image includes location information for points of the three-dimensional point cloud,
The texture image includes color information on points of the 3D point cloud. 14. The method of claim 13,
and a decoder for receiving the output bitstream and performing decoding and rendering on the 2D frame to generate 3D stereoscopic content.

Images