KR20220135939A - Transmission apparatus, receiving apparatus and method for providing 3d image by using point cloud data - Google Patents

Abstract

A transmission apparatus for providing a 3D image using point cloud data includes: a receiving part for receiving a depth image of an object from an image capturing device; a virtual object generation part for converting the depth image into point cloud data in a virtual space to generate a virtual object corresponding to the object; a viewpoint image generation part for generating a plurality of viewpoint images of the virtual object; a synthesized image generation part for generating a synthesized image obtained by synthesizing the plurality of viewpoint images; and a transmission part for transmitting the synthesized image to a receiving apparatus. The synthesized image may be converted into a 3D image and output through a display of the receiving apparatus.

Description

Transmitting device, receiving device, and method for providing a 3D image using point cloud data

The present invention relates to a transmitting apparatus, a receiving apparatus, and a method for providing a 3D image using point cloud data.

Conventional two-dimensional telepresence (Telepresence) is a technology implemented to make objects located at a remote place feel as if they are in the same space. This two-dimensional telepresence technology is mainly used in video conferences and performances.

Referring to FIG. 1 , the images of each participant located in different areas are projected on a pseudo hologram stage through a projector, and the remote participants are demonstrated as if they were present in one space. At this time, the voices of each participant are output through the stage speakers, creating an effect such as talking to each other in one space.

This is based on the Pepper's Ghost hologram stage technology to make the telepresence image look like a hologram, which is a pseudo-hologram technology that reflects a two-dimensional image.

Meanwhile, most general stereoscopic image display apparatuses display a stereoscopic image using the principle of binocular disparity. Since the images reflected by the left and right eyes are different from each other, the viewer can obtain three-dimensionality and a sense of space through the perception of the parallax by the left and right eyes.

According to a method of separating the left and right images, there are a glasses method and a glasses-free method. However, in the case of the glasses method, since there is an inconvenience of using a physical device (eg, an HMD device, 3D glasses) for viewing a 3D stereoscopic image, the stereoscopic visualization technology goes beyond the glasses-type stereo display and goes beyond the glasses-type stereo display and is a non-glasses-type multi-view display method. research is being actively conducted.

Unlike a two-dimensional display (or glasses-type display) in which pixels emit the same light in all directions, the light field display, which is a glasses-free multi-viewpoint display method, differs from each other by varying the amount of light depending on the direction. It is a display that projects images of a viewpoint.

As such, the light field display is not suitable for projecting an existing two-dimensional image due to the specificity of the display, and a process of generating content for the light field is absolutely necessary.

There are two methods for creating content for the light field: a method using live-action shooting and a method using 3D rendering technology in a virtual space.

Acquiring content for light field through live-action shooting requires a complex device and a lot of space, and there is a high barrier to popularization due to the difficulty of synchronization and alignment mismatch between viewpoints of each camera.

Japanese Patent Publication No. 6512575 (Registered on April 19, 2019)

The present invention is to solve the problems of the prior art described above, by converting a composite image obtained by synthesizing a plurality of viewpoint images of a virtual object corresponding to a remote object into a 3D image and outputting it in a real space in a 3D face-to-face We want to provide an experience.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the transmitting device for providing a 3D image using point cloud data according to the first aspect of the present invention is a receiving unit for receiving a depth image of an object from an image capturing device. ; a virtual object generator converting the depth image into point cloud data in a virtual space to generate a virtual object corresponding to the object; a viewpoint image generator generating a plurality of viewpoint images of the virtual object; a synthesized image generator for generating a synthesized image obtained by synthesizing the plurality of viewpoint images; and a transmitter configured to transmit the synthesized image to a receiving device, wherein the synthesized image may be converted into the 3D image and output through a display of the receiving device.

A receiving apparatus for providing a 3D image using point cloud data according to a second aspect of the present invention includes: a receiving unit for receiving a composite image in which a plurality of viewpoint images of a virtual object corresponding to an object are synthesized from a transmitting apparatus; a conversion unit converting the synthesized image into the 3D image; and an output unit outputting the 3D image through a display of the receiving device, wherein the virtual object corresponding to the object may be generated by converting a depth image of the object into point cloud data in a virtual space.

According to a third aspect of the present invention, a method for providing a 3D image using point cloud data performed by a transmitting apparatus includes: receiving a depth image of an object from an image capturing apparatus; generating a virtual object corresponding to the object by converting the depth image into point cloud data in a virtual space; generating a plurality of viewpoint images of the virtual object; generating a synthesized image obtained by synthesizing the plurality of viewpoint images; and transmitting the synthesized image to a receiving device, wherein the synthesized image may be converted into the 3D image and output through a display of the receiving device.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention converts a 3D image obtained by synthesizing a plurality of viewpoint images of a virtual object corresponding to a remote object into a 3D image and outputs it in real space. Dimensional face-to-face service can be provided.

Through this, the user can have a conversation with the object in real time through a 3D image corresponding to the object output through the display, without the need for the user to wear a separate physical device (eg, HMD device, 3D glasses), and the object The user may be provided with the same experience as being in the same space.

1 is a view showing a holographic image using the conventional remote telepresence (Telepresence).
2 is a block diagram of a 3D image providing system according to an embodiment of the present invention.
3 is a block diagram of the transmitting apparatus shown in FIG. 2 according to an embodiment of the present invention.
4A to 4C are diagrams for explaining a method of generating a composite image according to an embodiment of the present invention.
5 is a flowchart illustrating a method of providing a 3D image through a transmitting apparatus according to an embodiment of the present invention.
6 is a block diagram of the receiving apparatus shown in FIG. 2 according to an embodiment of the present invention.
7A to 7D are diagrams for explaining a method of converting a synthesized image into a 3D image according to an embodiment of the present invention.
8 is a flowchart illustrating a method of providing a 3D image through a receiving device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, detailed contents for carrying out the present invention will be described with reference to the accompanying configuration diagram or process flow diagram.

2 is a block diagram of a 3D image providing system according to an embodiment of the present invention.

Referring to FIG. 2 , the 3D image providing system may include a transmitting device 100 and a receiving device 110 . However, since the 3D image providing system of FIG. 2 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 2 , and may be configured differently from FIG. 2 according to various embodiments of the present invention. have.

In general, each component of the 3D image providing system of FIG. 2 is connected through a network (not shown). A network refers to a connection structure that allows information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound communication, Visible Light Communication (VLC), LiFi, and the like, but are not limited thereto.

The transmitting apparatus 100 may receive a depth image of the object from an image capturing apparatus (not shown).

The transmitting apparatus 100 may convert a depth image of an object into point cloud data in a virtual space, and generate a virtual object corresponding to the object by using the point cloud data.

The transmitting apparatus 100 may generate a plurality of viewpoint images of a virtual object corresponding to the object. For example, the transmitting device 100 arranges a plurality of virtual cameras centering on a virtual object located in a virtual space, and based on the arrangement information of the plurality of virtual cameras, a plurality of virtual objects for the virtual object through each virtual camera. A viewpoint image can be created.

The transmitting apparatus 100 may generate a synthesized image obtained by synthesizing a plurality of viewpoint images of a virtual object, and may transmit the generated synthesized image to the receiving apparatus 110 .

The receiving device 110 may receive a composite image of an object located remotely from the transmitting device 100 .

The receiving device 110 may convert the synthesized image of the object into a 3D image, and then output the 3D image through the display of the receiving device 110 .

Hereinafter, the operation of each component of the 3D image providing system of FIG. 2 will be described in more detail.

3 is a block diagram of the transmitting apparatus 100 shown in FIG. 2 according to an embodiment of the present invention.

Referring to FIG. 3 , the transmitter 100 may include a receiver 300 , a virtual object generator 310 , a viewpoint image generator 320 , a synthesized image generator 330 , and a transmitter 340 . have. However, the transmitting apparatus 100 shown in FIG. 3 is only one implementation example of the present invention, and various modifications are possible based on the components shown in FIG. 3 .

Hereinafter, FIG. 3 will be described with reference to FIGS. 4A to 4C.

According to the present invention, an object located in another remote space can be expressed in three dimensions in the space where the user is currently located by using point cloud data and light field technology.

First, the receiver 300 may receive a depth image of an object from an image capturing apparatus (not shown).

Here, the depth image of the object may include distance data between an image capturing apparatus (not shown) and the object and color data including a plurality of pixel information on the object.

Here, the image capturing apparatus (not shown) may include, for example, an image sensor, an infrared emitter, and an infrared depth sensor. In addition, the image capturing apparatus (not shown) may further include a processing device and a memory for storing instructions. When the instructions are executed by the image capturing apparatus (not shown), the instructions may cause the processing apparatus to perform a plurality of operations. For example, the instructions may include determining image data based on visible light captured by the image sensor, determining distance data based on infrared light transmitted by an infrared emitter and captured by an infrared depth sensor. may include

The receiver 300 may further receive audio data of the object from an image photographing apparatus (not shown).

The virtual object generator 310 may generate a virtual object corresponding to the object by converting the depth image of the object into point cloud data in the virtual space.

Here, the point cloud data is a set of points in a three-dimensional space, and may represent one point in three dimensions as a value compared with a two-dimensional image. Such point cloud data is a set of a vector form that may include position coordinate information and color information for a point.

The virtual object generator 310 may extract distance data and color data from the depth image of the object.

The virtual object generator 310 may convert the depth image into point cloud data based on color data and distance data of the depth image.

For example, referring to FIG. 4A , the virtual object generator 310 converts a depth image of an object into three-dimensional space coordinates (x coordinate, y coordinate, and z coordinate) in a virtual space based on distance data with respect to the object. It can be mapped and converted into point cloud data corresponding to the object.

That is, point cloud data constituting a spatial configuration may be generated by collecting location data for innumerable points included in the object. As the density of the point cloud data increases, it becomes more and more specific data and has a meaning as a virtual object.

The virtual object generator 310 may convert distance data of the object into point cloud data using a meshing technique.

For example, the virtual object generator 310 may convert faces corresponding to polygons connecting points of the object into a set. Also, the virtual object generator 310 may generate a virtual object in a virtual space by applying a color texture to a set of faces corresponding to each polygon using coordinate information of each face and color data of the object.

When a depth image and a color image of an object are received in real time from an image capturing apparatus (not shown), the virtual object generator 310 may generate a virtual object in real time according to the received depth image and color image in real time. .

On the other hand, when the point cloud data is not used, the virtual object generating unit 310 uses the camera arrangement information of the N image photographing apparatuses (not illustrated) for photographing the object received from the N image photographing apparatuses (not illustrated). A virtual object corresponding to the object may be generated by synthesizing the depth image of the object in the form of a tile to be described later.

However, when camera arrangement information of N image photographing apparatuses (not shown) is used, the cost of creating a virtual object increases in proportion to the number of image photographing apparatuses (not illustrated), and the location of each image photographing apparatus (not illustrated) There is a technical problem of synchronizing the output of each imaging device (not shown) with a calibration issue for the . In addition, in the case of photographing by an image photographing apparatus (not shown), it is not easy to remove the background behind the object to be photographed to add another spatial background effect or to synthesize it with another 3D graphic object.

To solve this problem, in the present application, point cloud data is obtained from a depth image of an object received from an imaging device (not shown), and the object is generated as a virtual object existing in a virtual space.

In addition, since the point cloud data representing the 3D data occupies a significant amount of memory, a special compression method is required to transmit the point cloud data to the receiving device 110 . When the point cloud data is compressed using a general compression method, the original data is deformed due to quantization.

Accordingly, in the present invention, the point cloud data is used only to generate a virtual object corresponding to the object, and the point cloud data is not compressed and transmitted to the receiving device 110 .

Meanwhile, the viewpoint image generator 320 may generate a plurality of viewpoint images of the virtual object corresponding to the object.

For example, referring to FIG. 4B , the viewpoint image generating unit 320 may obtain arrangement information of a plurality of virtual cameras for photographing the virtual object 40 centered on the virtual object 40 located in the virtual space. have.

Also, the viewpoint image generator 320 may generate a plurality of viewpoints for the virtual object 40 through a plurality of virtual cameras corresponding to each of the plurality of viewpoints for the virtual object 40 based on the arrangement information of the plurality of virtual cameras. You can create an image. Here, the number of virtual cameras for capturing a plurality of viewpoint images may be determined based on the resolution of the 3D image to be output from the receiving device 110 and relationship information between the transmitting device 100 and the receiving device 110 . In addition, the number of virtual cameras for capturing a plurality of viewpoint images may be further determined based on information on the display of the receiving device 110 to which the 3D image is to be output. For example, the number of virtual cameras may be determined based on characteristic information of a lenticular Fresnel lens constituting the display of the receiving device 110 .

Meanwhile, a plurality of viewpoint images of the virtual object may be transmitted to the receiving device 110 through different video channels. 110), in constructing one 3D image frame (eg, a light field image frame), there is a problem in that each viewpoint image must be synchronized. In addition, when loss or delay of one viewpoint image occurs in the process of transmitting a plurality of viewpoint images, it is difficult for the receiving device 110 to generate a 3D image or a delay occurs in generation of the 3D image.

To solve this problem, the synthesized image generator 330 generates a synthesized image obtained by synthesizing a plurality of viewpoint images of a virtual object.

That is, in order to easily synchronize the output images (ie, a plurality of viewpoint images) of each virtual camera in the receiving device 110 , the synthesized image generator 330 converts the plurality of viewpoint images into a set of image frame columns in a tile format. A composite image can be created.

Here, the number and size of the tile shape may be determined by the size information of the display of the receiving device 110 and the number of virtual cameras.

For example, referring to FIG. 4C , the display resolution of the receiving device 110 is 8K, and the number of virtual cameras is 30 (in this case, 6 virtual cameras in the case of a horizontal view, and 5 in the case of a vertical view) virtual camera), the synthesized image generator 330 derives size information (7680 X 4320) of the display from the display resolution, and based on the horizontal information 7680 of the display and the number of horizontal viewpoints (6 virtual cameras) Thus, the horizontal resolution of the tile (1280=7680/6) can be calculated. Also, the synthesized image generator 330 may calculate the vertical resolution (864=4320/5) of the tile based on the vertical information 4320 of the display and the number of vertical viewpoints (five virtual cameras). For example, the horizontal resolution of each tile may be calculated through [Equation 1], and the vertical resolution of each tile may be calculated through [Equation 2].

[Equation 1]

Tile Horizontal Resolution (WR) = Display Horizontal (W) / Number of Horizontal Views (WN)

[Equation 2]

Tile's Vertical Resolution (HR) = Display Height (H) / Number of Vertical Views (HN)

The synthesized image generator 330 may generate a synthesized image obtained by synthesizing a plurality of viewpoint images in a tile format based on the size information of the display of the receiving device 110 and the number of virtual cameras. For example, referring to FIG. 4C , the synthesized image generator 330 captures 40 viewpoint images according to an arrangement of a plurality of virtual cameras based on the size information of the display of the receiving device 110 and the number of virtual cameras. It is possible to create a composite image synthesized in a 6*5 tile format.

The synthesized image generator 330 may encode the synthesized image using a preset image compression method.

The transmitter 340 may transmit the synthesized image to the receiver 110 .

In order for the receiving device 110 to easily generate the 3D image, the transmitter 340 corresponds to the total resolution (R) of the synthesized image and the number of total viewpoint images (N, the number of virtual cameras) used in the synthesized image. ), the number of horizontal views (WN), the number of vertical views (HN), the horizontal resolution (WR) of the tile, and the vertical resolution (HR) of the tile to the receiving device 110 . can be transmitted

The transmitter 340 may transmit the encoded composite image and the voice data of the object to the receiver 110 . In this case, the encoded composite image and the audio data of the object may be transmitted to the receiving device 110 according to a standard transmission standard.

In this case, the synthesized image may be converted into a 3D image and output through the display of the receiving device 110 .

Each viewpoint image included in the composite image may be converted into a 3D image and output based on the pixel arrangement of the display of the receiving device 110 . In this case, pixel data for each viewpoint image may be determined based on width information and tilt information of the optical lens included in the display of the receiving device 110 .

Meanwhile, for those skilled in the art, each of the receiver 300 , the virtual object generator 310 , the viewpoint image generator 320 , the synthesized image generator 330 , and the transmitter 340 may be implemented separately, or at least one of them It will be fully understood that this may be integrated and implemented.

5 is a flowchart illustrating a method of providing a 3D image through the transmitting apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 5 , in operation S501 , the transmitting apparatus 100 may receive a depth image of an object from an image capturing apparatus.

In operation S503, the transmitting apparatus 100 may generate a virtual object corresponding to the object by converting the depth image into point cloud data in a virtual space.

In operation S505, the transmitting apparatus 100 may generate a plurality of viewpoint images of the virtual object.

In operation S507, the transmitting apparatus 100 may generate a synthesized image obtained by synthesizing a plurality of viewpoint images.

In step S509 , the transmitting device 100 may transmit the synthesized image to the receiving device 110 . Here, the synthesized image transmitted to the receiving device 110 may be converted into a 3D image and output through the display of the receiving device 110 .

In the above description, steps S501 to S509 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

6 is a block diagram of the receiving device 110 shown in FIG. 2 according to an embodiment of the present invention.

Referring to FIG. 6 , the receiving device 110 may include a receiving unit 600 , a converting unit 610 , and an output unit 620 . However, the receiving device 110 shown in FIG. 6 is only one embodiment of the present invention, and various modifications are possible based on the components shown in FIG. 6 .

Hereinafter, FIG. 6 will be described with reference to 7a to 7d.

The receiving device 110 may output the synthesized image received from the transmitting device 100 to form a 3D image in space through an optical lens (eg, a lenticular Fresnel lens).

In detail, the receiver 600 may receive, from the transmitter 100 , a composite image in which a plurality of viewpoint images of a virtual object corresponding to the object are synthesized and audio data of the object. Here, the virtual object corresponding to the object is an object generated by converting a depth image of the object into point cloud data in a virtual space. Here, the plurality of viewpoint images are images generated through a plurality of virtual cameras arranged around a virtual object located in a virtual space.

The receiver 600 receives the total resolution of the synthesized image from the transmitting device 100 , the number of total viewpoint images used in the synthesized image (corresponding to the number of virtual cameras), the number of horizontal viewpoints, the number of vertical viewpoints, and the number of horizontal views of the tile. Information on the composite image including the resolution and the vertical resolution of the tile may be further received.

The synthesized image received from the transmitting apparatus 100 may be an image in which a plurality of viewpoint images are synthesized in a tile format based on the size information of the display of the receiving apparatus 110 and the number of virtual cameras. Here, the number of the plurality of virtual cameras may be determined based on the resolution of the 3D image to be output from the receiving device 110 and relationship information between the transmitting device 100 and the receiving device 110 . The number of the plurality of virtual cameras may be further determined based on information on the display of the receiving device 110 to which the 3D image is to be output.

After decoding the encoded composite image, the converter 610 may convert the decoded composite image into a 3D image.

The converter 610 may convert the decoded synthesized image into a tile-shaped view image. In this case, each view image in the form of a tile may be configured in the form shown in FIG. 4C, for example.

The converter 610 may convert the synthesized image into a 3D image based on information on the synthesized image received from the transmitting apparatus 100 .

The converter 610 may convert each viewpoint image included in the synthesized image into a 3D image based on the pixel arrangement of the display of the receiving device 110 .

For example, the conversion unit 610 converts each viewpoint image in the form of a tile using a unique synthesis algorithm according to the characteristics of an optical lens (eg, lenticular Fresnel lens) included in the display of the receiving device 110 to the display of the display. It can be composited into an array of pixels. In this case, the pixel arrangement criterion may be different depending on the characteristics of the optical lens.

The conversion unit 610 provides information on the composite image received from the transmitting device 100 (full resolution, the number of total viewpoint images used in the synthesized image, the number of horizontal viewpoints, the number of vertical viewpoints, the horizontal resolution of the tile, and Each view image may be segmented based on the vertical resolution of the tile). For example, when the total number of viewpoint images is a viewpoint image composed of 30 viewpoints, an index for each viewpoint image is 1 to 30, and pixel data of each viewpoint image is displayed on the display of the receiving device 110 as shown in FIG. 7A . It may be determined based on width information and tilt information of the optical lens included in the .

FIG. 7A is a diagram illustrating a pixel layout for synthesizing a 3D image (light field image) based on the display resolution of the receiving device 110 using each view image in the form of a tile having a total number of views of 30. Referring to FIG.

Referring to FIG. 7A , each pixel number indicates an index for a viewpoint image for each viewpoint in the form of a tile. Viewpoint images may be arranged according to inclination information and width information of a prism of an optical lens constituting the display of the receiving device 110 . Here, a group having the same color is a group having the same pixel data (coordinate information) of pixels in each view image. Since the calculation on whether pixel data belongs to the same group requires a large amount of calculation, it may be processed in parallel in the GPU module of the receiving device 110 .

7B is a view for analyzing a position of a reconstructed composite image along a vertical line of a display panel in a display of the receiving device 110 using an optical lens, and the inclination of the optical lens arranged to be deflected based on the number of 7 total viewpoints and a view showing the display according to the width.

Referring back to FIG. 6 , the output unit 620 may output a 3D image through the display of the receiving device 110 . Also, the output unit 620 may output a 3D image and audio data of an object together.

For example, referring to FIG. 7C , the output unit 620 outputs a 3D image having N viewpoint images by allowing the N viewpoint images to converge in space using the prism principle of an optical lens included in the display. can do.

FIG. 7D shows a 3D image obtained by outputting the synthesized image in a tile format for each viewpoint of FIG. 4C using the method of FIG. 7A .

Meanwhile, those skilled in the art will fully understand that the receiving unit 600 , the converting unit 610 , and the output unit 620 may be implemented separately, or one or more of them may be integrated.

8 is a flowchart illustrating a method of providing a 3D image through the receiving device 110 according to an embodiment of the present invention.

Referring to FIG. 8 , in step S801 , the receiving device 110 may receive a composite image in which a plurality of viewpoint images of a virtual object corresponding to an object are synthesized from the transmitting device 100 . Here, the virtual object corresponding to the object is an object generated by converting a depth image of the object into point cloud data in a virtual space.

In step S803, the receiving device 110 may convert the synthesized image into a 3D image.

In step S805 , the receiving device 110 may output a 3D image through the display of the receiving device 110 .

In the above description, steps S801 to S805 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The foregoing description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: sending device
110: receiving device
300: receiver
310: virtual object generator
320: viewpoint image generating unit
330: synthesized image generating unit
340: transmission unit
600: receiver
610: conversion unit
620: output unit

Claims (18)

In the transmitting device for providing a three-dimensional image using point cloud (Point Cloud) data,
a receiver configured to receive a depth image of an object from an image capturing apparatus;
a virtual object generator converting the depth image into point cloud data in a virtual space to generate a virtual object corresponding to the object;
a viewpoint image generator generating a plurality of viewpoint images of the virtual object;
a synthesized image generator for generating a synthesized image obtained by synthesizing the plurality of viewpoint images; and
and a transmitter for transmitting the synthesized image to a receiving device,
The synthesized image is converted into the 3D image through a display of the receiving device and outputted.
The method of claim 1,
and the virtual object generator converts the depth image into the point cloud data based on color data and distance data of the depth image.
The method of claim 1,
and the viewpoint image generator generates the plurality of viewpoint images through a plurality of virtual cameras arranged around the virtual object located in the virtual space.
4. The method of claim 3,
The transmitting device, wherein the number of the virtual cameras is determined based on a resolution of the 3D image to be output from the receiving device and relationship information between the transmitting device and the receiving device.
5. The method of claim 4,
The number of the virtual cameras will be determined further based on information on a display of the receiving device to which the 3D image is to be output.
6. The method of claim 5,
The synthesized image generating unit generates the synthesized image in a tile format from the plurality of viewpoint images based on the size information of the display and the number of the virtual cameras.
The method of claim 1,
The receiving unit further receives the voice data of the object,
The synthesized image generator encodes the synthesized image using a preset image compression method,
The transmitter transmits the encoded composite image and the audio data of the object to the receiver.
7. The method of claim 6,
Each viewpoint image included in the synthesized image is converted into the 3D image based on the pixel arrangement of the display.
9. The method of claim 8,
The pixel data for each viewpoint image will be determined based on the width information and the inclination information of the optical lens included in the display, the transmitting device.
In the receiving device for providing a three-dimensional image using point cloud (Point Cloud) data,
a receiver configured to receive a composite image obtained by synthesizing a plurality of viewpoint images of a virtual object corresponding to an object from a transmitter;
a conversion unit converting the synthesized image into the 3D image; and
an output unit for outputting the 3D image through a display of the receiving device
including,
The virtual object corresponding to the object is generated by converting a depth image of the object into point cloud data in a virtual space.
11. The method of claim 10,
The plurality of viewpoint images are generated through a plurality of virtual cameras arranged around the virtual object located in a virtual space.
12. The method of claim 11,
The reception device, wherein the number of the virtual cameras is determined based on a resolution of the 3D image to be output from the reception device and relationship information between the transmission device and the reception device.
13. The method of claim 12,
The number of the virtual cameras will be determined further based on information on a display of the receiving device to which the 3D image is to be output.
14. The method of claim 13,
The receiving apparatus of claim 1, wherein the plurality of viewpoint images are generated as the composite image in a tile format based on the size information of the display and the number of the virtual cameras.
11. The method of claim 10,
The receiving unit further receives the voice data of the object from the transmitting device,
The output unit outputs the 3D image and audio data of the object.
11. The method of claim 10,
The receiving device of claim 1, wherein the converting unit converts each viewpoint image included in the synthesized image into the 3D image based on the pixel arrangement of the display.
17. The method of claim 16,
The pixel data for each viewpoint image will be determined based on the width information and the inclination information of the optical lens included in the display, the receiving device.
In the method of providing a three-dimensional image using point cloud data performed by a transmitting device,
receiving a depth image of the object from an image capturing apparatus;
generating a virtual object corresponding to the object by converting the depth image into point cloud data in a virtual space;
generating a plurality of viewpoint images of the virtual object;
generating a synthesized image obtained by synthesizing the plurality of viewpoint images; and
transmitting the synthesized image to a receiving device;
The synthesized image is converted into the three-dimensional image through a display of the receiving device and outputted.

Images