KR20200072321A - System and method for providing holoportation - Google Patents

Abstract

According to the embodiments of the present invention, provided is a system for providing a holoportation comprising: a first sensor unit which senses a first object; a first restoration server for generating first model information by restoring a 3D model of a first object through the sensing and transmitting the generated first model information; a first relay server which receives and converts the first model information and transmits the converted first model information; and a first terminal device which receives the converted first model information and displays the first object. Therefore, the system for providing the holoportation of the present invention can support various types of terminal devices.

Description

System and method for providing holographic information{SYSTEM AND METHOD FOR PROVIDING HOLOPORTATION}

The present invention relates to a system and method for providing a holoportion, and more particularly, to provide a holoportation service capable of supporting various types of terminal devices and enabling bidirectional transmission.

Holoportation refers to telepresence or virtual teleportation using AR (Augmented Reality)/VR (Virtual Reality). Here, telepresence or virtual teleportation is a technique that makes a person or object at a remote location feel as if they are in the same space, and is mainly used for teleconferencing.

Let's look at the prior art related to holoportion.

Korean Registered Patent No. 10-1659849 relates to a system for providing a telepresence using an avatar, and in a telepresence system using an avatar, human X exists in the first space, which is the current space, and human Y, in the second space, which is a remote location. In the initial state in which the avatar Y'corresponding to the human Y exists in the first space and the avatar X'corresponding to the human X exists in the second space, a change in the position of the human Y in the second space If is detected, (i) spatial information of the first space and the second space, (ii) the location information of the human X, the human Y, and the avatar X'in the first space and the second space And (iii) a reference point search unit for determining a point where the avatar Y'is located in the first space with reference to the gaze information of the human X and the human Y in the first space and the second space. And an avatar motion generating unit generating motion of the avatar Y'with reference to information on a location where the avatar Y'will be located.

However, the prior art does not propose a method capable of supporting various AR/VR devices in a holoport providing system or a specific configuration for providing bidirectional holoportion.

In order to solve the above-described problems, embodiments of the present invention have an object to provide a system and method for providing a holographic service in various types of terminal devices.

In addition, embodiments of the present invention has an object to provide a system for providing a bidirectional holographic.

One embodiment of the present invention for achieving the above object is a first sensor unit for sensing the first object; A first restoration server that generates first model information and transmits the generated first model information by restoring the three-dimensional model of the first object through the sensing; A first relay server that receives the first model information, converts it, and transmits the converted first model information; And a first terminal device that receives the converted first model information and displays a first object.

In one embodiment, the first relay server may include a converter that converts the format of the first model information or the quality, size, or type of a 3D model corresponding to the first model information.

In one embodiment, the converter may convert the format of the first model information according to the format of 3D model information supported by the first terminal device.

In one embodiment, the converter may convert the quality of the 3D model by adjusting the number of vertices of the 3D model corresponding to the first model information according to the performance of the first terminal device.

In one embodiment, the first terminal device measures depth information for a space in which the first terminal device is used, and the converter sets a position or distance at which the first object is to be displayed based on the depth information, and The size of the 3D model corresponding to the first model information can be converted.

In one embodiment, the converter may convert a 3D model corresponding to the first model information into a 3D point cloud or 3D mesh according to the type of 3D model supported by the first terminal device.

In one embodiment, the holoportation providing system includes a second sensor unit for sensing a second object; A second restoration server that restores a 3D model of the second object through the sensing, generates second model information, and transmits the generated second model information; A second relay server for receiving the second model information, converting it, and transmitting the converted second model information; And a second terminal device receiving the converted second model information and displaying a second object, the first terminal device displaying the first object to the second object, and the second terminal device providing the first object to the first object. Two objects can be displayed.

In one embodiment, the first restore server transmits the first model information to the first relay server via the second relay server, and the second restore server via the first relay server to the second The second model information can be transmitted to the relay server.

In addition, another embodiment of the present invention for achieving the above object is a sensing step of sensing the first object to generate two-dimensional image information or distance information; A restoration step of restoring a 3D model of the first object based on the 2D image information or distance information to generate first model information and transmitting the generated first model information; A relay step of receiving and converting the first model information and transmitting the converted first model information; And a display step of receiving the converted first model information and displaying the first object.

As described above, embodiments of the present invention, the first sensor unit for sensing the first object, through the sensing to restore the three-dimensional model of the first object to generate the first model information and transmit the generated first model information A first restoration server, a first relay server that receives and converts the first model information, and transmits the converted first model information, and a first terminal device that receives the converted first model information and displays the first object By providing a system for providing holoportation, including, a holoportment service can be provided in various types of terminal devices.

In addition, the embodiments of the present invention include a converter that converts the format of the first model information or the quality, size, or type of a 3D model corresponding to the first model information, thereby providing a holographic port in various types of terminal devices. Service may be provided.

In addition, the embodiments of the present invention can provide a holographic service in various types of terminal devices by converting the format of the first model information according to the format of the 3D model information supported by the first terminal device.

In addition, according to the embodiments of the present invention, the number of vertices of the 3D model corresponding to the first model information is adjusted according to the performance of the first terminal device to convert the quality of the 3D model, thereby allowing the holophor in various types of terminal devices Presentation service may be provided.

In addition, embodiments of the present invention by setting the location or distance to display the first object based on the depth information for the space in which the first terminal device is used, and converting the size of the 3D model corresponding to the first model information , Holographic services may be provided in various types of terminal devices.

In addition, embodiments of the present invention converts a 3D model corresponding to the first model information into a 3D point cloud or a 3D mesh according to the type of 3D model supported by the first terminal device, thereby various terminal devices. Holocaust service may be provided.

In addition, embodiments of the present invention includes a second sensor unit for sensing the second object; A second restoration server that restores a 3D model of the second object through the sensing, generates second model information, and transmits the generated second model information; A second relay server for receiving the second model information, converting it, and transmitting the converted second model information; And a second terminal device receiving the converted second model information and displaying a second object, the first terminal device displaying the first object to the second object, and the second terminal device providing the first object to the first object. By displaying the two objects, a bidirectional holographic service can be provided.

Further, in the embodiments of the present invention, the first restoration server transmits the first model information to the first relay server via the second relay server, and the second restore server to the second relay server via the first relay server. By transmitting the second model information, a private IP address can be assigned to other configurations except the first relay server and the second relay server, thereby reducing the cost by minimizing the number of public IP addresses required to build a network. It is not limited to the number of public IP addresses available, and the network inside the private network can be expanded.

It is not limited to the above-mentioned effects, and other effects that are not mentioned can be clearly understood by those skilled in the art from the following description.

FIG. 1 is a diagram exemplarily showing a system for providing a unidirectional holography according to an embodiment, and FIG. 2 is a diagram illustrating a system for providing a bidirectional holography.
3 is a diagram illustrating an exemplary restoration server according to an embodiment.
4 is a diagram illustrating a relay server according to an embodiment.
5 is a diagram illustrating a bidirectional holographic provision system according to another embodiment.
6 is a view showing an example of adjusting the size of an object in a holographic providing system according to an embodiment.
7 is a view exemplarily showing an example of data flow in a holoportation providing system according to an embodiment.
8 is a flowchart illustrating a method for providing holography according to an embodiment of the present invention.

Since the description of the present invention is merely embodiments for structural or functional description, the scope of the present invention should not be interpreted as being limited by the embodiments described in the text. That is, since the present embodiments can be variously modified and have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing technical ideas. In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such an effect, and the scope of the present invention should not be understood as being limited thereby.

In addition, the accompanying drawings are provided to help understanding of the present invention, and provide embodiments with detailed description. However, the technical features of the present invention are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to form a new embodiment.

The holoportion providing system disclosed in the following embodiments will be described in more detail with reference to each drawing.

FIG. 1 is a diagram exemplarily showing a system for providing a unidirectional holography according to an embodiment, and FIG. 2 is a diagram illustrating a system for providing a bidirectional holography.

Here, holoportation refers to a technology that makes a person or object at a remote location feel as if they are in the same space as himself using AR (Augmented Reality)/VR (Virtual Reality). can do.

Referring to FIG. 1, a unidirectional holoportation providing system 10 according to an embodiment includes a sensor unit 100, a restoration server 200, a relay server 300, and a terminal device 400.

Referring to FIG. 2, the bidirectional holographic presentation system 20 according to an embodiment includes first/second sensor units 100a and 100b, first/second restoration servers 200a and 200b, and first/ It comprises a second relay server (300a, 300b), the first / second terminal device (400a, 400b).

Unlike FIG. 1, FIG. 2 has two sensor units 100, a restoration server 200, a relay server 300, and a terminal device 400 in both directions (100a/) in order to provide a bidirectional holographic service. 100b, 200a/200b, 300a/300b, 400a/400b).

A brief look at the unidirectional and bidirectional holographic provision systems 10 and 20.

In the unidirectional holoportation providing system 10 of FIG. 1, the first object 500a is sensed by the sensor unit 100 and is processed through the restoration server 200, the relay server 300, and the terminal device 400. It can be displayed to the two objects 500b. Therefore, it can be recognized that only the second object 500b is located in the same space as the first object 500a through the terminal device 400.

Here, the terminal device 400 is a Hololens (Hololens), HMD (Head Mounted Display), smart watches, notebooks, mobile phones, smart phones (2G/3G/4G/LET, smart phone), PMP (Portable Media Player) , PDA (Personal Digital Assistant) and a tablet PC (Tablet PC). Also, here, the first object 500a and the second object 500b may correspond to a person. Meanwhile, in FIG. 1 and FIG. 2, the communication network may refer to a wide concept network including a wired or wireless communication network.

However, in this case, since the interaction between the first object 500a and the second object 500b is impossible, both the first object 500a and the second object 500b can simultaneously interact using the holographic service. There is a need for a bidirectional holographic presentation system 20 of FIG.

In FIG. 2, (i) the first object 500a is sensed by the first sensor unit 100a, and then through the first restoration server 200a, the first relay server 300a, and the first terminal device 400a. It may be displayed on the second object 500b. Accordingly, it can be recognized that the second object 500b is located in the same space as the first object 500a through the first terminal device 400a.

In addition, in the same way (ii) the second object 500b is also sensed by the second sensor unit 100b, and the second restoration server 200b, the second relay server 300b, and the second terminal device 400b are sensed. Through this, it may be displayed to the first object 500a. Accordingly, it can be recognized that the second object 500b is located in the same space as the first object 500a through the second terminal device 400b.

Each configuration of FIGS. 1 and 2 will be described. At this time, the first sensor unit 100a and the second sensor unit 100b of FIG. 2, the first restoration server 200a and the second restoration server 200b, the first relay server 300a and the second relay server (300b), the first terminal device 400a and the second terminal device 400b are configured with the sensor unit 100, the restoration server 200, the relay server 300, and the terminal device 400 of FIG. 1, respectively. Since the action and effect are similar, the focus will be on the configuration of FIG. 1.

The sensor unit 100 may be composed of a plurality of cameras, a binocular camera, a sensor such as a laser scanner, and may generate 2D image information or distance information by sensing the first object 500a or the second object 500b. have.

With respect to the restoration server 200, look at Figure 3.

3 is a diagram illustrating an exemplary restoration server according to an embodiment.

Referring to FIG. 3, the restoration server 200 according to an embodiment includes a restoration unit 210, a compression unit 220, an encryption unit 230, a first communication unit 240, a first database 250, and It comprises a control unit 260.

The restoration unit 210 restores the three-dimensional model of the first object 500a or the second object 500b based on the image information or distance information generated through sensing, so that the first model for the first object 500a is restored. Information or second model information for the second object 500b may be generated. The first model information or the second model information may correspond to a 3D point cloud or mesh information. Hereinafter, the first model information or the second model information is mixed with the 3D model information.

The compression unit 220 may compress the first model information or the second model information to quickly transmit data through a network. 3D model information such as first model information or second model information may include geometric information, color information, normal information, or connection information of a vertex of a 3D model. As the 3D model becomes more sophisticated, 3D model information The size can grow exponentially. Accordingly, the compression unit 220 may compress and transmit the 3D model, thereby reducing network usage and rapidly transmitting 3D model information.

The encryption unit 230 may encrypt the first model information, the second model information, the compressed first model information, or the compressed second model information to prevent forgery or tampering. At this time, the encryption unit 230 may perform an encryption process on the payload excluding the header from the compressed first model information or the compressed second model information using the AES (Advanced Encryption Standard) encryption method. The encryption unit 230 may transmit the key generated in the encryption process to the relay server 300.

Also, the encryption unit 230 may insert a watermark into the first model information, the second model information, the compressed first model information, or the compressed second model information.

The first communication unit 240 may receive image information or distance information generated through sensing from the sensor unit 100, and may include first model information or second model information or compression unit 220 generated by the restoration unit 210. ) Alternatively, the first model information or the second model information compressed or encrypted by the encryption unit 230 may be transmitted to the relay server 300.

Looking at the transmission of the first communication unit 240, the first communication unit 240 of the restoration server 200, as shown in Figure 1, the first model information is connected to the terminal device 400 to display the first model information It can be transmitted directly to the relay server 300.

Therefore, unlike FIG. 2, the first communication unit 240 of the first restoration server 200a directly transmits the first model information to the first relay server 300a connected to the first terminal device to which the first model information is displayed. The first communication unit 240 of the second restoration server 200b may directly transmit the second model information to the second relay server 300b connected to the second terminal device on which the second model information is to be displayed.

However, as shown in FIG. 2, the first communication unit 240 of the first restoration server 200a transmits the first model information to the second relay server 300b and the first communication unit 240 of the second restoration server 200b. ) May transmit the second model information to the first relay server 300a.

That is, the first restoration server 200a can transmit the first model information to the first relay server 300a via the second relay server 300b, and the second restoration server 200b provides the second model information. It may be transmitted to the second relay server 300b via the first relay server 300a. In this regard, the connection management unit 310 of the relay server 300 looks in detail.

The first database 250 may include image information or distance information generated through sensing, first model information or second model information generated by the restoration unit 210, or compressed or encrypted first model information or second model information. Can be saved.

The first control unit 260 controls the overall operation of the restoration server 200, the restoration unit 210, the compression unit 220, the encryption unit 230, the first communication unit 240 and the first database 250 You can control the control flow or data flow between them.

With respect to the relay server 300, look at Figure 4.

4 is a diagram illustrating a relay server according to an embodiment.

Referring to FIG. 4, the relay server 300 according to an embodiment includes a connection management unit 310, a decryption unit 320, a decompression unit 330, a converter 340, a terminal device management unit 350, and rendering It includes a unit 360, a player unit 370, a second communication unit 380, a second database 390, and a second control unit 395.

The connection management unit 310 enables a communication connection between a local and a remote and manages the communication connection. Here, look at Figure 5 with respect to local and remote locations.

5 is a diagram illustrating a bidirectional holographic provision system according to another embodiment.

Referring to FIG. 5, in the bidirectional holographic presentation system 20 according to another embodiment, (i) the first sensor unit 100a, the first restoration server 200a, the second relay server 300b, and The second terminal device 400b is connected to a private network A, and (ii) the second sensor unit 100b, the second restoration server 200b, the first relay server 300a, and the first terminal device 400a. Is connected to another private network B, and (iii) the first relay server 300a and the second relay server 300b are connected to a public network.

The connection management unit 310 of the first/second relay servers 300a and 300b enables communication between a local and a remote, and manages a communication connection, where local is a second relay The inside of the private network A to which the server 300b belongs or the inside of the private network B to which the first relay server 300a belongs belongs to and the remote location may mean the outside of the private network based on each relay server 300.

For example, the connection management unit 310 of the second relay server 300b is between the first restoration server 200a existing locally and the first relay server 300a or the first terminal device 400a existing at a remote location. By enabling communication and managing the established communication connection, the 3D model information generated by the local restoration server 200 can be transmitted to the terminal device 400 existing at a remote location.

In addition, the connection management unit 310 of the second relay server 300b communicates between the second restore server 200b or the first relay server 300a existing in the remote location and the second terminal device 100b existing locally. By enabling this and managing the established communication connection, the 3D model information generated by the restoration server 200 existing at a remote location can be received by the terminal device 400 existing locally.

Meanwhile, as shown in FIG. 5, the first restoration server 200a may transmit the first model information to the first relay server 300a via the second relay server 300b, and the second restoration server 200b may 2 The model information may be transmitted to the second relay server 300b via the first relay server 300a.

Public IP addresses should be assigned to the first/second relay servers 300a and 300b connected to the public network through such a configuration, but the first/second restore servers 200a and 200b connected to the private network A or the private network B are the first. /As the second terminal device (400a, 400b) can be assigned a private IP address used only in private network A or private network B, it is possible to reduce the cost by minimizing the number of public IP addresses required to build the network, and to be available The network inside the private network can be expanded without being limited to the number of IP addresses.

In addition, the connection management unit 310 can manage the type and list of the local restoration server 200 or the device 400, and the terminal device 400 and the relay server 300 existing in a remote location to access them. ) Or the restoration server 200 may be notified of whether the local restoration server 200 or the terminal device 400 is in a connectable state or a state in which a function can be provided.

In addition, the connection management unit 310 has a specific communication connection between the local restoration server 200 or device 400 and the terminal device 400, relay server 300 or restoration server 200 existing in a remote location. After being set as a condition, it is possible to maintain a communication connection meeting a specific condition.

Here, specific conditions may include, for example, general network settings, format of 3D model information to be described later, quality, size or type of 3D model.

In addition, the connection management unit 310 may provide an IP switching function to improve the transmission speed of data passing through the relay server 300. Here, IP switching may refer to a routing technology capable of transmitting data faster than conventional routing by using OSI layer 3 switches.

In addition, the connection management unit 310 when a plurality of terminal devices 400 or relay servers 300 existing in a local or remote location requests 3D model information from the restoration server 200 existing in a local or remote location 3 Various network technologies for efficiently transmitting dimensional model information can be performed.

For example, the connection management unit 310 merges the three-dimensional model information requests of the plurality of terminal devices 400 or the relay server 300 and requests the three-dimensional model information to the restoration server 200 only once, then the restoration server The 3D model information received from the 200 may be transmitted to the plurality of terminal devices 400 or the relay server 300.

The decryption unit 320 may receive and decrypt the encrypted 3D model information and encryption key, and detect the inserted watermark to determine whether the received 3D model information is authentic.

The decompression unit 330 may receive the compressed 3D model information and decompress it.

The converter 340 may include a format conversion unit 342, a 3D model quality conversion unit 344, a 3D model size conversion unit 346, and a 3D model type conversion unit 348.

The converter 340 may convert the format of the first model information or the second model information or the quality, size, or type of the 3D model corresponding to the first model information or the second model information. Each configuration of the converter 340 will be described in detail.

The format conversion unit 342 may receive a format of 3D model information supported by each terminal device 400 from the terminal device management unit 350, which will be described later.

Also, the format conversion unit 342 receives the 3D model information from the connection management unit 310, the encryption/decryption unit 320, or the decompression unit 330, and receives the 3D model information. ) Can convert the format of the received 3D model information according to the format of the 3D model information supported.

However, when the format of the received 3D model information is supported by the terminal device 400 to receive the 3D model information, the format conversion unit 342 does not need to convert the format of the 3D model information.

Here, the format of the 3D model information may mean various formats such as OBJ, MTL, STL, PLY, 3DS, COLLADA, XML, VRML, X3D, FBX, U3D, and O3D representing 3D model information.

The 3D model quality conversion unit 344 may receive information on the performance of each terminal device 400 from the terminal device management unit 350 to be described later. Here, the performance of the terminal device 400 may be determined based on the number of vertices or faces of the 3D model that can be displayed in real time on the terminal device 400.

In addition, the 3D model quality conversion unit 344 receives the 3D model information from the connection management unit 310, the encryption decryption unit 320, the decompression unit 330, etc., the terminal device to receive the 3D model information According to the performance of 400, the number of vertices of the 3D model corresponding to the 3D model information may be adjusted to convert the quality of the 3D model.

Specifically, the 3D model quality converter 344 adjusts the quality of AR/VR displayed on the terminal device 400 by reducing the number of vertices of the 3D model in proportion to the performance of the terminal device 400. Alternatively, the 3D model quality conversion unit 344 sees the total number of vertices of the 3D model corresponding to the received 3D model information as 1, and the ratio of the 4 steps according to the performance of the terminal device 400 ( 4/5, 3/5, 2/5, 1/5) can reduce the number of vertices of the 3D model at an appropriate ratio.

Here, the 3D mesh is a type of a 3D model composed of vertices and faces of a 3D model, and geometric information about coordinates of vertices and a connection relationship between vertices constituting a face. It may be expressed as 3D model information including connectivity information corresponding to.

When the number of vertices of a 3D mesh decreases, the number of faces may decrease, so not only the geometric information but also the connection information decreases. The number of faces (connection information) is reduced to about twice the reduced number of vertices (geometric information). The quality conversion unit 344 effectively converts the quality of the 3D model according to the performance of the terminal device 400 to receive the 3D model information by simply adjusting the number of vertices of the 3D model corresponding to the 3D model information. can do.

The 3D model size conversion unit 346 may receive depth information about a space in which each terminal device 400 is used from the terminal device management unit 350 to be described later.

In addition, the 3D model size conversion unit 346 receives the 3D model information from the connection management unit 310, the decryption unit 320, the decompression unit 330, and the like, a terminal device to receive the 3D model information. Set the location or distance to display the first object 500a or the second object 500b based on the depth information of the space in which the 400 is used, and the size of the 3D model corresponding to the received 3D model information You can convert

For example, the 3D model size converting unit 346 sets the distance of displaying the 3D model and the size of the 3D model smaller as the space in which the terminal device 400 is used is smaller, and the terminal device 400 is used. The larger the space, the larger the distance and 3D model size can be set. At this time, the 3D model size conversion unit 346 may set the minimum distance at which the reduced 3D model is displayed to 1.25m and the maximum distance at which the enlarged 3D model is displayed to 5m.

Here, the distance displaying the 3D model may mean a distance between the 3D model displayed on the terminal device 400 and the reference coordinates set on the terminal device 400.

In addition, here, that the space in which the terminal device 400 is used is narrow may mean that an obstacle is located close to the front of the terminal device 400 and that the space in which the terminal device 400 is used is large is It may mean that there is no obstacle in the front of the device 400 or it is located in a distant place.

6 is a view showing an example of adjusting the size of an object in a holographic providing system according to an embodiment.

Referring to FIG. 6, the 3D model size conversion unit 346 in the holoportation providing system 10 and 20 according to an embodiment displays a 3D model in a space where the terminal device 400 is used in a narrow space. The distance and the size of the 3D model can be set to a small size (left drawing), and the distance displaying the 3D model and the size of the 3D model can be set large in the open space where the terminal device 400 is used. (Right drawing)

The 3D model type conversion unit 348 may receive a type of a 3D model supported by each terminal device 400 from the terminal device management unit 350, which will be described later.

Here, the type of the 3D model is, for example, a 3D point cloud or a 3D mesh. The 3D mesh is as described above, and the 3D point cloud is a type of 3D model composed of only the vertices of the 3D model, and can be expressed as 3D model information including geometric information about vertex coordinates. Can be.

In addition, the 3D model type conversion unit 348 receives the 3D model information from the connection management unit 310, the decryption unit 320, the decompression unit 330, and the like, a terminal device to receive the 3D model information The 3D model corresponding to the 3D model information may be converted into a 3D point cloud or a 3D mesh according to the type of 3D model supported by 400.

At this time, the 3D model type conversion unit 348 may generate new connection information using the geometric information of the 3D point cloud in order to convert the 3D point cloud into a 3D mesh.

Here, the 3D model type conversion unit 348 converts a 3D model corresponding to 3D model information into a 3D point cloud or a 3D mesh, and there are two methods of selection transformation and automatic transformation.

The selective transformation is that the 3D model type conversion unit 348 converts the 3D model into a 3D point cloud or a 3D mesh according to the type of the 3D model selected by the terminal device 400. However, the 3D model type conversion unit 348 checks the performance of the terminal device 400 and determines the 3D model only when it is determined that the type of the 3D model selected by the terminal device 400 operates normally in the terminal device 400. It may be restricted to convert to the type of the selected 3D model.

Here, the performance of the terminal device 400 may be determined by whether the terminal device 400 can display a 3D point cloud or a 3D mesh.

In addition, here, for example, in order to check the performance of the terminal device 400 or to determine whether the selected 3D model type is operating normally in the terminal device 400, for example, the 3D model type conversion unit 348 is the terminal device 400. The test data corresponding to the selected 3D model type may be transmitted to the terminal device 400 to check whether the test data is normally performed on the terminal device 400.

The automatic conversion is not limited by the type of the 3D model selected by the 3D model type conversion unit 348 in the terminal device 400, and after checking the performance of the terminal device 400, selects a suitable 3D model type and selects the selected 3D It converts a 3D model into a 3D point cloud or 3D mesh depending on the type of model.

For example, even if a 3D mesh is selected as the 3D model type in the terminal device 400, the 3D model type conversion unit 348 checks the performance of the terminal device 400 and then the 3D mesh is displayed in the terminal device 400. If it is determined that it cannot be displayed normally, the 3D model type conversion unit 348 may convert a 3D model corresponding to 3D model information into a 3D point cloud.

As described above, the relay server 300 includes a converter 340 that converts the format of the 3D model information or the quality, size, or type of the 3D model corresponding to the 3D model information. 10 and 20 may support various types of terminal devices 400.

The terminal device manager 350 may receive the identifier or IP address of the terminal device 400 and the format of the 3D model information supported by the terminal device 400 from each terminal device 400. Specifically, when the terminal device 400 accesses the relay server 300, the terminal device management unit 350 provides the terminal device 400 with the identifier or IP address of the terminal device 400 and the terminal device 400 supported by the terminal device 400. The information on the format of the dimensional model information may be requested, or the terminal device 400 may be controlled to extract information on the format of the 3D model information supported by the terminal device 400 from the terminal device 400 to actively grasp it. have.

Also, the terminal device management unit 350 may receive an identifier or an IP address of the terminal device 400 and information on the performance of the terminal device 400 from each terminal device 400. Here, the performance of the terminal device 400 is as described above. Specifically, when the terminal device 400 accesses the relay server 300, the terminal device management unit 350 provides the terminal device 400 with the identifier or IP address of the terminal device 400 and the performance of the terminal device 400. Information may be requested, or the terminal device 400 may be controlled to extract information about the performance of the terminal device 400 from the terminal device 400 to actively grasp it.

In addition, the terminal device management unit 350 may receive an identifier or IP address of the terminal device 400 and depth information about a space in which the terminal device 400 is used, from each terminal device 400. Specifically, the terminal device management unit 350 when the terminal device 400 accesses the relay server 300, the terminal device 400 to the identifier or IP address of the terminal device 400 and the space where the terminal device 400 is used Depth information for the may be requested, or the terminal device 400 may be controlled to extract depth information about a space in which the terminal device 400 is used from the terminal device 400 to actively grasp it.

In addition, the terminal device management unit 350 may receive information on the type of the 3D model supported by the terminal device 400 from each terminal device 400. Specifically, when the terminal device 400 accesses the relay server 300, the terminal device management unit 350 provides the terminal device 400 with the identifier or IP address of the terminal device 400 and the terminal device 400 supported by the terminal device 400. Information on the type of the dimensional model may be requested, or the terminal device 400 may be controlled to extract information on the type of the 3D model supported by the terminal device 400 from the terminal device 400 to actively grasp it.

In addition, the terminal device management unit 350 may receive various attribute information or status information of the terminal device 400 from the terminal device 400 in addition to the functions or capabilities of the above-mentioned terminal device 400, and the received terminal device ( 400) may transmit attribute information such as function and performance, or status information to the connection management unit 310 or the converter 340.

The rendering unit 360 renders the 3D model information received and transmitted from the connection management unit 310, the encryption/decryption unit 320, the decompression unit 330, the converter 340, and the like, the 3D model information. Thus, the 2D image information can be generated and the generated 2D image information can be transmitted to the player unit 370, which will be described later.

Here, rendering may refer to a process of automatically generating a photorealistic or non-realistic image from a 3D model through a computer program.

The player unit 370 receives the 3D model information from the connection management unit 310, the encryption/decryption unit 320, and the decompression unit 330, or receives the 2D image information from the rendering unit 360 and receives the 3D model. Information or two-dimensional image information can be displayed on the screen. The player unit 370 may have basic control functions such as playing or stopping 3D model information or 2D image information.

The player unit 370 may be used to check whether an error occurs or an error degree of the 3D model information received from the connection management unit 310, the decryption unit 320, and the decompression unit 330.

Also, the player unit 370 may be used in the converter 340 to check 3D model information corresponding to the original before being converted according to each device.

In addition, the player unit 370 may provide a user interface for setting camera coordinates, coordinate centers of the 3D model, and the like for the 3D model information, so that the converter 340 can utilize the set values.

The second communication unit 380 may receive status information, attribute information, and the like of the terminal device 400 from the local terminal device 400, and the remote server's restoration server 200 or the relay server 300 existing in the remote site or The terminal device 400 may receive a connection, a state in which a function can be provided, and the like, or the first model information or the second model information generated or compressed or encrypted by the local or remote restoration server 200. Model information can be received.

In addition, the second communication unit 380 can transmit the local restoration server 200 or the terminal device 400 to a state in which access is possible, a state in which a function can be provided, and the like, and the local restoration server 200 ), the first model information or the second model information, which is generated or compressed, or encrypted, may be transmitted, and the 3D model information may be transmitted to the terminal device 400.

The second database 390 can store status information, attribute information, etc. of the local or remote restore server 200 or the terminal device 400, and is generated, compressed, or encrypted in the local or remote restore server 200. One model information or second model information may be stored.

The second control unit 395 controls the overall operation of the relay server 300, the connection management unit 310, the decryption unit 320, the decompression unit 330, the converter 340, the terminal device management unit 350 , It may control the control flow or data flow between the rendering unit 360, the player unit 370, the second communication unit 380 and the second database 390.

As described above, the terminal device 400 may include a hololens, a head mounted display (HMD), a smart watch, a laptop, a mobile phone, a smart phone, a portable media player (PMP), a personal digital assistant (PDA), or a tablet PC ( Tablet PC).

In addition, the terminal device 400 may receive the converted first model information or second model information from the relay server 300 and display the first object 500a or the second object 500b on a screen.

In addition, the terminal device 400 displays various attribute information or status information of the terminal device 400 such as the format of the 3D model information supported by the terminal device 400, the type of the 3D model, and the performance of the terminal device 400. It can be transmitted to the relay server 300.

In addition, the terminal device 400 may measure depth information about the space in which the terminal device 400 is used by using sensors such as a binocular camera, an RGB-D camera, and a laser scanner provided in the terminal device 400. The measured depth information may be transmitted to the terminal device management unit 350.

For example, when an obstacle such as a wall surface is located near the front of the terminal device 400, the terminal device 400 may determine the shortest distance to the wall as depth information and transmit it to the terminal device manager 350. When there is no obstacle to the far front side of the device 400, the terminal device 400 may determine a predetermined longest distance as depth information and transmit it to the terminal device management unit 350.

In addition, a private IP as well as a public IP may be set in the terminal device 400 as described above.

7 is a view exemplarily showing an example of data flow in a holoportation providing system according to an embodiment.

Referring to FIG. 7, in the holoportation providing system 10 and 20 according to an embodiment, the restoration server 200 includes a connection management unit 310 and a second communication unit 380 of a relay server 300 existing locally. The first model information or the second model information generated or compressed or encrypted by the restoration server 200 may be transmitted to the remote site, and the terminal device 400 is generated by the restoration server 200 at the remote site through the repeater 300. The first model information or the second model information, which is compressed or encrypted, may be received.

At this time, the terminal device 400 is remote through the second communication unit 380, the connection management unit 310, the decryption unit 320, the decompression unit 330, the converter 340, the second communication unit 380 The first model information or the second model information generated or compressed or encrypted by the restoration server 200 may be received. Here, since the relay server 300 needs to receive the 3D model information from a remote location and transmit it to the terminal device 400 existing locally, the relay server 300 undergoes the second communication unit 380 twice (reception, transmission).

8 is a flowchart illustrating a method for providing holography according to an embodiment of the present invention.

Referring to FIG. 8, a method 600 for providing holoportion according to an embodiment may include a sensing step (S610), a restoring step (S620), a relaying step (S630), and a displaying step (S640). .

In the sensing step (S610), the holoportion providing system 100 senses the first object 500a or the second object 500b by sensors such as a plurality of cameras, binocular cameras, and laser scanners, to provide two-dimensional image information or Distance information can be generated.

In the restoration step (S620), the holoportion providing system 100 restores the 3D model of the first object 500a or the second object 500b based on the 2D image information or distance information generated through sensing. By doing so, the first model information for the first object 500a or the second model information for the second object 500b may be generated.

In addition, the holoportation providing system 100 can compress the first model information or the second model information to quickly transmit data through a network, and the first model information and the second model information to prevent forgery or tampering. , The compressed first model information or the compressed second model information may be encrypted or a watermark may be inserted.

In addition, the holoportation providing system 100 may transmit the generated first model information or second model information or compressed or encrypted first model information or second model information to the relay server 300. Hereinafter, the first model information or the second model information is mixed with the 3D model information.

In the relaying step (S630), the holoportion providing system 100 may receive and convert 3D model information and transmit the converted 3D model information.

Specifically, the holoportation providing system 100 can receive and decrypt the encrypted 3D model information and the encryption key, detect the inserted watermark, and determine whether the received 3D model information is authentic. Also, it is possible to decompress by receiving the compressed 3D model information.

Also, the holoportation providing system 100 may convert the format of the first model information or the second model information, or the quality, size, or type of the 3D model corresponding to the first model information or the second model information. have.

More specifically, the holoportation providing system 100 may convert the format of the 3D model information according to the format of the 3D model information supported by the terminal device 400 to receive the 3D model information.

In addition, the holoportation providing system 100 adjusts the number of vertices of the 3D model corresponding to the 3D model information according to the performance of the terminal device 400 to receive the 3D model information to improve the quality of the 3D model. Can be converted.

In addition, the holoportation providing system 100 may display a first object 500a or a second object 500b based on depth information about a space in which the terminal device 400 to receive 3D model information is used. The location or distance can be set and the size of the 3D model corresponding to the 3D model information can be converted.

In addition, the holoportation providing system 100 uses a 3D point cloud or 3D model corresponding to 3D model information according to the type of 3D model supported by the terminal device 400 to receive 3D model information. Can be converted to dimensional mesh.

In the display step S640, the holoportion providing system 100 may receive the converted first model information or second model information and display the first object 500a or the second object 500b on a screen or the like. .

As described above, although described with reference to preferred embodiments of the present application, those skilled in the art variously modify the present application without departing from the spirit and scope of the present invention described in the following claims. And can be changed.

10: Unidirectional holoportation system
20: bi-directional holographic provision system
100: sensor unit 200: restore server
300: relay server
310: connection management unit 320: password decryption unit
330: decompression unit 340: converter
342: format conversion unit 344: 3D model quality conversion unit
346: 3D model size conversion unit 348: 3D model type conversion unit
350: terminal device management unit
400: terminal device

Claims (9)

A first sensor unit for sensing the first object;
A first restoration server that generates first model information and transmits the generated first model information by restoring the three-dimensional model of the first object through the sensing;
A first relay server that receives the first model information, converts it, and transmits the converted first model information; And
And a first terminal device that receives the converted first model information and displays a first object.
According to claim 1,
The first relay server comprises a converter for converting the format of the first model information, or a converter for converting the quality, size or type of a 3D model corresponding to the first model information.
According to claim 2,
And the converter converts the format of the first model information according to the format of the 3D model information supported by the first terminal device.
According to claim 2,
The converter provides a holographic porting system, characterized in that, by adjusting the number of vertices of the 3D model corresponding to the first model information according to the performance of the first terminal device, the quality of the 3D model is converted.
According to claim 2,
The first terminal device measures depth information about a space in which the first terminal device is used,
The converter provides a holoportion providing system, wherein the first object is set based on the depth information or a distance to be displayed and the size of the 3D model corresponding to the first model information is converted.
According to claim 2,
The converter provides a holographic porting system, wherein the 3D model corresponding to the first model information is converted into a 3D point cloud or a 3D mesh according to the type of 3D model supported by the first terminal device. .
According to claim 1,
The holoportation providing system,
A second sensor unit for sensing the second object;
A second restoration server that restores a 3D model of the second object through the sensing, generates second model information, and transmits the generated second model information;
A second relay server for receiving the second model information, converting it, and transmitting the converted second model information; And
And a second terminal device for receiving the converted second model information and displaying a second object,
The first terminal device displays a first object to the second object, and the second terminal device displays a second object to the first object.
The method of claim 7,
The first restoration server transmits first model information to the first relay server via the second relay server,
The second restoration server is a holoportation providing system, characterized in that for transmitting the second model information to the second relay server via the first relay server.
A sensing step of sensing the first object to generate 2D image information or distance information;
A restoration step of restoring a 3D model of the first object based on the 2D image information or distance information to generate first model information and transmitting the generated first model information;
A relay step of receiving and converting the first model information and transmitting the converted first model information; And
And a display step of receiving the converted first model information and displaying a first object.

Images