WO2020032434A1

WO2020032434A1 - Method for providing three-dimensional image contents and program

Info

Publication number: WO2020032434A1
Application number: PCT/KR2019/009142
Authority: WO
Inventors: 김태욱; 이하람; 정덕영
Original assignee: 클릭트 주식회사
Priority date: 2018-08-07
Filing date: 2019-07-24
Publication date: 2020-02-13
Also published as: KR20200016657A; KR102090009B1; TW202008312A

Abstract

A method for providing three-dimensional image contents is provided. The method comprises the steps, performed by a server, of: dividing three-dimensional image data into a plurality of overlay items; assigning each of the plurality of overlay items to each of a plurality of layers so as to generate layer data for each of the plurality of layers; and transmitting each piece of layer data to a client through respective channels.

Description

Method and program for providing 3D image contents

The present invention relates to a method and a program for providing three-dimensional image content.

Increasingly, contents that can provide various services using images in a three-dimensional space such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and the like, are increasing.

Contents such as virtual reality, augmented reality, mixed reality, and the like generally provide a user with a three-dimensional virtual world similar to the real world created by a computer, and can be freely manipulated with the virtual world in real time. It is a technology that enables artificial experiences and experiences by providing a means of sensory feedback that provides actual senses in response.

Recently, a method of coupling a mobile terminal to a device (eg, a VR device) for providing 3D image content, and performing a processing of information for generating a VR image by the mobile terminal, and then delivering the same to a VR device has emerged. . This is because the mobile terminal, which is a computer that performs information processing, and the VR device to be played back are connected by wire, but the mobile terminal can be coupled to the VR device, thereby solving the problem caused by the cable. However, the mobile terminal has a problem that it is difficult to drive a high-end program compared to a PC, it is difficult to drive a high-end VR game or play a high-resolution VR video. In addition, as well as the method of coupling the mobile terminal to the VR device, there is a problem that it is difficult to drive a VR program of high specification and high resolution as compared to a PC even in a stand-alone VR device.

The problem to be solved by the present invention is to provide a method and program for providing three-dimensional image content.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present disclosure, a method of providing 3D image content by a server includes: dividing 3D image data into a plurality of overlay items, and assigning each of the plurality of overlay items to each of the plurality of layers. Generating layer data for each of the layers, and transmitting each layer data to the client through each channel.

In an embodiment of the present disclosure, the dividing into the plurality of overlay items may include dividing the at least one object into the plurality of overlay items based on characteristics of at least one object included in the 3D image data. can do.

In an embodiment of the present disclosure, generating layer data for each of the plurality of layers may include configuring metadata for each of the plurality of overlay items and each layer data including the metadata. It may include the step of generating.

In one embodiment of the present invention, the metadata includes location information for each of the plurality of overlay items, the location information, the location where each of the plurality of overlay items should be displayed on the screen of the client Information.

In one embodiment of the present invention, the metadata further includes additional information according to the characteristics of at least one object included in each of the plurality of overlay items, the additional information, normal for displaying a control object It may include at least one of a vector, a compression method, a correction state, direction information of the 3D image data, information for representing a 3D object, and information on a specific object that needs to be dynamically synthesized.

A method for providing 3D image content by a client according to an embodiment of the present invention includes receiving a plurality of layer data from a server through a plurality of channels, and assigning each of the plurality of layer data to each of the plurality of layers. And generating three-dimensional image data by synthesizing the plurality of layers.

In an embodiment of the present disclosure, the assigning to each of the plurality of layers may include assigning each of the plurality of overlay items included in each of the plurality of layer data to each of the plurality of layers.

The generating of the 3D image data may include extracting metadata from the plurality of layer data, and extracting each of the plurality of overlay items from the client based on the metadata. And synthesizing the plurality of layers by allocating the positions to be displayed on the screen.

In an embodiment of the present disclosure, the generating of the 3D image data may further include additional information according to characteristics of at least one object included in each of the plurality of overlay items. The method may further include processing the overlay item based on the additional information included in the metadata.

In one embodiment of the present invention, when the client generates at least one object included in the 3D image data, generating the at least one object and assigning it to a separate layer, and the separate layer And synthesizing with the plurality of layers to finally generate the 3D image data.

A computer program according to an embodiment of the present invention is combined with a computer, which is hardware, and stored in a computer-readable recording medium for executing a method for providing 3D image content performed by the server and the client.

According to an embodiment of the present invention, since the objects constituting the 3D image data may be divided into a plurality of overlay items according to characteristics and transmitted individually through each channel, all layers or all data constituting the 3D image data. There is no need to send all at once. Accordingly, the network load and the video delay can be reduced, thereby providing higher quality 3D video content to the user.

According to an embodiment of the present invention, since various metadata may be provided in generating 3D image data, it is easy to configure a layer suitable for each overlay item, and the data may be processed in various ways for each layer.

According to an embodiment of the present invention, since the three-dimensional image data is processed into a plurality of layers according to the characteristics of the objects, it is possible to perform the operation by extracting only the layers that need correction or need an immediate response, so that the entire image There is no need to deal with. Therefore, the 3D image data can be effectively processed even by a client having a lower performance than the server.

Effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram of a system for transmitting and receiving 3D image content according to an embodiment of the present invention.

2 is a flowchart illustrating a method of providing 3D image content by the server 100 according to an embodiment of the present invention.

3 is a diagram illustrating an example of dividing 3D image data into a plurality of overlay items and allocating the corresponding 3D image data to corresponding layers according to an embodiment of the present invention.

4 is a diagram illustrating an example in which a first overlay item allocated to a first layer is displayed on a screen of a client 200 according to an embodiment of the present invention.

5 is a flowchart illustrating a method for providing 3D image content by the client 200 according to an exemplary embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be embodied in various different forms, only the present embodiments are intended to complete the disclosure of the present invention, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and / or “comprising” does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be the second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention;

Referring to FIG. 1, a 3D image content transmission and reception system according to an embodiment of the present invention may include a server 100 and a client 200.

The server 100 may be a computer device that generates 3D image content. 3D image content may be content that can provide various services using images in a 3D space, for example, virtual reality (VR), augmented reality (AR), mixed reality (Mixed Reality) ; MR) or the like.

The server 100 performs information processing therein to generate 3D image content to be provided to the client 200. For example, when the 3D image content is a VR image according to a specific game driving, the server 100 may drive a game program to generate appropriate image data and transmit it to the client 200 through wireless communication.

The server 100 generates image data (hereinafter, referred to as 3D image data) corresponding to 3D image content, divides the image data into a plurality of overlay items, and assigns each of the divided overlay items to each layer to make a layer for each layer. You can generate data. In addition, the server 100 may transmit layer data for each layer to the client 200 through each channel using wired or wireless communication. A detailed operation process thereof will be described later.

The client 200 may be a device that receives and reproduces image data corresponding to 3D image content. That is, the client 200 plays a function of reproducing 3D image data received from the server 100 and providing the same to the user. For example, the client 200 may be a device for playing 3D image content, for example, a VR device, an AR device, or an MR device itself, or a mobile terminal may be coupled to the device. For example, when the VR device and the mobile terminal are combined to configure the client 200, the mobile terminal receives the 3D image data generated from the server 100, and the VR device through a wired cable or short-range wireless communication connected thereto. Can be displayed on the screen.

The client 200 may be configured in various forms and reproduce and display 3D image data on a screen. For example, the client 200 may display image data including each image suitable for both eyes on one display unit, and generate the 3D image by the fisheye lens in each eye direction. As another example, the client 200 may include two display units that provide an image corresponding to each eye.

The client 200 receives a plurality of layer data from the server 100 through a plurality of channels, and allocates each received layer data to each layer. The client 200 may generate three-dimensional image data by synthesizing each layer. A detailed operation process thereof will be described later.

The server 100 and the client 200 may be connected through wired or wireless communication. Here, the Wi-Fi method, cellular communication, etc. may be utilized as the wireless communication method. For example, if the server 100 is a computer located in a specific space (eg, home, virtual reality experience space, etc.) where the user is located, the client 200 may be connected through a wireless AP (eg, Wi-Fi AP). Communication between the server and the server 100 may be performed. In addition, when the server 100 is a computer located at an external remote location, the remote server 100 may transmit 3D image data to the client 200 through cellular communication or LAN communication. The client 200 may receive 3D image data from a base station through cellular communication or receive 3D image data from a wireless AP through a WLAN. Through this, if the user possesses the client 200 capable of wireless communication, the user may receive and use 3D image data provided from the server 100 even if the user does not have a location close to the server 100.

Hereinafter, a method of providing 3D image content in the server 100 and the client 200 according to an exemplary embodiment of the present invention will be described with reference to the accompanying drawings.

Meanwhile, the image data corresponding to the 3D image content, that is, the 3D image data according to the exemplary embodiment of the present invention may be composed of a plurality of layers and displayed on the display screen. In this case, each layer may be logically configured according to a specific function and vertically overlapped.

Referring to FIG. 2, the server 100 may divide 3D image data into a plurality of overlay items (S100).

In an embodiment, the server 100 may divide the 3D image data into a plurality of overlay items based on the characteristics of at least one object included in the 3D image data.

The plurality of overlay items may be composed of objects having logically specific functions in the 3D image data, respectively. For example, the server 100 extracts an object (eg, a background image) that is displayed statically from 3D image data, sets it as a first overlay item, and dynamically displays an object (eg, an image in which movement occurs). May be extracted and set as the second overlay item. Alternatively, the server 100 extracts an object (eg, a user interface) that displays graphic or text-based information, sets it as a third overlay item, and displays an object (eg, a crosshair displayed according to the operation of the controller) that displays control information. , Pointer) may be extracted and set as the fourth overlay item. Alternatively, the server 100 dynamically obtains information from the 3D image data, extracts a specific object (eg, a billboard for displaying an advertisement, etc.) to reflect the information on a specific area, and sets the fifth overlay item. The object displayed as the dimensional image may be extracted and set as the sixth overlay item.

The server 100 may allocate each of the plurality of overlay items to each of the plurality of layers, and generate layer data for each of the plurality of layers (S110).

That is, the server 100 assigns the first to nth overlay items divided according to the characteristics of the objects included in the 3D image data to the first to nth layers corresponding to the first to nth layers, respectively. You can create layer data for. For example, a first overlay item including an object that is statically displayed in 3D image data may be allocated to the first layer and configured as first layer data. 3 is a diagram illustrating an example of dividing 3D image data into a plurality of overlay items and allocating the corresponding 3D image data to corresponding layers according to an embodiment of the present invention.

In an embodiment, the server 100 may configure metadata for each of a plurality of overlay items, and generate each layer data including each metadata. That is, the server 100 may configure information necessary for playing on the client 200 side as metadata according to the characteristics of each object included in the first to nth overlay items, and include it in the layer data.

For example, the metadata may include location information about each of the plurality of overlay items, and the location information may be location information where each of the plurality of overlay items is to be displayed on the screen of the client 200.

Referring to FIG. 4, the server 100 may allocate a first overlay item to a first layer, and the first overlay item may include at least one object having a specific function in 3D image data. For example, if the first overlay item includes objects corresponding to the user interface (eg, 1-1, 1-2, 1-3, 1-4 objects), each object (eg, 1-1) , 1-2, 1-3, 1-4 objects) may be allocated to the same layer and transmitted to the client 200. In this case, each object (eg, 1-1, 1-2, 1-3, 1-4 object) may be displayed at a specific position on the screen of the client 200. In this case, the server 100 may display each object ( For example, 1-1, 1-2, 1-3, 1-4 objects may configure location information to be displayed on the screen of the client 200 as metadata about a corresponding overlay item (ie, a corresponding layer). . The location information may be displayed as coordinate information. For example, the location information may be displayed as (x, y) coordinates to be positioned on the screen of the client 200. Also, if each object has depth information, it may further include z-order information.

In addition to the location information described above, the metadata may further include additional information according to at least one object property included in the plurality of overlay items. The additional information may include normal vector information for displaying a control object including control information, information for representing an object including a three-dimensional image, a compression method for each overlay item (ie, each layer), a correction method, It may include direction information of the 3D image data, information for representing a specific object that needs to be dynamically synthesized, and the like.

For example, if the first overlay item includes a control object including control information, the server 100 may allocate the first overlay item to the first layer and display the control object on the screen of the client 200. Layer data for the first layer may be generated by configuring metadata including the normal vector information. In one embodiment, the server 100 receives the operation information (including the position information of the controller and the direction indicated by the controller) of the controller controlling the control object from the client 200, and based on the received operation information It may determine a specific point in the 3D image data indicated by and the normal vector for the object present at the specific point. Thereafter, the server 100 may configure the determined specific point and normal vector information as metadata.

As another example, when the second overlay item includes an object including a 3D image, the server 100 may allocate the second overlay item to the second layer and display the 3D image on the screen of the client 200. The layer data for the second layer may be generated by configuring information, for example, texture information, polygon mesh information, and the like, for improving the three-dimensional effect of the object.

As another example, when the third overlay item includes a specific object that needs to be dynamically synthesized, the server 100 may allocate the third overlay item to the third layer and display the specific object on the screen of the client 200. The information may be configured as metadata to generate layer data for the third layer. In one embodiment, the method includes an object that dynamically obtains information from the 3D image data and reflects the information on a specific area, and includes an object such as a billboard that displays an advertisement in the 3D image data. This may be the case. Such an advertisement board object may generate layer data by including advertisement information in advance in the server 100, but separate from an external device such as an advertisement server in real time when the client 200 receives and plays 3D image data. You can also receive information and dynamically synthesize the object. Therefore, the server 100 may additionally generate location information, a normal vector of the object, information about an external device, and the like, for the overlay item including the object such as an advertisement board and configure the metadata.

As another example, the server 100 may set different compression methods for a plurality of overlay items, and in this case, the server 100 may generate each layer data by configuring the compression method set for each overlay item as metadata. Can be. For example, the server 100 may divide a background object that is statically displayed in the 3D image data and a dynamic object that is dynamically displayed, set each overlay item, allocate the same to each layer, and compress the same by using different compression methods. . Thereafter, the compression method performed on each layer may be configured as metadata to generate layer data for each layer.

As another example, the server 100 may set different correction methods for a plurality of overlay items, and in this case, the server 100 may generate each layer data by configuring the correction method set for each overlay item as metadata. Can be. For example, the server 100 classifies the overlay items that need to be delayed like the background object from the 3D image data and the overlay items that need to be immediately reacted to by the user's movement. Metadata may be configured including information necessary for correction.

The server 100 may transmit layer data for each layer to the client 200 through each channel (S120).

As described above, the server 100 may transmit layer data to the client 200 through wired or wireless communication. For example, the server 100 may transmit first layer data through a wired or wireless communication through a first channel and second layer data through a second channel.

Referring to FIG. 5, the client 200 may receive a plurality of layer data from the server 100 through a plurality of channels (S200).

In an embodiment, the client 200 receives a plurality of layer data from the server 100 through wired or wireless communication, receives first layer data through a first channel, and second layer data through a second channel. Can be received.

The client 200 may allocate each of the plurality of layer data to each of the plurality of layers (S210).

That is, the client 200 may allocate each of the plurality of overlay items included in each of the plurality of layer data to each of the plurality of layers. For example, the first layer data may be allocated to the first layer, and the first layer item is allocated to the first layer including at least one object having a specific function (that is, the same or similar characteristics).

The client 200 may generate 3D image data by synthesizing the plurality of layers (S220).

In one embodiment, the client 200 extracts metadata from the plurality of layer data, and allocates each of the plurality of overlay items to a position to be displayed on the screen of the client 200 based on the extracted metadata, and then each layer. Can be synthesized.

As described above, the metadata may include location information about each of the plurality of overlay items. For example, the client 200 obtains location information of at least one object included in the first overlay item from metadata, and allocates at least one object on the first layer based on the obtained location information. Next, the 3D image data may be finally generated by synthesizing the same. Since a detailed process thereof has been described with reference to FIG. 4, the detailed description thereof will be omitted.

In addition to the location information, the metadata may further include additional information according to at least one object property included in the plurality of overlay items. For example, the metadata may include normal vector information for displaying a control object including control information, information for representing an object including a three-dimensional image, a compression scheme for each overlay item (ie, each layer), and correction. A method, direction information of the 3D image data, and information for expressing a specific object that needs to be dynamically synthesized. That is, the client 200 may obtain information (ie, the above-described additional information) necessary for displaying the corresponding overlay item assigned to each layer on the screen from the metadata, and based on the obtained additional information, each layer Each layer can be combined by displaying the corresponding overlay item on the screen.

According to an embodiment, the client 200 may generate at least one object included in the 3D image data. For example, this may correspond to a case where the client 200 configures a layer itself without receiving layer data (ie, information on at least one layer) from the server 100.

In this case, the client 200 may generate at least one object and assign it to a separate layer, and synthesize the separate layer with the layer information received from the server 100 to finally generate 3D image data. have. For example, if it includes an object that needs to react immediately as a user moves (eg, a control object including control information displayed as the user performs an operation through a controller), the client 200 Generates a separate layer displaying control information of the controller and determines a specific point in the 3D image data indicated by the control object and a normal vector of an object present at the specific point based on the controller's motion information. It can be displayed on a separate layer. Thereafter, the client 200 may finally combine 3D image data with another layer and another layer received from the server 100.

In addition, when the location is changed only for a specific object in the 3D image data, the client 200 receives only the corresponding overlay item including the changed location information for the specific object from the server 100 to update the 3D image data. It may be. In addition, when correction is required only for a specific layer in the 3D image data, the client 200 may update the 3D image data by receiving only information on the specific layer that needs correction from the server 100.

According to the embodiment of the present invention as described above, since the objects constituting the 3D image data can be divided into a plurality of overlay items according to characteristics and transmitted individually through respective channels, There is no need to transfer a layer or all of the data all at once. Accordingly, the network load and the video delay can be reduced, thereby providing higher quality 3D video content to the user.

In addition, according to the embodiment of the present invention as described above, since it is possible to provide a variety of metadata in generating the three-dimensional image data, it is easy to configure a layer suitable for each overlay item and the data in various ways for each layer Can be processed.

In addition, according to the embodiment of the present invention as described above, since the three-dimensional image data is processed by separating into a plurality of layers according to the characteristics of the object, only the layer that needs to be corrected or immediately reacts to perform the operation You do not need to process the entire image. Therefore, the 3D image data can be effectively processed even by a client having a lower performance than the server.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. The software module may be a random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may realize the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

In the method for providing three-dimensional image content by the server,

Dividing the 3D image data into a plurality of overlay items;

Assigning each of the plurality of overlay items to each of a plurality of layers to generate layer data for each of the plurality of layers; And

And transmitting the respective layer data to a client through a respective channel.
The method of claim 1,

The dividing into the plurality of overlay items may include:

And dividing the at least one object into the plurality of overlay items based on characteristics of at least one object included in the 3D image data.
The method of claim 1,

Generating layer data for each of the plurality of layers may include:

Constructing metadata for each of the plurality of overlay items; And

And generating the layer data by including the metadata.
The method of claim 3,

The metadata includes location information for each of the plurality of overlay items.

The location information is a method of providing three-dimensional image content, characterized in that each of the plurality of overlay items are location information that should be displayed on the screen of the client.
The method of claim 4, wherein

The metadata may further include additional information according to characteristics of at least one object included in each of the plurality of overlay items.

The additional information may include a normal vector for displaying a control object, a compression scheme, whether to correct the correction information, direction information of the 3D image data, information for representing a 3D object, and information about a specific object that needs to be dynamically synthesized. Method for providing three-dimensional image content, characterized in that it comprises at least one.
In the method for the client to provide three-dimensional image content,

Receiving a plurality of layer data through a plurality of channels from a server;

Allocating each of the plurality of layer data to each of the plurality of layers; And

And synthesizing the plurality of layers to generate three-dimensional image data.
The method of claim 6,

Assigning to each of the plurality of layers,

And assigning each of a plurality of overlay items included in each of the plurality of layer data to each of the plurality of layers.
The method of claim 7, wherein

Generating the 3D image data,

Extracting metadata from the plurality of layer data; And

And synthesizing the plurality of layers by assigning each of the plurality of overlay items to a position that should be displayed on the screen of the client based on the metadata.
The method of claim 8,

Generating the 3D image data,

When the metadata further includes additional information according to the characteristics of at least one object included in each of the plurality of overlay items,

And processing a corresponding overlay item based on the additional information included in the metadata.
The method of claim 6,

When the client generates at least one object included in the 3D image data,

Generating the at least one object and assigning the at least one object to a separate layer; And

And synthesizing the separate layers with the plurality of layers to finally generate the 3D image data.
A computer program, coupled with a computer in hardware, stored on a computer readable recording medium for carrying out the method of any one of claims 1 to 10.