KR20230078369A - Method for providing 3D image service using hybrid transmission system - Google Patents

Abstract

A method for providing a three-dimensional (3D) image service from a server is disclosed. The method includes the steps of: sending a description file to a user terminal; receiving a request message from the user terminal requesting content selected by the user based on the description file; generating 3D image content or 2D image stream content according to the request message; and transmitting the generated 3D image content or 2D image stream content to the user terminal.

Description

Method for providing 3D image service using hybrid transmission system

The present invention relates to a transmission system technology capable of reproducing and rendering a 3D image stored in a server in real time.

Since a three-dimensional (3D) image includes spatial coordinates (z) as well as x and y coordinates, unlike a two-dimensional (2D) image, it is stored in a separate file format different from that of a 2D image. Since 3D video renders a two-dimensional (2D) video from the user's point of view, in order to receive this service, users must have a technology (hereinafter referred to as a renderer) that parses and renders a file of the corresponding format. .

In the case of complex 3D images (high resolution or high precision), a high-end hardware terminal is required, and when the data capacity is large, real-time rendering may not be possible or initial delay and service interruption may occur depending on the terminal specifications.

Recently, in order to solve this problem, a technology for streaming 2D images after performing 2D rendering according to a user's point of view in a server has been developed. In this case, there is a problem in that the server may be overloaded when there are many users because the server renders and transmits the view point of each user in real time.

Therefore, a system structure that enables service regardless of whether the terminal is a renderer, minimizes initial delay, and allows users to select a service type according to terminal specifications is required.

The present invention proposes a service structure in which a 3D video file is directly transmitted or a server renders a 3D video and then streams the result to a 2D screen according to the user's required content complexity and network environment when providing a 3D video service.

The service proposed according to the present invention provides the user with a description file listing the content type, media type, data acquisition path (URL, etc.), etc., and the user requests/acquires data suitable for himself/herself to play. When rendering is impossible or the initial delay is expected in the network due to the large size of data, streaming service can be received on a 2D screen, providing high-speed 3D video service regardless of the type of terminal.

The above-mentioned objects and other objects, advantages and characteristics of the present invention, and a method of achieving them will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

A method for providing a 3D image service according to an aspect of the present invention for achieving the above object is a method for providing a 3D image service in a server that performs socket communication with a user terminal of a user. Transmitting a description file to a terminal; Receiving a request message requesting content selected by the user based on the description file from the user terminal; generating 3D video content or 2D video stream content according to the request message; and transmitting the generated 3D video content or 2D video stream content to the user terminal.

In an embodiment, the description file may be file information specifying information related to content type, media type, content capacity, and access path (URL) serviceable by the server based on XML.

In an embodiment, the generating may include checking whether the user request is a 3D video content request or a 2D video stream content request according to the request message; generating the 3D video contents when the user request is a request for 3D video contents; and generating the 2D video stream contents when the user request is the 2D video stream contents request.

In an embodiment, the generating of the 3D image content may include obtaining 3D image data corresponding to the 3D image content from a 3D image database; and processing the acquired 3D image data to generate a 3D image data packet as the 3D image contents.

In an embodiment, the generating of the 2D video stream content may include obtaining 3D video data corresponding to the 3D video content from a 3D video database; rendering the 3D image data; generating a 2D video stream by encoding the rendered 3D video data; and generating a 2D video stream packet generated by processing the generated 2D video stream as the 2D video stream contents.

A method for providing a 3D image service according to an aspect of the present invention is a method for providing a 3D image service in a user terminal that performs socket communication with a server, comprising the steps of downloading a description file from a server; transmitting a request message requesting content selected based on the description file to the server; receiving 3D video data or 2D video stream from the server according to the request message; and reproducing the 3D video data or the 2D video stream.

In an embodiment, the description file may be XML-based specified file information related to content type, media type, content capacity, and access path (URL).

In an embodiment, the transmitting of the request message to the server may include, when the content selected based on the description file is low-capacity content, transmitting the request message requesting 3D image data to the server; and when the content selected based on the description file is high-capacity content, transmitting the request message requesting a 2D video stream to the server.

In an embodiment, the step of transmitting the request message to the server may include the request message requesting 3D image data when the user terminal supports 3D image rendering and the content selected based on the description file is low-capacity content. Transmitting to the server; and transmitting the request message requesting a 2D video stream to the server when the user terminal supports the 3D video rendering and the content selected based on the description file is high-capacity content.

In an embodiment, the transmitting of the request message to the server may include transmitting the request message requesting a 2D video stream to the server when the user terminal does not support 3D video rendering.

In an embodiment, the reproducing may include rendering the 3D image data; and reproducing the rendered 3D image data.

In an embodiment, the reproducing may include decoding the 2D video stream; and reproducing the decoded 2D video stream.

According to the present invention, since a user requests a content type and a URL suitable for him/herself, a server load can be reduced by simplifying a response from the server. In addition, 3D image service is possible regardless of the performance or type of terminal, so that various additional services can be provided.

1 is a block diagram showing basic configurations of a hybrid transmission system for providing a three-dimensional (3D) video service according to an embodiment of the present invention.
FIG. 2 is a block diagram showing the main components of the 3D video hybrid server shown in FIG. 1 in more detail.
FIG. 3 is a flowchart illustrating a process of processing a response to a request of a user terminal by the 3D video hybrid server shown in FIG. 2 .
FIG. 4 is a block diagram showing the main components of the user terminal shown in FIG. 1 in more detail.
5 is a flowchart illustrating a request processing process of the user terminal shown in FIG. 4 .

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The present invention relates to a transmission system technology capable of reproducing and rendering a 3D image stored in a server in real time, including a server and client technology capable of providing a service of a 3D image to a user, and providing a user with one It is characterized in that it is configured to include an XML-based descriptor transmission technology (DASH-MPD) that provides a serviceable list for 3D images.

In addition, the present invention is characterized in that it is configured to include a capture stream and scene control technology for streaming 3D video to 2D video, and a socket communication technology for low-latency service.

In addition, when providing a 3D video service, the present invention is characterized by a service in which a 3D video file is directly transmitted or a server renders a 3D video and then streams the result to a 2D screen according to the user's requested content complexity and network environment. .

In addition, since the present invention provides the user with a description file listing content types, media types, data acquisition paths (URL, etc.), and the user requests/obtains and plays data suitable for himself, 3D image rendering is impossible or When the initial delay is expected in the network due to the large size of data, streaming service can be received on a 2D screen, so it is possible to provide low-latency 3D video service regardless of the type of terminal.

The present invention can selectively stream one 3D image after 3D file transmission or 2D rendering in a server, provides a list of service types provided for 3D images, and makes it possible to request a service suitable for the user terminal environment. In this respect, it can be differentiated from the prior art.

1 is a block diagram showing basic configurations of a hybrid transmission system for providing a three-dimensional (3D) video service according to an embodiment of the present invention.

Referring to FIG. 1 , a hybrid transmission system 300 according to an embodiment of the present invention includes a 3D video hybrid server 100 and a user terminal (client or user device) 200 .

The 3D image hybrid server 100 and the user terminal 200 exchange information in the form of packets, files and/or messages in real time through a network (not shown in FIG. 1), and a communication method between them 100 and 200 may be a socket communication that is advantageous for real-time video streaming, real-time data transmission, and two-way communication.

Both the hybrid server 100 and the user terminal 200 may be referred to as 'computing devices' having a communication function, but there may be differences in processing performance. For example, the hybrid server 100 may be a computing device with higher processing performance than the user terminal 200 .

A computing device basically includes at least one processor (eg, at least one CPU and/or at least one GPU), at least one memory (eg, DRAM, SRAM, DDR, etc.), and at least one storage medium (eg, hard drive). disk, etc.), an input interface (eg, a key input unit), an output interface (eg, a display unit, etc.), and a communication interface.

The 3D video hybrid server 100 is a device for transmitting 3D video contents to the user terminal 200 regardless of the specifications of the user terminal 200, and provides a service of providing 3D video contents to the user. The 3D video hybrid server 100 includes a descriptor ('description file manager' or description file generator) 110, a 3D video renderer 1120, and a 2D stream encoder 130. In addition, the 3D video hybrid server 100 interworks with the 3D video database 140 storing 3D video contents.

The client 200 is a device for playing 3D video content provided from the 3D video hybrid server 100, and includes a description file parser 210 and a player 230, and the player 230 is a 3D video renderer 232 and a 2D stream decoder 234.

The 3D image hybrid server 100 provides the user with a content-type, media-type, access path (URL), etc. that the user terminal 200 can access through a description file. It is provided to the terminal 200, and the user terminal 200 selects an access path to receive data (contents) according to its own device environment and network environment to receive 3D video content service.

The 3D video hybrid server 100 generates a description file through the descriptor (or 3D video description generator) 110 based on the 3D video database 140 . The description file is based on XML like DASH-MPD [2] and can include content type and media type. Here, DASH stands for 'Dynamic Adaptive Streaming over HTTP', and MPD stands for 'Media Presentation Description'.

The user terminal 200 downloads the description file, parses the downloaded description file using the description file parser 210, and requests content suitable for the user environment. Through this, the user terminal 200 can determine the complexity of the 3D image, and requests and acquires 3D image content (or 3D image file) when real-time rendering is possible and the network environment is stable. In this case, the user terminal 200 can control (moving a 3D screen, etc.) the downloaded content in real time without delay on its own device.

When real-time 3D rendering is impossible or when network delay is expected due to a very large content data size, the user can request 3D video in the form of a 2D video stream. In this case, the 3D video hybrid server 100 renders and encodes the corresponding content in 2D using the 3D video renderer 120 and the 2D stream encoder 130, and streams the rendered and encoded corresponding content to the user terminal 200 do.

When the user terminal 200 controls the video, control information is transmitted to the 3D video hybrid server 100 through a control channel, and the 3D video hybrid server 100 reflects the control information to perform 2D rendering, and then renders the corresponding result stream to the user.

FIG. 2 is a block diagram showing the main components of the 3D video hybrid server shown in FIG. 1 in more detail.

Referring to FIG. 2, the 3D video hybrid server 100 according to an embodiment of the present invention includes the components 110, 120, 130, and 140 in the 3D video hybrid server 100 shown in FIG. 1 as well as a request controller 150. ), a 3D image data buffer controller 160, a stream buffer controller 170, and a response controller 180 may be further included.

The request controller 150 processes a user's request received in the form of a packet and/or message from the user terminal 200 . This request controller 150 may be a communication interface that performs receiving functions and messaging (or packet) processing functions.

The 3D image data buffer controller 160 generates 3D image data packets by processing 3D image data (3D image contents) desired by a user among 3D image data (3D image contents) stored in the 3D image database 140.

The renderer 120 renders a 3D object included in 3D image content desired by a user among 3D image contents stored in the 3D image database 140 .

The encoder 130 encodes a result rendered by the renderer 120 to generate a 2D video stream.

The stream buffer controller 170 processes the 2D video stream encoded by the encoder to generate media stream packets.

The response controller 180 processes the response to the user's request and the generated 3D image data packet and/or media stream packet and transmits them to the user terminal 200 . The response controller 180 may be a communication interface that performs a transmission function and a messaging (or packet) processing function.

The 3D image description generator 110 generates a description file in XML format based on the 3D image database and transmits it to the user terminal 200 through the response controller 180 .

The user of the user terminal 200 may request a 3D image service or a 2D stream service after receiving a description file for desired content, and may request control or modification of the content after receiving the service for the requested content. .

The response controller 180 delivers a request message and packet/file data for the request to the user.

FIG. 3 is a flowchart illustrating a process of processing a response to a request of a user terminal by the 3D video hybrid server shown in FIG. 2 .

Referring to FIG. 3, first, in step 301, the request controller 150 in the server 100 divides the user request from the user terminal 200 into a content request, a modification request, and a control request, and processes the corresponding request. .

Next, in step 302, the request controller 150 checks whether the request type according to the user's request is a content request.

Next, in step 303, if the request type is a content request, it is checked whether the media type request according to the user request is a 2D video stream.

Next, in step 304, if the media type request is a 2D video stream, the renderer 120 in the server 100 renders 3D video data that the user wants to receive in a 2D streaming method. Here, steps 303 and 304 may be integrated into one step. In this case, the integrated step may be a step of determining whether the user's request is a 3D video data (3D video content) request or a 2D video stream request.

Next, in step 305, the encoder 130 encodes the rendering result (rendered 3D video) to generate a 2D video stream.

Next, in step 306, the stream buffer controller (or stream packet generator) 170 processes the 2D video stream to generate media stream packets.

Next, in step 308, the response controller 180 in the server 100 transmits the generated media stream packet to the user terminal 200.

Meanwhile, in step 303, if the media type request is not 2D streaming, the process proceeds to step 309, and in step S309 it is checked whether the media type request is a 3D video (3D video content) request.

Next, in step 310, when the media type request is a 3D image request, the 3D image data buffer controller 160 processes the 3D image (3D image acquired from the 3D image database 140) requested by the user to obtain a 3D image data packet. And, in step 308, the response controller 180 in the server 100 transmits the generated 3D image data packet to the user terminal 200.

Meanwhile, when the media type request is not a 3D video (3D video content) request in step S309, the process proceeds to step 311, and in step S311, it is checked whether the media type request is a description file request.

In step 312, if it is checked that the media type request is a description file request, the 3D video description generator (or descriptor) 110 creates an XML format description file based on the 3D video database, and in FIGS. 1 and 2 Although not shown, after the description file manager generates (converts) the created description file into a packet (description file packet), in step 308, the response controller 180 transmits the description file packet to the user terminal 200. .

Meanwhile, in step 302, if the request type is not a content request, the process proceeds to step 313, and if the request type is a modification request, the 3D video (3D video content) is changed, the media type is checked, and the modified object is transmitted. .

In step 314, if the request type is a control request, since the media type is streaming, the request is reflected in 3D image data, and then encoded into a 2D stream and transmitted as a packet. All responses can send a response message about the request processing result together.

The steps described above are only listed in chronological order to facilitate understanding of the description, and some steps may be performed in parallel or simultaneously or integrated into one step. For example, steps 302 , 303 , 309 , 311 , 313 , and 314 of FIG. 3 , which are judged, are only classified to help understanding of the description, and may be integrated into one step.

FIG. 4 is a block diagram showing the main components of the user terminal shown in FIG. 1 in more detail.

Referring to FIG. 4, a user terminal 200 according to an embodiment of the present invention

Request Controller (201), Response Controller (202), Description File Parser (210), File/Data Buffer Controller (203), 3D Image Data Buffer Controller (204), Stream Buffer Controller (205), Renderer (232), Decoder 234 and event controller 236.

The request controller 201 calculates the currently available network capacity (bandwidth), selects an appropriate path based on the calculated current available network capacity according to whether the terminal has a renderer capability and the user's object content selection, and transmits the request message to the server Send to (100). This request controller 201 may be a communication interface that performs transmission functions and messaging (packet) processing functions.

The response controller 202 processes a response to the requested result. The response controller 202 may be a communication interface that performs a receiving function and a messaging (packet) processing function. The response controller 202 controls packets according to the type of requested content.

The file/data buffer controller 203 extracts a description file by processing the description file packet received from the server 100 through the response controller 202 and transfers it to the description file parser 210 .

The description file parser 210 parses (analyzes) the delivered description file, and provides list information indicating content type, media type, content size, access path (URL), etc. that can be serviced by the server 100 to the user interface. It is delivered to the user through (UI: User Interface). Here, the description file may specify detailed information on media types and contents serviceable by the server 100 as XML-based file information such as MEPG-DASH:MPD.

Based on the list information displayed through the user interface, the user selects a path through which data (contents) can be provided according to the user's device environment and network environment, and sends the server 100 the selected data (contents). ) is requested.

When the content type requested by the user is 3D video data, the 3D video data buffer controller 204 processes the 3D video data packet received from the server 100 through the response controller 202 to generate 3D video data (3D video content). ) is generated and passed to the renderer 232.

The renderer 232 renders 3D image data (3D image content), and the rendered 3D image data (3D image content) is played by a player.

When the content type requested by the user is a 2D stream, the stream buffer controller 205 processes the media stream packet received from the server 100 through the response controller 202 to generate an encoded 2D video stream, and decodes it ( 234).

The decoder 234 decodes the encoded 2D video stream to generate a 2D video stream, and the 2D video stream is played by a player.

The event controller 236 generates control information for controlling content according to a control command input through the user interface and transmits the control information to the server 100 through the request controller 201, and the server 100 sends the control information to the control information. The 3D image data controlled according to the method is rendered, and the rendered result is transmitted to the user terminal 200 again in a streaming manner. Here, the control information may include information related to capture stream and scene control for streaming 3D video to 2D video.

5 is a flowchart illustrating a request processing process of the user terminal shown in FIG. 4 .

Referring to FIG. 5 , first, in step 501, the user terminal 200 downloads (receives) a 3D video description file desired by the user from the server 100 through the response controller 202.

Next, in step 502, the description file parser 210 in the user terminal 200 parses the downloaded (received) 3D video description file to obtain a requestable content type, media type, content capacity, access path (URL), etc. is confirmed, and the user selects content suitable for the user terminal environment (device environment) using the user interface. Thereafter, the user interface generates a request message requesting the selected content and transmits it to the server 100 through the request controller 201 .

Subsequently, in step 503, the user terminal 200 determines whether it can perform (support) 3D image rendering (determines whether it has a 3D image rendering function or a renderer). If 3D image rendering is possible, the process proceeds to step 504; otherwise, it proceeds to step 511.

First, in step 511, if 3D video rendering is impossible, the request controller 201 in the user terminal 200 transmits a request message requesting a 2D video stream to the server 100.

Next, in step 512, the response controller 202 in the user terminal 200 receives the 2D video stream packet from the server 100.

Next, in step 513, the stream buffer controller 205 in the user terminal 200 processes the 2D video stream packet to generate a 2D video stream, and then the decoder 234 in the user terminal 200 processes the generated 2D video stream. Decode the stream.

Subsequently, in step S508, the player in the user terminal 200 reproduces the decoded 2D video stream.

Meanwhile, if the user terminal 200 is capable of rendering 3D images, the process proceeds from step 503 to step 504, and in step 504, it is determined whether the content capacity desired by the user is low capacity contents or high capacity contents. judge Content capacity can be confirmed from list information acquired by parsing the description file.

If the content capacity is low-capacity content, the process proceeds from step 504 to step 505, and in step 505, the user creates a request message requesting 3D image data using the user interface, and sends it to the server (through the request controller 201) 100).

Next, in step 506, the response controller 202 in the user terminal 200 receives the 3D video data packet from the server 100, and the 3D video data buffer controller 204 processes the 3D video data packet to generate 3D video data. generate

Next, in step 507, the renderer 232 in the user terminal 200 renders 3D image data.

Next, in step 508, the player in the user terminal 200 reproduces the rendered 3D video data as 3D video content.

Meanwhile, in step 503, even if the user terminal 200 is capable of rendering 3D video, a 2D video stream type may be requested instead of 3D video data according to content capacity and available network capacity of the current device. That is, when the capacity of the 3D video content desired by the user is high-capacity content, the process proceeds from step 504 to step 511, and steps 512 and 513 are sequentially performed.

On the other hand, after step 508, if the user requests control of the content through the user interface during playback of 3D image content in step 509, the user terminal can directly control and render the image being played.

When a user controls content during playback of a 2D video stream, the corresponding control information is transmitted to the server 100, and the user terminal 200 receives and reproduces the rendered result from the server 100.

Each step included in the processing methods according to the embodiment of the present invention described above may be implemented as a program executed by a computer. These programs may include codes coded in computer languages such as C, C++, JAVA, Ruby, and machine language that can be read by a computer's processor (CPU, GPU, MCU, etc.) through a device interface of the computer.

These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do.

In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is.

In addition, if the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code is how to communicate with any other remote computer or server using the communication module of the computer. It may further include communication-related codes for what information or media should be transmitted/received during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected by a network, and computer readable codes may be stored in a distributed manner.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or the components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (12)

A method of providing a three-dimensional (3D) image service in a server that performs socket communication with a user's user terminal,
transmitting a description file to the user terminal;
Receiving a request message requesting content selected by the user based on the description file from the user terminal;
generating 3D video content or 2D video stream content according to the request message; and
transmitting the generated 3D video content or 2D video stream content to the user terminal;
Method for providing a 3D image service comprising a. In paragraph 1,
The description file,
The method for providing a 3D video service, which is file information specifying information related to a content type, media type, content capacity, and access path (URL) serviceable by the server based on XML. In paragraph 1,
The generating step is
checking whether a user request is a 3D video content request or a 2D video stream content request according to the request message;
generating the 3D video contents when the user request is a request for 3D video contents; and
generating the 2D video stream contents when the user request is the 2D video stream contents request;
Method for providing a 3D image service comprising a. In paragraph 3,
The step of generating the 3D video content,
obtaining 3D image data corresponding to the 3D image content from a 3D image database; and
generating a 3D image data packet as the 3D image contents by processing the acquired 3D image data;
Method for providing a 3D image service comprising a. In paragraph 3,
The step of generating the 2D video stream contents,
obtaining 3D image data corresponding to the 3D image content from a 3D image database;
rendering the 3D image data;
generating a 2D video stream by encoding the rendered 3D video data; and
generating a 2D video stream packet generated by processing the generated 2D video stream as the 2D video stream contents;
Method for providing a 3D image service comprising a. A method of providing a three-dimensional (3D) image service in a user terminal performing socket communication with a server,
Downloading a description file from a server;
transmitting a request message requesting content selected based on the description file to the server;
receiving 3D video data or 2D video stream from the server according to the request message; and
Reproducing the 3D video data or 2D video stream
Method for providing a 3D image service comprising a. In paragraph 6,
The description file,
A method for providing a 3D video service that is XML-based specified file information related to content type, media type, content capacity, and access path (URL). In paragraph 6,
Sending the request message to the server,
transmitting the request message for requesting 3D image data to the server when content selected based on the description file is low-capacity content; and
Transmitting the request message requesting a 2D video stream to the server when the content selected based on the description file is high-capacity content
Method for providing a 3D image service comprising a. In paragraph 6,
Sending the request message to the server,
transmitting the request message requesting 3D image data to the server when the user terminal supports 3D image rendering and the content selected based on the description file is low-capacity content; and
Transmitting the request message requesting a 2D video stream to the server when the user terminal supports the 3D video rendering and the content selected based on the description file is high-capacity content.
Method for providing a 3D image service comprising a. In paragraph 6,
Sending the request message to the server,
and transmitting the request message requesting a 2D video stream to the server when the user terminal does not support 3D video rendering. In paragraph 6,
In the regeneration step,
rendering the 3D image data; and
Reproducing the rendered 3D image data
Method for providing a 3D image service comprising a. In paragraph 6,
In the regeneration step,
decoding the 2D video stream; and
Reproducing the decoded 2D video stream
Method for providing a 3D image service comprising a.

Images