KR102537163B1 - Terminal, cloud server, system and method for transreceive video on cloud augmented reality platform - Google Patents

Abstract

The present invention provides a terminal for transmitting network status and image data; and a cloud AR platform including a cloud server, wherein the cloud server receives the network state and the image data, and restores the image data based on the received network state. Provides an image transmission and reception system of

Description

Video transmission and reception system of terminal, cloud server and cloud augmented reality (AR) platform and its method

The present invention relates to a system for transmitting and receiving a video of a cloud AR platform and a method thereof, and more particularly, the present invention provides a system and method for transmitting and receiving a video of a cloud AR platform including a terminal, a cloud server, and a cloud server.

Augmented reality (AR) is a field of virtual reality (VR) and is a computer graphics technique that synthesizes virtual objects or information in a real environment to make them look like objects in the original environment. Recently, various devices to which AR platforms to which such an augmented reality technique is applied are being developed. However, there was a problem that the AR platform could not be used in low-end devices due to the difference in performance of the GPU (graphics processing unit) of the device.

In order to provide this AR platform to consumers in real time regardless of hardware specifications of the device, the AR platform may be provided based on a cloud server rather than a fixed server. Accordingly, from the user's point of view, even if they do not have a high-end device, they can easily enjoy the AR platform using the cloud service.

However, since the AR images provided by the AR platform through the cloud server are provided in a fairly large capacity to implement augmented reality, a smooth network condition is required to use the cloud AR platform in the terminal.

At this time, if the network condition is not good, delay time inevitably occurs when transmitting and receiving large-capacity AR images in the terminal, and it is difficult to use the cloud AR platform.

The present invention relates to a video transmission/reception system and method of a cloud AR platform, and an object of the present invention is to reduce network dependence or load when video is transmitted from a video input device (eg, mobile terminal) to a server.

In addition, an object of the present invention is to provide optimization of latency when transmitting and receiving images on a cloud AR platform.

According to one aspect of the present invention, a terminal for transmitting network status and image data; and a cloud AR platform including a cloud server, wherein the cloud server receives the network state and the image data, and restores the image data based on the received network state. Provides an image transmission and reception system of

In addition, the cloud server restores the image data using a deep-learning module, and the deep-learning module is characterized in that it performs at least one function of increasing resolution or restoring color.

In addition, the deep learning module may set the reconstructed image data as input data and set the image data before restoration as target data to learn a deep learning model.

In addition, the cloud AR platform further includes a session server, and the cloud AR platform transmits the network state and the image data to the cloud server through the session server.

In addition, the video transmission/reception system of the cloud AR platform further includes a set-top box (STB), and the cloud AR platform transmits the restored video data to the set-top box.

Also, the set-top box is characterized in that it decodes the received video data and reproduces the decoded video data on a display device.

According to one aspect of the present invention, an Android container running an AR application; at least one incoding container; and a container manager that manages the Android container and the incoding container, wherein the incoding container decodes video data, restores the decoded video data, and stores the restored video data and the data received from the AR application. A cloud server is provided that synthesizes AR content and encodes the synthesized video.

In addition, the incoding container restores the image data using a deep-learning module, and the deep-learning module is characterized in that it performs at least one function of increasing resolution or restoring color.

In addition, the deep learning module may set the reconstructed image data as input data and set the image data before restoration as target data to learn a deep learning model.

In addition, the container manager is characterized in that it executes a first indy-coding container including a preset deep learning module based on the AR application.

Also, the image data is characterized in that it is received from a server different from the cloud server.

According to one aspect of the present invention, a camera for obtaining an image; communications department; and a control unit including a network check module and an image conversion module, wherein the control unit checks a network state received from the communication unit through the network check module, converts the acquired image through the image conversion module, It provides a terminal, characterized in that for transmitting image data corresponding to the confirmed network state and the converted image to a cloud AR platform.

In addition, the control unit may transmit the network status and the video data to the cloud AR platform as a service provided by the cloud AR platform starts.

In addition, the control unit is characterized in that the image is converted based on the network state.

In addition, the control unit is characterized in that it transmits the performance information of the camera to the cloud AR platform.

In addition, when the performance of the camera is equal to or higher than a predetermined level, the control unit converts the image data into a first image quality based on the network condition and transmits the converted image data to the cloud AR platform.

In addition, when the performance of the camera is lower than the predetermined level, the control unit converts the image data into a second quality and transmits the image data to the cloud AR platform.

In addition, when the performance of the camera is equal to or less than the predetermined level, the control unit converts the image data converted to the second quality into grayscale and transmits the converted image data to the cloud AR platform.

According to one aspect of the present invention, receiving network status and image data from a terminal; and restoring the image data using a deep-learning module based on the received network state, wherein the deep-learning module performs at least one function of increasing resolution or restoring color. Characteristically, it provides a method for transmitting and receiving images of a cloud AR platform.

According to an embodiment of the present invention, there is an advantage that the cloud AR platform can be used even with low network quality.

In addition, according to an embodiment of the present invention, service usability can be increased by improving the latency of the cloud AR platform.

In addition, according to an embodiment of the present invention, since the super resolution technique is processed in the server, there is an advantage that the performance of the camera or GPU of the terminal is not affected.

1 is a diagram explaining the configuration of a video transmission/reception system of a cloud AR platform according to an embodiment of the present invention.
2 is a flowchart illustrating a video transmission/reception system of a cloud AR platform according to an embodiment of the present invention.
3 is a diagram for explaining the configuration of a terminal according to an embodiment of the present invention.
4 is a diagram illustrating the configuration of a session server according to an embodiment of the present invention.
5 is a diagram illustrating a configuration of a cloud server according to an embodiment of the present invention.
6 is a flowchart illustrating a cloud server according to an embodiment of the present invention.
7 is a flowchart illustrating a video transmission/reception method of a cloud AR platform according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves.

In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another. It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be.

On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The following bit rate is required to stream the camera image obtained from the current terminal.

category 15 fps 30 fps 1080p 4 Mbps 6 Mbps 720p 2.5 Mbps 3.5 Mbps 480p 1.2 Mbps 2 Mbps

In other words, because a bandwidth of 6 Mbps is required for 1080p (30 fps), if the network condition of the terminal is not good, a smooth cloud AR platform cannot be used. Accordingly, if the network conditions are not good, delay time occurs when transmitting and receiving large-capacity AR images in the terminal, which makes it difficult to provide services of the cloud AR platform.

The present invention is to solve the above problems, and the present invention will be described in detail with reference to FIGS. 1 to 7 below.

1 is a diagram explaining the configuration of a video transmission/reception system of a cloud AR platform according to an embodiment of the present invention.

Referring to FIG. 1 , a video transmission/reception system 100 of a cloud AR platform according to an embodiment of the present invention may include a terminal 110, a cloud AR platform 120, and a set-top box 130 (hereinafter referred to as STB). there is. The components shown in FIG. 1 are not essential to implement the video transmission and reception system 100 of the cloud AR platform, so the system described in this specification may have more or fewer components than the components listed above. can

Here, the terminal 110 may include a video transmission application 111, a communication unit 112, and a camera 113. According to an embodiment of the present invention, the terminal 110 may transmit network status and image data.

To this end, the video transmission application 111 may include a network check module 111a and an image conversion module 111b. At this time, the video transmission application 111 is not shown in the figure, but may be controlled by the control unit of the terminal 110.

The network check module 111a may check the network status received from the communication unit 112 . At this time, the network check module 111a may check the network status at preset time intervals. In addition, in one embodiment of the present invention, the network check module 111a may check the network state of the terminal 110 and transmit it to the image conversion module 111b.

The image conversion module 111b may convert an image acquired through the camera 113 . To this end, the image conversion module 111b may receive an image acquired through the camera 113 . At this time, the image conversion module 111b may convert the obtained image based on the network state. For example, when the performance of the camera 113 is equal to or higher than a predetermined level, image data may be converted into a first quality image based on a network condition. On the other hand, if the performance of the camera 113 is below a predetermined level, the image data may be converted into the second quality. In this case, the first image quality may correspond to a better image quality than the second image quality. In addition, in one embodiment of the present invention, the image conversion module 111b may convert the image data converted to the second quality into grayscale when the performance of the camera 113 is below a predetermined level.

The communication unit 112 may externally transmit image data corresponding to the confirmed network state and the converted image. The communication unit 112 may check the network status of the terminal 110 and transmit corresponding information to the network check module 111a.

The camera 113 may obtain an image. In one embodiment of the present invention, the terminal 110 may transmit network status and image data to the cloud AR platform 120 as a service provided by the cloud AR platform 120 starts.

This will be described in more detail in FIG. 3 .

In addition, the cloud AR platform 120 may include a session server 121 and a cloud server 122 .

Here, the session server 121 may include a network status checking module 121a and an image input module 121b. In one embodiment of the present invention, the cloud AR platform 120 may receive network status and video data from the terminal 110 through the session server 121, and the received network status and video data may be transmitted to the cloud server 122. ) can be transmitted.

In addition, the cloud server 122 may include an incoding container 1221, an Android container 1222, and a container manager 1223. In an embodiment of the present invention, the cloud server 122 may receive network status and image data from the terminal 110 . In this case, the cloud server 122 may receive the network status and video data through the session server 121 without directly receiving the network status and video data from the terminal 110 . Also, the cloud server 122 may restore image data based on the received network state.

The incoding container 1221 may include an encoding module 1221a, a deep learning module 1221b, and a decoding module 1221c. In addition, in FIG. 1, only one incoding container 1221 is shown, but the cloud server 122 may include a plurality of incoding containers.

The encoding module 1221a may synthesize the reconstructed image data and the AR content received from the AR application 1222a. Also, the encoding module 1221a may encode the synthesized video.

In one embodiment of the present invention, the cloud server 122 may restore image data using the deep learning module 1221b. In this case, the deep learning module 1221b may perform at least one function of increasing resolution or restoring color.

In addition, the deep learning module 1221b may learn a deep learning model by setting restored image data as input data and setting image data before restoration as target data.

The decoding module 1221c may decode image data.

The Android container 122 may include an AR application 1222a and an Android framework 1222b. In one embodiment of the invention, Android container 1222 may run AR application 1222a.

The container manager 1223 may manage the incoding container 1221 and the Android container 1222. In one embodiment of the present invention, the container manager 1223 may execute the incoding container 1221 including the preset deep learning module 1221b based on the AR application 1222a. As described above, the cloud server 122 may include a plurality of incoding containers 1221 . In this case, the container manager 1223 may execute the incoding container 1221 including an appropriate deep learning module 1221b based on the AR application 1222a.

This will be described in more detail with reference to FIGS. 4 and 5 .

In addition, the STB 130 may include a player application 131 and a decoding module 132 . In one embodiment of the present invention, the cloud AR platform 120 may transmit restored video data to the STB 130.

The STB 130 may decode the received image data through the decoding module 132 . Also, the STB 130 may reproduce the decoded video data on a display device through the player application 131 .

2 is a flowchart illustrating a video transmission/reception system of a cloud AR platform according to an embodiment of the present invention.

Referring to FIG. 2 , in step S210, the terminal may transmit network status and image data of the terminal to the session server. At this time, the terminal may transmit network status and video data to the session server of the cloud AR platform as the terminal and the cloud AR platform are connected. For example, as a service provided by the cloud AR platform starts, the terminal may transmit network status and image data to the session server of the cloud AR platform. Also, the image data may correspond to data converted based on the terminal's network state. In addition, inside the terminal, the network status may be checked through the communication unit (network transmission/reception unit) of the terminal, and the checked network status may be transmitted to the network check module. In addition, the camera of the terminal may obtain an image and transmit the acquired image to the image conversion module. Accordingly, the terminal may transmit the network status to the network status checking module of the session server through the network check module, and transmit video data to the video input module of the session server through the video conversion module.

In step S220, the session server may transmit the network status and image data received from the terminal to the cloud server. More specifically, the session server may transmit the network status received from the terminal through the network status check module to the decoding module of the cloud server, and transmit the video data received from the terminal through the video input module to the decoding module of the cloud server. there is.

In step S230, the cloud server may restore image data based on the received network state. At this time, the cloud server may restore image data using a deep learning module. The deep learning module may perform at least one function of increasing resolution or restoring color to restore image data. In addition, the deep learning module may learn a deep learning model by setting reconstructed image data as input data and setting unreconstructed image data as target data. In addition, although not shown in the figure, inside the cloud server prior to step S230, the container manager may execute an indy coding container equipped with a deep learning module suitable for AR service. This will be described once more in FIG. 6 .

In step S240, the cloud server may transmit the encoded AR synthesized video to the decoding module of the STB. Thereafter, the decoding module of the STB may decode the received AR synthesized video and reproduce the decoded video data on the display device. For example, the set-top box can play the received AR synthesized video on a TV.

3 is a diagram for explaining the configuration of a terminal according to an embodiment of the present invention.

Referring to FIG. 3 , a terminal 300 may include a video transmission application 310 , a communication unit 320 and a camera 330 . In addition, the components shown in FIG. 3 are not essential to implement the terminal, so the terminal described in this specification may have more or fewer components than the components listed above.

The video transmission application 310 may include a network check module 311 and an image conversion module 312 . Also, the video transmission application 310 may be controlled by the control unit of the terminal 300 .

The network check module 311 may check the network status received through the communication unit 320 . The network check module 311 may transfer the received information on the network state to the image conversion module 312 . In addition, the network check module 311 may transmit information about the network status to the network status check module of the session server.

The image conversion module 312 may convert an image acquired through the camera 330 . At this time, the image conversion module 312 may convert the resolution of the obtained image and perform color conversion. In particular, the image conversion module 312 may transform the acquired image based on the identified network state. At this time, the image conversion module 312 may determine a resolution to be converted based on the network state information received from the network check module 311 . For example, the image conversion module 312 may convert a video to 1080p when the network status is “high”, and convert an image to 720p when the network status is “medium”. In the case of “bottom”, the video can be converted to 480p. Also, the image conversion module 312 may convert an RGB format image into a grayscale format image. Since 480p and 30 fps grayscale images use 2 Mbps of network bandwidth, and 1080p and 30 fps RGB images use 6 Mbps network bandwidth, grayscale images occupy 1/9 of the network bandwidth compared to RGB images. (This is because grayscale image data requires only 1-channel data, but RGB image data requires 3-channel data.) Accordingly, when image data is converted to grayscale image and transmitted to the cloud AR platform, network use is avoided. can be optimized. Also, the video conversion module 312 may transmit the converted video data to the video input module of the session server.

The communication unit 320 may include one or more modules enabling wireless communication between the terminal 300 and a wireless communication system, between the terminal 300 and other terminals, or between the terminal 300 and an external server. Also, the communication unit 320 may include one or more modules that connect the terminal 300 to one or more networks. To this end, the communication unit 320 may include at least one of a broadcast reception module, a mobile communication module, a wireless Internet module, a short-distance communication module, and a location information module, but this may be included in the communication unit 320 of the general terminal 300. As a module, this description will be omitted. In one embodiment of the present invention, the communication unit 320 may transmit and receive the network status of the terminal 300 and report the network status to the network check module 311 . At this time, the communication unit 320 may classify the state of the network into high/medium/low. Of course, this is just an example, and the communication unit 320 can be further subdivided according to design. In addition, the communication unit 320 may periodically check the network status when using the service of the cloud AR platform. The terminal 300 may transfer the network status received from the communication unit 320 to the network check module 311 . Then, the communication unit 320 may transmit the network status and converted image data to the cloud AR platform under the control of the controller. Also, the communication unit 320 may transmit spec information of the camera 330 to the cloud AR platform.

The camera 330 may process an image frame such as a still image or a moving image obtained by an image sensor. In one embodiment of the present invention, a camera may obtain an image. Here, the image includes still images (image) or moving images (video) and is not limited to any one. Also, the processed image frame may be displayed on a display (not shown) or stored in a memory (not shown). That is, in one embodiment of the present invention, the acquired image may include an image stored in a memory. In one embodiment of the present invention, the terminal 300 sets the quality of image data transmitted from the terminal 300 to 1 of 1080p/720p/480p based on a network condition when the performance of the camera 330 is equal to or higher than a preset level. It can be selected and transmitted to the cloud AR platform server at 15 fps. On the other hand, when the performance of the camera 330 is below a predetermined level, the terminal 300 may fix the quality of image data to 480p and 15 fps and transmit the same to the cloud AR platform server. In addition, the terminal 300 may transmit the quality of image data in grayscale. In this case, there is an effect of reducing the network bandwidth by 1/3 compared to the existing RGB image. Thereafter, the cloud AR platform server may retransmit the received grayscale image data to the terminal 300 after restoring the received grayscale image data to 1080p and 30 fps including RGB colors by utilizing a pre-learned deep learning model (eg, SRGAN model). This will be described in detail in the following drawings.

Accordingly, the terminal 300 periodically checks the network state of the terminal 300 to transmit a low-resolution video in a situation where the network quality is low, and transmit a high-resolution video in a situation where the network quality is high.

4 is a diagram illustrating the configuration of a session server according to an embodiment of the present invention.

Referring to FIG. 4 , the session server 400 may include a network status check module 410 and an image input module 420 .

The network status check module 410 may receive information (data) about the network status of the terminal from the network check module of the terminal. Thereafter, the network status checking module 410 may transmit the received information on the network status to the decoding module of the cloud server.

The image input module 420 may receive converted image data from the terminal from the image conversion module of the terminal. Then, the image input module 420 may transmit the converted image data to the decoding module of the cloud server.

That is, the session server 400 plays a role of transferring the video received from the terminal and information on the network state to the cloud server.

5 is a diagram illustrating a configuration of a cloud server according to an embodiment of the present invention.

Referring to FIG. 5 , the cloud server 500 may include an incoding container 510, an Android container 520, and a container manager 530.

The incoding container 510 may include an encoding module 511 , a deep learning module 512 , and a decoding module 513 .

The encoding module 511 may encode AR content and image data. More specifically, the encoding module 511 may encode AR content received from the AR application 521 and image data synthesized. In addition, the encoding module 511 may transmit encoded AR content and synthesized image data to the decoding module of the player application of the STB.

The deep learning module 512 may increase resolution and restore color of the decoded image data through deep learning. More specifically, the deep learning module 512 may restore low resolution image data transmitted from the session server to high resolution image data in real time by using a deep learning model (eg, SRGAN). For example, the deep learning module 512 may restore the received image data to 1080p and 30fps RGB image data by using the SRGAN model based on the received network state information. In addition, the deep learning module 512 may use a deep learning model trained differently for each AR application 521 . For example, in the case of service A, if there is a scenario in which parents film their children at home through a camera and combine them with AR content (dinosaur, animal, etc.), the deep learning model A photo data set with . can be used as training data. In addition, the deep learning module 512 may restore RGB color of image data converted to grayscale. That is, low-resolution grayscale image data may be restored to high-resolution RGB image data through the deep learning module 512 . In one embodiment of the present invention, the restored image data may be transmitted to the AR application 521 operating in the cloud server. In addition, the deep learning module 512 may learn a deep learning model by setting restored image data as input data and setting image data before restoration as target data.

The decoding module 513 may decode video data received from the session server. At this time, the decoding module 513 may transmit the decoded image data to the deep learning module 512 together with network state information.

The Android container 520 may include an AR application 521 and an Android framework 522. That is, the Android container 520 may drive the AR application 521.

The AR application 521 may synthesize the AR content and the received image data and transmit the result to the encoding module 511 .

The container manager 530 may correspond to a manager that manages execution or suspension of each container. In one embodiment of the present invention, the container manager 530 may execute an indie coding container loaded with a pre-learned deep learning model based on the service of an AR application running in the Android container 520 . For example, the container manager 530 executes an A' indie coding container including an A1 deep learning module optimized for service A and executes an indie coding container B' including a B1 deep learning module optimized for service B. can In addition, the deep learning models used in the A1 deep learning module and the B1 deep learning module can learn scenarios in advance to be optimized for the scenario of the service.

In addition, various deep learning models may be applied for super-resolution conversion. That is, various other deep learning models may be used for resolution conversion in addition to SRGAN as an example described above. For example, various deep learning models such as SRCNN, VDSR, EDSR, and MDSR may be used. The reason why various deep learning models are used is that their performance varies significantly depending on the training data. Accordingly, in one embodiment of the present invention, the performance of deep learning may be improved by using a deep learning model optimized for a service scenario of an AR application running in the Android container 520. For example, in the case of a specific AR application, there is a scenario in which AR contents such as dinosaurs and animals are synthesized and displayed when a parent shines on a child. In this case, the deep learning model may be trained by configuring an image data set of a category in which a person is indoors as training data. In addition, the deep learning module may learn a deep learning model by setting reconstructed image data as input data and setting unreconstructed image data as target data. More specifically, the deep learning module configures a training data set by setting the original image data of the selected category of data set as the target data and the image data converted to grayscale after lowering the resolution as the input data to train the deep learning model. can Also, as another embodiment, the deep learning module may learn by utilizing a video data set for each category provided by a third party. (Reference: https://research.google.com/youtube8m )

In addition, the above-described SRGAN model is briefly described as follows.

The GAN model is composed of a generator and a discriminator. The generator generates data as similar as possible to the actual input data, and the discriminator aims to distinguish generated data from actual sample data as much as possible. . In addition, the existing GAN model was able to obtain a high peak signal to noise ration (PSNR) by setting the mean squared error (MSE) as the loss function and proceeding with optimization. There was a problem that I couldn't express well.

To overcome this, the SRGAN model is trained by defining a perceptual loss function instead of PSNR. That is, the SRGAN model has a lower PSNR value than the existing GAN model, but records a high mean opinion score (MOS), so that a more sophisticated image can be created in human terms.

An embodiment of the present invention can use such an SRGAN model in a deep learning module to minimize network dependence and latency of the cloud AR platform.

6 is a flowchart illustrating a cloud server according to an embodiment of the present invention. 6 describes in detail various embodiments implemented by a container manager, an encoding container, and an Android container in a cloud server.

Referring to FIG. 6 , in step S610, the container manager may execute a first incoding container including a first deep learning module based on the AR application. That is, the cloud server may include at least one incoding container, and the container manager may execute an incoding container including an appropriate deep learning module in consideration of the AR content to be synthesized.

In step S620, the executed incoding container may decode the received image data.

In step S630, the incoding container may restore image data based on the received network state. At this time, the indycoding container may restore image data using a deep learning module. The deep learning module may perform at least one function of increasing resolution or restoring color to restore image data. In addition, the deep learning module may learn a deep learning model by setting reconstructed image data as input data and setting unreconstructed image data as target data. This is as described above.

In step S640, the incoding container may transmit restored image data to the Android container. For example, the incoding container can transmit RGB image data of 1080p and 30 fps (high resolution) to the AR application of the Android container.

In step S650, the Android container may transmit an image obtained by synthesizing the restored image data and AR content to the incoding container. More specifically, the AR application of the Android container may transmit an image in which the restored image data and AR contents are synthesized to the encoding module of the incoding container.

In step S660, the incoding container may encode the reconstructed video data and AR content. The encoding module of the incoding container may encode the received video data and AR content, and then transmit the encoded AR synthesized video data to the decoding module of the STB.

7 is a flowchart illustrating a video transmission/reception method of a cloud AR platform according to an embodiment of the present invention.

Referring to FIG. 7 , in step S710, the cloud AR platform may receive network status and image data from the terminal. The cloud AR platform may receive a network state from the network check module of the terminal, and image data converted to at least one of low resolution and grayscale from the image conversion module.

In step S720, the cloud AR platform may restore image data based on the received network state. At this time, the cloud AR platform may restore image data through an appropriate deep learning module in consideration of the AR content to be synthesized. A method of restoring an image through the deep learning module is the same as in the above-described embodiment. Thereafter, the cloud AR platform may synthesize the restored image data and AR contents and transmit the result to the STB.

In addition, it goes without saying that the embodiments described through the above drawings can be implemented as a method as shown in FIG. 7 .

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. , and also includes those implemented in the form of a carrier wave (eg, transmission over the Internet).

Although the video transmission/reception system and method of the terminal, cloud server, and cloud AR platform according to the embodiment of the present invention have been described as specific embodiments, this is only an example and the present invention is not limited thereto, and the basic idea disclosed herein should be construed as having the widest scope according to A person skilled in the art may implement an embodiment that is not indicated by combining or substituting the disclosed embodiments, but this also does not deviate from the scope of the present invention. In addition, those skilled in the art can easily change or modify the disclosed embodiments based on this specification, and it is clear that such changes or modifications also fall within the scope of the present invention.

100: Video transmission and reception system of cloud AR platform
110: terminal
120: Cloud AR platform
130: STB

Claims (19)

a terminal that transmits network status and image data; and
Including a cloud AR platform including a cloud server,
The cloud server
An Android container running an AR application;
at least one incoding container; and
Including a container manager that manages the Android container and the indie coding container,
The container manager executes the incoding container based on the AR application,
Receiving the network state and the image data from the terminal;
The indicoding container is
Restoring the image data based on the received network state;
The cloud server restores the image data using a deep-learning module, and the deep-learning module performs a color restoration function. According to claim 1,
The deep learning module is a video transmission and reception system of a cloud AR platform, characterized in that it performs a resolution increase function. According to claim 2,
The deep learning module sets the restored image data as input data and sets the image data before restoration as target data to learn a deep learning model. According to claim 1,
The cloud AR platform further includes a session server,
The cloud AR platform, characterized in that for transmitting the network state and the image data to the cloud server through the session server, video transmission and reception system of the cloud AR platform. According to claim 1,
The video transmission and reception system of the cloud AR platform further includes a set-top box (STB),
The cloud AR platform transmits and receives the image data of the cloud AR platform, characterized in that for transmitting the restored image data to the set-top box. According to claim 5,
The set top box
decoding the received video data;
A video transmission and reception system of a cloud AR platform, characterized in that the decoded video data is reproduced on a display device. An Android container running an AR application;
at least one incoding container; and
A container manager for managing the Android container and the indie coding container;
The container manager
Execute the incoding container based on the AR application;
The indicoding container is
decoding the video data;
Restoring the decoded video data;
The indie-coding container restores the image data using a deep-learning module included in the indi-coding container, and the deep-learning module performs a color restoration function;
synthesizing the restored image data and AR contents received from the AR application;
Characterized in that for encoding the synthesized video, the cloud server. According to claim 7,
Characterized in that the deep learning module performs a resolution increase function, the cloud server. According to claim 8,
The cloud server, characterized in that the deep learning module learns a deep learning model by setting the restored image data as input data and setting the image data before restoration as target data. According to claim 7,
Characterized in that, the container manager executes a first indy-coding container including a preset deep learning module based on the AR application. According to claim 7,
Characterized in that the image data is received from a server different from the cloud server, the cloud server. a camera that acquires an image;
communications department; and
Including a control unit including a network check module and an image conversion module,
The control unit
Checking the network status received from the communication unit through the network check module;
Transmitting the performance information of the camera to the cloud AR platform,
Converting the acquired image through the image conversion module;
Transmitting image data corresponding to the confirmed network state and the converted image to the cloud AR platform,
Characterized in that, when the performance of the camera is below a predetermined level, the image data is converted to grayscale and transmitted to the cloud AR platform. According to claim 12,
The terminal, characterized in that the control unit transmits the network status and the image data to the cloud AR platform as the service provided by the cloud AR platform starts. According to claim 12,
Characterized in that the control unit converts the image based on the network state, the terminal. delete According to claim 12,
The terminal, characterized in that, when the performance of the camera is equal to or higher than a predetermined level, the control unit converts the video data into a first quality image based on the network condition and transmits the image data to the cloud AR platform. 17. The method of claim 16,
The terminal, characterized in that, when the performance of the camera is equal to or less than the predetermined level, the control unit converts the image data into a second quality and transmits the converted image data to the cloud AR platform. delete In the video system of the cloud AR platform including the cloud server,
Executing an AR application through an Android container included in the cloud server;
Executing an indie coding container based on the AR application through a container manager included in the cloud server and managing the indie coding container;
Receiving network status and image data from a terminal; and
Restoring the image data using a deep-learning module based on the received network state, characterized in that the deep-learning module performs a function of increasing resolution and restoring color. AR platform video transmission and reception method.

Images