KR101779527B1 - Web server for processing 360° VR video having high resolution and method thereof - Google Patents

Abstract

Disclosed are a web server and a video processing method for high-resolution 360 virtual reality video processing. The web server according to one embodiment includes a rendering unit for rendering an original image of a 360 virtual reality video, a video player for playing the rendered 360 virtual reality video, an encoding unit for encoding a part of a 360 virtual reality video screen played through the video player, and a transmitting unit for transmitting a part of the encoded area to a client terminal. Accordingly, the present invention can maintain image quality at an ultra-high resolution level.

Description

Technical Field [0001] The present invention relates to a web server for processing high-resolution 360 ° virtual reality video,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique, and more particularly to an image quality control and transmission technique applied to a virtual reality.

A 360 ° video is a video captured by looking at 360 ° around the camera, or a video that gives such a feeling. Since it includes side and back image information more than general video, it can be applied to various application fields. In particular, 360 ° motion pictures can be applied to virtual reality. Users can enjoy realistic and immersive virtual reality through 360 ° virtual reality video.

Based on the same resolution, 360 ° VR movie files are much larger than regular video files. For example, a 360 ° VR movie file at HD resolution (1280 × 720p) is three to four times larger than a regular movie file.

On the side of the display for playing a moving picture, in order to display a 360 ° moving picture on one screen, it is necessary to have a display device capable of displaying a 360 ° moving picture at once, which is constituted by a cylindrical shape, There is a restriction that a display device having a very large size so as to be displayed on the image is provided or that the size of the moving image is reduced.

According to one embodiment, a web server and a video processing method for high resolution 360 ° virtual reality video processing are proposed.

The web server according to one embodiment includes a rendering unit for rendering an original image of a 360 ° virtual reality video, a video player for playing the rendered 360 ° virtual reality video, and a 360 ° virtual reality video output An encoding unit encoding an area on the screen, and a transmission unit transmitting the encoded part of the area to the client terminal. Some of the areas may be the center section or the section requested by the client terminal among the moving image output screens divided into a plurality of sections.

The rendering unit may convert the resolution of the original image of the 360 ° virtual reality video into a first resolution and render the first resolution may be higher or equal to the resolution of the original image of the 360 ° virtual reality video. The encoding unit converts the resolution of a partial area of the 360 ° virtual reality video output screen into a second resolution and encodes the partial resolution, and the transmitting unit transmits a partial area having the second resolution to the client terminal. Resolution than the original image.

The rendering unit can render the original image of a 360 ° virtual reality video using GPU or hardware acceleration. The encoding unit can encode some areas of the video output screen of the video player using the GPU or hardware acceleration.

The transmitting unit may transmit the encoded partial region to the client terminal in a cloud streaming manner.

The web server further includes a receiver for receiving the terminal resolution information from the client terminal. The encoding unit converts and encodes the resolution of a part of the video output screen of the video player according to the terminal resolution using the terminal resolution information of the client terminal, The transmitting unit may transmit the encoded partial area to the client terminal.

According to an exemplary embodiment, problems that may occur when processing a 360 ° VR moving image at a client terminal, for example, inefficient video streaming or download transmission usage problem, image degradation problem, problems of a client terminal demanding high performance And so on. At this time, the 360 ° VR video streaming or downloading transmission usage is the resolution level corresponding to the client terminal (which is also high resolution), but the image quality quality can be maintained at the ultra high resolution level.

FIG. 1 is a reference view showing a 360 ° virtual reality video output screen for facilitating understanding of the present invention;
FIG. 2 is a reference view showing a 360 ° moving image output screen for explaining an example of selecting and displaying a part of a playback screen of a moving picture player according to an embodiment of the present invention;
FIG. 3 is a configuration diagram of a high-resolution 360 ° VR moving image processing system according to an embodiment of the present invention,
4 is a configuration diagram of a web server according to an embodiment of the present invention;
5 is a reference diagram illustrating a processing process of a web server for processing 360 DEG VR moving images according to an embodiment of the present invention;
FIG. 6 is a reference view showing a processing process of a web server using a GPU to enhance the performance of a web server providing 360 ° VR moving picture according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. , Which may vary depending on the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowcharts may be performed by computer program instructions (execution engines), which may be stored in a general-purpose computer, special purpose computer, or other processor of a programmable data processing apparatus The instructions that are executed through the processor of the computer or other programmable data processing equipment will generate means for performing the functions described in each block or flowchart of the block diagram.

These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of the flowchart.

And computer program instructions may be loaded onto a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the data processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or step may represent a portion of a module, segment, or code that includes one or more executable instructions for executing the specified logical functions, and in some alternative embodiments, It should be noted that functions may occur out of order. For example, two successive blocks or steps may actually be performed substantially concurrently, and it is also possible that the blocks or steps are performed in the reverse order of the function as needed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the following embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to enable those skilled in the art to more fully understand the present invention.

FIG. 1 is a reference view showing a 360 ° virtual reality (VR) video output screen for facilitating understanding of the present invention.

Referring to FIG. 1, a 360 ° VR video file has a much larger file size than a regular video file based on the same resolution. For example, a 360 ° VR movie file at HD resolution (1280 × 720p) is three to four times larger than a regular movie file. 360 ° VR videos provide augmented reality with 360 ° stereoscopic video.

FIG. 2 is a reference view showing a 360 ° moving image output screen for explaining an example of selecting and displaying a part of a playback screen of a moving picture player according to an embodiment of the present invention.

Generally, a general program such as a 360 ° VR video player is installed in the client terminal. In this case, the 360 ° VR video player of the client terminal embodies the 360 ° VR video content in a circle or the like as shown in FIG. 2 and divides it into equal parts, for example, nine equal parts, Only the center cross section 5 is displayed on the video player screen. 9 is one embodiment only, and it can be subdivided according to the resolution of the video player such as 18 equal parts.

The above-described 360 ° VR video processing method has the following problems. For example, a 360 ° VR video file at the same resolution, for example at HD resolution (1280 × 720p), is three to four times larger than a regular video file. Therefore, at the same resolution, 360 ° VR video uses 3 to 4 times more streaming or downloading transmission than ordinary video. The 360 ° VR video player, which displays 3 ~ 4 times more transmission than general video, actually displays the center section (5). When the client terminal resolution is HD resolution, 1/9 level (426 × 240) Of low resolution.

Assuming that the screen size of the client terminal executing the 360 ° VR video player is HD resolution (1280 × 720p), the section having a resolution of 1/9 level (420 × 240) is multiplied again to 3 times If it is displayed on the screen, the image quality deterioration necessarily occurs in the process.

On the other hand, the hardware requirements for the 360 ° VR video player to handle user input smoothly are also very high. Basically, 3D acceleration, 3D processing, and 3D rendering must be done with hardware for 360 ° VR processing.

Thus, the present invention implements a 360 ° VR video player on a web server rather than on a client terminal. Then, only a part of the executed screen, for example, the central section is transmitted to the client terminal. Accordingly, it is possible to solve all of the inefficient video streaming or download transmission usage problem, image quality degradation problem, and problem of the client terminal demanding high performance, which occurs in the process of processing the 360 ° VR moving image by the client terminal.

In order to improve the image quality of a 360 ° VR movie, the web server according to an exemplary embodiment of the present invention may convert a moving image file of HD quality into a high definition 360 ° (1080p), Ultra HD (4K) VR video, the video is reproduced inside the web server, and only a part of the area of the reproduced output screen is transmitted to the client terminal at a resolution of HD class or lower. Accordingly, the video streaming or downloading transmission usage amount is the resolution level (which is also high resolution) corresponding to the client terminal, but the image quality quality can be maintained at the ultra high resolution level. Hereinafter, a high-resolution 360 ° VR moving image processing technique having the above-described features will be described with reference to the following drawings.

3 is a block diagram of a high-resolution 360 ° VR moving image processing system according to an embodiment of the present invention.

3, the web server 2 and the client terminal 3 are connected via the network 1. [ In the web server 2, a high-resolution 360 ° VR movie is reproduced through a moving picture player executed and executed, and only a part of the reproduced moving picture is captured and encoded and transmitted to the client terminal 3.

Here, some of the regions may be a center cross-section or a cross-section requested by the client terminal 3 among the moving image output screens divided into a plurality of cross-sections. An example of the central section has been described above in Fig. As an example of the section requested by the client terminal 3, when the client terminal 3 selects the size and the cross section of the cross section for outputting the 360 ° VR movie in consideration of the terminal resolution and requests the web server 2, (2) transmits the cross section selected by the client terminal (3) to the client terminal (3) in accordance with the cross sectional size.

The web server 2 is connected to the client terminal 3 through the network 1 to transmit and receive all data based on the web. To this end, the client terminal 3 may request the web server 2 through a web client such as a web browser, receive the request result, and provide it to the user. The web server 2 and the client terminal 3 can perform bi-directional communication through a web-based networking method such as a web socket. In this case, the web server 1 can depart from the communication method of responding to the request of the client terminal 3.

The web server 2 may be a cloud streaming web server. In this case, the web server 2 provides a cloud streaming service to the client terminal 3. The cloud streaming service means that the software is executed in the web server 2 and the execution result is transmitted to the client terminal 3. At this time, the client terminal 3 transmits the user input again to the web server 2 and reflects it to the software of the web server 2. The software may be a program executable in the web server 2, such as a multimedia player, such as a video player. The client terminal 3 can receive the result of execution of the web server 2 rotor moving picture player and output the received execution result.

The client terminal 3 includes all the user terminals that the web client can install or download. The web client may be a web browser or a hybrid app. A hybrid app is an app that runs in an Internet browser. Hereinafter, for convenience of explanation, the web client is limited to a web browser, but the present invention is not limited thereto. The client terminal 3 may be various terminals such as an information communication device, a multimedia terminal, a wire terminal, a fixed terminal and an IP (Internet Protocol) terminal. The client terminal 3 may be a mobile terminal, a portable multimedia player (PMP), a mobile Internet device (MID), a smart phone, a desktop, a tablet PC, a notebook, book, and information communication devices, which have various mobile communication specifications.

The network 1 performs a transmission / reception operation for data transmission and information exchange between the web server 2 and the client terminal 3. The network 1 may use various types of communication networks such as a wireless LAN (WLAN), a Wi-Fi, a Wibro, a Wimax, a High Speed Downlink Packet Access (HSDPA) Or a wired communication method such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to the Curb) and FTTH (Fiber To The Home) . Meanwhile, the network 1 is not limited to the communication method described above, and may include any other known or later-developed communication methods in addition to the communication methods described above.

4 is a configuration diagram of a web server according to an embodiment of the present invention.

4, the web server 2 includes a video player 20, a processor 22, a transmitting unit 24, a receiving unit 26, and a memory 28. [

The video player 20 reproduces the 360 ° VR video content. The processor 22 executes various commands for service provision. The processor 22 executes the video player 20. The memory 28 may store data including instructions executed through the processor 22 and data generated by instruction execution. In the memory 28, a 360 ° VR movie file source to be reproduced through the moving picture player 20 may be stored. The processor 22 can transmit the reproduction screen of the moving picture player 20 to the client terminal through the transmission unit 24. [

The processor 22 according to one embodiment includes a rendering unit 220 and an encoding unit 222.

The rendering unit 220 renders an original image of the 360 ° VR moving picture through the moving picture player 20. The 360 ° VR movie rendered through the rendering unit 220 is reproduced through the video player 20. Rendering refers to the process of creating a three-dimensional image by injecting a sense of reality into a two-dimensional image in consideration of external information such as a light source, a position, and a color.

The rendering unit 220 converts the resolution of the original image of the 360 ° VR moving image into the first resolution and renders it. At this time, the first resolution is higher or equal to the resolution of the original image of the 360 ° VR moving image. For example, a moving image file of an HD quality image is reproduced through the moving image player 20 by increasing resolution to an ultra high resolution 360 ° VR moving image file such as Full HD, 4K, and 8K class.

The rendering unit 220 according to an exemplary embodiment may use an original image of a 360 ° VR movie using a graphics processing unit (GPU) or hardware (HW) Render. The way of using GPU can use surface rendering method using DirectX (DirectX) technology or rendering method using OpenGL library (OpenGL library). DirectX is the application interface you need to create and manage graphical images and multimedia effects. Surface rendering is the process of giving a three-dimensional texture to the surface of an object. The built-in functions that can be called through this API include features such as hidden face removal, transparency, anti-aliasing, texture mapping, pixel manipulation, modeling for transformation, and atmospheric effects (fog, smoke, haze, etc.).

The rendering unit 220 can maximize the available amount of the web server 2 by rendering a 360 ° VR movie using a plurality of GPUs at the same time. An embodiment of this will be described below with reference to Fig.

The encoding unit 222 captures and encodes a part of the 360 ° VR moving picture output screen reproduced through the video player 20. Some areas can be fixed in advance, for example, the center section. For this purpose, the encoding unit 222 embodies an output screen reproduced through the moving picture player 20 as a circle or the like as shown in FIG. 2, divides it into equal parts, for example, nine parts, do. 9 is an embodiment of the present invention, and can be subdivided according to the resolution of the video player 20, such as eighteen equal parts. The central section may be a screen at another location in one embodiment of the present invention. Also, it may be fixedly set in advance, but the user may select and encode a section of an angle desired by the user at the request of the client terminal.

The encoding unit 222 according to an embodiment converts the resolution of a part of the moving image output screen to a second resolution and encodes the resolution. The second resolution is less than or equal to the resolution of the original image of the 360 ° VR movie. For example, when a video source file of HD quality is targeted, an image of an ultra-high resolution 360 ° VR movie file such as Full HD, 4K, and 8K class is increased in resolution, Encode by lowering the resolution to HD class or lower. The transmitting unit 24 transmits the encoded region through the encoding unit 222 to the client terminal. Accordingly, the video streaming or downloading transmission usage amount is the resolution level (which is also high resolution) corresponding to the client terminal, but the image quality quality can be maintained at the ultra high resolution level.

The encoding unit 222 according to an exemplary embodiment encodes a portion of a playback screen of the moving picture player 20 using a GPU or HW acceleration. An embodiment of this will be described below with reference to Fig. The encoding unit 222 may separately encode the execution picture and audio reproduced through the moving picture player 20, respectively.

The receiving unit 26 may receive the content request from the client terminal and receive the user input. Also, the receiving unit 26 can receive the terminal resolution information from the client terminal. The encoding unit 222 encodes the resolution of a part of the output screen of the moving picture player 20 according to the resolution of the terminal using the terminal resolution information of the client terminal and the transmitting unit 24 encodes the encoded partial area To the client terminal.

5 is a reference diagram showing a processing process of a web server for processing 360 ° VR moving picture according to an embodiment of the present invention.

Referring to FIG. 5, a 360 ° VR movie is provided from the web server 2 to the client terminal 3. The 360 ° VR movie source file may be located in the memory of the Web server 2, for example, a local disk, and may be located in an accessible NAS (Network-Attached Storage) or a SAN (storage area network). The web server 2 executes the VR video player 20 to render a 360 ° VR movie original file. Rendering can be performed using the GPU using method or the HW acceleration method at the time of rendering. The way of using GPU can use surface rendering method using DirectX (DirectX) technology or rendering method using OpenGL library (OpenGL library).

The encoding unit 222 encodes the rendering screen reproduced by the 360 ° VR video player 20. The encoding unit 222 can encode only the central picture, especially the rendering picture. The encoding unit 222 may encode the captured image through the video codec using the GPU using method or the HW acceleration method. The way of using the GPU can use the encoding method using DirectX (DirectX) technology or the HW acceleration encoding method using the video codec. In HW acceleration encoding via GPU, the picture quality for service can be increased by using H.265 codec. In addition, the audio output reproduced by the 360 ° VR video player 20 is encoded by the audio codec. You can use the encoding method using DirectX (DirectX) technology or the HW acceleration encoding method using audio codec.

The encoded screen and audio are streamed or downloaded to the client 30 of the client terminal 3 via the network. Examples of the client 30 are a web browser, a hybrid app, or a simple player.

FIG. 6 is a reference view showing a process of a web server using a GPU to enhance the performance of a web server providing 360 ° VR moving picture according to an embodiment of the present invention.

Referring to FIG. 6, 360 ° VR video rendering and some area encoding are simultaneously processed by a plurality of GPUs, thereby reducing the CPU occupation rate throughout the 360 ° VR video processing and simultaneously processing 360 ° VR video processing processed by the web server 2 Increase capacity. A plurality of GPUs (GPU # 1, GPU # 2, ..., and GPU #n) are mounted on the web server 2 in one web server 2, GPU # n) at the same time, it is possible to maximize the available amount of the web server 2 by processing the 360 DEG VR video rendering and the partial region encoding.

The web server 2 transmits the 360 ° VR moving picture result screen to the client 30 of the client terminal 3 in a streaming or download transmission manner. In addition, the web server 2 outputs the 360 ° VR movie and transmits the audio to the client 30 of the client terminal 3 in a streaming or download transmission manner.

Gestures such as touch, tab, pinch, zoom, scroll, and tilt input from the client terminal 3 and input such as a mouse and a keyboard can be transmitted to the web server 2 by the client terminal 3. In this case, the web server 2 reflects the user input to the executed 360 ° VR video player.

The present invention converts an original image quality of a 360 ° VR movie file into an ultra high resolution image and reproduces it through a video player of the web server 2 in order to enhance the image quality of a 360 ° VR movie. For example, in the HD (1280 × 720p) class, the image quality is improved for the 360 ° VR video service by increasing the resolution by Full HD (1080p), Ultra HD (4K), and 8K class. On the other hand, when the image is provided to the client terminal 3, the image quality of the video player output screen having an ultra-high resolution, for example, an original image quality of 4K is lowered (this is also high resolution) Adjust transmission quality. Accordingly, both the image quality and the service quality for transmission can be increased. In addition, in the HW encoding through the GPU, the image quality for the service is increased by using the H.265 codec. As another example, the quality of service can be improved by transmitting the resolution of the client terminal to the 360 ° VR video player of the web server 2 to match the output screen of the 360 ° VR video player.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

1: Network 2: Web server
3: client terminal 20: video player
22: processor 24:
26: Receiving unit 28: Memory
220: rendering unit 222: encoding unit

Claims (8)

A rendering unit for rendering an original image of a 360 ° virtual reality video;
A video player that plays rendered 360 ° virtual reality video;
An encoding unit for encoding a part of a 360 ° virtual reality video output screen reproduced through the video player; And
A transmitting unit for transmitting the encoded partial region to the client terminal;
The web server comprising: The method according to claim 1,
Wherein the partial area is a center section or a section requested by the client terminal among the moving image output screens divided into a plurality of sections. The apparatus of claim 1, wherein the rendering unit
Converts the resolution of the original image of the 360 ° virtual reality video into a first resolution,
Wherein the first resolution is higher or equal to the resolution of the original image of the 360 ° virtual reality video. The method according to claim 1,
The encoding unit converts the resolution of a part of the 360 ° virtual reality video output screen into a second resolution and encodes the resolution,
Wherein the transmitting unit transmits a partial area having a second resolution to the client terminal,
Wherein the second resolution is lower than or equal to the resolution of the original image of the 360 ° virtual reality video. The apparatus of claim 1, wherein the rendering unit
Wherein the original image of the 360 ° virtual reality video is rendered using GPU or hardware acceleration. The apparatus of claim 1, wherein the encoding unit
Wherein a portion of the video output screen of the video player is encoded using a GPU or hardware acceleration. The apparatus of claim 1,
And transmits the encoded partial region to the client terminal in the cloud streaming transmission. The system according to claim 1, wherein the web server
A receiving unit for receiving terminal resolution information from the client terminal; Further comprising:
The encoding unit converts the resolution of a part of the moving picture output screen of the moving picture player according to the terminal resolution using the terminal resolution information of the client terminal,
Wherein the transmitting unit transmits the encoded partial region to the client terminal.

Images