WO2019013415A1 - Method and system for live-streaming vr content - Google Patents

Abstract

A system for providing a live streaming service for a VR content is provided. A system for live-streaming a VR content according to an embodiment of the present invention may comprise: a rendering server for executing a second broadcasting VR game content identical to a first broadcasting VR game content executed by a streamer terminal, synchronizing the second broadcasting VR game content with the first broadcasting VR game content by using game progress information relating to the first broadcasting VR game content, which is provided from the streamer terminal, and rendering a 360-degree panoramic image on the basis of the synchronized second broadcasting VR game content; and a broadcasting management server for allowing the rendered 360-degree panoramic image to be streamed to a viewer terminal.

Description

Live streaming method and system for VR content

The present invention relates to a live streaming method for VR content and a system for performing the method. More specifically, the present invention relates to a method of efficiently streaming a 360-degree panorama image to broadcast VR content in order to provide seamless live streaming service for broadcast VR content, a method of streaming live streaming And a system for performing the methods.

Recently, a field of great interest from around the world is the field of virtual reality (hereinafter abbreviated as "VR"). Some believe that the 2020 VR content market will come in the wake of the hardware market.

VR refers to a specific environment or situation, or technology itself, that is similar to the reality created by artificial technology using computers, but not actual. At this time, the created virtual environment or situation stimulates the user's five senses and makes the user experience the spatial and temporal experiences similar to reality, thereby freely bringing the boundary between reality and imagination free. In addition to being immersed in virtual reality, users can also interact with things implemented in virtual reality, such as manipulating or commanding them using real devices. VR is distinguished from a one-sided simulation in that it can interact with users and create user experience.

On the other hand, as Internet broadcasting platforms such as Africa TV, Twot, and YouTube are established, streamers that broadcast personal internet over various contents are increasing day by day. In recent years, streamers have become more diverse, including advertising sponsors (such as banner ads from companies) and YouTube streaming revenues. As a result, not only personal hobby broadcasting but also streamers for personal internet broadcasting to be.

However, despite the fact that VR content is a big issue all over the world, it is difficult for VR contents to be transmitted to viewers through the pre-established Internet broadcasting platform. to be.

For example, assuming that a streamer wearing a VR device such as an HMD (head mounted device) is going to broadcast Internet for a VR game content through an established Internet broadcasting platform, a broadcasting client installed in the streamer terminal The game screen is captured and streamed to the viewer terminal as it is. In addition, since the viewing client installed on the viewer terminal displays the captured screen image on the viewer terminal as it is, even the viewer wearing the HMD can not experience the sense of realism and immersion of the original VR game content.

In addition, even if the viewer wearing the VR device turns the viewpoint by turning his or her head to the left or right, the viewpoint of the game image displayed on the viewer terminal is not changed, which also deteriorates the sense of reality and immersion of the VR content.

Therefore, there is a demand for a live streaming method and system that can share experiences with streamers and viewers with VR content, and improve the sense of reality and immersion of viewers.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a live streaming system for VR content.

Another technical problem to be solved by the present invention is to provide a method for efficiently streaming a 360 degree panorama image and a system for performing the method in order to provide seamless live streaming service to viewers.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a live streaming system designed to allow streamers and viewers to share experiences on VR content.

It is another object of the present invention to provide a live streaming method designed to improve the immersion feeling of viewers watching streaming broadcasts for broadcast VR content and a system for performing the method.

The technical objects of the present invention are not limited to the above-mentioned technical problems, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a live streaming system for a VR content, the system comprising: a VR server for executing a VR game content for a second broadcast identical to a first broadcast VR game content executed in a streamer terminal, Synchronizes the second broadcast VR game content with the first broadcast VR game content in response to receipt of the game progress information for the first broadcast VR game content from the streamer terminal in real time, A rendering server for rendering a 360 degree panorama image based on the game content, and a broadcast management server for allowing the rendered 360 degree panorama image to be streamed to a viewer terminal.

In one embodiment, the rendering server renders the 360-degree panoramic image using an image photographed by a virtual camera positioned adjacent to a virtual object in a virtual space built on the second broadcast VR game content , The virtual object is a virtual object controlled according to input of the streamer, and the virtual camera can be moved together with the virtual object in the virtual space.

In one embodiment, in response to receipt of viewpoint information and viewing angle information of the viewer from the viewer terminal, the rendering server generates a first partial panorama image corresponding to the viewpoint information and the viewing angle information from the rendered 360 degree panorama image To the broadcast management server.

In one embodiment, the rendered 360-degree panorama image is a first 360-degree panorama image, and the streamer terminal displays a second 360-degree panorama image rendered based on a capture screen for the first broadcast VR game content One of the first 360 degree panorama image and the second 360 degree panorama image may be streamed to the viewer terminal.

In one embodiment, the broadcast web server includes: a member information management module for managing information on a streamer member; an authentication module for performing authentication on the streamer using information about the streamer member; And a billing module for performing a billing process in response to the request for purchase of the broadcast right in response to a request for purchase of a license ticket of the trimmer terminal. At this time, the streaming request of the streamer terminal that has not purchased the broadcasting right can be blocked.

According to another aspect of the present invention, there is provided a live streaming system for a VR content, the system comprising: a streamer terminal for providing a web page service to a streamer for streaming broadcast virtual reality (VR) content; And a broadcasting management server for broadcasting a 360-degree panorama image of the VR content for broadcasting to a viewer terminal using a pre-built Internet broadcasting platform in response to a streaming request of the streaming terminal have.

According to another aspect of the present invention, there is provided a live streaming system for VR content, comprising: a first virtual camera disposed in a virtual space constructed on broadcast VR content; A second virtual camera disposed at a position spaced apart by a predetermined interval, rendering a left eye panorama image based on the image photographed by the first virtual camera, and projecting, on the basis of the image photographed by the second virtual camera, Rendering a panoramic image, wherein the left panoramic image and the right panoramic image are 360-degree panoramic images; And a broadcast management server for allowing each of the left panoramic image and the right panoramic image to be streamed to a viewer terminal.

According to another aspect of the present invention, there is provided a live streaming method for VR content for broadcasting, which is performed by a live streaming system, comprising the steps of: Receiving a streaming request for the broadcast VR content from a streamer terminal that is broadcasting; And a step of, in response to the streaming request, streaming the 360-degree panorama image for the broadcast VR content to the viewer terminal using the pre-built Internet broadcast platform.

According to another aspect of the present invention, there is provided a computer program for causing a computer to execute the steps of: receiving a streaming request for a broadcast VR content from a streamer terminal for streaming broadcast VR content; And in response to the streaming request, streaming the 360 degree panorama image for the broadcast VR content to the viewer terminal using the pre-built Internet broadcast platform, the stream may be recorded on a computer readable recording medium .

According to another aspect of the present invention, there is provided a live streaming method for VR content for broadcasting performed by a live streaming system, comprising the steps of: Receiving, in real time, game progress information on a first broadcast VR game content, in response to receipt of the game progress information in real time, using the game progress information to transmit the second broadcast VR game content to the first broadcast VR Synchronizing with the game content, rendering a 360 degree panorama image based on the synchronized second broadcast VR game content, and streaming the rendered 360 degree panorama image to a viewer terminal.

According to another aspect of the present invention, there is provided a computer program product for causing a computer to execute the steps of: receiving game progress information for a first broadcast VR game content executed in a streamer terminal in real time, Synchronizing the second broadcast VR game content with the first broadcast VR game content using the game progress information in response to the game progress information being received in real time, And a step of streaming the rendered 360 degree panorama image to a viewer terminal.

According to another aspect of the present invention, there is provided a method for live streaming VR content, the method comprising: placing a first virtual camera in a virtual space established on the VR content for broadcasting; The method comprising the steps of: arranging a second virtual camera at a position spaced apart from the first virtual camera by a predetermined interval; rendering a left panoramic image based on the image photographed by the first virtual camera; Rendering the left eye panorama image and the right eye panorama image to a viewer terminal, and rendering the left eye panorama image and the right eye panorama image to a viewer terminal.

According to another aspect of the present invention, there is provided a computer program product including a computer-readable recording medium having a computer-readable recording medium storing a program for causing a computer to execute the steps of: arranging a first virtual camera in a virtual space built on a broadcast- The method comprising the steps of: arranging a second virtual camera at a position spaced apart from the first virtual camera by a predetermined interval; rendering a left panoramic image based on the image photographed by the first virtual camera; Rendering the left eye panorama image and the right eye panorama image to a viewer terminal, and reproducing the left eye panorama image and the right eye panorama image, respectively, on a computer readable recording medium.

According to the above-described embodiments of the present invention, synchronization is performed between the first broadcast VR content running in the streamer terminal and the second broadcast VR content running in the rendering server, and the rendering server transmits the second broadcast VR content The 360 degree panorama image is rendered based on the rendered 360 degree panorama image, and the rendered 360 degree panorama image can be streamed to the viewer terminal. Accordingly, the experience of the broadcast VR content can be shared between the viewer and the streamer watching the streaming broadcast. In addition, the streamer terminal does not perform the rendering of the 360-degree panorama image, so that the performance requirements for live streaming are alleviated, which may have the effect of inducing streamer participation.

In addition, when the streaming broadcast is performed for the VR game contents, the 360 degree panorama image is rendered using the virtual camera disposed near the virtual object controlled by the streamer. Thus, the sense of immersion of the viewer in the streaming broadcast can be maximized.

Also, since the 360-degree panoramic image is provided, the viewpoint of the viewer is not completely dependent on the viewpoint of the streamer, so that the viewer's sense of reality and immersion of the VR content can be maximized.

In addition, a streaming broadcasting service for VR contents can be provided using a pre-established Internet broadcasting platform, and the system construction cost can be greatly reduced.

In addition, since streaming broadcasting can be performed on a new topic called VR content, personal internet broadcasting is more active.

In addition, the entire 360-degree panorama image may not be streamed but the partial panorama image corresponding to the view field of the viewer may be streamed. Thus, the bandwidth of the network is saved, and a seamless live streaming service can be provided.

Also, among the partial panorama images, an image outside the visible region of the viewer can be encoded and streamed in a low quality. Thus, the bandwidth of the network is further saved, and a seamless live streaming service can be provided.

In addition, if the game is not in progress by the streamer, a separate 360 degree panorama image rendered at the streamer terminal can be streamed. Accordingly, the feeling of immersion of the viewer can be maintained, and a seamless live streaming service can be provided.

In addition, streaming broadcasting is permitted only for streamers purchased through a billing process, whereby a new profit structure can be formed for a provider of a live streaming system and / or an Internet broadcasting platform, and the effect of mass- have.

Further, the left eye panorama image and the right eye panorama image can be provided by using a plurality of virtual cameras arranged at a distance apart from each other. Accordingly, the viewer can appreciate the stereoscopic panoramic image according to the binocular parallax, and the immersion feeling of the viewer for the broadcast VR content can be further improved.

Further, the viewpoint of the virtual camera can be automatically adjusted according to the distance between the first virtual object controlled by the streamer and the second virtual object not controlled on the broadcast VR content. For example, as the second virtual object approaches the first virtual object, the viewpoint of the virtual camera may be adjusted so that a convergence angle formed by the viewpoint of the virtual camera becomes larger. Accordingly, since the panorama image rendered based on the second virtual object approaching the first virtual object can be provided to the viewer, the immersion feeling of the viewer for the broadcast VR content can be further improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood to those of ordinary skill in the art from the following description.

1 is a configuration diagram of a live streaming system for VR contents according to a first embodiment of the present invention.

FIGS. 2 to 5 are exemplary views of a screen that can be displayed as the game client module is executed in the streamer terminal.

6 is a block diagram illustrating a rendering server that is a component of a live streaming system according to an embodiment of the present invention.

FIGS. 7 and 8 are views for explaining a 360-degree panoramic image rendering method that may be referred to in some embodiments of the present invention.

9 is a block diagram illustrating a broadcast management server, which is a component of a live streaming system according to an embodiment of the present invention.

10 is a configuration diagram of a live streaming system for VR contents according to the second embodiment of the present invention.

11-13 illustrate a second 360 degree panoramic image rendering method that may be referenced in some embodiments of the present invention.

14 is a configuration diagram of a live streaming system according to the third embodiment of the present invention.

15 is a block diagram illustrating a relay server that is a component of a live streaming system according to an embodiment of the present invention.

16 is a block diagram illustrating a broadcast web server that is a component of a live streaming system according to an embodiment of the present invention.

17 is a flowchart showing the operation of the live streaming system according to the third embodiment.

18 is a flowchart illustrating a live streaming method for VR content according to an embodiment of the present invention.

19 to 23 are diagrams for explaining a live streaming method considering network conditions according to the first to third embodiments of the present invention.

24 and 25 are views for explaining a process of rendering left and right panoramic images in the live streaming method according to the fourth embodiment of the present invention.

26 to 30 are views for explaining a process of automatically adjusting a viewpoint of a virtual camera based on a virtual object in the live streaming method according to the fourth embodiment of the present invention.

31 is a flowchart illustrating a process of streaming left and right panoramic images in a live streaming method according to a fourth embodiment of the present invention.

32 is a hardware block diagram of an exemplary computing device capable of implementing various devices that constitute a live streaming system for VR content in accordance with some embodiments of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise. The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification.

It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

Prior to the description of the present specification, some terms used in this specification will be clarified.

In this specification, an Internet broadcasting platform means a platform for providing a streaming broadcasting service using the Internet. For example, the Internet broadcasting platform may be, but is not limited to, an African TV, a YouTube, a Twot, and a Cacao TV.

In this specification, a streamer means a person who performs Internet broadcasting through an Internet broadcasting platform. The streamer is named as BJ (broadcasting jockey) in African TV, PD (play director) in Cacao TV, streamer in twin TV and creator in YouTube, and various other terms such as broadcaster However, in the present specification, it is referred to as a streamer for convenience.

In this specification, virtual reality (VR) refers to a specific environment or situation, or technology itself, that is similar to the reality produced by an artificial technique using a computer or the like, but is not actual. VR can be used in various technical terms such as artificial reality, cyberspace, virtual worlds, virtual environment, synthetic environment, artificial environment, It should be noted that they may be used interchangeably but may refer to the same object.

In this specification, an instruction is a set of instructions, which are grouped by function, indicating that they are executed by a processor and a component of a computer program.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a live streaming system for VR contents according to a first embodiment of the present invention.

1, a live streaming system according to a first embodiment of the present invention includes a streamer terminal 100, a rendering server 300, a broadcasting management server 500, at least one Internet broadcasting platform 700, And may include one viewer terminal 900. However, it should be understood that the present invention is not limited to the above-described embodiments, and that various changes and modifications may be made without departing from the scope of the present invention. It should be noted that each component of the live streaming system shown in FIG. 1 shows functional elements that are functionally distinct, and that at least one component may be implemented in a form that is integrated with one another in an actual physical environment. For example, the rendering server 300 and the broadcast management server 500 may be implemented as a single integrated server device. Hereinafter, the components of the live streaming system according to the first embodiment will be described.

In the live streaming system according to the first embodiment, the streamer terminal 100 is a computing device used by the streamer 1 to proceed with live streaming broadcasting. Here, the computing device may be a notebook, a desktop, a laptop, a smart phone, and the like, but it is not limited thereto and may include any type of device having a computing function and a communication function can do.

The streamer terminal 100 can execute live broadcast for VR content for broadcasting through the streaming platform (e.g., the rendering server 300, the broadcast management server 500, the Internet broadcast platform 700, etc.). In the following description, it is assumed that the type of the VR content for broadcasting is the VR game content. However, this is for the purpose of illustrating some embodiments of the present invention, and the scope of the present invention is not limited to the above examples.

The streamer terminal 100 can execute the broadcast VR game contents through the pre-installed game client module and proceed with live streaming broadcasting. Here, the game client module may be understood as a software module in which a broadcasting function is additionally implemented in the VR game contents. In order to provide more convenience of understanding, an example in which the game client module is executed when the VR game content for broadcasting is an FPS (first-person shooter) game will be briefly described.

As shown in FIG. 2, when the broadcast VR game content is executed by the game client module, the main interface 5 for the broadcast VR game content can be displayed. Also, if the field of view is changed to a specific direction (e.g., right side) according to the input of the broadcast streamer 1, the broadcast interface 7 related to live streaming broadcast can be displayed as shown in FIG. Alternatively, if the current time is changed to the third person view according to the input of the streamer 1 as shown in FIG. 4, the broadcast interface 7 may be displayed. For reference, a button disabled (e.g., YouTube) in FIG. 3 indicates an Internet broadcasting platform that is currently unavailable for some reason.

The broadcast interface 7 may include, for example, a selection interface for an Internet broadcast platform on which live broadcasting is to be performed, a broadcast start / end interface, and the like. Therefore, the streamer 1 can start live streaming broadcasting at a desired time while playing the VR game contents for broadcasting.

When the live broadcast starts, an indicator 9a indicating " broadcasting " can be displayed on the screen of the streamer terminal 1 as shown in Fig. If the broadcast is impossible due to a network problem or the like, an indicator 9b indicating " no broadcast " may be displayed.

Referring again to FIG. 1, the streamer 1 may be a broadcast host who wears a VR device such as an HMD (head mounted device) and plays VR game contents for broadcasting. When the live streaming broadcast starts, the game image of the broadcast VR game content played by the streamer 1 can be streamed to the viewer terminal 900. [ More specifically, a 360 degree panorama image for the broadcast VR game content may be streamed to the viewer terminal 900 through at least one Internet broadcast platform 700 selected by the streamer 1. [

In one embodiment, the 360 degree panorama image may be rendered based on the broadcast VR game content running on the streamer terminal 100. According to the present embodiment, since a separate rendering server 300 need not be provided, a live streaming system can be constructed at a low cost. However, since high-performance computing power is generally required to execute VR game contents, it is preferable that a separate rendering server 300 is installed to provide seamless live streaming service.

In another embodiment, it may be determined based on the performance of the streamer terminal 100 whether a 360 degree panorama image is to be rendered in the streamer terminal 100. For example, if the streamer terminal 100 is a high-performance computing device satisfying predetermined performance conditions, the 360-degree panoramic image can be performed in the streamer terminal 100. In the opposite case, the 360-degree panorama image may be rendered in the rendering server 300. According to this embodiment, the computing capability of the rendering server 300 is provided to only some streamer terminals that lack computing capability. Therefore, the burden on the rendering server 300 can be reduced.

In yet another embodiment, the streamed 360 degree panorama images may all be rendered by the rendering server 300. According to the present embodiment, the burden on the streamer terminal 100 can be alleviated. Hereinafter, in order to facilitate understanding, it is assumed that the 360 degree panorama image is rendered in the rendering server 300, and the description will be continued. However, the scope of the present invention is not limited thereto.

In the live streaming system according to the first embodiment, the rendering server 300 renders the 360-degree panorama image 3 based on the VR game content for broadcasting.

According to the embodiment of the present invention, the rendering server 300 generates the VR game content for broadcasting (hereinafter referred to as " first broadcast VR game content ") executed by the streamer terminal 100 in order to render the 360- (Hereinafter, referred to as " second broadcast VR game content ") that is the same as the broadcast VR game content. The rendering server 300 synchronizes the second broadcast VR game content with the first broadcast VR game content using the game progress information received in real time from the streamer terminal 100 during the streaming broadcast, 2 < / RTI > VR game content. In the present embodiment, the game progress information includes all information required for synchronization of two VR game contents. For example, the game progress information includes a virtual object controlled by the streamer 1, a state of the virtual object, But it is not limited to the above-listed examples. A more detailed description of this embodiment will be given later with reference to Figs. 6 and 11. Fig.

In the live streaming system according to the first embodiment, the broadcast management server 500 includes a module (or or) 602 for allowing the 360-degree panorama image 3 rendered by the rendering server 300 to be streamed through the Internet broadcast platform 700 Device).

According to an exemplary embodiment of the present invention, the broadcast management server 500 may include at least one broadcast client module operatively associated with the Internet broadcast platform 700. Here, the broadcast client module is a module for transmitting a streaming object image to a streaming server of an Internet broadcasting platform, for example, a software module provided by the Internet broadcasting platform. More specifically, for example, when the Internet broadcasting platform is " Africa TV ", the broadcasting client module may indicate a software module installed in a terminal of the BJ which broadcasts via Africa TV. A more detailed description of the configuration and operation of the broadcast management server 500 will be given later with reference to FIG.

In the live streaming system according to the first embodiment, the Internet broadcasting platform 700 means a platform for providing a live broadcasting streaming service over the Internet.

According to the embodiment of the present invention, a pre-established Internet broadcasting platform such as Africa TV, Twot Site, YouTube, etc. can be used as the Internet broadcasting platform 700. That is, according to the present embodiment, a separate streaming server is not constructed to stream real-time VR game contents, but a legacy system used by existing viewers can be utilized. Accordingly, the system construction cost is greatly reduced, and existing viewers using the Internet broadcasting platform can be secured as potential viewers of VR contents broadcasting.

In the live streaming system according to the first embodiment, the viewer terminal 900 means a terminal used by a viewer to view a live streaming broadcast. The viewer terminal 900 may be, for example, a computing device equipped with a viewing client module linked with the Internet broadcasting platform 700. At this time, the computing device may be implemented as an arbitrary device such as a notebook, a desktop, a laptop, a smart phone, or an HMD.

The viewing client module may be, for example, a software module installed to use a live streaming broadcasting service provided by an Internet broadcasting platform.

Each component that constitutes a live streaming system according to some embodiments of the present invention may communicate over a network. Here, the network may be any kind of wired / wireless network such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, a wibro Can be implemented.

Hereinafter, the components of the live streaming system will be described in more detail with reference to FIGS. 6 to 9. FIG.

6 is a block diagram illustrating a rendering server 300 that is a component of a live streaming system for VR content according to the first embodiment of the present invention.

6, the rendering server 300 includes a content execution module 6100, a content synchronization module 330, a panorama image rendering module 350, an encoding module 370, and a panorama image transmission module 390 Lt; / RTI > However, only the components related to the embodiment of the present invention are shown in Fig. Therefore, it will be understood by those skilled in the art that other general-purpose components other than those shown in FIG. 6 may be further included. Hereinafter, each component of the rendering server 300 will be described.

The content execution module 6100 is a module for executing the second broadcast VR game content. The second broadcast VR game content is the same VR game content as the first broadcast VR game content executed in the streamer terminal 100 as described above.

For example, the content execution module 6100 receives information (e.g., name, identifier, etc.) of the first broadcast VR game content from the streamer terminal 100, The second broadcast VR game content identical to the VR game content can be executed.

Next, the content synchronization module 330 synchronizes the second broadcast VR game content and the first broadcast VR game content using the content progress information of the first broadcast VR game content. Here, the content progress information may be received from the streamer terminal 100 in real time.

Next, the panorama image rendering module 350 is a module for rendering a 360-degree panorama image streamed to the viewer terminal 900 based on the second broadcast VR game content. Hereinafter, a method of rendering a 360-degree panoramic image according to an embodiment of the present invention will be described in detail with reference to FIGS. 7 and 8. FIG.

FIG. 7 illustrates an exemplary virtual space constructed on the second broadcast VR game content. 7 shows a case where the first virtual object 10 and the second virtual object 20 exist in the virtual space and the second virtual object 20 is a virtual object played by the streamer 1 As shown in FIG.

As shown in FIG. 7, the game image is photographed by the virtual camera 30 disposed adjacent to the second virtual object 20 in the virtual space. Then, the panorama image rendering module 350 renders the 360-degree panorama image using the captured game image.

8, the virtual camera 30 rotates 360 degrees around the second virtual object 20, captures the game image 40, and the panorama image rendering module 350 displays the game image 40 The panoramic panoramic image can be rendered using the panoramic panoramic image 40. However, the method of rendering the 360 degree panorama image by the panorama image rendering module 350 is not limited to this, and it may be rendered in any way.

In this embodiment, when the second virtual object 20 moves in the virtual space according to the input of the streamer 1, the virtual camera 30 can also be moved along the second virtual object 20. [ According to the present embodiment, since the virtual camera 30 moves along the play object 20 and shoots the game video image, viewers watching the broadcast VR game content can receive the feeling of playing the VR game content directly. Therefore, the immersion feeling for the live streaming broadcast can be maximized.

In addition, in the present embodiment, since the 360-degree panoramic image is provided to the viewer, the viewer's view may not be completely dependent on the field of view of the streamer 1. For example, if the view direction of the viewer changes from the first viewpoint to the second viewpoint, an image corresponding to the second viewpoint among the 360 degree panorama images is displayed on the viewer terminal 900, regardless of the viewpoint of the streamer 1 . Therefore, the field of view of the viewer is not completely dependent on the field of view of the streamer 1. According to this embodiment. The viewer who watches the broadcast VR content can experience the same sense of reality and immersion as the streamer 1 without installing the high-volume VR game content on the viewer terminal 900. [

Referring back to FIG. 6, description will be continued.

The encoding module 370 is a module for encoding the rendered 360 degree panorama image.

In one embodiment, the encoding module 370 may adaptively encode the network state between the Internet broadcast platform 700 and the viewer terminal 900. [ For example, if the network condition is unstable (e.g., if there is a wireless network section) and the available bandwidth is not sufficient, the encoding module 370 may encode the 360 degree panorama image with a lower quality image than the preset reference quality . In the opposite case, the encoding module 370 may encode the 360 degree panorama image with a predetermined reference image quality or a higher image quality.

In one embodiment, the encoding module 370 may perform encoding taking into account the computing capabilities of the viewer terminal 900. [ For example, if the viewer terminal 900 is a device with a poor computing capability, such as a mobile terminal, the encoding module 370 may perform encoding with a picture quality lower than a predetermined reference picture quality.

In one embodiment, the encoding module 370 may encode a 360 degree panorama image using a scalable video coding (SVC) technique to selectively select an image quality. In this case, the image quality of the 360-degree panoramic image can be adjusted based on at least one of conditions of the computing ability and the network state of the viewer terminal 900. At this time, the adjustment of the image quality may be performed by any one of the components of the panorama image transmission module 390, the broadcasting management server 500, or the Internet broadcasting platform 700.

In addition, according to the embodiment of the present invention, the encoding module 370 can encode and provide one panorama image with a plurality of image qualities. The description of this embodiment will be given later with reference to Figs. 21 to 23. Fig.

Next, the panorama image transmission module 390 transmits the encoded 360 degree panorama image to the broadcasting management server 500. The transmitted 360-degree panoramic image can be streamed to the viewer terminal 900 by the broadcast management server 500.

In accordance with an embodiment of the present invention, the panorama image transmission module 390 receives viewer's viewpoint information and viewing angle information from the viewer terminal 900, and in response thereto, Only the panoramic image can be transmitted to the viewer terminal 900. According to the present embodiment, network bandwidth is saved and seamless live streaming service can be provided. A detailed description of this embodiment will be given later with reference to Figs. 19 and 20. Fig.

FIG. 9 is a block diagram illustrating a broadcast management server 500, which is a component of a live streaming system for VR content according to the first embodiment of the present invention.

Referring to FIG. 9, the broadcast management server 500 may include a panorama image receiving module 510 and a broadcast client managing module 530. However, only components that are related to the embodiment of the present invention are shown in Fig. Therefore, it will be understood by those skilled in the art that other general-purpose components other than the components shown in FIG. 9 may be further included.

Referring to the respective components, the panorama image receiving module 510 receives the 360-degree panorama image rendered by the rendering server 300. Here, the 360-degree panorama image may be a properly encoded 360-degree panorama image.

The broadcast client management module 500 is a module for managing at least one broadcast client module 531 to 535 linked with the Internet broadcast platform 700 that has been constructed.

According to an embodiment of the present invention, the broadcast client management module 530 may be configured to broadcast the broadcast client information to the corresponding broadcast client based on broadcast information of the stored streamer member and / or selection information of the Internet broadcast platform selected by the streamer member through the broadcast interface, You can run the module. Also, the broadcast client management module 530 may stream the 360 degree panorama image to the viewer terminal 900 using the broadcast client module executed.

Each of the components shown in FIGS. 6 and 9 or the components shown in FIGS. 15 and 16 to be described later may be implemented by software, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC) And the like. However, the components are not limited to software or hardware, and may be configured to be addressable storage media, and configured to execute one or more processors. The functions provided in the components may be implemented by a more detailed component, or may be implemented by a single component that performs a specific function by combining a plurality of components.

1 to 9, a live streaming system for VR contents according to the first embodiment of the present invention has been described. The 360-degree panorama image for the broadcast VR game content is rendered in the rendering server 300, and the rendered 360-degree panorama image is streamed to the viewer terminal 900. FIG. Therefore, the experience of the broadcast VR game contents can be shared between the viewer and the streamer watching the streaming broadcast.

Next, a live streaming system for VR contents according to a second embodiment of the present invention will be described with reference to FIGS. 10 and 13. FIG.

10 is a configuration diagram of a live streaming system for VR contents according to the second embodiment.

As shown in FIG. 10, the configuration of the live streaming system according to the second embodiment is the same as the system according to the above-described first embodiment. However, the live streaming system according to the second embodiment differs from the system according to the first embodiment in that streaming service is provided using two kinds of 360 degree panorama images. To avoid redundant explanations, let's go on to explain the difference.

The live streaming system according to the second embodiment provides a live streaming service using the first 360 degree panorama image 18 and the second 360 degree panorama image 19. [ The first 360 degree panorama image 18 may be an image rendered by the rendering server 300 and the second 360 degree panorama image 19 may be an image rendered by the streamer terminal 100. [

According to the embodiment of the present invention, the second 360 degree panorama image 19 can be streamed to the viewer terminal 900 when the streamer 1 does not have the first broadcast VR game content in progress. That is, when the streamer 1 is playing the first broadcast VR game contents, the first 360-degree panorama image 18 is streamed. Otherwise, the second 360-degree panorama image 19 is streamed .

Here, when the first broadcast VR game content is not in progress, a case of loading a specific content is exemplified. When the specific content is loaded, the viewer can feel that the streaming broadcast is cut off because the rendering server 300 can not render the 360 degree panorama image. Assuming that the loading screen is streamed as it is, the viewer will not feel that the streaming broadcast is cut off, but the feeling of immersion in the VR content for broadcasting may be hindered by the loading screen. Therefore, in such a case, by streaming the second 360 degrees panorama image 19, the immersion feeling of the viewer can be maintained.

Hereinafter, a method of rendering the second 360 degree panorama image 19 will be described with reference to Figs. 11 to 13. Fig. The method of rendering the first 360 degree panorama image 18 will be described with reference to the description of FIG. 7 and FIG.

Figs. 11 and 12 are diagrams for explaining a method of rendering the second 360 degree panorama image 19. Fig.

FIG. 11 shows a first image 21a generated by capturing a first broadcast VR game content displayed on the screen of the streamer terminal 100. FIG. 11, the case where the size of the first image 21a is " 1920x1080 " is shown as an example.

12, the streamer terminal 100 converts the size of the first image 21a into a predetermined size to render the second 360 degrees panorama image 20, and generates the second image 21b do. In FIG. 12, the predetermined size is exemplified as " 640x360 ", but it may vary as much as possible.

The streamer terminal 100 arranges the second image 21b at the center of the 360 degree panorama image 20 and sets the region 23 excluding the second image 21b in the 360 degree panorama image to a predetermined color for example, black) to render a 360 degree panoramic image 20.

The rendered 360 degree panorama image 20 may be as shown in FIG. 13, for example. Specifically, when the 360-degree panoramic image 20 is streamed, when the viewer 30 looks at the front, the second image 21b is seen. When the viewpoint is changed, a space painted with a predetermined color is seen.

Up to now, a live streaming system for VR contents according to the second embodiment of the present invention has been described with reference to FIGS. 10 to 13. FIG. Next, a live streaming system for VR content according to the third embodiment of the present invention will be described with reference to FIG. 14 to FIG.

14 is a configuration diagram of a live streaming system for VR content according to the third embodiment of the present invention.

Referring to FIG. 14, the live streaming system according to the third embodiment of the present invention may further include a relay server 200 and a broadcast web server 400. However, it should be understood that the present invention is not limited to the above-described embodiments, and that various changes and modifications may be made without departing from the scope of the present invention. It should be noted that each component of the live streaming system shown in FIG. 14 shows functional elements that are functionally separated, and that at least one component may be implemented in a form that is integrated with each other in an actual physical environment.

In the live streaming system according to the third embodiment, the relay server 200 generates a broadcast channel to provide an independent broadcast environment to the streamer 1, and receives live streaming data from the streamer terminal 100 during live streaming And relays the game progress information to the rendering server 300. For example, in response to a request from a streamer terminal 100 (eg, an authenticated streamer or a streamer that has purchased a broadcast license) in which live streaming broadcasting is permitted, the relay server 200 generates and provides a broadcast channel for each streamer do. Here, the broadcasting room may refer to an online space shared by viewers watching live streaming broadcasting of the streamer and the streamer.

The relay server 200 can further provide various functions necessary for smooth broadcast progress such as supporting a chat function for each broadcast. Since the broadcasting room is a well-known concept in the related art, a detailed description thereof will be omitted.

For reference, the relay server 200 may be implemented as a photon server or the like, but the scope of the present invention is not limited thereto. A more detailed description of the relay server 200 will be given later with reference to FIG.

In the live streaming system according to the third embodiment, the broadcast web server 400 provides a web page service to the user. For example, the broadcasting web server 400 provides a web page in response to a request from the streamer terminal 100, and transmits the web page via the web page to the broadcasting web server 400, such as membership registration, login, member information management, Function can be provided. A more detailed description of the broadcast web server 400 will be given later with reference to FIG.

Hereinafter, the relay server 200 and the broadcast web server 400 will be described in detail with reference to FIG. 15 and FIG.

FIG. 15 is a block diagram illustrating a relay server 200 that is a component of a live streaming system for VR content according to the third embodiment of the present invention.

Referring to FIG. 15, the relay server 200 may include a content information receiving module 210, a content advance relay module 230, and a broadcast management module 250. Fig. 15 shows only the components related to the embodiment of the present invention. Accordingly, those skilled in the art will recognize that other general-purpose components other than those shown in FIG. 15 may be further included.

The content information receiving module 210 receives information (e.g., game progress information) about the first broadcast VR game content from the streamer terminal 100, and the content information relay module 230 And transmits information on the first broadcast VR game contents to the rendering server 300.

For example, the content information receiving module 210 receives game progress information for the first broadcast VR game content in real time from the streamer terminal 100, and the content information relay module 230 receives the game progress information To the rendering server 300. At this time, the rendering server 300 may synchronize the second broadcast VR game content with the first broadcast VR game content using the game progress information, and may render the 360-degree panorama image (or the first 360-degree panorama image) have.

Next, the broadcasting management module 250 generates a broadcasting room indicating an online space in which the streamer is broadcast in response to a broadcasting start request of the streamer terminal 100 to which the streaming broadcasting is permitted. In addition, the broadcasting management module 250 may terminate the corresponding broadcasting channel in response to a broadcasting end request from the streamer terminal 100. [

According to the embodiment of the present invention, the streamer in which the streaming broadcast is permitted may be a streamer that has been successfully authenticated and purchased a broadcast right. The broadcast start request of the streamer in which the streaming broadcast is not permitted may be blocked in this embodiment.

16 is a block diagram illustrating a broadcast web server 400 that is a component of a live streaming system for VR content according to the third embodiment of the present invention.

Referring to FIG. 16, the broadcast web server 400 may include a member information management module 410, an authentication module 430, a broadcast information management module 450, and a billing module 470. However, only the components related to the embodiment of the present invention are shown in Fig. Therefore, it will be understood by those skilled in the art that other general-purpose components other than the components shown in FIG. 16 may be further included.

Referring to the respective components, the membership information management module 410 is a module for managing information on streamer members. For example, the member information management module 410 may manage the streamer member information by DB. Here, the information about the streamer member may include authentication information such as a member ID and a password.

The authentication module 430 is a module that performs authentication for a streamer member. The authentication may be performed, for example, on an ID / password basis, but may be performed in any other manner.

The broadcast information management module 450 is a module for managing broadcast information on a streamer member. The broadcast information management module 450 can organize the broadcast information into a DB. Here, the broadcast information may include, for example, setting information for an Internet broadcasting platform, information on VR content for broadcasting, information on broadcasting rights, and the like.

The billing processing module 470 is a module for performing a billing process in accordance with the purchase of a broadcasting right in response to a streaming member purchase request of a streaming member. The method of performing the billing process may be performed by any method such as point, account transfer, card settlement, and the like.

According to the embodiment of the present invention, a live broadcast request of a streamer that has not purchased a broadcast right can be blocked. According to the present embodiment, streaming broadcasting is allowed only for a streamer purchased through a billing process, so that a new profit structure can be provided to a provider of a live streaming system and / or an Internet broadcasting platform. In addition, streamer's indiscreet penetration of broadcasting is blocked, and high-quality Internet broadcasting can be mass-produced.

Next, for better understanding, an operation method of the live streaming system according to the third embodiment will be described.

17 is a flowchart showing the operation of the live streaming system according to the third embodiment. However, it should be understood that the present invention is not limited thereto and that some steps may be added or deleted as needed. In particular, FIG. 17 illustrates an example in which the rendering server 300 and the broadcasting management server 500 are implemented as a single server device.

Referring to FIG. 17, a game client module is executed in the streamer terminal 100 as a first step for performing live broadcasting on broadcast VR game contents (S10). The game client module includes the first VR game content for broadcasting, and the first VR game content for the broadcasting is also executed in step S10.

Next, an authentication procedure may be performed between the streamer terminal 100 and the broadcast web server 400 (S20). That is, through the authentication procedure, it is determined whether or not the streamer is a registered member, and live broadcasting can be permitted only for the registered member.

Next, broadcast information about the streamer authenticated by the broadcast web server 400 is searched (S30), and the broadcast client module is executed in the broadcast management server 500 based on the searched broadcast information (S50) . Alternatively, depending on the embodiment, the broadcast client module for the Internet broadcast platform selected by the streamer may be executed (see FIG. 3).

For reference, in step S50, a plurality of broadcast client modules may be executed. In this case, live streaming broadcasting can be performed through a plurality of Internet broadcasting platforms 700. [

When the broadcast client module is executed, a broadcast generation request is transmitted to the relay server 200, and the relay server 200 generates a broadcast stream for the streamer in response to the broadcast generation request (S70). In addition, the relay server 200 may connect the broadcasting client module and the game client module to the broadcasting room so that the game progress information may be shared between the two modules (S80, S90).

Next, when the streamer terminal 100 plays the first broadcast VR game content by the streamer, the game progress information is transmitted to the rendering server 300 through the relay server 200 in real time (S100 , S110, S120).

In response to receiving the game progress information, the rendering server 300 synchronizes the second broadcast VR game content with the first broadcast VR game content (S130), and the 360-degree panorama An image is generated (S140).

Finally, the generated 360 degree panorama image is streamed to the viewer terminal 900 through the internet broadcasting platform 700 (S150, S160).

Up to now, a live streaming system for VR contents according to the third embodiment of the present invention and a method of operating the system have been described with reference to FIG. 14 to FIG. Hereinafter, a live streaming system for VR content according to a fourth embodiment of the present invention will be described.

The configuration of the live streaming system for VR contents according to the fourth embodiment of the present invention may be the same as or similar to the live streaming system according to the first to third embodiments described above.

However, in the live streaming system according to the fourth embodiment, the rendering server 300 may be configured so that the panorama image rendering module 350 accurately displays the left panorama image 350 using a plurality of virtual cameras arranged at predetermined intervals, And a right-eye panoramic image, respectively. According to the present embodiment, not the one 360 degree panorama image but the left eye panorama image and the right eye panorama image are provided to the viewer terminal 900, so that the viewer can appreciate the stereoscopic panorama image through the binocular parallax. Accordingly, the immersion, realism, realism, and the like that the viewer feels about the VR content for broadcasting can be greatly improved. A detailed description of how the panoramic image rendering module 350 renders the left and right panoramic images will be described in detail with reference to FIGS. 24 to 30. FIG.

In the live streaming system according to the fourth embodiment, the rendering server 300 accurately determines whether the panoramic image transmission module 390 merges the left panoramic image and the right panoramic image according to the type of the viewer terminal 900 Or may transmit them, respectively. A detailed description of this embodiment will be given later with reference to Fig.

Up to now, a live streaming system for VR content according to the fourth embodiment of the present invention has been described. Hereinafter, a live streaming method for VR content according to an embodiment of the present invention will be described in detail with reference to FIG.

Each step of the live streaming method according to an embodiment of the present invention to be described below may be performed by a system composed of a single computing device and / or a plurality of computing devices. For example, the system may be a live streaming system according to some embodiments of the present invention. However, for the sake of convenience of description, description of the operation subject of each step included in the live streaming method may be omitted. In addition, each step of the live streaming method may be implemented as an operation of a computer program executed by a processor.

18 is a flowchart of a live streaming method for VR content according to an embodiment of the present invention. However, it should be understood that the present invention is not limited to the above-described embodiments, and that various changes and modifications may be made without departing from the scope of the present invention.

Referring to FIG. 18, game progress information on the first broadcast VR game content executed in the streamer terminal 100 is received in real time (S200). For example, the rendering server 300 may receive game progress information for the first broadcast VR game content in real time.

Next, in response to receiving the game progress information, the second broadcast VR game content and the first broadcast VR game content are synchronized using the game progress information (S220). For example, the rendering server 300 may perform the synchronization using the game progress information.

Next, a 360-degree panorama image is rendered based on the synchronized second broadcast VR game content (S240). For example, the rendering server 300 may render a 360-degree panorama image based on the synchronized second broadcast VR game content. The method of rendering the 360-degree panoramic image is as described above, and a description thereof will be omitted.

According to the embodiment of the present invention, the rendering server 300 can render the left eye panorama image and the right eye panorama image based on the synchronized second broadcasting VR game contents. A detailed description of this embodiment will be given later with reference to FIG. 24 to FIG.

For reference, the reason why the rendering server 300 renders the 360-degree panorama image is because the execution of the high-capacity VR content itself is a large burden on the streamer terminal 100 itself. In other words, by rendering a 360-degree panorama image on a separate computing device using the rendering server 300, a load imposed on the streamer terminal 100 can be minimized and a smooth live streaming service can be provided. In addition, because the performance requirements of the streamer terminal 100 are relaxed, the streamer participation of the streamer can be induced.

Next, the rendered 360 degree panorama image is streamed to the viewer terminal 900 (S260). For example, the broadcast management server 500 may stream the rendered 360-degree panorama image to the viewer terminal 900 using the pre-built Internet broadcast platform.

According to the embodiment of the present invention, when the left eye panorama image and the right eye panorama image are rendered, the left eye panorama image and the right eye panorama image may be respectively transmitted or combined based on the viewer terminal type. A detailed description of this embodiment will be given later with reference to Fig.

Up to now, a live streaming method for VR content according to an embodiment of the present invention has been described with reference to FIG. According to the above-described method, synchronization is performed between the first broadcast VR content being executed in the streamer terminal 100 and the second broadcast VR content being executed in the rendering server 300. In addition, the 360-degree panorama image is rendered on the basis of the second broadcast VR content in the rendering server 300, and the rendered 360-degree panorama image can be streamed to the viewer terminal 900. Accordingly, the experience of the streamer with respect to the VR content can be shared with the viewer.

Hereinafter, a live streaming method according to some embodiments of the present invention for providing a seamless live streaming service will be described with reference to FIGS. 19 to 23. FIG.

Because VR content provides all of the 360 degree panoramic images to support the Omni View, it has a very large capacity and requires significant network bandwidth compared to general video content. Therefore, if the network state is unstable, a problem of frequent interruption may occur in the viewer terminal 900, which may be an important factor that deteriorates the immersion feeling and interest of viewers. It is to be understood that some embodiments to be described below are for solving such a problem.

19 and 20 are diagrams for explaining a live streaming method according to the first embodiment of the present invention.

FIG. 19 shows a viewer wearing an HMD and watching a 360-degree panorama image for broadcast VR content.

19, the entire area of the 360 degree panorama image is not displayed on the screen of the viewer terminal 900, but only the partial panorama image 40 corresponding to the field of view 111 of the viewer is displayed Can be confirmed.

Therefore, according to the first embodiment, only the partial panorama image corresponding to the viewing angle of the viewer is streamed to each viewer terminal 900 without the entire 360-degree panorama image being streamed to the viewer terminal 900. [ For example, when the rendering server 300 receives the viewer's current viewpoint information (gaze direction) and viewing angle information from the viewer terminal 900, the rendering server 300 extracts the partial panorama image using the current viewpoint information and the viewing angle information , And the extracted partial panorama image may be streamed to the viewer terminal 900. Alternatively, the above-described functions may be performed by the broadcast management server 500 or a separate server, which may vary depending on the embodiment.

20 shows an example in which the partial panoramic images 121 to 125 are streamed to each viewer terminal based on the current viewpoint information and the viewing angle information of the viewer according to the first embodiment. In FIG. 20, a partial panorama image accurately corresponding to a viewing angle of a viewer is extracted and streamed. However, a partial panorama image having a wider area than a viewer's viewing angle may be streamed considering a viewpoint change of the viewer. A detailed description thereof will be given later with reference to FIG.

Up to now, a live streaming method according to the first embodiment of the present invention has been described with reference to FIGS. 19 and 20. FIG. Next, a live streaming method according to the second embodiment of the present invention will be described with reference to FIGS. 21 and 22. FIG.

According to the second embodiment, partial panorama images encoded with different image qualities can be streamed to the viewer terminal 900.

21, an image of the first region 64 corresponding to the front view point of the viewer among the entire region of the partial panorama image 60 is encoded with the first image quality, and the first region 64 The images of the second regions 61 and 65 may be encoded with the second image quality. At this time, the first image quality may be higher quality than the second image quality.

According to the embodiment of the present invention, the first image quality and the second image quality can be variably adjusted according to the state of the network between the Internet broadcasting platform 700 and the viewer terminal 900. [ For example, if the available bandwidth of the network is greater than or equal to a predetermined threshold value, at least one of the first image quality and the second image quality may be encoded with a higher image quality. On the other hand, if the available bandwidth is less than a preset threshold value, the image quality of at least one of the first image quality and the second image quality can be encoded with a lower quality.

According to an embodiment of the present invention, a first region 63 encoded in high picture quality and a second region 61 and 65 encoded in a low picture quality according to the state of the network between the Internet broadcasting platform 700 and the viewer terminal 900 The proportion occupied can be adjusted variably. For example, when the available bandwidth of the network is greater than a predetermined threshold value, the ratio of the first area 63 in the partial panorama image 60 is increased and the ratio of the second area 61 and 65 is decreased . That is, as the available bandwidth increases, the size of the first area 63 increases and the size of the second area 61 and 65 decreases relatively.

FIG. 22 shows a case where the current time of the viewer is changed to the left as the viewer turns his / her head to the left side as shown in FIG.

Referring to FIG. 22, the partial panorama image 70 corresponding to the changed point in the 360-degree panorama image may be streamed as the current point of view of the viewer changes from left to right. In addition, the region 73a encoded in the second image quality in the previous image is changed to the first image quality, the region 75 encoded in the first image quality no longer corresponds to the front viewpoint of the viewer, . ≪ / RTI >

Up to now, the live streaming method according to the second embodiment of the present invention has been described with reference to FIG. 21 and FIG. Next, a live streaming method according to the third embodiment of the present invention will be described with reference to FIG.

According to the third embodiment, the first partial panorama image corresponding to the viewing angle of the viewer and the second partial panorama image positioned in the invisible region outside the viewing angle and located adjacent to the first partial panorama image can be streamed together. That is, even if the viewer's viewpoint is changed, the predicted partial panorama image can be transmitted so that continuous streaming can be performed.

23, the partial panorama image 80 according to the third embodiment includes the first partial panorama images 83 to 87 corresponding to the viewing angle of the viewer and the first partial panorama images 83 to 87 located in the non- And two-part panoramic images 81 and 89. [ 23, it is exemplified that the second partial panorama images 81 and 89 are adjacent to the left and right sides of the first partial panorama images 83 to 87. However, the second partial panorama image may be located above and / Or may be an image located at the lower side.

According to the embodiment of the present invention, the second partial panorama images 81 and 89 may be images encoded with the same or lower image quality than the first partial panorama images 83 to 87. 23, the first area 85 of the first partial panorama image is encoded with the first image quality, and the second area 83, 87 is encoded with the second image quality lower than the first image quality, for example, The second partial panorama images 81 and 89 may be the second image quality or the image encoded with the third image quality lower than the second image quality.

According to the embodiment of the present invention, the size and image quality of the second partial panorama image can be variably adjusted according to the state of the network between the Internet broadcasting platform 700 and the viewer terminal 900. For example, if the available bandwidth of the network is greater than or equal to a predetermined threshold value, the size of the second partial panorama images 81 and 89 may be larger. Alternatively, if the available bandwidth of the network is greater than or equal to a predetermined threshold value, the second partial panorama image 81, 89 may be encoded with higher image quality.

Up to now, a live streaming method according to the first to third embodiments of the present invention has been described with reference to FIGS. 19 to 23. FIG. According to the above-described method, only some of the panoramic images are streamed or the image quality is adjusted, thereby saving bandwidth of the network and providing seamless live streaming service.

Hereinafter, a live streaming method according to a fourth embodiment of the present invention will be described with reference to FIG. 24 to FIG.

The live streaming method according to the fourth embodiment provides a live streaming service based on a left eye panorama image and a right eye panorama image. First, the process of rendering the left and right panoramic images in the live streaming method according to the fourth embodiment will be described with reference to FIGS. 24 and 25. FIG.

24 is a flowchart illustrating a process of rendering left and right panoramic images in the live streaming method according to the fourth embodiment.

Referring to FIG. 24, the rendering server 300 disposes the first virtual camera and the second virtual camera in a virtual space constructed on the second broadcast VR game content by a predetermined interval (S242, S244) . Here, the predetermined interval may be an appropriately set value so that the binocular disparity can be generated.

Next, the rendering server 300 renders the left eye panorama image based on the image photographed by the first virtual camera, and renders the right eye panorama image based on the image photographed by the second virtual camera (S246 , S248).

The rendered left eye panorama image and the right eye panorama image can be respectively streamed to the viewer terminal 900. For example, as shown in Fig. 25, when the left panoramic image 97 and the right panoramic image 95 generated based on the two virtual cameras 91 and 93 placed on the virtual space are streamed The left panoramic image 97 and the right panoramic image 95 are output to the left and right displays of the HMD of the viewer, respectively. Accordingly, the viewer can appreciate the three-dimensional game image, so that the feeling of immersion and realism of the streaming broadcast can be greatly improved.

Hereinafter, with reference to FIGS. 26 to 29, a method of adjusting the viewpoint of the virtual camera to improve the immersion feeling of the viewer in some embodiments of the present invention will be described.

Referring to FIG. 26, the first and second virtual cameras 103a and 103b, which respectively photograph the left eye panorama image and the right eye panorama image, are adjacent to the virtual object 101 controlled by the streamer 1 as described above . The point at which the viewpoints of the first and second virtual cameras 103a and 103b meet becomes a convergence point and the angle formed by the viewpoints of the first and second virtual cameras 103a and 103b is convergence angle.

According to the embodiment of the present invention, the convergence point may basically coincide with the timing of the streamer 1. Accordingly, the viewer who watches the VR game content for broadcasting can enjoy the panorama image rendered at the viewpoint of the streamer 1, and thus can experience the same experience as playing the game directly.

On the other hand, as shown in FIG. 27, when the streamer 1 is within the predetermined distance from the first virtual object 111 controlled by the streamer 1 during the progress of the broadcast VR game contents, 115) may appear frequently. The second virtual object 115 may be, but is not limited to, a virtual object that is not controlled by the streamer 1, such as a non-player character (NPC).

According to the embodiment of the present invention, when a second virtual object is displayed as shown in FIG. 27, the second virtual object 115 is a convergence point, and the first and second virtual objects 111, The viewpoints of the second virtual cameras 113a and 113b can be adjusted. For example, as shown in FIG. 28, if the convergence angle of the first and second virtual cameras is originally the " convergence angle A ", the second virtual object 115 is used as a convergence point, The viewpoints of the cameras 113a and 113b are adjusted so that the convergence angle can be changed to the " convergence angle B ".

According to the embodiment of the present invention, the viewpoints of the first and second virtual cameras 113a and 113b can be adjusted based on the distance between the first virtual object 111 and the second virtual object 115. [ 29, when the distance between the first virtual object 111 and the second virtual object 115 approaches from "distance B" to "distance C", the first and second virtual objects 111, The convergence angle formed by the cameras 113a and 113b can be made larger to the " convergence angle C ".

According to the embodiment described above, when a second virtual object approaching the first virtual object controlled by the streamer 1 appears during the reproduction of the broadcast VR game content, the panorama rendered automatically centering on the second virtual object Since the video can be provided to the viewer, the immersion feeling of the viewer watching the broadcast VR content can be further improved.

FIG. 30 shows a case where a plurality of virtual objects 123 and 125 appear within a predetermined distance from the first virtual object 121. FIG. In particular, FIG. 30 shows an example in which a second virtual object 123 and a third virtual object 125, which are two virtual objects, are shown for convenience of explanation.

In one embodiment, the viewpoints of the first and second virtual cameras can be adjusted using a virtual object of either the second virtual object 123 or the third virtual object 125 as a convergence point. For example, based on the distance between the first virtual object 121 and each of the virtual objects 123, 125, the virtual object 125 located at a closer distance may be the convergence point of the first and second virtual cameras 125, The viewpoint can be adjusted. In another example, based on the relative importance of two virtual objects on the broadcast VR game content, any one virtual object can be determined as the convergence point. For example, if the second virtual object 123 is a generic virtual object and the third virtual object 125 is an important virtual object (eg, a boss, an important NPC on a scenario, etc.) The viewpoints of the first and second virtual cameras can be adjusted.

In one embodiment, the viewpoints of the first and second virtual cameras may be adjusted using the average coordinates of the position coordinates of the second virtual object 123 and the position coordinates of the third virtual object 125 as convergence points. According to the present embodiment, the panorama image including both the second and third virtual objects 123 and 125 can be rendered and provided to the viewer.

Meanwhile, according to the embodiment of the present invention, the viewpoints of the first and second virtual cameras may be adjusted in response to only some virtual objects of all the virtual objects appearing in the VR game content for broadcasting. For example, the viewpoints of the first and second virtual cameras are adjusted so that only when a virtual object (eg, a boss, a virtual object having a high level of importance such as an important NPC in a scenario) appears in the VR game content for broadcasting It is possible.

Up to now, a method of adjusting a viewpoint of a virtual camera according to an embodiment of the present invention has been described with reference to FIG. 26 to FIG. According to the above-described method, the immersion feeling of the viewer can be further improved.

Hereinafter, the process of streaming the left and right panoramic images will be described with reference to FIG.

31 is a flowchart illustrating a process of streaming left and right panoramic images in the live streaming method according to the fourth embodiment.

Referring to FIG. 31, when the viewer terminal 900 does not correspond to a preset type, the left panoramic image and the right panoramic image are merged, and the merged panoramic image can be streamed (S262, S264, S268). For example, if the viewer terminal 900 is a terminal that does not have a module for displaying left and right panorama images, the merged panorama image can be streamed. Here, the merging of the two panoramic images may be performed right and left, up and down, or merged in any manner.

On the other hand, if the viewer terminal 900 corresponds to a preset type, the left panoramic image and the right panoramic image may be streamed (S266).

Up to now, a live streaming method according to the fourth embodiment of the present invention has been described with reference to FIGS. Hereinafter, an exemplary computing device 600 capable of implementing various devices (e.g., rendering server 300, broadcasting management server 500, and the like) constituting a live streaming system for VR content will be described.

32 is a hardware block diagram of the exemplary computing device 600 of FIG.

32, a computing device 600 includes one or more processors 610, a bus 650, a network interface 670, a memory 630 that loads a computer program 691 performed by the processor 610, And a storage 690 for storing a computer program 691 and a computer program 691. However, only the components related to the embodiment of the present invention are shown in Fig. Therefore, it will be understood by those skilled in the art that other general-purpose components other than the components shown in FIG. 32 may be further included.

The processor 610 controls the overall operation of each configuration of the computing device 600. The processor 610 may comprise a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), a graphics processing unit (GPU), or any type of processor well known in the art . The processor 610 may also perform operations on at least one application or program to perform the method according to embodiments of the present invention. The computing device 600 may include one or more processors.

The memory 630 stores various data, commands, and / or information. The memory 630 may load one or more programs 691 from the storage 690 to perform the method according to some embodiments of the invention. The RAM is shown as an example of the memory 630 in Fig.

The bus 650 provides communication functionality between components of the computing device 600. The bus 650 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The network interface 670 supports wired and wireless Internet communication of the computing device 600. In addition, the network interface 670 may support various communication methods other than Internet communication. To this end, the network interface 670 may comprise a communication module well known in the art.

The storage 690 may store the one or more programs 691 non-temporarily.

The storage 690 may be a nonvolatile memory such as a ROM (Read Only Memory), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a flash memory or the like, a hard disk, a removable disk, And any form of computer-readable recording medium known in the art.

The computer program 691 may be loaded into the memory 630 and include instructions to cause the processor 610 to execute a method in accordance with some embodiments of the present invention.

For example, the computer program 691 may include an instruction to receive game progress information for the first broadcast VR game content executed in the streamer terminal in real time, an instruction to play the game progress information in response to the game progress information being received in real- Instructions for synchronizing a second broadcast VR game content with the first broadcast VR game content, instructions for rendering a 360 degree panorama image based on the synchronized second broadcast VR game content, and instructions for rendering the rendered 360 degree panorama image To the viewer terminal.

In another example, the computer program 691 may include instructions for placing a first virtual camera in a virtual space built on broadcast VR content, instructions for placing a second virtual camera at a location spaced a predetermined distance from the first virtual camera in the virtual space, An instruction to render a left eye panorama image based on the image photographed by the first virtual camera; an instruction to render the right eye panorama image on the basis of the image photographed by the second virtual camera; And an instruction to stream each of the left eye panorama image and the right eye panorama image to the viewer terminal.

In another example, the computer program 691 may include an instruction to receive a streaming request for the VR content for broadcasting from a streamer terminal for streaming broadcasting for the VR content for broadcasting, And an instruction to stream 360-degree panoramic images of the VR content for broadcasting to a viewer terminal using the broadcasting platform.

The concepts of the invention described above with reference to Figures 1 to 32 can be implemented in computer readable code on a computer readable medium. The computer readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) . The computer program recorded on the computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

Although the operations are shown in the specific order in the figures, it should be understood that the operations need not necessarily be performed in the particular order shown or in a sequential order, or that all of the illustrated operations must be performed to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various configurations in the above-described embodiments should not be understood as such a separation being necessary, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products .

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, I can understand that. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (20)

1. A second broadcast VR game content that is the same as the first broadcast VR game content executed in the streamer terminal, and that uses the game progress information on the first broadcast VR game content provided from the streamer terminal, A rendering server for synchronizing the broadcast VR game content with the first broadcast VR game content and rendering a 360 degree panorama image based on the synchronized second broadcast VR game content; And

   And a broadcast management server for allowing the rendered 360 degree panorama image to be streamed to a viewer terminal.

   Live streaming system for VR content.
2. The method according to claim 1,

   The rendering server,

   Rendering the 360-degree panoramic image using an image photographed by a virtual camera positioned adjacent to a virtual object in a virtual space built on the second broadcast VR game content,

   The virtual object includes:

   A virtual object controlled according to an input of the streamer,

   The virtual camera includes:

   Wherein the virtual object is moved along with the virtual object in the virtual space.

   Live streaming system for VR content.
3. The method according to claim 1,

   The rendering server,

   And transmitting a first partial panorama image corresponding to the viewpoint information and the viewing angle information from the rendered 360 degree panorama image to the broadcast management server in response to receipt of viewpoint information and viewing angle information of the viewer from the viewer terminal As a result,

   Live streaming system for VR content.
4. The method of claim 3,

   The rendering server,

   And transmits the first partial panorama image corresponding to the changed viewpoint information among the rendered 360 degree panorama images to the broadcast management server in response to reception of the changed viewpoint information of the viewer from the viewer terminal.

   Game content streaming system.
5. The method of claim 3,

   The rendering server,

   A first partial panorama image encoding unit that encodes a first region corresponding to a front view point of the viewer with a first image quality and a second region that is not the first region with a second image quality,

   Wherein the first image quality is higher quality than the second image quality.

   Live streaming system for VR content.
6. The method of claim 3,

   The rendering server,

   Further transmitting a second partial panorama image of a preset size adjacent to the first partial panorama image and located in an invisible region in the rendered 360 degree panorama image,

   Wherein the first partial panorama image is an image encoded with a first image quality, the second partial panorama image is an image encoded with a second image quality,

   Wherein the first image quality is higher quality than the second image quality.

   Live streaming system for VR content.
7. The method according to claim 1,

   The rendered 360 degree panorama image is a first 360 degree panorama image,

   The streamer terminal,

   Generating a second 360 degree panorama image rendered based on a capture screen for the first broadcast VR game content,

   And the panorama image of any one of the first 360 degree panorama image and the second 360 degree panorama image is streamed to the viewer terminal.

   Live streaming system for VR content.
8. 8. The method of claim 7,

   The second 360 degree panorama image is a panorama image,

   When the first broadcast VR game content is not in progress according to an input of the streamer, the stream is streamed to the viewer terminal.

   Live streaming system for VR content.
9. 8. The method of claim 7,

   The streamer terminal,

   Wherein the converted capture screen is arranged in a first area corresponding to the center of a 360 degree image area and a second area in the 360 degree image area, which is not the first area, And the second 360 degree panorama image is generated by processing the first 360 degree panorama image with a predetermined color.

   Live streaming system for VR content.
10. A broadcast web server for providing a web page service to a streamer for streaming broadcast virtual reality (VR) contents at a streamer terminal; And

    And a broadcasting management server for broadcasting the 360-degree panoramic image of the VR content for broadcasting to the viewer terminal using the pre-built Internet broadcasting platform in response to the streaming request of the streamer terminal.

    Live streaming system for VR content.
11. 11. The method of claim 10,

    Wherein the broadcast VR content is a first broadcast VR game content being executed by the streamer terminal,

    And a rendering server for rendering the 360-degree panoramic image,

    The rendering server,

    A content synchronization module for synchronizing a second broadcast VR game content identical to the first broadcast VR game content and being executed in the rendering server with the first broadcast VR game content in response to receiving game progress information from the streamer terminal, ,

    And a panorama image rendering module for rendering the 360-degree panorama image based on the second broadcasting VR game content.

    Live streaming system for VR content.
12. 12. The method of claim 11,

    Wherein the panorama image rendering module comprises:

    Rendering the 360-degree panoramic image using an image captured by a virtual camera disposed adjacent to a virtual object in a virtual space constructed on the second broadcast VR game content,

    The virtual object includes:

    A virtual object controlled according to an input of the streamer,

    The virtual camera includes:

    Wherein the virtual object is moved along with the virtual object in the virtual space.

    Live streaming system for VR content.
13. 11. The method of claim 10,

    The broadcast web server includes:

    A member information management module for managing information on a streamer member;

    An authentication module for authenticating the streamer using information about the streamer member; And

    And a billing module for performing a billing process in response to the request to purchase a broadcast right in response to a request for purchase of a broadcast right of the streamer terminal,

    And the streaming request of the streamer terminal that has not purchased the broadcasting right is blocked.

    Live streaming system for VR content.
14. 11. The method of claim 10,

    The streamer terminal,

    Receiving a selection input for at least one streaming Internet broadcasting platform for streaming broadcasting for the broadcasting VR content among the established Internet broadcasting platforms,

    The broadcast management server includes:

    And a plurality of broadcast client modules linked with the pre-established Internet broadcast platform,

    Selecting at least one broadcast client module operable with the at least one internet broadcasting platform for streaming among the plurality of broadcast client modules based on the selection input,

    And the 360 degree panorama image is streamed using the selected at least one broadcast client module.

    Live streaming system for VR content.
15. A first virtual camera is disposed in a virtual space constructed on a broadcast VR content, a second virtual camera is disposed at a position spaced apart from the first virtual camera by a predetermined interval, and a video captured by the first virtual camera Wherein the left panoramic image and the right panoramic image are rendered on a right eye panorama image based on an image photographed by the second virtual camera, wherein the left panoramic image and the right eye panorama image are 360-degree panoramic images, ; And

    And a broadcast management server for allowing each of the left eye panorama image and the right eye panorama image to be streamed to a viewer terminal.

    Live streaming system for VR content.
16. 16. The method of claim 15,

    Wherein the first virtual camera and the second virtual camera,

    Wherein the first virtual object is located adjacent to the first virtual object existing in the virtual space,

    Wherein the first virtual object comprises:

    It is a virtual object that is controlled according to streamer input.

    Wherein the first virtual camera and the second virtual camera,

    Wherein the first virtual object is moved along with the first virtual object in the virtual space.

    Live streaming system for VR content.
17. 17. The method of claim 16,

    In response to a determination that a second virtual object exists within a predetermined distance based on the first virtual object in the virtual space, the second virtual object is set as a convergence point, and the first virtual camera and the second virtual camera Is adjusted,

    Wherein a convergence angle between the first virtual camera and the second virtual camera increases as the distance between the first virtual object and the second virtual object becomes closer.

    Live streaming system for VR content.
18. 18. The method of claim 17,

    When a second virtual object and a third virtual object exist within a predetermined distance based on the first virtual object in the virtual space,

    Wherein the viewpoints of the first virtual camera and the second virtual camera are adjusted by using the average coordinates of the position coordinates of the second virtual object and the position coordinates of the third virtual object as convergence points.

    Live streaming system for VR content.
19. 16. The method of claim 15,

    If the viewer terminal does not correspond to the pre-designated type,

    The rendering server,

    The panorama image, the left panorama image, and the right panorama image,

    The broadcast management server includes:

    So that the merged 360 degree panorama image is streamed.

    Live streaming system for VR content.
20. 16. The method of claim 15,

    The rendering server,

    Extracting a first partial panorama image corresponding to the viewpoint information and the viewing angle information from the left panorama image in response to receipt of viewpoint information and viewing angle information of the viewer from the viewer terminal, Extracting a second partial panorama image corresponding to the viewing angle information,

    The broadcast management server includes:

    The first partial panorama image and the second partial panorama image are streamed to the viewer terminal.

    Live streaming system for VR content.

Images