KR102568415B1 - HMD-based PC game expansion system - Google Patents

Abstract

The present invention relates to an HMD-based PC game expansion system, comprising: an expansion server that provides virtualization for PC game execution, captures game data according to the execution, and transmits the captured game data to an HMD terminal through a wireless network; and an HMD terminal that receives the game data from the expansion server and reproduces corrected game video and sound according to a control signal, wherein the control signal includes an orientation signal and a controller signal of the HMD terminal. includes

Description

HMD-based PC game expansion system

The present invention relates to a game expansion technology, and more particularly, to an HMD-based PC game expansion system that virtualizes and executes a high-end PC game on a server and wirelessly transmits it to an interlocked HMD terminal so that high-quality games can be easily used. it's about

A head-mounted display is a display device that can present an image directly in front of the user's eyes by attaching it to the head. In 1968, Ivan Sutherland of the University of Utah made the first HMD, and in the case of the first HMD, 3D graphics were displayed on monitors mounted on both eyes.

Since the use of HMD is to enjoy or execute high-definition video, virtual practice and games using virtual reality techniques in stereoscopic images, technology for real-time transmission and playback of large-capacity contents is absolutely necessary.

In addition, wireless communication should be smooth so as not to restrict a user's activity at the same time, and an HMD that is continuously attached to the head during content playback time needs to be implemented lightly and small.

In addition, in the case of using a short-distance wireless communication technology such as WiFi, a data signal must be processed according to a protocol, and for this purpose, the data must be compressed, packetized, and transmitted. In the HMD, a codec process of storing the transmitted signal as a data packet in memory and decompressing the received signal is required. In order to speed up the codec, high-speed encoders and decoders may be required.

Korean Patent Publication No. 10-2017-0058758 (2017.05.29)

An embodiment of the present invention is to provide an HMD-based PC game expansion system that can easily use high-quality games by virtualizing and executing high-end PC games on a server and wirelessly transmitting them to an interlocked HMD terminal.

Among embodiments, an HMD-based PC game extension system includes an extension server that provides virtualization for PC game execution, captures game data according to the execution, and transmits the captured game data to the HMD terminal through a wireless network; and an HMD terminal that receives the game data from the expansion server and reproduces corrected game video and sound according to a control signal, wherein the control signal includes an orientation signal and a controller signal of the HMD terminal. includes

The expansion server may include a virtualization unit that creates a container corresponding to the PC game, configures virtualization resources according to specifications of the PC game, and allocates them to the container; a capture unit that extracts a rendered image while the PC game is being executed on the container and captures the game image and sound as the game data based on the profile information of the HMD terminal; an encoder unit generating encoded data by encoding the game data according to a streaming method; and a first network unit transmitting the encoded data to the HMD terminal through the wireless network.

The capture unit includes a DLL insertion module for inserting a Dynamic Link Library (DLL) into a process of the PC game to extract the game image and sound; a post-processing module that performs a post-processing operation of changing the type and resolution of the rendered image or applying at least one filter according to the profile information; and a capture module that captures the game image and sound from the rendered image to which the post-processing operation is applied.

The encoder unit may determine a bit rate and an encoding method according to the streaming method and change an encoding option according to a load condition of the wireless network.

The HMD terminal includes a second network unit receiving the encoded data from the expansion server through the wireless network; a decoder unit which decodes the encoded data to generate decoded data and stores the decoded data in a texture buffer; a rendering unit that updates the display screen by extracting the decoded data from the texture buffer according to the screen refresh rate; and a control data processing unit that receives the direction signal, the position signal, and the control signal and transmits them to the rendering unit and the extension server, respectively.

The rendering unit may remove screen shake due to latency by correcting a current frame of the display screen using signals received from the control data processing unit.

The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all of the following effects or only the following effects, so it should not be understood that the scope of rights of the disclosed technology is limited thereby.

The HMD-based PC game expansion system according to an embodiment of the present invention virtualizes and executes a high-end PC game on a server and wirelessly transmits it to an interlocked HMD terminal so that high-quality games can be easily used.

1 is a diagram illustrating a game expansion system according to the present invention.
FIG. 2 is a diagram explaining the system configuration of the expansion server of FIG. 1 .
3 is a diagram illustrating the functional configuration of a game expansion system according to the present invention.
FIG. 4 is a diagram explaining the functional configuration of the capture unit of FIG. 3 .
5 is a diagram illustrating the operation of an expansion server according to the present invention.
6 is a diagram explaining the operation of the HMD terminal according to the present invention.
7 is a diagram illustrating an embodiment of a game expansion system according to the present invention.

Since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

Meanwhile, the meaning of terms described in this application should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may exist in the middle. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

Expressions in the singular number should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as “comprise” or “having” refer to an embodied feature, number, step, operation, component, part, or these. It should be understood that it is intended to indicate that a combination exists, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identification code (eg, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step clearly follows a specific order in context. Unless otherwise specified, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as consistent with meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

1 is a diagram illustrating a game expansion system according to the present invention.

Referring to FIG. 1 , the game extension system 100 may include an HMD device 110 , an extension server 130 and a database 150 .

The HMD terminal 110 may correspond to a head mounted display device. The HMD device 110 may receive game data from the expansion server 130 and reproduce corrected game images and sounds according to a user's control signal. To this end, the HMD device 110 may be connected to the extension server 130 through a wireless network, and a plurality of HMD devices 110 may be connected to the extension server 130 at the same time.

The expansion server 130 may be implemented as a computing device capable of virtualizing a PC game and transmitting game data according to an execution result to the HMD terminal 110 or as an independent server device performing the same. The expansion server 130 may interwork with the HMD terminal 110 through a wireless network, and the wireless network may include Bluetooth, WiFi, LTE, and the like. In addition, the expansion server 130 may operate in conjunction with a separate external system to provide additional functions. For example, the expansion server 130 may be implemented to provide an independent function in the service provision process in conjunction with a referral system such as an authentication server for user authentication and a payment server for service payment.

The database 150 may correspond to a storage device for storing various information necessary for the operation process of the expansion server 130 . For example, the database 150 may store information about a high-end PC game and store virtualization information for execution of a PC game, but is not necessarily limited thereto, and the expansion server 130 may store the HMD terminal ( 110), information collected or processed in various forms can be stored.

Meanwhile, in FIG. 1, the database 150 is shown as a device independent of the expansion server 130, but is not necessarily limited thereto, and is included in the expansion server 130 as a logical storage device of the expansion server 130. Of course, it can be implemented.

FIG. 2 is a diagram explaining the system configuration of the expansion server of FIG. 1 .

Referring to FIG. 2 , the expansion server 130 may be implemented by including a processor 210 , a memory 230 , a user input/output unit 250 and a network input/output unit 270 .

The processor 210 may execute a procedure for processing each step in the operation of the expansion server 130, manage the memory 230 that is read or written throughout the process, and the memory 230 You can schedule synchronization time between volatile memory and non-volatile memory in . The processor 210 can control the overall operation of the expansion server 130, and is electrically connected to the memory 230, the user input/output unit 250, and the network input/output unit 270 to control data flow between them. there is. The processor 210 may be implemented as a central processing unit (CPU) of the expansion server 130 .

The memory 230 is implemented as a non-volatile memory such as a solid state drive (SSD) or a hard disk drive (HDD) and may include auxiliary storage used to store all data necessary for the expansion server 130, and RAM. It may include a main memory device implemented as a volatile memory such as (Random Access Memory).

The user input/output unit 250 may include an environment for receiving user input and an environment for outputting specific information to the user. For example, the user input/output unit 250 may include an input device including an adapter such as a touch pad, a touch screen, an on-screen keyboard, or a pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input/output unit 250 may correspond to a computing device connected through a remote connection, and in such a case, the expansion server 130 may be implemented as an independent server.

The network input/output unit 270 includes an environment for connecting to an external device or system through a network, and includes, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and a VAN ( An adapter for communication such as Value Added Network) may be included.

3 is a diagram illustrating the functional configuration of a game expansion system according to the present invention.

Referring to FIG. 3 , the game extension system 100 may operate through interworking between the HMD device 110 and the extension server 130 . Here, the expansion server 130 may include a virtualization unit 310, a capture unit 320, an encoder unit 330, and a first network unit 340, and the HMD terminal 110 may include a second network unit. 350, a decoder unit 360, a rendering unit 370, and a control data processing unit 380 may be included.

More specifically, the virtualization unit 310 may generate a container corresponding to the PC game, configure virtualization resources according to specifications of the PC game, and allocate the virtualized resources to the container. That is, the virtualization unit 310 may create a virtualized container to create a virtual space for independently executing a high-end PC game on the expansion server 130 . In addition, the virtualization unit 310 may create instances of computing resources required for smooth execution based on specification information of the PC game and place them in containers as virtualization resources. In this case, computing resources for virtualization may include a network, a CPU, a GPU, a memory, a DB, and the like.

Meanwhile, the virtualization unit 310 may operate in conjunction with a separate cloud server to perform virtualization operations for execution of PC games. That is, the virtualization unit 310 may request virtualization for game execution to the cloud server according to a request from the HMD terminal 110, and as a response, may execute a PC game using virtualized resources.

In one embodiment, the virtualization unit 310 may configure a virtualization environment for game execution according to a virtualization scenario in response to a game execution request from the HMD terminal 110, and a network connection state with the HMD terminal 110. , The virtualization environment may be reconfigured to an appropriate size according to the requested game specifications and the profile information of the HMD device 110 .

The capture unit 320 may extract a rendered image while the PC game is being executed on the container and capture the game image and sound as game data based on the profile information of the HMD terminal 110 . To this end, the capture unit 320 may be implemented by including independent modules. Specifically, the capture unit 320 may be implemented by including a DLL insertion module, a post-processing module, and a capture module, which will be described in more detail with reference to FIG. 4 .

The encoder unit 330 may generate encoded data by encoding game data according to a streaming method. The extension server 130 and the HMD device 110 may interwork through a wireless network, and a data compression process may be performed to transmit high-quality images at high speed. The encoder unit 330 may determine a predetermined encoding method according to a streaming method through a wireless network. In addition, the encoder unit 330 may dynamically determine an encoding method in consideration of a network environment, load conditions of the extension server 130 and the HMD device 110, and the like.

In one embodiment, the encoder unit 330 may determine a bit rate and an encoding method according to a streaming method and change an encoding option according to a load condition of a wireless network. For example, encoding schemes may include H.264 (AVC) and H.265 (HEVC), and may be divided into tile-based encoding and slice-based encoding. Also, the encoder unit 330 may change an encoding option according to a network load condition. Encoding options include the Adaptive Bitrate Control method, which applies a bitrate suitable for the network condition, and the Adaptive Intra Refresh method, which applies a refresh cycle and refresh time suitable for the network condition. can include

The first network unit 340 may transmit encoded data to the HMD terminal 110 through a wireless network. More specifically, the first network unit 340 may divide the encoded frame data into packet data according to a wireless network protocol and transmit the divided data. The first network unit 340 may separately process streaming data and control signal data, and may selectively use a transmission protocol suitable for a wireless network.

Meanwhile, the first network unit 340 may perform data transmission using multi-channels according to the state of the network. For example, the first network unit 340 may increase the number of channels used for data transmission when the load of the network is high. Conversely, the first network unit 340 may reduce the number of channels used for data transmission when the network load is low. In addition, the first network unit 340 may provide a synchronization function between multiple channels when using multiple channels.

In one embodiment, the first network unit 340 may be implemented by including a NAL unit. A NAL unit may correspond to part of the H.264/AVC and HEVC video coding standards. That is, the first network unit 340 is implemented as a network abstraction layer unit corresponding to the encoder method of the encoder unit 330 and can efficiently transmit game images to the HMD device 110 .

The second network unit 350 may receive encoded data from the expansion server 130 through a wireless network. That is, the second network unit 350 may assemble the received data to match the frame information. In the case of multi-channel transmission, the second network unit 350 may assemble data received through each channel according to frame information. In addition, the second network unit 350 may separate and process streaming data and control signal data. The second network unit 350 may selectively use a transmission protocol suitable for a wireless network, and may use the same transmission protocol as the first network unit 340 .

In one embodiment, the second network unit 350 may be implemented by including a NAL parser. The NAL parser may correspond to part of the H.264/AVC and HEVC video coding standards. The second network unit 350 may process the NAL header of the NAL unit through the NAL parser, extract remaining data, and transmit the extracted data to the decoder unit 360 .

The decoder unit 360 may decode the encoded data to generate decoded data and store the decoded data in a texture buffer. The decoder unit 360 may decode and restore encoded data received and assembled by the second network unit 350 . The reconstructed image data may be textured and then stored in a texture buffer. When the texture buffer is full, the rendering unit 370 may be notified that generation of an image for the next frame has been completed. Meanwhile, the decoder unit 360 may decode sound separately from the game video, and the decoded sound may be played along with the game video.

The rendering unit 370 may update the display screen by extracting decoded data from the texture buffer according to the screen refresh rate. When the texture buffer is full, the rendering unit 370 may receive a signal related thereto and perform an operation of updating the display screen according to the screen refresh rate of the HMD device 110 . The game image of the next frame stored in the texture buffer may be played by the screen update operation.

In one embodiment, the rendering unit 370 may remove screen shaking due to latency by correcting the current frame of the display screen using the signals received from the control data processing unit 380 . An error between an actual playback screen and a current user movement may occur due to a delay caused by data transmission between the server and the terminal, and accordingly, a flickering phenomenon may occur in a screen played on the HMD terminal 110 . The rendering unit 370 may correct the current frame based on tracking information about the user's movement in order to remove the screen shake.

The control data processing unit 380 may receive a direction signal, a position signal, and a control signal and transmit them to the rendering unit 370 and the expansion server 130 , respectively. That is, the control data processing unit 380 may collect movement information of the HMD device 110 according to the user's movement and control information of a controller according to the user's control, and the movement information is the direction of the HMD device 110. It may include orientation information and position information.

Meanwhile, the expansion server 130 controls the current direction and position of the HMD device 110 based on the previous end-to-end latency based on the user's movement information provided by the control data processing unit 380. can be predicted and applied to the process of game operation and rendering image generation. In addition, the expansion server 130 predicts the current position of the controller based on the previous end-to-end latency based on the position information of the controller provided by the control data processing unit 380 to control and control the game. Each can be applied to the rendering image generation process.

FIG. 4 is a diagram explaining the functional configuration of the capture unit of FIG. 3 .

Referring to FIG. 4 , the extension server 130 may capture game data according to game execution and transmit it to the HMD terminal 110 through a wireless network. At this time, the capture unit 320 of the extension server 130 may be implemented by including a plurality of modules to perform a related operation. More specifically, the capture unit 320 may include a DLL insertion module 410 , a post-processing module 420 and a capture module 430 .

The DLL insertion module 410 may insert a dynamic link library (DLL) into a PC game process to extract game images and sounds. DLL injection may correspond to an operation of forcibly inserting a DLL file for extracting images and sounds into a game process. The DLL insertion module 410 may insert a DLL file into the memory space allocated to the PC game process, and through this, the capture unit 320 may extract game images or sounds during the PC game execution process.

The post-processing module 420 may perform a post-processing operation of changing the type and resolution of a rendered image or applying at least one filter according to profile information. For example, the post-processing module 420 may receive setting information of the HMD terminal 110 as profile information and generate a 3D image as a rendered image. In addition, the post-processing module 420 may generate a rendered image according to the game resolution, and may apply various filters as needed.

The capture module 430 may capture a game image and sound from a rendered image to which a post-processing operation is applied. The capture module 430 may designate an area corresponding to a specific direction and a specific viewing angle in the rendered image and capture it as a game image, and may also store sound data at the time of the capture.

5 is a diagram illustrating the operation of an expansion server according to the present invention.

Referring to FIG. 5 , the expansion server 130 may provide virtualization for PC game execution, and may execute the PC game using virtualized resources. The expansion server 130 may capture and transmit game images and sounds to the HMD terminal 110 while the PC game is being executed.

More specifically, the extension server 130 may insert a DLL file into the game process through DLL injection (step S510). DLL files can be used to capture game images and sounds during game execution. The expansion server 130 may process the rendered image into a desired form through a post-processing operation (step S530), and may capture video and audio for transmission to the HMD terminal 110 (step S550).

In addition, the expansion server 130 may encode the captured video and audio data in a method suitable for streaming (step S570). Finally, the extension server 130 may divide the encoded video into packet data of a protocol and transmit (step S590).

6 is a diagram explaining the operation of the HMD terminal according to the present invention.

Referring to FIG. 6 , the HMD device 110 may receive game data from the extension server 130 and reproduce corrected game images and sounds according to control signals.

More specifically, the HMD device 110 may receive data captured by the expansion server 130 through a wireless network (step S610). The HMD device 110 may assemble the received data according to frame information. If multi-channel transmission is used, the HMD terminal 110 may assemble data received for each channel according to frame information. In addition, the HMD terminal 110 may independently process streaming data and control signal data.

The HMD device 110 may decode and restore the assembled streaming data (step S630). Thereafter, the decoded image data may be textured and stored in a texture buffer (step S650), and a preparation signal for a new frame may be transmitted to the rendering unit. When the video for the next frame is prepared through the rendering unit, the HMD terminal 110 can reproduce the game video by updating the display screen according to the screen refresh rate (step S690). At this time, the HMD device 110 may correct the current frame based on the tracking information on the user's movement.

The game expansion system 100 according to the present invention may be configured as shown in FIG. 7, and may virtualize a high-definition PC game and transmit it to the HMD device wirelessly (eg, WiFi or 5G Mobile Network). Through this, users can wirelessly link games that run only on high-end equipment and freely use high-quality games at their own location.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: game expansion system
110: HMD terminal 130: Expansion server
150: database
210: processor 230: memory
250: user input/output unit 270: network input/output unit
310: virtualization unit 320: capture unit
330: encoder unit 340: first network unit
350: second network unit 360: decoder unit
370: rendering unit 380: control data processing unit
410: DLL insertion module 420: post-processing module
430: capture module

Claims (6)

An expansion server that provides virtualization for PC game execution, captures game data according to the execution, and transmits the captured game data to the HMD terminal through a wireless network; and
An HMD terminal for receiving the game data from the expansion server and reproducing corrected game video and sound according to a control signal;
The extension server
A container corresponding to the PC game is created and a virtualization environment for game execution is configured according to the virtualization scenario for the PC game, but the network connection state with the HMD terminal, the recommended specifications of the PC game, and the profile information of the HMD terminal a virtualization unit that reconfigures virtualization resources for the virtualization environment and allocates them to the container;
a capture unit that extracts a rendered image while the PC game is being executed on the container and captures the game image and sound as the game data based on the profile information of the HMD terminal;
an encoder unit generating encoded data by encoding the game data according to a streaming method; and
A first network unit transmitting the encoded data to the HMD terminal through the wireless network;
the capture unit
A DLL insertion module for accessing a memory space allocated to the process of the PC game among the virtualization resources to extract the game image and sound, and inserting a Dynamic Link Library (DLL) into the process;
a post-processing module that performs a post-processing operation of changing the type and resolution of the rendered image or applying at least one filter according to the profile information; and
A capture module for capturing the game image and sound from the rendered image to which the post-processing operation is applied;
The HMD-based PC game extension system, characterized in that the control signal includes an orientation signal and a controller signal of the HMD terminal.
delete delete The method of claim 1, wherein the encoder unit
HMD-based PC game extension system, characterized in that the bit rate and the encoding method are determined according to the streaming method and the encoding option is changed according to the load condition of the wireless network.
The method of claim 1, wherein the HMD terminal
a second network unit receiving the encoded data from the expansion server through the wireless network;
a decoder unit which decodes the encoded data to generate decoded data and stores the decoded data in a texture buffer;
a rendering unit that updates the display screen by extracting the decoded data from the texture buffer according to the screen refresh rate; and
The HMD-based PC game extension system comprising a; control data processing unit which receives the direction signal, the position signal and the controller signal and transfers them to the rendering unit and the expansion server, respectively.
The method of claim 5, wherein the rendering unit
The HMD-based PC game extension system, characterized in that, by using the signals received from the control data processor to correct the current frame of the display screen to remove screen shake due to latency.