KR102543136B1 - Cloud vr-based 3d image provision method - Google Patents

Abstract

The present invention relates to a cloud VR-based 3D image providing method, comprising the steps of executing a game implemented to provide a 2D image; hooking and storing rendering data of the 2D image during the execution of the game; generating the 2D image based on the rendering data; obtaining depth information of the 2D image; determining positions of vertices of a pair of binocular images based on the depth information; rendering the binocular images respectively using the rendering data based on the positions of the vertices; and generating a single 3D image by integrating the images of both eyes.

Description

Cloud VR-based 3D image provision method {CLOUD VR-BASED 3D IMAGE PROVISION METHOD}

The present invention relates to a 3D image generation technology, and more particularly, to a cloud VR-based 3D image providing method capable of converting a 2D image of a general PC game into a 3D image and providing the same.

When a general PC game is executed through an HMD device, the game image is only played through a virtual monitor in front of the user's eyes, and it is difficult to provide a realistic three-dimensional effect to the user. Although there are games that support VR mode, it is not easy to play general PC games realistically in a cloud VR environment.

The 2D video image generated through the execution of the game corresponds to a flat image of the objects in the scene observed from a single point of view, and since it cannot express the depth information of each object, it has a three-dimensional effect. It is difficult to express the image.

To generate a 3D image, a method of converting an object into a 3D image using two or more images taken from different angles or calculating depth information of an object using a preset database may be used. .

However, in this method, it is very difficult to provide 3D images through the execution of a general PC game in that it is necessary to use a special shooting device using two or more cameras or to store a database suitable for each image in advance. .

Korean Patent Registration No. 10-1054043 (2011.07.28)

An embodiment of the present invention is to provide a cloud VR-based 3D video providing method capable of converting a 2D video of a general PC game into a 3D video and providing the same.

An embodiment of the present invention provides a more immersive VR image while improving the problem of conventional parallax mapping by moving the geometry itself like a parallax mapping algorithm using the z value among the positional information of vertices for each of the binocular images. It is intended to provide a cloud VR-based 3D image providing method that can be used.

Among the embodiments, it relates to a cloud VR-based 3D image providing method, comprising: executing a game implemented to provide a 2D image; hooking and storing rendering data of the 2D image during the execution of the game; generating the 2D image based on the rendering data; obtaining depth information of the 2D image; determining positions of vertices of a pair of binocular images based on the depth information; rendering the binocular images respectively using the rendering data based on the positions of the vertices; and generating a single 3D image by integrating the images of both eyes.

The executing may include executing the game through a VR device.

The storing may include hooking scene data including vertex information, texture information, and shader information of the 2D image as the rendering data.

The determining step may include determining positions of vertices of the 2D image by moving the vertices of the 2D image by an offset based on an eye vector of each of both eyes using the z value related to the depth information. can include

The storing step includes generating two rendering data sets by copying the rendering data, and the determining step includes matching the two rendering data sets to the left eye and the right eye, respectively, and then determining the positions of the vertices independently. It may include the step of determining as.

The generating of the 3D image may include generating a frame image of the 3D image by merging identical pixels in each frame of the binocular image.

The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all of the following effects or only the following effects, so it should not be understood that the scope of rights of the disclosed technology is limited thereby.

A cloud VR-based 3D image providing method according to an embodiment of the present invention may convert a 2D image of a general PC game into a 3D image and provide the converted 3D image.

The cloud VR-based 3D image providing method according to an embodiment of the present invention moves the geometry itself like a parallax mapping algorithm using the z value among the positional information of vertices for each of the binocular images, thereby solving the problem of conventional parallax mapping. While improving, it is possible to provide a more immersive VR image.

1 is a diagram illustrating an image providing system according to the present invention.
FIG. 2 is a diagram explaining the system configuration of the video providing server of FIG. 1 .
FIG. 3 is a diagram explaining the functional configuration of the video providing server of FIG. 1 .
4 is a flowchart illustrating a cloud VR-based 3D video providing method according to the present invention.
5 is a diagram explaining a method of generating a 3D image through parallax mapping.
6 is a diagram for explaining an embodiment of a 3D image providing method according to the present invention.

Since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

Meanwhile, the meaning of terms described in this application should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may exist in the middle. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

Expressions in the singular number should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as “comprise” or “having” refer to an embodied feature, number, step, operation, component, part, or these. It should be understood that it is intended to indicate that a combination exists, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identification code (eg, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step clearly follows a specific order in context. Unless otherwise specified, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as consistent with meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

1 is a diagram illustrating an image providing system according to the present invention.

Referring to FIG. 1 , an image providing system 100 may include a VR device 110 and an image providing server 130 .

The VR device 110 may correspond to a user terminal capable of reproducing AR/VR images. Here, the VR device 110 may be basically implemented as a Head Mounted Display (HMD) terminal, but is not necessarily limited thereto, and may be implemented as various devices capable of reproducing VR/AR images, of course. The VR device 110 may be connected to the video providing server 130 through a network to exchange data.

In addition, the VR device 110 may be implemented to operate through the cloud VR-based 3D image providing method according to the present invention. To this end, the VR device 110 may operate in conjunction with the video providing server 130 . That is, the VR device 110 may execute a game implemented to provide a 2D video by interworking with the video providing server 130, and during the execution of the game, specific operations for providing a 3D video based on cloud VR according to the present invention may be performed. can be performed.

For example, the VR device 110 may generate a game execution request signal and transmit it to the video providing server 130, and the video providing server 130 receives the request signal, executes a corresponding game, and executes the game. After converting the 2D game image according to the 3D image, it can be transmitted to the VR device 110 . At this time, the video providing server 130 may transmit the converted 3D video to the VR device 110 after encoding, and the VR device 110 receives the 3D video, decodes it, and then can be played

In one embodiment, the VR device 110 may be implemented by including 6 Degrees of Freedom (DoF) sensors related to user motion information. In addition, the VR device 110 may be implemented by further including various sensors as needed. For example, the VR device 110 may further include a GPS sensor, a motion sensor, and the like. In another embodiment, the VR device 110 may receive motion information of the user from an external 6 DoF sensor.

The video providing server 130 may be implemented as a server corresponding to a computer or program that converts a 2D video of a game executed in the VR device 110 into a 3D video and transmits the video through a network. In addition, the video providing server 130 may receive a request from the VR device 110 to directly execute the game, automatically convert the corresponding game video into a 3D video and transmit it to the VR device 110. The video providing server 130 may be connected to the VR device 110 through a wireless network such as Bluetooth, WiFi, or 5G communication, and may transmit/receive data with the VR device 110 through the network.

In addition, the video providing server 130 may receive a 6 DoF signal as the user's movement information from the VR device 110, and based on this, generate an image in response to the user's movement and transmit it to the VR device 110. there is. To this end, the video providing server 130 may install and run a remote application that operates in conjunction with the VR device 110.

FIG. 2 is a diagram explaining the system configuration of the video providing server of FIG. 1 .

Referring to FIG. 2 , the image providing server 130 may be implemented by including a processor 210, a memory 230, a user input/output unit 250 and a network input/output unit 270.

The processor 210 may execute a procedure for processing each step in the process of the image providing server 130 operating, and manage the memory 230 read or written throughout the process, and the memory 230 ), you can schedule the synchronization time between volatile memory and non-volatile memory. The processor 210 can control the overall operation of the image providing server 130, and is electrically connected to the memory 230, the user input/output unit 250, and the network input/output unit 270 to control data flow between them. can The processor 210 may be implemented as a central processing unit (CPU) of the image providing server 130 .

The memory 230 may include a secondary storage device implemented as a non-volatile memory such as a solid state drive (SSD) or a hard disk drive (HDD) and used to store all data necessary for the image providing server 130, It may include a main memory implemented as a volatile memory such as RAM (Random Access Memory).

The user input/output unit 250 may include an environment for receiving user input and an environment for outputting specific information to the user. For example, the user input/output unit 250 may include an input device including an adapter such as a touch pad, a touch screen, an on-screen keyboard, or a pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input/output unit 250 may correspond to a computing device connected through a remote connection, and in such a case, the image providing server 130 may be performed as an independent server.

The network input/output unit 270 includes an environment for connecting to an external device or system through a network, and includes, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and a VAN ( An adapter for communication such as Value Added Network) may be included.

FIG. 3 is a diagram explaining the functional configuration of the video providing server of FIG. 1 .

Referring to FIG. 3 , the image providing server 130 includes a game execution unit 310, a rendering data storage unit 320, a 2D image generator 330, a depth information acquisition unit 340, and a vertex adjustment unit 350. , a binocular image rendering unit 360, a 3D image generator 370, and a controller (not shown in FIG. 3).

The game execution unit 310 may execute a game implemented to provide a 2D image. That is, the game may basically correspond to a general PC game, but is not necessarily limited thereto, and may include a game in which a game image is generated as a 2D flat image. In addition, the game execution unit 310 may execute media files, programs, applications, etc. that provide 2D flat images in addition to games.

In one embodiment, the game execution unit 310 may execute the game through the VR device 110 . The game execution unit 310 corresponds to one component of the video providing server 130, and the operation of the game execution unit 310 may be executed under the control of the video providing server 130. Accordingly, game execution by the game executor 310 may correspond to an operation on the video providing server 130 . However, the game execution unit 310 may operate in conjunction with the VR device 110, and accordingly, the game may be executed through the VR device 110. Meanwhile, the VR device 110 may operate independently of the video providing server 130 and may process game execution according to its own control process.

The rendering data storage unit 320 may hook and store rendering data of a 2D image during a game execution process. The rendering data storage unit 320 may secure basic data necessary for generating a 3D image by hooking various rendering information required for image reproduction during a game execution process. The rendering data storage unit 320 may store acquired data in a database.

In one embodiment, the rendering data storage unit 320 may hook scene data including vertex information, texture information, and shader information of a 2D image as rendering data. In one embodiment, the rendering data storage unit 320 may create two sets of rendering data by copying the rendering data. That is, the rendering data storage unit 320 may hook and store information necessary for generating a game image as rendering data during a game execution process, and equally generate two rendering data for a binocular image necessary for generating a 3D image. there is. In this case, the rendering data storage unit 320 may manage the generated rendering data by grouping them into a set.

The 2D image generator 330 may generate a 2D image based on the rendering data. The 2D image generator 330 may generate 2D images for each frame as the game progresses, and the generated 2D images may be temporarily stored in a memory according to a generation order.

In one embodiment, the 2D image generator 330 may perform filtering on the 2D image. If depth information is obtained directly from a 2D image, the 2D image generator 330 may filter the 2D image so that depth information can be extracted more effectively. For example, by removing a specific color region or emphasizing a specific color through image filtering, boundaries between regions may be more clearly defined.

The depth information acquisition unit 340 may obtain depth information of a 2D image. Here, the depth information may include z value information for each pixel of the 2D image based on a view point. The depth information may be obtained from information stored in a depth buffer in the process of the 2D image generator 330 generating a 2D image based on the rendering data, but is not limited thereto, and is obtained from the 2D image. Depth information may be obtained directly. The depth information acquisition unit 340 may store depth information in association with the corresponding 2D image.

The vertex adjusting unit 350 may determine positions of vertices of a pair of binocular images based on the depth information. The vertex adjusting unit 350 may perform preprocessing of moving vertices used to generate binocular images, that is, left-eye images and right-eye images in advance, in order to render the binocular images in a step before generating the 3D images. In this case, the position of the vertex may be determined according to a parallax mapping algorithm.

In one embodiment, the vertex adjusting unit 350 moves the vertices of the 2D image by an offset based on the eye vectors of each of both eyes using the z-value related to the depth information to adjust the positions of the vertices. can decide For example, in the case of a left eye image, the vertex adjuster 350 may adjust the positions of corresponding vertices by applying a predetermined offset according to a direction of an eye vector toward a view point of the left eye. That is, when the z' value at the corresponding position is obtained after moving the position corresponding to the z value of the specific vertex in the direction of the eye vector, the z value of the specific vertex is changed to the z' value so that the position can be adjusted.

In one embodiment, the vertex adjuster 350 may match the two sets of rendering data to the left eye and the right eye, respectively, and then independently determine the positions of the vertices. Here, the rendering data set may include a vertex set. More specifically, vertices of the 2D image may be copied in two sets in advance to generate each of the binocular images, and the vertex adjusting unit 350 may move the respective vertices according to the left eye image and the right eye image.

The binocular image rendering unit 360 may render binocular images respectively using rendering data based on the positions of vertices. The binocular image rendering unit 360 may independently generate a left-eye image and a right-eye image based on the set of vertices at the adjusted position. While a 2D image is generated through one rendering during the course of the game, the binocular image rendering unit 360 performs independent rendering based on the positions of the vertices adjusted through preprocessing to generate binocular images, respectively. can

The 3D image generator 370 may generate a single 3D image by integrating the images of both eyes. In an embodiment, the 3D image generator 370 may generate a frame image of the 3D image by merging identical pixels in each frame of the binocular image. That is, the 3D image generator 370 may determine the RGB value of the corresponding pixel of the 3D image by integrating the RGB values of the corresponding pixels in the binocular image.

In an embodiment, the 3D image generator 370 may generate a 3D image by replacing the damaged image with an image of both eyes prior to the corresponding point in time when one of the images of the both eyes at the same point in time is damaged. Image damage may occur due to an error in the rendering process or loss of a normally generated image. The 3D image generator 370 performs an operation of integrating images of both eyes at the same time point, and if necessary, when some images are damaged as described above, a 3D image may be generated based on a previous image of both eyes at the time point. .

The controller (not shown in FIG. 3) controls the overall operation of the image providing server 130, and the game execution unit 310, the rendering data storage unit 320, the 2D image generator 330, and depth information acquisition A control flow or data flow between the unit 340 , the vertex adjustment unit 350 , the binocular image rendering unit 360 and the 3D image generator 370 may be managed.

4 is a flowchart illustrating a cloud VR-based 3D video providing method according to the present invention.

Referring to FIG. 4 , the video providing server 130 may execute a game implemented to provide a 2D video through the game execution unit 310 (step S410). The image providing server 130 may hook and store the rendering data of the 2D image during the game execution process through the rendering data storage unit 320, and based on the rendering data through the 2D image generator 330, 2 A dimensional image may be generated (step S430).

Also, the image providing server 130 may obtain depth information of the 2D image through the depth information obtaining unit 340 (step S450). The image providing server 130 may determine the positions of the vertices of the pair of binocular images based on the depth information through the vertex adjusting unit 350, and determine the positions of the vertices through the binocular image rendering unit 360 based on the depth information. Images of both eyes may be respectively rendered using the rendering data (step S470). As a result, the image providing server 130 may generate a single 3D image by integrating the images of both eyes through the 3D image generator 370 (step S490).

Thereafter, the video providing server 130 may transmit the 3D video to the VR device 110 and play it through the VR device 110 . On the other hand, the method of providing a 3D image according to the present invention can be performed independently through the VR device 110, and accordingly, the VR device 110 converts a 2D image into a 3D image in the process of running its own game, and thus VR Even general PC games that do not support mods can provide highly immersive VR images.

5 is a diagram for explaining a method of generating a 3D image through parallax mapping, and FIG. 6 is a diagram for explaining an embodiment of a method for providing a 3D image according to the present invention.

Referring to FIG. 5 , parallax mapping may correspond to a technique of making a 2D image look like a 3D image by performing pixel movement as much as a depth. The image providing server 130 may convert a 2D image into a 3D image through parallax mapping.

In FIG. 5 , the image providing server 130 may obtain rendering data for a 2D image while the game is running by executing the game. The image providing server 130 may perform game rendering based on the obtained rendering data, and may obtain depth buffer information acquired during the rendering process. The image providing server 130 may generate images corresponding to the left and right eyes by moving the position of each pixel to generate a 3D image based on the 2D image and depth information, and integrating each image into one to create a 3D image. You can create video.

However, when converting to a 3D image through parallax mapping, a problem in that image quality may deteriorate due to an increase in pixels in a specific region may occur. The 3D image providing method according to the present invention can solve the problem of parallax mapping through geometry parallax mapping. More specifically, the method of providing a 3D image according to the present invention is a method of pre-processing rather than post-processing as shown in FIG. 5 . That is, while general games perform rendering once, in the case of the present invention, rendering can be performed once for each of the left and right eyes.

As a result, the 3D image providing method according to the present invention acquires various information generated for rendering in the course of a game operation, then converts all the data into two identical pieces to generate a binocular image, and then vertex position information. Among them, a 2D image can be converted into a 3D image by using a method of moving the geometry itself using the z value in the same way as the parallax mapping algorithm.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: video providing system
110: VR device 130: video providing server
210: processor 230: memory
250: user input/output unit 270: network input/output unit
310: game execution unit 320: rendering data storage unit
330: 2D image generator 340: depth information acquisition unit
350: vertex adjustment unit 360: binocular image rendering unit
370: 3D image generator

Claims (6)

Executing a game implemented to provide a two-dimensional image through a VR device;
During the execution of the game, scene data including vertex information, texture information, and shader information of the 2D image is hooked as rendering data, and the rendering data is copied to create the same two rendering data Creating and storing sets;
generating the 2D image based on the rendering data, and removing a specific color region or emphasizing a specific color region by performing filtering on the 2D image;
obtaining depth information of the 2D image;
matching the two sets of rendering data to the left eye and the right eye, respectively, and then independently determining positions of vertices of the pair of binocular images based on the depth information;
independently rendering the binocular images using the rendering data based on positions of vertices of the pair of binocular images that are independently determined; and
Including; generating a single 3D image by integrating the images of both eyes,
The determining step may include locating a position corresponding to a z value of each vertex in a direction of an eye vector of each of both eyes by using the z value of the depth information for the vertices of the pair of images of both eyes that are independently determined. Independently determining the positions of the vertices by moving them by an offset,
The generating of the 3D image includes generating a frame image of the 3D image by merging the same pixels for each frame of the binocular image, wherein one of the binocular images at the same time point is damaged. case, generating the 3D image by replacing the damaged image with an image of both eyes before the corresponding point in time.
delete delete delete delete delete

Images