KR102461478B1 - Clould vr device for reducing object jittering of image - Google Patents

Abstract

The present invention relates to a cloud virtual reality (VR) device for reducing an object jittering in an image. The device comprises: a depth map generation unit which generates a depth map with respect to an image rendered in a cloud VR server; a depth information transmission processing unit which allocates and transmits an RGB value of a rendered color to an RGB channel part and the generated depth map information to an alpha (A) channel part when transmitting the rendered image, in an RGBA format, to a client VR terminal; a depth information reception processing unit which receives the image transmitted in the RGBA format and performs decoding in the client VR terminal; and a screen jittering correction unit which calculates the amount of movement of a pixel using a difference from depth information of a previous frame based on the decoded depth map information and corrects a jittering. According to the present invention, the device can reduce an object jittering occurring in an image by using depth information.

Description

CLOUD VR DEVICE FOR REDUCING OBJECT JITTERING OF IMAGE

The present invention relates to cloud VR technology, and more particularly, to reduce object shake in an image, which can reduce object shake in an image by the received depth information by transmitting depth information together through an alpha channel during image transmission It relates to a cloud VR device for

Although virtual reality (VR) devices have recently appeared, they are not as widely distributed as smartphones due to problems such as high prices, low resolutions that hinder immersion, and lack of VR content.

In particular, even if the head orientation due to latency and jitter is corrected in the cloud environment, vibration may occur when the controller or other objects move. The phenomenon is related to the items displayed at a close distance to the user while moving in conjunction with the user's gaze such as GUI (Graphical User Interface) and HUD (Head Up Display) among the rendering items of the game or application. Because it appears so well, it may be a problem that needs to be addressed.

Korean Patent Registration No. 10-2166158 (10.08.2020)

An embodiment of the present invention provides a cloud VR device for reducing object shake in an image, which can reduce object shake in an image by transmitting depth information through an alpha channel together with the received depth information when transmitting an image want to

An embodiment of the present invention generates a depth map at the rendering time in the cloud VR server and transmits the rendered RGB image to the client VR terminal by allocating depth information to the alpha channel part to transmit object tremor in the image It is intended to provide a cloud VR device for reduction.

An embodiment of the present invention is a cloud VR for reducing object shake in an image that can be shared by applying a scale factor to specify a distance value range to be expressed in a rendered image, and transmitting the applied scale factor as metadata. We want to provide a device.

In embodiments, the cloud VR device for reducing object vibration in an image includes a depth map generator that generates a depth map for an image rendered in a cloud VR (Virtual Reality) server, and a client VR terminal that converts the rendered image into RGBA format. Depth information transmission processing unit for allocating RGB values of the rendered color to the RGB channel part and allocating the generated depth map information to the alpha (A) channel part to transmit the image, receiving the image transmitted in the RGBA format and a depth information reception processing unit for decoding in the client VR terminal, and a screen shake correction unit for correcting the shake by calculating the movement amount of the pixel using the difference from the depth information of the previous frame based on the decoded depth map information do.

The depth map generator may generate the depth map with depth information for each pixel of each frame included in the source data used for the rendering.

The depth information transmission processor may apply a scale factor to the generated depth map information to designate a range of perspective distance values to be expressed in the rendered image.

The depth information transmission processing unit may include the scale factor in the meta data to transmit and share the RGBA image when transmitting the RGBA image.

The depth information reception processing unit may restore the depth map information included in the alpha channel portion of the RGBA format image received through the client VR terminal by applying the inverse of the scale factor included in the metadata.

When the movement amount of the pixel is equal to or greater than a specific reference, the screen shake correcting unit may correct the screen shake by adjusting the depth information and rendering the adjusted depth information and the user's head movement.

The disclosed technology may have the following effects. However, this does not mean that a specific embodiment should include all of the following effects or only the following effects, so the scope of the disclosed technology should not be construed as being limited thereby.

The cloud VR device for reducing object shake in an image according to an embodiment of the present invention transmits depth information together through an alpha channel when transmitting the image, so that object shake in the image can be reduced by the received depth information. .

The cloud VR device for reducing object shake in an image according to an embodiment of the present invention generates a depth map at the rendering time in the cloud VR server and transmits the rendered RGB image to the client VR terminal in the alpha channel part. Depth information can be allocated and transmitted.

A cloud VR device for reducing object shake in an image according to an embodiment of the present invention specifies a range of distance values to be expressed in a rendered image by applying a scale factor, and transmits and shares the applied scale factor as metadata can do.

1 is a view for explaining a cloud VR system according to the present invention.
2 is a view for explaining a functional configuration of a cloud VR device according to the present invention.
3 is a flowchart illustrating a process for transmitting and receiving depth information and correcting screen shake according to the present invention.
4 is an exemplary diagram illustrating a case in which a scale factor is applied to depth information according to an embodiment.
5 is a diagram for explaining depth information transmission/reception processing according to the present invention.

Since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiment is capable of various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

On the other hand, the meaning of the terms described in the present application should be understood as follows.

Terms such as “first” and “second” are for distinguishing one component from another, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected to the other component, but it should be understood that other components may exist in between. On the other hand, when it is mentioned that a certain element is "directly connected" to another element, it should be understood that the other element does not exist in the middle. On the other hand, other expressions describing the relationship between elements, that is, "between" and "between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The singular expression is to be understood to include the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" or "have" refer to the embodied feature, number, step, action, component, part or these It is intended to indicate that a combination exists, and it should be understood that it does not preclude the possibility of the existence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

Identifiers (eg, a, b, c, etc.) in each step are used for convenience of description, and the identification code does not describe the order of each step, and each step clearly indicates a specific order in context. Unless otherwise specified, it may occur in a different order from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer-readable codes on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Terms defined in the dictionary should be interpreted as being consistent with the meaning of the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present application.

1 is a view for explaining a cloud VR system according to the present invention.

Referring to FIG. 1 , the cloud VR system 100 may include a client VR terminal 110 and a cloud VR server/PC 130 .

The client VR terminal 110 may correspond to a user terminal capable of playing AR/VR images. For example, the client VR terminal 110 may be typically implemented with an HMD, VR/AR glasses, etc., but is not limited thereto, and may be implemented with various devices capable of reproducing AR/VR images. The client VR terminal 110 may be connected to the cloud VR server/PC 130 through a network to exchange data.

In one embodiment, the client VR terminal 110 may be implemented including the cloud VR device according to the present invention. In this case, the cloud VR device according to the present invention may correspond to a dedicated application executed on the client VR terminal 110 . That is, the cloud VR device according to the present invention may be implemented by being included in the client VR terminal 110 as an independent module that performs a predetermined function, and interworking with the cloud VR server/PC 130 to interwork with the image according to the present invention. It is possible to perform a specific operation for reducing the vibration of an object within.

In addition, when the cloud VR device according to the present invention is implemented by being included in the client VR terminal 110 , some operations may be implemented by being included in the cloud VR server/PC 130 . For example, the operation of encoding the video image may be performed in the cloud VR server/PC 130 , and the client VR terminal 110 receives and plays the video image encoded by the cloud VR server/PC 130 . action can be performed.

In an embodiment, the client VR terminal 110 may receive and decode the encoded video image from the cloud VR server/PC 130 and then play the image. In particular, the client VR terminal 110 may receive an image transmitted in RGBA format and correct screen shake through depth information allocated to the alpha part.

The cloud VR server/PC 130 may be implemented as a server corresponding to a computer or program that generates an AR/VR image reproduced in the client VR terminal 110 and transmits it through a network. The cloud VR server/PC 130 may be connected to the client VR terminal 110 through a wireless network such as Bluetooth, WiFi, 5G communication, etc., and may transmit/receive data to and from the client VR terminal 110 through the network.

In addition, the cloud VR server/PC 130 may perform image rendering, and in this case, may generate a depth map for the corresponding image. Thereafter, the cloud VR server/PC 130 may perform screen capture and video encoding. The cloud VR server/PC 130 may transmit the generated video to the client VR terminal 110 through a wireless network. In this case, the cloud VR server/PC 130 may transmit the depth information contained in the generated depth map to the alpha part of the RGBA transmission format and transmit it together. The wireless network may include LTE, 5G, WiFi, and the like. The cloud VR server/PC 130 may install and execute a remote application that operates in conjunction with the client VR terminal 110 .

2 is a view for explaining a functional configuration of a cloud VR device according to the present invention.

Referring to FIG. 2 , the cloud VR device 200 includes a depth map generation unit 210 , a depth information transmission processing unit 230 , a depth information reception processing unit 250 , a screen shake correcting unit 270 , and a control unit 290 . may include In an embodiment, each functional component constituting the cloud VR device 200 may be implemented by being included in the client VR terminal 110 or the cloud VR server/PC 130 , and the client VR terminal 110 and the cloud VR It may be organically coupled by interworking between the server/PC 130 .

The depth map generator 210 may be generated at a rendering time in the cloud VR server 130 .

Rendering refers to the process of creating an image from a model (scene data) using a rendering program. Scene data may include an arrangement of objects representing a virtual scene, viewpoints, texture mapping, lighting, shading, and the like. Rendering is the process of converting model data into image data on a 2D screen. Because rendering is a process that takes a lot of time and resources, the cloud VR server 130 with excellent processing performance renders and the 2D image that has been rendered is transmitted to the client VR terminal 110 .

In an embodiment, the depth map generator 210 may generate a depth map with depth information included in source data used for rendering in a rendering process for a corresponding image through the cloud VR server 130 . A depth map may correspond to one image containing information related to a distance from an observation viewpoint to an object (object) surface in 3D computer graphics. The depth map generator 210 may generate a depth map using depth information for each pixel existing in each frame of a rendered image. The depth image generated by the depth map generator 210 is an image representing a distance between an object located in a 3D space and a camera photographing the object in units of black and white.

When transmitting the 2D image rendered by the cloud VR server 130 to the client VR terminal 110 in RGBA format, the depth information transmission processing unit 230 allocates the RGB value of the rendered color to the RGB channel part and the depth to the alpha channel part. It can be processed to be transmitted by allocating map information. In an embodiment, the depth information transmission processing unit 230 may transmit depth map information by including the depth map information in the alpha part because the alpha part is not used in RGBA, which is a transmitted image format. An image may be expressed in various ways. Typically, an image may be expressed as an RGB channel. RGB channels represent colors. The alpha channel may mean auxiliary data different from color expression data in each pixel. Here, when the depth information transmission processing unit 230 transmits the image using the RGBA channel method, the depth image related to the RGB image may be included in the alpha channel to be transmitted together.

In the depth information transmission processing unit 230, the size of data to which the depth is applied is 1 byte, and a value that can be expressed may be 0 to 255. In this case, the depth information transmission processing unit 230 may apply a scale factor to uniformly or differentially designate the perspective range of the image. In an embodiment, the depth information transmission processing unit 230 divides the section from near to far to be expressed in the rendered image equally into 256 when the scale factor is applied as “1” and divides the depth information into values of 0 to 255. can be expressed When the scale factor is applied as “100”, the depth information transmission processing unit 230 may express depth information in a value of 0 to 25500 to increase the resolution of the entire image. Here, the depth information transmission processing unit 230 may increase the short-range image to a high resolution by subdividing the range of the short-range value relative to the far-field value by applying a different scale factor according to the long-distance and the short-distance. The depth information transmission processing unit 230 may create and apply a scale factor using a constant function, an exponential function, a log function, and the like.

The depth information transmission processing unit 230 may include the scale factor applied to the depth information as metadata and transmit and share the RGBA image when transmitting the same.

The depth information reception processing unit 250 may receive an image transmitted in RGBA format from the cloud VR server 130 and decode it through the client VR terminal 110 . The depth information reception processing unit 250 decodes the RGBA format image received through the client VR terminal 110 and applies the inverse of the scale factor included in the metadata to the depth information included in the alpha channel to receive the depth information. restoration can be processed.

The screen shake correcting unit 270 calculates the difference from the depth information of the previous frame based on the depth information in the process of reproducing the decoded image to obtain the movement amount of the pixel, and then returns the depth information when the movement amount of the pixel is greater than or equal to a specific standard. By adjusting it, it is possible to compensate for screen shake by rendering with the adjusted depth information. When the difference between the actual direction and the predicted direction of the user's head in the decoding process is greater than or equal to a preset specific standard, the screen shake correcting unit 270 performs correction in the canvas rendering process for the video image based on the actual direction to reduce the shake. can The screen shake correcting unit 270 may perform shake compensation during the canvas rendering process on the video image based on the actual direction, and the size of the region rendered at high resolution in the foveated rendering process may be determined by determining the difference between the predicted direction and the actual direction. It can be determined dynamically based on The screen shake correcting unit 270 may reduce shake for objects in an image, particularly GUI and HUD items displayed at a close distance to the user, by performing screen shake correction according to depth information and the user's head movement. .

The controller 290 controls the overall operation of the cloud VR device 200 , and operates between the depth map generation unit 210 , the depth information transmission processing unit 230 , the depth information reception processing unit 250 , and the screen shake correcting unit 270 . It can manage control flow or data flow.

3 is a flowchart illustrating a process of transmitting depth information and correcting screen shake according to the present invention.

Referring to FIG. 3 , the cloud VR device 200 may generate a depth map at a rendering time in the cloud VR server 130 through the depth map generator 210 (step S310 ). The cloud VR device 200 applies a scale factor to the depth map information generated through the depth information transmission processing unit 230 to designate a range of perspective distance values of the image, and then transmits the RGB channel to the client VR terminal 110 in RGBA format. It is possible to input the RGB value of the rendered color to the part, allocate depth map information to the alpha (A) channel part, and transmit it (steps S330 and S350).

In addition, the cloud VR device 200 may decode the RGBA format image received from the client VR terminal 110 through the depth information reception processing unit 250 (step S350 ). The cloud VR device 200 may correct the shake by calculating the movement amount of the pixel by using the difference from the depth information of the previous frame based on the depth information decoded through the screen shake correcting unit 270 (step S390).

4 is an exemplary diagram illustrating a case in which a scale factor is applied to depth information according to an embodiment.

Referring to FIG. 4 , the cloud VR system 100 reduces tremor of objects in an image that may occur due to latency and jitter between the cloud VR server 130 and the client VR terminal 110 through transmission and reception of depth information. can do it In this case, the depth information may be generated during the rendering process of the cloud VR server 130 , and a depth range may be designated by applying a scale factor. The depth information may have a range of values depending on the size. When the size is 1 byte, the depth value range may be 0 to 255.

In the case of FIG. 4 , the depth value may be configured as a near-field value and a far-field value according to the distance from the user's point of view. Here, near corresponds to the closest distance value to be expressed in the rendered image, and far corresponds to the furthest distance value to be expressed in the rendered image. By applying a scale factor to the depth information, the near and far ranges to be expressed in the image can be equally or differentially specified. When the scale factor is “1”, with “128” as an intermediate value, near becomes “0” and far becomes “255”. When the scale factor is “2”, with “255” as an intermediate value, near becomes a value of “0” and far becomes a value of “511”. In addition, by applying a scale factor as a function, the distance value can be expressed by designating the high-resolution level in the near section and the low-resolution level in the far section.

5 is a diagram for explaining depth information transmission/reception processing according to the present invention.

Since the cloud VR server has superior CPU and graphic card performance compared to the client VR terminal, basically, the cloud VR system renders all the content to be displayed on the screen in the cloud VR server and then sends the resulting image to the client VR through the network. It is transmitted to the terminal, and only the received result image is displayed on the screen in the client VR terminal. In this case, the shaking phenomenon of the object displayed on the screen of the client VR terminal may occur due to latency, jitter, etc. on the network. In particular, the closer the object, the greater the tremor.

In the cloud VR system 100 according to an embodiment, after rendering through the cloud VR server 130 , the rendering result may be transmitted to the client VR terminal 110 through a network together with metadata. At this time, the cloud VR server 130 inputs the RGB value of the color of the rendered 2D image to the RGB channel part, allocates the depth information to the alpha channel part, and transmits it in RGBA format. Here, the depth information can designate the range of near and far values to be expressed in the rendered image by applying the scale factor, and information about the applied scale factor is included in the metadata to be shared with the client VR terminal 110 . can The client VR terminal 110 may receive and decode the RGBA image and then play it back. At this time, the depth information is restored by applying the inverse of the scale factor, and the amount of movement of the pixel through the difference between the restored depth information and the depth information of the previous frame. By obtaining and correcting , it is possible to reduce or remove the shaking phenomenon of the object displayed on the screen of the client VR terminal 110 .

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

100: Cloud VR system
110: client VR terminal 130: cloud VR server / PC
200: Cloud VR device
210: depth map generation unit 230: depth information transmission processing unit
250: depth information reception processing unit 270: screen shake correction unit
290: control unit

Claims (6)

a depth map generator for generating a depth map for an image rendered in a cloud VR (Virtual Reality) server;
When the rendered image is transmitted to the client VR terminal in RGBA format, the RGB value of the rendered color is allocated to the RGB channel part, and the generated depth map information is allocated to the alpha (A) channel part and transmitted. Depth information transmission processing unit;
a depth information reception processing unit that receives the image transmitted in the RGBA format and performs decoding in the client VR terminal; and
And a screen shake correction unit for correcting the shake by calculating the movement amount of the pixel by using the difference from the depth information of the previous frame based on the decoded depth map information,
The screen shake correction unit
A cloud VR device for reducing object shake in an image, characterized in that when the amount of movement of the pixel is greater than or equal to a specific criterion, the depth information is adjusted, and the screen shake is corrected by rendering by reflecting the adjusted depth information and the user's head movement.
The method of claim 1, wherein the depth map generator
A cloud VR device for reducing object shake in an image, characterized in that the depth map is generated with depth information for each pixel of each frame included in the source data used for the rendering.
According to claim 1, wherein the depth information transmission processing unit
A cloud VR device for reducing object shake in an image, wherein a range of perspective distance values to be expressed in the rendered image is designated by applying a scale factor to the generated depth map information.
The method of claim 3, wherein the depth information transmission processing unit
Cloud VR device for reducing object shake in an image, characterized in that the scale factor is included in the metadata so that it is transmitted and shared when the RGBA image is transmitted.
The method of claim 4, wherein the depth information reception processing unit
In the RGBA format image received through the client VR terminal, the depth map information included in the alpha channel part is restored by applying the inverse of the scale factor included in the metadata to reduce object shake in the image. Cloud VR device.
delete

Images