KR20220049891A - Body movement based cloud vr device and method - Google Patents

Abstract

The present invention relates to a body movement-based cloud virtual reality (VR) device and method to minimize operation delay. According to the present invention, the body movement-based cloud video recording device comprises: a body motion measurement unit measuring a motion of a body associated with a user's central axis through a first gyro sensor; a head motion measurement unit measuring a head motion associated with the user's head through a second gyro sensor; a field of view (FOV) image determination unit determining a first FOV image with respect to the central axis and determining a second FOV image within the first FOV image according to the head movement; and an image processing unit tracking the body movement and updating and providing the first FOV image on the basis of the central axis when a change in the central axis is detected according to the body movement.

Description

Cloud VR device and method based on body movement {BODY MOVEMENT BASED CLOUD VR DEVICE AND METHOD}

The present invention relates to cloud VR technology, and more particularly, to a body movement-based cloud VR device and method that can implement fast MTP latency and provide high FPS even when the central axis is changed according to a user's movement will be.

Although virtual reality (VR) devices have recently appeared, they are not as widely distributed as smartphones due to problems such as high prices, low resolutions that hinder immersion, and lack of VR content. In particular, resolving physical discomfort such as dizziness that may occur due to a mismatch between a user's head movement and a visible VR image may be an essential problem to be solved. In order to solve this, it is necessary to reduce the MTP (Motion-to-Photon) redundancy to within 10 to 20 ms, and there are technical difficulties in achieving this in a wireless environment (eg, WiFi or mobile network, etc.).

Korean Patent Publication No. 10-08201232 (2008.04.01) relates to a method and system for image intra-prediction mode estimation, transmission and organization, estimating a pixel prediction mode used in an image encoding or decoding process, and Disclosed is a technique capable of improving the efficiency of a coding process by passing a pixel prediction mode between an encoder and a decoder, and providing a method and system for ordering the intra-prediction mode of a pixel.

Korean Patent Publication No. 10-08201232 (2008.04.01)

An embodiment of the present invention is to provide a cloud VR apparatus and method based on body movement that can implement fast MTP latency and provide high FPS even when the central axis is changed according to a user's movement.

An embodiment of the present invention is to provide a cloud VR apparatus and method based on body movement that can minimize computational delay by processing in a server with high computational power when a change in a user's movement exceeds a specific standard.

An embodiment of the present invention is to provide a body motion-based cloud VR device and method capable of providing real-time image processing through hybrid interworking between a cloud server and a VR device.

In embodiments, a body motion-based cloud VR device includes a body motion measurement unit that measures a body motion associated with a central axis of the user through a first gyro sensor, and a head of the user through a second gyro sensor ) a head motion measurement unit for measuring a head motion associated with the FOV, determining a first FOV (Field Of View) image with respect to the central axis, and determining a second FOV image within the first FOV image according to the head motion and an image determining unit and an image processing unit that tracks the body movement and updates and provides the first FOV image based on the central axis when a change in the central axis is detected according to the body movement.

The body motion measurement unit may generate body coordinate data related to the body motion based on a signal measured from the first gyro sensor and transmit it to the head motion measurement unit through a communication interface.

The head motion measurement unit determines a gaze direction of the user based on a head motion measurement module that generates head coordinate data related to the head motion based on a signal measured from the second gyro sensor, the head coordinate data, and the corresponding gaze and a stereoscopic image rendering module for rendering the second FOV image based on a direction and a display module for displaying the second FOV image.

The stereoscopic image rendering module may calculate the position of the central axis according to the body coordinate data and, when the amount of position change exceeds a threshold value, transmit the body coordinate data to the image processing unit to receive the updated first FOV image.

The stereoscopic image rendering module recrystallizes and renders the second FOV image within the first FOV image when the position change amount is less than the threshold value and the user's gaze direction deviates from the actual angle of view of the second FOV image there is.

The stereoscopic image rendering module increases the resolution of the second FOV image up to the actual angle of view by performing foveated rendering on the first FOV image when the user's gaze direction is maintained for a predetermined time or longer. can do it

The image processing unit receives body coordinate data and head coordinate data from the head motion measurement unit, and a FOV determination module that determines the FOV according to the central axis and the gaze direction of the user, and captures a gaze image related to the FOV. It may include a gaze image capturing module and an image encoding module encoding the gaze image and transmitting it to the FOV image determiner as the first FOV image.

Among embodiments, the body motion-based cloud VR method includes measuring, by the body motion measuring unit, a body motion associated with a central axis of the user through a first gyro sensor, by the head motion measuring unit , measuring a head movement associated with the user's head through a second gyro sensor, determining a first FOV (Field Of View) image with respect to the central axis by the FOV image determiner, and determining the head determining a second FOV image within the first FOV image according to movement; and tracking the body movement by the image processing unit, and when a change in the central axis is detected according to the body movement, based on the corresponding central axis and updating and providing the first FOV image.

The disclosed technology may have the following effects. However, this does not mean that a specific embodiment should include all of the following effects or only the following effects, so the scope of the disclosed technology should not be understood as being limited thereby.

The cloud VR apparatus and method based on body movement according to an embodiment of the present invention can minimize computational delay by allowing a server with high computational power to process a change in a user's movement beyond a specific standard.

The cloud VR apparatus and method based on body movement according to an embodiment of the present invention may provide real-time image processing through hybrid interworking between a cloud server and a VR device.

1 is a conceptual diagram illustrating a cloud VR device based on body movement according to the present invention.
2 is a view for explaining a functional configuration of a cloud VR device according to the present invention.
FIG. 3 is a view for explaining a functional configuration of the second system of FIG. 1 .
FIG. 4 is a view for explaining a functional configuration of the third system of FIG. 1 .
5 is a flowchart illustrating a cloud VR process based on body movement according to the present invention.
6 is a flowchart illustrating a stereoscopic image generation and restoration process according to the present invention.

Since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiment may have various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

On the other hand, the meaning of the terms described in the present application should be understood as follows.

Terms such as “first” and “second” are for distinguishing one component from another, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected to the other component, but it should be understood that other components may exist in between. On the other hand, when it is mentioned that a certain element is "directly connected" to another element, it should be understood that the other element does not exist in the middle. Meanwhile, other expressions describing the relationship between elements, that is, "between" and "between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The singular expression is to be understood as including the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" or "have" refer to the embodied feature, number, step, action, component, part or these It is intended to indicate that a combination exists, and it should be understood that it does not preclude the possibility of the existence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

In each step, identification numbers (eg, a, b, c, etc.) are used for convenience of description, and identification numbers do not describe the order of each step, and each step clearly indicates a specific order in context. Unless otherwise specified, it may occur in a different order from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be embodied as computer-readable codes on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. . Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless otherwise defined. Terms defined in the dictionary should be interpreted as being consistent with the meaning of the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present application.

The cloud VR device 100 based on body movement according to the present invention may correspond to a device capable of minimizing dizziness while providing realistic VR/AR images to the user by effectively reflecting the change of the central axis according to the user's movement. . The cloud VR device 100 based on body movement utilizes gyro information that can be obtained from various sensors mounted or attached to the user's body to generate realistic images synchronized with the user's actual movement to provide the user with can provide

Accordingly, the cloud VR device 100 based on body movement can provide realistic images to the user while solving the problems that existing VR devices have due to limitations in physical space.

1 is a conceptual diagram illustrating a cloud VR device based on body movement according to the present invention.

Referring to FIG. 1 , the cloud VR apparatus 100 based on body movement may include a first system 110 , a second system 130 , and a third system 150 .

The first system 110 may collect body coordinates data related to the user's body movement through the 6 DoF sensor included therein, and may be transmitted to the VR device worn by the user through a communication interface. can be transmitted Meanwhile, the first system 510 may be attached to the user's body and used to track the user's movement information, and may be implemented in the form of a wearable device such as a body suite.

The second system 130 may typically correspond to VR/AR glasses (or HMD) worn on the user's head, and the user may enjoy VR/AR images by directly wearing the VR/AR glasses. The second system 130 may be implemented to include a GPS sensor and a 6 DoF sensor by itself, and may display an image generated through a local application.

The second system 130 typically includes (a) a lens, (b) a display device, (c) Interface, (d) 6 DoF Gyro sensor, (e) Motion Tracking sensors, and in addition (f) CPU, (g) GPU, (h) network modem, (i) battery, and (j) Interface device may be included as a built-in or external type. The second system 130 may be configured in a standalone or all-in-one form including all of (f) to (j), or may be separated and configured as a separate external device.

Meanwhile, the local app 2-1 operating in the second system 130 may perform the following operation. (a) Head coordinate data generated from the 6 DoF sensor 2-4 of the second system 130 may be used for positioning in the local app 2-1. (b) Body coordinate data generated from the 6 DoF sensor 1-1 of the first system 110 is finally transferred to the remote application 3-1 through the second system 130, where It can be used for positioning.

On the other hand, the local app 2-1 of the second system 130 additionally performs operations such as color space conversion (RGB-YUV), video decoding, business logic processing, graphics rendering based on the local GPU, and synthesizing graphics and video signals. can

The third system 150 may correspond to a cloud VR server. The third system 150 may determine the FOV with respect to the user's gaze direction through the remote app 3-1, and may perform image rendering based on the FOV. In this case, the rendered image may correspond to an image up to a spare angle of view greater than the actual angle of view of the user.

Thereafter, the third system 150 may perform screen capture 3-2 and video encoding 3-3. The third system 150 may transmit the generated video to the second system 130 through a wireless network. The wireless network may include LTE, 5G, WiFi, and the like.

2 is a view for explaining a functional configuration of a cloud VR device according to the present invention.

Referring to FIG. 2 , the body motion-based cloud VR apparatus 100 includes a body motion measurement unit 210 , a head motion measurement unit 230 , an FOV image determination unit 250 , an image processing unit 270 , and a control unit 290 . ) may be included.

The body motion measurement unit 210 may generate data regarding the user's motion by using a first gyro sensor (eg, a 6 DoF sensor) installed on the user's body. More specifically, the body motion measurement unit 210 may collect 3-axis gyro/acceleration information regarding the user's movement through the 6 DoF sensor. Meanwhile, the body motion measurement unit 210 may be implemented as a physical device attached to the user's body and operated, and may include, for example, smart wear such as a bodysuit, and is not necessarily limited thereto. It may include a device that is attached to or implanted in the body.

In an embodiment, the body motion measurement unit 210 may generate body coordinate data related to body motion based on a signal measured from the first gyro sensor and transmit it to the head motion measurement unit 230 through a communication interface. In this case, the communication interface used for data communication between the body movement measurement unit 210 and the head movement measurement unit 230 may include USB, BT (BlueTooth), or the like.

The head motion measurement unit 230 may measure a head motion associated with the user's head through a second gyro sensor (eg, a 6 DoF sensor). The head motion measurement unit 230 may be implemented by itself including a GPS sensor and a 6 DoF sensor in order to operate independently of the body motion measurement unit 210 . That is, the head motion measurement unit 230 may be attached to the user's head to collect information about the user's head motion.

For example, the head motion measurement unit 230 may correspond to a device that is mounted on the user's head and operates. In this case, information about the user's head motion is obtained for generating an image responding to the head motion. can be collected Meanwhile, the head motion measurement unit 230 is basically a device mounted on the user's head and operated, and may include a lens, a display device, and an interface motion tracking sensor. In addition, if necessary, the head motion measurement unit 230 may further include a CPU, a GPU, a network modem, a battery, and an interface device as a built-in or an external device.

In one embodiment, the head motion measuring unit 230 is a head motion measuring module that generates head coordinate data related to head motion based on a signal measured from the second gyro sensor, and a user's gaze direction based on the head coordinate data. It may include a stereoscopic image rendering module that determines and renders a second FOV image according to the corresponding gaze direction, and a display module that displays the second FOV image. This will be described in more detail with reference to FIG. 3 .

The FOV image determiner 250 may determine a first Field Of View (FOV) image with respect to the central axis of the user and determine a second FOV image within the first FOV image according to the user's head movement. The first FOV image may correspond to a reference image determined according to the movement of the user's central axis, and the second FOV image may correspond to a gaze image generated based on a gaze direction determined according to the user's head movement. The first FOV image may correspond to an image corresponding to a spare angle of view greater than the user's actual angle of view, and the second FOV image may correspond to an image corresponding to the user's actual angle of view.

Meanwhile, the FOV image determiner 250 may be implemented by being included in the same device as the head movement measurement unit 230 , and may operate in conjunction with the head movement measurement unit 230 . For example, the FOV image determiner 250 and the head motion measurement unit 230 may be implemented by being included in the HMD, and determine the gaze image corresponding to the user's gaze direction based on the reference image received from the cloud server. to create a FOV image.

The image processing unit 270 tracks the user's body movement, and when a change in the central axis is detected according to the body movement, the first FOV image may be updated and provided based on the corresponding central axis. For example, the image processing unit 270 may serve as a cloud server, and as a task that requires an excessive amount of computation to be processed by the HMD, a reference image (ie, the first FOV image) that adapts to the change of the central axis is generated. It can be processed instead of the update operation.

That is, when processing is performed by the HMD, delay or interruption of the image displayed to the user may occur due to processing delay, which may cause the user to feel dizzy. Accordingly, the image processing unit 270 may be implemented as a large-capacity cloud server device with high computational processing capability to support the operation of the HMD, thereby providing high MTP latency in the image reproduced by the HMD.

In an embodiment, the image processing unit 270 receives the body coordinate data and the head coordinate data from the head motion measurement unit 230 and determines the FOV (Field Of View) according to the user's central axis and gaze direction. , a gaze image capturing module that captures a gaze image related to the FOV, and an image encoding module that encodes the gaze image and transmits it to the FOV image determiner 250 as a first FOV image. This will be described in more detail with reference to FIG. 4 .

The controller 290 controls the overall operation of the cloud VR device 100 based on body motion, and the body motion measurement unit 210 , the head motion measurement unit 230 , the FOV image determiner 250 , and the image processing unit 270 . ) can manage the control flow or data flow between

FIG. 3 is a view for explaining a functional configuration of the second system of FIG. 1 .

Referring to FIG. 3 , the second system 130 may include a head motion measurement module 310 , a stereoscopic image rendering module 330 , a display module 350 , and a first control module 370 . Meanwhile, the head motion measurement unit 230 is a component of a computing device capable of detecting a user's gaze direction according to the user's head movement and reproducing a stereoscopic image in the corresponding gaze direction, implemented with a smartphone, VR/AR HMD, etc. can be

The head motion measurement module 310 may generate head coordinate data related to head motion based on a signal measured from the second gyro sensor. The head motion measurement module 310 may use the second gyro sensor and the GPS sensor together if necessary. The head coordinate data may include data regarding a head center, a direction, and an inclination. The head motion measurement module 310 may perform validation on signals received from each of the sensors, and may perform data normalization or pre-processing on data conversion into a specific format if necessary.

The stereoscopic image rendering module 330 may determine the user's gaze direction based on the head coordinate data and render the second FOV image according to the corresponding gaze direction. Meanwhile, the stereoscopic image rendering module 330 may determine the user's gaze direction by using the body motion information received from the body motion measurement unit 210 as necessary. When the user's gaze direction is determined, the stereoscopic image rendering module 330 may determine a user's field of view (FOV) for generating a second FOV image, and may generate a second FOV image according to the determined field of view. .

In an embodiment, the stereoscopic image rendering module 330 may determine horizontal and vertical FOVs based on head and body movements. More specifically, when determining the FOV by considering only the user's head movement, the range of the FOV is based on the premise that the user's body is fixed. may be limited. Accordingly, the stereoscopic image rendering module 330 may more accurately reflect the user's dynamic movement by determining the FOV in consideration of not only the user's head movement but also the user's body movement. That is, the stereoscopic image rendering module 330 may determine the horizontal and vertical FOVs, respectively, based on acceleration of the user's head and body movements.

In an embodiment, the stereoscopic image rendering module 330 calculates the position of the central axis according to the body coordinate data and transmits the body coordinate data to the third system 150 when the amount of position change exceeds a threshold value to update the reference image The first FOV image may be received as . The stereoscopic image rendering module 330 may determine the user's position based on the body coordinate data, and may determine an axis perpendicular to the horizontal plane as the center at the corresponding position. The stereoscopic image rendering module 330 may track the position of the central axis, and may calculate a change amount with respect to the position of the central axis for a unit time.

If the amount of position change exceeds a preset threshold, it may be necessary to update the first FOV image determined based on the central axis of the user, and as a result, the 3D image rendering module 330 is updated to the third system 150 . The request can be forwarded. Meanwhile, the third system 150 may be synchronized with the stereoscopic image rendering module 330 to immediately process a response to the request. Therefore, the stereoscopic image rendering module 330 allows the third system 150 with excellent computational power to process a task with an excessive amount of computation for image processing, and receives and processes only the result of the image processing delay. It can minimize the symptoms of dizziness.

In an embodiment, the stereoscopic image rendering module 330 recrystallizes and renders the second FOV image within the first FOV image when the amount of position change is less than the threshold and the user's gaze direction deviates from the actual angle of view of the second FOV image. can The first FOV image includes an image up to an extra angle of view greater than the actual angle of view, but when the user's gaze direction is out of the range of the corresponding extra angle of view, the third system 150 updates the image for the deviated area. It is necessary to receive the first FOV image.

However, there is a possibility that a delay may occur due to communication between the second system 130 and the third system 150 depending on the network connection state. can create That is, the stereoscopic image rendering module 330 may receive, as the first FOV image, a spare image up to a spare angle of view greater than a specific reference point or larger than the actual angle of view of the second FOV image from the third system 150 and store it by itself. By updating the second FOV image based on the FOV image, it is possible to seamlessly provide an image immediately reflecting a change in the user's gaze.

On the other hand, the stereoscopic image rendering module 330 generates the second FOV image based on the first FOV image when the amount of position change is less than the threshold and the user's gaze direction is within the first range exceeding the actual angle of view of the second FOV image. Update on the first FOV image, which is the reference image, to the third system 150 while updating the second FOV image based on the first FOV image when the user's gaze direction is within the second range exceeding the first range Requests can be sent together.

If the user's gaze direction exceeds the second range, the stereoscopic image rendering module 330 may render the second FOV image as a stereoscopic image after receiving the first FOV image updated by the third system 150 . can

In one embodiment, the stereoscopic image rendering module 330 performs foveated rendering on the first FOV image when the user's gaze direction is maintained for a certain period of time or more to the second FOV image up to the actual angle of view. resolution can be increased. When the user's gaze direction is maintained for a predetermined time or more, it may be determined that the user is looking at the image in the corresponding gaze direction, and the stereoscopic image rendering module 330 may perform foveated rendering on the first FOV image. .

Here, the foveated rendering may correspond to an image processing method of processing a region that the user gazes with interest according to the user's gaze at a high resolution and processing a portion other than the corresponding region in a low resolution. The stereoscopic image rendering module 330 may determine whether or not to gaze the user on the basis of whether the user's gaze direction changes, and more specifically, when the rate of change of the gaze direction for a unit time is less than the reference rate of change and continues for a specific time. can be determined by the user's attention.

When it is determined that the user is watching, the stereoscopic image rendering module 330 renders the real area the user is looking at in high resolution, and the remaining area, that is, the area other than the second FOV image in the first FOV image, is rendered in low resolution. Rendering can be performed. As a result, the stereoscopic image rendering module 330 may differentially apply the resolutions of the real image up to the actual angle of view and the remaining images to the spare image, and may apply a relatively higher resolution to the real image. Meanwhile, the stereoscopic image rendering module 330 may generate a binocular image as a stereoscopic image.

In an embodiment, the stereoscopic image rendering module 330 may dynamically adjust the threshold value regarding the rate of change of position by reflecting the network speed between the second system 130 and the third system 150 . For example, the 3D image rendering module 330 adjusts the threshold value when there is a high possibility of network delay due to the reception of a request and a response from the third system 150 , so that the number of updates of the first FOV image according to the position change can reduce On the contrary, if the network speed is sufficient, the 3D image rendering module 330 increases the number of updates of the first FOV image according to the position change by adjusting the threshold value in order to maximize the high computational power of the third system 150 . can do it

The display module 350 may display the second FOV image. For example, the display module 350 may be implemented to include a display panel for image reproduction, and may reproduce a stereoscopic image generated by the stereoscopic image rendering module 330 in real time. Meanwhile, the stereoscopic image rendering module 330 may receive the updated first FOV image from the third system 150 , and the display module 350 may receive and reproduce the image.

The first control module 370 controls the overall operation of the second system 130 , and manages the control flow or data flow between the head motion measurement module 310 , the stereoscopic image rendering module 330 , and the display module 350 . can do.

FIG. 4 is a view for explaining a functional configuration of the third system of FIG. 1 .

Referring to FIG. 4 , the third system 150 may include a FOV determination module 410 , a gaze image capture module 430 , an image encoding module 450 , and a second control module 470 .

The FOV determination module 410 may receive the body coordinate data and the head coordinate data from the second system 130 to determine a field of view (FOV) according to the position of the central axis and the user's gaze direction. The FOV determination module 410 may adaptively determine the user's field of view (FOV) based on the user's head direction and body direction. The third system 150 may update the stereoscopic image based on the angle of view determined by the FOV determination module 410 .

More specifically, the FOV determination module 410 may determine the horizontal and vertical FOVs based on acceleration of the user's head and body movements. In this case, the body movement may correspond to a change in the central axis.

For example, the FOV determination module 410 may set 210 degrees in the horizontal direction and 130 degrees in the vertical direction as default angles of view (FOV). The FOV determination module 410 may adjust the horizontal angle of view (HFOV) between 210 and 360 degrees and the vertical angle of view (VFOV) between 130 and 180 degrees. The angle of view adjustment may be determined based on acceleration of head movement and acceleration of movement of body, respectively.

The gaze image capturing module 430 may capture a gaze image related to the FOV. That is, the region corresponding to the FOV determined by the FOV determination module 410 may be captured in the gaze image. Meanwhile, the gaze image capturing module 430 may generate a binocular image constituting a stereoscopic image as a gaze image.

The image encoding module 450 may encode the gaze image and transmit it to the FOV image determiner 250 as the first FOV image. More specifically, the image encoding module 450 may largely perform three types of encoding. The image encoding module 450 may perform standard AVC (Advanced Video Coding) encoding, and may perform tile-based High Efficiency Video Cdoing encoding, and redundancy related to a stereoscopic image is removed. encoding can be performed. The image encoding module 450 does not encode the entire 360-degree image around the user, but encodes only the image including the corresponding FOV according to the FOV dynamically determined based on the movement of the user's head and body. You can reduce the amount of data.

The second control module 470 controls the overall operation of the third system 150 , and manages the control flow or data flow between the FOV determination module 410 , the gaze image capture module 430 , and the image encoding module 450 . can do.

5 is a flowchart illustrating a cloud VR process based on body movement according to the present invention.

Referring to FIG. 5 , the body motion-based cloud VR apparatus 100 may measure the body motion associated with the central axis of the user through the first gyro sensor through the body motion measurement unit 210 (step S510). The body movement-based cloud VR apparatus 100 may measure the head movement associated with the user's head through the second gyro sensor through the head movement measurement unit 230 (step S530).

In addition, the body movement-based cloud VR device 100 determines a first FOV (Field Of View) image about the central axis of the user through the FOV image determiner 250, and according to the head movement, within the first FOV image. A second FOV image may be determined (step S550). The body movement-based cloud VR device 100 tracks the body movement through the image processing unit 270, and when a change in the central axis is detected according to the body movement, the first FOV image is updated and provided based on the central axis. It can be done (step S570).

6 is a flowchart illustrating a process of generating and restoring a stereoscopic image according to the present invention.

Referring to FIG. 6 , a user input signal may be generated from an HMD device or a wearable device, and the cloud server may receive it and calculate acceleration with respect to the movement of the user's head and body. The cloud server may dynamically calculate the user's FOV based on acceleration with respect to the movement of the user's head and body. Also, the cloud server may perform graphics rendering for generating a stereoscopic image based on the calculated FOV.

In an embodiment, the graphic rendering may consist of an adaptive FOV selection step and a foveated rendering step. The step of selecting the adaptive angle of view may correspond to a step of dynamically determining the angle of view based on the acceleration of the head movement and the acceleration of the body movement within a specific range based on the default angle. The foveated rendering step may correspond to a step of performing image processing in a high resolution area for the user's gaze based on the user's angle of view and low resolution for other areas.

In addition, the cloud server may perform image capturing for capturing the rendered image, and may generate image data to be wirelessly transmitted through video encoding. Video encoding may be performed according to the existing standard Advanced Video Coding or High Efficiency Video Coding to generate a stereoscopic image. The encoded data generated by the cloud server may be transmitted and played back to the HMD device through wireless communication.

The HMD device may perform video decoding using encoded data received from the cloud server, and may restore an encoded image through video decoding. Here, video decoding may be performed in a manner similar to video encoding, and a stereoscopic image reconstructed by the HMD device may be reproduced through a display panel.

That is, the HMD device may generate and reproduce a stereoscopic image according to the FOV determined according to the body movement as well as the user's head movement. For example, the HMD device may expand or cut an image to fit the displayed angle of view of the user and then reproduce the image through the display panel.

Although the above has been described with reference to the preferred embodiment of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as described in the claims below. You will understand that it can be done.

100: Cloud VR device based on body movement
110: first system 130: second system
150: third system
210: body movement measurement unit 230: head movement measurement unit
250: FOV image determining unit 270: image processing unit
290: control unit
310: head motion measurement module 330: stereoscopic image rendering module
350: display module 370: first control module
410: FOV determination module 430: Gaze image capture module
450: video encoding module 470: second control module

Claims (8)

a body motion measurement unit measuring a body motion associated with a central axis of the user through the first gyro sensor;
a head motion measurement unit measuring a head motion associated with the user's head through a second gyro sensor;
an FOV image determining unit that determines a first Field Of View (FOV) image with respect to the central axis and determines a second FOV image within the first FOV image according to the head movement; and
and an image processing unit that tracks the body movement and updates and provides the first FOV image based on the central axis when a change in the central axis is detected according to the body movement.
According to claim 1, wherein the body motion measurement unit
The body movement-based cloud VR device, characterized in that based on the signal measured from the first gyro sensor, body coordinate data related to the body movement is generated and transmitted to the head movement measurement unit through a communication interface.
According to claim 1, wherein the head movement measuring unit
a head motion measurement module that generates head coordinate data related to the head motion based on a signal measured from the second gyro sensor;
a stereoscopic image rendering module that determines the user's gaze direction based on the head coordinate data and renders the second FOV image based on the corresponding gaze direction; and
Body movement-based cloud VR device comprising a display module for displaying the second FOV image.
The method of claim 3, wherein the stereoscopic image rendering module
A cloud based on body movement, characterized in that when a position of a central axis is calculated according to the body coordinate data and the amount of position change exceeds a threshold value, the body coordinate data is transmitted to the image processing unit to receive an updated first FOV image VR device.
5. The method of claim 4, wherein the 3D rendering module comprises:
Body motion-based, characterized in that the second FOV image is re-determined and rendered within the first FOV image when the position change amount is less than the threshold value and the user's gaze direction deviates from the actual angle of view of the second FOV image of cloud VR devices.
5. The method of claim 4, wherein the 3D rendering module comprises:
Body movement, characterized in that when the user's gaze direction is maintained for a predetermined time or more, foveated rendering is performed on the first FOV image to increase the resolution of the second FOV image up to the actual angle of view. based cloud VR device.
According to claim 1, wherein the image processing unit
an FOV determination module for receiving body coordinate data and head coordinate data from the head motion measurement unit and determining an FOV according to a central axis and a gaze direction of the user;
a gaze image capturing module for capturing a gaze image related to the FOV; and
and an image encoding module that encodes the gaze image and transmits it to the FOV image determiner as the first FOV image.
measuring a body movement associated with a central axis of a user through a first gyro sensor;
measuring a head movement associated with the user's head through a second gyro sensor;
determining a first Field Of View (FOV) image with respect to the central axis and determining a second FOV image within the first FOV image according to the head movement; and
and tracking the body movement and, when a change in the central axis is detected according to the body movement, updating and providing the first FOV image based on the corresponding central axis.

Images