KR102489632B1 - Method of dynamic overfilling for virtual reality and system performing the same - Google Patents

Abstract

Disclosed are a dynamic overfilling method for virtual reality and a performing system thereof. The overfilling method according to various embodiments comprises the operations of: calculating a first overfilling size for performing the overfilling of an original image based on a field of view corresponding to a first plane when a user sees an image in the virtual reality at a first angle, and a field of view corresponding to a second plane when the user sees the input image at a second angle; obtaining an angular velocity from the first angle to the second angle; generating a scaling factor based on the angular velocity; and generating a second overfilling size generated by adjusting the first overfilling size based on the first overfilling size and the scaling factor, wherein the original image may be a base in generating the image. Therefore, provided is an overfilling technique using a minimal overfilling size capable of solving a blank region for each frame.

Description

METHOD OF DYNAMIC OVERFILLING FOR VIRTUAL REALITY AND SYSTEM PERFORMING THE SAME}

Various embodiments of the present invention relate to a method for dynamic overfilling for virtual reality and a system for performing the same.

In the field of VR, latency (latency) occurs significantly due to high computing overhead compared to existing content, and when a gap occurs between the rendered image and the user's vision due to latency, it can cause discomfort such as motion sickness to the user. For example, the angle of the user's head moves while rendering the image predicted by the user's field of view, so that the user's sense of equilibrium perceives and actually looks at it differently. Image reprojection may be performed to compensate for such a prediction error, but as a result of image reprojection, a black border may occur in a viewed image.

The background art described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known art disclosed to the general public prior to the present application.

Overfilling may be performed to render an image larger than the field of view (FOV) of the VR headset in order to minimize a blank area in the viewed image. The overfilling size can be calculated using a mathematical model, but in some cases, the mathematical model may i) calculate an overfilling size that is too large, making it difficult to apply in practice, or ii) calculate an overfilling size that is far from the actual value, resulting in inaccurate results. can have consequences. On the other hand, large overfilling consumes more computing power. Accordingly, a technique for performing overfilling with an optimal overfilling size by adjusting the overfilling size obtained from the mathematical model may be required.

Various embodiments may provide a technique for performing overfilling using a minimum overfilling size capable of resolving blank areas for each frame.

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

The overfilling method according to various embodiments provides a field of view corresponding to a first plane when a user views an image in the virtual reality at a first angle and a second plane when the user views the input image at a second angle. An operation of calculating a first overfilling size for performing overfilling of an original image based on a corresponding field of view, an operation of obtaining an angular velocity from the first angle to the second angle, and a scaling factor based on the angular velocity and generating a second overfilling size obtained by adjusting the first overfilling size based on the first overfilling size and the scaling factor, wherein the original image is used as a basis for generating the image. it may be

The operation of acquiring the angular velocity includes obtaining the angular velocity and a minimum value, a threshold value, and a maximum value of the angular velocity, and the operation of generating the scaling factor predicts a minimum scaling factor based on the minimum value of the angular velocity. Operation of predicting a maximum scaling factor based on the threshold value of the angular velocity, operation of predicting an extreme scaling factor based on the maximum value of the angular velocity, and the minimum scaling factor, the maximum scaling factor, and the threshold value of the angular velocity It may include an operation of generating the scaling factor based on.

The generating of the second overfilling size may include generating the second overfilling size by dividing the first overfilling size by the scaling factor when the scaling factor is greater than the minimum scaling factor and smaller than the maximum scaling factor. operation, when the scaling factor is greater than or equal to the maximum scaling factor, generating the second overfilling size by dividing the first overfilling size by the extreme scaling factor, and when the scaling factor is smaller than the minimum scaling factor An operation of generating a second overfilling size by scalar multiplication of the first overfilling size may be included.

The generating of the scaling factor includes generating the scaling factor by using the angular velocity as an exponential of an exponential function, and the generating of the second overfilling size includes using the reciprocal of the scaling factor as an exponential of the first overfilling factor. It may include an operation of multiplying the filling size.

An apparatus according to various embodiments includes a memory including instructions, and a processor electrically connected to the memory and executing the instructions, and when the instructions are executed by the processor, the processor enables a user to first Overfilling of the original image is performed based on the field of view corresponding to the first plane when viewing the image in the virtual reality at an angle and the field of view corresponding to the second plane when the user views the input image at a second angle. Calculate a first overfilling size for the above, obtain an angular velocity from the first angle to the second angle, generate a scaling factor based on the angular velocity, and generate a scaling factor based on the first overfilling size and the scaling factor. Thus, a second overfilling size obtained by adjusting the first overfilling size may be generated, and the original image may be a basis for generating the image.

1 is a diagram for explaining a black border generated when a virtual reality image is generated.
2 is a diagram for explaining an operation of dynamically overfilling an original image to prevent a blank area, according to various embodiments.
3 is a diagram for explaining an operation of overfilling an original image for each pitch, roll, and yaw axis of a user's head, according to various embodiments.
4 is a diagram for explaining an operation of dynamically overfilling an original image by a server according to various embodiments.
5 is a diagram for explaining a correlation between a first overfilling size and angular velocity of a user's head for each pitch, roll, and yaw axis, according to various embodiments.
6 shows an example of an apparatus, according to various embodiments.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram for explaining a black border generated when a virtual reality image is generated.

Referring to FIG. 1 , due to latency during rendering, the angle of the head of the user 100 viewing the virtual reality image is changed while rendering the original image, which is the basis for generating the virtual reality image. It can change. For example, the angle of the head of the user 100 at the time of rendering the original image is the first angle 141, and the angle of the head of the user 100 is the second angle when the user views the rendered image in virtual reality. (142). When the angle of the head of the user 100 is the first angle 141 according to the change in the angle of the head of the user 100, the user 100 has a field of view (FOV) corresponding to the first plane 110 ), and when the head angle of the user 100 is the second angle 142, the user may have a field of view corresponding to the second plane 120.

After the original image is rendered, it may be reprojected onto the second plane 120 in order to correct the user's 100 field of view, but the rendered original image is a portion 121 of the second plane 120. Pixel information about the partial section 111 on the first plane 110 corresponding to may not be included. As a result, since the reprojected image does not include pixel information about the partial section 121 of the second plane 120, the reprojected image 150 that the user 100 sees in virtual reality is a blank area (black border). ) 155, the completeness of the reprojected image may be deteriorated.

2 is a diagram for explaining an operation of dynamically overfilling an original image to prevent a blank area according to various embodiments, and FIG. It is a diagram for explaining the operation of overfilling for each yaw axis.

Referring to FIG. 2 , according to various embodiments, an original image may be dynamically overfilled to prevent a blank area. The original image includes pixel information about a partial section 211 on the first plane 110 corresponding to the partial section 221 of the second plane 120 when rendered by performing overfilling to increase the resolution. can do. When the overfilled original image is rendered and reprojected onto the second plane 120, the reprojected image 250 that the user 100 sees in virtual reality is a reprojected image when overfilling is not performed (eg : Unlike the reprojection image 150 of FIG. 1 , a blank area (eg, the blank area 155 of FIG. 1 ) may not be included. In order to perform overfilling according to the angular velocity from the first angle 141 to the second angle 142 in the original image, overfilling may be dynamically performed by using a differently generated overfilling size. Since overfilling can be performed dynamically on the original image by performing overfilling with the minimum overfilling size for each angular velocity, compared to the case where overfilling is performed only with a fixed overfilling size, computing power is saved while blank areas (black border) ) can solve the problem.

Referring to FIG. 3 , according to various embodiments, a first angle (eg, first angle 141 of FIG. 2 ), a second angle (eg, second angle 142 of FIG. 2 ), and angular velocity are all It may include values on the three axes of pitch, roll, and yaw, and in the original image, the angular velocity from the first angle 141 to the second angle 142 for each of the three axes of pitch, roll, and yaw Overfilling may be performed dynamically by using differently generated overfilling sizes to perform overfilling according to the above. Since overfilling can be performed on the original image using the minimum overfilling size for each of the three axes, compared to the case of performing uniform overfilling in all directions using a fixed overfilling size, space is saved while saving computing power. The black border problem can be solved.

4 is a diagram for explaining an operation of dynamically overfilling an original image by a server according to various embodiments.

Operations 411 to 416 are for describing operations in which the server 410 generates an overfilled image by performing the overfilling described with reference to FIGS. 2 and 3 on the original image, and renders the overfilled image. can

In operation 411, the server 410 determines a first plane when a user (eg, user 100 of FIG. 2) views an image in virtual reality at a first angle (eg, first angle 141 of FIG. 2). (eg: the first plane 110 in FIG. 2 ) and the user 100 at a second angle (eg, the second angle 142 in FIG. 2 ) when viewing the image in virtual reality. A first overfilling size (eg, model margin) for performing overfilling of the original image may be calculated based on a field of view corresponding to a plane (eg, the second plane 120 of FIG. 2 ). The original image may be a basis for generating an image in virtual reality by being reprojected on the second plane 120 after being rendered. The server 410 may calculate the first overfilling size using a mathematical model (eg, a trigonometric function). The first angle 141 may be the angle of the user's head when the user 100 renders the original image, and the second angle 142 may be the angle of the user's head when the user 100 actually views the image in virtual reality. , and the first angle 141 and the second angle 142 may be obtained (eg, measured) using an inertial measurement unit (IMU). Both the first angle 141 and the second angle 142 may include values on three axes of pitch, roll, and yaw, and the server 410 may include values on three axes of pitch, roll, and yaw. A first overfilling size can be calculated.

In operation 412, the server 410 determines the angular velocity of the user's head angle from the first angle 141 when rendering the original image to the second angle 142 when the user 100 actually views the image in virtual reality. can be obtained (eg calculated). The angular velocity may include values on the three axes of pitch, roll, and yaw, and the server 410 acquires the angular velocity on the three axes of pitch, roll, and yaw to statistically obtain a minimum value of the angular velocity (e.g., velocity minimum ), a threshold value of angular velocity (eg, velocity threshold), and a maximum value of angular velocity (eg, velocity maximum) may be acquired (eg, determined). For example, the minimum angular velocity value may be the smallest angular velocity value that affects generation of the blank area, and the angular velocity threshold value may be the largest angular velocity that the user 100 can generate. The maximum value of the angular velocity may be a scalar multiplication (eg, 100 times) of the threshold value of the angular velocity, or may be an infinite value.

At operation 413, the server 410 may generate a scaling factor based on the angular velocity. The server 410 predicts an applicable scaling factor (eg, minimum scaling factor) corresponding to the minimum value of the angular velocity, predicts an applicable scaling factor (eg, maximum scaling factor) corresponding to the threshold value of the angular velocity, and Corresponding to the maximum value, an applicable scaling factor (eg, extreme scaling factor) may be predicted. The extreme scaling factor may be a scalar multiple (eg, 100 times) of the maximum scaling factor in order to greatly reduce the first overfilling size proportional to the angular velocity as the angular velocity has the maximum value.

In operation 413, the server 410 may generate a scaling factor based on the minimum scaling factor, the maximum scaling factor, and the threshold value of the angular velocity. For example, the scaling factor may be defined by Equation 1.

[Equation 1]

는 각속도의 역치값이고,

는 최대 스케일링 팩터이고,

은 최소 스케일링 팩터일 수 있다. 스케일링 팩터의 역수는 [수학식 2]에서와 같이 각속도와 exponential 함수의 관계성을 가질 수 있다.In Equation 1, F is a scaling factor, V is an angular velocity,

is the threshold value of the angular velocity,

is the maximum scaling factor,

may be the minimum scaling factor. The reciprocal of the scaling factor may have a relationship between the angular velocity and the exponential function as in [Equation 2].

[Equation 2]

In operation 414, the server 410 generates a second overfilling size obtained by adjusting the first overfilling size from the first overfilling size and the scaling factor according to the pitch, raw, and yaw axes by any one of a scalar algorithm and an exponential algorithm. can do. When the scaling factor is greater than the minimum scaling factor and less than the maximum scaling factor according to the scalar algorithm, the server 410 may generate the second overfilling size by dividing the first overfilling size by the scaling factor. When the scaling factor is greater than or equal to the maximum scaling factor according to the scalar algorithm, the server 410 may generate the second overfilling size by dividing the first overfilling size by the extreme scaling factor. The server 410 may generate a second overfilling size by scalar multiplication (eg, 0.875 times) the first overfilling size when the scaling factor is smaller than the minimum scaling factor according to a scalar algorithm. The server 410 may generate a second overfilling size by multiplying the first overfilling size by an inverse of the scaling factor according to an exponential algorithm.

In operation 415, the server 410 may generate an overfilling image by reflecting the second overfilling size. The overfilled image may be one in which the resolution of the original image is increased by reflecting the second overfilling size generated for each of the pitch, raw, and yaw axes.

At operation 416, server 410 may render the overfilled image. The server 410 transmits the rendered image to the client 430 over the network, and the client 430 has the first angle when the user (eg, the user 100 of FIG. 2 ) renders the overfilled image. A first plane (eg, the first plane 110 of FIG. 2 ) viewed from 141 and a second plane viewed from a second angle 142 (eg, the first plane 110 of FIG. 2 ) when the user 100 actually views an image in virtual reality. The rendered image may be reprojected onto the second plane 120 so as not to feel a sense of separation between the second planes 120 of FIG. 2 . The reprojected image 250 may be output (eg, displayed) by the client 430 (eg, a VR device).

5 is a diagram for explaining a correlation between a first overfilling size and angular velocity of a user's head for each pitch, roll, and yaw axis, according to various embodiments.

Referring to FIG. 5 , according to various embodiments, the first overfilling size may be proportional to angular velocities 541 , 542 , and 543 of the user's head for each pitch, roll, and yaw axis. The first overfilling size ratio 510 is the ratio of the number of pixels exceeding the number of pixels included in the original image to the number of pixels included in the original image, and the first overfilling size ratio 510 is 1 It can be a measure of the increase or decrease of the size of overfilling. Since the ratio 510 of the first overfilling size tends to increase as the angular velocity (541, 542, 543) for each pitch, roll, and yaw axis of the user's head increases, the first overfilling size is the pitch, roll, and yaw of the user's head As the angular velocities 541, 542, and 543 for each axis increase, the second overfilling size adjusted (eg, reduced) to a larger width may be generated.

6 shows an example of an apparatus, according to various embodiments.

Referring to FIG. 6 , according to various embodiments, a device 600 may be substantially the same as the server 410 of FIG. 4 .

The memory 610 may store instructions (eg, programs) executable by the processor 630 . For example, the instructions may include instructions for executing an operation of the processor 630 and/or an operation of each component of the processor 630 .

The processor 630 may process data stored in the memory 610 . The processor 630 may execute computer readable code (eg, software) stored in the memory 610 and instructions triggered by the processor 630 .

The processor 630 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program.

For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

An operation performed by the processor 630 may be substantially the same as that of the server 410 described with reference to FIGS. 2 to 5 . Accordingly, detailed descriptions will be omitted.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

In the overfilling method for virtual reality,
based on a field of view corresponding to a first plane when a user views an image in the virtual reality at a first angle and a field of view corresponding to a second plane when the user views the image in the virtual reality at a second angle calculating a first overfilling size for overfilling the original image;
obtaining an angular velocity from the first angle to the second angle;
generating a scaling factor based on the angular velocity; and
An operation of generating a second overfilling size obtained by adjusting the first overfilling size based on the first overfilling size and the scaling factor
including,
The original image,
is the basis for creating the image
The operation of obtaining the angular velocity,
Obtaining the angular velocity and the minimum, threshold, and maximum values of the angular velocity
including,
The operation of generating the scaling factor,
predicting a minimum scaling factor based on the minimum value of the angular velocity;
predicting a maximum scaling factor based on the threshold value of the angular velocity;
predicting an extreme scaling factor based on the maximum value of the angular velocity; and
Generating the scaling factor based on the minimum scaling factor, the maximum scaling factor, and the threshold value of the angular velocity
Including, overfilling method.
delete According to claim 1,
The operation of generating the second overfilling size,
generating the second overfilling size by dividing the first overfilling size by the scaling factor when the scaling factor is larger than the minimum scaling factor and smaller than the maximum scaling factor;
generating the second overfilling size by dividing the first overfilling size by the extreme scaling factor when the scaling factor is greater than or equal to the maximum scaling factor; and
Generating a second overfilling size by scalar multiplication of the first overfilling size when the scaling factor is smaller than the minimum scaling factor
Including, overfilling method.
According to claim 1,
The operation of generating the scaling factor,
An operation of generating the scaling factor by using the angular velocity as an exponent of an exponential function
including,
The operation of generating the second overfilling size,
Multiplying the reciprocal of the scaling factor by the first overfilling size
Including, overfilling method.
In the device for virtual reality,
memory containing instructions; and
A processor electrically connected to the memory and configured to execute the instructions
including,
When the instructions are executed by the processor, the processor:
based on a field of view corresponding to a first plane when a user views an image in the virtual reality at a first angle and a field of view corresponding to a second plane when the user views the image in the virtual reality at a second angle Calculate a first overfilling size for performing overfilling of the original image;
Obtaining an angular velocity from the first angle to the second angle and a minimum value, a threshold value, and a maximum value of the angular velocity,
predicting a minimum scaling factor based on the minimum value of the angular velocity;
predicting a maximum scaling factor based on the threshold value of the angular velocity;
predict an extreme scaling factor based on the maximum value of the angular velocity;
Generating a scaling factor based on the minimum scaling factor, the maximum scaling factor, and a threshold value of the angular velocity;
generating a second overfilling size obtained by adjusting the first overfilling size based on the first overfilling size and the scaling factor;
The original image,
What is the basis for generating the image, the device.

Images