KR20180042419A - A method of generating virtual reality of a head mount display and a head mount display - Google Patents

Abstract

A method for generating a virtual reality of a head mounted display includes the steps of: (a) generating a reference image associated with a first eye through ray tracing; (b) Generating a provisional image by modifying the reference image; and (c) generating a virtual image by performing ray tracing on pixels below a threshold value among the pixels in the provisional image.

Description

A method of generating virtual reality of a head mount display and a head mount display

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual reality technology, and more particularly, to a method of creating a virtual reality on a head-mounted display capable of realizing a virtual reality in a head-mounted display (HMD) .

Virtual Reality, which can be referred to as immersive multimedia or computer simulated life, replicates an environment that simulates the physical reality that exists in the real world or virtual world and allows user interaction in such a world. Virtual reality artificially creates a sensory experience that can include vision, hearing, touch, smell, and taste. Most recent virtual reality environments are displayed on a computer screen or a specific stereoscopic display, and some simulations include additional sensory information and focus on real sound through targeted speakers or headphones for VR users. Currently, some advanced haptic systems include tactile information known as force feedback in medical, game, and military applications. Virtual reality can also be achieved by the use of standard input devices such as keyboards and mice, or via remote devices that provide the virtual reality of a user as concepts of telepresence and telemetricity, or virtual artifacts, through multimodal devices such as wired gloves or omnidirectional treadmills. It covers the communication environment.

The simulation environment can be very similar to the real world created for vivid experiences such as pilot or combat training, or very much like reality, like VR games.

Korean Patent Registration No. 10-0453225 relates to a client system for implementing a 3D virtual reality and a virtual reality implementation method using the same. The client system includes an applet, a communication module, a browser, and an ASP support module. The applet allows the client system to communicate X3D data with the server. The communication module connected to the applet enables communication between the client system and the server. The browser analyzes the X3D data sent by the server through the communication module, creates a scene graph, The scene graph is rendered according to the preset type, and the rendered scene graph is displayed on the display. The ASP support module processes the data transmitted by the server in the external window according to the user event generated in the VR implemented by the browser.

Korean Patent No. 10-0453225 (October 2006)

One embodiment of the present invention seeks to provide a head-mounted display technology capable of generating precise virtual reality in a head-mounted display through ray tracing.

According to an embodiment of the present invention, a head mount display technology capable of reducing the complexity of operation by generating another gaze image using the first gaze image as ray tracing is proposed. Here, the first line-of-sight image (one line-of-sight image) may be an image displayed with one eye (for example, the left eye), and the other eye image may be a video image displayed with the other eye .

An embodiment of the present invention is to provide a head mount display technology capable of real-time interface and reducing eye fatigue.

Among the embodiments, a method for generating a virtual reality of a head-mounted display includes the steps of (a) generating a reference image associated with a first eye through ray tracing, (b) generating a reference image associated with the second eye, Generating a temporary image; and (c) performing ray tracing on pixels below a threshold value among the pixels in the temporary image to generate a virtual image.

In one embodiment, the method may further comprise obtaining a viewpoint formed by the binocular and comprising first and second eyelines and a parallax prior to step (a).

In one embodiment, the step (a) may include generating the reference image by associating the first line of sight with the ray ray using ray tracing.

 The step (b) may include warping the reference image by reflecting a parallax formed by binocularity on the reference image.

 In the step (c), the second line of sight is generated in the reference image in association with the eye ray using ray tracing. When the second line of sight is viewed in three dimensions in the reference image, And performing ray tracing on the corresponding blank space.

In one embodiment, the step (b) includes the steps of (b1) dividing the reference image into a plurality of sub reference images, and (b2) And warp the reference image.

In one embodiment, the step (b1) may further include the step of variably dividing the reference window into a plurality of sub reference images through a plurality of sub windows dividing the reference window.

In one embodiment, the step (c) includes the steps of: (c1) associating the second line of sight with the eye ray using ray tracing to generate a blank space corresponding to the semi-solid area of the plurality of sub- And performing ray tracing on at least one sub reference image including the at least one reference reference image.

In one embodiment, the step (c1) may further comprise performing parallel ray tracing on the plurality of warped reference images.

In one embodiment, the step (b) includes the steps of: (b1) dividing the reference image into a plurality of sub reference images or dividing the reference image into a plurality of first sub reference images including an anti- (B2) warping the plurality of second sub reference images to reflect the parallax formed by both sides of each of the plurality of second sub reference images have.

 The step (c) may include (c1) performing ray tracing on the plurality of first sub reference images by associating the second line with the ray tracing ray tracing.

In one embodiment, the step (c) includes: (c2) generating the virtual image by synthesizing the plurality of first sub reference images generated through the ray tracing and the plurality of wired second sub reference images, As shown in FIG.

Among the embodiments, the head-mounted display includes a point acquiring unit formed by binocular and acquiring a viewpoint including parallax with first and second eyelines, a temporary image generating unit for generating a temporary image associated with the second eye and modifying the reference image and a temporary image generating unit for generating a temporary image related to the pixel below the threshold in the temporary image, And a ray tracing unit for performing ray tracing.

The virtual reality generation method of the head mount display and the head mount display performing the same according to an embodiment of the present invention can generate precise virtual reality in the head mount display through ray tracing.

The method of generating a virtual reality of a head mount display and the head mount display performing the same according to an embodiment of the present invention can reduce the complexity of operation by generating another visual image using one visual image as a light ray. Here, one gaze image may be an image represented by one eye (for example, the left eye), and another eye image may be an image represented by the other eye (for example, the right eye).

The method of generating a virtual reality of a head mount display according to an embodiment of the present invention and the head mount display performing the same can provide a real-time interface to reduce eye fatigue.

FIG. 1 is a conceptual diagram illustrating a configuration of a head-mounted display for performing a method of generating a virtual reality according to an embodiment of the present invention.
Figure 2 is a block diagram illustrating one control in the head-mounted display of Figure 1;
3 is a block diagram illustrating another control in the head-mounted display of Fig.
4 is a flowchart illustrating a virtual reality generation method of the head mount display shown in FIG.
5 is a conceptual diagram for explaining a stereoscopic region and a semi-stereoscopic region.

The description of the present invention is merely an example for structural or functional explanation, and the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas. Also, the purpose or effect of the present invention should not be construed as limiting the scope of the present invention, since it does not mean that a specific embodiment should include all or only such effect.

Meanwhile, the meaning of the terms described in the present application should be understood as follows.

The terms "first "," second ", and the like are intended to distinguish one element from another, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

It is to be understood that the singular " include " or "have" are to be construed as including the stated feature, number, step, operation, It is to be understood that the combination is intended to specify that it does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

In each step, the identification code (e.g., a, b, c, etc.) is used for convenience of explanation, the identification code does not describe the order of each step, Unless otherwise stated, it may occur differently from the stated order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

The present invention can be embodied as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system . Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also implemented in the form of a carrier wave (for example, transmission over the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used predefined terms should be interpreted to be consistent with the meanings in the context of the related art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present application.

The present invention is supported by the Seoul Industrial Promotion Agency as a 'patent technology commercialization technology development support project'. From September 1, 2013 to August 31, 2014, a software kit for development of ray tracing rendering content by SILICONARTS, INC. Development 'project.

[National R & D Tasks Supporting the Present Invention]

[Project Serial Number] PA130017

[Department] Seoul Metropolitan Government

[Research Management Organization] Seoul Industrial Promotion Agency

[Research Project] Seoul Industry-Academia Collaboration Project (2013 Patent Technology Commercialization Technology Development Support Project)

[Research Subject] Development of a software kit for developing ray tracing rendering contents

[Contribution rate] 1/1

[Main Organization] Silicon Arts Co., Ltd.

[Research period] September 1, 2013 to August 31, 2014

1 is a conceptual diagram showing a system of a head-mounted display for performing a method of generating a virtual reality according to an embodiment of the present invention.

Referring to FIG. 1, a head-mounted display 10 system includes a virtual object 50 and a head-mounted display (HMD) 100.

The virtual object 50 may correspond to a virtual focusing area formed by the HMD 100. [ That is, the virtual object 50 may be a 3D object that is not actually, but three-dimensionally expressed. In one embodiment, the virtual object 50 may be generated entirely by the HMD 100. In another embodiment, the virtual object 50 may be created by combining an actual object located in front of the HMD 100 with an object partially generated by the HMD 100. [

The HMD 100 includes a left display unit 110, a right display unit 120, a network interface unit 130, and a control unit 140. The HMD 100 can be worn on the head or as part of a helmet, and there is a small display optic in front of each eye (binocular HMD).

The left and right display portions 110 and 120 may correspond to a display optics in front of the eye. In one embodiment, the left display unit 110 generates a left eye image and the right display unit 120 generates a right eye image. Accordingly, the left and right display units 110 and 120 are virtual And generates a stereoscopic image.

The network interface unit 130 can transmit left and right images to an external device and can receive stereoscopic images or left and right images from an external device. In one embodiment, the network interface unit 130 may be implemented as Bluetooth, WiFi, or the like.

The control unit 140 may generate a reference image associated with the first eye through ray tracing. The control unit 140 may transform the reference image to generate a temporary image related to the second eye and perform ray tracing on the error pixel (i.e., below the threshold pixel) in the temporary image to complete the other side image for the other eye . Finally, the control unit 140 may combine the side image and the side image to generate a virtual image. The control unit 140 will be described with reference to Figs. 2 and 3. Fig.

2 is a block diagram illustrating one control of the head-mounted display of Fig.

Referring to FIG. 2, a head mounted display (HMD) 100 includes a view obtaining unit 210, a reference image generating unit 220, a temporary image generating unit 230, a ray tracing unit 240, And an HMD display unit 250.

The viewpoint acquiring unit 210 acquires a viewpoint formed by the binocular and including first and second eyelines and a parallax. For example, the first line of sight corresponds to the line of sight of the left eye, and the second line of sight corresponds to the line of sight of the right eye. The parallax may correspond to a binocular parallax caused by looking at the virtual object 50 at a position spatially different from the left eye and the right eye. The reference image generator 220 generates a reference image associated with a first eye through ray tracing. In one embodiment, the reference image generator 220 generates the reference image by associating the first line of sight with the eye ray using ray tracing. For example, the reference image generating unit 220 may perform ray tracing based on a line of sight of the first eye (e.g., the left eye), i.e., the first line of sight, (Reference image). Ray tracing sets an object to be rendered, a light source, and a line of sight, and simulates a ray reflected from a light source from a virtual object 50 to calculate a color in each pixel, thereby rendering an image .

The temporary image generating unit 230 generates a temporary image of the second eye by modifying the reference image. For example, the temporary image generating unit 230 may change the viewpoint of the reference image from the first eye to the second eye to generate the temporary image. The temporary image may be incomplete for the second child due to some area visible only in the first child (i.e., not visible in the second child).

In one embodiment, the temporary image generator 230 may generate a temporary image by warping the reference image by reflecting the parallax formed by the binocular on the reference image. The temporary image generating unit 230 may distort the reference image into a temporary image, and may use a time difference for changing the viewpoint from the first eye to the second eye. This can calculate the stereoscopic direction and the lighting effect for the virtual object 50. [ That is, the temporary image generating unit 230 may generate a temporary image by performing image warping on the reference image, thereby reflecting the distortion that may occur due to the parallax difference in the reference image.

The ray tracing unit 240 performs ray tracing by relating a second line of sight to an error pixel (i.e., a sub-threshold pixel) in the temporary image with the ray. Here, the error pixel may include a pixel that improperly reflects the color value or a pixel that improperly reflects the stereoscopic direction.

When a temporary image is generated by distorting the reference image, an error pixel may be generated due to the parallax. Then, the ray tracing unit 240 may perform ray tracing for pixels below the threshold value among the pixels of the temporary image to obtain pixel values to correct the error pixels. In one embodiment, error pixels due to parallax may be grouped into a plurality of regions. One of the areas corresponds to a blank area indicating an semi-stereoscopic area that is not visually seen through the first line when the warped reference image (i.e., temporary image) is generated.

The HMD display unit 250 combines the reference image generated by the reference image generating unit 220 and the reference image generated by the ray tracing unit 240 and warped to the first eye, Can be provided.

3 is a block diagram illustrating another control portion of the head-mounted display of Fig. 3 shows an example of dividing a reference image into a plurality of sub-images, warping each of the divided sub-images, and correcting the divided sub-images through ray tracing to generate a virtual image.

3, the head mount display 100 includes a viewpoint acquisition unit 310, a reference image generation unit 320, a video segmentation unit 332, and a temporary temporal image generation unit 332, An image generating unit 330, a ray tracing unit 340, and an HMD display unit 350.

The functions of the viewpoint acquisition unit 310, the reference image generation unit 320, the ray tracing unit 340, and the HMD display unit 350 are the same as those of the corresponding configuration of FIG. Hereinafter, for convenience of explanation, the main difference will be mainly described.

3, the temporary image generating unit 330 may include an image dividing unit 332 and a divided temporary image generating unit 334. The image divider 332 divides the reference image into a plurality of sub reference images (for example, a left image). In one embodiment, the image divider 332 may variably divide the reference image into a plurality of sub-reference images through a plurality of sub-windows dividing the reference window, have. For example, if the size of the reference window is 16 * 16 (pixels * pixels), the size of the plurality of subwindows may include 4 * 4, 8 * 8 and 16 * 16.

In one embodiment, the image divider 332 may divide the reference image using a plurality of sub-windows of one kind. For example, the image divider 332 may divide the reference image into four sub-reference images using an 8 * 8 sub-window. In another embodiment, the image divider 332 may divide the reference image using a plurality of sub-windows of various kinds. For example, the image dividing unit 332 divides the reference image into three 8 * 8 size sub reference images and four 4 * 4 sub-reference images using a sub-window of 8 * 8 size and a sub- It may be divided into sub-reference images of four sizes.

The split temporal image generating unit 334 generates a temporary image by warping the parallax formed by the binaural eye 330 with respect to a plurality of sub reference images. The divided temporal image generating unit 240 may generate a temporal image by performing image warping on each of the plurality of sub reference images. That is, the divided temporal image generating unit 334 may perform a video image warping on each of the plurality of sub-reference images to generate a temporary image.

The ray tracing unit 340 associates a second line of sight with a wired sub reference image having one error pixel (i.e., a sub-threshold pixel) or error pixel among a plurality of warped sub reference images, Perform tracing. In one embodiment, the ray tracing unit 340 may perform ray tracing in parallel. Since a ray tracing process for an error pixel is described in the ray tracing unit 240, a ray tracing process for a wafed sub reference image having an error pixel is described.

In one embodiment, the ray tracing unit 340 generates a ray traced image for at least one of sub-reference images including an error pixel or a blank region (e.g., an area below a threshold pixel) Ray tracing may be performed by associating the two eyes with the eye ray. Here, the blank region may represent an anti-stereoscopic region for a plurality of wafed sub reference images representing an area that is not visible from the other eye when viewed from one eye in the reference image.

In another embodiment, the image divider 332 divides the reference image into a plurality of sub reference images or divides the reference image into a plurality of first sub reference images including a semi-solid area and a plurality of non- Two sub reference images can be distinguished. The divided temporal image generating unit 334 may generate a temporal image by warping a plurality of second sub-reference images by reflecting the parallax formed by the binocular.

In this case, the ray tracing unit 340 performs ray tracing by associating the second line of sight with respect to the plurality of first sub reference images with the eye ray used for ray tracing, And generate a ray traced waffer image by combining the plurality of second sub reference images.

Fig. 4 is a flowchart illustrating a virtual reality generation method of the head-mounted display system shown in Fig. 1, and Fig. 5 is a conceptual diagram showing a stereoscopic and semi-stereoscopic region.

 4, after the viewpoint acquiring unit 210 or 310 acquires the viewpoint including the first and second eyes and the parallax formed by binocular vision, the reference image generator 220 or 320 generates the viewpoint To generate a reference image associated with the first child (step S410).

The temporary image generation unit 230 or 330 generates a temporary image associated with the second child and modifying the reference image (step S320). In one embodiment, the temporary image generator 230 or 330 may reflect the parallax formed by binocularity in the reference image and warp the reference image to generate a temporary image. The temporary image generating unit 230 or 330 may transform the reference image in whole or in part.

The ray tracing unit 240 or 340 performs ray tracing using the second eye line with respect to the error pixel (i.e., the sub-threshold pixel) in the temporary image (step S430).

In one embodiment, the ray tracing unit 240 or 340 performs ray tracing for each error pixel in the temporal image or for each of the error subblocks therein, each of which is segmented with error pixels And shows the converted reference image. Finally, the ray tracing unit 240 or 340 generates ray-traced and transformed reference images.

5, if the reference image is a left image viewed from the left eye, the temporary image generating unit 230 or 330 transforms the reference image to generate a right eye temporary image. The steric region 510 may include an appropriately modified region as viewed in an amount. The semi-solid area 250 may include invisible and inappropriately deformed areas in the left eye when viewed in the right eye. The ray tracing unit 240 or 340 is included in the semi-solid area 520 and searches for an error pixel in a temporary image in which ray tracing is performed. In one embodiment, the error pixel may be used as a sub-block containing an error pixel.

The HMD display unit 250 or 350 generates a virtual image in an amount. That is, the HMD display unit 250 or 350 provides the reference image to one display unit (e.g., the left display unit 110), and outputs the ray-traced reference image to another display unit (e.g., The right display unit 120).

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the present invention as defined by the following claims It can be understood that

100: Head Mounted Display (HMD)
110:
120: right display unit
130: Network interface unit
140:
210:
220: Reference image generating section
230: temporary image generating unit
240: ray tracing unit
250: HMD display unit

Claims (13)

(a) generating a reference image associated with a first eye through ray tracing;
(b) generating a temporary image associated with the second child and generated by transforming the reference image; And
(c) generating a virtual image by performing ray tracing on a pixel below a threshold value among pixels in the temporary image to generate a virtual image of the HMD (Head-Mounted Display). The method of claim 1,
Further comprising the step of acquiring a viewpoint formed by both eyes and including a parallax with first and second eyelines prior to step (a) Generation method. The method of claim 1, wherein step (a)
And generating the reference image by associating an eye ray used for ray tracing with a first eyeline. 2. The method of claim 1, wherein step (b)
And warp the reference image by reflecting a parallax formed by binocular vision in the reference image. 5. The method of claim 4, wherein step (c)
Performing ray tracing for a blank area by associating an eye ray and a second eyeline used for ray tracing,
Wherein the blank region corresponds to a non-stereoscopic region that is not visually displayed through the first eyeline when the reference image is stereoscopically displayed. 2. The method of claim 1, wherein step (b)
(b1) dividing the reference image into a plurality of sub reference images; And
(b2) warping the reference image by reflecting parallax generated by binocularity on each of the plurality of sub reference images. 7. The method of claim 6, wherein the step (b1)
Further comprising the step of variably dividing the plurality of sub reference images into a plurality of sub reference images through a combination of a plurality of sub windows,
Wherein the combination completes a reference window. 7. The method of claim 6, wherein step (c)
(c1) performing ray tracing on at least one sub reference image by associating an eye ray and a second eye line used in ray tracing,
Wherein the at least one sub reference image includes a blank area corresponding to a non-stereoscopic area among the plurality of wafted sub reference images. 9. The method of claim 8, wherein step (c1)
Further comprising performing parallel ray tracing on the plurality of warped sub reference images. ≪ RTI ID = 0.0 > 8. < / RTI > 2. The method of claim 1, wherein step (b)
(b1) before or after dividing the reference image into a plurality of sub reference images, a plurality of first sub reference images having a divided region and a plurality of second sub reference images having no divided region are divided into a semi-dimensional region A step of discriminating; And
(b2) warping the plurality of second sub reference images reflecting the parallax formed by both eyes on each of the plurality of second sub reference images . 11. The method of claim 10, wherein step (c)
(c1) performing ray tracing on the plurality of first sub reference images by associating a second eyeline with an eye ray used for ray tracing; and Wherein the HMD is a virtual reality. 12. The method of claim 11, wherein step (c)
(c2) combining the plurality of first sub reference images generated through the ray tracing and the plurality of warped second sub reference images to generate the virtual image How to create virtual reality in HMD. A point obtaining unit for obtaining a viewpoint formed by both eyes and including a first and second eyelines and a parallax;
A reference image generator configured to generate a reference image associated with a first eye through ray tracing;
A temporary image generation unit associated with the second child and generating a temporary image obtained by modifying the reference image; And
And a ray tracing section for performing ray tracing on pixels below the threshold among the pixels in the temporary image.

Images