KR20020053986A - Three-dimensional display apparatus and method with gaze point feedback - Google Patents

Abstract

PURPOSE: A stereo image display apparatus and method is provided to prevent eye fatigue by permitting the user to gaze the desired point and change the gaze point in a free manner. CONSTITUTION: A stereo image display apparatus comprises a three-dimensional model storage unit(11) for creating and storing a three-dimensional model for an object to be displayed in a virtual reality space; a head and eye movement detection unit(16) for detecting the position of head(face) of a user and extracting images of eyes of the user; a gaze direction and distance measurement unit(12) for extracting information for the current gaze point of the user, from the position of head and images of eyes output from the head and eye movement detection unit; an image creating unit(13) for generating the stereo image corresponding to the current gaze point extracted from the gaze direction and distance measurement unit, on the basis of the three-dimensional model of the object stored in the three-dimensional model storage unit; and display units(14,15) for displaying left and right side images created by the image creating unit.

Description

Three-dimensional display apparatus and method with gaze point feedback

The present invention relates to a stereoscopic image display apparatus that follows a gaze point, and a method and a computer readable recording medium recording a program for realizing the method. More specifically, the present invention relates to a viewer's head (face) position and eye movement. By actively displaying the stereo image corresponding to the gaze point based on the extracted gaze point information, the viewer can observe the desired point in the field of view and change the gaze point arbitrarily by removing the constraint that the viewer needs to adjust the focal length. The present invention relates to a three-dimensional image display device for removing eye fatigue and to provide a natural image when viewing a three-dimensional image, and a method and a computer-readable recording medium having recorded thereon a program for realizing the method.

A person has two left and right eyes, and the positions of the two eyes are different so that the image formed on the retina of the right eye and the image formed on the retina of the left eye are different. However, each object entering the field of view also differs in the position of the image on the left and right eyes depending on the distance from the viewer. In other words, the closer the object is, the more the image is formed on both eyes, while the farther the object is, the less the image is on the eyes. Therefore, the information on the distance can be restored from the difference between the images formed in the left and right eyes, thereby feeling a three-dimensional effect.

By applying this principle, different images can be seen by both eyes, thereby realizing stereoscopic images. This method is currently used in the implementation of stereoscopic movies and virtual reality.

However, in spite of the excellent reality provided by the three-dimensional image, such a device is not widely used because there is a problem that eyes are easily tired when viewing the three-dimensional image. The reason why the eyes are tired easily is that in the conventional stereoscopic image display method, the left and right images are predetermined at every moment, forcing the viewer to follow the gaze point according to the given image.

However, in daily life, people move their faces or eyes to see where they want to go, and following the gaze point as specified is very unnatural and puts a lot of strain on the eyes.

As described above, in the conventional stereoscopic image display method, since the focal length is unilaterally given in accordance with the focal length in the two stereo images representing the stereoscopic image, the eye is forced to follow the eye even when the stereoscopic image is viewed for a short time. This hurts, head dizzy phenomenon occurs.

The present invention has been proposed in order to solve the above problems, a stereoscopic image display device for extracting a gaze point by tracking the movement of the face (eye) and eyeball every moment, and to display a stereo image corresponding to the gaze point; It is an object of the present invention to provide a computer-readable recording medium that records the method and a program for realizing the method.

That is, the present invention actively displays stereo images corresponding to the gaze point based on gaze point information extracted from the spectator's head (face) position and eyeball movement, thereby eliminating the constraint that the viewer needs to adjust the focal length. 3D image display device and method and a program for realizing the above method to provide a natural image by eliminating eye fatigue when viewing 3D image by allowing the user to gaze at a desired point in the field of view and change the gaze point arbitrarily. Its purpose is to provide a computer-readable recording medium having recorded thereon.

1 is a block diagram of a virtual reality stereoscopic image display device according to an embodiment of the present invention.

2 is a flowchart illustrating a virtual reality stereoscopic image display method according to an embodiment of the present invention.

3 is a block diagram of a three-dimensional image display device for a natural image according to another embodiment of the present invention.

4 is a flowchart illustrating a stereoscopic image display method for a natural image according to another embodiment of the present invention.

* Explanation of symbols for the main parts of the drawings

11: 3D model storage unit 12, 33: gaze direction and distance measuring unit

13: Image generating unit 14,35: Left image display unit

15,36: Right image display

16,37: head and eye movement detector and stereo image display

31: Representative image storage unit for each gaze point 32: Image selection unit

34: disparity adjusting unit

According to an aspect of the present invention, there is provided a stereoscopic image display apparatus which follows a gaze point, comprising: three-dimensional model storage means for generating and storing a three-dimensional model of an object to be displayed in a virtual reality space; Head and eye movement detecting means for extracting a head (face) position of a user and extracting images of two eyes; Gaze direction and distance measuring means for extracting information on the current gaze point of the user from the head position and eyeball image received from the head and eye movement detecting means; Image generating means for generating a stereo image corresponding to a current gaze point extracted by the gaze direction and distance measuring means based on a three-dimensional model of an object stored in the three-dimensional model storage means; And display means for displaying left and right images generated by the image generating means.

In addition, another apparatus of the present invention, a stereoscopic image display apparatus that follows the gaze point, the gaze point representative image storage means for storing the reference stereo images for the reference gaze point; Head and eye movement detecting means for extracting a head (face) position of a user and extracting images of two eyes; Gaze direction and distance measuring means for extracting information on the current gaze point of the user from the head position and eyeball image received from the head and eye movement detecting means; Image selection means for selecting a reference stereo image closest to the current gaze point extracted by the gaze direction and distance measuring means from the representative image storing means for each gaze point; Disparity adjusting means for correcting the position of the image in a direction of reducing the difference between the current gaze point extracted by the gaze direction and the distance measuring means and a focal length (stare point) of the reference stereo image selected by the image selecting means; And display means for displaying left and right images corrected by the disparity adjusting means.

On the other hand, the method of the present invention, a stereoscopic image display method applied to the three-dimensional image display device along the gaze point, the first step of generating and storing a three-dimensional model of the object to be displayed in the virtual reality space; A second step of extracting information about a gaze point by detecting movement of a face (head) and an eye of a user; Generating a stereo image of the extracted gaze point based on the 3D model of the generated and stored object; And a fourth step of displaying left and right images of the generated stereo image.

In addition, the method of the present invention, in order to obtain the exact correspondence between the gaze point that the user actually wants to see and the three-dimensional image in the virtual reality space, first to sequentially view the objects at a predetermined position to the user It further comprises a fifth step of adjusting the reference point.

According to another aspect of the present invention, there is provided a stereoscopic image display method applied to a stereoscopic image display apparatus following a gaze point, the method comprising: a first step of storing reference stereo images corresponding to a reference gaze point; A second step of extracting information about a gaze point by detecting movement of a face (head) and an eye of a user; Selecting a reference stereo image closest to the extracted current gaze point from the stored reference stereo images; A fourth step of correcting the position of the image in a direction of reducing a difference between the extracted current gaze point and a focal length (stare point) of the selected reference stereo image; And a fifth step of displaying left and right images of the corrected stereo image.

Meanwhile, the present invention provides a stereoscopic image display device having a processor, comprising: a first function of generating and storing a 3D model of an object to be displayed in a virtual reality space; A second function of detecting movement of a user's face (head) and eyes and extracting information on a gaze point; A third function of generating a stereo image of the extracted gaze point based on the 3D model of the generated and stored object; And a computer-readable recording medium having recorded thereon a program for realizing a fourth function of displaying left and right images of the generated stereo image.

In addition, the present invention, in order to obtain the exact correspondence between the gaze point that the user actually wants to see and the 3D image in the virtual reality space, the user first to sequentially see the objects at a predetermined position to the reference point A computer readable recording medium having recorded thereon a program for further realizing a fifth function of adjusting the function is provided.

The present invention also provides a stereoscopic image display device having a processor, comprising: a first function of storing reference stereo images corresponding to a reference gaze point; A second function of detecting movement of a user's face (head) and eyes and extracting information on a gaze point; A third function of selecting from the stored reference stereo images a reference stereo image closest to the extracted current gaze point; A fourth function of correcting a position of an image in a direction of reducing a difference between a focal length (stare point) of the extracted current gaze point and the selected reference stereo image; And a computer readable recording medium having recorded thereon a program for realizing a fifth function of displaying left and right images of the corrected stereo image.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a virtual reality stereoscopic image display device according to an embodiment of the present invention.

As shown in FIG. 1, a virtual reality stereoscopic image display apparatus according to the present invention includes a three-dimensional model storage unit for generating and storing a three-dimensional model of an object existing in a virtual reality space to be viewed by a user in advance. 11) the user from the head and eye movement detection unit 16, the head and eye movement detection unit 16 for extracting the position of the head (face) and extract the image of the two eyes, the head and eye movement detection unit 16 The gaze direction and distance measuring unit 12 for extracting information about the current gaze point of the gaze direction and distance measuring unit based on the three-dimensional model of the object stored in the three-dimensional model storage unit 11 ( An image generator 13 for generating a stereo image corresponding to the current gaze point extracted in 12) and a left image for displaying a left image generated by the image generator 13; Display the left and right images from the display unit 14, the right image display unit 15 for displaying the right image generated by the image generating unit 13 and the left and right image display units 14 and 15 on the actual screen A stereo image display unit 16 is included.

Next, operation of the virtual reality stereoscopic image display device according to the present invention will be described.

As shown in the figure, a three-dimensional model of an object existing in the virtual reality space to be seen by the user is generated in advance and stored in the three-dimensional model storage unit 11. The user wears a device including a stereo image display unit 16 through which a corresponding stereo image is projected to the left and right eyes. In addition to the image projection device, the device includes a head and eye movement detector 16 for detecting the position of the head (face) and extracting images of two eyes.

Then, the gaze direction and distance measuring unit 12 extracts the current gaze point of the user from the eyeball image and the head position transmitted from the device mounted by the user. At this time, it is necessary to first adjust the reference point in order to obtain an accurate correspondence between the gaze point that the user actually wants to see and the 3D image in the virtual reality space. This is done by letting the user view the objects in the predetermined location sequentially.

Next, the image generating unit 13 generates a stereo image corresponding to the gaze point extracted by the gaze direction and distance measuring unit 12 based on the 3D object model stored in the 3D model storage unit 11. do. Thereafter, the image generated by the image generating unit 13 is projected onto the corresponding display device through the left image display unit 14 and the right image display unit 15.

This operation is performed periodically, and the period is set at an interval required according to the application.

2 is a flowchart illustrating a virtual reality stereoscopic image display method according to an embodiment of the present invention. Since this method has been described in detail with reference to FIG. 1, the processing procedure will be briefly described herein.

First, a 3D model of an object to be displayed in the virtual reality space is generated in advance and stored (21).

Thereafter, the movement of the face (head) and the eye of the viewer (user) is detected to extract information about the gaze point (22).

Next, a stereo image of the extracted gaze point is generated based on the 3D model of the object, which is generated and stored in advance (23).

Thereafter, the left and right images of the generated stereo image are projected and displayed on an actual screen (24).

3 is a block diagram of a stereoscopic image display apparatus for a natural image according to another exemplary embodiment of the present invention.

As shown in FIG. 3, the stereoscopic image display apparatus for a natural image according to the present invention includes a representative image storage unit 31 for each gaze point, which stores reference stereo images corresponding to a representative gaze point as a reference. The current gaze point of the user from the head and eyeball motion detection unit 37 and the head and eyeball motion image received from the head and eyeball motion detecting unit 37 for extracting the position of the face and extracting the images of the two eyeballs. The gaze direction and distance measuring unit 33 for extracting information about the gaze direction and distance measuring unit 33 extracts the reference stereo image closest to the current gaze point representative image storage unit 31 for each gaze point. ) The current gaze point extracted by the gaze direction and distance measuring unit 33 and the device selected by the image selector 32. Disparity adjuster 34 for correcting the position of the image in a direction of reducing the difference in focal length (stare point) of the quasi-stereo image, and left image display for displaying the left image corrected by the disparity adjuster 34 ( 35) a right image display unit 36 for displaying a right image corrected by the disparity adjuster 34 and left and right images from the left and right image display units 35 and 36 on an actual screen; Stereo image display unit 37 is included.

Next, the operation of the stereoscopic image display apparatus for the natural image according to the present invention will be described.

As shown in the figure, reference stereo images corresponding to a representative gaze point as a reference are stored in the representative image storage unit 31 for each gaze point. The user wears a device including a stereo image display unit 37 for projecting the stereo image on the left and right eyes. In addition to the image projection apparatus, the apparatus includes a head and eye movement detector 37 for detecting the position of the head (face) and extracting images of two eyes.

The gaze direction and distance measuring unit 33 extracts the current gaze point of the user from the eyeball image and the head position transmitted from the device mounted by the user. However, unlike an example of generating an image corresponding to an arbitrary gaze point based on a 3D model in a virtual reality space, a natural image is difficult to generate an image.

Therefore, the image selector 32 selects the reference stereo image closest to the current gaze point from among the reference stereo images corresponding to the representative gaze point which is the stored reference. Thereafter, the disparity adjuster 34 reduces the difference between the current gaze point extracted by the gaze direction and the distance measurer 33 and the focal length (stare gaze point) of the reference stereo image selected by the image selector 32. To correct the position of the image.

Thereafter, the image corrected by the disparity adjuster 34 is projected onto the corresponding display device through the left image display unit 35 and the right image display unit 36.

This operation is performed periodically, and the period is set at an interval required according to the application.

4 is a flowchart illustrating a stereoscopic image display method for a natural image according to another embodiment of the present invention. Since this method has been described in detail with reference to FIG. 1, the processing procedure will be briefly described herein.

First, reference stereo images corresponding to a representative gaze point as a reference are stored (41).

Thereafter, the movement of the face (head) and the eye of the viewer (user) is detected to extract information about the gaze point (42).

Next, a reference stereo image closest to the extracted current gaze point is selected from the stored reference stereo images (43).

Thereafter, the position of the image is corrected in a direction of reducing the difference between the extracted current gaze point and the focal length (stare point) of the selected reference stereo image (44).

Next, the left and right images of the corrected stereo image are projected and displayed on an actual screen (45).

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be apparent to those of ordinary knowledge.

The present invention as described above, by actively displaying the stereo image corresponding to the gaze point based on the gaze point information extracted from the head (face) position and eyeball movement of the spectator, the limitation of the existing method that the viewer must focus the focal length By eliminating the spectators can observe the desired point in the field of view arbitrarily and change the gaze point arbitrarily, there is an effect that can eliminate the fatigue of eyes and provide a natural image when viewing the stereoscopic image.

In addition, the present invention can be applied to various fields such as stereoscopic movies, computer games for providing stereoscopic images, virtual reality display, virtual reality communication, stereoscopic display of still images in the exhibition hall.

Claims (11)

In the stereoscopic image display apparatus that follows the gaze point, Three-dimensional model storage means for generating and storing a three-dimensional model for the object to be displayed in the virtual reality space; Head and eye movement detecting means for extracting a head (face) position of a user and extracting images of two eyes; Gaze direction and distance measuring means for extracting information on the current gaze point of the user from the head position and eyeball image received from the head and eye movement detecting means; Image generating means for generating a stereo image corresponding to a current gaze point extracted by the gaze direction and distance measuring means based on a three-dimensional model of an object stored in the three-dimensional model storage means; And Display means for displaying left and right images generated by the image generating means Stereoscopic image display device comprising a. The method of claim 1, The display means, Left image display means for displaying a left image generated by said image generating means; Right image display means for displaying a right image generated by said image generating means; And Stereo image display means for displaying the left and right images from the left and right image display means on an actual screen Stereoscopic image display device comprising a. The method according to claim 1 or 2, The gaze direction and distance measuring means, In order to obtain the exact correspondence between the gaze point that the user actually wants to see and the 3D image in the virtual reality space, the user first adjusts the reference point by sequentially viewing the objects at a predetermined position. Stereoscopic image display device. In the stereoscopic image display apparatus that follows the gaze point, Representative image storage means for each gaze point storing reference stereo images with respect to the gaze point; Head and eye movement detecting means for extracting a head (face) position of a user and extracting images of two eyes; Gaze direction and distance measuring means for extracting information on the current gaze point of the user from the head position and eyeball image received from the head and eye movement detecting means; Image selection means for selecting a reference stereo image closest to the current gaze point extracted by the gaze direction and distance measuring means from the representative image storing means for each gaze point; Disparity adjusting means for correcting the position of the image in a direction of reducing the difference between the current gaze point extracted by the gaze direction and the distance measuring means and a focal length (stare point) of the reference stereo image selected by the image selecting means; And Display means for displaying left and right images corrected by the disparity adjusting means Stereoscopic image display device comprising a. The method of claim 4, wherein The display means, Left image display means for displaying a left image corrected by the disparity adjusting means; Right image display means for displaying the right image corrected by the disparity adjusting means; And Stereo image display means for displaying the left and right images from the left and right image display means on an actual screen Stereoscopic image display device comprising a. In the stereoscopic image display method applied to the stereoscopic image display device along the gaze point, A first step of generating and storing a 3D model of the object to be displayed in the virtual reality space; A second step of extracting information about a gaze point by detecting movement of a face (head) and an eye of a user; Generating a stereo image of the extracted gaze point based on the 3D model of the generated and stored object; And A fourth step of displaying left and right images of the generated stereo image Stereoscopic image display method comprising a. The method of claim 6, A fifth step of adjusting a reference point by first allowing the user to sequentially view objects at a predetermined position in order to obtain an accurate correspondence between the gaze point that the user actually wants to see and the 3D image in the virtual reality space The stereoscopic image display method further comprising. In the stereoscopic image display method applied to the stereoscopic image display device along the gaze point, A first step of storing reference stereo images corresponding to a reference gaze point; A second step of extracting information about a gaze point by detecting movement of a face (head) and an eye of a user; Selecting a reference stereo image closest to the extracted current gaze point from the stored reference stereo images; A fourth step of correcting the position of the image in a direction of reducing a difference between the extracted current gaze point and a focal length (stare point) of the selected reference stereo image; And A fifth step of displaying left and right images of the corrected stereo image Stereoscopic image display method comprising a. In a stereoscopic image display device having a processor, A first function of generating and storing a 3D model of an object to be displayed in the virtual reality space; A second function of detecting movement of a user's face (head) and eyes and extracting information on a gaze point; A third function of generating a stereo image of the extracted gaze point based on the 3D model of the generated and stored object; And A fourth function of displaying left and right images of the generated stereo image A computer-readable recording medium having recorded thereon a program for realizing this. The method of claim 9, A fifth function of adjusting a reference point by first allowing the user to sequentially view objects at a predetermined position in order to obtain an accurate correspondence relationship between a gaze point that the user actually wants to see and a 3D image in a virtual reality space A computer-readable recording medium that records a program for further realization. In a stereoscopic image display device having a processor, A first function of storing reference stereo images corresponding to a reference gaze point; A second function of detecting movement of a user's face (head) and eyes and extracting information on a gaze point; A third function of selecting from the stored reference stereo images a reference stereo image closest to the extracted current gaze point; A fourth function of correcting a position of an image in a direction of reducing a difference between a focal length (stare point) of the extracted current gaze point and the selected reference stereo image; And A fifth function of displaying left and right images of the corrected stereo image A computer-readable recording medium having recorded thereon a program for realizing this.