JPH0415772A - Visual line following type high speed image generation/ display method - Google Patents

Abstract

PURPOSE:To reduce the calculation quantity and to generate and display the images of high quality at a high speed by producing a difference of modelling hierarchies between the vicinity and the periphery of a gazed subject. CONSTITUTION:The balls having the prescribed radii STh1 and STh2 are supposed around the spatial gazed point positions eo1 and eo2. A model having the finer hierarchy by one stage s used at the inside of the STh1, and a model having a hierarchy more rough by one stage is used at the outside of the STh2 respectively. In other words, the inside of the STh1 is changed to a model (a) with a subject obj1, and the inside of a deformed STh1 is changed to a model (b) with a subject obj2 respectively. Thus the subject is modelled relatively fine and rough at the points close to and far from 0(w) respectively. Meanwhile the subject is modelled relatively fine and rough at the vicinity and the periphery of a space gazing point eo(w) respectively. As a result, the calculation quantity is reduced.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a line-of-sight tracking type high-speed image generation and display method. This invention relates to a follow-up high-speed image generation and display method.

[Prior art] Generating and displaying a three-dimensional image by a computer,
So-called computer graphics technology is being used in a variety of fields, including simulations of scientific and technical calculations. Currently, no technology has been found that satisfies this requirement. In other words, if you try to generate a high-quality image, a huge amount of calculation time is required for image generation, and on the other hand, if you aim for high-speed, real-time display such as displaying 10 images per second, it will require rough calculations. The quality of the generated images will be poor.

On the other hand, the field of application of computer graphics is also expanding to the field of image communication called intelligent coded communication. In this method, the transmitting side uses image recognition to extract three-dimensional structural information of the transmission target, parameterizes its characteristics, and transmits the information. On the receiving side, three-dimensional structure information (database) is prepared in advance, and the three-dimensional database is converted and displayed at high speed based on the transmitted feature parameters. In these communication methods, the receiving side communicates with the sender's 3
Since it is possible to freely generate and display images from any viewpoint based on dimensional structure information, it has many advantages such as the following.

■ Motion vision can be realized by detecting the movement of the recipient's viewpoint and displaying an image that corresponds to this movement. Here, kinesthetic vision refers to a person's potential ability to perceive space from changes in the image reflected on the retina of the eyeball when moving the head. Trying to raise awareness through movement is a common experience. That is, by artificially realizing motion vision using computer graphics, it is expected that the three-dimensional effect of images on the screen will be improved, making it possible to display images with a rich sense of realism.

■ Binocular stereopsis can be achieved by using the receiver's eyes as two viewpoints and generating and displaying images from each viewpoint. Since the human eyes are located approximately 6 cm apart, different images are projected onto the winter eye's retinas even without moving the head. The correspondence between these two different images is called parallax information, and people use this to obtain a three-dimensional sense of space. By creating an image with this parallax information using computer graphics, the three-dimensional effect of the image on the screen is further improved.

■ In addition, it is also possible to easily generate images that show the person on the other end (sender) having a conversation while making eye contact.

However, there are currently some difficulties in implementing such a communication method. The first is image recognition and feature parameter extraction on the transmitting side, and the second is an image generation and display method on the receiving side that displays a wide field of view at high speed and with high resolution.

In communication, real-time processing is essential, and there is a higher demand for faster processing than in conventional computer graphics.

The reason why high-speed, high-precision display is difficult will be explained in detail below.

FIG. 21 is a diagram for explaining a typical conventional three-dimensional image generation and display method. In FIG. 21, the screen S1 is represented by a display reference coordinate system X(w)-Y(w)Z(w) having the center of the screen S1 as its origin. Note that this coordinate system has a function of describing the positional relationship of each of the following coordinate systems, and may be provided at a different location from the screen S1. Viewpoint coordinate system X (e) −
Y (e)Z (e) is the display reference coordinate system X (w)
-Y(w)-Z(w) The position and rotation angle of the coordinate axes are known. It is assumed that the display target object obj has a three-dimensional structure, and the structure point Pi(obj) of this target is represented by the target coordinate system X (obD −Y (obj)−Z (obj)).

In such a situation, when the structural point P1 is viewed from the viewpoint ○e, consider where this point Pi is displayed on the screen S1. Display reference coordinate system X (w)
-Y (w) -Z (w), target coordinate system
(ob D -Y (ob D -Z(obj) is known, so the target coordinate system X(obD -Y (o
bj)-Z (structural point Pi(pbj
) is the display reference coordinate system X(w)'-Y (w)-Z (w
) and is expressed as P i (w) −Ml·Pi(obj).

Here, Ml is a transformation matrix for translation and rotation. Also, the viewpoint coordinate system X (e) −Y(e) −Z
(e) also displays reference coordinate system X (w) -Y(W)-Z
(W), so if this transformation matrix is M2, then the display reference coordinate system
(W) -Y (w) -Z Structural point P seen from (w)
i(w) is the viewpoint coordinate system X(e)Y(e)-Z(e
), P i (e)=M2-' ・P i (w)=M
It can be expressed as 1 aM21·Pi(obj). In this way, the display reference coordinate system
(w) -Y (w) -Z (w) and target coordinate system X
(ob j) -Y (ob j) -Z (ob
D Head display reference coordinate system x (w) -Y (w)
-Z (w) and viewpoint coordinate system X (e) -Y (e)
-Z (e), the structure point Pi(obj) indicated in the object coordinate system can be freely expressed in the viewpoint coordinate system.

Here, in order to find the point on the screen of the structure point Pi(e) indicated by the viewpoint coordinate system, consider X (np) - Y (np) - Z (np), which is called a regular perspective coordinate system. This coordinate system has its origin 0 (np) on the Z (e) axis of the viewpoint coordinate system, and areas A, B, and C, called visual field pyramids, are notched in the viewpoint coordinate system.

D, A (-), B (oo), C (-),
D (Beautiful) - Rectangular parallelepipeds A, B, indicated by dashed and dotted lines
C, D, A' (oO), B' (-), C'
(oO), there is a correspondence that matches D' (-). It is assumed that the Z (np) axis direction of the rectangular parallelepiped is normalized to 1. That is, the point Z (np) = 0 is Z (e)
Corresponding to axis h, the point Z (np) = 1 is Z (e)
= corresponds to beauty.

Under these conditions, the rod-shaped object obj shown in FIG. 21 is -
As shown by the dotted chain line, the cross section becomes smaller on the larger Z(np) axis. The transformation between this viewpoint coordinate system and the normal perspective coordinate system can be expressed by a 4×4 matrix M3. Therefore, P 1 (np) can be expressed as Pi (np)=M:lPj (e). Here, Pi(np) is A, B, C
, ξ1 (S) shown by the dotted line can be obtained by parallel projection onto the D plane.

In FIG. 21, for ease of explanation, the regular perspective coordinate system
It is assumed that (np)-Y (np)-Z (np) and the display reference coordinate system X (W) -Y (w) -Z (w) overlap in origin and axis direction. therefore,
The Z(e) axis of the viewpoint coordinate system is the Z(w) axis of the display reference coordinate system.
) in the same direction as the axis.

In this case, the rectangular parallelepipeds A, B, C, and D correspond to the size of the screen. Therefore, ξ1 (S) is the projected image on the screen seen from viewpoint 0 (e). In this way, three matrix operations and parallel projection are required from the structure point Pi(obj) to ξi(s).

Next, a case where the viewpoint deviates from the Z (e) axis of the display reference coordinate system will be described. In Figure 21, 0' (
e) is the moved viewpoint position. A new viewpoint coordinate system X' (e) Y' (e) - with this moved point as the origin and the O' (e) - 〇 (w) line as the Z (e) axis
Consider Z' (e). Display reference coordinate system X (w)
-Y (w) -Z (w) If the position of the new viewpoint coordinate system is known, then X (e) -Y(e)-Z (
e) to X'(e)-Y'(e)-Z' (
Conversion to e) is possible via matrix M4.

Therefore, the structure point Pi seen from the new viewpoint coordinate system is Pi
(e)'=M4・Pi (e)=M4・Ml・M
It is expressed as 2-1·Pi(obj).

FIG. 22 is a view seen from above the Y(w) axis in FIG. 21.

Multiplying Pi(e)' by the normal perspective projection transformation matrix M3 described above yields Pi' (np), and parallel projection of this yields ξi" (s). Here,
New viewpoint coordinate system X' (e) − seen from the display reference coordinate system
Since the position of Y' (e) - Z' (e) is known,
It is easy to find the point where the line connecting ξi'' (S) and O' (e) intersects the screen.This point ξi'
In other words, (s) is an image projected onto the screen as seen from viewpoint O'. In this way, four matrix operations, parallel projection, etc. are required from the structural point Pi (obj) to ξi' (s).

As mentioned above, the display reference coordinate system X(w)-Y(w)-Z
In (w), the object coordinate system X(obj) −Y (
ob j) −Z (ob j) (7) The display object whose position is described, and each structural point Pi(
The display object for which the coordinates of (obj) are described can be perspectively projected onto the screen from any viewpoint. However, as described above, it is necessary to perform many matrix operations on all constituent points to be displayed, resulting in an enormous amount of calculation time. Matrix calculations can be made considerably faster by using hardware, but real-time processing becomes difficult when the number of constituent points exceeds several thousand. Further, depending on the position of the display object, a lot of unnecessary processing is performed, which limits the processing speed. This is explained below.

Figure 23 shows the coordinate system shown in Figure 21 viewed on the Y (e) axis and in the Z (e) axis direction, and Figure 24 shows the display target shown in Figure 23. be.

In FIG. 23, the screen S1 consists of a large number of pixels gi. The display object has a three-dimensional structure as shown in FIG. 24, and is composed of a structural point Pi and a structural surface Li created by the point. The case where this display target is nearby in the display reference coordinate system is shown in Fig. 23 (A), and the case where it is far away is shown in Fig. 23 (B).
, obj2. By the method described in FIGS. 21 and 22, the display object h- is displayed on the screen S1.

bjl and h-obj2 are projected. A signal can be sent to the corresponding pixel on the screen coordinates of each component point and displayed. Since obj2 is located far away, the distance between each component point is smaller than that of h-objl.

If this interval is sufficiently smaller than the pixel interval, there is no need to calculate all the constituent points within that pixel. In addition, the human eye has visual acuity characteristics, and it is meaningless to calculate each component point on the screen 81 at an interval that is less than the resolution of the eye.

Existing algorithms calculate all constituent points of the target regardless of the position of the display target and without considering the user's visual acuity characteristics, resulting in slow display speed and the ability to display complex targets in real time. This becomes difficult.

[Problems to be Solved by the Invention] The above-mentioned problems are particularly problematic when realizing stereoscopic computer graphics. Stereoscopic computer graphics is a system that uses the user's right and left eyes as viewpoints to separately calculate the perspective images on the screen when looking at a 3D display object, and uses time sharing to calculate the perspective images for the user's left and right eyes. This is a method of creating a display that gives the user a three-dimensional impression by projecting it onto a screen that corresponds to the eyes.

In stereoscopic display, the depth interval is clearly recognized by the user, so it becomes necessary to model large objects (such as buildings) as thick (and small objects (such as insects) as small). In non-stereoscopic computer graphics, it is possible to make objects appear larger by placing them close to the viewpoint without having to model them large.However, in stereoscopic viewing, the depth interval or parallax information is given, so it is possible to make small objects appear larger. Even if it is in the foreground, it will not be recognized as a large object, and it will appear as though a small object is in the foreground.

When an object is modeled to its original size, a large object becomes
If we assume that the object will come close to the viewpoint and spread out on the screen, it will be necessary to model it down to the smallest detail. On the other hand, when the object is far away from the viewpoint, the same amount of calculation is required as when the object is nearby, even though the entire object is displayed on a part of the screen, as shown in Figure 23 (B). There is an absurdity in requiring it.

The need for high-speed computer graphics has been described above, but even if computer graphics is real-time, there are disadvantages depending on the application. In immersive communication, a virtual space is artificially created using computer graphics, and users can interact with the virtual space in various ways as if they were actually there. What is required here is not just high speed, but also the ability to change the screen image in synchronization with the observer's movements.

However, in computer graphics, there is always time left to generate images, so even if it is possible to generate and display several tens of images per second, there is a delay in generating each image relative to the movement of the observer. If this happens, the user may feel uncomfortable and cannot feel a sense of unity with the virtual space.

Furthermore, as will be described later, human visual acuity is high near the point of gaze, but significantly deteriorates around it.

One of the reasons for this is the inherent property of the eye that the retina cannot be focused in areas that are off the optical axis of the eyeball. Therefore, when artificially generating and displaying an image, it is necessary to generate a screen image so that the image reflected on the retina is close to the image viewed in real space. In other words, from the perspective of realistic display, it is not necessarily appropriate to display the entire screen in high definition using a uniform algorithm.

Therefore, the main purpose of this invention is to provide real-time and highly accurate computer graphics display necessary for intelligent encoded communication and stereoscopic computer graphics.
An object of the present invention is to provide a line-of-sight tracking high-speed image generation and display method that can be realized by significantly reducing the amount of calculation.

Another object of the present invention is to reduce the time delay between image generation and display in computer graphics that realizes linked parents, and to realize a high-speed eye-tracking type image that can realize a natural display without any discomfort. It is an object of the present invention to provide a generation and display method.

Still another object of the present invention is to generate an image that gives the same feeling as when viewing real space by generating a screen image so that the image reflected on the retina is similar to the image when viewing real space. An object of the present invention is to provide an eye-tracking high-speed image generation and display method that can be realized.

[Means for Solving the Problem] The invention according to the first claim is a line-of-sight-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the constituent plane of the object is hierarchically described in the target coordinate system and with the size of the area as at least one element,
When projecting the constituent surface of the target when viewed from an arbitrary viewpoint onto a two-dimensional screen, the user's line of sight or the point of gaze where the line of sight intersects with the two-dimensional screen is detected, and the line of sight or the point of gaze of the two-dimensional screen and the line of sight are detected. The point at which the line segment connecting the first intersects the constituent surface of the object in the display coordinate system is the spatial point of interest, and the distance from the spatial point of interest or a feature point near that point that is necessary to express the display object. The hierarchical level is selected for each space with at least one parameter.

The invention according to claim 5 is a gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the constituent plane of the object is in the object coordinate system and the size of the area is is described hierarchically as at least one element,
When projecting the target constituent surface onto a two-dimensional screen when viewed from an arbitrary viewpoint, detect the gaze point where the user's line of sight or line of sight intersects with the two-dimensional screen, and detect the line segment connecting the line of sight or the gaze point with the viewpoint. The hierarchy level is selected using the distance to the target as at least one parameter.

The invention according to claim 9 is a gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the constituent plane of the object is in the object coordinate system and the size of the area is at least It is described hierarchically as one element, and when projecting the object constituent surface onto a two-dimensional screen when viewed from an arbitrary viewpoint, the display reference coordinate system at the previous image display time from the movement of the line of sight of both parties. The system is configured to estimate a spatial gaze point that is a gaze point or its vicinity, select a hierarchy degree using the distance from the spatial gaze point or its vicinity as at least one parameter, and generate a projection image in advance.

More preferably, in addition to the invention according to the first or third claim, the invention according to the second or sixth claim provides a method for detecting an arbitrary point of the object expressed in the object coordinate system from the origin or viewpoint of the display reference coordinate system. The hierarchy level is selected using the distance to the second parameter as a second parameter.

Furthermore, more preferably, in the invention according to the third or seventh aspect, the vicinity of the optical system principal point of the left and right eyes is set as a viewpoint, and a projected image viewed from that point is generated.

4th. The invention according to claim 8 or 10 is characterized by the following features: 1. ], 5. In addition to the ninth invention, the point where the line of sight and the two-dimensional screen intersect is the point of gaze, and the position of the point of gaze is the starting point,
It is configured to apply a high-frequency cutoff filter to the periphery and display the image.

[Operations] The invention according to claim 1 utilizes the characteristic that the visual acuity of the human eye is high at the point of gazing and rapidly decreases in the periphery, so that the display target can be adjusted to the size of the area, that is, the scale. By modeling hierarchically as one parameter, displaying the gaze part of the display target up to a detailed modeling hierarchy, and displaying the display target part corresponding to the peripheral vision that is outside the gaze area in a coarse modeling hierarchy, the peripheral vision part The aim is to reduce the amount of calculation and speed up the process.

The invention according to claim 5 selects the degree of hierarchy by using the distance from the line segment connecting the line of sight or the gaze point of the two-dimensional image and the viewpoint as at least one parameter to the object.
To generate stereoscopic computer graphics images at high speed and high quality.

The invention according to claim 9 detects the movement of a person's line of sight,
To solve the problem of display discomfort caused by a delay in image generation time when this detection result is reflected in computer graphics, and to obtain a more natural image.

The inventions according to the second and sixth claims display fine modeling hierarchies when the object is close to the origin or viewpoint of the display reference coordinate system, and display coarse modeling hierarchies when the object is far away. , which combines the second feature of reducing the amount of calculation and speeding up calculations for distant objects with the inventions according to the first and fifth claims, and which is far from the display reference coordinate origin or viewpoint and is far from the peripheral vision. We will significantly reduce the amount of calculations and speed up the parts corresponding to .

The invention according to the third and seventh claims generates a projected image viewed from a viewpoint near the principal point of the optical system of the observer's eye.

4th. 8th. The tenth aspect of the invention is to display each part of the display target corresponding to the periphery in a coarse modeling hierarchy by applying a high-frequency cutoff filter to the periphery of the gazing point and displaying the gazing point as a starting point. compensate for the unnaturalness caused by When displayed using a coarse modeling hierarchy, there will be many discontinuities between the constituent surfaces to be displayed in the screen projection image, and high frequency components will occur in these parts. By applying this product, the components that cause this discomfort can be removed.

[Embodiments of the invention] Fig. 1 is a diagram showing the principle of this invention, and Fig. 2 is a diagram showing the principle of the invention.
FIG. 3 is a diagram illustrating hierarchical modeling of the displayed objects shown in the figure.

The principle of this invention will be explained with reference to FIG.

The screen S1 of the display device has a display reference coordinate system X (w) −Y (w) − with its origin 0 (w) as a reference.
It is expressed as Z (w). Note that the display reference coordinate system X (
w) -Y (w) -Z (w) is a coordinate system that serves as a reference for other coordinate systems shown below, and may be provided at a different location than the screen S1 in FIG. Display target o
bjl and obj2 have different structure data compared to the conventional example shown in FIG.

That is, as shown in FIG. 2, the display target is a plurality of structural points Pi and a structural surface (
In FIG. 2, it is defined as a surface surrounded by four structural points). In addition, each structure point is in the target coordinate system X (.

b j) - Y (obD -Z (obj). The hierarchy setting in Figure 2 is based on the distance between each structure point or the size of the structure surface, or the number of structure points each time the distance exceeds a predetermined size. As shown in (a) to (d),
There are a total of four levels of hierarchy. A data structure represented by such structural points and line segments connecting the points will be referred to as a wire frame structure.

Vectors VOI and VO2 are each in the display reference coordinate system X
(w) -Y (w) -Z (w) origin 0 (w)
From the display target objl, the target coordinate system origin ob of obj2
jl, a vector to obj2. In Figure 1,
User 2, who is an observer of screen S1, has both eyes.
, eye2 to view display targets objl and obj2. The line of sight 10 intersects the screen S1 at a point ep. This point ep is called the screen gaze point. X (e) −Y (
e) −Z (e).

X'(e)-Y'(e)-Z'(e) is the viewpoint coordinate system, and eol is the point where the line segment connecting O'(e) and the screen gaze point ep intersects with the display target.

The structure point Pi(obj) is a feature point in the rough hierarchical display target model shown in FIG.

Here, the line of sight 10 can determine the position in the display target coordinate system by capturing the feature points of the eyeballs with two cameras and performing stereo image measurement from the position information of the camera imaging plane. As the feature point, a corneal reflection image created when a light source set at the center of the pupil or a known point in the display target coordinate system is reflected by the cornea may be used. For details, regarding feature point extraction, refer to "Non-contact line of sight detection device" (Japanese Patent Application No. 63-289761) by the inventors of the present application.

It is possible to measure the position of the pupil by using an "image photographing device" (Japanese Patent Application No. 1-181387) and a "line-of-sight detection method" (Japanese Patent Application No. 1-296900). In addition, the eol point in FIG. 1 or the feature point near it (for example, Pi(obj)) is
It can be obtained by inverse transformation of the method of perspective projection transformation of point i onto screen $1.

FIGS. 3 and 4 are diagrams for explaining a method for finding a point on the model corresponding to the screen gaze point ep(s).

In FIGS. 3 and 4, Z(
w) The case where there is a viewpoint o on the axis (e) is shown. The display target is an object as shown in Fig. 2, and to find the position of the target corresponding to the screen gaze point ep(s),
A coarse modeling hierarchy is used to speed up the process. 2nd E
The object shown in (d) is the object obj in FIG. The normal perspective transformation point of each component point Pi(w) is pi(np)
shall be. This conversion can be done quickly if the model is coarse. Note that the screen S of the normal perspective transformation point pi (np)
Let the parallel projection point to l (ABCD) be ξ1(S).

Here, as shown in FIG. 4, the screen gaze point ep(s
), starting from this point, a search line 20 parallel to the Z(np) axis can be searched to find the point where the search line 20 intersects the orthogonally projected object. If the constituent points 301 to 303 of the normally projected object 30 are given by the normal perspective coordinate system It's easy to ask.

Furthermore, even if the constituent plane is not given, as shown in FIG. You can also ask for Here, the points eh (np), pi
Display (np) reference coordinate system X (w) −Y (w)
−Z (w), the screen gaze point ep
It is possible to obtain the spatial gaze point eo(w) corresponding to (s) or Pi(w) located in the vicinity thereof.

Regarding Pi(w), the screen gaze point ep
This is a corresponding point slightly distant from (s). The reason why a model with a detailed hierarchy is not used is that the search in the example shown in Figure 3 assumes that normal perspective coordinate transformation data for each structure point to be displayed already exists, so using a detailed modeling hierarchy is speedy. This is because there is a contradiction in terms of. However, in one embodiment of the present invention, a case where several to several dozen images are displayed per second is considered, and at such a speed, it is considered that there is little change between two temporally consecutive models.

Furthermore, modeling information for generating the image immediately before being displayed next and this regular perspective transformation data already exist. Therefore, by searching using this previous data, it is possible to accurately determine the display target position corresponding to the screen gaze point ep (s). In this way, it is possible to find the spatial gaze point eO(W) corresponding to the screen gaze point ep(s) or a position in the vicinity thereof.

Note that the vicinity of the spatial gaze point eo(w) in this invention refers to the component point 2 component plane of the target that is near the line of sight and is determined to be gazed at, or the origin of the target coordinate system. The above search can be performed using simple theoretical calculations and coordinate transformation calculations, and can be implemented in hardware.

FIG. 5 is a diagram for explaining a method for selecting a model hierarchy. With reference to Figure 5, the origin of the display reference coordinate system is 0.
(W) or the absolute value of the vector ■o1 or VO2 from the viewpoint 0'(e) to the origin of the target coordinate system, that is, the distance is a predetermined distance (Thl or Th
2) Select the model hierarchy depending on whether it is within the range.

For object 0bj1, it is inside the distance Thl and the hierarchy shown in FIG. 2(b) is selected, and for obj2, it is inside the distance Th2 and the hierarchy shown in FIG. 2(c) is selected.

Next, consider the spatial gaze point positions (two examples are shown in Figure 5), and set each to eo4. e.

2, and a predetermined radius STh l5Th around this point.
Assume a ball of 2. In the example shown in FIG. 5, consider the case where eol and Pi (○bj) shown in FIG. 1 overlap. For the inside of the radius 5Th1, a model with one level of finer hierarchy is used. Although not shown in FIG. 5, the area outside the radius 5Th2 is changed to a hierarchical model that is one step coarser. That is, for the object objl, the inside of the radius 5Thl is changed to the model shown in FIG. 2(a), and for the object obj2, the inside of the radius 5Th1 is changed to the model shown in FIG. 2(b). As described above, the near from O(w) is relatively fine, the far is coarse, and the spatial gaze point e
An object modeled relatively finely in the vicinity of o(w) and relatively coarsely modeled in the vicinity is prepared. A display image is obtained by projecting each constituent point of the model onto the screen S1 using a conventional method.

FIG. 6 is a diagram for explaining a method for selecting a model hierarchy in another example of the present invention. Referring to FIG. 6, eol and eo2 are the positions of the spatial gaze points, Thi is the radius centered on eol, and the distance d+(absolute value of V+) from O(w) to the origin of the target coordinate system ) is a function of The function increases the radius Thl when di is small;
When the distance d1 is large, it acts to reduce the radius Thl. In the example shown in FIG. 6, the position e of the spatial gaze point is
T h +-1 around ol is larger than the radius T h t-2 of the spatial gaze point e02. Within radius Thl, FIG. 2(a) is selected, between radius Thl and Th2, FIG. 2(b) is selected, between radius Th2 and Th3, FIG. 2(C) is selected, Between the radius Th3 and Th4, the one shown in FIG. 2(d) is selected. As described above, a model with a degree of definition that differs in stages around the spatial gaze point eo(w) can be obtained.

In addition, in the examples shown in FIGS. 1, 5, and 6, the display reference coordinate system X (w) −Y (w) −Z(w
), the distance from the origin 0(w) to any point in the target coordinate system was taken as a parameter, but the viewpoint origin (0(e)
or O' (e), etc.) may be used as the starting point. In this case, if the display object does not move but the viewpoint moves, there is a need to change the hierarchy each time, but considering that the visual acuity characteristics of the person who recognizes the details of the object greatly depends on the distance from the viewpoint. , it can be said that a more rational hierarchy selection can be made.

Furthermore, as will be described later, in binocular stereoscopic computer graphics, the display object may be displayed in front of the screen S1, or in such a case, as shown in FIG. If the distance from w) is used as a parameter, it would be unreasonable that a rougher layer would be selected than an object immediately behind (in the back) of screen S1, even though the object is near viewpoint 0 (e). become. Therefore, a method using the distance from the viewpoint to the origin of the target coordinate system as a parameter is also effective, depending on the complexity of calculation.

FIG. 7 is a diagram showing quadratic and cubic Bezier curved surfaces. The example shown in FIG. 7(a) is quadratic and cubic (n=2.
3), which is expressed by the following equation.

R(t:3)=(1-t+tE)' PaBez
When 1=0, the ier curve starts from the control point of Po,
When t=1, the end control point (when n=2, P2.n
=3, P3) is reached. The curve smoothly connects the area near the control point. Increasing the number of control points corresponds to increasing the degree, which increases the expressiveness of the curve.

A Bizier surface is expressed as a product of Bezier curves as shown in the following equation.

S (u: m, v: n) = (1-u+uE)Q
(IV+vF)” Po.

(0≦U≦1) (0≦V≦1) E and F are shift operators, E P , = P t−
1,,FP,=P,,,,,.

Figures 7(c) and (d) show the quadratic and cubic Bezie
It shows an r-curved surface. For 3rd order, 16 from POO to P33
By preparing these control points, a free-form surface that smoothly connects them can be generated.

In this way, by adaptively determining a partial region and selecting enough control points to describe this region, a surface with a fine and smooth surface can be obtained. On the other hand, if the partial area is made large and the number of control points is decreased, the surface will be rough. An example of hierarchizing using the number of control points in a partial region as a parameter has been shown above.

Furthermore, in the Bezier surface equation described above, the structure points are determined by the control points and how the values of u and v are taken.

Therefore, a predetermined control point may be determined and the interval between u and v may be used as a hierarchy parameter. That is, in FIG. 7(d), even if the number of control points is constant at 16, u,
Depending on how the interval Δ of v is determined, the constituent surfaces can be coarse or dense. If the interval Δ is made smaller, the constituent planes will be further divided than in FIG. 7(d), and many small constituent planes will be generated. In this way, it is also possible to hierarchize the size of the area using Δ as a parameter.

Next, as for the display method on the screen, as shown in Figure 1, in addition to the simple method of displaying points and lines obtained by perspectively projecting the wire frame, the method of displaying color data on the structural surface is also possible. and colorize it when displayed. ■For each structural surface, a pattern called a texture map is prepared as data, and this pattern can be rotated, moved, or moved depending on the direction and position of the structural surface.
Reduced.

Perform processing such as enlargement and display on the screen.

■Assuming a light source at a predetermined location in the display reference coordinate system,
It is possible to calculate and display the shadow of each constituent surface by the light as seen from the viewpoint for each constituent surface.

As an example, a case will be described in which a plaster cast of a person with equidistant fire is modeled using a Bezier curved surface. Approximately 5,000 control points were required to model this plaster image in such a way that every detail could be seen. That is, as shown in FIG. 7(d), these 5000 control points are divided into 4 x 4 control point groups, and the partial area determined by these 4 x 4 control points is divided into u, v. By selecting the value of , it is subdivided into 10 parts. In this way, a structured surface with 500,000 structured points is formed. By creating a display surface that is divided to this extent, an image that can be recognized in fine detail can be obtained.

On the other hand, as long as it is possible to determine whether it is a person or not, the number of control points may be approximately several tens to 100, and furthermore, U.

■ can also be divided into several parts. In other words, it can be expressed using about 100 structural points. In this way, even for objects as small as people, the amount of data varies by a factor of 100 to several thousand times depending on the display environment and necessity. In other words, the layering can be selected in various ways regarding the size of the area.

FIG. 8 is a diagram showing still another example of the present invention, in which the viewpoint is shown at the optical principal point of the observer's eyeball. The optical principal point can be approximated by the center position of the pupil or iris, and the position and direction of the line of sight are known as described above in the display reference coordinate system.

In FIG. 8, display targets objll, obj12, o
bj13 and obj14 are modeling objects of the same size with the hierarchy shown in Figure 2, and ○ (obj12) is obj1.
2, and el is the screen gaze point ep (s)
is a line segment (i.e. line of sight) that connects and viewpoint ○(e), and d
3. d4. d5゜d6 are the distances from the line segment eL to the origin of the target coordinate system of each target object objll to obj14 (for example, 0 (obj12)), and V12 is the viewpoint 0
It is a vector from (e) to 0 (obj12), and 2
21.222 are each 0 (objll) on the line segment el
, O(obj13).

Since both the target coordinate system and line of sight of each display target are expressed in the display reference coordinate system, it is possible to obtain each of the above-mentioned distances. In this way, in FIG. 8, the display target objl
l and obj12 are relatively close to the viewpoint 0 (e), and display targets obj13 and obj14 are far away. In addition, display targets obj11 and obj13 are located near the line segment eL, and display targets obj12 and obj14 are located away from the line segment et.An example of a model hierarchy selection method will be described.

The absolute value of the vector V12 is defined as a distance d12, which is one parameter. Further, the distance d5 from the line segment et to the origin 0 (obj12) of the target obj12 is used as a second parameter, and the hierarchy of the model is selected according to the value of the function f (d12.d5).

The smaller the value of distance d12 or d5, the smaller f becomes. Therefore, for an object objll whose target coordinate system is near the viewpoint and close to the line segment et, the hierarchy shown in FIG.
In bj14, the hierarchy shown in FIG. 2(d) is selected. Also, the object obj12. which is close in distance from the viewpoint or distance from the line segment el and the other is far. 2nd in obj13
The hierarchy shown in Figure (c) is selected.

In addition, in FIG. 8, the distance from the viewpoint 0 (e) to 221 or 222 is taken as one parameter, and the distance d5
You may also create a function called f with f as the second parameter.

In FIG. 8, the distance d3 from the line segment eL to the origin of the target coordinate system. d4. d5. d6 is used as a parameter, but this parameter does not need to be the origin. For example, as shown in FIG. 9, if it is given at the position of each constituent surface or the center position of that surface, this parameter Distance can also be used as a parameter.

In FIG. 9, line segment 251 is near line segment el, so
The corresponding surface 300 is modeled in the hierarchy shown in FIG. 2(b), and since the line segments 252 and 253 are somewhat distant from the line segment eL, the model in the hierarchy shown in FIG. 2(C) is selected. .

Furthermore, the display target may be composed of a plurality of structures, and may have a main object coordinate system corresponding to the main body and a sub-object coordinate system corresponding to a part of the main body or an appendage. In this case, it goes without saying that the distance between the origin of the sub-object coordinate system and the line segment eL can be used as a parameter.

FIG. 10 is a diagram showing the relationship between the angle of deviation from the point of gaze and the relative visual acuity, measured by measuring visual acuity around a point when the human eye gazes at a flat point. In FIG. 10, the horizontal axis is the deviation angle from the point of fixation, and the vertical axis is the relative visual acuity normalized by the visual acuity of the point of fixation.

The two curves shown in FIG. 10 show that visual acuity characteristics change depending on the indicator presentation time.

A moving image displayed at a speed of about 30 frames per second is considered to fall within the range of these two curves. It can be seen that the longer the image is presented, the better the visual acuity characteristics are, but in both cases the visual acuity is significantly reduced in the periphery. The reason for this is that peripheral visual objects that are away from the direction of the line of sight are not optically imaged on the retina, resulting in blurring, and that the resolution is low in parts of the retina that are away from the fovea of the retina in the direction of the line of sight. It is.

Furthermore, when capturing an object that changes in depth, the camera changes the convergence and adjusts the lens to focus.

In other words, it is a daily experience that even if the line of sight is pointing in the direction of the object, if it is not in focus, blur will still occur. From the above, it can be said that the part that is clearly visible in real space is a very small part where line of sight and accommodation are controlled. Other parts are greatly blurred.

Based on this situation, when generating the space that a person is looking at using computer graphics, we need to: ■ detect the spatial gaze point relative to the screen gaze point, and use the distance from that point as one parameter; ■ determine the target from the line of sight. It is reasonable to use the distance to an arbitrary point (for example, the origin of the target coordinate system) as one parameter and to weight the fineness of the surface of the model from the viewpoint of reducing the amount of calculation. Furthermore, displaying in detail a portion that is originally out of focus and blurred may create a sense of discomfort because it differs from reality.

Presenting a blurred image may be closer to reality.

In this way, the method of the present invention is effective not only in speeding up processing but also in realizing a sense of realism.

FIG. 11 is a diagram showing still another example of the present invention. Referring to FIG. 11, if the relationship between the target coordinate system and the display reference coordinate system of the display target obj is known, each constituent point and constituent surface data of the display target obj is set in the display reference coordinate system X (
w) −Y (w) −Z(W). The two viewpoints 0(e)IL and O(e)r correspond to the viewpoints of the left and right eye optical systems of the observer, respectively. The position of the viewpoint and the direction of the line of sight can be determined in the display reference coordinate system by the aforementioned stereo image measurement method or the like. Once the lines of sight of the left and right eyes are known, the point where the two lines of sight intersect is the target position that the observer is paying attention to. Although this point may include errors depending on the measurement accuracy of the line of sight, the method for finding objects near the line segment connecting the viewpoint and the point of fixation shown in Fig. 8 (equal to the line of sight in Fig. 11) can be applied to both the left and right eyes. If you follow the line of sight, it is easier to identify the object you are gazing at than in the case of FIG.

Let this spatial gaze point (or its vicinity) be eo. Assuming a sphere with radius Th around the spatial gaze point eo, the inside of the sphere is modeled in the hierarchy shown in Figure 2(b), and the outside of the sphere is modeled in the hierarchy shown in Figure 2(d). Ru. Left and right images ho
b j (L), h-.

bj(R) are each projected onto a screen.

In this example, two viewpoints 0(e)L, O(e)R
and the line of sight constantly changes depending on the observer's movement.
This change is detected in real time, and the left and right images h-ob
j (L) and h-ob j (R) are similarly played back and displayed in real time. This allows the observer to move his head and view the display object obj from various directions, such as left and right, and up and down. Also, at this time, the two images are h-obj (L) and h-obj (R) l
↓Since it has parallax information, the observer can compare objects obj three-dimensionally.

Furthermore, it is also possible to create an image assuming that the object obj has come in front of the screen S1.

This method does not tolerate display delays. In other words, since the viewpoint is the observer's eyes, it will feel strange unless the image is generated and displayed in synchronization with the observer's movement. The allowable delay is considered to be about 1 to 2/30 seconds. therefore,
The method of this invention, which is advantageous in high speed, works effectively.

Note that in this embodiment, a binocular stereoscopic display device can be used to display the generated images. A specific example of this device is a "stereoscopic display device" (Japanese Patent Application No. 2-18051) by the inventors of the present application. The configuration of this device is that a semicircular lens sheet called a lenticular lens is provided on the display surface, and pixels for the right eye and pixels for the left eye are arranged on both sides of the focal point on the back of each lens. be. In other words, 1 pixel or 1 set of 2 pixels
It is placed facing two kamaboko lenses. The light emitted from the two pixels is spatially separated by the lens and separately enters the observer's left and right eyes. In this way, stereoscopic viewing becomes possible without having to wear any special equipment.

FIG. 12 is a diagram showing still another example of this invention,
This is an example of further reducing the delay in relation to the observer's movements. Referring to FIG. 12, display target obj1. obj2 is the model object shown in Figure 2, and the display reference coordinate system
It is expressed as (w) -Y (w) -Z(W). The viewpoint and the viewpoint 0 (e) of the eyeball are approximated by the center position of the pupil or iris. The lines of sight el (1) and el (0) change as the eyeball rotates. The pupil position and line of sight are represented by a display reference coordinate system X(W)-Y(W)-Z(W). The spatial gaze point eO is on the extension of the line of sight. The distance from the line of sight to the display object ob j ] is d (-1),
It is expressed as d (0).

Fig. 12 shows a case where about 10 to 30 images are generated and displayed every second, so (0), (
-1) and (+1) points are spaced approximately 100 to 33 m5ec apart. The hierarchy in the current state (0) and the previous state (-1) is selected as follows. The distance from the line of sight to the object described in FIG. 8 is used as one of the parameters. At (-1), the display target obj2 is away from the viewpoint but is on the line of sight, so the hierarchy shown in FIG. 2(c) is selected. The display target objl is near the viewpoint, but since it is a little far from the viewpoint, it is similarly displayed in Figure 2 (
The hierarchy shown in C) is selected. In this way, h-
ob jl (-1), h-ob j2 (-1)
is generated and displayed. Also, in (0), the display target o
Although bj2 does not change, the display target objl moves further away from the line of sight, so the hierarchy shown in FIG. 2(d) is selected. In this way, h-objl (0), h-ob
j2 (0) is generated and displayed.

Next, a display that predicts the movement of the line of sight will be described. Eyeball movements can be broadly divided into smooth follow-up movements when tracking a moving target, and jumping movements seen when searching for an interesting object. In voluntary movement, visual acuity decreases less than when looking at a stationary indicator. On the other hand, in a jumping motion, visual acuity decreases significantly immediately after the point of gaze moves.

Figure 12 shows a state where the head is moving while gazing at the spatial gaze point, but the movement of the line of sight in such a case is smooth, and the line of sight display reference coordinate system at the time when the next image is displayed from the trajectory X (w) −Y (w) Z (
w) is easy to obtain.

That is, O(e)(-i) to O(e)(0).

e L (-i) ~ e l (0) to O(e) (+
1) It is possible to obtain el(+1) by extrapolation. Therefore, perspective projection images h-objl (+1) and h-obj2 (+1) of the object viewed from O(e)(+1) are generated in advance and displayed at time (+1).

In addition, in the case of this example, the spatial gaze point at times (-1) and (0) is eo, and it has not changed, so it is assumed for the time being that the gaze is at the same point at (+1). h-objl (+1), h-obj2 (
It is also possible to generate +1). At this time, since it is possible to infer that the observer is interested in obj2, the modem hierarchy can be raised one level from the case (0) to the hierarchy shown in FIG. 2(b). In this way, it is also possible to identify the target of attention and adaptively display the target in a detailed hierarchy. In addition, various knowledge such as the positional relationship of display objects, the meaning of the objects, and the transition of the observer's gaze object can be used in conjunction with the identification of the gaze object.

The method according to the present invention has few problems when the line of sight is moving smoothly because it is difficult to predict, but when a jumping motion is mixed in, the prediction becomes incorrect, which makes it seem unreasonable. However, there is no problem for the following reasons. That is, as mentioned above, visual acuity is closely related to the movement of the line of sight, and immediately after a jumping movement, visual acuity deteriorates and requires more than 200 m5eC to recover. Therefore, even if the predicted viewpoint is wrong, the observer's visual acuity will be greatly reduced, so the observer will not feel any discomfort in the displayed image.

It is only necessary to detect the correct pupil position and line of sight and generate and display a perspective projection image from that point until vision is restored.

By the way, in the above embodiment, the screen gaze point ep(
A model with a fine layer is selected in the vicinity of s) (central vision section), and a model with a coarse layer is selected and displayed in the peripheral vision section. At this time, since the structural planes become larger in the peripheral vision area, boundaries may be created between the structural planes, and continuity with the central vision area may be disrupted. Since such a discontinuous surface has a high spatial frequency, it is easily perceived in peripheral vision and may cause a sense of discomfort. Therefore, in order to eliminate this sense of discomfort, the screen gaze point ep (
One possible method is to blur the image by covering the periphery of s) with a high-threshold cutoff filter.

FIG. 13 is a diagram showing a high-threshold cutoff filter that is placed around the screen's gaze point. Referring to Figure 13,
6. No filter within 14 degrees around the screen gaze point ep(s). The original image of 6 c p d is displayed as is), and in the area of 14 to 20 degrees, 4.5 cp
d, 1°0 cpd above 20° (note that c
pd is the number of spatial frequencies that can be displayed within 1° viewing angle). This filter can be configured with hardware, and can be applied to the displayed image in real time by following the movement of the screen gaze point ep(s). In designing this filter, the issues are how much the resolution difference between the central vision section and the peripheral vision section and the viewing angle of the central vision section should be.

FIG. 14 is a diagram showing the result of applying the filter shown in FIG. 13 to an original image captured by a camera. As is clear from this Fig. 14, it has the viewing angle division shown in Fig. 13,
When a camera captures a filter using the peripheral vision resolution as a parameter, the maximum spatial frequency cpd of the peripheral image becomes 3.
It can be seen that when the value is 0 or more, it cannot be distinguished from the original image. Therefore, in a filter with a central viewing area of 14 to 20 degrees,
It is possible to lower the maximum surrounding spatial frequency to about 3 cpd. By using such a filter,
The discomfort caused by discontinuity in the structure is reduced.

FIG. 15 is a schematic block diagram of an embodiment of the present invention. In the embodiment shown in FIG. 15, the viewpoint is the eyeball, which is moving, and for the sake of simplicity, it is assumed that the display object is stationary in the display reference coordinate system. In addition,
The thick solid line portion shown in FIG. 15 is a novel means that has not existed in the past.

Referring to FIG. 15, the viewpoint and line of sight detecting means 10 detects the viewpoint and line of sight of the eyeball in the display reference coordinate system. The target coordinate system setting means 20 specifies the origin position and rotation angle (angle) of the display target in the display reference coordinate system. The distance calculation means 30 calculates the distance from the viewpoint to the origin of the target coordinate system according to the detection output of the viewpoint and line of sight detection means 10, and the origin position and rotation angle specified by the target coordinate system setting means 20. The distance calculation means 40 uses the line of sight data of another eye not shown in FIG. Find the origin of the target coordinate system at , and use that origin as the spatial gaze point, or calculate the distance of a perpendicular line drawn from the origin of the target coordinate system to the line of sight.

In the three-dimensional hierarchical database 80, the size of the area of the constituent plane of the display target object is hierarchically described as one parameter. The constituent point/constituent plane generating means 50 generates a three-dimensional model of the object at a position specified in the display reference coordinate system. That is, the generating means 50 generates information such as the distance from the viewpoint to the object origin determined by the distance calculation means 30, the distance from the spatial gaze point to the object constituent plane determined by the distance calculation means 40, the distance from the object origin to the line of sight, etc. is a parameter, the value of this parameter is compared with a reference value to determine the hierarchy of the target constituent surface, and the 3D hierarchical database 8
3 in the display reference coordinate system by receiving data from 0.
Generate a dimensional model.

Note that the part surrounded by the dotted line in FIG. 15 constitutes a processing means 101 that determines the hierarchy level using the above-mentioned parameters.

The perspective projection conversion means 60 receives the viewpoint position and the above-mentioned three-dimensional model as input, and perspectively projects the display object onto the two-dimensional screen. The display image generation means 70 creates a pattern such as coloring on the target constituent surface while referring to the constituent plane data obtained by specifying objects and nine layers of constituent planes in the three-dimensional hierarchical database 80. to attach.

The high-frequency cutoff filter means 90 cuts off high-frequency spatial frequencies that overlap the peripheral visual field of the screen gaze point. Display means 100
A lenticular screen or the like is used to display the target object.

FIGS. 16 and 17 are flowcharts for explaining the operation of one embodiment of the present invention.

Next, with reference to FIGS. 15 to 17, a specific operation of an embodiment of the present invention will be described.

Now, using the above-mentioned FIG. 8 as an example, the display reference coordinate system X(w)
-Y (w) -Z (w), object objll~
Consider generating an image on the screen S1 in which obj14 appears to be at a predetermined location. Note that the position of the viewpoint is the pupil position, and the viewpoint is assumed to be moving. In addition, the object is stationary in the display reference coordinate system to simplify the explanation, and the three-dimensional structure data of the object is described in the object coordinate system as shown in Figure 2, and the constituent surfaces are The area size is hierarchized as a parameter.

First, in step SPl (abbreviated as SP in the figure), the target coordinate system setting means 20 determines the position vector V of the origin of the target coordinate system. bl-+ = (x-bl-1+ Y
obl-1g Z. bl-1) and the rotation angle R of the target coordinate system
abl-1= (α.bl-1, β.bI-1,+
bi-1) γrito is set and provided to the processing means 101. In step SP2, the viewpoint position aVoe detected by the viewpoint and line of sight detection means = (Xoe.

Yo e, Xo e) are provided to the processing means 101.

In step SP3, the distance calculation means 30 reads the viewpoint position vector Voe at that time given from the viewpoint and line of sight detection means 10, and calculates the position vector V of the origin of the object coordinate system. 6. −1 and the viewpoint position vector Voe, the distance between two points dislobi−1=l V. bi-
Calculate I V Oe I. In step SP4, the processing means 40 calculates the distance dis2°bl-1 from the line of sight shown at d3 to d6 in FIG. 8 to the origin of the target coordinates.

Next, the generating means 50 generates the parameter disl determined by the distance calculating means 30. , I-3 and the parameter d I S 2 obj-1 determined by the processing means 40
Perform the following calculation using .

dis3ob+-+ = f (disl.b-+ d
is2abl-1) = A' d I S 1a)z-
10B' d l s2 The above equation is the parameter d ls
This is the simplest example of creating 3 ob+-1. In addition, A
, B are constants here. Note that a more complicated formula may be created by taking human visual acuity characteristics into account.

Next, in step SP6 shown in FIG. 17, the processing means 101 sets the parameters dis3°, 1-1 and the reference value Th
3 and the parameter dis3°.

If it is larger than the reference value Th3, that is, if it is far away and away from the line of sight like the object obj14 in FIG. In step SP8, the position vector V of the origin of the target coordinate system is selected. , 1-1, a three-dimensional model is generated at the rotation angle Ra bl-1 of the target coordinate system.

The processing means 101 determines the parameter d in step SP6.
is3゜6. -1 is smaller than the reference value Th3, and furthermore, when it is determined in step SP9 that it is larger than the reference value Th2, that is, the target 01)j12. In the case of obj13, the hierarchical model shown in FIG. 2(c) is selected from the three-dimensional hierarchical database 80 in step 5 PIO. but,
The processing means 101 determines the parameter d in step SP9.
When it is determined that I S 3 obi-1 is smaller than the reference value Th2, in step 5P12, the process shown in FIG.
Select the hierarchical model shown in b), that is, the model of the object obj14 as shown in FIG.

As described above, the processing means 101 reads model data for all objects (1 to i) from the three-dimensional hierarchical database 80. In step 5P14, the perspective projection transformation means 60 converts the perspective projection transformation image of the object.

bl-11+ h abl-! □, h. bl-13v
Generate h abL-14. In step 5P15, the display image generation means 70 reads each constituent surface pattern data stored in the three-dimensional hierarchical database 80, and
Add a pattern to the target configuration surface of the dimensional screen. Furthermore, the high-frequency cutoff filter means 90 cuts off the high-frequency spatial frequencies that overlap the peripheral visual field of the screen gaze point, and the display means 100 displays the target image generated in step 5P16.

FIG. 18 is a flowchart for explaining the operation of another embodiment of the present invention, in which the distance from the viewpoint to the origin of each target coordinate system is the first parameter, and the distance from the line of sight to the origin of each target coordinate system. In this example, the closest object origin is estimated as the spatial gaze point, and the distance from this point is used as the second parameter. Steps SPI to SP4 are the same as in FIG. 16, and steps 5P21 to This differs from FIG. 16 in that the process of 5P23 is performed. Step SP
4, after finding the distance d I Sobl and yu from the line of sight to the origin of the target coordinates shown in d3 to d6 in FIG.
In step 5P21, the distance di s2°5. -1, the smallest obj-min is found, and the object origin is estimated as the spatial gaze point. To explain with reference to FIG. 8, in step 5P22, the origin of 0bjll is temporarily set as a spatial gaze point, and other objects obj12. obj13
．． Distance to the origin of obj14 dis4°1. -1 is calculated. The distance calculation means 40 has its parameter dis1
．． Using bI and dis4°, j-, the following equation is calculated.

dis3. +z-1=f (disl.b+=)
・g(d i s 46bl-+) = C-d
is 1゜bl-1・diS4゜,.

The above equation has parameters dis3°6. This is the simplest example of creating -I. C is set as a constant here.

Hereinafter, steps SP6 to 5P16 shown in FIG.
processing is performed. Note that even in this embodiment, a more complicated formula may be created by taking human visual acuity characteristics into consideration.

FIG. 19 is a schematic block diagram showing another embodiment of the invention. This example is expanded so that not only the origin of the target coordinate system but also the target constituent planes can be used in creating parameters for selecting the hierarchical level of the constituent planes, except for the following points. This is the same as FIG. 15 described above.

That is, the processing means 102 includes an upper layer generation means 21 and distance calculation means 31 and 41. Upper layer generation means 21
When viewed from the display reference coordinate system, the object coordinate system is set, and a rough structure of the object, that is, a configuration plane with a coarse level of hierarchy (higher hierarchy) is generated at the name position. Since the hierarchy level is high, this processing can be performed at high speed. Distance calculation means 31
calculates the distance from the viewpoint to the base point of the schematic constituent surface, for example, the center of the surface. The distance calculation means 41 sets the point where the line of sight intersects the schematic constituent plane as a spatial gaze point, or calculates the base point of the constituent plane in the vicinity of this point, or calculates the distance of a perpendicular line drawn from the base point of the schematic constituent plane to the line of sight. calculate. The component point/component surface generation means 50 generates a three-dimensional model of the object, and in the case of (2), the distance from the spatial gaze point is used as a parameter and compared with the reference value Th, while Data at an appropriate level of hierarchy is imported from the system, and constituent points/planes of the target are generated. In the case of (3), the distance of the perpendicular line dropped to the line of sight is used as a parameter, and while comparing it with the reference value Th, data of an appropriate level of hierarchy is imported from the three-dimensional hierarchical database 80, and the constituent points/planes of interest are generate. In this embodiment, the high-frequency cutoff filter means 90 shown in FIG. 15 is omitted.

FIG. 20 is a diagram showing an example of application of the present invention.

The example shown in FIG. 20 allows a computer graphics image to be observed with a rich sense of realism. The viewpoint corresponds to the center position of the pupil or iris. A viewpoint detection device (not shown) performs a viewpoint detection 1 on this viewpoint or line of sight, and a detection device (not shown) performs a detection 2 of the position and shape of the hand. These data are given to the intention understanding processing unit 3 and analyzed.
The object the observer is gazing at and the task he or she is about to perform next are estimated. Based on this information, the three-dimensional model world 6 is created while referring to the three-dimensional shape database 4.
An update 5 is performed, a three-dimensional model world 6 is generated at high speed, an image is generated 7 using real-time three-dimensional computer graphics, and the image is projected onto a screen 8.
displayed on the screen in real time.

[Effects of the Invention] As described above, according to the present invention, the modeling hierarchy is differentiated between the vicinity of the gaze target and the periphery thereof, so that the amount of calculation can be significantly reduced. Conversely, this extra time allows for precise calculation of the vicinity of the gaze point. the result,
This enables high-speed, high-quality image generation and display. When simulating the case where a display object with a small depth, that is, a small component in the Z-axis direction, is projected onto a screen using the method according to the present invention, the processing time is reduced to less than 1/3 of the conventional method for a screen with a viewing angle of 30°, which is 60 1/4 in case of ゜
It became the following. For objects that have depth, the processing time can be further reduced by the square of this value.

Furthermore, the amount of calculation can be further significantly reduced for objects that are far from the origin or viewpoint of the display reference coordinate system. Conversely, this extra time can be used to calculate the details of nearby objects. As a result, even faster and higher quality image generation and display becomes possible. If the image quality is the same as in the past, the processing speed will be significantly reduced to 1/100 or less.

By detecting line of sight, it is possible to adaptively control the definition of each part of the displayed image, so it is possible to reproduce an image similar to what appears on the retina when looking at real space when looking at the screen. Natural and powerful images can be generated and displayed. Furthermore, it becomes possible to display binocular stereoscopic computer graphics at high speed and with high quality.

Furthermore, the display that predicts the movement of the line of sight generates an image that is synchronized with the movement of the eyeballs, etc., so there is less discomfort due to delays in image display, and a sense of unity with the image is improved.

Further, the present invention is particularly effective in the computer graphics processing portion of intelligent encoded communication with a rich sense of reality, and in stereoscopic computer graphics. In addition, it can be used in many computer graphics fields that require high processing speed and high quality images.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the principle of this invention. Figure 2 is the first
FIG. 3 is a diagram illustrating hierarchical modeling of the displayed objects shown in the figure. Figures 3 and 4 are screen gaze points ep(s)
FIG. 3 is a diagram for explaining a method for finding points on a model corresponding to . FIG. 5 is a diagram for explaining a method for selecting a model hierarchy. FIG. 6 is a diagram for explaining a method for selecting a model hierarchy in another example of the present invention. FIG. 7 is a diagram showing quadratic and cubic Bezier curves. FIG. 8 is a diagram showing still another example of the present invention, in which the viewpoint is the optical viewpoint of the observer's eyeball. FIG. 9 is a diagram showing an embodiment in which the position of the constituent surface to be displayed is given by the center position of the surface. FIG. 10 is a diagram showing the relationship between the angle of deviation from the point of gaze and the relative visual acuity, measured by measuring the visual acuity around the point when the human eye gazes at a point on a plane. FIG. 11 is a diagram showing still another example of the present invention. FIG. 12 is a diagram illustrating the principle of still another example of the present invention, and is a diagram illustrating an example in which the delay due to the movement of the observer is further reduced. FIG. 13 is a diagram showing a high-threshold cutoff filter that is placed around the screen's gaze point. FIG. 14 is a diagram showing the result of applying the filter shown in FIG. 13 to an original image captured by a camera. FIG. 15 is a schematic block diagram of an embodiment of the present invention. FIGS. 16 and 17 are flowcharts for explaining the operation of one embodiment of the present invention. FIG. 18 is a flow diagram for explaining the operation of another embodiment of the present invention. FIG. 19 is a schematic block diagram of another embodiment of the invention. FIG. 20 is a diagram showing an example of application of the present invention. FIG. 21 is a diagram showing a typical conventional three-dimensional image generation method. Figure 22 shows the coordinate system shown in Figure 21 as Y (w
) This is a diagram seen from above the axis. FIG. 23 is a diagram of the coordinate system shown in FIG. 21 viewed from the Y (e) axis in the Z (e) axis direction. Figure 24 is the second
4 is a diagram of the display target shown in FIG. 3; FIG. In the figure, obj is a display object known in the display reference coordinate system, Pi (obj) is a structural point of the display object indicated in the object coordinate system, Sl is a screen, and h-. bj is a perspective projection image, ○(e) is a viewpoint, ep(s) is a screen gazing point, eo(w) is a spatial gazing point, and Th is a distance. Patent applicant A.T.R. Communication System Research Institute Rin 10 z2 Ward HiYhiit -01 (b) (Electronic Wing 1) Ryusho” S8 Figure 10 Ward 1 year old!,, 9, from In the wound (degree) Island 2 times 0fel + - 11 Island ward eL eR plexus ``Ibihi o 30 50 strokes 霜周VL
Wholesale 1i is also for space; ＆+cpd1-575= Mizu Mochi 24 ffi Gourd 2 Mo 23

Claims (9)

[Claims] (1) A line-of-sight tracking high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the constituent planes of the object are in the object coordinate system and the size of the area is determined by at least one Detecting a gaze point where a user's line of sight or line of sight intersects with the two-dimensional screen when the component plane of the object is projected onto the two-dimensional screen when viewed from an arbitrary viewpoint, which is hierarchically described as an element; A point at which a line segment connecting the line of sight or a point of gaze on the two-dimensional screen and the line of sight first intersects a constituent surface of the object in the display coordinate system is defined as a spatial point of gaze, and a point at or near the point of spatial gaze . A line-of-sight tracking high-speed image generation and display method, characterized in that the level of hierarchy is selected for each space in which at least one parameter is a distance from a feature point necessary for expressing a display target. (2) The hierarchical level is selected using a distance from the origin of the display reference coordinate system or the viewpoint to an arbitrary point of the object expressed in the object coordinate system as a second parameter. The eye-tracking high-speed image generation and display method described above. (3) The line-of-sight tracking type high speed according to claim 1 or 2, further characterized in that a projection image viewed from a point near the optical system principal point of the left and right eyes is generated as a viewpoint. Image generation and display method. (4) A point where the line of sight intersects with the two-dimensional screen is set as a point of interest, and the periphery of the point of interest is displayed with a high-frequency cutoff filter applied thereto. 3. The eye-tracking high-speed image generation and display method according to any one of items 1 to 3. (5) A gaze-following high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, the constituent planes of the object being in the object coordinate system and having at least one area size. Described hierarchically as elements, when projecting the object constituent surface onto the two-dimensional screen when viewed from an arbitrary viewpoint, detects a gaze point where the user's line of sight or line of sight intersects with the two-dimensional screen, and determines the line of sight. Alternatively, a line-of-sight tracking high-speed image generation and display method, characterized in that the level of hierarchy is selected using, as at least one parameter, a distance from a line connecting the gaze point of the two-dimensional screen and the line of sight to the object. (6) The hierarchical level is selected using a distance from the origin or viewpoint of the display reference coordinate system of the two-dimensional screen to an arbitrary point on the object expressed in the object coordinate system as a second parameter. 3. The eye-tracking high-speed image generation and display method according to item 5. (7) Line-of-sight tracking type high-speed image generation according to claim 5 or 6, characterized in that a projected image viewed from a point near the optical system principal point of the left and right eyes is generated. Display method. (8) The eye-tracking high-speed image according to any one of claims 5 to 7, characterized in that the eye-tracking high-speed image is displayed using the position of the gaze point as a starting point and applying a high-frequency cutoff filter to the periphery thereof. Generation display method. (9) A line-of-sight tracking high-speed image generation and display method for projecting and displaying an object having a three-dimensional structure on a two-dimensional screen, wherein the constituent surfaces of the object are in the object coordinate system and the size of the area is determined by at least one Hierarchically described as elements, when projecting the target constituent surface onto the two-dimensional screen when viewed from an arbitrary viewpoint, the display standard of the two-dimensional screen at the image display time after the movement of the user's line of sight. Estimate the spatial gaze point or its vicinity, which is the gaze point in the coordinate system,
The distance from the spatial gaze point or its vicinity is at least 1
A gaze-following high-speed image generation and display method, characterized in that the hierarchical level is selected as one parameter and a projection image is generated in advance.