JPH0831140B2 - High-speed image generation and display method - Google Patents

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to computer graphics for displaying a high-speed and realistic image on a screen, that is, a high-speed image generation and display method.

[Prior Art] A technique of so-called computer graphics, in which a three-dimensional image is generated and displayed by a computer, is being used in various fields such as simulation of scientific and technological calculation. High quality and high speed are contradictory conditions, and at the same time, there is no technology to satisfy them. In other words, when trying to generate a high-quality image, a huge amount of calculation time is required for image generation, and conversely, for the purpose of high-speed, real-time display, for example, displaying several tens of images per second, rough calculations must be performed. And the quality of the generated image deteriorates.

On the other hand, the application field of computer graphics is expanding to the field of image communication called intelligent coded communication. In this method, the transmitting side extracts the three-dimensional structural information of the transmission target by image recognition, parameterizes the characteristics, and transmits the parameterized information. On the receiving side, three-dimensional structure information (database) is prepared in advance, and the three-dimensional database is converted at high speed and displayed based on the transmitted characteristic parameters. In these communication methods, the receiving side can freely generate and display an image from any viewpoint based on the three-dimensional structural information of the transmitting side, and thus has many advantages as described below.

Motion can be realized by detecting the movement of the recipient's viewpoint and displaying an image corresponding to this movement. Here, kinetic vision is a person's potential ability to perceive space from the change in the image displayed on the retina of the eyeball when the head is moved. Trying to move and raise awareness is a common experience. That is, it is expected that the stereoscopic effect of the image on the screen will be improved by artificially realizing the motion vision by computer graphics, and the display with a high sense of presence can be realized.

Binocular stereoscopic vision can be realized by generating and displaying an image from each of the two eyes of the recipient as two viewpoints. Since both eyes of a person are separated by a distance of about 6 cm, separate images are projected on the retinas of each eye without moving the head. The correspondence between these two different images is called parallax information, and a person uses this to obtain a stereoscopic sense of space. By realizing the image having the parallax information by computer graphics, the stereoscopic effect of the image on the screen is further improved.

In addition, it is possible to easily generate an image in which the person on the other side (sending side) has a line of sight and has a conversation.

However, at present, there are some difficulties in realizing such a communication method. The first is image recognition and extraction of characteristic parameters on the transmission side, and the second is
This is an image generation and display method for displaying a wide field of view at high speed and high resolution on the receiving side. In communication, real-time processing is an essential proposition, and there is a high demand for higher speed than conventional computer graphics.

The reason why high-speed and high-precision display is difficult will be described in detail below.

FIG. 15 is a diagram for explaining a conventional representative three-dimensional image generating and displaying method. In FIG. 15, the screen S1
Is represented by a display reference coordinate system X (w) -Y (w) -Z (w) whose origin is the center of the screen S1. In addition,
This coordinate system has a function of describing the positional relationship of the following coordinate systems, and the origin may not be the center of the screen S1. The viewpoint coordinate system X (e) -Y (e) -Z (e) is
The position and the rotation angle of the coordinate axis are known as viewed from the display reference coordinate system X (w) -Y (w) -Z (w). The display target object obj has a three-dimensional structure, and the target structure point Pi (obj)
Is represented by the target coordinate system X (obj) -Y (obj) -Z (obj).

In such a situation, when the structural point Pi is viewed from the viewpoint Oe, the position on the screen S1 at which the Pi point is displayed will be considered. Display reference coordinate system X (w) -Y
For (w) -Z (w), the target coordinate system X (obj) -Y
Since (obj) -Z (obj) is already known, the structural point P represented by the target coordinate system X (obj) -Y (obj) -Z (obj)
i (pbj) is the display reference coordinate system X (w) -Y (w) -Z
In (w), it is shown as Pi (w) = M1 · Pi (obj).

Here, M1 is a transformation matrix for movement and rotation. Also, the viewpoint coordinate system X (e) -Y (e) -Z (e)
Is also assumed to be represented by the display reference coordinate system X (w) -Y (w) -Z (w), so if this conversion matrix is M2, the display reference coordinate system X (w) -Y ( w)
-The structural point Pi (w) viewed from Z (w) is the viewpoint coordinate system X.
From the viewpoint of (e) -Y (e) -Z (e), it can be expressed as Pi (e) = M2 ⁻¹ · Pi (w) = M1 · M2 ⁻¹ · Pi (obj). In this way, the display reference coordinate system X
(W) -Y (w) -Z (w) and target coordinate system X (obj)-
Y (obj) -Z (obj) and display reference coordinate system X (w)-
Y (w) -Z (w) and viewpoint coordinate system X (e) -Y (e)-
If the relationship with Z (e) is known, the structure point Pi (obj) shown in the target coordinate system can be freely expressed in the viewpoint coordinate system.

Here, X (np) -Y (np) -Z (np) called a normal perspective coordinate system is considered in order to find the point on the screen of the structural point Pi (e) shown in the viewpoint coordinate system. This coordinate system has an origin O (np) on the Z (e) axis of the viewpoint coordinate system, and is a region A called a field pyramid hatched in the viewpoint coordinate system.
B, C, D, A (∞), B (∞), C (∞), D (∞) are rectangular parallelepiped A, B, C, D, A ′ (∞), B ′ ( ∞), C ′
There is a correspondence that matches (∞), D ′ (∞). It is assumed that the direction of the Z (np) axis of the rectangular parallelepiped is normalized to 1. That is, Z (np) = 0 point corresponds to h on the Z (e) axis,
The point of (np) = 1 corresponds to Z (e) = ∞. Under this condition, the rod-shaped object obj shown in FIG. 10 becomes the normal perspective coordinate system X (np) −
In Y (np) -Z (np), Z (n
The larger the p) axis, the smaller the cross section. The conversion between this viewpoint coordinate system and the normal perspective coordinate system is a 4 × 4 matrix M3.
Can be shown as Therefore, Pi (np) can be represented by Pi (np) = M3 · Pi (e). Here, when Pi (np) is parallel-projected on the A, B, C, and D planes, ξi (s) shown by a dotted line is obtained.

In FIG. 15, the normal perspective coordinate system X is shown for simplicity of explanation.
(Np) -Y (np) -Z (np) is the display reference coordinate system X (w)
It is assumed that -Y (w) -Z (w) overlaps the origin and the axial direction. Therefore, Z (e) of the viewpoint coordinate system
The axis is in the same direction as the Z (w) axis of the display reference coordinate system.

In this case, the rectangular parallelepipeds A, B, C and D correspond to the size of the screen. Therefore, ξi (s) is the viewpoint O
It is a projected image on the screen seen from (e). As described above, three matrix operations and parallel projection are required from the structural point Pi (obj) to ξi (s).

Next, a case where the viewpoint deviates from the Z (e) axis of the display reference coordinate system will be described. In FIG. 15, O '(e) is the moved viewpoint position. With this moved point as the origin,
Consider a new viewpoint coordinate system X '(e) -Y' (e) -Z '(e) with the O' (e) -O (w) line as the Z '(e) axis. If the position of the new viewpoint coordinate system viewed from the display reference coordinate system X (w) -Y (w) -Z (w) is known, X (e)-
From Y (e) -Z (e) to X '(e) -Y' (e) -Z '
Conversion to (e) is possible via matrix M4.
Therefore, the structural point Pi seen from the new viewpoint coordinate system is Pi
(E) ′ = M4 · Pi (e) = M4 · M1 · M2 ⁻¹ · Pi (obj)

FIG. 16 is a view as seen from the Y (w) axis of FIG. Pi
Multiplying (e) ′ by the above-mentioned normal perspective projection transformation matrix M3, Pi ′ (np) is obtained.
i ^* (s) is obtained. Here, since the position of the new viewpoint coordinate system X ′ (e) −Y ′ (e) −Z ′ (e) viewed from the display reference coordinate system is known, ξi ^* (s) and O ′ (e) It is easy to find the point where the line connecting the and intersects the screen.
This point ξi ′ (s) is the projected image on the screen viewed from the viewpoint O ′. Thus, the structure point Pi (ob
Four matrix operations and parallel projection are required from j) to ξi '(s).

As described above, the display reference coordinate system X (w) -Y (w)-
In Z (w), the target coordinate system X (obj) -Y (obj)-
The display target in which the position of Z (obj) is described and the display target in which the coordinates of each structural point Pi (obj) in the target coordinate system are described can be perspectively projected onto the screen from an arbitrary viewpoint.
However, as described above, it is necessary to perform many matrix calculations for all the constituent points to be displayed, so the calculation time becomes huge. The operation of the matrix will be considerably faster by using hardware, but the number of constituent points will be tens.
When the score is 00 or more, real-time processing becomes difficult. In addition, a lot of useless processing is performed depending on the position of the display target, limiting the processing speed. This is shown below.

17 is a view of the coordinate system shown in FIG. 15 on the Y (e) axis and in the Z (e) axis direction, and FIG. 18 is a view showing the display object shown in FIG. is there.

In FIG. 17, the screen S1 is composed of a large number of pixels gi. The display target has a three-dimensional structure as shown in FIG. 18, and is composed of a structural point Pi and a structural surface Li formed by the point. FIG. 17 (A) shows the case where this display object is near in the display reference coordinate system, and FIG. 17 (B) shows the case where it is far away in the display reference coordinate system, which are designated as obj1 and obj2 respectively. First
According to the method described in FIGS. 5 and 16, the screen S1
The display objects h-obj1 and h-obj2 are projected on. A signal can be sent to the corresponding pixel on the screen coordinate of each constituent point for display. Since obj2 is located far away, the distance between the constituent points is smaller than that of h-obj1. If this distance is sufficiently smaller than the pixel distance, it is not necessary to calculate all the constituent points included in the pixel. In addition, human eyes have visual acuity characteristics, and it is meaningless to calculate each constituent point on the screen S1 surface at intervals equal to or lower than the resolution of the eyes.

The existing algorithm calculates all the constituent points of the target regardless of the position of the display target and without considering the visual acuity characteristics of the user, so the display speed becomes slow and a complicated target is displayed in real time. Becomes difficult.

[Problems to be Solved by the Invention] The above-mentioned problems are particularly problematic when realizing stereoscopic computer graphics. Stereoscopic computer graphics is a method in which a perspective image on a screen when a three-dimensional display target is viewed is calculated separately from the user's right eye and left eye as viewpoints, and this is calculated by time division or the like for each of the left and right users. This is a method of displaying a stereoscopic image by projecting it on a screen corresponding to the eyes.

In the stereoscopic display, since the depth interval is clearly recognized by the user, it is necessary to model a large object (such as a building) that is originally large and a small object (such as a bug) to be small. In computer graphics that is not stereoscopic, it is possible to make a great idea that the object is placed close to the viewpoint without modeling it greatly. However, in stereoscopic view, since the depth interval is given by parallax information, even if a small object is in the foreground, it is not recognized as a large object, and the small object appears to be in the foreground.

When an object is modeled in its original size, it is necessary to model a large object in detail, assuming that the object comes near the viewpoint and spreads over the screen. On the other hand, when the object is distant from the viewpoint, as shown in FIG. 17 (B), even though the entire screen is displayed on a part of the screen, the same amount of calculation as when the object is near is achieved. There is an irrational need.

The need for high-speed computer graphics has been described above, but even if the computer graphics are real-time, there are some inconveniences depending on the application. In the realistic communication, a virtual space is artificially created by computer graphics, and various interactions with the virtual space occur as if the virtual space were actually present. In this case, what is required is not only high speed, but also the property of changing the image on the screen in synchronization with the movement of the observer.
However, in computer graphics, the time to generate images always remains, so even if it is possible to generate and display several tens of images per second, if there is a delay time to generate each image with respect to the observer's motion. , I feel uncomfortable and cannot get a sense of unity with the virtual space.

Therefore, a main object of the present invention is to realize a high-speed image generation capable of realizing highly accurate real-time computer graphics display required for intelligent coded communication and stereoscopic computer graphics by significantly reducing the amount of calculation. It is to provide a display method.

Another object of the present invention is a high-speed image generation and display method capable of realizing a natural display without discomfort by reducing the time delay from the generation of an image to the display in computer graphics for realizing motion vision. Is to provide.

[Means for Solving the Problems] According to the present invention, an object having a three-dimensional structure is described at a plurality of different levels of detail with the size of a surface as at least one criterion, and (a) an operator's viewpoint is detected, (B) The operator's viewpoint at the next image display time is predicted, (c) the detail level of the target is set with the distance from the predicted viewpoint to the target as at least one parameter, and (d) the set detail level. An image obtained by projecting the target on a two-dimensional screen is generated and displayed based on the model of the target designated by the above and the predicted viewpoint, and (a) to (d) are repeated.

[Operation] The high-speed image generation and display method according to the present invention detects the operator's viewpoint, predicts the operator's viewpoint at the next image display time, and determines at least one distance from the predicted viewpoint to the target. By setting the detail level of the target as a parameter and repeatedly generating and displaying an image obtained by projecting the target on a two-dimensional screen based on the target model specified by the set level of detail and the predicted viewpoint, The amount of calculation is greatly reduced for distant objects to increase the speed,
Get a natural image.

[Embodiment of the Invention] FIG. 1 is a diagram showing the principle of the present invention, and FIG. 2 is a diagram showing an example of hierarchical modeling of the display target shown in FIG.

First, the principle of the present invention will be described with reference to FIG. Vectors VO1 and VO2 are the display reference coordinate system X
It is a vector from the origin O (w) of (w) -Y (w) -Z (w) to the target coordinate system origin (obj1, obj2) of the display objects obj1, obj2. Also, the difference from the conventional example shown in FIG. 10 is how to hold the structural data to be displayed. FIG. 2 shows an example of hierarchical modeling of the display target. In the example shown in FIG. 2, the display target is a plurality of structural points (k _1j , k _2j , k _3j).
...; j = 1 ...) and a structural surface (l _1j ) formed by this set of structural points is defined as a surface surrounded by four structural points). Each structural point is the target coordinate system X
It is represented by (obj) -Y (obj) -Z (obj). In the hierarchy setting in FIG. 2, the number of structure points is reduced each time the distance between the structure points or the size of the structure surface exceeds a predetermined size, and there are three levels of hierarchy as a whole. A data structure represented by such structure points and a line segment connecting the points is called a wireframe structure.

In the example shown in FIG. 1, the hierarchical degree of the display object hierarchically arranged as shown in FIG. 2 is the origin O (w) of the display reference coordinate system.
The distance from to the origin of the target coordinate system is adaptively selected and displayed as one parameter.

FIG. 3 shows this state, and is a diagram showing an example in which the distance from the origin O (w) of the display coordinate system to the origin of the target coordinate system is adaptively selected and displayed as one parameter. is there. In FIG. 3, areas Th ₁ , Th ₂ , and Th ₃ are areas in which the distance from the origin O (w) of the display reference coordinate system is a predetermined value. The object including the origin of the target coordinate system inside the area Th ₁ (for example, obj1) uses the modeling hierarchy shown in FIG. 2B, and the object including the origin of the target coordinate system inside the area Th ₃ is the second H-obj2 and h-obj3 respectively using FIG.
Is displayed.

Although the display target is the same size as shown in FIG. 2,
When it is near, it is displayed on the screen S1 as large as h-obj1 and when it is far, it is displayed as small as h-obj3. The modeling hierarchy at this time is selected finely when it is close, and coarsely when it is far.
Therefore, the observer feels that both h-obj1 and h-obj3 are fine and necessary. The calculation speed of the perspective projection transformation of obj2 and obj3 is much faster than that of obj1, and therefore, the amount of calculation required to display the entire screen S1 is significantly smaller than in the past. This effect becomes higher as the display target of the background becomes more complicated. In real space, there are far more objects far away than those near the observer. Therefore, it is possible to generate an image with a high sense of realism at high speed.

In the examples shown in FIG. 1 and FIG. 3 described above, the distance from the origin O (w) of the display reference coordinate system to the origin of the target coordinate system is used as a parameter. Since the distance from the system origin to the structural surface of the object becomes large, the distance between a certain structural surface and another structural surface may be greatly separated, and one may be located in front and the other may be located far away. In such a case, it may be more rational to use the distance to a point other than the origin of the target coordinate system as a parameter.

FIG. 4 is a diagram showing an example in which the display object is large and is formed by combining a plurality of structures, and an individual target coordinate system can be set for the structure of each part. In FIG. 4, the origin or viewpoint position of the display reference coordinate system is represented by K, the main target coordinate system is represented by Cor-M-obj, and this position vector is represented by VO, and the main target coordinate system is represented. A part of Cor-M-obj or a sub-target coordinate system of the target obj2 connected to the main target coordinate system Cor-M-obj is represented by Cor-S-obj. Sub-target coordinate system Cor as seen from the main target coordinate system Cor-M-obj
The position of -S-obj is given by the vector VO1. In this example, the position of the target obj2 is represented by V0 + Vo1 = V2. It is assumed that the fine and non-fine modeling layers are divided by the distance Th from the viewpoint position K. Since the origin O (obj2) of the object obj2 is inside the distance Th, a model having a close interval between the structure points is selected. Since the origin O (obj1) of the target obj1 is outside the distance Th, a rough model is selected. The broken line in the target obj1 is a candidate point (line) that has not been selected.

FIG. 5 is a schematic block diagram of one embodiment of the present invention. In the embodiment shown in FIG. 5, it is assumed that the viewpoint is an eyeball and the eyeball is moving, and the display target is stationary in the display reference coordinate system for simplification of the description. The portion shown by the thick solid line in FIG. 5 is a novel means which is not in the prior art.

Referring to FIG. 5, the viewpoint detecting means 10 detects the viewpoint of the eyeball in the display reference coordinate system. The target coordinate system setting means 20 specifies the origin position and rotation angle (angle) of the display target in the display reference coordinate system. The distance calculation means 30 calculates the distance from the viewpoint to the origin of the target coordinate system according to the detection output of the viewpoint detection means 10 and the origin position and rotation angle designated by the target coordinate system setting means 20. In the three-dimensional hierarchical database 80, the constituent surface of the display target object is hierarchically described with the size of the area as one parameter. The component point / component surface generation means 40 receives the origin position of the target coordinate system to be displayed and the above-mentioned distance as inputs, and refers to the component surface data shown by the three-dimensional hierarchical database 80 in the display reference coordinate system. Generate a three-dimensional model with a three-dimensional structure. The portion shown by the dotted line in FIG. 5 constitutes processing means 100 for selecting the hierarchy level using the distance as a parameter.

The perspective projection conversion means 50 receives the viewpoint position and the above-described three-dimensional model, and perspective-projects the display target on a two-dimensional screen. The display image generating means 60 refers to the component plane data obtained by designating the object, the component plane, and the layer in the three-dimensional hierarchical database 80, and the pattern such as coloring of the target component plane is obtained. Add a note. A lenticular screen or the like is used as the display unit 70 to display the target object. Incidentally, in the embodiment shown in FIG.
When generating the constituent points / structural surfaces, the definition of the constituent surfaces is weighted by the distances described above. This operation is performed by the constituent point / structural surface generating means 40 with a predetermined reference value Th and distance. Is performed by extracting the constituent surface of a predetermined hierarchy from the three-dimensional hierarchical database 80 while comparing

FIG. 6 is a flow chart for explaining a specific operation of the embodiment of the present invention.

Next, the specific operation of the embodiment of the present invention will be described with reference to FIG. Now, as shown in FIG. 1 and FIG. 3 above, in the display reference coordinate system, the object ob
It is assumed that j1, obj2, and obj3 generate images on the screen that appear to be at predetermined positions. The position of the viewpoint is the pupil position, and therefore, the viewpoint moves. Also,
For simplicity of explanation, the object is stationary in the display reference coordinate system, the three-dimensional structural data of the object is described in the object coordinate system, and the constituent surface is the size of the area, as shown in FIG. Is parameterized as a parameter.

In step (abbreviated as SP in the figure) SP1, the target coordinate system setting means 20 sets the position vector V _obj-i = (X _obj-i , Y _obj-i , Z _obj-i ) of the target coordinate system origin. Rotation angle of the target coordinate system R _obj-i = (α _obj-i , β _obj-i ,
γ _obj-i ) and are provided to the processing means 100. In step SP2, the viewpoint position Voe = (Xoe, Yoe, Zoe) detected by the viewpoint detecting means 10 is given to the processing means 100. In step SP3, the processing means 100 reads the viewpoint position vector Voe given from the viewpoint detection means 10 at that time, and the position vector V _{obj-i of the} origin of the target coordinate system.
And the viewpoint position vector Voe, the distance dis between the two points dis
_obj-i = | V _obj-i −Voe | is calculated. In step SP4, the processing means 100 compares the calculated distance dis _obj-i with the reference value Th2. When the processing means 100 determines that the distance dis _obj-i is larger than the reference value Th2, the processing means 100 determines 3 in step SP5.
Dimension hierarchical database 80 to target obj3 shown in FIG.
As described above, the model with a low hierarchy shown in FIG. 2 (c) is selected. Then, the processing means 100 sets the rotation angle R _obj-i of the target coordinate system at the position of the position vector V _obj-i of the origin of the target coordinate system.
Generate a three-dimensional model at an angle of.

The processing means 100 determines the distance di between the two points in step SP4.
If it is determined that s _obj-i is smaller than the reference value Th2, the distance between the two points dis _obj-i and the reference value Th1 are determined in step SP7.
Compare with When the processing means 100 determines that the distance dis _obj-i between the two points is larger than the reference value Th1, it selects the model of the hierarchy degree shown in FIG. 2 (b) from the three-dimensional hierarchical database 80 in step SP8. Then, in step SP9, a three-dimensional model is generated in the same manner as in step SP6 described above. Furthermore, the processing means 100 determines the distance dis _obj-i between the two points.
Is smaller than the reference value Th1, a model with a high hierarchy shown in FIG. 2A is selected from the three-dimensional hierarchical database 80 in step SP10, and a three-dimensional model is generated in step SP11.

In this way, when the three-dimensional model is generated in any of SP6, SP9, and SP11, the perspective projection conversion means 50 causes the perspective projection conversion image h _obj-1 , h _obj-2 , h _obj-3 of the target. To generate.
In step SP13, the display image generation means reads the pattern data of each constituent surface stored in the three-dimensional hierarchical database 80 and applies the pattern to the target constituent surface of the two-dimensional screen. The target image generated in this way is displayed on the display means 70 in step SP14.

FIG. 7 is a diagram showing another example in which the distance between points other than the origin is used as a parameter in the target coordinate system, and FIG. 8 is a diagram showing a hierarchically modeled display target.

In FIG. 7, the display object obj is modeled hierarchically as shown in FIG. 8, and further, FIG. 8 (c).
It is assumed that the constituent surfaces a ₁ , a ₂ ... A _i of the hierarchy shown in are represented such that the center position of the surface is represented by Va ₁ , Va ₂ ... Va _{i in} the target coordinate system. In FIG. 5, to make each constituent surface a _i ,
The distance to the constituent plane is the position vector VO of the origin of the target coordinate system.
And the center position Va _i, the VO + Va _i is calculated as the distance Th1,
Select the hierarchy of the constituent plane in comparison with Th2. The constituent surface a ₁ displayed as shown in Figure No. 8 (a) is selected, configured surface a ₂
Is displayed as shown in Figure No. 8 (b) is selected, the constituent surface a ₃
The display target shown in FIG. 8C is selected for. In this way, the distance to any point in the target coordinate system can be used as a parameter.

In the example shown in FIG. 1, the display reference coordinate system X
Starting from the origin O (w) of (w) -Y (w) -Z (w), the distance to an arbitrary point in the target coordinate system was used as a parameter, but the origin of the viewpoint (O (e) or O ') was used. (E) or the like) may be used as a starting point. In this case, if the viewpoint moves even if the display target does not move, there is a lot of trouble to change the hierarchy level each time, but considering that the visual acuity characteristics of the person who recognizes the details of the target greatly depend on the distance from the viewpoint. It can be said that a more reasonable hierarchy level can be selected. In binocular stereoscopic computer graphics, the display target is the screen S1.
In some cases, if the distance from the origin O (w) is used as a parameter as shown in FIG. 1, an object near the viewpoint O (e) will be displayed. Regardless, an absurdity arises that a coarser hierarchy is selected than an object immediately behind (inside) the screen S1. Therefore, although the calculation is complicated, it is also effective to use the distance from the viewpoint to the origin of the target coordinate system as a parameter.

In the present invention, using the size of the region as one parameter means that the scale related to the region such as the distance between the structure points, the structure area, and the volume has a predetermined size, as described in FIG. 2 above. It means layering in which the number of data is reduced each time the number exceeds. Further, parameters other than the size of the area may be used as other parameters. For example, when the display target is a human figure, the eyes and mouth are more important parts for determining the facial expression of a person than the cheeks, nose, and ears. Therefore, it makes sense to model these parts more finely than the other parts. In this way, in addition to the size of the area, the meaning of the display portion can be added to the parameter such as the degree of importance.

Further, as shown in FIG. 2, the structural point may be characteristic data defining the structure of the partial area to be displayed, in addition to the point actually displayed. For example, when the partial region is a free-form surface such as a cheek of a face, a free-form curved surface descriptive expression may be used, and the control parameter of the curved surface represented by the expression may be used as hierarchical data. As an example, when the partial area shown by the dot in FIG. 2 is a curved surface, the Bezier surface formula is used to describe this,
This control point may be hierarchical data.

The Bezier surface can be represented by a product surface of Bezier curves shown by the following equation. The Bezier curve is expressed by the following equation.

R (t: n) = (1-t + tE) ⁿ P _O ; (0 ≦ t ≦ 1) Here, E is called a shift operator and represents EP _j = P _{j + 1} . P _j is called a control point, and its position is given in the form of coordinates. The Bezier curve is in the form of a linear combination of control points.

FIG. 9 is a diagram showing quadratic and cubic Bezier curved surfaces. The example shown in FIG. 9 (a) is the secondary and tertiary (n = 2,3)
And is expressed by the following equation.

R (t: 3) = ( 1-t + tE) when ⁿ P _O Bezier curve t = 0, starting from the control points P _O, t
When = 1, the end control point (P ₂ when n = ₂ , P ₃ when n = ₃ ) is reached. The curves connect smoothly between them near the control points. Increasing the number of control points corresponds to increasing the degree, and the expressiveness of the curve increases.

Bezier surface is expressed by the following equation in the form of product of Bezier curves.

S (u: m, v: n) = (1-u + uE) ^m (1-V + vF) ⁿ P _O0 (0 ≦ u ≦ 1) (0 ≦ v ≦ 1) E and F are shift operators And EP _ij = P _{i + 1, j} , FP _ij = P
_It acts as _{i, j + 1} .

9C and 9D show quadratic and cubic Bezier curved surfaces. In the case of the third order, by preparing 16 control points from P _OO to P ₃₃ , a free-form surface smoothly connecting these is generated.

In this way, if a partial area is adaptively determined and control points sufficient to describe this area are selected, a surface having a fine smooth surface is obtained. On the other hand, if the partial area is made large and the number of control points is reduced, the surface becomes rough. So far, an example has been shown in which the number of control points in the partial area is hierarchized as a parameter.

Furthermore, in the above Bezier surface equation, the structural points are control points.
It depends on how u and v are taken. Therefore, a predetermined control point may be determined, and the interval taken by u and v may be used as the hierarchical parameter. That is, in FIG. 9 (d), the control point is 16
Even if the number is constant for each individual, there are rough and dense component surfaces depending on how to determine the interval Δ between u and v. If the distance Δ is made smaller, the constituent surface is further divided from that shown in FIG. 9 (d), and many small constituent surfaces are produced. In this way, the size of the area can be hierarchized using Δ as a parameter.

Next, as the method of displaying on the screen, as shown in FIG. 1, in addition to the simple method of displaying the points and lines obtained by perspective projection of the wire frame, color data is prepared on the structural surface, Colored when displayed. A pattern called a texture map is prepared as data on each structure surface, and the pattern is rotated, moved, reduced, enlarged, or the like according to the direction and position of the structure surface and displayed on the screen. In the display reference coordinate system, it is possible to assume a light source at a predetermined place, calculate the shadow of each component surface from the viewpoint by the light, and display it.

As an example, a case of modeling a life-sized person plaster image with Bezier curved surfaces will be described. To model this plaster image in detail, the number of control points is about 5000.
Was needed. That is, the 5000 control points are divided into a group of 4 × 4 control points as shown in FIG. 9 (d), and the partial area determined by the 4 × 4 control points is further divided.
By selecting the values of u and v, it is divided into 10 parts. In this way, a structural surface is formed by 500,000 structural points. When the display surface is created by dividing the image into such parts, an image in which details can be recognized in detail can be obtained.

On the other hand, as long as it can be discriminated whether it is a person or not, the control points may be several tens to 100, and u and v may be divided into several. In other words, it can be expressed with about 100 structural points. In this way, 100 to several 1 depending on the environment and need to display even objects such as people.
There is a data amount difference of 000 times. That is, hierarchization can be variously selected with respect to the size of the area.

FIG. 10 is a block diagram for embodying the principle shown in FIGS. 7 to 9 described above. The block diagram shown in FIG. 10 is the same as that shown in FIG. 5 except for the following points. That is, the processing means 101 is means 3 for generating a higher hierarchical level of the target component surface as compared with the processing means 100 shown in FIG.
1, the distance forming means 32 calculates the distance from the viewpoint to the base point of the target constituent surface. The generation means 31 generates, for example, the main surfaces of the objects obj1, obj2, obj3 shown in FIG. Where is the main aspect? It is a component surface of a rough model as shown in FIG. 8 (c) or (d). In FIG. 8, the position of each constituent surface is represented by a vector to the center (base point) of the constituent surface in the target coordinate system, and the angle of the constituent surface is the angle formed by the normal direction of the surface and the position vector. It is represented by. Constituent plane ob expressed in the target coordinate system
j-i-j is the position V _obj-i and the angle R _obj-i of the target coordinate system.
If you know it, you can obtain it by a simple coordinate transformation.

FIG. 11 is a flow chart for explaining the operation of the embodiment shown in FIG. The flow chart shown in FIG. 11 is newly added to the flow chart shown in FIG.
21 is provided, and SP22 is provided instead of step SP3. In step SP21, when the target coordinate system position V _obj-i and the angle R _obj-i are set by the target coordinate system setting means 20 in step SP1, for each target obj-i,
Three degrees of hierarchy (c) at the angle R _obj-i with respect to the target coordinate system position V _obj-i
A dimensional model is generated, and the base point position V _obj-ij of each component surface j of the target obj-i is calculated. When the viewpoint position Voe is detected by the viewpoint detection means 10 in step SP2, the distance calculation means 32 of the processing means 101 in step SP22 causes the distance dis _obj-ij from the viewpoint to the base point of each constituent surface of each target.
= | V _obj-ij −Voe | is calculated. And step SP4
At SP11, the distance is compared with the reference value Th _k (k = 1,2). The other operations are the same as in FIG.

FIG. 12 is a diagram showing another embodiment of the present invention. 12th
In the figure, display objects obj1 and obj2 are the same as those shown in FIG. For each constituent point and constituent plane data, the target coordinate system X (obj _i ) −Y (obj _i ) −Z (obj _i ) of the target is displayed as the reference coordinate system X (w) −Y (w) −Z (w If the relationship of) is known, it can be expressed in the display reference coordinate system. The two viewpoints O (e) R and O (e) L correspond to the principal points of the left and right eye optical systems of each observer. Also, the subscript (-of O (e) R (-
0) is the current viewpoint, (+1) is the next moving viewpoint, and (-1) and (-2) are the previous viewpoints. FIG. 12 shows a case where about 10 to 30 images are generated and displayed per second, and therefore, (+1) (0)
The points (-1) and (-2) are at intervals of about 100 to 33 msec. The position vectors V (obj1) and V (obj2) are the position vectors of the origin of the symmetrical coordinate system of the display target viewed from the viewpoint. The length of this position vector, that is, the distance can be easily obtained.

Here, the selection method of the modeling hierarchy is determined by the distance as in the case shown in FIG. Viewpoint is O
(E) Display objects obj1 and obj2 when in the position of R (0)
In the model, the display objects shown in FIGS. 6 (c) and 6 (d) are selected according to the lengths of V (obj1) (0) and V (obj2) (0), and h-obj1-R is displayed on the screen S1. (0), h-ob
It is projected as j2-R (0).

In this embodiment, the viewpoint is the viewpoint of the eyeball. Since the viewpoint position is close to the center position of the pupil, if the pupil can be captured by two cameras, this position can be easily obtained in the display reference coordinate system by stereo image measurement or the like. Note that an efficient method of extracting the pupil is described in "Image photographing device" by the present inventors (Japanese Patent Application No. 1-18138).
7), the position of the pupil can be measured by using the "line-of-sight detection method" (Japanese Patent Application No. 1-296900).

Next, the display in which the movement of the viewpoint is predicted will be described.
The rotational movement of the eyeball is closely related to visual acuity. When the target slowly moves with the head fixed, the line of sight moves smoothly in the same direction in conjunction with the movement of the target. On the contrary, when the head is moved while gazing at the fixed target, the line of sight moves smoothly in the opposite direction. In this way, when the line of sight moves smoothly, the human visual acuity does not significantly change from the case of observing the stopped target. At this time, the movement of the pupil seen in the display reference coordinate system is also smooth. Therefore, it is possible to predict the next movement of the pupil from the locus of the position and velocity of the pupil.
Therefore, a perspective image to be displayed is generated in advance using such a predicted position of the pupil as a viewpoint, so that the image can be displayed in synchronization with the movement of the pupil, that is, without delay time.

In FIG. 12, O (e) R (-2), O (e) R (-
1) and O (e) R (-0) are the positions of the pupil that change smoothly. The next image is displayed from this position data.
It is possible to predict the viewpoint position O (e) R (+1) after 100 to 33 msec. In FIG. 12, the viewpoint approaches the screen over time, that is, the display target,
Therefore, V (obj1) (+ 1), V (obj2) (+ 1)
Has become shorter. Therefore, as the model hierarchy, FIG. 8B is selected for the display target obj1, and FIG. 8B is selected for the display target obj2. Therefore, as a perspective projection image on the screen S1 viewed from O (e) R (+1), h-obj1-R as shown in FIG.
(+1), h-obj2-R (+1) are prepared. This image is finer by one layer than that of h-obj1-R (0) and h-obj2-R (0). This image is 100
Since it is presented after ~ 30msec, the observer does not feel uncomfortable due to display delay.

Here, when the observer's interested subject changes and the line of sight or face suddenly moves, the predicted position of the pupil is different, so this method seems irrational at first glance. However, there is no problem for the following reasons. That is, as described above, the visual acuity is closely related to the movement of the line of sight, and when the line of sight changes suddenly,
Human eyesight deteriorates, and it takes more than 200 msec to recover. Therefore, even if the predicted viewpoint is wrong, the observer's visual acuity in the generated and displayed image is greatly reduced, so that no discomfort occurs. The correct pupil position may be measured and a perspective projection image from that point may be generated and displayed before the eyesight is restored.

In this embodiment, the two images are generated in real time following the movement of the eyeball. Therefore, the observer can move his / her head and see the display object obj from various directions in the vertical and horizontal directions. Also, since the two images have parallax information, this object can be perceived stereoscopically.

Further, in this embodiment, a binocular stereoscopic display device can be used to display the generated image. As a specific example of this device, a "stereoscopic display device" (Japanese Patent Application No. 2-18051) by the inventors of the present application is possible. The structure of this device is such that a kamaboko-shaped lens sheet called a lenticular lens is provided on the display surface, and pixels for the right eye and pixels for the left eye are arranged on both sides of the back of each lens with a focus in between. . That is, two pixels are arranged as one set so as to face one kabuki lens. The light emitted from the two pixels is spatially separated by the lens and separately enters the left and right eyes of the observer. In this way, stereoscopic viewing is possible without mounting a special device.

FIG. 13 is a block diagram for embodying the embodiment shown in FIG. In this embodiment, an image is generated and displayed in synchronization with the movement of the viewpoint based on the principle shown in FIG. For this reason, in this embodiment, compared with the embodiment shown in FIG. 10 described above, a prediction viewpoint detection means 11, a viewpoint movement database 12, a distance calculation between the origin of the target coordinate system and the prediction viewpoint is newly performed. Means 33, constituent point / constituent surface generation means 41 of the target one screen ahead, and perspective projection conversion means 51 for generating a secondary projection screen at the display time one screen ahead are provided.
The predicted viewpoint detection means 11 estimates the viewpoint at the next image display time, that is, the position of the pupil, according to the detection output of the viewpoint detection means 10 while referring to the database 12 regarding the behavior of the viewpoint. The distance calculating means 33 between the origin of the target coordinate system and the predicted viewpoint calculates the distance between the position of the origin of the target coordinate system and the predicted viewpoint. The component point / component surface generating means 41, similar to the generating means 40, generates a three-dimensional model at the position of the target coordinate system designated by the target coordinate system setting means 20. Then, the generation means 41 fetches the constituent surface data of an appropriate hierarchy from the three-dimensional hierarchical database 80 while comparing the distance predicted by the distance calculation means 33 with the reference value Th, and the three-dimensional data at the display time of the next screen. Generate a model. The perspective projection conversion means 51 generates a two-dimensional projection screen at the display time of the next screen. The other operations are the same as those shown in FIG.

FIG. 14 is a diagram showing an application example of the present invention. This 14th
In the example shown in the figure, a computer graphics image can be observed in a highly realistic manner. The viewpoint corresponds to the center position of the pupil or iris. For this viewpoint or line of sight, viewpoint detection 1 is performed by a sight line detection device (not shown), and position 2 and shape 2 of the hand is detected by a detection device (not shown). These data are given to the intention comprehension processing unit 3 and analyzed to estimate the object the observer is gazing at and the work to be performed next. Based on this information, the 3D model world 6 is updated 5 while referring to the 3D shape database 4, the 3D model world 6 is generated at high speed, and the image generation 7 is performed by real-time stereoscopic computer graphics. It is performed, is projected and converted to the screen 8, and is displayed on the screen in real time.

EFFECTS OF THE INVENTION As described above, according to the present invention, it is possible to significantly reduce the amount of calculation for an object that is far from the origin of the display reference coordinate system or the viewpoint. On the contrary, in this extra time, it is possible to finely calculate the details of the nearby objects.
These enable high-speed, high-quality image generation and display. Further, the present invention is more effective as the background becomes more complicated. As an example, when a meeting room is created with computer graphics and a person image is created in the meeting room, the complicated background is roughly modeled, so it is several hundred times faster. Can be expected. Moreover, with the pupil as the viewpoint, by predicting this movement, an image synchronized with the movement of the eyeball is generated, so there is little discomfort due to the delay in image display, and high-quality motion vision with an improved sense of unity with the image. Computer graphics image display becomes possible. Furthermore, high-speed and high-quality display of binocular stereoscopic computer graphics is also possible. Further, according to the present invention, the utilization effect is particularly great for the computer graphics processing part of intelligent coded communication and the stereoscopic computer graphics which are rich in realism. It can be used for many computer graphics fields that require high speed and high quality.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the principle of the present invention. FIG. 2 is a diagram showing hierarchical modeling of the display target shown in FIG. FIG. 3 is a diagram showing an example in which the hierarchy is adaptively selected and displayed with the distance from the origin O (w) of the display coordinate system to the origin of the target coordinate system as one parameter. FIG. 4 is a diagram showing an example in which the display object is large and is formed by a combination of a plurality of structures, and an individual target coordinate system can be set for each partial structure. FIG. 5 is a schematic block diagram of one embodiment of the present invention. FIG. 6 is a flow chart for explaining the operation of the embodiment shown in FIG. FIG. 7 is a diagram showing another example in which the distance of points other than the origin is used as a parameter in the target coordinate system. FIG. 8 is a diagram showing a display object modeled hierarchically. FIG. 9 is a diagram showing quadratic and cubic Bezier curved surfaces. FIG. 10 is a schematic block diagram showing another embodiment of the present invention. Figure 11 shows
FIG. 11 is a flowchart for explaining the operation of the embodiment shown in FIG. FIG. 12 is a diagram showing another example of the present invention. Thirteenth
The figure is a block diagram for realizing the example shown in FIG. FIG. 14 is a diagram showing an application example of the present invention. FIG. 15 is a diagram showing a conventional representative three-dimensional image generation / display method. FIG. 16 is a view of the coordinate system shown in FIG. 15 viewed from the Y (w) axis. FIG. 17 shows the Y coordinate system shown in FIG.
This is viewed on the (e) axis and in the Z (e) axis direction. First
FIG. 18 is a diagram showing the display object shown in FIG. In the figure, obj1 is the display target known in the display reference coordinate system, P
i (obj) is the structural point to be displayed in the target coordinate system, S1
Is a screen, h-obj is a perspective projection image, O (e) is a viewpoint, ep is a screen gazing point, eo is a spatial gazing point, and Th is a hierarchy selection criterion.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Haruo Takemura, Haruo Takemura, Kyoto Prefecture, Seika-cho, Osamu, Osamu, Osamu, No. 5, Mihiratani, ATR Communications Systems Laboratories, Inc. (72) Satoshi Ishibashi, Soraku-gun, Kyoto Prefecture Seika-cho Osamu Osamu Osamu 5 Hiratani No. 5 AT R Communication System Research Institute, Inc. (72) Inventor Kenji Akiyama 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation (72) Invention Shinji Tetsuya 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation (72) Inventor Hiroyuki Yamaguchi 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation (56) References Japanese Patent Laid-Open No. 1-205277 (JP, A)

Claims (1)

[Claims] 1. An object having a three-dimensional structure is described at a plurality of different levels of detail with at least one surface size as a reference, (a) the operator's viewpoint is detected, and (b) the next image is displayed. Predict the operator's viewpoint at time, and (c) at least 1 distance from the predicted viewpoint to the target.
The detail level of the target is set as one parameter, and (d) the image of the target projected on the two-dimensional screen is generated and displayed based on the model of the target specified by the set detail level and the predicted viewpoint. A high-speed image generation and display method characterized by repeating the steps (a) to (d).