JPH09147142A - Displaying method and device therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To smoothly perform a work by using the result from that an observer freely observes by detecting the location and attitude of each of the observer and an object to be observed and determining the location/attitude and the size by using the result. SOLUTION: When a pallet 11 moves in a three-dimensional direction, a robot 10 transmits the location/attitude information on an object 70 to be observed to a 6-degree of freedom robot controller 20. When an arbitrary three- dimensional direction by an observer 90 exists, a display device 60 transfers the location/attitude information on the display device 60 to a 6-axis positioning mechanism controller 40 via a positioning mechanism 50. A location/attitude detector 80 for observer acquires the location/attitude information on the observer 90. A display device controller 30 performs a coordinate conversion by using the location/attitude information on each of an object 70 to be observed from the robot controller 20 and the display device 60 from the positioning mechanism controller 40 and the observer 90 from the location/attitude detector 80 for observer and the computer graphics model of the object 70 to be observed is made to be displayed on the display device 60.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a display method and device, and more particularly to a human interface used in a robot field, a device maintenance field, a surgical medical field, etc., in which a computer image is used to support an observer's work. A display method and device.

In recent years, with the rapid progress of computer hardware, computer graphics, image processing technology, etc., images processed or generated by computers in the fields of telerobotics, equipment maintenance, surgery and medical fields are used for work support. The technology to do so is being actively introduced. Above all, real-world information that humans perceive,
The concept of augmented reality is becoming popular in which observation and work are effectively performed by complementarily using image information processed by a computer.

[0003]

2. Description of the Related Art In the following conventional techniques, the field of medical application will be described as an example. A display device in a conventional medical application field will be described. FIG. 7 is a diagram for explaining a conventional display device. As an example of a conventional display device in the medical application field, “trial display of an organ in three dimensions using a three-dimensional plotter method and trial of synthetic observation with a living body” (image information (M),
Vol. 26, No. 20, p. 1169). In this technique, a special display 2 for forming an image of a bright spot in space is used to draw a three-dimensional solid figure 3 based on medical diagnosis data, which is passed through a half mirror 4 to a part of a human body 5 which is an object to be observed. Observer 1 (doctor)
Provide useful information to human beings. This device has an advantage that the observer 1 can superimpose the human body 5 as the observed object and the computer-processed image based on the medical data under a certain condition.

[0004]

However, in the above conventional display device, the display is fixed, and
Since there is no means for observing the position of the observer or the position / orientation of the display, the observer always has a certain position / orientation.
In addition, it is necessary to look at the indicator fixed in a fixed position / posture, which is a problem in terms of workability.

As described above, in the conventional technique, the observer must look at the display unit fixed at a predetermined position / posture and at a fixed position / posture.
Moreover, there is a problem that it is difficult to smoothly perform the work while utilizing the observation result.

The present invention has been made in view of the above points, and an observer always has to look at a display unit fixed at a fixed position / posture at a certain fixed position / posture. It is an object of the present invention to provide a display method and a device which solve the above-mentioned drawbacks, allow an observer to observe with a degree of freedom, and use the observation result while smoothly performing work.

[0007]

SUMMARY OF THE INVENTION The present invention is a display method for presenting an image to an observer by displaying, detecting the respective positions and orientations of the observer and the observed object, and detecting the respective positions. -Determine the position, orientation and size of the image to be displayed using the orientation.

Further, according to the present invention, the computer graphics model of the observed object is displayed as a part of the image when the image is displayed. Further, according to the present invention, when displaying, the captured real image of the observed object is displayed as a part of the image.

Further, according to the present invention, the position / orientation and size of the computer graphics model are determined and displayed so that at least one of the position / orientation and size of the observed object viewed from the observer matches. Further, the present invention detects at least the position / orientation of the image pickup device, images the object to be observed, uses the picked-up real video as a part of the image, and displays at least the position / posture and size of the real video to be displayed. 1
The position, orientation, and size of the computer graphics model are determined so that they match at least one.

FIG. 1 is a diagram showing the principle of the present invention. The present invention is a display device for presenting an image to an observer 200, a display unit 300 for displaying an image, and a position / position for detecting the position and orientation of each of the observer 200 and the observed object 100. The posture detection unit 400 and the determination unit 500 that determines the position / posture and size of the image on the display unit 300 using the respective positions / postures detected by the position / posture detection unit 400 are included.

Further, the present invention further comprises image adding means for using the computer graphics model of the observed object 100 as a part of the image. Further, the image adding means of the present invention, the position of the computer graphics model on the display means, so as to match at least one or more of the position, orientation and size of the observed object seen from the observer. Determine posture and size.

Further, the present invention further has an image pickup means for picking up an image of the object to be observed, and the actual image picked up by the image pickup means is used as a part of the image. Further, a semi-transmissive type is used as the display means of the present invention. The present invention uses an image pickup means for picking up an object to be observed, a detection means for detecting the position / orientation of the image pickup means, a real image picked up by the image pickup means as a part of an image, and The position of the computer graphics model on the display means so that at least one of the position / orientation and size of the actual image matches.
It has a computer graphics model determining means for determining a posture and a size.

According to the present invention, since the position and orientation of the observer and the object to be observed are automatically detected, it is not necessary for the observer to be in a fixed position, and the object to be observed can be observed at any place. It will be possible. Further, since the display device is also movable, the observer can move the display device to an arbitrary position to observe the observed object.

Further, since the captured real image is displayed as a part of the image on the display device together with the real world information by using the image pickup means, the work can be performed while utilizing the observation result of the image pickup. It is also possible to do it.

[0015]

FIG. 2 is a block diagram of a display device of the present invention. The display device shown in the figure holds the object to be observed 70 and is movable along the x, y, and z axes, the robot 10, the pallet 11 on which the object to be observed 70 is installed, the position of the robot 10.
6-DOF robot controller 20 for detecting and controlling posture, 6-DOF positioning mechanism 50 for holding the display 60, and display 6 from the 6-DOF positioning mechanism 50
An observer position / orientation detector 80 for detecting the position / orientation of 0 and controlling the set position of the display 60, a 6 free positioning mechanism 40, and an observer's position / orientation using the transmitter 90, It is composed of a display device controller 30.

The robot 10 is connected to the pallet 11, and when the pallet 11 moves in the x, y, and z directions, the information is transmitted to the 6-DOF robot controller 20 as position / orientation information of the object 70 to be observed. The 6-DOF robot controller 20 transfers the position / orientation information of the observed object 700 from the robot 10 to the display device controller 30.

The display unit 60 displays an arbitrary 3 by the observer 90.
It is possible to move in the dimensional directions (x, y, z), and when there is a movement, the position / orientation information of the display 60 is transferred to the positioning mechanism 50. The positioning mechanism 50 is the display 6
The position / orientation information transferred from the display device 60 is transferred to the 6-axis positioning mechanism controller 40.

The 6-axis positioning mechanism controller 40 transfers the position / orientation information of the display device 60 at a certain point of time to the display device controller 30. Observer position / orientation detector 8
0 acquires the position / orientation information of the observer 90 and transfers it to the display device controller 30. The observer position / orientation detection unit 80 may be any device that can recognize the position / orientation information of the observer 90.

The display device controller 30 includes a display device 60.
A display control unit 31 that determines the position / orientation and size of the image on the above, a graphics generation unit 32 that generates a computer graphics model of the observed object, and a position of the observer
The coordinate conversion unit 33 performs coordinate conversion using the posture data, and is realized by a personal computer. The display device controller 30 controls the position / orientation information of the observed object input from the 6-DOF robot controller 20,
The position / orientation information of the display device 60 input from the axis positioning mechanism controller 40 and the position / orientation information input from the observer position / orientation detector 80 are used to perform coordinate conversion, and a computer of the observed object 70 is displayed. The graphics model is controlled to be displayed on the display 60.

In addition, the observer 90 observes the position / orientation detector 80 for observer, which outputs a position detection signal, for example, in response to the detection of the position / orientation from the observer position / orientation detector 80. Any means that can detect the position / orientation of the person may be used. This display device is used as follows. That is, an observer who assembles, maintains, maintains, or repairs the observed object 70 corrects the position / orientation of the observed object or adjusts the device parts according to the image information displayed on the display 60. It is possible to confirm the correct position, confirm the missing parts, and verify the instruction of the assembly order while comparing the observed object that is a real object and the image information.

[0021]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 3 shows the configuration of the display position controller according to the first embodiment of the present invention. Display position controller 30 shown in FIG.
Is an object to be observed which stores position / orientation information of the object to be observed 70 input from the 6-degree-of-freedom robot controller 20 in addition to the display controller 31, the graphics generator 32, and the coordinate converter 33 shown in FIG. The position / orientation storage unit 34, the observer position / orientation storage unit 35 that stores the position / orientation information of the observer that is input from the observer position / orientation detection unit 80, and the display 60 that is input from the display unit 60. The display position / orientation storage unit 36 stores the position / orientation information.

As a result, the coordinate conversion unit 33 acquires the current position / orientation information of the observed object 70 from the observed object position / orientation storage unit 34 and sets the observed object coordinates Σw. The coordinate conversion unit 33 also sets the observer viewing coordinates Σ _view based on the current position / orientation information of the observer 90 from the observer position / orientation storage unit 35. Further, the coordinate conversion unit 33 acquires the current position / orientation information of the display unit 60 from the display unit position / orientation storage unit 36 and displays the display unit coordinates Σ.
Set _disp .

The graphics generation unit 32 includes a coordinate conversion unit 3
3, the computer graphics image of the observed object 100 is generated from the relationship between the observer viewing coordinate Σ _view that the observer 90 is currently viewing and the display device coordinate Σ _disp . The display controller 31 includes a graphics generator 32.
The computer graphics image generated by the control unit 6 controls the position on the display unit 60, the display size, etc.
Control for displaying 0.

FIG. 4 schematically shows an observation system for observing an object to be observed according to the first embodiment of the present invention. All coordinate systems handled below are orthogonal coordinate systems. First, global coordinates Σg (X, Y, Z) are taken as reference coordinates in the observation space. The Z axis is aligned with the vertical direction. The operation will be described based on the configuration diagram of the display device shown in FIG. 2 and the configuration diagram of the display device controller 30 shown in FIG.

The position / orientation of the object 70 to be observed is held on the pallet 11 at the hand of the robot 10 at a known position / orientation. In order to achieve this purpose, the encoder signal of the joint of the robot 10 is obtained through the A / D converter (not shown) to obtain the joint angle, and then the forward kinematics calculator (not shown) is used. Position / orientation is calculated, and this value is stored in the observed object position / orientation storage unit 34. Here, the observed object coordinate Zw is approximately at the center of the observed object 70.
Set. Therefore, from the relationship between the position and orientation of the hand of the robot 10 and the observed object coordinates Zw, the position of the observed object
The posture is identified as a relationship of the observed object coordinates Zw viewed from the reference coordinates Zg. The operation here corresponds to modeling conversion in general 3D graphics processing.

The three-dimensional computer graphics model of the observed object is described in the same relationship as the observed object coordinates Zw. Here, the wire frame model is adopted in consideration of high-speed processing and display. The position / orientation of the display device 60 held by the positioning mechanism 50 having the manual 6-axis moving mechanism is also identified as the relationship of the display device coordinates Z _disp as viewed from the reference coordinates Zg, and the display device position / orientation storage unit 36 is also provided. Stored in. This mechanism is back-drivable, and the observer 90 can grab the display 60 and set the desired position and orientation. As shown in FIG. 2, the display device 60 is also movable in the x-, y-, and z-axis directions.

The position / orientation of the observer 90 is a representative point of the observer.
As the observer viewing coordinates Σ _viewExpressed by This view
Inspector viewing coordinates Σ_viewThe origin of the view reference point Ov
And as the position vector represented by the reference coordinate Σg,
It is expressed as.^g P_view = [P_xp_yp_z1]^T The position / orientation of the observer 90 is the position / orientation using image processing.
Observer view from the reference coordinate Σg by the force detector 80
It is identified as the relationship of the ing coordinates Σview. Position and posture inspection
The output device 80 is a plurality of CCD cameras with infrared filters.
85 is placed in the work space and the helmet head of the observer 90
Measures the position of multiple infrared LEDs 95 mounted on the top
Then, the posture is calculated from the plurality of positions.

Next, description will be made with reference to FIG. Observer 9
The observed object 70 that appears at 0 is the coordinate conversion unit 3 shown below.
It can be considered that perspective projection is performed on the virtual projection plane (hereinafter, projection plane A) by the viewing conversion processing of No. 3. At this time,
Photographing the reference point O _R (virtual source point) are intended to be distant from the viewer viewing coordinate Σview the projection plane A.

First, assuming that the observer 90 always looks directly at the direction of the observed object 70 when observing, the z-axis of the observer viewing coordinates Σview coordinate axes is used to observe the line of sight to the observed object 70 and the remaining x. , A plane parallel to the plane defined by the y-axis
Is determined. The z-axis always coincides with the direction from the observer viewing coordinate Σview to the observed object coordinate Σw, and the line-of-sight direction unit vector n on the axis is given by the following equation.

N = [n _{x ny nz} 0] ^{T The} y-axis is in the vertical plane including the z-axis, and the direction vector v orthogonal to the line-of-sight direction unit vector n is similarly given by the following equation. v = [v _X v _Y v _Z 0] ^{T The} remaining x-axis direction vector u is represented by u = v × n,
It is expressed by the following equation.

U = [u _X u _Y u _Z 0] ^T Therefore, if ^g R _view = [u v n], the conversion from the reference coordinate Σg to the observer viewing coordinate Σ _view is expressed by the following equation. It

[0032]

(Equation 1)

Further, from the view reference point O _v to the projection reference point O
_Let r be the distance to _R and L be the distance to the projection plane A.
Now, assuming that the point Q on the observed object 70 is represented by ^g Q in the reference coordinate Σg, the ^view Q represented by the observer viewing coordinate Σ _view is ( ^view Q 1) ^T = ( ^g T _view ). ^-1 · ^(g Q 1) Next, 'When the point Q' points point Q is projected Q is an intersection of the projection plane a and a straight line connecting the point Q and the projection reference point O _R.

[0034] In addition, the point Q 'is the projection reference point O _R and the point Q r
+ L: | ^view q _z | -L. Where ^view q _z
Is the z component of the point Q on the observed object 70. Therefore, the point Q ′ is represented by the following equation in the observer viewing coordinate Σ _view . ( ^View Q ′ 1) ^T = ^A T _view ( ^view Q 1) ^T where

[0035]

(Equation 2)

Further, d = (r + L) / (R + | ^view q _z
|) Therefore, an arbitrary point ^g Q on the observed object represented by the reference coordinate Σg is calculated as follows:
It is represented by _view and is projected on the projection plane A as ^view Q '.

(^viewQ'1)^T=^AT_view・ (^gT
_view)^-1・ (^gQ 1)^T A point on the projection plane A is represented by x and y coordinates. Therefore,
The object^viewq_xas well as ^viewq_yWith a simple procedure using
It is projected on a virtual projection plane A for the observer.

Now, consider the relationship between the observer and the actual display 60. The display 60 is perpendicular to the line of sight of the observer, that is,
If it is assumed that it is perpendicular to the vector n of the observer viewing coordinates Σ _view , the computer graphics image corresponding to the observed object 70 generated by the graphics generating unit 32 is projected.

Now, the display unit coordinate Σ _disp is set at the center of the display unit 60 (virtual projection plane A). From the perspective projection perspective, the display coordinate Σ _disp here is the observer viewing coordinate Σ _view.
The position is at a distance L from, and the posture thereof coincides with Σ _view . Now, considering the relationship between the actual display device 60 and the observer 90, the display device 60 is not always perpendicular to the observer 90. That is, the display device 60 can actually take an arbitrary position / orientation independently of the observer viewing coordinate Σ _view . Therefore, the display coordinates considering the actual posture of the display device 60 are newly set as the display device coordinates Σ _disp ′, and the plane including these coordinates is set as the projection plane B again. That is, the projection plane B represents the actual display device 60 itself.

[0040] Here, a size that is rotated with respect to sigma _disp indicator coordinates sigma _disp 'does not have a significant effect on the optical visibility of the display unit 60, their angle vector φ = (φ _x φ _y φ _z ) ^T. At this time, the point Q ′ projected on the projection plane A is re-projected on the projection plane B to become a point Q ″, and from the observer 90, that is, the observer viewing coordinate Σ.
_{From the view} , the projected point Q ″ is as follows.

( ^View Q ″ 1) ^T = ^B T _A ( ^view Q ′ 1) ^T where

[0042]

(Equation 3)

K = (q _x ' ² + q _y ' ² ) ^1/2 Finally, in order to display the projection diagram on the display 60, the display control unit 31 converts it into the display coordinate Σ _disp 'as follows. . ( ^{Disp '} Q ", 1) ^T = ^disp' T _view ( ^view Q", 1)
^The display control unit 31 can arbitrarily convert the position and size on the display 60 based on this data. In particular, the size to be displayed is automatically set based on the relationship with the window of the display device 60 as in the case of the normal graphics method.

The image information is not limited to the object to be observed, but the size of the object to be observed,
Various kinds of information such as parts specifications can also be displayed at arbitrary positions and orientations as graphics information. The procedure is exactly the same as in the case of the model of the observed object.

As described above, when the observer looks at the display device 60, a computer graphics model image that substantially matches the image seen when the observed object 70 is viewed regardless of the position and orientation of the display device 60. Can be seen. In principle, this result does not depend on the positional relationship between the observed object 70 and the observer 90, and the position of the observer 90 and the display 60 is
It doesn't depend on the posture. That is, the observer can observe the object to be observed from any position angle, and at the same time, the display 6
Since 0 can be freely placed, the efficiency of observation and work is dramatically improved.

Next, a second embodiment of the present invention will be described.
FIG. 5 shows the configuration of the display device according to the second embodiment of the present invention. The figure shows an example in which the display 60 is of a semi-transmissive type. The display 60 includes a display unit including a half mirror 61 and a C
The projector 62 includes an RT, a lens, and an optical system. Although the graphics image from the projector 62 is projected on the half mirror 61, since it is the half mirror 61, it is 50% to 60% from the viewer through the display 60.
It is possible to observe the object to be observed while ensuring the transmittance of.

As a result, the graphics model on the display can be seen in a superposed manner with the real image of the observed object. Unlike conventional head-mounted displays, the observer does not always wear the indicator,
There is an advantage that workability is high without feeling a blocked state.

Next, a third embodiment of the present invention will be described.
FIG. 6 shows the configuration of a display device according to the third embodiment of the present invention. In this example, a CCD camera 65, which is an image pickup device for picking up an object to be observed 70, is mounted on a 6-DOF positioning mechanism,
A real video imaged from a predetermined position and orientation can be used as a part of the display image. Illumination may be used with the camera 65 if desired. In FIG. 6, the observed object 70
The camera image of is displayed on the display 60. In this case, the graphics models can be overlaid so as to match the image, and can be alternately displayed at substantially the same position on the display so that each can be clearly observed. Further, in some cases, they may be displayed side by side in parallel.

The present invention is not limited to the above embodiment, but can be variously modified and applied within the scope of the claims.

[0050]

As described above, according to the display device of the present invention, when an observer looks at the display, it is reflected when the observed object is seen regardless of the position and orientation of the display. Image and
Since the effect of being able to see a substantially matched computer graphics model image is produced, the observer can always look at the display fixed at a certain fixed position / orientation as in the conventional case. It is possible to carry out the work while observing with a degree of freedom and utilizing the observation result without causing the drawback of having to do so.

Furthermore, it is possible to see a real image taken by using a movable camera whose installation position is not fixed together with a computer graphics model image on the display, and use various observation results to perform various operations smoothly. It becomes possible to do it.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a configuration diagram of a display device of the present invention.

FIG. 3 is a configuration diagram of a display position controller according to the first embodiment of this invention.

FIG. 4 is a diagram showing an outline of an observation system of an object to be observed according to the first embodiment of the present invention.

FIG. 5 is a configuration diagram of a display device according to a second embodiment of the present invention.

FIG. 6 is a configuration diagram of a display device according to a third embodiment of the present invention.

FIG. 7 is a diagram for explaining a conventional display device.

[Explanation of symbols]

 10 robot 20 6-DOF robot controller 30 display position controller 31 display control unit 32 graphics generation unit 33 coordinate conversion unit 34 observed object position / orientation storage unit 35 observer position / orientation storage unit 36 indicator position / orientation storage unit 40 6-axis positioning mechanism controller 50 Positioning mechanism 60 Display 61 Half mirror 62 Projector 66 Camera controller 70 Observed object 80 Observer position / orientation detector 80 85 CCD camera 90 Observer 95 Infrared LED 100 Observed object 200 Observer 300 Display means 400 Position / orientation detection means 500 Determination means

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Hikaru Umeno 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Inside Nippon Telegraph and Telephone Corporation

Claims (16)

[Claims] 1. A display method for presenting an image to an observer by displaying, detecting the respective positions and orientations of an observer and an observed object, and displaying using the detected respective positions and orientations. A display method characterized by determining the position, orientation, and size of an image. 2. The position / orientation of the observed object is identified, an observed object coordinate system is set substantially at the center of the observed object, and the observed position of the observed object is observed from reference coordinates. The display method according to claim 1, wherein the identification is performed based on the relationship of the object coordinate system. 3. The display method according to claim 2, wherein the three-dimensional computer graphics model of the observed object is displayed in the same relationship as the observed object coordinate system. 4. The relationship between the position / orientation of the display device is identified as the relationship of the display device coordinate system viewed from the reference coordinates, and the position / orientation of the observer is related to the observer line-of-sight coordinate system viewed from the reference coordinate system. The display method according to claim 2, which is identified as. 5. The display method according to claim 2, wherein the three-dimensional computer graphics model of the object to be observed is perspectively projected on the display device by a viewing transformation viewed from the observer's line-of-sight coordinate system. 6. The display method according to claim 1, wherein the computer graphics model of the observed object is displayed as a part of an image when the image is displayed. 7. The display method according to claim 1, wherein when the image is displayed, a real video image of the observed object is displayed as a part of the image. 8. The position of the observed object seen from the observer
The display method according to claim 2, wherein the position / orientation and size of the computer graphics model are determined and displayed so that at least one of the attitude and the size matches. 9. The position / orientation of an imager is detected, the observed object is imaged, and the imaged real image is used as a part of an image, and at least the position / orientation and size of the real image to be displayed. 7. The display method according to claim 6, wherein the position / orientation and size of the computer graphics model are determined so as to match one or more. 10. A display device for presenting an image to an observer, a display unit for displaying the image, and a position / orientation for detecting respective positions and orientations of the observer and the observed object. A display device comprising: a detection unit; and a determination unit that determines the position / posture and size of the image on the display unit by using the respective positions / postures detected by the position / posture detection unit. . 11. The display device according to claim 10, further comprising an image adding unit that uses a computer graphics model of the observed object as a part of the image. 12. The computer graphics model on the display unit, wherein the image adding unit matches at least one of a position / posture and a size of the observed object viewed from the observer. The display device according to claim 11, wherein the position / posture and size of the display device are determined. 13. The display device according to claim 10, further comprising an image pickup unit for picking up the observed object, and using an actual image picked up by the image pickup unit as a part of the image. 14. The display device according to claim 10, wherein the display means is a transflective type. 15. An image pickup means for picking up the object to be observed, a detecting means for detecting a position / orientation of the image pickup means, and a real image picked up by the image pickup means as part of the image, and Position of the actual image on the display means
The display device according to claim 11, further comprising computer graphics model determining means for determining the position / orientation and size of the computer graphics model on the display means so that at least one of the orientation and the size matches. 16. The position / orientation detecting means sets an observed object position / orientation detecting means for detecting the position / orientation of the observed object, and sets an observed object coordinate system substantially at the center of the observed object. ,
11. The display according to claim 10, further comprising an observed object identifying means for identifying the position / orientation of the observed object obtained by the observed object position / orientation detecting means as a relationship in the observed object coordinate system viewed from the reference coordinates. apparatus.