JP2006003772A - Stereoscopic image pickup display system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereoscopic image pickup display system capable of displaying a stereoscopic image viewed from an optional viewing point by reconstructing three-dimensional shape based on a parallactic image picked up by an observation device capable of stereoscopic observation. <P>SOLUTION: The stereoscopic image pickup display system has a three-dimensional reconstructing part 31 equipped with an image forming optical system 1 forming the image of an object O on two imaging devices 4<SB>1</SB>and 4<SB>2</SB>and three-dimensionally reconstructing the shape of the object by searching a corresponding point from the parallactic image picked up by the imaging devices 4<SB>1</SB>and 4<SB>2</SB>. The display system is equipped with a pair of display elements 11<SB>1</SB>and 11<SB>2</SB>for displaying the parallactic image, a pair of projection means for projecting the displayed parallactic image on the same plane through an exit pupil, and a display panel 15 arranged on or near the plane and having an image forming means and a diffusing means, and is constituted so that the parallactic image obtained by viewing the object three-dimensionally reconstructed by the three-dimensional reconstructing part 31 with both eyes at the position of the optional viewing point can be displayed on the display elements 11<SB>1</SB>and 11<SB>2</SB>. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a stereoscopic imaging display system, and in particular, stereoscopic imaging display having a function of reconstructing a three-dimensional shape and displaying a stereoscopic image viewed from an arbitrary viewpoint simultaneously with stereoscopic observation based on a parallax image from an observation device. It is about the system.

Conventionally, there are binocular electronic image microscopes disclosed in Patent Literature 1, Patent Literature 2, and Patent Literature 3. These binocular stereomicroscopes can capture a three-dimensional image, but cannot measure the shape, and observation with both eyes requires special observation such as wearing shutter glasses. Patent Document 4 discloses a method of measuring a shape from two electronic images, although only the shape, particularly the height.
Japanese Patent No. 3,209,543 JP-A-7-222754 JP-A-8-36134 Japanese Patent No. 2,526,543 Japanese Patent No. 3,401,167 JP 2001-59949 A Takashi Matsuyama et al. “Computer Vision: Technical Review and Future Prospects” (issued on June 15, 1998 by New Technology Communications Inc.) "Technical CG Standard Textbook" supervised by the Technical CG Standard Textbook Editorial Committee (issued by the Association for Promotion of Image Information Education on March 1, 1999) Tomahiro Imama “CG Basic Seminar 3DCG / Video Media Becomes Stronger” (November 25, 2002, published by Ohm Corporation) Xugang et al., “3D Vision” (published April 20, 1998, Kyoritsu Publishing Co., Ltd.)

  However, conventionally, there has been no observation apparatus that can acquire a three-dimensional shape while observing a stereoscopic image in a wide viewing zone without requiring glasses or the like.

  The present invention has been made in view of such a situation in the prior art, and an object of the present invention is to perform stereoscopic observation from a parallax image captured by an observation apparatus capable of stereoscopic observation and to perform three-dimensional observation based on the parallax image. It is an object of the present invention to provide a stereoscopic imaging display system that can reconstruct a shape and display a stereoscopic image viewed from an arbitrary viewpoint.

The stereoscopic imaging display system of the present invention that achieves the above object forms at least two optical axes having parallax with respect to an observation object, and forms an image of the observation object on at least two corresponding imaging elements. A three-dimensional reconstruction unit that includes an imaging optical system having a positive refractive power as a whole, and that three-dimensionally reconstructs the shape of an object by searching for corresponding points from parallax images captured by the at least two image sensors. In a stereoscopic imaging display system having
A pair of display elements for displaying left and right parallax images, a pair of projection means for projecting parallax images displayed on the pair of display elements on the same plane through an exit pupil, and the plane or the vicinity thereof A display panel, and an imaging means for forming an image of an exit pupil of each projection means at an observation pupil position and a diffusion action, and an image of the exit pupil of each projection means at an observation pupil position A display panel having diffusing means for expanding,
The parallax image obtained by viewing the object three-dimensionally reconstructed by the three-dimensional reconstructing unit with both eyes at an arbitrary viewpoint position can be displayed on the pair of display elements.

  In this case, it is desirable that the imaging means is constituted by a Fresnel lens or a Fresnel mirror, and the optical axis of the Fresnel lens or Fresnel mirror is arranged eccentrically with respect to the center of the display panel.

  In addition, it is desirable to include a recording unit that records a parallax image captured by at least two image sensors or object shape data that is three-dimensionally reconstructed by a three-dimensional reconstruction unit.

Further, the difference Δφ between the azimuth angle when the three-dimensional reconstructed stereoscopic image displayed on the display panel is projected on the horizontal plane on which the observation object is placed and the azimuth angle of the observation object is:
−10 ° <Δφ <10 ° (1)
It is desirable to satisfy.

Further, the difference between the look-down angle θ determined by the direction connecting the center of the display panel and the center of the observation object and the vertical line, and the look-down angle θ of the line of sight with respect to the three-dimensional reconstructed stereoscopic image displayed on the display panel. Δθ is
−30 ° <Δθ <30 ° (2)
It is desirable to satisfy.

In addition, the look-down angle θ with respect to the vertical line of sight with respect to the three-dimensional reconstructed stereoscopic image displayed on the display panel is
0 ° <θ <60 ° (3)
It is desirable to satisfy.

  In addition, it is desirable to include means for changing the three-dimensionally reconstructed stereoscopic image displayed on the display panel to a stereoscopic image viewed with both eyes at an arbitrary viewpoint position.

  Further, the display panel is configured to be able to change the position with respect to the observation object, and the viewpoint position of the three-dimensional reconstructed stereoscopic image displayed on the display panel is determined according to the position of the display panel with respect to the observation object. It is desirable to make changes.

  In addition, a plurality of display panels are provided, and a three-dimensional reconstructed stereoscopic image displayed on each display panel is viewed with both eyes at an arbitrary viewpoint position according to the position of each display panel relative to the observation object. It is desirable to change to a stereoscopic image.

Furthermore, in the above, the inward angle α formed by the observation object position between at least two optical axes having parallax with respect to the observation object of the imaging optical system is
α <12 ° (4)
It is desirable to satisfy

  In the present invention, the three-dimensional object shape of the observation object is reconstructed based on the parallax image captured by a device capable of capturing a parallax image, and viewed from an arbitrary viewpoint position from the reconstructed three-dimensional object shape. Since a stereoscopic image is reproduced on a display panel that can be easily observed in a relatively wide observation range without using glasses or the like having a shutter function, an observation object reconstructed from a parallax image can be in any direction. 3D can be displayed as a 3D image viewed from the above.

  The stereoscopic imaging display system of the present invention will be described below based on examples.

A binocular microscope is a type in which the left and right optical paths are parallel with a single objective lens, and a type having two variable power optical systems on the left and right (FIG. 1 (a)) and a common variable power optical system on the left and right. There is an example of the type (FIG. 1B) and the like. A type (FIG. 2) in which the left and right optical paths are configured in a V shape is also known. In any case, at least two imaging elements are arranged for at least two imaging optical systems, and a two-dimensional image having at least two parallaxes is obtained from each imaging element. 1 and 2, the binocular microscope itself is denoted by reference numeral 10, the objective lens is denoted by reference numeral 1, the imaging lens is denoted by 2 ₁ , 2 ₂ , and the variable magnification optical system is denoted by 3, 3 ₁ , 3 ₂ . The image sensors are denoted by 4 ₁ and 4 ₂ , respectively. The subscripts “ ₁ ” and “ ₂ ” of these symbols are for distinguishing the arrangement position (optical path) when there are a plurality of these elements. In the case of two left and right optical paths, the distinction between the left and right elements is made. means. 1 and 2, the observation object is indicated by the symbol O. In the configuration of FIG. 2, the left and right variable magnification optical systems 3 ₁ and 3 ₂ also serve as left and right objective lenses.

Video signals from at least two image sensors 4 ₁ and 4 _{2 of} such a binocular microscope 10 are displayed on two two-dimensional display elements 11 ₁ and 11 ₂ as shown in FIG. The parallax images can be simultaneously projected onto the display panel 15 by the projection optical systems 12 ₁ and 12 ₂ , respectively, so that the left and right eyeballs (pupils) EL and ER can be seen separately, and the object O can be stereoscopically observed. To.

  Here, such a binocular stereoscopic observation device 30 used in the present invention will be described.

4 displays video signals from at least two imaging elements 4 ₁ and 4 ₂ of the parallax image imaging apparatus (binocular microscope) 10 as shown in FIGS. 1 and 2 on two two-dimensional display elements 11 ₁ and 11 ₂ . Then, the configuration of the binocular stereoscopic observation device 30 that enables the two-dimensional parallax images to be simultaneously projected onto the display panel 15 to enable the stereoscopic display of the object O is shown. 4A is a schematic configuration diagram of the transmission type binocular stereoscopic observation device, and FIG. 4B is a schematic configuration diagram of the reflection type binocular stereoscopic observation device. In FIG. 4B, only the configuration for the right eye is shown for convenience, and the configuration for the left eye is omitted.

The binocular stereoscopic observation device 30 shown in FIGS. 4A and 4B is in close contact with the display elements 11 ₁ and 11 ₂ , the projection optical systems 12 ₁ and 12 ₂ , the imaging optical system 13, and the observation side. Alternatively, it is configured to include a diffusing plate 14 disposed slightly apart. The imaging optical system 13 and the diffusing plate 14 constitute a display panel 15.

The projection optical systems 12 ₁ and 12 ₂ project the parallax images displayed on the display elements 11 ₁ and 11 ₂ on the viewer side. At this time, the optical system is configured so that the projected image is projected onto the same display surface of the display panel 15. The imaging optical system 13 is disposed in the vicinity of the same display surface. Then, the exit pupils 16 ₁ and 16 ₂ of the projection optical systems 12 ₁ and 12 ₂ are projected on the viewer side. The observer can observe the images displayed on the display elements 11 ₁ and 11 ₂ by matching the pupils EL and ER of the left and right eyes with the position of the projected exit pupil image. The diffusion plate 14 has an action of enlarging the observation pupil. Further, the imaging optical system 13 and the diffusion plate 14 are disposed at the display surface position.

The display surface position is an image formation position of the image of the display elements 11 ₁ and 11 ₂ projected by the projection optical systems 12 ₁ and 12 ₂ . As the imaging optical system 13 disposed at this imaging position, in the transmission type binocular stereoscopic observation device (FIG. 4A), the Fresnel lens (convex lens) is replaced by the reflection type binocular stereoscopic observation device (FIG. In 4 (b), a Fresnel mirror (concave mirror) is provided. The diffusing plate 14 includes a directional HOE scattering film and a directional diffusing surface formed on the incident surface of the Fresnel mirror or the exit surface of the Fresnel lens.

The Fresnel mirror and the Fresnel lens are designed to form images of _two exit pupils 16 ₁ and 16 _{2 on} the viewer side, respectively. Since these Fresnel surfaces are arranged at the display surface position (or in the vicinity thereof), the image quality of the projected image can be observed with high resolution without deterioration. Moreover, unlike a concave mirror, it is arrange | positioned at flat form.

  FIG. 5A is an explanatory diagram illustrating the principle by which the exit pupil image (observation pupil) is magnified in the binocular stereoscopic observation device 30. In FIG. 5A, the configuration of a transmission type binocular stereoscopic observation apparatus is used.

A diffusion plate 14 is disposed together with the imaging optical system 13 at or near the planar display surface position of the display panel 15. In FIG. 5A, the imaging optical system 13 has a diameter Φ ₀ of the exit pupils 16 ₁ , 16 ₂ of the left and right projection optical systems 12 ₁ , 12 ₂ having a size of Φ ₀ ′ and a predetermined value on the viewer side. It has the effect | action which forms an image in this position. This predetermined position is where the observer's eyeball (pupil) pupils EL and ER are located. Here, the diffusing plate 14 has a pupil image of the exit pupils 16 ₁ and 16 ₂ (images of the left and right projection optical systems 12 ₁ and 12 ₂₎ that should be imaged with a size of Φ ₀ ′. 16 ₁ and 16 ₂ ) are enlarged to the size of Φ ₁ . Note that the positions and diameters of the left and right exit pupil images magnified by the diffusion plate 14 are set so as not to overlap each other. The diffusing action of the diffusing plate 14 acts only once because it passes through the diffusing plate 14 provided at the display surface position in the transmissive binocular stereoscopic observation device, so that it acts only once. (Not shown) acts twice because it passes twice through the diffusion optical system provided at the display surface position.

FIGS. 5B and 5C show the brightness distribution in the left and right observation pupils. The type of distribution in the pupil may be uniform as shown in FIG. 5B or close to a Gaussian distribution as shown in FIG. In any case, it is important to make the light intensity of the overlapping portion of the left and right pupils sufficiently small so that there is substantially no overlapping. In particular, as shown in FIGS. 5B and 5C, it is desirable that the light intensity of the overlapping portion of the left and right pupils is less than 1/10. Therefore, the images of the left and right projection devices 20 ₁ and 20 ₂ can be presented to the observer's left and right pupils EL and ER in an independent state, and stereoscopic observation without glasses is possible.

FIG. 6 is a diagram showing an example of the binocular stereoscopic observation device 30 used in the present invention. 6A is a schematic configuration diagram viewed from above, and FIG. 6B is a side view of FIG. 6A. The binocular stereoscopic observation device 30 of this example is configured as a transmission type. An imaging optical system 13 is disposed at the display surface position of the display panel 15. The imaging optical system 13, the exit pupil 16 _1, 16 ₂ of the projection optical system 12 _1, 12 _2, projected to the observer side. By making the observer's eyeballs (pupils) EL and ER coincide with the projected position, the observer can observe the image.

Here, as the imaging optical system 13, a Fresnel lens having a Fresnel surface 13a facing the viewer side is used. In the vicinity of the Fresnel lens 13, a diffusion plate 14 for magnifying the pupil is disposed, and these constitute a transmissive display panel 15. In this example, the Fresnel lens surface 13a is disposed at the imaging position of the projected image projected by the projection optical systems 12 ₁ and 12 ₂ . For this reason, there is no deterioration in image quality due to the Fresnel lens surface 13a. The diffusion surface 14 a of the diffusion plate 14 is provided on the Fresnel lens surface 13 a side of the Fresnel lens 13. The diffusing surface 14a is disposed close to the Fresnel lens surface 13a, and blur is reduced to suppress image quality deterioration.

  In this example, the Fresnel lens surface 13a of the transmissive display panel 15 is an eccentric Fresnel lens surface, and the optical axis of the Fresnel lens surface 13a is positioned below the center as shown in FIG. 6B. is doing. The Fresnel lens surface 13a has a positive refractive power.

  If the transmissive display panel 15 is configured with a decentered optical system as in this example, the display panel surface itself does not have to be thick. Therefore, it is possible to dispose the display panel 15 without interfering with it. Further, as in this example, it is preferable to dispose the diffusing surface 14a and the Fresnel surface 13a as close as possible to the display surface position because image quality deterioration is less.

  FIG. 7 is an explanatory diagram showing another example of the binocular stereoscopic observation device 30 used in the present invention. FIG. 7A is a perspective view, and FIG. 7B is a side view.

The binocular stereoscopic observation device 30 of this example is configured as a reflection type. The display panel 15 includes an imaging optical system 13 and a diffusion plate 14 for pupil enlargement. The imaging optical system 13 is specifically a Fresnel mirror 14. The imaging optical system 13, the exit pupil 16 _1, 16 ₂ of the projection optical system 12 _1, 12 _2, projected to the observer side. The observer can observe the image by matching the pupils EL and ER to the projection position.

By the way, in the case of a reflection type binocular stereoscopic observation device, it is necessary to arrange each optical member so that the projection optical systems 12 ₁ and 12 ₂ do not interfere with the face of the observer. In addition, it is easier for an observer to observe the display panel when viewed from the front. Therefore, in this example, an angle Θ is provided between the incident optical axis of the projection light incident on the center of the display panel 15 and the outgoing optical axis of the light beam emitted from the center of the display panel 15. Further, the optical axis of the Fresnel mirror 13 is decentered in the vertical direction (upward in FIG. 7) with respect to the center of the display panel 15.

FIG. 8 is a side view showing a more specific example of FIG. In the example of FIG. 8, a spherical lens system is used for the projection optical system 12 ₂ (12 ₁ ), and the display element 11 ₂ (11 ₁ ) is arranged at a position decentered from the optical axis of the lens. By doing so, the projection optical system 12 ₂ (12 ₁ ) and the observer's face are prevented from interfering with each other. The display panel 15 is arranged perpendicular to the observer's eyes and the projection optical system 12 ₂ (12 ₁ ). An aspheric Fresnel mirror is used on the display panel surface.

  As the diffuser plate 14 constituting the display panel 15 of the binocular stereoscopic observation device 30 used in the present invention, a directional HOE scattering film as described above can be used. As the HOE scattering film, a transmission hologram can be adopted, and for this purpose, a volume hologram (volume hologram) can be used. Further, the diffusing plate 14 can be constituted by a directional diffusing surface formed on the entrance surface of the Fresnel mirror or the exit surface of the Fresnel lens, for example, by spraying glass beads on a randomly polished brass surface, A minute concave arrangement surface having a mirror surface and a random arrangement can be used as a mold and transferred to a Fresnel concave mirror made of plastic.

Thus, in the binocular stereoscopic observation device 30 used in the present invention, the two exit pupils 16 ₁ and 16 _{2 included} in the projection optical systems 12 ₁ and 12 ₂ that project the left and right parallax images are connected to the display panel 15. The image optical system 13 and the diffusing plate 14 cause the image to be projected in the vicinity of the left and right eyeballs (pupils) EL and ER of the observer, and the observer can project from the projected _two exit pupils 16 ₁ and 16 ₂ to the projection optical system 12. Among the two images having a parallax projected by ₁ and 12 ₂ , an image for the right eye can be observed with the right eye ER, and an image for the left eye can be observed with the left eye EL. A stereoscopic image can be easily observed in a relatively wide observation range without wearing glasses or the like on the face.

Furthermore, by arranging the optical axes of the Fresnel lens and the Fresnel mirror of the imaging optical system 13 to be decentered, the projection devices 20 ₁ and 20 ₂ can be projected from an oblique direction with respect to the display surface. It is possible to avoid flare in the image due to the reflected light on the display surface.

By the way, a corresponding point between images having parallax is searched by a well-known stereo measurement method from images having at least two parallaxes captured by the imaging elements 4 ₁ and 4 ₂ of the binocular microscope 10, and the three-dimensional object O is displayed. The shape can be acquired. This measurement method is described in detail, for example, on pages 80 to 96 ("Epipolar geometry in computer vision") of Non-Patent Document 1.

FIG. 9 shows a system configuration diagram of one embodiment of the stereoscopic imaging display system of the present invention provided with a three-dimensional reconstruction unit that acquires the stereoscopic shape of the object O. Also in this case, as in the configuration of FIG. 3, the video signals from the at least two imaging elements 4 ₁ , 4 ₂ of the binocular microscope 10 are sent to the binocular stereoscopic observation device 30 as described above, and the stereoscopic of the object O Enables observation.

Further, video signals from at least two image sensors 4 ₁ , 4 _{2 of} the binocular microscope 10 are input to a three-dimensional reconstruction unit 31 that performs three-dimensional reconstruction of the shape of the object O, and at least two images are included therein. The corresponding points are searched for, and three-dimensional coordinate data is created based on the corresponding points. Based on this, a surface object is generated, the texture is projected onto the surface of the surface model, and the three-dimensional reconstruction is completed (for the surface model, texture projection, etc., pages 55-86 and 165 of Non-Patent Document 2). To page 171).

  When the surface model constructed in this way is converted into a format such as VRML, the three-dimensional model reconstructed three-dimensionally by an Internet browser or the like can be rotated or enlarged to be easily observed ( (For the format such as VRML, see page 236 of Non-Patent Document 3).

  Therefore, the binocular stereoscopic observation device 30 as described above is connected to the three-dimensional reconstruction unit 31, and the stereoscopic model reconstructed three-dimensionally by the three-dimensional reconstruction unit 31 is viewed with both eyes from a desired position (viewpoint). It can be displayed as a left and right two-dimensional image having the observed parallax.

Furthermore, video signals from at least two image sensors 4 ₁ , 4 _{2 of} the binocular microscope 10 are connected to recording devices 32 ₁ , 32 ₂ , and two two-dimensional images having parallax are directly recorded on the recording devices 32 ₁ , 32 ₁ , 32 ₂ is recorded.

  The three-dimensional reconstruction unit 31 is connected to a reconstruction model recording device 33 that records the three-dimensionally reconstructed three-dimensional model.

Further, as shown in FIG. 10A, recording devices 32 ₁ and 32 ₂ for recording two two-dimensional images input as video signals from at least two imaging elements 4 ₁ and 4 ₂ of the binocular microscope 10 are provided. A two-dimensional image reproducing device 34 ₁ connected directly to the binocular microscope 10 to record a two-dimensional image for the time being and connected to a separate three-dimensional reconstruction unit 31 as shown in FIG. 34 ₂ is input to the three-dimensional reconstruction unit 31 by playing the image with at least two parallax is recorded in, where reconstructed 3-dimensional solid model, further connected to the three-dimensional reconstruction unit 31 It is also possible to display the stereoscopic model on the binocular stereoscopic observation device 30 as a left and right two-dimensional image having a parallax observed from a desired position (viewpoint). Also in this case, the reconstructed model recording device 33 that records the three-dimensionally reconstructed stereo model may be connected to the three-dimensional reconstructing unit 31. Alternatively, the binocular microscope 10 may be directly connected to the binocular microscope 10 to enable direct stereoscopic observation of the object O.

  Normally, an observer recognizes an image having two parallaxes with respect to an object as a stereoscopic image in the brain, and constructs a stereoscopic shape in the brain. However, this case is not suitable for observing a stereoscopic image by a plurality of people. In addition, there is no easy method for recording a stereoscopic image, and a special three-dimensional display device has to be used. In the present invention, it is possible to perform stereoscopic observation in real time without such a three-dimensional display device, or to reconstruct a stereoscopic model from recorded parallax images so that a stereoscopic image viewed from an arbitrary viewpoint can be obtained. It is also possible to observe. Furthermore, it is possible to save and record a three-dimensionally reconstructed surface model or a three-dimensional model itself.

FIG. 11 shows a perspective view of one embodiment of the binocular microscope apparatus constituting the stereoscopic imaging display system of the present invention. This binocular microscope apparatus includes a sample mounting table 41 on which an observation object O is mounted, a support bar 42 standing on the rear end of the sample mounting table 41, and a vertical adjustment screw 44 along the support bar 42. A movable body 43 that can be moved and attached to the lower part of the body 43, and a barrel 45 that mainly stores the objective lens 1 of the binocular microscope optical system, and an upper part of the body 43 to be three-dimensionally reconstructed. The transmission type binocular stereoscopic observation device 30 that displays a stereoscopic image of the object O directly on the basis of the obtained shape data or directly on the basis of video signals from the image sensors 4 ₁ and 4 ₂ . The mirror body 43 is provided with a magnification adjusting screw 47 for adjusting the magnification varying optical systems 3 ₁ and 3 ₂ to change the magnification. In this example, an optical system similar to that shown in FIG. 1A is used as the binocular microscope optical system, and the variable magnification optical system has an optical path of approximately 90 between 3 ₁ , 3 ₂ and the image sensors 4 ₁ , 4 _2. Deflection mirrors 5 ₁ and 5 _{2 to} be bent are arranged and configured.

The transmission binocular stereoscopic observation device 30 includes two-dimensional display elements 11 ₁ and 11 ₂ that display left and right parallax images, corresponding projection optical systems 12 ₁ and 12 ₂ that are not shown, and a left-right common transmission type. And a display panel 15.

The three-dimensional reconstruction unit 31 shown in FIG. 9 and the parallax image recording devices 32 ₁ and 32 ₂ are connected to such a configuration.

In this embodiment, parallax images captured by the left and right imaging elements 4 ₁ and 4 ₂ of the binocular microscope optical system are input to the three-dimensional reconstruction unit 31. When the input parallax images are recorded as they are as two parallax images (moving images), they are output only to the recording devices 32 ₁ and 32 ₂ . When performing stereoscopic display, the two parallax images are output as they are to the display elements 11 ₁ and 11 ₂ , and projected to the vicinity of the Fresnel lens 13 and the diffusion plate 14 of the transmissive display panel 15 by the projection optical systems 12 ₁ and 12 _2. Is done. Accordingly, the observer can observe the parallax image with the left and right eyes, and stereoscopic observation is possible.

  The two parallax images input to the three-dimensional reconstruction unit 31 are aligned between the two images, a corresponding point search is performed, the parallax amount is converted into three-dimensional coordinates, and converted into a surface model. Is done. Finally, the surface image of the two-dimensional image is pasted on the surface model and converted to VRML.

Left and right two-dimensional images having parallax obtained by observing the three-dimensional reconstructed three-dimensional model from a desired position (viewpoint) are output to the display elements 11 ₁ and 11 ₂ of the transmission binocular three-dimensional observation apparatus 30, and three-dimensional This makes it possible to perform stereoscopic observation of the reconstructed stereoscopic model viewed from a desired position.

By the way, in the stereoscopic imaging display system of the present invention, since the two-dimensional electronic images are acquired by the imaging elements 4 ₁ and 4 ₂ , respectively, it is possible to electronically invert the video. Therefore, there is no need to create an upright image with the optical system, and there is an advantage that the number of optical components can be reduced. However, as illustrated in FIG. 11, in particular, when the binocular stereoscopic observation device 30 is integrated, the direction in which the observer observes the observation object O and the display of the stereoscopic image displayed on the binocular stereoscopic observation device 30 are displayed. Matching the direction is particularly important in applications where the object O is operated (for example, surgery).

  In the state as shown in FIG. 12, that is, when the observation object O is a three-dimensional character “three-dimensional”, the stereoscopic image I displayed on the display panel 15 of the binocular stereoscopic observation device 30 is shown in FIG. As shown, when projected onto the sample mounting table 41, it is necessary to display so as to have the same azimuth angle φ as the observation object O. Here, the X and Y coordinates are taken on the surface of the sample mounting table 41, the Z axis is taken in the imaging direction (vertical direction) by the binocular microscope optical system, and the azimuth angle φ is defined as an angle measured from the Y axis. The

  In particular, in a small binocular microscope apparatus in which the binocular stereoscopic observation apparatus 30 is integrated, the observer rotates the direction of the mirror 43 or the lens barrel 45 to change the direction of the stereoscopic image I displayed as the observation object O. Matching is a very difficult task. For this reason, it is very important that the orientation of the stereoscopic image I displayed on the binocular stereoscopic observation device 30 side is matched with the orientation of the observation object O in advance.

  By doing so, when an operation is performed on the observation object O, the actual operation direction and the operation direction on the display panel 15 coincide with each other, so that the operation can be performed intuitively. Note that the viewpoint conversion of the three-dimensionally reconstructed solid model is performed according to the method described on pages 86 to 90 (“hand / eye calibration”) of Non-Patent Document 4. That is, it is possible by converting the coordinate system of the imaging optical axis in the direction of the display panel 15.

  Explaining this point a little further, the viewpoint for the three-dimensional reconstructed stereoscopic model displayed on the display panel 15 of the binocular stereoscopic observation device 30 (in this case, it is considered as the center of the eyes EL and ER of the left and right eyes). It is important that the observer matches the direction of observing the observation object O through the display panel 15. It is important to align the viewpoint direction, which is the observation direction of the three-dimensionally reconstructed surface model, with the direction connecting the center of the display panel 15 and the center of the observation object O. In the state as shown in FIG. 12, the reconstructed stereoscopic image is displayed as shown in FIG. 13, and the direction of the line of sight (intermediate line of the left and right lines of sight) with respect to the three-dimensional reconstructed solid model to be displayed is As shown in FIG. 13, it is necessary to select and display such that the azimuth angle φ and the look-down angle θ are the same as the direction of the line of sight (intermediate line of the left and right lines of sight) with respect to the observation object O.

In this case, the difference Δφ between the azimuth angle φ when the stereoscopic image I displayed on the display panel 15 is projected on the sample mounting table 41 and the azimuth angle φ of the observation object O is:
−10 ° <Δφ <10 ° (1)
It is preferable to satisfy the following condition.

Also, the look-down angle θ determined by the direction connecting the center of the display panel 15 and the center of the observation object O, and the look-down angle of the line of sight (intermediate line of the left and right lines of sight) for the three-dimensional reconstructed three-dimensional model displayed on the display panel 15 The difference Δθ from θ is
−30 ° <Δθ <30 ° (2)
It is preferable to satisfy the following condition.

Also, the look-down angle θ (FIG. 13) defined by the angle measured from the Z axis of the line of sight (intermediate line of the left and right lines of sight) for the three-dimensional reconstructed solid model is
0 ° <θ <60 ° (3)
It is desirable to satisfy the following conditions.

  The difference Δθ between the azimuth angle difference Δφ and the look-down angle θ affects the sense of direction when working on the XY plane, which is the mounting table 41, and if the upper and lower limits of ± 10 ° and ± 30 ° are exceeded, A shift occurs between the direction in which the target is set with respect to the object O and the direction in which the object O is to be moved, and operations such as gripping the object O cannot be performed.

  Further, the upper limit of 60 ° of the looking-down angle θ must be estimated by interpolation because a three-dimensional image cannot be obtained for a portion that is shadowed by the object O even if three-dimensional reconstruction is performed. However, when the upper limit is exceeded, the interpolated portion is mainly displayed, so that the three-dimensional reproducibility is significantly reduced. If the lower limit of 0 ° is exceeded, there is no difference from observing a two-dimensional image, and a stereoscopic effect cannot be obtained.

As for the azimuth angle difference Δφ,
−45 ° <Δφ <45 ° (1-1)
Moreover, the difference Δθ in the look-down angle θ is worst,
−45 ° <Δθ <45 ° (2-1)
It is preferable to satisfy the following condition.

Also, for the looking down angle θ,
10 ° <θ <45 ° (3-1)
It is more desirable to satisfy the following conditions.

  By the way, it is desirable that the viewpoint position with respect to the three-dimensionally reconstructed stereoscopic model displayed on the display panel 15 of the binocular stereoscopic observation device 30 can be changed once displayed. For this purpose, as shown in FIG. 14, a control unit 48 such as a joystick for freely changing the observation position (viewpoint) for the three-dimensionally reconstructed three-dimensional model is connected to the three-dimensional reconstruction unit 31 to While observing the stereoscopic image I displayed on the display panel 15 of the stereoscopic observation device 30, the viewpoint can be freely changed, so that the three-dimensional shape of the observation object O can be grasped more easily while viewing the display panel 15. It becomes possible to do.

  In addition, the binocular stereoscopic observation device 30 that displays the three-dimensionally reconstructed stereoscopic model can change any angle in the azimuth angle φ direction and the look-down angle θ direction. In that case, a means for detecting the azimuth angle φ and the look-down angle θ of the binocular stereoscopic observation device 30 is provided, and the viewpoint for the three-dimensional reconstructed three-dimensional model is converted with the detected azimuth angle φ and the look-down angle θ. It is important to make it. FIG. 15 shows the binocular stereoscopic observation device 30 that can be rotated by an arbitrary angle around the Z axis in the imaging direction. The binocular stereoscopic observation device 30 ′, the display panel 15 ′, and the displayed stereoscopic image after the change are shown. The image is indicated by I ′. Even when observed at the position after the change, the positional relationship between the displayed stereoscopic image I and the object O observed with the naked eye is substantially the same, and it is easy to operate, particularly in applications where the object O is operated.

  Further, for example, as shown in FIG. 16, when a plurality of binocular stereoscopic observation devices 30 a and 30 b that display a stereoscopic image of a three-dimensionally reconstructed stereoscopic model are provided, imaging with respect to the observation object O is performed. Recognizing the relationship between the coordinates of the display directions of the display panels 15a and 15b of the binocular stereoscopic observation devices 30a and 30b with respect to the coordinates of the system, and converting the viewpoint of the three-dimensionally reconstructed stereoscopic model according to the relationship. Is desirable. That is, as shown in FIG. 17, when a plurality of binocular stereoscopic observation devices 30a and 30b are provided, the stereoscopic image Ia of the three-dimensional reconstructed stereoscopic model displayed on the respective display panels 15a and 15b. , Ib combines the azimuth angle φ and the look-down angle θ formed by the binocular stereoscopic observation devices 30a and 30b with the object O together with the azimuth angle φ and the look-down angle θ in the line-of-sight direction with respect to the three-dimensionally reconstructed solid model, It is important to display different stereoscopic images Ia and Ib on the respective display panels 15a and 15b.

  For example, as shown in FIGS. 18A and 18B, when there are a plurality of observers Ma and Mb, the direction in which the object O is operated differs from each observer Ma and Mb. In such a case, if the three-dimensional image of the three-dimensionally reconstructed three-dimensional model viewed from the viewpoint according to each observation direction is not displayed, the observers Ma and Mb cannot work together. . Further, when the observers Ma and Mb move and the binocular stereoscopic observation devices 30a and 30b are rotated with respect to the object O, for example, means for detecting the rotation angle of the binocular stereoscopic observation devices 30a and 30b is provided. Based on information such as the azimuth angle φ from the rotation angle detection means, the azimuth that matches the observation direction of the observers Ma and Mb who are viewing the display panels 15a and 15b of the binocular stereoscopic observation devices 30a and 30b It is more preferable to display a stereoscopic image such as the angle φ.

  By the way, when a parallax image at an arbitrary viewpoint is formed from the stereoscopic model three-dimensionally reconstructed in the binocular stereoscopic observation device 30, the left and right parallax images are the same as the inward angle of the parallax image displayed on the display panel 15. It is important to display the image as a parallax image having an inward angle. If the inward angle of the imaging unit and the inward angle of the observation unit are different, the stereoscopic scale and the horizontal scale are different, and stereoscopic distortion occurs. In order to avoid this, first, a three-dimensional model is reconstructed (three-dimensional data, VRML, polygon data, etc.), and again from this data, a two-dimensional parallax image having an inward angle corresponding to the inward angle of the observation unit is obtained. By generating two images and displaying them on the binocular stereoscopic observation device 30, it is possible to observe an observation image without stereoscopic distortion.

By the way, when the inward angle that is the angle formed by the object O position of the optical axis of the left and right optical paths in the binocular microscope optical system in the stereoscopic imaging display system of the present invention is α (see FIGS. 1 and 2).
α <12 ° (4)
It is preferable to satisfy the following conditional expression:

  Exceeding the upper limit of 12 ° of conditional expression (4), the captured left and right images have too much parallax, and depending on the shape, a portion “occlusion” that cannot be seen by the object itself occurs. When three-dimensional reconstruction is performed, there are many portions where the corresponding point search cannot be performed, and the three-dimensional shape cannot be sufficiently reconstructed.

More preferably,
1 ° <α <8 ° (4-1)
It is more preferable that the following conditional expression is satisfied.

In the stereoscopic imaging display system of the present invention, a stereoscopic image generated from a stereoscopic model reconstructed three-dimensionally in a binocular stereoscopic observation device or a stereoscopic image based on two parallax images captured by the imaging elements 4 ₁ and 4 _2. When displaying an image, a parallax image before any processing is stored in the object O, and a stereoscopic image based on the parallax image and a stereoscopic image based on the parallax image after processing can be displayed in multiple layers (superimposed display). By doing so, for example, the machining process of the object can be accurately understood.

  Note that, in the stereoscopic imaging display system of the present invention, instead of the binocular stereoscopic observation device 30, observation is possible through a three-dimensional display unit that processes two parallax images so as to distribute them to the left and right eyes of the observer. be able to. As such a three-dimensional display unit, a parallax barrier / lenticular system proposed in Patent Document 5, a micropole system proposed in Patent Document 6 and the like, a surface sequential display, and a liquid crystal shutter are used. You may use the stereoscopic observation apparatus of the system which combined spectacles.

  As described above, the stereoscopic imaging display system of the present invention has been described based on the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made.

It is a figure for demonstrating two types of the binocular microscope which can be used for the three-dimensional imaging display system of this invention. It is a figure for demonstrating another type of binocular microscope which can be used for the stereoscopic imaging display system of this invention. 1 is a system configuration diagram of a basic configuration of a stereoscopic imaging display system of the present invention. It is a figure for demonstrating the structure of the binocular stereoscopic observation apparatus used in this invention. It is explanatory drawing which shows the principle in which an exit pupil image (pupil for observation) is expanded in a binocular stereoscopic observation device. It is explanatory drawing which shows an example of the binocular stereoscopic observation apparatus used in this invention. It is explanatory drawing which shows the other example of the binocular stereoscopic observation apparatus used in this invention. It is a side view which shows the example which actualized the example of FIG. 7 more. 1 is a system configuration diagram of one embodiment of a stereoscopic imaging display system of the present invention. FIG. It is a system block diagram of another Example of the three-dimensional imaging display system of this invention. It is a perspective view of one Example of the binocular microscope apparatus which comprises the three-dimensional imaging display system of this invention. It is a figure which shows an example of the arrangement | positioning relationship of the stereoscopic image displayed on the display surface of the observation object and the display apparatus. It is a figure which shows the azimuth of an observation object, the azimuth of a three-dimensional image displayed on the display surface of a display apparatus, and a look-down angle. It is a conceptual diagram of the binocular microscope apparatus similar to FIG. 11 provided with the control part which changes an observation position (viewpoint). It is a perspective view of an example of a binocular microscope device in which the binocular stereoscopic observation device can rotate at an arbitrary angle around the Z axis in the imaging direction. It is a figure for demonstrating the azimuth | direction angle and look-down angle | corner of the gaze direction of the stereo image which should be displayed on each binocular stereoscopic observation apparatus in case the some binocular stereoscopic observation apparatus is provided. It is a figure for demonstrating that the azimuth angle and the look-down angle of the stereo image displayed on the display panel of each binocular stereoscopic observation device in FIG. 16 should be matched with the azimuth angle and the look-down angle with respect to the observation object. It is a figure for demonstrating that a three-dimensional image should be displayed from the viewpoint match | combined with the observation direction of each observer, when the several binocular stereoscopic observation apparatus is provided.

Explanation of symbols

O ... Observation object EL, ER ... Eyeball (pupil)
I, I ', Ia, Ib ... stereoscopic image Ma, Mb ... observer 1 ... objective lens 2 ₁ , 2 ₂ ... imaging lens 3, 3 ₁ , 3 ₂ ... variable magnification optical system 4 ₁ , 4 ₂ ... imaging device 5 ₁ , 5 ₂ ... deflection mirror 10 ... binocular microscope 11 ₁ , 11 ₂ ... two-dimensional display element 12 ₁ , 12 ₂ ... projection optical system 13 ... imaging optical system 13a ... Fresnel lens surface 14 ... diffusion plate 14a ... diffusion surface 15, 15 ', 15a, 15b ... display panels 16 ₁ , 16 ₂ ... exit pupils 20 ₁ , 20 _{2 of} projection optical system ... projection devices 30, 30', 30a, 30b ... binocular stereoscopic observation device 31 ... three-dimensional re- Construction unit 32 ₁ , 32 ₂ ... Recording device 33... Reconstruction model recording device 34 ₁ , 34 ₂ ... 2D image reproduction device 41. 47 ... Magnification adjustment screw 48 ... Control unit

Claims (10)

An imaging optical system that forms at least two optical axes having parallax with respect to the observation object and has an overall positive refractive power that forms an image of the observation object on at least two corresponding imaging elements In addition, in the stereoscopic imaging display system having a three-dimensional reconstruction unit that three-dimensionally reconstructs the shape of the object by searching for corresponding points from the parallax images captured by the at least two imaging elements,
A pair of display elements for displaying left and right parallax images, a pair of projection means for projecting parallax images displayed on the pair of display elements on the same plane through an exit pupil, and the plane or the vicinity thereof A display panel, and an imaging means for forming an image of an exit pupil of each projection means at an observation pupil position and a diffusion action, and an image of the exit pupil of each projection means at an observation pupil position A display panel having diffusing means for expanding,
A stereoscopic imaging display system configured to be able to display a parallax image obtained by viewing the object reconstructed three-dimensionally by the three-dimensional reconstruction unit with both eyes at an arbitrary viewpoint position on the pair of display elements. . 2. The three-dimensional image according to claim 1, wherein the image forming means is composed of a Fresnel lens or a Fresnel mirror, and an optical axis of the Fresnel lens or the Fresnel mirror is decentered with respect to the center of the display panel. Imaging display system. 3. The recording apparatus according to claim 1, further comprising: a recording unit that records a parallax image captured by the at least two imaging elements or object shape data three-dimensionally reconstructed by the three-dimensional reconstruction unit. Stereoscopic imaging display system. The difference Δφ between the azimuth angle when the three-dimensional reconstructed stereoscopic image displayed on the display panel is projected on the horizontal plane on which the observation object is placed and the azimuth angle of the observation object is:
−10 ° <Δφ <10 ° (1)
The stereoscopic imaging display system according to any one of claims 1 to 3, wherein: The difference between the look-down angle θ determined by the direction connecting the center of the display panel and the center of the observation object and the vertical line, and the look-down angle θ of the line of sight with respect to the three-dimensional reconstructed stereoscopic image displayed on the display panel Δθ is
−30 ° <Δθ <30 ° (2)
The stereoscopic imaging display system according to any one of claims 1 to 4, wherein: The look-down angle θ with respect to the vertical line of sight with respect to the three-dimensional reconstructed stereoscopic image displayed on the display panel is:
0 ° <θ <60 ° (3)
The stereoscopic imaging display system according to any one of claims 1 to 5, wherein: 7. The apparatus according to claim 1, further comprising a unit that changes the three-dimensionally reconstructed stereoscopic image displayed on the display panel to a stereoscopic image viewed with both eyes at an arbitrary viewpoint position. The stereoscopic imaging display system according to the item. The display panel is configured such that the position of the display panel can be changed, and the viewpoint position of the three-dimensionally reconstructed stereoscopic image displayed on the display panel according to the position of the display panel with respect to the observation object The stereoscopic imaging display system according to claim 7, wherein: A plurality of the display panels are provided, and a three-dimensional reconstructed stereoscopic image displayed on each display panel is displayed with both eyes at an arbitrary viewpoint position according to the position of each display panel with respect to the observation object. The stereoscopic imaging display system according to claim 8, wherein the stereoscopic imaging display system is changed to a viewed stereoscopic image. An inward angle α formed by an observation object position between at least two optical axes having parallax with respect to the observation object of the imaging optical system,
α <12 ° (4)
The stereoscopic imaging display system according to any one of claims 1 to 9, wherein:

Images