JP5441527B2 - Stereoscopic image forming apparatus for stereoscopic video display apparatus - Google Patents

Description

  The present invention relates to an apparatus for forming a stereoscopic scale image used for estimating an actual size of an object displayed in an image in a stereoscopic video display apparatus that displays an image acquired by a stereoscopic imaging apparatus.

  In recent years, stereoscopic image display devices have been remarkably developed. In the stereoscopic video display device, for example, a pair of left and right imaging units are arranged so as to have parallax, and the captured pair of left and right images are displayed on a monitor for stereoscopic viewing. For stereoscopic viewing, the left and right images are alternately displayed in a time-division manner, and the left and right images are alternately observed with glasses equipped with a liquid crystal shutter, and the left and right images are passed through a polarizing filter whose polarization axes are orthogonal to each other. In addition, devices based on various principles, such as a polarization filter glasses system that displays polarized light on the same plane and separates left and right images using polarization filter glasses whose polarization axes are orthogonal to each other, are used. A method for observing a stereoscopic image with the naked eye has also been developed.

The ability to discriminate the subject is remarkably improved by making the video stereoscopic. An example of using this advantage to try to identify an intruder that is difficult to automatically determine in a two-dimensional image has been reported. In particular, if a stereoscopic endoscope camera can be used, a great effect is expected in supporting diagnosis and surgery for a built-in disease.
Furthermore, if the dimensions of an object in a stereoscopic image can be easily known, it will greatly contribute to accurate diagnosis and surgery.

  However, the video obtained from a stereoscopic endoscope camera is mainly information on the shape, layout, and color, and information on the size of the projected video cannot be obtained directly, but is reflected in the display video. It is obtained based on a comparison object such as forceps whose dimensions are known in advance. However, since the positional relationship with the comparison object cannot always be obtained accurately, there is no certainty about the size and distance of the object in the video.

There are a parallel method and a crossover method for photographing with a stereoscopic camera. In particular, in a stereoscopic endoscope, since the interval between two cameras to be stereoscopically viewed is small, in the case of the parallel method, the line-of-sight intersection angle becomes small and the stereoscopic effect is poor. On the other hand, for the purpose of using the stereoscopic endoscope, it is preferable that the observable area is wide and the stereoscopic effect is high. Therefore, in a stereoscopic endoscope, an image picked up by a wide-angle lens using a crossover method is generally stereoscopically displayed so that the field of view can be expanded and the distance in the depth direction can be displayed large.
Such a method for generating a distance image for a video photographed by the crossover method has not been realized. In addition, a simple method for acquiring the distance and size of a stereoscopically displayed video in a stereoscopic video display device is not yet known.

In diagnosis or surgery using a stereoscopic endoscope device, an operator observes the affected area displayed in a stereoscopic image photographed and displayed by a stereoscopic endoscope camera, for example, the shape of the affected area. And surrounding conditions, color, etc. are important judgment materials for diagnosis. However, although there is no means for accurately estimating the depth dimension of the affected part and a stereoscopic image that makes the depth feel as a stereoscopic image is obtained, there is only a method for estimating the dimension based on past experience. In addition, the size of the affected area can only be estimated by a comparison object such as forceps, which has been imaged together, but the positional relationship is not clear and cannot be estimated accurately.
In addition, such a situation is not limited to a stereoscopic endoscope apparatus, but also in a stereoscopic video display apparatus using a commonly used cross-type stereoscopic camera, similarly, it is difficult to accurately determine the dimensions of things in the video. .

Patent Document 1 describes a stereoscopic imaging device in which a measurement scale having an optimum size is drawn on a monitor by a measurement scale generation circuit. The scale generation circuit for measurement disclosed in Patent Document 1 calculates the magnification of an image according to the distance to a test object separately measured for a stereoscopic camera arranged according to the parallel method, and creates a scale corresponding to the magnification. generate.
In the stereoscopic imaging device described in Patent Document 1, when the specimen to be observed is brought to the center of the image, the distance to the specimen is automatically measured and measured by the newly provided stereoscopic measuring device. A measurement scale corresponding to the photographing magnification of the subject is formed according to the distance, and the formed measurement scale is automatically displayed on the monitor screen.

JP 2007-187590 A (in particular, paragraphs 0027 and 0030)

However, the scale display described in Patent Document 1 is intended for a stereoscopic camera arranged according to the parallel method, and the scale fits only on the test object brought to the center of the image. Then, it is necessary to move the camera each time so that the test object comes to the center of the image.
Therefore, the problem to be solved by the present invention is to make it easy to measure a depth such as a depth (distance), a width, and a height of an object existing at an arbitrary position of a stereoscopic image captured by a stereoscopic camera. It is an object of the present invention to provide a stereoscopic scale image forming apparatus that forms a scale and displays it on a stereoscopic video display apparatus.

  In particular, it is to provide a stereoscopic image forming apparatus that forms and displays a stereoscopic scale for numerically expressing the stereoscopic size of an object projected on an image display apparatus in a stereoscopic endoscope apparatus. An operator can perform diagnosis or surgery by knowing the actual size of an object using a three-dimensional scale.

  In order to solve the above problem, a stereoscopic image forming apparatus according to the present invention in a stereoscopic video display apparatus including a first imaging apparatus, a second imaging apparatus, and a stereoscopic image display apparatus is the first imaging apparatus. And a second scale image in the image of each of the second imaging devices, forming a three-dimensional scale image, and according to the distance between the position of the three-dimensional scale in the image and the principal point plane of the first and second imaging devices, A stereoscopic scale image forming apparatus that adjusts the viewing angle of the stereoscopic scale and the size of the length scale and displays the stereoscopic scale on the stereoscopic image display apparatus, and stereoscopically displays the stereoscopic scale in the video, and the position of the displayed stereoscopic scale. And an operation panel provided with an operation end to be moved in the video.

There are many types of stereoscopic cameras used in stereoscopic video display devices, using lenses that match the angle of view and the like. Further, even when parts of the same product number are used, the optical characteristics are often slightly different, and the optical characteristics may change due to adjustment. Furthermore, it is necessary to make adjustments (convergence, etc.) according to the characteristics of the stereoscopic camera and the intended use. Therefore, it is necessary to set a reference for each lens used and shooting conditions. The first imaging device and the second imaging device in the stereoscopic imaging device of the present invention may be arranged so that the optical axes intersect each other using an intersection method.
In the stereoscopic scale image forming apparatus according to one aspect of the present invention, a length scale having a size corresponding to a real object is provided at a video depth corresponding to a convergence angle that is an optical axis crossing angle of a stereoscopic camera using a crossing method. Since the provided 3D scale is displayed in the image, it is possible to know the depth (distance or length), horizontal length (width), and vertical height (length) of the subject at the same distance from the imaging device. Can do.
Further, the three-dimensional scale can be displayed at an appropriate position, and an image is picked up at an appropriate position by adjusting the scale according to the distance between the display position of the three-dimensional scale and the principal point planes of the first and second imaging devices. Since a three-dimensional scale that can be applied to a subject can be displayed, the actual size of the subject can be easily estimated.

  Furthermore, the viewer can easily adjust the display position or posture of the three-dimensional scale using the operation panel. The observer adjusts the depth of the stereoscopic scale to the depth of the subject while observing the target subject and the video on which the stereoscopic scale is displayed, and measures the actual size using the displayed scale. The operation panel is provided with an operation end that can be immediately returned to the reference position by one operation in addition to an operation end that designates three axes in the Cartesian coordinate system or six axes including three rotation axes. Is convenient. The reference position of the three-dimensional scale is preferably the intersection of the optical axes of the two photographing devices, that is, the convergence point.

  It is preferable that an operation panel as a three-dimensional scale man-machine interface that is overridden by an image projected by the three-dimensional camera can be easily operated. For example, it is required that an assistant can easily operate according to an operator's instruction, or can be easily operated when analyzing a recorded stereoscopic video at a later date. In particular, considering that the shape and orientation of the observation object cannot be specified in advance, the operation panel according to the present invention moves and rotates the scale of the three-dimensional scale by designating each axis by an operation based on the local coordinate system. It is preferable that it is possible. Moreover, since it may be necessary to use each axis | shaft restraint depending on a use, you may comprise so that each axis may interlock | cooperate by operating the target axis | shaft.

The stereoscopic scale displayed on the image of the stereoscopic image display device is data obtained by calibrating the scale in comparison with the actual dimensions based on the image obtained by photographing the reference stereoscopic scale set at the target position with the stereoscopic imaging device. It can be accumulated and stored in a storage device as a calibration matrix, and can be formed and displayed based on the stored calibration matrix.
It should be noted that the scale of the three-dimensional scale can be obtained by calculation as the position and orientation change. On the other hand, for the area where the three-dimensional scale is used, the point position covering the area is preliminarily calculated or compared with the actual pattern of the reference pattern to form data that can be rescaled and stored as a calibration matrix. It can be stored in the device.
When the scale pattern is photographed and calibrated by contrasting the scale pattern real image with the scale of the three-dimensional scale image, it is possible to easily form a three-dimensional scale according to the actual situation using the calibration matrix. When the values of the calibration matrix cannot be used as they are, they can be supplemented by using an interpolation method or an extrapolation method.

Since the 3D scale image forming apparatus of the 3D image display apparatus according to the present invention generates a 3D scale at the position of the subject, the viewer accurately measures the size of the subject shown in the 3D image with reference to the 3D scale. be able to.
In particular, when performing observation or surgery of an affected area using a stereoscopic endoscope apparatus, the operator can easily know the dimensions of the affected area displayed in the image by using the stereoscopic image forming apparatus according to the present invention. It is convenient because it can.

1 is a configuration diagram of a stereoscopic video display apparatus using a stereoscopic scale image forming apparatus according to an embodiment of the present invention. It is a conceptual diagram explaining the principle of the stereoscopic vision by a human eye. It is a conceptual diagram which shows the display position of the image of the target object in the image | video of a stereo image display apparatus. It is a conceptual diagram which shows the relationship between arrangement | positioning of the three-dimensional scale which concerns on this embodiment, and the eyes | visual_axis of a three-dimensional camera. It is a conceptual diagram explaining the relationship between the position of the three-dimensional scale which concerns on this embodiment, and the display position in a three-dimensional image display apparatus. It is a perspective view which shows an example of the operation panel used for the three-dimensional scale image forming apparatus which concerns on this embodiment. It is a conceptual diagram explaining the movement condition of each axis | shaft of the solid scale which concerns on this embodiment. It is a conceptual diagram which shows an example of the production method of the reference | standard solid scale which concerns on this embodiment. It is a conceptual diagram explaining the correction method of the geometric distortion of the lens system in this embodiment. It is a conceptual diagram which shows the calibration data matrix used for correction | amendment of the geometric distortion of the lens system in this embodiment. It is a lineblock diagram of a stereoscopic video display device showing an example at the time of applying a stereoscopic scale image forming device concerning this embodiment to a stereoscopic endoscope device.

  Hereinafter, a stereoscopic scale image forming apparatus of a stereoscopic video display apparatus according to the present invention will be described in detail with reference to the drawings. In the drawings, components having the same functions are denoted by the same reference numerals to simplify the description and avoid duplication.

  FIG. 1 is a configuration diagram of a stereoscopic video display apparatus using a stereoscopic scale image forming apparatus according to an embodiment of the present invention. The first imaging device 1 and the second imaging device 2 are cameras having substantially the same performance, and are coupled with a fixed distance between the optical axes so that the optical axes intersect at a predetermined convergence point 8 ahead. Is arranged. A surface that passes through the convergence point 8 and is perpendicular to the surface including the straight line 9 that connects the principal points of the two imaging devices and the two optical axes is referred to as a convergence surface 18. Note that an angle formed by intersecting the lines of sight of two imaging devices looking through one point on the object is called a line-of-sight angle α. The line-of-sight intersection angle with respect to the convergence point 8 is particularly called a convergence angle θ.

  Video signals formed by the first imaging device 1 and the second imaging device 2 are input to the image processing device 3. The image processing device 3 displays the video signals input from the imaging devices 1 and 2 on the stereoscopic image display device 4 separately for the right eye and the left eye. In the figure, the video signal supplied from the first imaging device 1 displays an image viewed with the left eye, and the video signal supplied from the second imaging device 2 displays an image viewed with the right eye.

The stereoscopic image display device 4 alternately displays a right-eye image and a left-eye image, and displays a stereoscopic image display in a time-division stereoscopic display method in which observation is performed using dedicated stereoscopic glasses (not shown) such as liquid crystal shutter glasses that are turned on and off in synchronization therewith. An apparatus is preferable because color reproducibility can be secured. In addition, the left and right images are projected through a polarizing filter with orthogonal polarization axes and projected on the same screen, and displayed on the same screen. An overhead display (OHD) or the like having both a display device and stereoscopic glasses functions can also be used.
In particular, in a stereoscopic image display apparatus used by an operator for observing an affected area or performing surgery, the color of the affected area is also important information, and thus a method in which the color of the image changes due to stereoscopic viewing is not preferable. .

The stereoscopic scale image calculation device 5 forms image signals for the left and right eyes of the stereoscopic scale and supplies them to the image processing device 4. In the image processing device 4, the left-eye signal of the stereoscopic image signal is superimposed on the video signal of the first imaging device 1, and the right-eye signal is superimposed on the video signal of the second imaging device 2. 4 is transmitted and displayed.
In this way, the stereoscopic image 10 and the stereoscopic scale 11 of the target object 7 are displayed on the stereoscopic image display device 4, and these images can be stereoscopically viewed when wearing stereoscopic glasses.

The stereoscopic scale image calculation device 5 is provided with an operation panel 6 operated by a person, and can specify the position and orientation of the stereoscopic scale 11 in the image. The three-dimensional scale image forming apparatus 20 includes a three-dimensional scale image calculation device 5 and an operation panel 6. By adjusting the stereoscopic scale 11 so that the image of the stereoscopic scale 11 overlaps the position of the stereoscopic image 10 of the object 7 using the operation panel 6, the dimension of the object 7 can be measured by reading the scale of the stereoscopic scale 11. it can.
In addition, when it is desired to know the dimensions in the oblique direction which is not the vertical or horizontal direction in the image 10 of the object, the scale is adjusted to follow the measurement direction by rotating the three-dimensional scale 11 about an appropriate axis by the operation panel 6. Thus, the dimensions can be measured by directly hitting the image 10 of the object.

FIG. 2 is a conceptual diagram illustrating the principle of stereoscopic vision by human eyes. The figure shows a case where objects with different distances are observed with a stereoscopic camera. FIG. 3 is a conceptual diagram showing the display position of the object image in the video of the stereoscopic image display device 4. The image viewed with the right eye is indicated by a solid line and the image viewed with the left eye is surrounded by a dotted line.
The first imaging device 1 imitating the human left eye and the second imaging device 2 imitating the right eye enable stereoscopic viewing using the crossing method.

  In FIG. 2, an object (cat) 12 that is closer to the imaging device than the convergence point 8 at which the optical axes of the imaging device intersect is an image 14 for the left eye generated by the first imaging device 1. 3 is displayed on the right side from the center of the stereoscopic image display device 4 as indicated by a dotted line, and the right-eye image 15 generated by the second imaging device 2 is from the center of the stereoscopic image display device 4 as indicated by a solid line. Displayed on the left side. By stereoscopically viewing these images with stereoscopic glasses, the object 12 is observed in the video of the stereoscopic image display device 4 so as to be at the position of the object 12 in the actual space.

  On the other hand, the object (church) 13 far from the convergence point 8 is on the left side of the center of the stereoscopic image display device 4 so that the image 16 for the left eye is indicated by a dotted line, contrary to the object 12 at a close position. The right-eye image 17 is displayed on the right side of the center of the stereoscopic image display device 4 as indicated by a solid line, and becomes a point of intersection of the lines of sight when viewing the images of the two imaging devices by stereoscopic viewing. The object is observed to be at the position of the object 13 in the actual space.

  As described above, when the distance between the imaging devices 1 and 2 and the object is shorter than the convergence plane, the left-eye image 14 displayed on the stereoscopic image display device 4 is displayed to the right of the right-eye image 15, and the imaging device 1. , 2 and the object are far from the convergence plane, the left-eye image 16 displayed on the stereoscopic image display device 4 is displayed to the left of the right-eye image 17. Further, the closer the object is, the larger the line-of-sight intersection angle α is, and the farther the object is, the smaller the line-of-sight intersection angle α is. The size of the image of the object is inversely proportional to the distance.

  The same can be said for the three-dimensional scale corresponding to the position of the three-dimensional scale in the video space. That is, the distance to the arrangement position of the stereoscopic image when the stereoscopic scale 11 is stereoscopically viewed in the stereoscopic image display device 4 is determined by determining the left-right arrangement of the left-eye image and the right-eye image and the line-of-sight intersection angle α. It will be. Note that the line-of-sight intersection angle α is an angle at which the line of sight of the first imaging device 1 and the line of sight of the second imaging device 2 intersect when looking through a certain point in the stereoscopic scale. The amount is related to the distance between the left-eye image and the right-eye image projected on the convergence plane.

FIG. 4 is a conceptual diagram showing the relationship between the arrangement of the stereoscopic scale according to the present embodiment and the line of sight of the stereoscopic camera, and FIG. 5 shows the relationship between the position of the stereoscopic scale according to the present embodiment and the display position in the stereoscopic image display device. It is a conceptual diagram to explain.
FIG. 4 is a conceptual diagram in which a stereoscopic scale displayed on the stereoscopic image display device is stereoscopically displayed. In FIG. 4, regarding the local coordinate system having the origin at the center of the three-dimensional scale, the horizontal axis is expressed as the X axis, the depth direction axis is expressed as the Y axis, and the vertical axis is expressed as the Z axis. In particular, the depth relationship was clearly shown. For this reason, in FIG. 4, the three-dimensional scales 21, 22, and 23 are placed on the Y axis of the absolute coordinate system that connects the intermediate point of the imaging device and the convergence point 8.

As shown in FIG. 4, the convergence plane perpendicular to the plane including the two optical axes at the position of the convergence point 8 formed by the optical axis of the first imaging device 1 and the optical axis of the second imaging device 2 intersecting each other. 18 exists. A reference three-dimensional scale 21 is formed on the convergence surface 18. The reference three-dimensional scale 21 has the same convergence angle θ (α0) as the crossing angle of the optical axes. The scale of the reference solid scale 21 is displayed at an actual size or a predetermined scale. Here, for example, the distance from the reference line 9 connecting the principal points of the first imaging device 1 and the second imaging device 2 to the convergence surface 18 is Dc.
The solid scale may be an appropriate one such as a plane on which a grid is formed or a cross in which scale bars are orthogonal to each other as shown in FIG.

The three-dimensional scale 22 is a three-dimensional scale when disposed at a position closer to the three-dimensional camera than the convergence surface 18. Here, when the distance from the reference line 9 connecting the first imaging device 1 and the second imaging device 2 to the three-dimensional scale 22 is Ds, the scale of the three-dimensional scale 22 is the scale of the reference three-dimensional scale 21. Dc / Ds times.
Note that the distance from the reference line 9 to the three-dimensional scale 22 is as described above, in the three-dimensional image display device 4, the left-right arrangement relationship between the left-eye image and the right-eye image of the three-dimensional scale 22, and the line-of-sight intersection angle α1, or It can be calculated as an amount related to the distance L1 between the left-eye image and the right-eye image projected on the convergence surface 18.

The three-dimensional scale 23 is a three-dimensional scale when arranged at a position farther from the three-dimensional camera than the convergence surface 18. Here, if the distance from the reference line 9 connecting the first imaging device 1 and the second imaging device 2 to the three-dimensional scale 23 is D1, the scale of the three-dimensional scale 23 is the scale of the reference three-dimensional scale 21. Dc / Dl times.
The distance from the reference line 9 to the three-dimensional scale 23 is projected on the left-right arrangement relationship between the left-eye image and the right-eye image on the three-dimensional scale 23 and the line-of-sight intersection angle α2 or the convergence plane 18 in the three-dimensional image display device 4. It can be calculated as an amount related to the distance L2 between the left-eye image and the right-eye image.

  FIG. 5 shows the display position and display image of the left-eye image L and the display position and display image of the right-eye image R in the stereoscopic image display device 4 for the three-dimensional scales 21, 22, and 23 shown in FIG. ing. In the figure, the center position of the screen is shown by a one-dot chain line, and the distance between the center position of the screen and the center position of the three-dimensional scale is shown.

FIG. 5A is a display screen of the stereoscopic image display device 4 showing the stereoscopic scale 22 closest to the stereoscopic camera. The left drawing represents the display unit of the stereoscopic image display device 4 on which the left-eye image of the stereoscopic scale 22 is displayed, and the right drawing represents the display unit on which the right-eye image of the stereoscopic scale 22 is displayed.
Although the stereoscopic scale 22 is stereoscopically displayed at the center of the stereoscopic image display device 4, the left-eye image is biased to the right and the right-eye image is biased to the left. The deviation amount of the one-eye image is L1 / 2, and the distance between them is L1. The scale of the three-dimensional scale 22 is Dc / Ds times larger than the scale of the reference three-dimensional scale 21.

FIG. 5B is a display screen of the stereoscopic image display device 4 showing the reference stereoscopic scale 21 formed on the convergence surface. The left drawing represents the display unit of the stereoscopic image display device 4 displaying the left eye image, and the right drawing represents the display unit displaying the right eye image.
In the reference stereoscopic scale 21, both the left-eye image and the right-eye image are represented at the screen center position, and there is no deviation in the center position of the standard stereoscopic scale 21 image. The scale size of the reference three-dimensional scale 21 is a value calibrated against the actual three-dimensional scale template.

FIG. 5C is a display screen of the stereoscopic image display device 4 showing the stereoscopic scale 23 far from the convergence plane. The left drawing represents the display unit of the stereoscopic image display device 4 on which the left-eye image on the stereoscopic scale 23 is displayed, and the right drawing represents the display unit on which the right-eye image on the stereoscopic scale 23 is displayed.
Although the stereoscopic scale 23 is displayed at the center of the image of the stereoscopic image display device 4, the left-eye image is biased to the left, and the right-eye image is biased to the right. The deviation amount of the one-eye image is L2 / 2, and the distance between them is L2. The scale of the three-dimensional scale 23 is Dc / Dl times that of the reference three-dimensional scale 21.

The three-dimensional scale determines the shape and size of the three-dimensional scale, and the display position according to the display position designated by the viewer through the operation panel, for example, and generates a left-eye image and a right-eye image that conform to the three-dimensional scale. The image is displayed on the stereoscopic image display device 4.
FIG. 6 is a perspective view showing an example of the operation panel 6 used in the stereoscopic scale image forming apparatus according to the stereoscopic image display apparatus of the present embodiment.

As shown in FIG. 6, the operation panel 6 includes a cross key 31 that swings in two directions, a tracking ball 32, a seesaw key 33 that swings in the front-rear direction, and two push button switches 34 and 35. It has been.
The cross key 31 is a switch for instructing movement in the X-axis direction and the Z-axis direction. The horizontal arm of the cross key 31 performs control in the X-axis direction. If the right side of the arm is pushed in, the three-dimensional scale moves to the right in the horizontal direction (X-axis positive direction). Move in (X-axis negative direction). The vertical arm controls the Z-axis direction, and if the upper side of the arm is pushed in, the three-dimensional scale moves upward (Z-axis positive direction), and if the lower side is pushed in, the lower direction (Z-axis) Move in the negative direction. Note that the cross key 31 may be configured to move at a high speed in the direction determined as above by long press.

  The seesaw key 33 is a switch for instructing movement in the Y-axis direction. The seesaw key 33 is provided on the side surface of the operation panel 6, and when the end of the seesaw key 33 on the opposite side is pushed in, the three-dimensional scale is displayed far away (in the negative Y-axis direction). If the is pushed in, the three-dimensional scale comes closer to the front (moves in the positive direction of the Y axis). The seesaw key 33 may designate the movement direction and speed of the three-dimensional scale according to the pressing depth.

  The operation panel 6 can determine the center position of the three-dimensional scale at an arbitrary position in the video by using the cross key 31 and the seesaw key 33. The scale of the three-dimensional scale is determined based on the coordinates in the Y-axis direction. Therefore, by operating the cross key 31 and the seesaw key 33 on the operation panel 6 for an arbitrary object in the video and adjusting the display position of the three-dimensional scale to the position of the object, the position of the object Actual size measurement can be performed.

However, since only the cross key 31 and the seesaw key 33 move in parallel, the three-dimensional scale moves in parallel. If the direction to be measured for the object is not the vertical direction or the horizontal direction, the three-dimensional scale is used to know the size of the image of the object. I can not know the dimensions directly by hitting.
The trackball 32 is an operation end for instructing a movement in the three-axis rotation direction, and the yaw movement around the Z axis of the three-dimensional scale for the rotation around the vertical axis of the trackball, the pitch movement around the Y axis for the rotation in the left-right direction, The roll motion around the X axis is assigned to the rotation in the direction, and the rotational motion of the three-dimensional scale can be controlled.

For example, when it is desired to tilt the three-dimensional scale to the right in the video screen, the trackball 32 may be turned to the right. When it is desired to rotate the three-dimensional scale around the Z axis, the trackball 32 may be rotated in the horizontal plane. Furthermore, when it is desired to tilt the three-dimensional scale forward, the trackball 32 may be rotated forward. In any case, since the movement direction of the trackball and the movement direction of the three-dimensional scale have an intuitive relationship, an operator does not get lost when controlling the three-dimensional scale.
Thus, if the trackball 32 is added as an operation end, the six axes including the rotation of the three-dimensional scale can be easily controlled.

FIG. 7 is a conceptual diagram for explaining the movement state of each axis of the three-dimensional scale according to the present embodiment. The relationship for adjusting the position and posture of the three-dimensional scale using the cross key 31, the seesaw key 33, and the trackball 32 as operation ends. Indicates.
The cross key 31 controls parallel movement of the three-dimensional scale in the X-axis direction and parallel movement in the Z-axis direction. The seesaw key 33 controls the parallel movement of the three-dimensional scale in the Y-axis direction.

Further, the rotation of the three-dimensional scale is controlled by the trackball 32. The rotational motion governed by the trackball 32 is a three-axis rotational motion including a yaw motion about the Z axis, a pitch motion about the Y axis, and a roll motion about the X axis.
Thus, the three-dimensional scale has six axes of freedom and each can change independently, but the changed three-dimensional scale must have a scale corresponding to its position. That is, when moving in the Y-axis direction, since the distance Dy from the reference line 9 of the imaging device to the three-dimensional scale changes, the scale of the reference three-dimensional scale 21 located at the distance Dc from the reference line is used as a reference. A scale of Dc / Dy times must be provided.

  In addition, since control with a 6-degree-of-freedom apparatus is not easy, control can be simplified by constraining and reducing the degree of freedom. In the operation panel 6 of this embodiment, the X axis and the roll axis, the Y axis and the pitch axis, the Z axis and the yaw axis are grouped into two axes so that only two combined axes can be operated at the same time. The mode is adopted. In addition, when any one of the three axes X, Y, and Z is selected, a mode in which the roll axis, the pitch axis, and the yaw axis can be simultaneously operated can be prepared. The push button switch 34 of the operation panel 6 is a switch for selecting activation and cancellation of these shaft constraint functions. By providing these shaft restraint functions, the operability can be further improved.

When there is an enlargement / reduction of an image during shooting or switching of an object, it is necessary to rapidly change the position of the three-dimensional scale and the scale size displayed along the object. In such a case, it may be difficult to follow by operating the operation ends such as the trackball 32 and the seesaw key 33. For this reason, the operation panel 6 is provided with a push button switch 35 as a reset key for giving an instruction to return directly to the position of the reference solid scale 21. By pressing the reset key, the three-dimensional scale is changed to the reference three-dimensional scale at the reference position all at once.
Since this reset key can be used, the followability of the three-dimensional scale is improved when the image changes or the measurement object in the image is changed.

FIG. 8 is a conceptual diagram showing an example of a method for creating a reference solid scale according to the present embodiment.
The scale of the three-dimensional scale displayed on the three-dimensional image display device 4 can be obtained by calculation based on the optical arrangement relationship with the convergence surface 18 as a reference. However, if the optical parameters differ from the design values due to variations in the optical performance of the parts, assembly errors, fluctuations due to optical adjustment or use, etc., the reliability of the scale of the three-dimensional scale obtained by the calculation decreases.
On the other hand, as shown in FIG. 8, a method of calibrating the scale of the three-dimensional scale displayed in the video can be used based on the real three-dimensional scale template.

  In the reference three-dimensional scale 21 generated corresponding to the convergence surface 18 shown in FIG. 4, a first three-dimensional scale template 29 having an actual scale scale is placed on the convergence surface 18 of the three-dimensional camera. The image of the reference stereoscopic scale 21 is brought close to the image of the reference stereoscopic scale template 29 and the image of the actual stereoscopic scale template 29 in the image captured by the imaging device 1 and the second imaging device 2 and displayed on the stereoscopic image display device 4. The scales can be adjusted by expanding and contracting and aligning the scales with the unit length scales 30 carved in the solid scale template 29.

  Note that the convergence point 8 that defines the convergence surface 18 is a point determined by the optical system, and can be easily determined from a geometrical arrangement. For example, when an actual three-dimensional scale template is photographed by two imaging devices and displayed on the three-dimensional image display device 4, the position where the three-dimensional left-eye image and right-eye image overlap with each other corresponds to the convergence plane 18. Therefore, an accurate position can be determined relatively easily.

Since the reference stereoscopic scale 21 is disposed on the convergence plane 18, the center position of the left-eye image displayed on the stereoscopic image display device 4 and the center position of the right-eye image match and do not shift.
It is also possible to automatically detect the scale of the actual three-dimensional scale template 29 using an image processing technique, calculate the size, and use the scale for alignment.
Further, the dimension of the unit length scale 30 of the actual three-dimensional scale template 29 and the distance Dc from the reference line 9 connecting the principal points of the first imaging device 1 and the second imaging device 2 to the convergence surface 18 are detected. Then, the data may be recorded in the system, and then the data of the reference stereoscopic scale 21 may be formed using the data.

Next, when forming the three-dimensional scale 22 to be applied to an object located closer to the convergence surface 18, an actual three-dimensional scale template is placed at the position of the object, and the first and second imaging devices 1 and 2 are imaged. Scale adjustment is performed by expanding and contracting the image of the three-dimensional scale 22 in the image captured by the apparatus 2 and displayed on the three-dimensional image display apparatus 4 and aligning the scale with the unit length scale 30 engraved in the actual three-dimensional scale template 29. Do.
A three-dimensional scale image in which the scale size is determined based on the distance Ds from the reference line 9 connecting the first imaging device 1 and the second imaging device 2 to the position of the object is formed. A stereoscopic display may be performed at the position of the image. The scale is Dc / Ds times larger than the scale of the reference solid scale 21.

Similarly, when forming a three-dimensional scale 23 to be applied to an object located far from the convergence surface 18, an actual three-dimensional scale template is placed at the position of the object, and the first imaging device 1 and the second imaging device 1 are arranged. Scale adjustment by expanding and contracting the image of the three-dimensional scale 23 in the image captured by the imaging device 2 and displayed on the three-dimensional image display device 4 to match the scale with the unit length scale 30 engraved in the actual three-dimensional scale template 29. I do.
In addition, based on the distance Dl from the reference line 9 connecting the first imaging device 1 and the second imaging device 2 to the position of the object, a scale having a size Dc / Dl times the scale of the reference three-dimensional scale 21 is provided. The held three-dimensional scale image may be three-dimensionally displayed at the position of the object image.

In this way, with respect to the three-dimensional scales arranged other than the convergence plane, the geometric arrangement relationship can be obtained without using the actual three-dimensional scale template by using the data related to the scale of the three-dimensional scale arranged on the convergence plane. It can also be determined by calculation based on the above.
Once the calibration at each position has been calibrated, store the data in the system memory as a calibration data matrix, and select the calibration data matrix data as necessary to display the 3D scale. That's fine. The three-dimensional scale relating to the position that has not been calibrated may be displayed by forming an appropriate three-dimensional scale by interpolation or extrapolation using the position data that has been calibrated.

  FIG. 9 is a conceptual diagram illustrating a correction method for the geometric distortion of the lens system in the present embodiment. When the optical system is composed of an ideal thin lens, the scale of the three-dimensional scale may be formed by a square lattice. However, in a stereoscopic endoscope or the like, a wide-angle lens is used for the purpose of widening the visual field range, and therefore, as shown in FIG. 9, between the optical path passing through the center of the wide-angle lens 36 and the optical path passing through the periphery of the wide-angle lens 36. An optical path difference occurs. This optical path difference causes image distortion.

If an ideal lens is used, an image of an actual three-dimensional scale template displays an ideal scale 37 consisting of an undistorted grid on a flat screen 39, but when using a wide-angle lens, due to the optical path difference, A scale 38 having distortion is formed. In order to calibrate the scale 38 having distortion to make the scale 37 without distortion, it is necessary to form a distorted screen 40.
Since the stereoscopic scale in the present embodiment is applied to a stereoscopic endoscope apparatus in particular, it can be dealt with by forming a stereoscopic scale suitable for distortion in a distorted stereoscopic image without using a distorted screen. Can do.

FIG. 10 is a conceptual diagram showing a calibration data matrix used for correcting the geometric distortion of the lens system in this embodiment. The concept of the calibration data matrix formed when there is optical system distortion will be described.
The calibration data matrix is a matrix composed of numerical information about the scale size on the plane on the Y coordinate, and information on each axis scale of the three-dimensional scale exists on this matrix. Therefore, a three-dimensional scale at an arbitrary coordinate position can be formed by using information accumulated in the calibration data matrix. For example, even when the three-dimensional scale is rotated, scale creation information can be acquired from the calibration data matrix to determine the size of the scale to be engraved on the skewed arm. In addition, the desired information can be formed by using an interpolation method or an extrapolation method even at a position where there is no necessary information.

If it is known that there is optical system distortion at the time of configuring the system, create a calibration data matrix for correcting the optical system distortion so that the scale of the three-dimensional scale corresponds to the actual scale on the assumption of optical system distortion. And may be used for forming a three-dimensional scale.
The calibration data matrix 41 for optical system distortion correction is based on a stereoscopic image signal acquired by a stereoscopic camera (first imaging device and second imaging device) by placing an actual stereoscopic scale template at an appropriate position. Form. Insufficient data can be compensated by interpolation or extrapolation. The calibration data matrix 41 is preferably stored in a storage device built in or attached to the three-dimensional scale image calculation device.

  For example, in the reference 3D scale 43 generated corresponding to the convergence plane, the actual 3D scale template having the full scale is placed on the convergence plane of the 3D camera, and the first imaging apparatus and the second imaging apparatus. The scale is adjusted by expanding and contracting the image of the reference stereoscopic scale 43 and aligning the scale with the scale of the actual stereoscopic scale template, based on the image of the actual stereoscopic scale template in the image captured and displayed on the stereoscopic image display device. Do. The scale image thus obtained is stored in the memory as data constituting the calibration data matrix. In practice, the coordinates at the grid points of the scale plate can be used.

Similarly, for the stereoscopic scale 42 at the far point, the scale of the stereoscopic scale 42 at the far point is adjusted while placing the stereoscopic scale template of the entity at the position of the stereoscopic scale 42 at the far point and comparing it with the stereoscopic scale template image of the entity in the image. . The obtained scale data is stored as data constituting the calibration data matrix 41.
Further, scale data is formed in the same manner for the three-dimensional scale 44 at the near point. In the three-dimensional scale 44 at the near point, an optical system distortion appears due to the influence of the wide-angle lens. Measurement data reflecting such actual conditions is stored as data constituting the calibration data matrix 41.

The number of data for forming the calibration data matrix 41 is determined in consideration of the accuracy required for the three-dimensional scale and the amount of work in the configuration. In a region that has not been actually measured, the data obtained by actual measurement can be supplemented using interpolation or extrapolation.
In the calibration data matrix 41 obtained in this way, there is scale information in each axial direction of the three-dimensional scale. Even when the six axes of the three-dimensional scale are changed, a three-dimensional scale having an appropriate scale for dimension measurement can be formed using the data of the calibration data matrix 41 based on the position and orientation of the three-dimensional scale. .
Note that the calibration data matrix 41 can be created and used in the same way even when there is no optical system distortion, thereby improving the efficiency in forming a three-dimensional scale.

FIG. 11 is a configuration diagram of a stereoscopic video display apparatus showing an example in which the stereoscopic scale image forming apparatus according to the present embodiment is applied to a stereoscopic endoscope apparatus.
The stereoscopic endoscope 51 includes two cameras 55 and 56 having substantially the same performance, which are arranged so that the optical axes intersect at a predetermined convergence point 57 ahead.
A video signal formed by a stereoscopic camera composed of two cameras 55 and 56 is input to the stereoscopic display control unit 52. The stereoscopic display control unit 52 includes an image processing device and a stereoscopic scale image calculation device. The stereoscopic display control unit 52 causes the image signal input from the stereoscopic cameras 55 and 56 to be displayed separately on the stereoscopic liquid crystal monitor 53 for the right eye and the left eye by the built-in image processing device.

The stereoscopic liquid crystal monitor 53 alternately displays the image for the right eye and the image for the left eye, and observes it with dedicated stereoscopic glasses (not shown) that are turned on and off in synchronization with this, so that the affected part imaged with the stereoscopic endoscope 51 is displayed. Can be observed three-dimensionally.
The stereoscopic display control unit 52 forms image signals for the left and right eyes of the stereoscopic scale by the built-in stereoscopic scale image calculation device, and supplies the image signals to the built-in image processing device. In the image processing device, the left-eye signal of the stereoscopic image signal is superimposed on the video signal of the left-eye stereoscopic camera, and the right-eye signal is superimposed on the video signal of the right-eye stereoscopic camera and transmitted to the stereoscopic liquid crystal monitor 53. Is displayed.
In this way, the stereoscopic image of the affected area and the stereoscopic scale 58 are displayed stereoscopically on the stereoscopic liquid crystal monitor 53.

The stereoscopic display control unit 52 is provided with an operation panel 44 that is operated by a person, and an operator or an assistant can perform a stereoscopic scale in the image via a stereoscopic scale image calculation device built in the stereoscopic display control unit 52. 58 positions and postures can be designated.
Therefore, if the image of the three-dimensional scale 58 is adjusted so that it overlaps the position of the affected part in the stereoscopic image, the dimension of the affected part can be measured by reading the scale of the three-dimensional scale 58, and accurate diagnosis and surgery are possible. become.

In the embodiment described in this specification, a stereoscopic imaging device arranged in an intersecting manner is cited as an application destination of the stereoscopic scale image forming device. However, even in a stereoscopic imaging device arranged in a parallel manner, the position of the stereoscopic scale is used. The scale of the 3D scale can be adjusted to a size that matches the position and orientation of the 3D scale, and a 3D scale template is placed at a position that matches the display position of the 3D scale. Therefore, it is possible to apply a three-dimensional scale image forming apparatus that performs calibration of a three-dimensional scale.
Needless to say, the calibration data matrix is appropriately recalibrated and updated if there is disassembly cleaning, readjustment, or characteristic change of the imaging apparatus.

  The operation panel of the three-dimensional scale image forming apparatus is composed of a cross key, seesaw key, tracking ball, push button switch, etc., but it can also specify a three-dimensional position and even a three-dimensional posture at a high level. For example, the same mechanism as an appropriate switch operating device such as a control stick or a controller of a game machine can be used.

  By using the stereoscopic image forming apparatus of the present invention, it becomes possible to measure the size of an object displayed on a stereoscopic video apparatus in a stereoscopic manner, and in particular, diagnosis and surgery using a stereoscopic endoscope apparatus, etc. Is effective.

DESCRIPTION OF SYMBOLS 1 1st imaging device 2 2nd imaging device 3 Image processing device 4 Stereoscopic image display device 5 Stereoscopic scale image calculating device 6 Operation panel 7 Object 8 Convergence point 9 Reference line 10 which connects the main points of the imaging device Image 11 Three-dimensional scale image 12 Object (cat)
13 Object (Church)
14 Image for left eye 15 Image for right eye 16 Image for left eye 17 Image for right eye 18 Convergence surface 20 3D scale image forming device 21 3D scale image 22 3D scale image 23 3D scale image 29 Real 3D scale template 30 Unit Long scale 31 Cross key 32 Tracking ball 33 Seesaw keys 34, 35 Push button switch 36 Wide angle lens 37 Ideal scale 38 Scale obtained with wide angle lens 39 Display position without distortion 40 Display position with wide angle lens 41 Calibration data matrix 42 Far point Data 43 Convergence point data 44 Peripheral data 51 3D endoscope 52 3D display control unit 53 3D LCD monitor 54 Operation panel 55, 56 3D camera 57 Convergence point 58 3D scale

Claims (9)

In a stereoscopic video display device comprising a first imaging device, a second imaging device, a stereoscopic image processing device, and an image display device arranged so that their optical axes intersect with each other , the first imaging device and wherein in the second respective imaging device images, sandwiched optical axis includes the intersection of the optical axis of the a stereoscopic scale image marked with length scales first and second imaging devices corner Forming a three-dimensional scale image adjusted in size based on the distance difference when the right-eye image and the left-eye image of the three-dimensional scale are projected on a convergence plane perpendicular to a line that bisects the line. A three-dimensional scale image calculation device that adjusts an intersection angle and the size of the length scale according to the position of the three-dimensional scale and displays them on the image display device, and stereoscopically displays the three-dimensional scale, and the displayed three-dimensional scale Place of The includes an operation panel and having a handling end to move in the image, the stereoscopic scale image forming apparatus of the stereoscopic image display device.   The stereoscopic image display device according to claim 1, wherein the scale of the stereoscopic scale is adjusted in size according to a substantial distance between a display position of the stereoscopic scale and a principal point plane of the first and second imaging devices. 3D scale image forming apparatus.   The three-dimensional scale image is a calibration data formed by accumulating data in which the scale of the three-dimensional scale is calibrated in comparison with a photographed image obtained by arranging a pattern having a constant dimension at the display position of the three-dimensional scale. The three-dimensional image forming apparatus of the three-dimensional image display apparatus according to claim 1, which is formed based on a matrix. The control panel is the position of the stereo scale in the video and an operation end for moving the position of intersection of the optical axis, the three-dimensional scale image forming apparatus of the stereoscopic image display apparatus according to claim 1. The control panel includes an operation end for adjusting the position of the stereo scale in the video, stereoscopic scale image forming apparatus of the stereoscopic image display device according to any one of claims 1 to 4. The operation panel includes an operation end that independently moves the position of the three-dimensional scale in the image in the X-axis direction, the Y-axis direction, and the Z-axis direction of the orthogonal coordinate system, and a roll axis that rotates the three-dimensional scale in the image. An operation end for adjusting the attitude of the three-dimensional scale by independently adjusting around the pitch axis and the yaw axis, and the three-dimensional scale image forming apparatus is configured such that the roll axis, the pitch axis, and the yaw axis are orthogonal coordinates. in combination with one axis of the system, at the same time only 2 shafts in combination each having a mode that allows operation, the control panel comprises an operating end for designating the cancellation of the mode claims 1 to 3 The three-dimensional image forming apparatus of the three-dimensional image display apparatus as described in any one of these. A calibration data matrix in which numerical information for correlating the scale of the three-dimensional scale with the physical scale is stored in a storage device, and an image of the three-dimensional scale is obtained based on the numerical information of the calibration data matrix obtained according to the position and orientation of the three-dimensional scale. The stereoscopic scale image forming device of the stereoscopic video display device according to any one of claims 1 to 6 , wherein the stereoscopic image forming device is displayed and displayed on the image display device. The stereoscopic image forming apparatus for a stereoscopic video display apparatus according to any one of claims 1 to 7 , wherein the first imaging device and the second imaging device are cameras of a stereoscopic endoscope. A stereoscopic endoscope apparatus comprising the stereoscopic scale image forming apparatus of the stereoscopic video display apparatus according to claim 8 .

Images