JP4607043B2 - Endoscopic image forming method with display of gravity direction - Google Patents

Description

  The present invention relates to endoscopic image formation, in particular endoscopic image orientation, and the relationship of this orientation to the direction of gravity and the reference frame of the eyepiece.

  An endoscope is an elongated tubular structure that is inserted into body cavities to inspect these cavities. The endoscope includes a telescope with an objective lens at its distal end. A telescope typically includes an image transfer system. In a rigid endoscope, it is a series of spaced apart lenses. In a flexible endoscope, it is a bundle of very small optical fibers that are closely assembled. Some endoscopes include camera means such as a CCD or CMOS image sensor at the distal end portion and transfer images electronically.

  Many endoscopes only look straight ahead. Other endoscopes feature a fixed or movable reflector at the distal end portion that allows for off-axis viewing. Some of the most common flexible type endoscopes feature an actuated bend at the distal end. The present invention is applicable to all types of endoscopes in the axial direction, non-axial direction, and variable viewing direction.

  At the proximal end of the image transfer system, some endoscopes include an eyepiece that creates a virtual image for immediate human viewing. Often, camera devices such as CCD or CMOS chips are connected to the endoscope. It receives an image and generates a signal for video display. Some endoscopes have a camera device built directly into the endoscope.

  Although the surgeon can and often can see the endoscope directly through the eyepiece, it is more common for the surgeon to use an attached video camera and observe the image on the video screen. It has become the target. In a surgical or diagnostic procedure, the surgeon manually operates the endoscope. The surgeon pitches the endoscope around an axis perpendicular to the longitudinal direction and rolls it around the longitudinal axis. When these manual operations occur on an endoscope with an attached camera, the camera itself displays what is entering the eye from the endoscope as the upright axis of the image on the display. Faithfully associated with the vertical axis of This often results in a rotation of the image that entered the eye.

  This is just a problem. As the image rotates, the surgeon loses track of what actually goes up and down inside the endoscope cavity. Losing this sense of direction is one of the greatest enemies of the endoscope and has led to serious mistakes such as excision of the optic nerve that are believed to be different parts of the tissue during the procedure. It was. When the surgical procedure is a laparotomy rather than an endoscope, the surgeon can see the tissue directly, so there is no problem losing sense of direction. However, during the endoscopic procedure, the surgeon's viewpoint is different from the endoscope's viewpoint, and therefore the surgeon can continuously and interactively display the tissue image drawn in his own mind on the endoscopic image on the display. Try to be related. When making such an attempt, it is routine for the surgeon to observe the flow direction of the droplets in the endoscope cover window or to look for blood pools to get a sense of the direction inside the cavity. Therefore, there is a strong need to know which is up or down inside the endoscope cavity. Apart from being important for distinguishing tissue features that may look similar, knowing the up direction also helps to understand the position of the endoscope relative to the surrounding tissue. Ideally, the surgeon would be able to associate the endoscopic cavity as if his own eye was actually inside the cavity.

Attempted solutions to this problem have been proposed in the prior art patent literature to the bonnet. This document is incorporated herein by reference in its entirety (for example, see Patent Document 1). The object of the present invention was to maintain the orientation of the endoscopic image without the use of electrical detection and positioning devices. A pendulum fixed to the camera is rotatably attached to the endoscope. The pendulum maintains its orientation with respect to gravity about the longitudinal axis of the endoscope. As the endoscope rotates, the pendulum rotates the camera in the opposite direction relative to the endoscope. This is intended to keep the image in the proper orientation.
US Pat. No. 5,307,804

An endoscope equipped with rotational orientation correction is also suggested in another prior art patent document by Koninckx. This document is incorporated herein by reference in its entirety (see, for example, Patent Document 2). An electronic rotary pick-up means responsive to gravity detects the rotation of the camera about the longitudinal axis of the endoscope. The image rotator rotates the camera image by a rotation signal from the rotary pickup means.
US Pat. No. 5,899,851

Another endoscope and camera system equipped with rotational orientation correction is disclosed in yet another prior art patent document by Mattsson-Bose et al. This document is incorporated herein by reference in its entirety (see, for example, Patent Document 3). An electronic detection and positioning device detects and corrects the rotation of a camera that is rotatably mounted on the endoscope. The accelerometer fixed to the camera serves as an electronic rotary pickup means that reacts to gravity. The motor rotates the camera according to the signal from the accelerometer. This accelerometer and motor system is functionally equivalent to the pendulum described by the bonnet. The pendulum rotates depending on gravity, while the accelerometer measures gravity and the motor rotates the assembly accordingly. Therefore, the system can be considered as an electromechanical pendulum. Mattsson-Bose also noticed electronic image rotation to correct the image orientation, but actively stopped doing this for several reasons.
US Pat. No. 6,097,423

Another prior art patent document by Green et al. Discloses the same device as disclosed in Mattsson-Bose and suggests two alternative methods for image rotation (see, for example, US Pat. In the first method, an optical image rotator is used instead of a rotating camera. In the second method, electronic manipulation is used to correct image orientation. One or more gyroscopes are also suggested as another electronic rotary pickup means.
US Pat. No. 6,471,637

Yet another prior art patent document by Chatnever et al. Discloses the same apparatus disclosed in Mattsson-Bose and Green and suggests two alternative methods for electronic rotary pickup (eg, US Pat. reference). In the first method, image analysis is used to calculate the rotation signal. In the second method, the machine vision system is used to calculate the rotation signal.
US Patent Application No. 2002/0161280

Yet another prior art patent document by Schara et al. Teaches a general solution to the image orientation problem. This document is incorporated herein by reference in its entirety (see, for example, Patent Documents 6 and 7). Unlike the above disclosures, these disclosures apply to all scope types and configurations, regardless of the pitch and roll of the endoscope and any line that forms a field of view that is deviated from the axis of the endoscope. An endoscopic image leveled to the gravity with respect to can be provided.
US Patent Application No. 2005/02228230 US Patent Application No. 2005/0154260

  All of the above solutions teach only automatic reorientation and leveling of endoscopic images. From market research reports and discussions with surgeons from different disciplines, it is clear that even a vertical indicator without reorientation of the endoscopic image would be very useful. Surgeons become accustomed to manually reorienting the endoscopic camera during the procedure, and they do not necessarily need or even want an automatically corrected image. Simply providing a vertical indicator allows surgeons to maintain the practice of adjusting the camera on their own, and at the same time provides a visual indication of how much the camera must be rotated to obtain a truly vertical image. Give the key. Alternatively, the surgeon may choose to maintain the current camera orientation, but with the indicator, at least the indicator can see which is up. This is particularly relevant for modern chip-in-tip endoscopes that have a distal camera that cannot rotate.

  Also, with the exception of U.S. Patent Application No. 2005/0228230 and U.S. Patent Application No. 2005/0154260 by Schara et al., All of the above solutions only compensate for rolls about the longitudinal axis and For this reason, a rotation-corrected image is given only. These give an approximation of the correct orientation for a slightly oblique viewing endoscope held near horizontal, but only Schara et al. Correct straight, oblique, lateral, backward and variable viewing endoscopes Teach the solution to do. The current practice of endoscopes is that the surgeon attempts to hold the image vertically by rotating the proximal camera head so that the roll around the endoscope axis of the endoscope is at a horizontal level. . This is done regardless of whether the type of endoscope used is straight, diagonal or flexible. A widespread misconception is that this practice keeps the image horizontal. In fact, a leveled image is given in the case of a rigid linear field endoscope. For any other scope type, this practice does not give a leveled image and is therefore incorrect. This is because there is actually nothing that is believed to be a leveled image.

  Accordingly, it is an object of the present invention to present an indicator that provides an accurate vertical orientation (relative to the eyepiece) of an image viewed from an endoscope. A further object of the present invention is applicable to any axis, oblique, lateral, or rear view endoscope, as well as an endoscope having a variable direction view.

  According to a feature of the invention, the electronic rotary pick-up means is fixed to the endoscope housing. The electronic rotary pickup means generates a signal indicating the rotation of the endoscope. The microprocessor uses these signals to calculate a rotation indicator for endoscopic field orientation. This calculation includes factors to account for endoscope roll, endoscope pitch, and endoscope field of view direction. This indicator is displayed on the video display device. With this arrangement, the indicator is in which direction is up and how much the current image orientation is off-vertical, as if it were viewed by a surgeon standing in a vertical position or sitting. Alternatively, the user indicates how much the camera must be rotated in order to make the image “vertical” on the display.

  The present invention is a method for indicating an appropriate vertical orientation (relative to an eyepiece) of an image from an endoscope, comprising calculating a vertical direction, the calculating step comprising: endoscope pitch, endoscope A method comprising taking into account the effect of image orientation caused by the roll and endoscope viewing direction, and presenting an appropriate said vertical indicator.

  FIG. 1 schematically shows an endoscope. The endoscope includes a shaft 10 with elements that are conventionally provided. The shaft has a longitudinal axis 12.

  An objective optical system is provided at the distal end of the shaft to provide the endoscope with a field vector 14 and a field 16. The objective optical system includes components such as a lens, a prism, and a reflector. The objective optics can be adjusted or can be adjustably attached to make the field direction variable.

  The housing 18 is provided at the proximal end of the shaft 10. An image detection device or camera 20 is attached to the housing 18. The image detection device is configured to receive the image 22 from the objective optics. The housing 18 contains an electronic microprocessor 23 for performing calculations.

  Electronic rotary pick-up means, in the preferred embodiment, three accelerometers 24, 26, 28 responsive to gravity are mounted on the housing 18. Each accelerometer measures the component of gravity along a specific measurement axis. The accelerometer transmits pulse width modulated signals to a processing device that can convert each signal to a gravimeter. Variations in gravity measurements from the accelerometer are related to the rotation of the endoscope.

  In order to properly describe the method of the present invention, an appropriate mathematical configuration needs to be defined.

  The housing 18 has a horizontal longitudinal axis 30 when the housing is in its vertical position and a vertical axis 34 when the housing is in its vertical position. These axes 30, 32, 34 are orthogonal. Each accelerometer axis is aligned with the axis of housing 18. The first accelerometer 24 measures the component of gravity along the longitudinal axis 30. The second accelerometer 26 measures the component of gravity along an axis 32 perpendicular to the longitudinal direction. The third accelerometer 28 measures the component of gravity along the vertical axis 34. The force from the longitudinal accelerometer 24 is Z. The force from the accelerometer 26 in a direction perpendicular to the longitudinal direction is X. The force from the vertical accelerometer 28 is Y.

  The endoscope has a visibility vector 14. Camera vertical projection 36 is the projection of the default vertical axis 38 of camera 20 by optics and view vector 14.

  A viewing vector pivot axis 40 is defined at the distal end of the endoscope and is initially aligned with the vertical axis 34 of the housing. The pivot axis 40 cannot be present in the actual instrument of the endoscope, but is defined as part of the mathematical configuration. The pivot axis 40 can be realigned by rotating itself about the longitudinal axis 12. The variable theta is used to describe the angle of the pivot axis 40 relative to the vertical axis 34 as rotated about the longitudinal axis 12. The variable phi is used to describe the angle of view vector 14 relative to longitudinal axis 12 as it is rotated about pivot axis 40. The variable zeta is used to describe the angle of the camera vertical projection 36 relative to the pivot axis 40 as it is rotated about the field of view vector 14. It should be noted that the above parameterization uses ZYZ Euler angles.

  For simple oblique, lateral, or posterior endoscopes, the parameterized variable theta, phi, and zeta described above are fixed for each endoscope. The variable direction of the visual field endoscope requires that one or more of the variables change between operations that reflect the change in visual field direction.

  During use, the endoscope is positioned in a posture as shown in FIG. This attitude is parameterized as pitch and roll. The variable alpha is also used to describe the pitch angle of the longitudinal axis 12 relative to the horizontal 42. The variable beta is used to describe the roll angle of the endoscope about its longitudinal axis 12. Both pitch and roll can be adjusted during use. The microprocessor calculates the pitch and roll from the official accelerometer output.

  As shown in FIG. 3, the vertical projection 36 of the camera is offset from the gravity vertical portion 43 by the correction angle. The variable gamma is used to describe the correction angle around the view vector 14.

  As shown in FIG. 3, the vertical projection 36 of the camera is offset from the gravity vertical portion 43 by the correction angle. The variable gamma is used to describe the correction angle around the view vector 14. The microprocessor calculates gamma by formula.

  As shown in FIG. 4, the video display 44 is used to give the user an endoscopic image 45 in addition to the vertical indicator 46. Indicator 46 is a directional arrow in this embodiment, but may be any type of graphic object such as a dotted line or a line. An optional vertical stripe 48 indicates the physical top of the display 44 and provides a reference point for rotating the camera. If the user wants to position the endoscopic image 45 so that the upper direction of the endoscopic image is aligned with the upper direction of the display 44, the user will use the camera (until the indicator 46 is aligned along the stripe 48. Alternatively, the image itself) can be rotated if the system has some means of image rotation. Video display 44 may be any suitable device for displaying images from an endoscope.

  In addition to the image orientation indicator 46, a further set of indicators 50 may be used to give the user a sense of the orientation of the endoscope, as shown in FIG. 5A. In this case, these indicators 50 sliding along the outer periphery of the image 44 indicate whether the endoscope has left the user or is heading towards the user. Alternatively, a three-dimensional arrow indicator 52 can be used (FIG. 5B).

  In another embodiment, one or more gyroscopes may be used as the electronic rotary pickup means. The gyroscope output is used to determine the attitude of the endoscope. The gyroscope produces a signal representing a force proportional to the angular displacement relative to its rotational axis. Although described in methods for determining posture using a gyroscope, details of these methods are not necessary for an understanding of the present invention.

  In the layered embodiment of the present invention, a machine vision system is used to calculate the attitude of the endoscope. In such a system, the endoscope comprises at least one signaling element that emits a form of energy that is received by a receiver located at a location remote from the endoscope. Such a signal transmitting element is mounted on a tripod or the like or in a wall and is in the ceiling of the operation room. By analyzing the energy received from the signaling element, the receiver calculates the attitude of the endoscope. Signaling elements can generate energy on their own, such as light emitting diodes, magnets, etc., or to reflect energy emitted from a transmission source located at a location remote from the endoscope Of reflectors. Such a signal transmitting element is mounted on a tripod or the like or in a wall and is in the ceiling of the operation room. The transmission source thus transmits energy. This is reflected from the signaling element and received by the receiver. The energy can comprise, for example, infrared energy, visible spectrum light, magnetic energy, and the like.

  The invention has been described in terms of the most preferred embodiment so that an understanding of the invention can be conveyed. However, although there are many other arrangements for methods of providing gravity-referenced endoscopic images not specifically described herein, the present invention is applicable to these as well. For example, another mathematical framework describing an endoscope leads to another formula for calculating vertical orientation. There are also many ways to display the vertical direction. Furthermore, although the examples are given for endoscopes used in surgery, the present invention is equally applicable to borescopes and the like for use in various mechanical structures. Thus, the term “endscope” as used herein is an endoscope (used for medical surgery) or any similar such as a borescope, fiberscope, etc. Mention the device.

  The present invention is not limited to the embodiments shown in the drawings and described herein. These embodiments are illustrative and not limiting, but are subject to the appended claims.

1 is a schematic view of an endoscope useful in the present invention. FIG. It is a figure which shows the attitude | position of an endoscope. It is a figure which shows the angle amount which the endoscopic image deviated from the perpendicular direction by gravity. It is a figure which shows the displayed endoscopic image which has an indicator of a perpendicular direction. It is a figure which shows the displayed endoscopic image containing the further indicator which gives the information regarding the attitude | position of an endoscope. It is a figure which shows the displayed endoscopic image containing the further indicator which gives the information regarding the attitude | position of an endoscope.

Explanation of symbols

10 shaft 12 longitudinal axis 14 field of view vector 16 field of view 18 housing 20 camera 22 image 23 electronic microprocessor 24, 26, 28 accelerometer 30, 32, 34 axis 34 vertical axis 36 vertical projection 38 axis 40 rotation axis 42 horizontal 43 gravity Vertical portion 44 Image 45 Endoscopic image 46 Indicator 48 Stripe 50 Indicator 52 Three-dimensional arrow indicator

Claims (11)

A system showing the correct vertical orientation of an image,
A scope for acquiring an image, the scope comprising a field of view vector, a longitudinal axis, and a field of view vector that rotates about the field of view vector and is angularly deviated from the longitudinal axis. When,
At least three rotation sensors for monitoring rotation of the scope about three substantially orthogonal axes and generating signals therefor;
A processor coupled to the rotation sensor that receives the signal and calculates an image vertical direction based on the received signal;
A display device coupled to the scope for displaying an acquired image, wherein the display device is an image vertical 3 calculated based on rotation of the scope about the three substantially orthogonal axes; A display device for displaying a graphic display of a dimensional arrow indicator together with an image,
The calculated vertical direction is further based on a rotation of a field vector about a longitudinal axis of the scope.   The system of claim 1, wherein the rotation sensor is an accelerometer.   The system of claim 1, wherein the rotation sensor is at least one gyroscope.   The system of claim 1, wherein the graphic display comprises a three-dimensional arrow graphic display. A method of operating an endoscope to indicate the exact vertical direction of an image, comprising:
Obtaining an image with a scope comprising a field of view vector, a longitudinal axis, and a field of view vector rotated about the field of view vector, the field vector being angularly deviated from the longitudinal axis;
Monitoring the rotation of the scope about three substantially orthogonal axes;
Calculating an image vertical direction based on rotation of the scope about three substantially orthogonal axes, and rotating the visual field vector with respect to the longitudinal axis direction of the scope;
Displaying the acquired image on a display device;
Displaying, with the image, a graphic display of a three-dimensional arrow indicator in the image vertical direction calculated based on the rotation of the scope about the three substantially orthogonal axes;
Including operating method. 6. The operating method according to claim 5, wherein the graphic display includes a graphic display of a three-dimensional arrow. The method of operation according to claim 5 comprising measuring the rotation about the three axes that said substantially orthogonal using at least three acceleration sensors for monitoring the rotation of said scope. The method of operation according to claim 5 comprising measuring the rotation about the three axes that said substantially orthogonal using at least one gyroscope that monitors the rotation of the scope. A method of operating an endoscope to indicate the exact vertical direction of an image, comprising:
Providing a scope comprising a field vector, a longitudinal axis, and a field vector angularly deviated from the longitudinal axis about which the field vector rotates.
Acquiring an image with the scope;
Monitoring the rotation of the scope about three substantially orthogonal axes;
Calculating a tilt of the field vector when the field vector rotates with respect to the longitudinal axis of the scope, the calculation comprising the effect of image orientation due to rotation about the three substantially orthogonal axes; A step of calculating
Displaying the inclination of the visual field vector on the display device together with the acquired image;
Including operating method. Monitoring the rotation of the serial scope The operating method as claimed in claim 9 comprising measuring the rotation about the three axes that said substantially orthogonal using at least three acceleration sensors. The method of operation according to claim 9 comprising measuring the rotation about the three axes orthogonal using at least one gyroscope to the substantially monitoring the rotation of said scope.

Images