JP3041420B1 - Endoscope system and recording medium storing control program for detecting depth information of endoscope image - Google Patents

Abstract

Abstract: PROBLEM TO BE SOLVED: To provide an endoscope system capable of accurately obtaining depth information even in the case of a monocular optical system. SOLUTION: By using the principle of motion parallax, information corresponding to motion parallax is obtained by arithmetic processing from a movement of an endoscope in a depth direction and a change in a corresponding position of an endoscope image that changes with the movement, This is an endoscope system that enables depth information to be obtained with a single eye. This system includes an endoscope 1, an endoscope position / direction detection sensor 8 that measures the spatial position and direction of the endoscope itself in real time, and a plurality of continuous plurality obtained when the endoscope moves. A movement information extraction device 6 for calculating the movement of the image at each point in the corresponding image plane from the optical image, and each point visible in the image from the change in the spatial position and direction of the endoscope itself and the movement amount of each point in the image. And a monitor device 9 for superimposing and displaying the three-dimensional position of each point of the image in the endoscopic image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to relates to a recording medium which records depth information detection control program suitable endoscopic system and endoscope images industrial endoscope and medical endoscope.

[0002]

2. Description of the Related Art A conventional endoscope is often a monocular optical system due to its thin cylindrical shape, so that it is difficult to grasp the depth distance during operation. For this reason, it is difficult to determine to what depth the endoscope can be inserted. For this reason, operation under the endoscope is difficult, and if the operator does not pay close attention, there is a high risk that both the endoscope and the target object may be damaged or damaged. is there.

[0003]

[0008] The present invention has been made in view of the above, and its object is an endoscope system and obtained accurately depth information even in the case of monocular optical system
It is to provide a recording medium which records the depth information detection control program endoscope image.

[0004]

In order to achieve the above object, the invention of an endoscope system according to a first aspect of the present invention is directed to an endoscope and an endoscope that measures the spatial position and direction of the endoscope itself in real time. Endoscope position / direction detection means, and movement information extraction means for calculating the movement of an image at each point or region in the corresponding image plane from a plurality of continuous optical images obtained when the endoscope moves, From the change in the spatial position and direction of the endoscope itself obtained from the endoscope position / direction detection means, and the amount of movement of each point or region of the image obtained from the movement information extraction means, In the space of a point or area 3
Depth information calculating means for calculating a three-dimensional position, and three points of each point or region of the image obtained by the depth information calculating means.
Image / distance information display means for displaying the dimensional position superimposed on the endoscopic image.

[0005] Preferably, the endoscope is an endoscope provided with an imaging optical system at the tip of a tube using an optical fiber or a relay lens system or an image pickup device therein.

Preferably, the movement information extracting means calculates an in-image direction and a movement amount, so-called optical flow.

Preferably, the value of the three-dimensional position of each point or area of the image obtained by the depth information calculation means is compared with a predetermined value, so that the end of the endoscope and the object are compared. Comparing means for detecting that the distance between them has approached beyond a predetermined distance limit, and warning means for automatically generating a warning by voice, light, message display, or the like according to the comparison result of the comparing means. Have more.

Preferably, the endoscope is an industrial endoscope or a medical endoscope.

[0009]

[0010] To achieve the above object, the present invention provides a recording medium according to claim 6, there is provided a recording medium recording a control program for detecting the depth information of the endoscopic image using a computer, the control program Allows the computer to measure the spatial position and direction of the endoscope itself in real time, and from a plurality of continuous optical images obtained when the endoscope moves, at each point or area in the corresponding image plane The motion of the image is calculated, and the three-dimensional position in space of each point or region seen in the image is calculated from the change in the spatial position and direction of the endoscope itself and the movement amount of each point or region in the image. The three-dimensional position of each point or area of the image is superimposed and displayed in the endoscope image.

(Function) A function called motion parallax of the human eye can obtain a stereoscopic effect even with a single eye. By applying this principle, the present invention calculates information corresponding to motion parallax from the movement of the endoscope in the depth direction and the change of the corresponding position of the endoscope image that changes with the above configuration. Obtained by By superimposing the information corresponding to the motion parallax in the endoscope image, depth information can be accurately obtained even with a single eye.

Therefore, when the present invention is used, it is easy to grasp the depth distance when operating the endoscope, and it is possible to easily determine the depth direction in which the endoscope can be inserted. In addition, the operation under the endoscope becomes easy, and the risk of damaging or damaging both the endoscope and the object can be easily reduced.

[0013]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 shows a schematic system configuration of an endoscope system according to an embodiment of the present invention. As shown in FIG. 1, the endoscope 1 includes a coupling optical system 1b at a distal end of a tube 1a using an optical fiber or a relay lens system.
The image output of the endoscope 1 is input to a movement information extraction device (optical flow extraction device) 6. The movement information extracting device 6 calculates the image movement (direction and movement amount in the image plane, at each point (region) in a corresponding image plane) from a plurality of continuous optical images obtained when the endoscope 1 moves. Optical flow) is calculated.

The calculation output of the movement information extraction device 6 is further input to a depth information calculation device 7. Depth information calculation device 7
, An output from an endoscope position / direction detection sensor (also referred to as a spatial position / direction detection sensor) 8 attached to the endoscope 1 is also input. Endoscope position / direction detection sensor 8
Measures the spatial position and direction of the endoscope itself in real time.

Based on the output information from the movement information extracting device 6 and the output information from the endoscope position / direction detection sensor 8, the depth information calculation device 7 determines an area visible to the endoscope 1 from the endoscope. It is converted into distance information indicating how far away from the tip is actually. That is, the depth information calculation device 7 calculates the three-dimensional position in space of each point (region) visible in the image from the change in the spatial position and direction of the endoscope itself and the movement amount of each point in the image.

The depth information calculation device 7 calculates the 3
The distance information of the three-dimensional position is output, and the distance information is input to the monitor device 9, and when the operator needs it, the distance information (the image The distance information of the three-dimensional position of each point is superimposed and displayed. This makes it possible for the operator to simultaneously show the distance to each position visible in the image.

Reference numeral 10 denotes a light source for irradiating an object to be observed (for example, the inside of the esophagus or stomach) through the tube 1a of the endoscope 1. The operation means of the monitor device 9 including the above-mentioned instruction buttons, such as a keyboard and an infrared remote controller, are well known in the art, and a description thereof will be omitted.

While the overall configuration and operation of the present embodiment have been outlined above, the present embodiment will be described in detail below for each component.

As the endoscope position / direction detection sensor 8 for measuring the spatial position and direction of the endoscope itself in real time, a magnetic type and an optical type are already commercially available.

Various methods that can be used for the movement information extracting device 6, that is, methods for estimating how a corresponding point in an image has moved from among a plurality of images obtained continuously, that is, an optical flow, have been proposed. Have been. These documents include, for example, Makoto Sato and Hiroshi Sasaki, "Hierarchical Estimation of Motion Vectors in Moving Images" Transactions of the Institute of Electronics, Information and Communication Engineers '86 / 5 Vol.J69-D No.5, P771-776; Shizawa Masahiko,
Kenji Mase, "Multiple Optical Flows, Basic Constraint Equations and Unified Computational Theory of Motion Transparency and Motion Boundary Detection" Transactions of the Institute of Electronics, Information and Communication Engineers, '93 / 5 Vol.J76-D-II No.5, P987-1005;
Takeki Naoki, "Vector Field Reconstruction Using Resolution Control with Dispersion and Its Application to Optical Flow Estimation", IEICE Transactions on Information and Communication, '95 / 7 Vol.J78-D-II No.7, P1028-1038. Therefore, the movement information extracting device 6 can be provided by using the disclosed known technology.

The depth information calculator 7 estimates depth information from an optical flow. The basic principle will be described with reference to FIGS.

First, FIG. 2 shows a state in which the endoscope 1 is used to look inside the cylindrical shape 4 of the observation object or the inspection object. Here, the specific objects 5a and 5b of the cylindrical shape 4 are
The imaging surface 3 of the video camera via the lens 2 of the endoscope 1
Projected to An imaging lens such as a rod lens is used as the lens 2. For the imaging surface 3, a solid-state imaging device such as a CCD (charge coupled device) having an RGB filter is used.
Here, in order to facilitate the explanation, the imaging surface 3 is
Although depicted very close, the imaging surface 3 is often at a remote location through a well-known relay lens system.
Instead of this relay lens system, for example, a gradient index optical fiber generally known as Selfoc (trademark) can be used.

FIG. 3 shows that the endoscope 1 is moved forward from the position shown in FIG.
That is, a case is shown in which the object moves in a direction approaching the objects 5a and 5b. Although the objects 5a and 5b are projected on the imaging surface 3 of the endoscope 1 as in FIG. 2, the positions on the imaging surface slightly shift in proportion to the distance approaching the endoscope 1.

FIG. 4 shows this as viewed on the image pickup surface 3.
FIG. 4 shows images 5a ′ and 5a ′ of the objects 5a and 5b in the state of FIG.
5b ′ and images 5a ″ and 5b ″ in the state of FIG. 3 are illustrated. From the moving distances of the images of these objects, the endoscope 1 is calculated back as shown in FIG.
The relative positions of the objects 5a and 5b from the object can be obtained by geometrical optics. In FIG. 5, 3 ′ is the position of the imaging surface 3 in FIG. 2, 3 ″ is the position of the imaging surface in FIG.
Represents the moving distance of At this time, the position of the endoscope 1 itself,
If the direction is detected by the endoscope position / direction detection sensor 8 in FIG. 1, the absolute positions of the target objects 5a and 5b can be calculated by using the detected values, as described in detail below. .

FIG. 6 defines the coordinate system and the positional relationship between the endoscope 1 and the object in explaining the basic principle of the depth information calculation device 7. A coordinate system と し た is defined with O being the center of the lens of the endoscope 1 in FIG. The direction of the X axis of Σ is the horizontal direction (right is positive) on the endoscope screen, the direction of the Y axis is also the vertical direction (top is positive), and the direction of the Z axis is the optical axis direction (the front is positive). ). Similarly, a coordinate system Σ ′ is defined in which the center of the lens of the endoscope after the movement shown in FIG. 3 is O ′ and the origin is O ′.

In FIG. 6, the position of the object 5a in FIG. The coordinate vector of P viewed from the coordinate system Σ is → p
(Note that the symbol → in the vector symbol is originally described above p, but in the text of the specification except the drawings and the following arithmetic expressions, it is described in parallel on the left side of p for convenience. Hereinafter, the other vectors are similarly described.), And the coordinate vector of P viewed from the coordinate system Σ ′ is denoted by → p ′.
Then, the following equation 1 holds between → p and → p ′.

[0028]

(Equation 1)

Here, → d = (dx, dy, dz) is the position of O ′ as viewed from the coordinate system Σ, that is, the amount of movement of the endoscope 1, and M is a rotation matrix of 3 rows and 3 columns, that is, This is the amount of change in the posture of the endoscope 1. Therefore, both d and M are known values obtained by the endoscope position / direction detection sensor 8.
Obtaining the absolute position of the object 5a is equivalent to solving the equation (1).

FIG. 7 shows the definition of the object position in the endoscope image shown in FIG. 2, that is, before the endoscope 1 moves. For the sake of explanation, as an example, the endoscope 1 is a direct endoscope, and its field of view is circular. X-axis shown in FIG. 6, Y-axis, the origin O, respectively in the endoscopic image plane shown in FIG. 7, X _i axis, Y _i axis, corresponding to O _i (not the same position). Position 5a 'is in the endoscope visual field of the object 5a, the distance from the endoscope visual field center O _i r and X _i
It is determined by the angle α from the axis (positive counterclockwise).
Here, the radius of the field of view of the endoscope in the endoscope image is r ₀ , the viewing angle of the endoscope is 2ω, the coordinate vector of P in the coordinate system →→ p is the depth k as an unknown number, and Is shown in

[0031]

(Equation 2)

Similarly, the coordinate vector of P as viewed from the coordinate system Σ ′ → p ′ is expressed by the following equation (3). Where r '
Is the distance of the object from the center of the endoscope field of view, and α ′ is X
The angle from the _i- axis, k ', is the depth from the coordinate origin O'.

[0033]

(Equation 3)

From the above equations 1, 2 and 3, the following equation 4
Can be derived.

[0035]

(Equation 4)

Equation 4 is a simultaneous equation having two unknowns k and k '. By solving this simultaneous equation, k and k' are obtained. By substituting the obtained k into Equation 2, the absolute coordinates of the object viewed from the coordinate system →→ p
Can be obtained.

Further, as described above, the endoscope 1 is inserted into the hole 4, and when the depth information becomes necessary, the endoscope 1 is moved back and forth to calculate the depth information. The device 7 calculates the absolute coordinates → p of each object viewed from the coordinate system Σ, and uses the calculated values of the absolute coordinates as depth information. For example, as shown in FIG. By superimposing and displaying contour lines and the like on the endoscope image, it is possible to immediately understand, via the endoscope screen or the monitor device 9, which position of the screen viewed by the endoscope 1 and how much unevenness is present. Present it to the operator in a form that can be done.

FIG. 8A shows an original endoscope image, and FIG. 8B shows an endoscope image obtained by mapping depth information as contour lines.

(Other Embodiments) In addition, a comparison circuit is provided in the depth information calculation device 7, and based on the depth information, the distance between the end of the endoscope 1 and the object is limited to a predetermined distance. When approaching beyond the limit, a warning can be automatically generated by voice, light, message display, or the like.

Another object of the present invention is to supply a recording medium (storage medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer of the system or the apparatus. Needless to say, the present invention is also achieved when the CPU (or the CPU or the MPU) reads out and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the function of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.
As a recording medium for recording the program code and for recording variable data such as a table, for example, a floppy disk (FD), hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile A memory card (IC memory card), a ROM, and the like can be used.

[0041]

As described above, according to the present invention,
When the operator needs to display the distance information to the object on the endoscope screen or the external monitor using contour lines or the like, a safe movable range of the subsequent operation can be confirmed. Therefore, according to the present invention, not only safety can be ensured even in the case of a monocular optical system, but also spatial position information can be accurately grasped, so that it is possible to estimate even the actual size of an object, and for an operator. The amount of useful information obtained from the endoscope is significantly improved.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing a configuration of an endoscope system according to an embodiment of the present invention.

FIG. 2 is a schematic diagram showing an optical positional relationship between the endoscope and an object to be imaged when the endoscope is inserted into a hole of an object in one embodiment of the present invention.

FIG. 3 is a schematic diagram similarly showing a change in an optical positional relationship between the endoscope and a rest when the endoscope is moved in one embodiment of the present invention.

FIG. 4 is a conceptual diagram showing a change in an apparent position of an object as viewed from an imaging surface in one embodiment of the present invention.

FIG. 5 is a schematic diagram showing a principle of estimating a spatial position of an object from a change in the position of the endoscope and a change in the position of the object on the imaging surface at that time in the embodiment of the present invention.

FIG. 6 is a schematic diagram for defining a coordinate system and a positional relationship between an endoscope and an object in explaining a basic principle of the depth information calculation device in FIG. 1;

FIG. 7 is a schematic diagram showing a definition of an object position in an endoscope visual field before the endoscope of FIG. 2 moves.

FIG. 8A is a photograph showing an original endoscopic image (the form of a living body, that is, the inside of a human body), and FIG. 8B is an image in which depth information is mapped as contour lines. It is a photograph which shows an endoscope image (form of a living thing, ie, the inside of a human body).

[Explanation of symbols]

 Reference Signs List 1 endoscope 2 lens 3 imaging surface 4 inside of object 5a, 5b object 6 movement information extraction device 7 depth information calculation device 8 endoscope position / direction detection sensor 9 monitor device 10 light source

──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Juri Yamashita 1-3-1 Higashi, Tsukuba, Ibaraki Pref. Industrial Technology Institute Within Biotechnology Research Institute (72) Inventor Kazunori Yokoyama 11-7 Manabe Shinmachi, Tsuchiura City, Ibaraki Prefecture Hospital Tsuchiura Kyodo Hospital (56) References JP-A-10-243936 (JP, A) JP-A-8-275206 (JP, A) JP-A 1-145619 (JP, A) JP-A-63-160634 ( JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) A61B 1/00-1/32

Claims (8)

(57) [Claims] 1. An endoscope, endoscope position / direction detection means for measuring a spatial position and direction of the endoscope itself in real time, and a plurality of continuous endoscopes obtained when the endoscope moves Movement information extraction means for calculating the movement of the image at each point or region in the corresponding image plane from the optical image of the endoscope, the spatial position and direction of the endoscope itself obtained from the endoscope position / direction detection means Depth information calculation means for calculating a three-dimensional position in space of each point or area visible in the image from the change and the movement amount of each point or area of the image obtained from the movement information extraction means; and the depth information calculation means And an image / distance information display means for displaying the three-dimensional position of each point or region of the image obtained in step (1) superimposed on the endoscope image. 2. The endoscope according to claim 1, wherein the endoscope includes an optical fiber, a relay lens system, or an imaging device, and an imaging optical system at an end of a tube body. An endoscope system according to item 1. 3. The endoscope system according to claim 1, wherein the movement information extracting unit calculates an in-image direction and a movement amount, that is, an optical flow. 4. The distance between the tip of the endoscope and an object by comparing a value of a three-dimensional position of each point or area of the image obtained by the depth information calculating means with a predetermined value. And a warning means for automatically generating a warning by voice, light, message display, or the like according to the comparison result of the comparison means. The endoscope system according to any one of claims 1 to 3, wherein 5. The endoscope system according to claim 1, wherein the endoscope is an industrial endoscope or a medical endoscope. 6. A recording medium recording a control program for detecting depth information of an endoscope image using a computer, the control program instructing the computer to execute a spatial position and a direction of the endoscope itself. Is measured in real time, the movement of the image at each point or region in the corresponding image plane is calculated from a plurality of continuous optical images obtained when the endoscope moves, and the spatial position of the endoscope itself And calculating the three-dimensional position of each point or region in the image in space from the change in direction and the amount of movement of each point or region in the image, and endoscopically viewing the three-dimensional position of each point or region in the image A recording medium storing a control program for detecting depth information of an endoscope image, wherein the control program is displayed by being superimposed on a mirror image. 7. The control program for detecting depth information of an endoscope image according to claim 6 , wherein the control program causes a computer to calculate an in-image direction and an amount of movement, that is, an optical flow. The recording medium on which it was recorded. 8. The control program causes a computer to compare a value of a three-dimensional position of each point or area of the image with a predetermined value, thereby reducing a distance between the tip of the endoscope and an object. 8. The method according to claim 6 , wherein a detection is made that the vehicle has approached beyond a predetermined distance limit, and a warning is automatically generated by voice, light, message display, or the like according to the detection result. A recording medium on which a control program for detecting depth information of an endoscope image is recorded.