JPWO2016076262A1 - Medical equipment - Google Patents

Abstract

The medical apparatus includes an information acquisition unit that acquires position information and gaze direction information of a predetermined viewpoint, a position adjustment unit that aligns the position information and the line-of-sight information with a coordinate system of three-dimensional image information of a predetermined luminal organ, A position calculation unit that calculates a position that intersects the inner wall of a predetermined luminal organ when the line-of-sight direction of the predetermined viewpoint is extended; a distance calculation unit that calculates position information of the position calculation unit and distance information to the predetermined viewpoint; An index generator for generating an index defined based on distance information, and a three-dimensional image of a predetermined luminal organ or a two-dimensional model image in which a predetermined luminal organ is developed, An image generation unit for generating an associated image.

Description

  The present invention relates to a medical device that is inserted into a subject and acquires an image.

2. Description of the Related Art In recent years, endoscope apparatuses equipped with an endoscope as a medical apparatus that is inserted into a subject and observes the inside of the subject or performs a treatment using a treatment tool or the like are widely used. When observing the inside of a predetermined luminal organ to be observed (or examined) using an endoscope, if the surgeon can identify whether or not it has already been observed, the same region is observed repeatedly. There is an advantage that it can be prevented.
For example, in the conventional example of Japanese Patent Laid-Open No. 2013-27697, when observing the inside of a tubular body using a virtual endoscope, an observed area and an unobserved area are displayed on the image of the tubular body. An apparatus for displaying so as to be identifiable is disclosed.

However, since the conventional example does not distinguish between cases where the distance from the viewpoint to the site to be observed is different, it is an observed state that the operator has actually observed, or simply enters the visual field from a distance. Therefore, since it is already observed and it is not possible to determine whether or not the operator is substantially not observing, there is a drawback that the effectiveness when identified is low.
For this reason, it is desirable to be able to easily determine whether or not the observed state is substantially observed from an index or the like according to distance information from the viewpoint to the inner wall of the organ to be observed.
The present invention has been made in view of the above-described points, and an object of the present invention is to provide a medical device capable of generating an image that makes it easy to determine whether or not a substantially observed state has been observed.

  A medical device according to an aspect of the present invention includes an information acquisition unit that acquires position information and gaze direction information of a predetermined viewpoint provided in a predetermined luminal organ, the position information and the line-of-sight information, A positioning unit that matches the coordinate system of the three-dimensional image information of the cavity organ, and the position information and the line-of-sight information that are matched to the three-dimensional image information by the positioning unit. A position calculation unit that calculates a position that intersects the inner wall of the predetermined luminal organ when extending the line-of-sight direction, a distance calculation unit that calculates distance information between the position of the position calculation unit and the predetermined viewpoint, An index generation unit that generates an index defined based on the distance information acquired by the distance calculation unit, and a three-dimensional image of the predetermined luminal organ or the predetermined tube based on a calculation result of the position calculation unit Cavity organ In the two-dimensional model image obtained by developing, and an image generating unit that generates an image associated with the position information of the said indicator inner wall.

FIG. 1 is a diagram showing an overall configuration of a medical device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the image processing apparatus and the like in FIG. FIG. 3 is a diagram showing an association image displaying an index representing a field-of-view range observed on a 2D model image. FIG. 4 is a diagram showing a second coordinate system and the like set on a 3D image in which the bladder is approximated by a sphere. FIG. 5 is a diagram showing an endoscopic image displayed on a monitor and associated images on a 2D model image and a 3D image. FIG. 6A is a diagram showing an observation state in which the inner wall of the bladder is observed while changing the distance from the viewpoint. FIG. 6B is a diagram showing an association image obtained by adding (superimposing) an index whose display form is changed in accordance with a change in distance on the 2D model image corresponding to the observation state of FIG. 6A. FIG. 7 is a flowchart showing the overall operation of the first embodiment. FIG. 8 is a flowchart showing details of step S9 in FIG. FIG. 9A is a flowchart showing processing for generating a circular index according to the contents stored in advance in the table. FIG. 9B is an explanatory diagram of the operation of FIG. 9A. FIG. 9C is a diagram showing, in a tabular form, the contents of a table storing a relationship with a radius index set in accordance with the distance value referred to in the processing of FIG. 9A. FIG. 10A is a flowchart showing a part of processing in a case where selection to change the display form is made according to the angle formed by the line-of-sight direction and the inner wall surface. FIG. 10B is an explanatory diagram of the operation of FIG. 10A. FIG. 10C is a diagram showing an association image in which an index is displayed on the 2D model image by the process of FIG. 10A. FIG. 11A is a diagram showing a display form on the 2D model image in which the color of the index generated according to the value of the distance from the viewpoint to the inner wall is changed. FIG. 11B is a diagram showing a display form on the 2D model image in which the shape of the index is changed according to the angle formed by the line-of-sight direction and the perpendicular to the inner wall surface. FIG. 11C is a diagram showing a display form on the 2D model image in which the size of the index is changed according to the size of the visual field range on the inner wall. FIG. 12 is a diagram for explaining coordinates on the inner surface of a sphere in the second coordinate system according to a modification of the first embodiment of the present invention. Figure 13 is a position P ₂ and the direction V ₂ in the second coordinate system from the position and the direction vector of the tip, explanatory view for explaining the relationship between the position Ps.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
As shown in FIG. 1, the medical apparatus 1 according to the first embodiment of the present invention is an endoscope that observes (or examines) the inside of a predetermined luminal organ (in the specific example, the bladder B) inside a patient P as a subject. The mirror 2, the light source device 3, the processor 4, the image processing device 5, the monitor 6, and the magnetic field generator 7 are configured. The medical device 1 has a function of observing in two observation modes of normal light observation and special light observation. An operator as a user of the medical device 1 performs an internal view of the inside of the bladder B as a predetermined luminal organ (also simply referred to as a luminal organ or an organ) in the patient P placed on the bed 8 in a supine position or the like. Perform a mirror examination.
The endoscope 2 includes an operation unit 2a, a flexible insertion unit 2b, and a universal cable 2c. The endoscope 2 is configured by, for example, an endoscope for bladder examination.
Further, a light guide 9 as shown in FIG. 2 is inserted into the universal cable 2c, and the endoscope 2 passes illumination light from the light source device 3 through the light guide 9 and the distal end portion 2d of the insertion portion 2b. The inside of a predetermined luminal organ (as the bladder B) in the subject into which the distal end portion 2d of the insertion portion 2b is inserted is illuminated.

In addition, as shown in FIG. 2, the distal end portion 2d of the insertion portion 2b is provided with an objective optical system 10 and an imaging surface at its imaging position, and an optical image formed on the imaging surface is photoelectrically converted. An image sensor 11 that outputs an image signal is provided. The inner wall of the bladder B illuminated by the illumination light of the light source device 3 is imaged by the image sensor 11. Therefore, the objective optical system 10 and the imaging device 11 form an imaging unit (or imaging device) 12 that images the inside of a hollow organ. The imaging signal obtained by the imaging element 11 is input to the processor 4 that generates a captured image inside the bladder B as a luminal organ via a signal line in the universal cable 2 c, and the imaging signal is an image in the processor 4. Image generation processing is performed in the generation circuit 4a, and an endoscope image as a captured image obtained by imaging the inside of the bladder B is generated.
In the present embodiment, the image generation circuit 4a in the processor 4 converts the optical image of the subject imaged on the imaging surface of the imaging device 11 of the imaging unit 12 mounted on the distal end 2d into an electrical imaging signal. The captured image corresponding to the optical image is generated by conversion, and the generated captured image is displayed in the endoscope image display area on the monitor 6. In the configuration of FIG. 2, the endoscopic image generated by the image generation circuit 4 a passes through the image processing device 5 in the endoscope image display area A1 of the monitor 6 as shown in FIG. 5, for example. 5 is superimposed on or synthesized with another image generated by the process 5.

The position of the imaging unit 12 that images the inner wall of the bladder B and the imaging direction for imaging (the optical axis direction of the objective optical system 10) are the viewpoint position and the line-of-sight direction when the captured image is generated.
The processor 4 has a changeover switch 4b for switching the observation mode. An observation mode signal designated by the changeover switch 4b is input to the image generation circuit 4a. The image generation circuit 4a is an observation designated by the changeover switch 4b. An endoscopic image corresponding to the mode is generated. More specifically, when the normal light observation mode is specified, a normal light observation image captured under illumination of normal light (as white light) is generated, and the special light observation mode is specified In this case, a special light observation image (narrow band light observation image in a narrow sense) is generated.
The observation mode signal from the changeover switch 4b is input to the LED control circuit 3a of the light source device 3, and the LED control circuit 3a controls to generate illumination light according to the observation mode.

When the normal light observation mode is designated by the changeover switch 4b, the LED control circuit 3a performs control so that the white LED 3b serving as the light source for the normal light observation mode emits light, and the special light observation mode is designated. In this case, the LED control circuit 3a performs control so that the narrow-band blue LED 3c serving as the light source for the special light observation mode emits light.
When the narrow-band blue LED 3c emits light, the narrow-band blue light is selectively reflected by the dichroic mirror 3d disposed at an angle of 45 degrees on the optical path in which the narrow-band blue light travels, and then is collected by the condenser lens 3e. The condensed light is incident on the base end of the light guide 9. The narrow-band blue illumination light incident on the proximal end of the light guide 9 is transmitted by the light guide 9 and emitted from the illumination window to which the distal end of the light guide 9 is attached. In a narrow sense, illumination for a narrow-band light observation mode) is performed.
In addition, white light emitted from the white LED 3b is selectively transmitted by the dichroic mirror 3d disposed on the optical path, and most of the white light except for the narrow-band blue light is selectively transmitted by the condenser lens 3e. Light is incident on the proximal end of the light guide 9. White illumination light other than the narrow-band blue light incident on the base end of the light guide 9 is transmitted by the light guide 9 and emitted from the illumination window to which the tip of the light guide 9 is attached, for the normal light observation mode. Do the lighting.

Further, the endoscopic image generated by the processor 4 (the image generation circuit 4a thereof) is output from the processor 4 to the monitor 6, and for example, a live endoscopic image Ien is displayed on the monitor 6 as shown in FIG. Is displayed. An operator who performs endoscopy can insert the distal end portion 2d of the insertion portion 2b from the urethra of the patient P and observe the inside of the patient's bladder B (shown by a dotted line in FIG. 1).
Further, a magnetic sensor 13 as a position sensor is disposed at the distal end portion 2d of the insertion portion 2b. Specifically, as shown in FIG. 2, a magnetic sensor 13 is provided in the vicinity of the objective optical system 10 and the imaging element 11 that constitute the imaging unit 12 of the distal end portion 2 d. The magnetic sensor 13 is used to detect a three-dimensional position (simply a position) that is the viewpoint of the imaging unit 12 at the distal end portion 2d and the line-of-sight direction at that position.
In the endoscope 2 shown in FIG. 2, the optical axis direction of the objective optical system 10 constituting the imaging unit 12 mounted on the distal end portion 2d is parallel to the axial direction of the distal end portion 2d. The position and the line-of-sight direction can be approximated to the position of the tip 2d and its axial direction (simply the direction).

In the present embodiment, as shown in FIG. 2, an optical image of the subject is formed on the imaging surface of the imaging device 11 arranged at the tip 2d by the objective optical system 10 arranged at the tip 2d. Not only in the case of the endoscope 2, an optical image formed by the objective optical system 10 disposed at the distal end portion 2d is transmitted to the rear (base end) side of the insertion portion 2b, and is disposed at the proximal end side of the insertion portion 2b. The present invention can also be applied to an endoscope that captures an image with an image sensor that has been used.
As an expression including this case, the viewpoint position (and the line-of-sight direction) of the objective optical system 10 (arranged at the tip 2d) is used rather than the expression of the viewpoint position (and the line-of-sight direction) of the imaging unit. Is more reasonable. Therefore, in the following description, the viewpoint position and the line-of-sight direction are mainly used as the viewpoint position and the line-of-sight direction of the objective optical system 10, or instead of the viewpoint position and the line-of-sight direction as an approximate expression (objective optical system In some cases, the position and direction (of 10 or the tip 2d) are used.
As shown in the enlarged view of FIG. 1, the magnetic sensor 13 includes, for example, two coils 2e. The magnetic sensor 13 is a sensor that detects the position and direction of the tip 2d. A signal line 2 f of the magnetic sensor 13 extends from the endoscope 2 and is connected to the image processing device 5 (internal position / direction acquisition circuit 25).

The magnetic field generator 7 generates a magnetic field at a predetermined position that is known, and the magnetic sensor 13 detects the magnetic field generated by the magnetic field generator 7. The magnetic field detection signal is input from the endoscope 2 to the image processing device 5 (internal position / direction acquisition circuit 25) via the signal line 2f.
The position / direction acquisition circuit 25 receives the position information of the viewpoint as the position of the viewpoint of the objective optical system 10 disposed at the distal end portion 2d and the direction of the line of sight based on the amplitude and phase of the input detection signal (both viewpoint position information). And a function of an information acquisition unit (or information acquisition circuit) 25a that acquires the line-of-sight direction information. Note that in this embodiment, the luminal organ to be observed is the inner wall of the bladder B, and the bladder B can be approximated by a three-dimensional (3D) image of a sphere as described later. That is, the bladder B is approximated by a 3D image M1 (FIG. 4) represented by a sphere.
A release button (or release switch) 14 is provided on the operation unit 2 a of the endoscope 2. The release button 14 is a button to be pressed when a user such as an operator records (or stores) an endoscopic image. When the release button 14 is pressed, a release button operation signal is input to the processor 4, and the processor 4 generates a release signal and outputs it to the image processing device 5. The endoscopic image when the release button 14 is pressed is recorded (or stored) in a memory 22 (to be described later) of the image processing device 5.

Further, the endoscope 2 includes an ID generation unit (simply abbreviated as “ID” in FIG. 2) configured by a ROM (read only memory) that generates identification information (abbreviated as “ID”) unique to each endoscope 2. The ID generation unit 15 is input to the image processing device 5 (inside the position / direction acquisition circuit 25) via the processor 4.
As shown in FIG. 2, the ID may be input to the image processing apparatus 5 (the position / direction acquisition circuit 25) without going through the processor 4. The position / direction acquisition circuit 25 is configured by the imaging unit 12 such as the focal length of the objective optical system 10 from the ID, the number of pixels of the image sensor 11 that captures an optical image by the objective optical system 10, the size of the pixel, and the angle of view (viewing angle). A function of an imaging information acquisition unit (or imaging information acquisition circuit) 25b that acquires imaging information in the case of imaging is provided. The acquired imaging information may be used when generating an index to be described later.
The image processing apparatus 5 includes a central processing unit (hereinafter referred to as a CPU) 21, a memory 22, a display interface (hereinafter abbreviated as display I / F) 23, an image capturing circuit 24, and a position / direction acquisition circuit. 25 and a drive circuit 27. The CPU 21, the memory 22, the display I / F 23, the image capture circuit 24, and the position / direction acquisition circuit 25 are connected to each other via a bus 28.

In addition, the medical device 1 includes an input device 30 that allows the user to input and select various types of information to the image processing device 5.
The CPU 21 constitutes a control unit that controls processing of each unit in the image processing device 5 and is arranged at the distal end portion 2d using the viewpoint position information, the line-of-sight direction information, and the imaging information acquired by the position / direction acquisition circuit 25. Position calculation for calculating the viewpoint of the objective optical system 10 and the position (or position information including the position) intersecting the inner wall (surface) of the bladder B as a luminal organ (at least) on the line of sight from this viewpoint The position calculation circuit 21c which forms a part is provided.

  The distance calculation circuit 21a calculates a distance (or distance information including the distance) from the viewpoint to the position calculated by the position calculation circuit 21c. The CPU 21 includes an index generation circuit 21b that forms an index generation unit that generates an index defined (or determined) based on the distance information calculated by the distance calculation circuit 21a. The position calculation circuit 21c may be provided outside the distance calculation circuit 21a, or the position calculation circuit 21c may include the function of the distance calculation circuit 21a.

In addition, the CPU 21 generates a 3D image of the luminal organ (or a 3D model image as an approximate image thereof) M1 and a 2D model image M2 as a model image of a two-dimensional (2D) image in which the luminal organ is developed. An image (association) that associates the index and the position (information) on the inner wall of the luminal organ when the index is generated on the 3D / 2D image generation circuit 21d that performs the 3D / 2D image generation circuit 21d and the 3D image M1 or 2D model image M2. An image generation circuit 21e that forms an image generation unit that generates an image). When the luminal organ is the bladder B, the 2D model image M2 is also referred to as a bladder schema or simply a schema.
For example, as illustrated in FIG. 5, the image generation circuit 21e generates an image in which the index Mk1 is superimposed (associated) on the 2D model image M2, and also generates an image in which the index Mk2 is superimposed (associated) on the 3D image M1. You may do it.
Alternatively, as illustrated in FIG. 3, the image generation circuit 21e may generate an image in which the index Mk1 is superimposed (associated) on the 2D model image M2.

The CPU 21 obtains the first coordinate system (X ₀ Y ₀ Z as a coordinate system based on the magnetic field generator 7 when the position information and the line-of-sight direction information acquired by the position / direction acquisition circuit 25 are acquired. ₀ ) (see FIG. 1) has a function of a coordinate conversion circuit 21f for converting into an intermediate coordinate system (X ₁ Y ₁ Z ₁ ) (see FIG. 1) based on the entrance of the bladder.
As will be described later, the coordinate conversion circuit 21f also has a function of converting from the intermediate coordinate system (X ₁ Y ₁ Z ₁ ) to the second coordinate system (X ₂ Y ₂ Z ₂ ). Therefore, the coordinate conversion circuit 21f has a function of converting from the first coordinate system (X ₀ Y ₀ Z ₀ ) to the second coordinate system (X ₂ Y ₂ Z ₂ ). In addition, when performing coordinate conversion between two coordinate systems, the function of the alignment circuit 21g that forms an alignment unit that performs an alignment process for determining a conversion coefficient, a conversion matrix, and the like at positions known in both coordinate systems is provided. Have.
The coordinate conversion circuit 21f determines, for example, the position (PR in FIG. 4) and direction of the entrance of the bladder B as the reference position and reference direction for alignment, and acquires the position and direction according to the following expressions (1) and (2). The position / direction information of the circuit 25 is converted into the position / direction information of the intermediate coordinate system (X ₁ Y ₁ Z ₁ ) based on the entrance of the bladder B.

［数４］

よって、第１の座標系（X_０Y_０Z_０）上の点（ｘ_０，ｙ_０，ｚ_０）は、次の式（５）に示すように、中間座標系（X_１Y_１Z_１）上の点（ｘ_１，ｙ_１，ｚ_１）に変換される。P ₁ = R ₀₁ P ₀ + M ₀₁ (1)
_{V 1} = _R 01 V 0 ··· formula (2)
Here, P ₀ and V ₀ are respectively a position (vector notation) in the first coordinate system (X ₀ Y ₀ Z ₀ ), which is a coordinate system based on the magnetic field generator 7, and a magnitude of 1 Direction vector. R ₀₁ is a rotation matrix expressed by the following equation (3), and M ₀₁ is a parallel progression sequence expressed by the following equation (4).
[Equation 3]

[Equation 4]

Accordingly, the point (x ₀ , y ₀ , z ₀ ) on the first coordinate system (X ₀ Y ₀ Z ₀ ) is represented by the intermediate coordinate system (X ₁ Y ₁ Z) as shown in the following equation (5). ₁ ) is converted into a point (x ₁ , y ₁ , z ₁ ) on the top.

内視鏡２の先端部２ｄ（の対物光学系１０）の膀胱Ｂ内への挿入が検出されたときに位置方向取得回路２５により取得される位置と方向のベクトルをＰ´₀、Ｖ´₀とすると、並進行列Ｍ_０１は以下の式（６）により求められる。
Ｍ_０１＝−Ｐ´₀ ・・・式（６）
また、回転行列Ｒ_０１は以下の条件を満たすように求める。回転行列Ｒ_０１の満たす条件は、Ｚ₁軸が重力方向と平行であること、及びＺ₁軸に対して垂直なX₁Y₁平面にV´₀を投影し、その投影したベクトル方向をY₁軸、Y₁Z₁平面に垂直なベクトルをX₁軸とする、ことである。
更に、上記座標変換回路２１ｆは、次の式（７）と式（８）に従って、中間座標系（X_１Y_１Z_１）の位置ベクトルと方向ベクトルを、球形の３Ｄ画像Ｍ１の中心Ｏ_２を基準とする第２の座標系（X_２Y_２Z_２）における位置ベクトルと方向ベクトルに変換する。図４は、中間座標系（X_１Y_１Z_１）と第２の座標系（X_２Y_２Z_２）の関係を説明するための図である。[Equation 5]

_P'0 position and direction of the vector obtained by the position-direction acquisition circuit 25 when inserted into the bladder B has been detected in the distal end portion 2d of the endoscope 2 (the objective optical system 10), _V'0 Then, the parallel progression _M01 is obtained by the following equation (6).
M ₀₁ = −P ′ ₀ Formula (6)
Further, the rotation matrix R ₀₁ is obtained so as to satisfy the following conditions. The condition that the rotation matrix R ₀₁ satisfies is that the Z ₁ axis is parallel to the gravity direction, and that V ′ ₀ is projected onto the X ₁ Y ₁ plane perpendicular to the Z ₁ axis, and the projected vector direction is represented by Y _One axis, a vector perpendicular to the Y ₁ Z ₁ plane is taken as the X ₁ axis.
Further, the coordinate conversion circuit 21f converts the position vector and the direction vector of the intermediate coordinate system (X ₁ Y ₁ Z ₁ ) into the center O ₂ of the spherical 3D image M1 according to the following equations (7) and (8). Is converted into a position vector and a direction vector in the second coordinate system (X ₂ Y ₂ Z ₂ ). FIG. 4 is a diagram for explaining the relationship between the intermediate coordinate system (X ₁ Y ₁ Z ₁ ) and the second coordinate system (X ₂ Y ₂ Z ₂ ).

P ₂ = R ₁₂ P ₁ + M ₀₂ (7)
V ₂ = R ₁₂ V ₁ (8)
Here, P ₁ and V ₁ are a position (vector) and a direction vector at an arbitrary position in the intermediate coordinate system (X ₁ Y ₁ Z ₁ ), respectively, and P ₂ and V ₂ are the second A position vector and a direction vector when converted into the coordinate system (X ₂ Y ₂ Z ₂ ).
When P ₀ and V ₀ are set to a position (vector) representing the position of the viewpoint of the objective optical system 10 of the objective optical system 10 calculated from the position / direction acquisition circuit 25 and a direction vector in the line-of-sight direction, P ₀ and V ₀ are P ₁ and V ₁ are converted, and P ₁ and V ₁ are further converted into P ₂ and V ₂ .
That is, when P ₀ and V ₀ are acquired as the viewpoint position and the direction vector in the line-of-sight direction, the corresponding P ₂ and V ₂ are the viewpoints of the objective optical system 10 in the second coordinate system (X ₂ Y ₂ Z ₂ ). A position vector for Pv (also used for Pv) and a direction vector for the line-of-sight direction V (also used for V). R ₁₂ is a rotation matrix represented by the following equation (9), and M ₀₂ is a _{parallel progression} represented by the following equation (10).

［数１０］

よって、中間座標系（X_１Y_１Z_１）上の点（ｘ_１，ｙ_１，ｚ_１）は、次の式（１１）に示すように、第２の座標系（X_２Y_２Z_２）上の点（ｘ_２，ｙ_２，ｚ_２）に変換される。
［数１１］

X₁Y₁Z₁座標系をY₁軸方向にR₂移動した場合、並進M₁₂と回転R₁₂は、それぞれ、式（１２）と式（１３）のようになる。[Equation 9]

[Equation 10]

Therefore, a point on the intermediate coordinate system _{(X 1 Y 1 Z 1)} (x 1, y 1, z 1) , as shown in the following equation (11), a second coordinate system (X ₂ Y ₂ Z ₂ ) is converted to a point (x ₂ , y ₂ , z ₂ ) above.
[Equation 11]

When the X ₁ Y ₁ Z ₁ coordinate system is moved R ₂ in the Y ₁ axis direction, the translation M ₁₂ and the rotation R ₁₂ are as shown in Expression (12) and Expression (13), respectively.

［数１３］

以上のように、磁場発生装置７の第１の座標系（X_０Y_０Z_０）における位置Ｐ_０は、式（５）と式（１１）より、３Ｄ画像Ｍ１の中心Ｏ_２を基準とする第２の座標系（X_２Y_２Z_２）の位置Ｐ_２に変換され、第１の座標系（X_０Y_０Z_０）における方向V_０は、次の式（１４）に従って、第２の座標系（X_２Y_２Z_２）の方向V_２に変換される。
V_２＝Ｒ_１２Ｒ_０１V_０ ・・・式（１４）
また、図４は、内視鏡２の挿入部２ｂが膀胱Ｂ内に挿入された様子を３Ｄ画像Ｍ１で示し、対物光学系１０の視点Ｐｖの位置から視線方向Ｖが交差する球体の内壁面（膀胱Ｂの内壁面）の位置も第２の座標系（X_２Y_２Z_２）を用いて扱うことができる。[Equation 12]

[Equation 13]

As described above, the position P ₀ in the first coordinate system (X ₀ Y ₀ Z ₀ ) of the magnetic field generator 7 is based on the center O ₂ of the 3D image M1 based on the equations (5) and (11). To the position P ₂ of the second coordinate system (X ₂ Y ₂ Z ₂ ), and the direction V ₀ in the first coordinate system (X ₀ Y ₀ Z ₀ ) is expressed by the following equation (14). It is converted in the direction V ₂ of the coordinate system ₂ (X ₂ Y ₂ Z ₂ ).
_{V 2 = R 12 R 01 V} 0 ··· formula (14)
FIG. 4 shows a 3D image M1 in which the insertion portion 2b of the endoscope 2 is inserted into the bladder B, and the inner wall surface of a sphere where the line-of-sight direction V intersects from the position of the viewpoint Pv of the objective optical system 10 The position of (the inner wall surface of the bladder B) can also be handled using the second coordinate system (X ₂ Y ₂ Z ₂ ).

The center O _{2 that} is the origin of the second coordinate system (X ₂ Y ₂ Z ₂ ) is the position of the midpoint of the line segment that connects the neck PR serving as the entrance to the bladder B and the opposing portion PT. In the second coordinate system (X ₂ Y ₂ Z ₂ ), the left wall side in the bladder B is in the X ₂ direction, the right wall side is in the −X ₂ direction, the front wall (abdomen) side of the patient P is in the Z ₂ direction, and the patient P is set to be the rear wall (back) side is -Z ₂ direction.
Further, the 2D model image M2 shown in FIG. 3 or FIG. 5 is obtained by developing or projecting the 3D image shown in FIG. 4 (the inner wall surface of the bladder B approximated by) along a plane including X ₂ and Y _2. It becomes. In the 2D model image M2 in FIG. 5, the above-mentioned representative region is the neck PR, the top corresponding to the neck PR, the left side wall (Left), the right side wall (Right), the front wall (Front ) And back wall (Back).
As shown in FIG. 4, when the position of the viewpoint Pv and the line-of-sight direction V in the second coordinate system (X ₂ Y ₂ Z ₂ ) of the objective optical system 10 at the tip 2d are determined, the center of the imaging surface 11a of the imaging device 11 is determined. The position Ps on the inner wall surface of the sphere imaged at the position (representing the bladder B) is obtained using the radius R of the sphere as follows. A coefficient k satisfying the following equations (15) and (16) is calculated, and the coordinate position of the position Ps in the second coordinate system (X ₂ Y ₂ Z ₂ ) is obtained.

Ps = Pv + kV (15)
| Ps | = R (16)
The position calculation circuit 21c calculates the position Ps of the inner wall that is on the line-of-sight direction V at each viewpoint Pv using Expressions (15) and (16). When the position Ps is calculated, the distance calculation circuit 21a calculates | Pv−Ps | to calculate the distance (information) from the viewpoint Pv to the inner wall. The shape of the bladder B is a sphere having a radius R (the value of R is a predetermined value). For example, the operator contacts the distal end portion 2d with the inner wall (for example, the top) of the bladder where the distance from the bladder entrance is maximum. The value of the radius of the sphere may be set.
The memory 22 shown in FIG. 2 is formed using a ROM, a RAM (Random Access Memory), a flash memory, or the like. The memory 22 stores various processing programs executed by the CPU 21 and various data, and further generates an image. The information of the endoscopic image generated by the circuit 4a, the position information acquired by the position / direction acquisition circuit 25, the line-of-sight direction information, and the like are also stored.

That is, the memory 22 includes an endoscope image storage unit 22a that stores information on an endoscope image, and a position / direction information storage unit 22b that stores viewpoint position information and line-of-sight direction information.
The bladder B can be approximated by a three-dimensional (3D) image of a sphere, and parameters such as a radius representing the sphere are stored in the patient image information storage unit 22c in the memory 22 as a 3D image M1. Further, image information of the 2D model image M2 obtained by developing a three-dimensional image of the bladder B as a luminal organ is also stored in the patient image information storage unit 22c in the memory 22.
Therefore, the patient image information storage unit 22c has a function of a 3D / 2D image storage unit 22d of the 3D image M1 and the 2D model image M2. Note that the 3D / 2D image storage unit 22d may be formed in a storage area other than the patient image information storage unit 22c in the memory 22.

The 3D image M1 and 2D model image M2 stored in the memory 22 are read by, for example, the CPU 21 and displayed on the monitor 6 via the display I / F 23. In the present embodiment, for example, as shown in FIGS. 3 and 5, an image (or an index Mk1 representing an area on the inner wall of the bladder B observed along the line-of-sight direction from the viewpoint on the 2D model image M2 (or An image with the index Mk1 added or an image with the index Mk1 superimposed) is generated and displayed on the display screen of the monitor 6. The surgeon can grasp the observation state of the inside of the bladder B by viewing this image.
Here, the 3D model image M1 and the 2D model image M2 read from the patient image information storage unit 22c have been described, but the acquisition method of the 3D model image M1 and the 2D model image M2 is not limited to this. For example, the 3D / 2D image generation circuit 21d is imaged (observed) by the imaging unit 12 of the endoscope 2 using the observation position and gaze direction data based on the endoscope image Ien captured by the imaging unit 12. A 3D model image M1 or a 2D model image M2 corresponding to the dimensional image data may be generated.

  In this case, the 3D / 2D image generation circuit 21d responds from one two-dimensional image, such as the method described in Japanese Patent No. 5354494 or the well-known Shape from Shading method. A 3D shape may be estimated. Alternatively, a stereo method using two or more images, a three-dimensional shape estimation method using monocular movement vision, a SLAM method, or a method of estimating a 3D shape in combination with a position sensor may be used. When estimating the 3D shape, the 3D shape data may be constructed with reference to 3D image data acquired from a tomographic image acquisition apparatus such as an external CT apparatus.

Further, as shown in FIG. 5, an image with the index Mk1 added may be displayed on the 2D model image M2, and an image with the index Mk2 added may be displayed on the 3D image M1.
In the display example of FIG. 5, the monitor 6 includes an endoscope image Ien in the endoscope image display area A1 and an index Mk1 on the 2D model image M2 together with the 2D model image M2 in the 2D model image display area A2. Is displayed in the 3D image display area A3 together with the 3D image M1 and the image on which the index Mk2 is superimposed on the 3D image M1.

In FIG. 5, since the index Mk1 on the 2D model image M2 and the index Mk2 on the 3D image M1 correspond to each other, the operator can surely grasp the observation state by referring to both. Of course, the observation state can be grasped from one image.
The 3D / 2D image storage unit 22d of the memory 22 sequentially stores the 2D model image M2 to which the index Mk1 is added in time series and the 3D image M1 to which the index Mk2 is added.
The image capturing circuit 24 illustrated in FIG. 2 has a function of an image capturing unit that performs processing for capturing an endoscopic image generated by the processor 4 at a certain period. The image capture circuit 24 acquires, for example, 30 endoscopic images per second, which are the same as the frame rate, from the processor 4. The image capture circuit 24 also receives a release signal from the processor 4.

The position / direction acquisition circuit 25 controls the drive circuit 27 that drives the magnetic field generator 7 to generate a predetermined magnetic field in the magnetic field generator 7, detects the magnetic field by the magnetic sensor 13, and detects the detected magnetic field. From the detection signal, the position coordinates (x, y, z) serving as the viewpoint of the objective optical system 10 and the orientation (ie, Euler angles (ψ, θ, φ)) serving as the viewing direction, that is, the position information of the viewpoint and the viewing direction. Information is generated in real time in the first coordinate system (X ₀ Y ₀ Z ₀ ). That is, the position / direction acquisition circuit 25 acquires the position information and direction information from the magnetic sensor 13, and the position information and line-of-sight direction information of the viewpoint of the objective optical system 10 (simplified position line-of-sight information or position / direction information). The information acquisition part which acquires also is comprised.
The CPU 21 obtains (the information about) the endoscopic image captured by the image capturing circuit 24 and the position / direction information calculated from the position / direction information detected by the position / direction acquisition circuit 25 (for example, by using the coordinate conversion circuit 21f). The association information associated with the second coordinate system (X ₂ Y ₂ Z ₂ ) based on the center O ₂ is stored in the memory 22. Note that the data may be stored in the memory 22 without being converted to the second coordinate system (X ₂ Y ₂ Z ₂ ).

  The memory 22 stores, for example, a 3D / 2D image storage unit in the memory 22 associating information for associating the endoscopic image captured by the image capturing circuit 24 with the position / direction information when the endoscopic image is captured. 22d (example shown in FIG. 2), or the endoscope image storage unit 22a or the position / direction information storage unit 22b in the memory 22 may have a function of an association information storage unit. Alternatively, the association information storage unit may be formed by a storage device separate from the memory 22.

FIG. 6A shows a state in which the distal end portion 2d of the insertion portion 2b of the endoscope 2 is inserted into the bladder B, and the inner wall of the bladder B is observed by the objective optical system 10 or the like in the distal end portion 2d. A state in which an image to which an index is added (associated) is generated on the 2D model image M2 corresponding to the observation state is shown.
The inner wall of the bladder B is approximated by a sphere having a radius R with the center being O _2, and each position on the inner wall is a second coordinate system (X ₂ Y ₂ Z with the center O ₂ as a reference (origin)). ₂ ) It becomes easier to handle.

In addition, as described above, the information on the position of the viewpoint Pv and the line-of-sight direction (optical axis direction) V of the objective optical system 10 disposed at the distal end portion 2d is calculated using the position / direction acquisition circuit 25, and the second coordinates After being converted into system (X ₂ Y ₂ Z ₂ ) information, it is stored in association with the endoscopic image. The position / direction acquisition circuit 25 (the imaging information acquisition unit 25b thereof) acquires imaging information such as the focal length f of the objective optical system 10, the number of pixels of the imaging element 11, and the angle of view.
In the example shown in FIG. 6A, for example, the viewpoint (position) is changed from Pva to Pvb and the distance from the viewpoint to the inner wall is changed from La to Lb for observation. As shown in FIG. 6A, when the distance La from the viewpoint to the inner wall is large, the visual field range (observed observation range) Aa on the inner wall becomes large, and when the distance is small, the visual field range (observation on the inner wall) Range) Ab becomes smaller.
Thus, since the visual field range (observation range) on the inner wall changes according to the distance, in the present embodiment, the index generation circuit 21b depends on the size of the distance or the size of the visual field range on the inner wall. Changing the display color of the indicator display format for the indicator, in particular, if the. For this reason, the index generation circuit 21b has a function of an index color changing unit (index color changing circuit) that changes the display color of the index according to the size of the distance or the size of the visual field range. As shown in FIG. 6B, the index generation circuit 21b functions as an index size changing unit (index size changing circuit) that changes the size (size) of the index according to the size of the distance or the size of the visual field range. have.

Corresponding to the observation state of FIG. 6A, for example, an index Mk1 is generated on the 2D model image M2 as shown in FIG. 6B.
The index Mk1 on the 2D model image M2 shown in FIG. 6B is the index Mk1_Aa corresponding to the area Aa centered on the position Ps on the inner wall that is on the line-of-sight direction V from the viewpoint Pva in FIG. And an index Mk1_Ab corresponding to a region Ab centered on the position Ps on the upper inner wall.
Further, the index Mk1_Aa and the index Mk1_Ab on the 2D model image M2 are respectively positioned at positions on the 2D model image M2 corresponding to the position Ps on the inner wall (Ps ′ in FIG. 6B) with sizes corresponding to the areas Aa and Ab. Generated. Alternatively, the index Mk1_Aa and the index Mk1_Ab are respectively generated in a region on the 2D model image M2 corresponding to the region Aa on the inner wall and the region Ab (with a size reflecting the size of the region Aa and the region Ab).
For simplification, FIG. 6A shows a case where the distance is changed while the line-of-sight direction V is the same. However, if the line-of-sight direction V is different, the position Ps on the inner wall is also different, and thus the index Mk1_Aa on the 2D model image M2 , Mk1_Ab center positions and regions are also different.

Further, as shown in FIG. 6B, when the indices Mk1_Aa and Mk1_Ab overlap on the 2D model image M2, the index is set so that the index Mk1_Ab indicating the observed region observed with a smaller distance is on the upper side. Generated and displayed.
In addition, as shown in FIG. 6B, an image that displays an index with a different color is generated according to the distance to the inner wall.
For this reason, the image generation circuit 21e that superimposes the index generated by the index generation circuit 21b or the index generation circuit 21b in the 3D image M1 or the 2D model image M2 has a distance to the inner wall of a plurality of distance ranges prepared in advance. It has a function of a determination unit that belongs to (or is included in) which distance range. In the example illustrated in FIG. 6B, the distance range Ls in the case of displaying the superimposed index in the 3D image M1 or the 2D model image M2, for example, five distance ranges Li (i (Ls ≧ L5>L4>L3>L2> L1). = 1, 2, ..., 5).

The calculated distance L is red within the smallest distance range L1, orange in the next largest distance range L2, yellow in the next largest distance range L3, green in the next largest distance range L4, and next largest distance range L5. Let's assign the blue color. When the index Mk1 is displayed on the 2D model image M2 as shown in FIG. 6B, a color bar C indicating in which distance range the color representing the display mode of the index is included is displayed around the index Mk1. The diagonal line on the left side of the color such as red in the color bar C is actually assigned a color such as red.
In FIG. 6B, the arrow in the color bar C indicates that the warm color and the distance are small (close to the inner wall). As shown in FIG. 6B, the present invention is not limited to the case of dividing into five distance ranges, and may be divided into a smaller number or a larger number. In addition, a user such as an operator may be able to select the number to be sorted from the input device 30.

Further, as will be described later, an index of the radius r calculated from a preset table according to the distance L from the viewpoint Pv to the inner wall position Ps on the line-of-sight direction V is set on the 2D model image (bladder schema) M2. Or the like.
For this reason, for example, the memory 22 may store the table in which the value of the distance L is associated with the size of the radius r of the corresponding index. A storage device other than the memory 22 may include the above table. For example, the index generation circuit 21b or the image generation circuit 21e may include a storage device having a table. In FIG. 2, the distance calculation circuit 21a, the index generation circuit 21b, the position calculation circuit 21c, the 3D / 2D image generation circuit 21d, the image generation circuit 21e, the coordinate conversion circuit 21f, and the alignment circuit 21g are formed using the CPU 21. Although the configuration is shown, it may be configured using hardware such as a dedicated electronic circuit.

  The medical device 1 according to the first embodiment of the present invention includes a patient image information storage unit 22c (or memory 22) that forms a storage unit that records information representing the shape of a predetermined luminal organ, and the predetermined lumen. A position / direction acquisition circuit 25 forming an information acquisition unit for acquiring position information and line-of-sight direction information of a predetermined viewpoint provided in the organ, and the position information and the line-of-sight information are converted into a three-dimensional image of a predetermined lumen organ. Based on the position information and the line-of-sight information that are aligned with the three-dimensional image information by the alignment unit, the alignment circuit 21g that forms an alignment unit that is aligned with the information coordinate system, and the predetermined viewpoint A position calculation circuit 21c that forms a position calculation unit that calculates a position that intersects the inner wall of the predetermined luminal organ when the line-of-sight direction is extended, and the distance between the position of the position calculation unit and the predetermined viewpoint A distance calculation circuit 21a that forms a distance calculation unit that calculates information, an index generation circuit 21b that forms an index generation unit that generates an index defined based on the distance information acquired by the distance calculation unit, and the position Based on the calculation result of the calculation unit, on the three-dimensional image of the predetermined luminal organ or the two-dimensional model image in which the predetermined luminal organ is developed, an image (association) that associates the index with the position information of the inner wall And an image generation circuit 21e that forms an image generation unit for generating (image).

Next, the operation of this embodiment will be described below with reference to the flowchart of FIG.
When the medical device 1 is in the operating state, the operator performs initial setting in the first step S1. When the operator observes the inside of the bladder B from the input device 30, a condition whether or not to display an index indicating that the region has been observed (specifically, a distance from the viewpoint to the inner wall of the bladder B) Threshold TL). When the threshold value TL is set, an index is generated in a distance range within the threshold value TL. The surgeon can set the threshold value TL in the judgment of the surgeon, but a preset value (for example, 30 mm described later) may be used.
In the next step S2, the operator inserts the endoscope 2 into the urethra of the patient P, and aligns the coordinate system at a known position. Since the position information and the line-of-sight direction information acquired by the position / direction acquisition circuit 25 with respect to the magnetic sensor 13 of the distal end portion 2d become information under the first coordinate system (X ₀ Y ₀ Z ₀ ), the bladder Alignment with the second coordinate system (X ₂ Y ₂ Z ₂ ) used in B is performed.

In this embodiment, the first coordinate system (X ₀ Y ₀ Z ₀ ) and the intermediate coordinate system (X ₁ Y ₁ Z ₁ ) are aligned, and the intermediate coordinate system (X ₁ Y ₁ Z _1). ) And the second coordinate system (X ₂ Y ₂ Z ₂ ), thereby aligning the first coordinate system (X ₀ Y ₀ Z ₀ ) and the second coordinate system (X ₂ Y ₂ Z _2). ). Thereby, the position information and the line-of-sight direction information acquired by the position / direction acquisition circuit 25 are converted into the position information and the line-of-sight direction information in the second coordinate system (X ₂ Y ₂ Z ₂ ) by the coordinate conversion circuit 21f. Become.
In the next step S3, the surgeon inserts the distal end portion 2d of the endoscope 2 into the bladder B. As shown in step S4, the position / direction acquisition circuit 25 acquires information on the viewpoint and line-of-sight direction of the objective optical system 10 at the distal end portion 2d, and as shown in step S5, the coordinate conversion circuit 21f uses this information as the second information. The information is converted into information on the viewpoint and line-of-sight direction in the coordinate system (X ₂ Y ₂ Z ₂ ).
In the next step S6, the distance calculation circuit 21a calculates the position of the inner wall of the bladder B (position Ps in FIG. 4) in the line-of-sight direction (the above formulas (15) and (16)), and the inner wall (upper The distance L (La or Lb in FIG. 6A) is calculated.

In the next step S7, the distance calculation circuit 21a determines whether or not the calculated distance L is within the threshold value TL. When it is determined that the distance L is within the threshold TL, the index generation circuit 21b generates a color index (as a display form) corresponding to the distance L in the next step S8. For example, a color index such as the index Mk1_Ab illustrated in FIG. 6B is generated.
In the next step S9, the image generation circuit 21e generates an image in which an index is added (or superimposed) to a region corresponding to the visual field range on the inner wall on the 2D model image M2 and the 3D image M1. The generated image is output to the monitor 6 via the display I / F 23, and displayed on the display screen of the monitor 6 as shown in the next step S10. In step S11, the memory 22 stores the image generated in step S9. In step S12 after the process of step S11, the CPU 21 determines whether or not an instruction to end the observation is input from the input device 30, and if not input, returns to the process of step S4. Then, the processing of FIG.

If the determination result in step S7 is that the distance is larger than the threshold value TL, the CPU 21 determines in step S13 whether or not the index generation switch 30a of the input device 30 is turned on. Return to the process.
On the other hand, when the index generation switch 30a of the input device 30 is turned on, in the next step S14, the index generation circuit 21b generates a color index (as a display form) according to the value of the distance L, and the step The process proceeds to S9. In the case of step S14, for example, the viewpoint is indicated by Pvc in FIG. 6A. In this case, the distance indicated by Lc is larger than the threshold value TL. In this case, the region Ac forming the visual field range on the inner wall is a wide region, and the region on the 2D model image M2 corresponding to the visual field range Ac has a specific color (for example, as shown by a two-dot chain line in FIG. 6B). (Purple color) index Mk1_Ac.
FIG. 8 shows details of step S9 in FIG. When the process of step S9 is started, in step S21, the image generation circuit 21d reads a past (association) image previously generated in time stored in the memory 22, that is, an image to which a past index is added.

In the next step S22, the image generation circuit 21d determines whether or not the current index, that is, the index of the current observation state generated in step S8 immediately before step S9 overlaps the past index. In the case of overlapping determination results, in the next step S23, the image generation circuit 21d determines the current index so that the index with the smaller distance is on the upper side, in other words, the index with the smaller distance range is on the upper side. An image is generated by overlaying on the past index. Then, the process of FIG. 8 ends. The processing in step S23 has the contents shown in FIG. 6B.
On the other hand, if they do not overlap in step S22, the image generation circuit 21d adds the current index to the past index in step S24 to generate the current image, and ends the process of FIG.
In the above-described embodiment, the case where the index is added on the 2D model image M2 and the 3D image M1 has been described. However, the index may be added on the 2D model image M2 or the 3D image M1. good.

In the example described with reference to FIGS. 6A and 6B, the case has been described in which an index having a size that substantially matches the visual field range on the inner wall has been described. However, as illustrated in FIG. By referencing, the size of the index (more specifically, the circular size of the index) may be automatically determined from the value of the calculated distance L.
FIG. 9A shows processing for setting an index according to the distance L, FIG. 9B shows an explanatory diagram of the operation, and FIG. 9C shows the contents of the table.
When the process of FIG. 9A is started, in the first step S41, the position calculation circuit 21c obtains the position Ps of the inner wall of the bladder B that is on the visual line direction V from the viewpoint Pv, and the position X corresponding to the position Ps on the bladder schema. (Ps ′ in FIG. 6B) is obtained.
In the next step S42, the distance calculation circuit 21a calculates the distance L between the viewpoint Pv of the tip of the endoscope 2 and the position Ps.
In the next step S43, the distance calculation circuit 21a determines whether or not the distance L is 30 mm or less as a preset threshold value TL.

If the determination result is that the distance L is 30 mm or less, in the next step S44, the index generation circuit 21b refers to the table in FIG. 9C and obtains the radius r corresponding to the distance L.
In the next step S45, the index generation circuit 21b or the image generation circuit 21e draws (adds) an index Mk1 having a radius r centered on the position X on the schema. After step S45, the process returns to step S41. Note that if the distance L is greater than 30 mm in step S43, the process returns to step S41.
In the illustrated example of FIG. 9B, when the inner wall is observed from the visual field direction V perpendicular to the inner wall surface, a circular index Mk1 having a radius r close to the visual field range on the inner wall is generated. The size of the index that is automatically generated from the value of the distance L is set (determined) even when it is not vertical. Therefore, the index Mk1 can be generated easily and in a short time. In addition, the surgeon can confirm the observation range of the inner wall almost in real time.

According to this embodiment that operates in this way, when observing the inner wall of the bladder B as a luminal organ, an indicator of the display form corresponding to the distance reflecting the observation state of observing the inner wall is displayed on the 2D model image M2 or Since it is added on the 3D image M1, it is easy for the surgeon to determine whether the observed state is substantially observed from the indicator display form even in the observed region.
Therefore, when an operator intends to observe the observation target range such as the inner wall of the bladder B, the operator refers to the image with the index added on the 2D model image M2 or the 3D image M1, thereby reducing the range within the observation target range. It is possible to efficiently observe by preventing the region of the portion from being overlooked or excessively overlapping.
In the embodiment described above, for example, as shown in FIG. 2, the input device 30 is provided with a display form selection switch (or button) 30b that forms a display form selection unit for selecting a display form of the index, and selects the display form of the index. You may be able to do that. A selection signal generated by the selection operation of the display form selection switch 30b is input to the CPU 21, and the index generation circuit 21b generates an index corresponding to the selection signal.
For example, in addition to the display form in which the surgeon changes the color of the index according to the distance or field of view described above, the angle formed by the line-of-sight direction and the perpendicular line perpendicular to the inner wall surface facing the line-of-sight direction The display form of the indicator may be changed according to θ.
Hereinafter, in a state where the display form of the distance or the visual field range (on the inner wall) is selected, the processing when the display form is selected to be changed according to the angle θ formed by the line-of-sight direction and the perpendicular to the inner wall surface is performed. explain. In this case, the processing of FIG. 7 is slightly changed to the processing shown in FIG. 10A, and only different processing portions will be described.
FIG. 10A shows processing for changing the display form in accordance with an angle θ (also referred to as a line-of-sight angle) θ formed by a line-of-sight direction and a perpendicular to the inner wall surface. For example, in step S31 immediately before step S9 in FIG. 7, the CPU 21 determines whether or not the display form selection switch 30b is ON. That is, the CPU 21 determines whether or not the display form selection switch 30b that changes the display form according to the angle θ formed by the line-of-sight direction and the inner wall surface is ON.

When the display form is selected to be changed according to the angle θ formed between the line-of-sight direction and the inner wall surface, in the next step S32, the index generation circuit 21b calculates the angle θ formed between the line-of-sight direction and the inner wall surface.
FIG. 10B shows how the angle θ is calculated. The position calculation circuit 21c calculates the position Ps on the inner wall surface facing in the line-of-sight direction V from the above formulas (15) and (16). The plane of the inner wall at this position Ps is a plane perpendicular to the position vector Ps with the center O ₂ as the origin. For this purpose, for example, the position calculation circuit 21c calculates the angle θ by π / 2−θ ′, where θ ′ = arccos (V · Ps / | Ps |). The calculation result is sent to the index generation circuit 21b.
In the next step S33, the index generation circuit 21b determines whether or not the calculated angle θ is equal to or less than the threshold θth of the angle θ. The threshold value θth is set within a range of about 80 degrees to 95 degrees, for example.

When the angle θ calculated in step S33 is equal to or smaller than the threshold θth, in the next step S33, the index generation circuit 21b changes the shape of the index generated in step S8 or step S14 according to the value of the angle θ. The deformed index is sent to the image generation circuit 21e. For example, when the angle θ is large, the index generation circuit 21b generates an index having a shape close to a circle, and generates a flat oval index Mk as the angle θ decreases.
FIG. 10C shows an example in which an elliptical index Mk that is, for example, 10 to 20% narrower than the visual field range on the inner wall is generated. Since the observation is performed from an oblique direction, the entire visual field range in that case is not observed, but is reduced so that the inner region is shown as the observed region. The reduction ratio may be set in advance by table data according to the value of the angle θ.
The image generation circuit 21e generates an image in which the index Mk1 is added to the 2D model image M2 and the index Mk2 is added to the 3D image using the input index (Mk in the case of this description) as shown in step S9 of FIG. To do.

FIG. 10C shows an image in which the index Mk1 is added to the 2D model image M2 when the inner wall is observed in a state where the angle θ formed by the line of sight with the inner wall is smaller than the threshold θth as shown in FIG. 10B. In FIG. 10C, an index Mk1 ′ is also shown, and this index Mk1 ′ represents an index corresponding to the case where the inner wall is observed obliquely from a direction perpendicular to the paper surface in FIG. 10B, for example.
When the selection of the display form is not turned on in step S31 in FIG. 10A, the process proceeds to step S9 (similar to the case in FIG. 7). Further, when the angle θ is larger than the threshold θth in step S33, the observation is made from a direction perpendicular to the inner wall surface, and therefore the process proceeds to step S9 as in the case of FIG.
The other processes are the same as those in FIG. By performing the processing as shown in FIG. 10A, the operator can identify the state of observing the inner wall from an oblique direction. The operator can perform the operation on the region displayed in an oval shape in the image of FIG. 10C. This makes it possible to easily know whether or not to observe from a direction perpendicular to the inner wall.

Note that the display form of the index may be selected as shown in FIGS. 11A, 11B, and 11C by the selection by the display form selection switch 30b.
In FIG. 11A, for example, the display color of an index having a certain size is changed according to the value of the distance from the viewpoint to the inner wall.
In FIG. 11B, similar to the case shown in FIGS. 10A to 10C, the smaller the angle θ, the closer the shape of the index, and the larger the angle θ, the closer the shape of the index to an oval shape. To do.
In FIG. 11C, the smaller the visual field range according to the visual field range on the inner wall, the smaller the radius of the index, and the larger the visual field range, the larger the radius of the index.
Moreover, you may change the display form which combined the display form of the parameter | index shown in FIG. 11A-FIG. 11C.
In this embodiment, the sphere of the bladder B is used, but the shape of the bladder B may be a known shape such as an ellipsoid or a cone.

In the first embodiment described above, assuming that the sizes of the 3D image M1 and the 2D model image M2 are predetermined standard sizes, for example, the distance from the bladder entrance to the top by bringing the tip 2d into contact with the top, However, the accuracy of the position where the captured endoscopic image is pasted on the 2D model image and the accuracy of the position and shape of the tip 2d displayed on the 3D image are improved. Therefore, the shape of the patient's bladder B may be measured (measured) using the stereo measurement function of the image processing device 5, and the measured shape may be used as the 3D image M1. Hereinafter, modified examples using the stereo measurement function will be described.
A position on the 3D image M1 corresponding to the position of the endoscopic image of the bladder B (the position Pa (x _a , y _a , z _a ) of the lesioned part AA in FIG. 12) is present within the shape of the bladder B When projecting onto the inner surface of the sphere having the radius R2, the above formulas (15) and (16) are replaced by the following formulas (17) and (18), respectively. FIG. 12 is a diagram for explaining the coordinates on the inner surface of the sphere in the second coordinate system (X ₂ Y ₂ Z ₂ ). FIG. 13 is a diagram for explaining the position P ₂ and the direction V _{2 in} the second coordinate system (X ₂ Y ₂ Z ₂ ) and the position Ps on the inner wall surface from the position and direction vector of the tip 2d. .

When the position P ₂ and the direction V ₂ in the second coordinate system (X ₂ Y ₂ Z ₂ ) of the tip 2d are determined, the coordinates of the intersection point of the obtained endoscopic image with the inner surface of the sphere are obtained. For this purpose, a coefficient k that satisfies the following equations (17) and (18) is calculated, and a position Ps in the second coordinate system (X ₂ Y ₂ Z ₂ ) is obtained.

Based on the position of the obtained coordinates Ps, an index is added on the 2D model image M2 or the 3D image M1 as in the first embodiment.
Therefore, when an operator intends to observe the observation target range such as the inner wall of the bladder B, the operator refers to the image with the index added on the 2D model image M2 or the 3D image M1, thereby reducing the range within the observation target range. It is possible to efficiently observe by preventing the region of the portion from being overlooked or excessively overlapping.
Note that an embodiment configured by partially combining the contents described in the first embodiment including the above-described modification also belongs to the present invention.
Furthermore, the inner wall shape of the bladder B may be acquired using an image taken by an apparatus such as CT, MRI, or ultrasound before the examination.

  This application is filed on the basis of the priority claim of Japanese Patent Application No. 2014-229103 filed in Japan on November 11, 2014, and the above-mentioned disclosure content includes the present specification, claims, It shall be cited in the drawing.

One aspect of the medical device of the present invention is to provide an endoscope in a predetermined luminal organ inside unit, an information acquisition unit that acquires position information and eye direction information in the predetermined viewpoint, the positional information and the viewing direction An alignment unit that aligns information with a coordinate system of three-dimensional image information of a predetermined luminal organ, and the position information and the line-of-sight direction information that are aligned with the three-dimensional image information by the alignment unit A distance calculation unit that calculates distance information from the predetermined viewpoint to the inner wall of the predetermined luminal organ, and an index that generates a plurality of indexes defined based on the distance information calculated by the distance calculation unit a generating unit, before Symbol predetermined luminal organ of the 3-dimensional image or two-dimensional model image on which the expansion of the predetermined tubular organ, to associate the position information of the said plurality of indicators inner wall, said inner Visual If the position of the plurality of indicators overlap in accordance with the observation conditions, and an image generating unit that generates an image index the distance information is smaller it is related as is preferentially displayed.

A medical device according to an aspect of the present invention includes an information acquisition unit that acquires position information and line-of-sight direction information at a predetermined viewpoint with respect to an endoscope in a predetermined lumen organ, the position information, and the line-of-sight direction An alignment unit that aligns information with a coordinate system of three-dimensional image information of a predetermined luminal organ, and the position information and the line-of-sight direction information that are aligned with the three-dimensional image information by the alignment unit A distance calculation unit that calculates distance information from the predetermined viewpoint to the inner wall of the predetermined luminal organ, and an index that generates a plurality of indexes defined based on the distance information calculated by the distance calculation unit On the three-dimensional image of the predetermined luminal organ or the two-dimensional model image obtained by developing the predetermined luminal organ, the generating unit associates the plurality of indices with the position information of the inner wall, and If the position of the plurality of indicators overlap in accordance with the observation conditions, an image generating unit which index the distance information is small to produce an image superimposed current index so that the upper on the image in the past indicators Is provided.

Claims (14)

An information acquisition unit for acquiring position information and gaze direction information of a predetermined viewpoint provided in a predetermined luminal organ;
An alignment unit that aligns the position information and the line-of-sight information with a coordinate system of three-dimensional image information of a predetermined luminal organ;
Based on the position information and the line-of-sight information aligned with the three-dimensional image information by the alignment unit, when the line-of-sight direction of the predetermined viewpoint is extended, it intersects the inner wall of the predetermined lumen organ A position calculation unit for calculating a position;
A distance calculation unit for calculating distance information to the position of the position calculation unit and the predetermined viewpoint;
An index generation unit that generates an index defined based on the distance information acquired by the distance calculation unit;
Based on the calculation result of the position calculation unit, an image in which the index and the position information of the inner wall are associated with each other on a three-dimensional image of the predetermined lumen organ or a two-dimensional model image in which the predetermined lumen organ is developed An image generation unit for generating
A medical device comprising:   The medical apparatus according to claim 1, wherein the index generation unit changes a display form of the index so as to change according to the distance information.   The medical apparatus according to claim 2, wherein the index generation unit changes a display form of the index according to a distance calculated based on the distance information.   The index generation unit calculates a visual field range on the inner wall based on the distance information and the view angle information of the viewpoint, and changes a display form of the index according to the size of the visual field range. The medical device according to claim 2.   The medical apparatus according to claim 2, wherein the index generation unit associates the index with the inner wall from which the distance information calculated by the distance calculation unit is calculated.   The medical apparatus according to claim 4, wherein the index generation unit changes a display form of the index by changing a color or a size of the index. The viewpoint is a viewpoint of an objective optical system that forms a subject image in the subject,
The alignment unit aligns the position information and the line-of-sight direction information of the objective optical system acquired by the information acquisition unit with respect to a coordinate system of three-dimensional image information in the predetermined luminal organ. The medical device according to claim 1.   The medical apparatus according to claim 2, wherein the index generation unit changes a display form of the index according to an angle formed by the line-of-sight direction and the inner wall surface facing the line-of-sight direction.   The image generation unit superimposes the plurality of indices observed at different times over time on the three-dimensional image or the two-dimensional model image at positions corresponding to the inner walls, and positions of the indices and the inner walls. The medical device according to claim 2, wherein the image associated with information is generated.   When the image generation unit superimposes a plurality of indices on the three-dimensional image or the two-dimensional model image, the index when the distance is small is an upper position on the image. The medical device according to claim 9, wherein the arrangement processing is performed.   The medical device according to claim 2, further comprising a display form selection unit that selects a display form to be actually used from a plurality of types of display forms prepared in advance.   The index generating unit generates an index color changing unit that generates an index for changing the color of the index according to the size of the visual field range, or an index for changing the size of the index according to the size of the visual field range. The medical device according to claim 4, further comprising an index size changing unit to be generated.   The medical apparatus according to claim 3, wherein the index generation unit changes a size of the index or a display color of the index according to a distance calculated based on the distance information.   The medical device according to claim 3, wherein the index generation unit changes a display color of the index together with the size of the index according to the size of the distance calculated based on the distance information.

Images