JP2017205343A - Endoscope device and method for operating endoscope device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an endoscope device and the like that can accurately mark a position of a prescribed target object found during endoscopic observation.SOLUTION: An endoscope device includes: an endoscope 1 having an imaging part 11; a three-dimension model image generating part 22 for generating a three-dimension model image M3 of a subject; a position calculation part 26 for calculating a position of an object inside the subject on the basis of a position and orientation information of the endoscope 1 at the time of imaging and the three-dimension model image M3; and a first image processing part 23a for displaying the position of the object on the three-dimension model image M3.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an endoscope apparatus that displays a predetermined object existing inside a subject in association with a three-dimensional model image of the subject, and an operation method of the endoscope apparatus.

  When observing the inside of a lumen with an endoscope, an endoscopic image acquired as a two-dimensional image is displayed, and a three-dimensional model image of the lumen is generated and displayed, so that the observation state can be displayed by the user. An endoscope apparatus having a function as an observation support system to be presented is proposed.

  When a predetermined target such as a lesion or a calculus is found while observing with a two-dimensional endoscope image, marking the tip position of the endoscope at the time of discovery A technique for presenting the position of a predetermined object to a user has also been proposed.

  For example, in International Publication No. WO2013 / 132880, a position of a calculus is calculated based on previously acquired X-ray image data and X-ray imaging position data, and three-dimensional data in which the position of the calculus is superimposed is displayed as an image. Techniques for processing and creating a two-dimensional image for display on a display device are described.

  Japanese Patent Application Laid-Open No. 10-234662 describes a technique for creating a figure indicating a tumor or the like at coordinates stored in advance in a stereoscopic model image displayed on a sub-screen. Furthermore, this publication also describes that an indicator image indicating the distal state of the endoscope scope (the line-of-sight direction, the viewpoint position, and the operator's head direction) is displayed on the stereoscopic model image.

  Further, in International Publication No. WO2015 / 190186, position information acquired by a position detection device is aligned with a three-dimensional image, and angle-of-view display information is superimposed on a position indicated by the aligned position information. The technology to do is described. Further, the view angle display information is displayed in a freeze display by operating a freeze view angle display button.

  JP 2008-537730 A discloses a marking material containing a radiopaque contrast agent that can be seen by fluoroscopy and a coloring pigment that can be seen using an endoscope observation apparatus. Techniques for marking body cavities and providing visual indications are described.

International Publication WO2013 / 132880 Japanese Patent Laid-Open No. 10-234662 International Publication No. WO2015 / 190186 Special table 2008-537730 gazette

  However, in the techniques described in International Publication WO2013 / 132880 and Japanese Patent Application Laid-Open No. 10-234662, it is necessary to acquire the position of a predetermined object such as a calculus or a lesion by X-ray photography or the like in advance. However, it does not support marking the position of a predetermined object found during observation with an endoscope.

  Moreover, since the technique described in the said international publication WO2015 / 190186 or the said special table 2008-537730 is a technique which marks the front-end | tip of an endoscope, or its vicinity position, it is the vicinity of the marked position. Even if it can be understood that there is a predetermined object such as a calculus or a lesion part somewhere, it is difficult to grasp the exact position of the predetermined object.

  Thus, with the above-described conventional technology, it is difficult to accurately mark the position of a predetermined object found during observation with an endoscope. Therefore, even if the user tries to compare the luminal organ model generated in the previous procedure / examination with the observation status of the present procedure / examination, it is determined whether or not the comparison is the same object. It was not possible to judge clearly.

  The present invention has been made in view of the above circumstances, and provides an endoscope apparatus capable of accurately marking the position of a predetermined object discovered during endoscopic observation, and an operation method of the endoscope apparatus. The purpose is to do.

  An endoscope apparatus according to an aspect of the present invention includes an endoscope having an imaging unit configured to capture an image of a subject having a three-dimensional shape and acquire a captured image, and the three-dimensional of the subject. A three-dimensional model image generation unit configured to generate a model image, and position information and azimuth information of the endoscope when a predetermined object existing inside the subject is imaged by the imaging unit; A position calculation unit configured to calculate position information of the predetermined object based on the three-dimensional model image, and a position indicated by the position information of the predetermined object A first image processing unit configured to perform first image processing on a model image or a two-dimensionally developed image obtained by two-dimensionally developing the three-dimensional model image.

  According to an operation method of an endoscope apparatus according to an aspect of the present invention, an imaging unit included in an endoscope captures a captured image by imaging an inside of a subject having a three-dimensional shape, and a three-dimensional model image generation unit Generating a three-dimensional model image of the subject, and the position calculating unit when the predetermined object existing inside the subject is imaged by the imaging unit; The position information of the predetermined object is calculated based on the three-dimensional model image, and the first image processing unit determines the position indicated by the position information of the predetermined object as the three-dimensional model. First image processing is performed to display the image or a two-dimensional developed image obtained by developing the three-dimensional model image in two dimensions.

  According to the endoscope apparatus and the operation method of the endoscope apparatus of the present invention, it is possible to accurately mark the position of a predetermined object found during endoscopic observation.

The block diagram which shows the structure of the endoscope apparatus of Embodiment 1 of this invention. The figure which shows the example of a display of the display apparatus which displays an endoscopic image and a three-dimensional model image in the said Embodiment 1. FIG. The figure which shows the example of a display of the calculus position in the three-dimensional model image in the said Embodiment 1. FIG. The figure which shows an example of the positional relationship of an endoscope and the calculus in a luminal organ in the said Embodiment 1. FIG. The figure which shows the other example of the positional relationship of an endoscope and the calculus in a luminal organ in the said Embodiment 1. FIG. The flowchart which shows the process at the time of observation of the endoscope apparatus of the said Embodiment 1. FIG. The block diagram which shows the structure of the endoscope apparatus of Embodiment 2 of this invention. The figure which shows the example which marks the front-end | tip part position of an endoscope in the said Embodiment 2. FIG. The figure which shows the example which marks the predetermined area | region calculated | required from the front-end | tip part position of an endoscope in the said Embodiment 2. FIG. 7 is a flowchart showing processing during observation of the endoscope apparatus according to the second embodiment. The block diagram which shows the structure relevant to the control part of the endoscope apparatus of Embodiment 3 of this invention. The figure which shows the example of the movement vector of the calculus obtained based on the several captured image from which acquisition time differs in the said Embodiment 3, and the movement vectors other than a calculus. 10 is a flowchart showing processing during observation of the endoscope apparatus according to the third embodiment. The flowchart which shows the process of the stone type determination in FIG. 13 of the said Embodiment 3. FIG.

Embodiments of the present invention will be described below with reference to the drawings.
[Embodiment 1]

  1 to 6 show a first embodiment of the present invention, and FIG. 1 is a block diagram showing a configuration of an endoscope apparatus.

  The endoscope apparatus includes an endoscope 1 and a processing system 2, and may further include at least one of a database 3 and a display device 4 as necessary. Hereinafter, the case where the endoscope apparatus includes the display device 4 will be described as an example, and the case where the endoscope apparatus further includes the database 3 will be described as appropriate.

  The endoscope 1 is an image acquisition device that acquires an image inside a subject in order to observe the inside of the subject having a three-dimensional shape, and includes an imaging unit 11, an illumination unit 12, and a position / orientation detection unit. 13 and an operation unit 14. Among these, the imaging unit 11, the illumination unit 12, and the position / orientation detection unit 13 are disposed, for example, at the distal end of the insertion unit of the endoscope 1, and the operation unit 14 is connected to the proximal side of the insertion unit.

  In the present embodiment, the renal pelvic kidney gob is taken as an example of a subject having a three-dimensional shape. However, the present invention is not limited to this, and endoscope observation is possible with a plurality of ducts. Any subject can be widely applied.

  The illumination unit 12 irradiates illumination light toward the inside of the subject.

  The imaging unit 11 forms an optical image inside the subject irradiated with illumination light by an optical system and performs photoelectric conversion using an imaging element or the like, and generates and acquires a captured image.

  The position / orientation detection unit 13 detects the three-dimensional position of the distal end portion of the insertion portion of the endoscope 1 and outputs it as position information, and detects the direction in which the distal end portion of the insertion portion of the endoscope 1 faces to Output as information. For example, when the xyz coordinate system is set, the position information is represented by (x, y, z) coordinates, and the azimuth information is represented by an angle around the x axis, an angle around the y axis, and an angle around the z axis. (Therefore, the position / orientation detection unit 13 is also called, for example, a 6D sensor). Of course, the position information and orientation information of the endoscope 1 may be expressed using other appropriate methods (for example, a polar coordinate system).

  The operation unit 14 performs a bending operation or the like of the endoscope 1 and also serves as a member for performing a marking operation. The marking operation may be performed by a foot switch or the like, or may be performed by an operation unit provided in the processing system 2, and is not limited to be performed by a specific configuration.

  The processing system 2 controls the endoscope 1 and communicates with the database 3 as necessary to process the captured image, position information, and orientation information acquired from the endoscope 1 to display image data for display. Or image data for recording is generated and output to the display device 4 or the like. In addition, this processing system 2 may be comprised as a single apparatus, and may be comprised from several apparatuses, such as a light source device and a video processor.

  The processing system 2 includes an endoscope image generation unit 21, a three-dimensional model image generation unit 22, an image processing unit 23, an image synthesis output unit 24, an illumination control unit 25, and a control unit 27. ing.

  The endoscopic image generation unit 21 performs various types of image processing such as demosaicing processing (or synchronization processing), white balance processing, color matrix processing, and gamma conversion processing on the captured image output from the imaging unit 11. Then, an endoscopic image EI (see FIG. 2 etc.) is generated.

  The three-dimensional model image generation unit 22 generates a three-dimensional model image M3 (see FIG. 2 and the like) of the subject.

  For example, the three-dimensional model image generation unit 22 captures an endoscope image EI generated by the endoscope image generation unit 21 and a captured image that is a source for generating the endoscope image EI. The position information and the direction information detected by the position / orientation detection unit 13 are acquired for a plurality of frames via the control unit 27. The three-dimensional model image generation unit 22 generates the three-dimensional model image M3 while matching the positional relationship of the endoscopic images EI of a plurality of frames based on the position information and orientation information of each frame. . In this case, the three-dimensional model image M3 (see FIG. 2 and the like) is gradually constructed as the observation proceeds.

  However, the method of generating the three-dimensional model image M3 by the three-dimensional model image generation unit 22 is not limited to this. For example, if the endoscopic examination of the subject is performed for the second time or later, and the 3D model data generated by the past endoscopy is already recorded in the database 3, the 3D model data May be used. Alternatively, when data acquired by performing contrast CT imaging on the subject is already recorded in the database 3, the three-dimensional model image M3 may be generated using the contrast CT data. .

  The three-dimensional model image M3 is, for example, an image when a three-dimensional subject image is viewed from a certain line-of-sight direction, and the line-of-sight direction can be changed (that is, with the change of the line-of-sight direction, three-dimensional The model image M3 is rotated).

  The image processing unit 23 includes a first image processing unit 23a.

  The first image processing unit 23a uses a predetermined object (in this embodiment, the calculus 42 (see FIG. 2) as an example of the predetermined object). However, the first image processing unit 23a may be a diseased part or other object. The position indicated by the position information (which is not limited to a specific object) (the position information of the predetermined object is calculated by the position calculation unit 26 described later) is 3 The first image processing to be displayed on the three-dimensional model image M3 generated by the three-dimensional model image generation unit 22 is performed. Specifically, the first image processing unit 23a adds the calculus mark MK1 to the position indicated by the position information of the calculus 42 as the predetermined target on the three-dimensional model image M3 (that is, marking). Do. Therefore, the first image processing unit 23a is an object marking unit.

  Here, the first image processing unit 23a performs the process of displaying the position indicated by the position information of the predetermined object on the three-dimensional model image M3 as the first image process, but the present invention is not limited to this. It is not something. For example, as the first image processing, the first image processing unit 23a displays the position indicated by the position information of a predetermined object on a two-dimensional developed image obtained by two-dimensionally developing the three-dimensional model image M3. You may do.

  The image synthesis output unit 24 synthesizes the endoscope image EI generated by the endoscope image generation unit 21 and the three-dimensional model image M3 generated by the image processing unit 23 into one image and displays it. Output to device 4.

  The illumination control unit 25 controls on / off of the illumination light emitted by the illumination unit 12 and the amount of light. Here, the illumination control unit 25 may be a light source device and the illumination unit 12 may be a light guide or the like, the illumination control unit 25 may be a light emission control circuit, and the illumination unit 12 may be a light source such as an LED. It doesn't matter.

  The control unit 27 controls the entire processing system 2 and also controls the endoscope 1. The endoscope image generation unit 21, the three-dimensional model image generation unit 22, the image processing unit 23, The image composition output unit 24 and the illumination control unit 25 are connected.

  The control unit 27 includes a position calculation unit 26. The position calculation unit 26 is based on the position information and orientation information of the endoscope 1 when the predetermined object existing inside the subject is imaged by the imaging unit 11 and the three-dimensional model image M3. The position information of the object is calculated.

  The database 3 is connected to the processing system 2 via, for example, an in-hospital system. As described above, the contrast CT data of the subject, which is the original data when generating the three-dimensional model image, The three-dimensional model data of the subject generated by the mirror examination is recorded.

  The display device 4 displays an image including the endoscopic image EI and the three-dimensional model image M3 output from the image composition output unit 24.

  Next, FIG. 2 is a diagram illustrating a display example of the display device 4 that displays the endoscopic image EI and the three-dimensional model image M3.

  The display device 4 is provided with an endoscope image display unit 4a and a three-dimensional model image display unit 4b.

  An endoscopic image EI is displayed on the endoscopic image display unit 4a. This endoscopic image EI is an endoscopic image based on a captured image obtained by capturing an image of a calculus 42, which is a predetermined object, present inside the kidney cup 41, which is the subject, by the imaging unit 11. It is an image generated by the generation unit 21. In FIG. 2, the calculus 42 is shown in the center of the endoscopic image EI, and the calculus 42 is positioned almost on the optical axis of the optical system of the imaging unit 11.

  A 3D model image M3 generated by the 3D model image generation unit 22 is displayed on the 3D model image display unit 4b. Since FIG. 2 shows the three-dimensional model image M3 that is gradually constructed as the observation proceeds as described above, the existence of the unobserved region UOR indicates that there is a connection portion to the unobserved region UOR. Are displayed in different colors (for example, color (hue, saturation, brightness), pattern, or combination of color and pattern). To give some specific examples, the connection portion to the unobserved region UOR is displayed in red hue (red display), and the saturation of the connection portion to the unobserved region UOR is reduced (monochrome display). For example, the brightness of the connection portion to the unobserved area UOR is increased (highlighted display).

  On the three-dimensional model image M3, a calculus mark MK1 added by the first image processing unit 23a and an endoscope mark EL are displayed. In the example shown in FIG. 2, the calculus mark MK1 is, for example, spherical, and the endoscope mark EL has a curved shape along the curved shape of the insertion portion of the endoscope 1. The transmission brightness of the three-dimensional model image M3 is adjusted so that the calculus mark MK1 located inside the three-dimensional model image M3 and the endoscope mark EL can be visually recognized.

  FIG. 3 is a diagram illustrating a display example of a calculus position in the three-dimensional model image M3. As shown in FIG. 3, the endoscope mark EL need not be displayed. Alternatively, the user may switch on / off the display of the endoscope mark EL as desired.

  Here, calculation of position information of a predetermined object by the position calculation unit 26 will be described with reference to FIGS. 4 and 5. For example, there are the following two methods for calculating position information by the position calculation unit 26 using the three-dimensional model image M3 and the position information and orientation information of the endoscope 1.

  The first method is a simple method for calculating the position information of the predetermined object on the assumption that the predetermined object is located at the center of the endoscopic image EI.

  In addition, the second method is a method of calculating the position information of the predetermined object more accurately based on how much the predetermined object is displaced in which direction from the central portion of the endoscopic image EI. .

  Although the second method can obtain more accurate position information than the first method, it is necessary to detect the calculus 42 in the endoscopic image EI by image recognition, and the calculation processing becomes complicated. Therefore, when the calculus 42 is located at the center of the endoscopic image EI, it is sufficient to use the first method, and when the calculus 42 cannot be recognized, the first method is used.

  The first method and the second method are selectively used by, for example, the calculus 42 by the first method when a trigger signal for marking is generated by the user performing an operation from the operation unit 14 or the like. The position information of the calculus 42 is generated by the second method when the trigger signal for marking is generated by the discovery of the calculus 42 by the image recognition processing by the image processing unit 23. , Etc.

  The first method will be described with reference to FIG. FIG. 4 is a diagram showing an example of the positional relationship between the endoscope 1 and the calculus 42 in the hollow organ.

  In the example shown in FIG. 4, there is a calculus 42 as a predetermined object on the optical axis of the optical system of the imaging unit 11 of the endoscope 1.

  The position calculation unit 26 determines a position of a point where a straight line extended from the position indicated by the position information of the endoscope 1 in the direction indicated by the azimuth information of the endoscope 1 intersects the three-dimensional model image M3. It is calculated as position information of the calculus 42 that is the object.

  Specifically, the position calculation unit 26 calculates the distal end portion position P of the endoscope 1 represented by three-dimensional coordinates based on the position information acquired from the position / orientation detection unit 13. Further, the position calculation unit 26 calculates the optical axis direction vector VD of the optical system of the imaging unit 11 of the endoscope 1 based on the direction information acquired from the position / direction detection unit 13. Then, a point where a straight line extending along the optical axis direction vector VD intersects the inner wall of the kidney cup 41 in the three-dimensional model image M3 is calculated as position information of the calculus 42 from the distal end position P of the endoscope 1. To do.

  Next, the second method will be described with reference to FIG. FIG. 5 is a diagram illustrating another example of the positional relationship between the endoscope 1 and the calculus in the hollow organ.

  In the example illustrated in FIG. 5, a calculus 42 that is a predetermined object is present at a position shifted from the optical axis of the optical system of the imaging unit 11 of the endoscope 1.

  The position calculation unit 26 is based on the position of the calculus 42 in the captured image acquired by imaging the calculus 42 that is a predetermined object by the imaging unit 11 and the direction indicated by the orientation information of the endoscope 1. The line-of-sight direction toward the calculus 42 from the position indicated by the position information of the endoscope 1 is calculated, and a straight line extending in the line-of-sight direction from the position indicated by the position information of the endoscope 1 is the three-dimensional model image M3. The position of the intersecting point is calculated as position information of the calculus 42.

  Specifically, when it is assumed that there is a projection surface VEI of the endoscope image at a predetermined distance from the distal end position P of the endoscope 1, the distal end position P of the endoscope 1 changes to the optical axis direction vector VD. The straight line extending along the center passes through the center O of the projection plane VEI of the endoscopic image.

  On the other hand, it is assumed that the calculus image V42 on the projection plane VEI of the endoscopic image is at a position shifted from the center O of the projection plane VEI of the endoscopic image. In this case, the position calculating unit 26 determines the coordinates of the calculus image V42 with respect to the center O of the projection plane VEI of the endoscopic image and a predetermined distance from the distal end position P of the endoscope 1 to the projection plane VEI of the endoscopic image. Based on the distance, a line-of-sight vector VE from the distal end position P of the endoscope 1 toward the calculus image V42 is calculated. Then, a point where a straight line extending along the line-of-sight vector VE from the distal end position P of the endoscope 1 intersects the inner wall of the kidney cup 41 in the three-dimensional model image M3 is calculated as position information of the calculus 42.

  As described above, the first method of position information calculation by the position calculation unit 26 is a method using the three-dimensional model image M3 and the position information and orientation information of the endoscope 1. In addition, the second method of calculating the position information by the position calculating unit 26 is generated based on the three-dimensional model image M3, the position information and orientation information of the endoscope 1, and the captured image captured by the image capturing unit 11. And an endoscopic image EI.

  However, the calculation method of the position information by the position calculation unit 26 is not limited to these. For example, the position information of the calculus 42 is calculated using a stereo measurement technique using parallax, based on a plurality of captured images (or endoscopic images EI generated based on the captured images) having different shooting positions. It doesn't matter if you do.

  Next, FIG. 6 is a flowchart showing processing during observation of the endoscope apparatus.

  When this process is started, it is determined whether or not a new calculus 42 has been found during the loop from step S1 to step S7 (step S1).

  The discovery of the calculus 42 is performed by the user or automatically by the image recognition processing of the image processing unit 23.

  The determination to be a calculus 42 in the case of performing automatic determination is performed, for example, when the outline of the calculus 42 is entirely included in the endoscopic image EI. Furthermore, when performing automatic determination, when a calculus 42 is found outside a sphere having a predetermined radius r from the position of the calculus 42 that has already been found, it is determined that a new calculus 42 has been found. .

  By performing such processing, only one calculus mark MK1 is displayed even if there are a plurality of calculi 42 in a sphere having a predetermined radius r, and the display becomes complicated. Can be made easier to see.

  And when the new calculus 42 is discovered in step S1, the control part 27 produces | generates a trigger signal (step S2). The trigger signal is generated by the control unit 27 based on a marking operation performed by the user on the operation unit 14 or the like, or based on an image recognition result that a new calculus 42 has been found by the image processing unit 23.

  Then, the control unit 27 acquires the position information and the direction information of the distal end portion of the endoscope 1 at the time when the trigger signal is generated from the position and direction detection unit 13 (step S3), and the trigger signal is generated based on the user operation. If the trigger signal is generated based on the image recognition processing by the first method described above, the line-of-sight vector VE is calculated by the second method described above (step S4). Here, in the case of the first method, it corresponds to approximation when the optical axis direction vector VD coincides with the line-of-sight vector VE.

  Then, the coordinates of the point where the straight line extending from the distal end position P of the endoscope 1 in the direction of the line-of-sight vector VE intersects the inner wall of the kidney cup 41 in the three-dimensional model image M3 is calculated as position information of the calculus 42. (Step S5).

  The first image processing unit 23a adds, for example, a spherical stone mark MK1 to the position of the coordinates of the intersection calculated in step S5 on the three-dimensional model image M3, and also adds an endoscope mark EL as necessary. After that, the added three-dimensional model image M3 is output to the image composition output unit 24. However, for the endoscope mark EL, a real-time shape is added regardless of the presence or absence of a trigger signal.

  Here, since the three-dimensional model image M3 used by the first image processing unit 23a is an image in which the calculus mark MK1 has already been added in the loop of steps S1 to S6, the loop of steps S1 to S6 is repeated. Accordingly, the number of calculus marks MK1 will increase. However, any calculus mark MK1 may be canceled as desired in accordance with a user operation.

  The image synthesis output unit 24 synthesizes the endoscope image EI from the endoscope image generation unit 21 and the three-dimensional model image M3 from the image synthesis output unit 24 as shown in FIG. Output to. As a result, the three-dimensional model image M3 to which the calculus mark MK1 (further endoscope mark EL if necessary) is added is displayed on the display device 4 together with the endoscope image EI (step S6).

  When the process of step S6 is performed, or when no new calculus has been found in step S1, it is determined whether or not to end the endoscopic observation (step S7).

  Here, when it is determined that the endoscopic observation is not yet finished, the process returns to step S1 and the above-described processing is repeated. On the other hand, when it is determined that the endoscopic observation is finished, Finish the process.

  According to the first embodiment, since the position information of the predetermined object is calculated based on the position information and orientation information of the endoscope 1 and the three-dimensional model image M3, the endoscope The position of a predetermined object found during observation can be accurately calculated.

  Further, since the mark is displayed at the position of the predetermined object obtained by the calculation on the three-dimensional model image M3 or on the two-dimensional development image obtained by two-dimensionally developing the three-dimensional model image M3, a predetermined value is displayed. The position of the object can be accurately marked.

  In this way, it becomes possible to perform accurate marking of an object without requiring CT imaging before performing endoscopic observation, and the user can accurately grasp the position of a predetermined object. .

  Furthermore, since it becomes possible to compare the stone calculated in the current procedure / examination with the stone calculated in the next procedure / examination, more accurate diagnosis and treatment can be performed.

  If the three-dimensional model image M3 to which the calculus mark MK1 is added is used, the patient can be explained easily and accurately.

  Further, when the position of a point where a straight line extending from the distal end position P of the endoscope 1 in the direction of the optical axis direction vector VD intersects the three-dimensional model image M3 is calculated as position information of a predetermined object. Even when the image of the predetermined object cannot be recognized, the calculus mark MK1 or the like can be displayed.

On the other hand, a line-of-sight vector VE toward the predetermined object is calculated based on the position of the predetermined object in the captured image and the optical axis direction vector VD, and the line-of-sight vector VE from the distal end position P of the endoscope 1 is calculated. When the position of the point where the straight line extending in the direction intersects with the three-dimensional model image M3 is calculated as the position information of the predetermined object, the stone mark MK1 and the like can be displayed at a more accurate position. Become.
[Embodiment 2]

  7 to 10 show the second embodiment of the present invention, and FIG. 7 is a block diagram showing the configuration of the endoscope apparatus.

  In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  In the first embodiment described above, the calculus mark MK1 is added to the three-dimensional model image M3 or the like according to the trigger signal. However, in the present embodiment, the calculus mark MK1 and the endoscope 1 are changed according to the trigger signal. For example, one of the tip mark MK2 indicating the tip position is added to the three-dimensional model image M3 or the like.

  First, in the endoscope apparatus according to the present embodiment, as illustrated in FIG. 7, the control unit 27 includes an image processing switching unit 31 in addition to the position calculation unit 26, and the image processing unit 23 includes the first image processing unit. In addition to 23a, a second image processing unit 23b is provided. Here, the first image processing unit 23a and the second image processing unit 23b may be configured as independent and different circuits, but may be configured to share the same circuit.

  The second image processing unit 23b performs second image processing for displaying a predetermined area obtained from the position information of the endoscope 1 on, for example, the three-dimensional model image M3.

  The image processing switching unit 31 performs image processing performed on the three-dimensional model image M3 generated by the three-dimensional model image generating unit 22, the first image processing by the first image processing unit 23a and the second image processing unit. A process of switching to one of the second image processing by 23b is performed. The image processing switching by the image processing switching unit 31 is performed based on a switching operation by the user from the operation unit 14, for example.

  The predetermined area displayed by the second image processing unit 23b described above includes the position of the distal end portion of the endoscope 1 (the area of area 0 is a point), the kidney cup including the position of the distal end portion of the endoscope 1, Any of the kidney cup not including the position of the distal end portion of the endoscope 1 may be used.

  FIG. 8 is a diagram illustrating an example of marking the distal end portion position P of the endoscope 1.

  In the example shown in FIG. 8, a tip mark MK2 corresponding to the tip position P (see FIGS. 4, 5, etc.) of the endoscope 1 is added to the three-dimensional model image M3 together with the endoscope mark EL. Has been. Here, the tip mark MK2 shown in FIG. 8 is a hexahedron, which is different from the spherical shape of the calculus mark MK1 described above. As described above, the calculus mark MK1 added in the first image processing and the tip mark MK2 added in the second image processing are display modes such as the size, shape, color, pattern, or combination thereof. Should be different.

  FIG. 9 is a diagram illustrating an example of marking a predetermined region obtained from the distal end position P of the endoscope 1.

  In the example shown in FIG. 9, a tip mark MK2 'is added to the three-dimensional model image M3 so as to fill the area of the kidney cup including the tip position P of the endoscope 1.

  Next, FIG. 10 is a flowchart illustrating processing during observation of the endoscope apparatus.

  When this process is started, the control unit 27 determines whether the first image process or the second image process is selected by the image process switching unit 31 in accordance with the user's switching operation (step S2). S11).

  If it is determined that the first image processing is selected, steps S1 to S6 for adding the calculus mark MK1 to the three-dimensional model image M3, similar to those described with reference to FIG. Perform the process.

  On the other hand, when it is determined in step S11 that the second image processing is selected, it is determined whether or not a marking operation by the user from the operation unit 14 or the like has been performed (step S12).

  Here, when it is determined that the marking operation has been performed, the control unit 27 generates a trigger signal (step S13).

  Then, the control part 27 acquires the positional information on the front-end | tip part of the endoscope 1 at the time of a trigger signal being generated from the position and orientation detection part 13 (step S14).

  The second image processing unit 23b adds, for example, a hexahedral tip mark MK2 to the position of the coordinates indicated by the position information acquired in step S14 on the three-dimensional model image M3, and then adds the three-dimensional model image after the addition. M3 is output to the image composition output unit 24.

  Note that the tip mark MK2 of the endoscope 1 may also be displayed on a two-dimensional developed image obtained by two-dimensionally developing the three-dimensional model image M3 instead of being displayed on the three-dimensional model image M3. Is the same as in the case of the calculus mark MK1 of the first embodiment described above.

  In order to perform image composition as described above, the image composition output unit 24 displays the three-dimensional model image M3 to which the new tip mark MK2 has been added together with the endoscope image EI on the display device 4 ( Step S15).

  When the process of step S6 described above is performed, when no new calculus is found in step S1, when the process of step S15 is performed, or when it is determined that the marking operation is not performed in step S12 In step S7, it is determined whether or not to end the endoscopic observation by performing the process in step S7. If it is determined that the endoscopic observation is not yet finished, the process returns to step S11 and the above-described processing is repeated.

  Thus, as the loop of steps S11 to S7 is repeated, the calculus mark MK1 and the tip mark MK2 increase (the endoscope mark EL displays a real-time shape as described above. ). As described above, any calculus mark MK1 and any tip mark MK2 may be canceled as desired according to a user operation.

  On the other hand, if it is determined in step S7 that the endoscopic observation is to be terminated, this process is terminated.

  When performing a procedure or an inspection, the marking can be used not only for indicating the position of the calculus 42 but also for other purposes. For example, when marking a kidney cup, it is desired to leave information that there is no remaining stone in the kidney cup. Therefore, by enabling the second image processing for marking the position of the distal end portion of the endoscope 1, it is possible to meet such a demand.

  Further, for example, when there is only one calculus 42 that is not very large, a treatment may be performed in which the calculus 42 is crushed and taken out as it is. On the other hand, when the calculus 42 is large or there are a plurality of calculi 42, the position of the calculus 42 is memorized in advance, moved to a place where the calculus 42 can be easily taken out, and then the calculus 42 is crushed at the gathered place. In some cases, a removal procedure is performed. For this reason, when the position of the calculus 42 is memorized in advance, the calculus mark MK1 indicating the accurate position of the calculus 42 is useful. Moreover, when marking the place where the calculus 42 is put together, the tip mark MK2 can be used. Therefore, the configuration of the present embodiment can be effectively used for such applications.

According to the second embodiment, the first image processing for displaying the calculus mark MK1 and the like and the second image processing for displaying the tip mark MK2 and the like have the same effects as the first embodiment. Therefore, the tip mark MK2 can be used adaptively for marking of the goblet without the remaining stone, marking of the place where the calculus 42 is gathered, and the like. Thereby, it becomes possible to perform a procedure and a test | inspection more efficiently.
[Embodiment 3]

  11 to 14 show Embodiment 3 of the present invention, and FIG. 11 is a block diagram showing a configuration related to the control unit 27 of the endoscope apparatus.

  In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  In the present embodiment, a type of a predetermined object, specifically, a calculus 42 is determined, and the position of the calculus 42 is displayed in a different display mode according to the determined type.

  In the endoscope apparatus according to the present embodiment, as shown in FIG. 11, the control unit 27 includes a determination unit 35 in addition to the position calculation unit 26.

  The determination unit 35 determines the type of the predetermined object based on the captured image acquired by imaging the predetermined object by the imaging unit 11.

  In addition, the first image processing unit 23a changes the display mode when displaying the position indicated by the position information of the predetermined object according to the type of the predetermined object determined by the determination unit 35.

  Specifically, an example in the case where the calculus 42 is classified into an embedded calculus and a floating calculus will be described with reference to FIGS. Here, the buried stone is embedded in the subject, and the movement is integrated with the subject.For example, the buried stone is a floating stone that can move independently of the subject. Point to.

  First, FIG. 13 is a flowchart showing processing during observation of the endoscope apparatus.

  When this process is started, the above-described steps S1 to S5 are performed. However, in order to determine the type of the calculus 42 by the determination unit 35, image recognition of the calculus 42 is necessary. Therefore, here, a trigger signal is generated based on, for example, image recognition processing, and the line-of-sight vector VE is the second line of sight described above. (However, image recognition of the calculus 42 may be performed in accordance with a trigger signal generated based on a user operation).

  And after performing the process of step S5, the determination part 35 performs the process which determines the kind of calculus 42 (step S21).

  Here, FIG. 14 is a flowchart showing processing for determining the type of the calculus 42 in FIG.

  Upon entering this process, the determination unit 35, for example, a plurality of endoscopic images generated by the endoscopic image generation unit 21 based on a plurality of captured images obtained at different acquisition points acquired by the imaging unit 11. The movement vector V1 (see FIG. 12) of the calculus 42 as shown in FIG. 12 is calculated using the EI (step S31), and the movement vector V2 of the inner wall of the kidney cup 41 other than the calculus 42 (see FIG. 12). Is calculated (step S32).

  FIG. 12 is a diagram illustrating an example of the movement vector V1 of the calculus 42 and the movement vector V2 other than the calculus 42 obtained based on a plurality of captured images at different acquisition points.

  And the determination part 35 determines whether the movement vector V1 and the movement vector V2 correspond (step S33).

  Here, when the movement vector V1 and the movement vector V2 match (within an allowable range error), the determination unit 35 determines that the calculus 42 to be subjected to the type determination is an embedded calculus ( Step S34).

  On the other hand, when the movement vector V1 and the movement vector V2 do not coincide (exceeding an allowable range error), the determination unit 35 determines that the calculus 42 to be subjected to the type determination is a floating calculus ( Step S35).

  When the process of step S34 or step S35 is thus performed and the result of the type determination is obtained, the process returns to the process of FIG.

  Returning to the processing of FIG. 13, the first image processing unit 23a displays the calculus mark MK1 when displaying the position according to whether the calculus 42 as the predetermined object is an embedded calculus or a floating calculus. Different aspects. Specifically, the size, shape, color, pattern, or combination thereof of the stone mark MK1 is varied depending on whether the stone 42 is an buried stone or a floating stone.

  The image synthesis output unit 24 synthesizes the endoscope image EI from the endoscope image generation unit 21 and the three-dimensional model image M3 from the image synthesis output unit 24 as shown in FIG. Output to. Thereby, the three-dimensional model image M3 to which the stone mark MK1 or the like corresponding to the type of the stone 42 is added is displayed on the display device 4 together with the endoscope image EI (step S6A).

  Thereafter, the determination in step S7 is performed, and if not completed, the process returns to step S1, and if completed, the process is terminated, as in the process shown in FIG. 6 of the first embodiment.

  As a modification of the third embodiment, the size of a predetermined object may be determined. In this case, the determination unit 35 determines the size of the predetermined object, and the first image processing unit 23a changes the display mode when displaying the position according to the size of the predetermined object. .

  Specifically, the distance from the distal end position P of the endoscope 1 shown in FIG. 5 to the calculus 42 can be calculated using the three-dimensional model image M3. On the other hand, from the contour of the calculus 42, the size of the calculus 42 image in the endoscopic image EI can be calculated. Therefore, the actual size of the calculus 42 can be calculated based on the distance and the size on the image.

  Thus, when the actual size of the calculus 42 is calculated by the determination unit, the first image processing unit 23a displays the calculus mark MK1 according to the size, specifically, the size, shape, and color. , A pattern, or a combination of these may be displayed differently.

  Of course, in the configuration as shown in the third embodiment, a configuration capable of switching between the first image processing and the second image processing as described in the second embodiment may be added. .

  According to the third embodiment, the same effects as those of the first and second embodiments described above are obtained, and the type of the predetermined object is determined based on the captured image, and the display mode is determined according to the determined type. Therefore, it is possible not only to accurately know the position of the calculus 42 but also to understand what kind of calculus it is.

  Specifically, when determining whether the calculus 42 is an buried calculus or a floating calculus based on the difference in movement vector from the kidney cup, it is possible to recognize which is only by looking at the calculus mark MK1. Can do.

  Further, when determining the size of the calculus 42, it is possible to recognize the size of the calculus 42 only by looking at the calculus mark MK1.

  In addition, each part mentioned above may be comprised as a circuit. Any circuit may be mounted as a single circuit or a combination of a plurality of circuits as long as it can perform the same function. Furthermore, an arbitrary circuit is not limited to being configured as a dedicated circuit for performing a target function, and may be configured to perform a target function by causing a general-purpose circuit to execute a processing program. .

  In the above description, the endoscope apparatus has been mainly described. However, an operation method for operating the endoscope apparatus as described above may be used, or a process for causing a computer to perform the same process as the endoscope apparatus. It may be a program, a non-temporary recording medium readable by a computer for recording the processing program, or the like.

  Furthermore, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various aspects of the invention can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Endoscope 2 ... Processing system 3 ... Database 4 ... Display apparatus 4a ... Endoscope image display part 4b ... Three-dimensional model image display part 11 ... Imaging part 12 ... Illumination part 13 ... Position direction detection part 14 ... Operation part DESCRIPTION OF SYMBOLS 21 ... Endoscopic image generation part 22 ... Three-dimensional model image generation part 23 ... Image processing part 23a ... 1st image processing part 23b ... 2nd image processing part 24 ... Image composition output part 25 ... Illumination control part 26 ... Position calculation Unit 27 ... Control unit 31 ... Image processing switching unit 35 ... Determination unit 41 ... Kidney cup 42 ... Stone EI ... Endoscopic image EL ... Endoscopic mark M3 ... Three-dimensional model image MK1 ... Stone mark MK2, MK2 '... Tip Mark O: Center of projection plane of endoscopic image P: End position of endoscope UOR: Unobserved region V1, V2 ... Movement vector V42 ... Stone image of endoscopic image VD ... Optical axis direction vector VE ... Line of sight Baek Projection plane Le VEI ... endoscopic image

Claims (9)

An endoscope having an imaging unit configured to capture an image of a subject having a three-dimensional shape and acquire a captured image;
A three-dimensional model image generation unit configured to generate a three-dimensional model image of the subject;
The position of the predetermined object based on the position information and orientation information of the endoscope when the predetermined object existing inside the subject is imaged by the imaging unit, and the three-dimensional model image. A position calculator configured to calculate information;
The first image processing is performed to display the position indicated by the position information of the predetermined object on the three-dimensional model image or on a two-dimensional developed image obtained by two-dimensionally developing the three-dimensional model image. A first image processing unit configured to:
An endoscope apparatus comprising:   The position calculating unit is configured to determine a position of a point where a straight line extending in a direction indicated by the azimuth information of the endoscope from a position indicated by the position information of the endoscope intersects the three-dimensional model image. The endoscope apparatus according to claim 1, wherein the position information is calculated as the position information of the target object.   The position calculation unit is based on a position of the predetermined object in the captured image acquired by imaging the predetermined object by the imaging unit and a direction indicated by the azimuth information of the endoscope. Calculating a line-of-sight direction from the position indicated by the position information of the endoscope toward the predetermined object, and a straight line extending in the line-of-sight direction from the position indicated by the position information of the endoscope is The endoscope apparatus according to claim 1, wherein the position of a point that intersects the three-dimensional model image is calculated as the position information of the predetermined object. Second image processing is performed to display a predetermined region obtained from position information of the endoscope on the three-dimensional model image or on a two-dimensional developed image obtained by two-dimensionally developing the three-dimensional model image. A second image processing unit configured to:
An image processing switching unit configured to switch to any one of the first image processing by the first image processing unit and the second image processing by the second image processing unit;
The endoscope apparatus according to claim 1, further comprising:   The said 2nd image processing displays the said predetermined area | region calculated | required from the positional information on the said endoscope in a different aspect from the display by the said 1st image processing. Endoscopic device. A determination unit configured to determine the type of the predetermined object based on the captured image acquired by imaging the predetermined object by the imaging unit;
The first image processing unit changes a display mode when displaying the position indicated by the position information of the predetermined object according to the type of the predetermined object determined by the determination unit. The endoscope apparatus according to claim 1, configured as described above. The predetermined object is a calculus inside the subject,
The determination unit determines that a stone movement vector obtained based on a plurality of the captured images at different acquisition points and a movement vector other than a stone match an embedded stone, and if not, a floating stone Is configured to determine that
The said 1st image process part changes the display mode at the time of displaying a position according to whether the said predetermined target object is an embedded stone or a floating stone. Endoscope device. The determination unit is configured to determine a size of the predetermined object;
The endoscope apparatus according to claim 6, wherein the first image processing unit changes a display mode when displaying a position in accordance with a size of the predetermined object. The imaging unit of the endoscope captures a captured image by imaging the inside of the subject having a three-dimensional shape,
A three-dimensional model image generation unit generates a three-dimensional model image of the subject;
Based on the position information and azimuth information of the endoscope and the three-dimensional model image when the position calculation unit images a predetermined object existing inside the subject by the imaging unit. Calculate location information for
The first image processing unit displays the position indicated by the position information of the predetermined object on the three-dimensional model image or on a two-dimensional development image obtained by two-dimensionally developing the three-dimensional model image. An operation method of an endoscope apparatus, characterized by performing first image processing.

Images