WO2020194942A1 - Blood vessel recognition device, blood vessel recognition method, and blood vessel recognition system - Google Patents

Abstract

This blood vessel recognition device is provided with: an image input unit that is connected to an imaging device for capturing an image of an operation field and that inputs a captured image of the operation field from the imaging device; an image processing unit that recognizes a blood vessel reflected in the captured image on the basis of the inputted captured image; and an image output unit that generates a composite image obtained by superimposing, onto the captured image, information for teaching about the recognized blood vessel and that outputs the generated composite image to a monitor.

Description

Vascular recognition device, vascular recognition method and vascular recognition system

The present disclosure relates to a vascular recognition device, a vascular recognition method, and a vascular recognition system that recognize a vascular image reflected in a captured image of a surgical field.

Conventionally, when performing angiography in digital subtraction angiography (DSA), it is difficult to observe the blood vessel because the contrast medium is diluted only by displaying the live image obtained after injecting the contrast medium into the blood vessel. Therefore, in Patent Document 1, a subtraction image in which only blood vessels appear and a mask image or live image in which at least bones appear are used, and only the subtraction image is emphasized and the subtraction image and the mask image or live image are added. A DSA post-treatment blood vessel highlighting device is disclosed that obtains an image in which the positional relationship between a blood vessel and a bone can be clearly identified.

Japanese Patent Application Laid-Open No. 63-139532

Here, for example, the liver is known as an organ composed of the above-mentioned blood vessels intricately intertwined in a tissue. The hepatic veins or Glisson's capsule in the liver run in a mesh pattern while crossing intricately, and the running pattern varies from person to person. In addition, there are blood vessels (an example of a vessel) that are so thin that it is difficult to visualize them by CT scan (Computed Tomography) or MRI (Magnetic Resonance Imaging) processing before surgery, and during surgery, the pulse is caused by deformation of organs. The position of the tube may change slightly. The premonition that the vessel appears on the dissection surface to be excised during surgery and its cognitive ability largely depend on the observation eye (so-called tacit knowledge) cultivated by a skilled doctor with considerable experience, and the vessel such as the liver There is a problem that the difficulty of surgery for complicated and intricate organs is increasing.

This disclosure was devised in view of the above-mentioned conventional situation, and reproduces the vascular recognition function equivalent to the implicit knowledge of a skilled doctor who quickly recognizes the vascular appearing in the surgical field in real time and with high accuracy. An object of the present invention is to provide a vascular recognition device, a vascular recognition method, and a vascular recognition system that notify the location of a vascular appearing on a detached surface and support the provision of safe and secure surgery.

The present disclosure is a vessel recognition device connected to an imaging device that images a surgical field, and is based on an image input unit that inputs an image captured by the surgical field from the imaging device and the input image. An image processing unit that recognizes the vessels reflected in the captured image and a composite image in which the recognized information teaching the vessels is superimposed on the captured image are generated, and the generated composite image is used as a monitor. Provided is a vessel recognition device including an image output unit for outputting.

Further, the present disclosure is a vascular recognition method in a vascular recognition device connected to an imaging device that images the surgical field, and is a step of inputting an image captured in the surgical field from the imaging device. A step of recognizing a vessel reflected in the captured image based on the captured image, a step of generating a composite image in which information for teaching the recognized vessel is superimposed on the captured image, and the generated said Provided is a vascular recognition method having a step of outputting a composite image to a monitor.

Further, the present disclosure is a vessel recognition system in which an imaging device for imaging a surgical field and a vessel recognition device are connected to each other, and the vessel recognition device is an image captured of the surgical field from the imaging device. Is input, the vessel reflected in the captured image is recognized based on the input captured image, and a composite image in which the information for teaching the recognized vessel is superimposed on the captured image is generated and generated. Provided is a vessel recognition system that outputs the combined image to a monitor.

According to the present disclosure, the recognition function of the vessel, which is equivalent to the tacit knowledge of a skilled doctor who quickly recognizes the vessel appearing in the surgical field, is reproduced in real time and with high accuracy, and the location of the vessel appearing on the incision surface is notified. However, we can support the provision of safe and secure surgery.

The figure which shows the schematic configuration example of the medical image projection system which concerns on Embodiment 1. The figure which shows the concrete structure of the medical image projection system A flowchart showing an example of an operation procedure of the medical image projection system according to the first embodiment. The figure which shows the image of the organ which the vessel specific image is projected on the vessel detected by the image recognition function of the vessel. The figure which shows the schematic configuration example of the endoscope system which concerns on Embodiment 2. Diagram showing an overview of the endoscopic system Flow chart showing an example of operating procedure of the endoscope system according to the second embodiment The figure which shows the registration content of the scene determination table which shows the scene at the time of performing the laparotomy which concerns on the modification 2 of Embodiment 1. The figure which shows the screen of the monitor which the tube specific image which concerns on the modification 3 of Embodiment 1 is superposed on the organ and is displayed. The figure which shows the screen of the monitor which the ICG fluorescence image which concerns on the modification 4 of Embodiment 1 and the vessel specific image partially overlap and are superposed on the captured image including an organ. A diagram schematically showing an ICG fluorescence image and a vessel specific image displayed on a monitor when the ICG fluorescence image and the vessel specific image according to the modified example 5 of the first embodiment overlap with an organ included in the captured image.

Hereinafter, embodiments in which the configuration and operation of the vessel recognition device, the vessel recognition method, and the vessel recognition system according to the present disclosure will be specifically disclosed will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
First, as an example of the vascular recognition system according to the first embodiment, a medical imaging projection system (MIPS) will be illustrated and described. FIG. 1 is a diagram showing a schematic configuration example of the medical image projection system 5 according to the first embodiment. The medical image projection system 5 projects and maps the fluorescent image of the tumor part where fluorescent agents such as ICG (Indocyanine Green) are accumulated directly to the organ, and the surgeon such as a doctor keeps an eye on the surgical field to prevent blood flow and ischemia. It is a system that visualizes the boundaries of regions in real time. Here, the vessels include blood vessels through which blood flows in the body and lymph vessels through which lymph fluid flows. The Grisson capsule is a tissue that contains blood vessels and bile ducts that flow through the liver. In the first embodiment, the Grisson capsule is treated in the same manner as the vessel.

The medical image projection system 5 includes an image pickup irradiation device 20, a camera control unit 10 (CCU: Camera Control Unit) as an example of a vessel recognition device, and a monitor 30.

The imaging irradiation device 20 irradiates the surgical field with white light (that is, visible light), receives the reflected light reflected by the affected part (for example, an organ) of the subject (for example, a person), and captures a visible light image. To do. The imaging irradiation device 20 irradiates infrared light to excite the fluorescent agent accumulated in the affected area or the like, receives infrared light including fluorescence generated by the fluorescence emission of the fluorescent agent, and images a fluorescent image. Further, the image pickup irradiation device 20 projects a tube specific image that teaches the position of the tube on the affected part (for example, an organ). The image pickup irradiation device 20 includes a camera head 21, a projector 22, and a light source 23.

The camera head 21 as an example of the imaging device includes a visible light sensor unit 24A including a visible light image sensor 25A and an IR cut filter 26A, and an infrared light sensor including an infrared light image sensor 25B and a visible light cut filter 26B. Includes unit 24B.

The IR cut filter 26A blocks (cuts) the excitation light (IR light: Infrared Light) having an IR wavelength band that is irradiated from the light source 23 to the affected area (for example, an organ) and reflected by the affected area. The visible light image sensor 25A receives visible light that has passed through the IR cut filter 26A (that is, visible light reflected by the affected area) and captures a visible light image.

The visible light cut filter 26B irradiates the affected area (for example, an organ) from the light source 23 and blocks (cuts) the visible light reflected by the affected area. Further, the visible light cut filter 26B blocks (cuts) not only visible light but also IR light in the wavelength band of excitation light (for example, 690 nm to 820 nm). The infrared light image sensor 25B receives fluorescence that has passed through the visible light cut filter 26B (that is, fluorescence emitted based on the fluorescence emission of the fluorescent agent accumulated in the affected area), and captures a fluorescence image.

The camera head 21 captures, for example, a visible light image and a fluorescent image in a time-division manner. When the light source 23 emits white light, the camera head 21 switches the optical filter to the IR cut filter 26A. At this time, the visible light image sensor 25A receives the visible light that has passed through the IR cut filter 26A and captures a visible light image.

Further, when the light source 23 emits infrared light, the camera head 21 switches the optical filter to the visible light cut filter 26B. At this time, the infrared light image sensor 25B receives the fluorescence (see above) that has passed through the visible light cut filter 26B and captures a fluorescence image.

If the camera head 21 has a configuration in which the IR cut filter 26A and the visible light cut filter 26B can be used together, the visible light image and the fluorescent image can be captured at the same time. Further, as the image sensor constituting the visible light image sensor 25A and the infrared light image sensor 25B, for example, CMOS (Complementary Metal Oxide Semiconductor) is used, but CCD (Charged Coupled Device) may be used.

The light source 23 irradiates excitation light in the IR band (for example, light having a wavelength of 760 nm) for exciting a fluorescent agent (for example, indocyanine green). Further, the light source 23 irradiates white light (for example, light having a wavelength of 700 nm or less). The light source 23 can emit IR excitation light and white light in a time-division manner or at the same time.

The projector 22 generates a vascular specific image for teaching (in other words, characterizing) the shape of the vascular when the vascular is detected and recognized in the captured image of the affected area captured by the camera head 21. , The projected light corresponding to the vascular specific image is projected onto the affected area in the surgical field. The projector 22 may acquire a vessel specific image from the camera control unit 10. At this time, the projector 22 corresponds to the vascular specific image so that the position of the vascular specific image coincides with the position of the vascular appearing on the surface of the affected area, or the vascular specific image covers the outer shape of the vascular. Project the projected light. Hereinafter, the projection of the projected light corresponding to the vascular specific image is simply referred to as the projection of the vascular specific image. Since the configuration inside the projector 22 may be the same as the configuration of a well-known projector, detailed description thereof will be omitted, but the projector 22 generates a projection light source and a projection optical system for projecting projected light, and a vessel-specific image. Includes an image forming unit for The projector 22 may be a projector that employs any method such as a DLP (Digital Light Processing) method, a 3LCD (Liquid Crystal Display) method, and an LCOS (Liquid Crystal On Silicon) method.

The camera control unit 10 is electrically connected to the imaging irradiation device 20, the monitor 30, and the mouse 45 (see FIG. 2), and controls the operation of these devices in an integrated manner. The camera control unit 10 includes an image input unit 11, an image processing unit 12, and an image output unit 13.

The image input unit 11 inputs the data of the captured image at the time of surgery captured by the camera head 21. In addition to the dedicated image input interface, the image input unit 11 may use an HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus) Type-C interface capable of transferring video data at high speed.

The camera control unit 10 has a processor and a built-in memory, and the processor executes a program stored in the built-in memory to specifically realize the functions of the image processing unit 12 and the image output unit 13. The processor may be a GPU (Graphical Processing Unit) suitable for image processing. The image processing unit 12 uses a dedicated electronic circuit designed by an MPU (Micro Processing Unit), a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), etc., or an FPGA (Field Programmable Gate) instead of the GPU. It may be composed of an electronic circuit designed to be reconfigurable by Array) or the like.

The image processing unit 12 is equipped with artificial intelligence (AI: Artificial Intelligent) and has an image recognition function for detecting and recognizing a vessel reflected in the captured image input by the image input unit 11. The image processing unit 12 can realize the above-mentioned image recognition function by using a trained model equipped with artificial intelligence. The trained model is pre-generated before the start of actual operation of the medical image projection system 5. The procedure for generating the trained model is as follows.

First, the user (for example, an administrator or an operator) of the medical image projection system 5 prepares the data of the captured image at the time of surgery. It should be noted that the data of this captured image also includes audio data explained by a doctor by injecting findings including names of parts such as organs reflected in each captured image by voice. For example, about 1000 captured images are used per case, and the findings including the names of organs reflected in each captured image are explained by voice, and the findings for a total of about 100,000 captured images in 100 cases are explained. Is prepared.

The user of the medical image projection system 5 (see above) refers to the sound of the findings explained by the doctor in the acquired image, and puts the pulse into the vessel to be detected in each image. Annotation work such as adding information for teaching the tube (for example, a frame image) is performed, and the data of each captured image to which the annotation work is performed is prepared as teacher data. As a result, for example, a large amount of teaching data for detecting and recognizing the vessels reflected in the captured image is prepared.

Here, the image processing unit 12 inputs a large number of teacher data prepared in advance and data of the captured image at the time of surgery as described above, and performs machine learning such as deep learning (deep learning). In this machine learning, the weighting coefficient between the neurons of each layer (input layer, intermediate layer, output layer) of the neural network that composes the trained model for performing deep learning is reflected in the captured image during surgery. Is optimized so that it can be detected and recognized accurately.

The image processing unit 12 generates a trained model suitable for detecting and recognizing a vessel as a result of the machine learning described above. Using this trained model, the image processing unit 12 performs image recognition processing of the vascular tube that can be reflected in the captured image input from the image input unit 11 at the time of surgery, and colors the area surrounding the detected vascular tube. To generate a vessel-specific image. The vessel-specific image is superimposed on the captured image at the time of surgery by the image output unit 13. Further, the image processing unit 12 evaluates (scores) the result of the image recognition processing of the detected vessel (for example, the matching probability), and updates the trained model by machine learning in real time using the evaluation result. May be good.

The image output unit 13 generates a composite image in which the vessel-specific image generated by the image processing unit 12 is superimposed on the captured image at the time of surgery input from the image input unit 11. The image output unit 13 transmits the composite image data to the monitor 30. This composite image data further includes position information in the composite image of the vessel and color information for teaching the region of the vessel, in addition to the captured image at the time of surgery and the vessel specific image. The image output unit 13 may transmit text data related to the vessel in addition to the above-mentioned composite image data. The text data may include, for example, an evaluation value (score value) of an image recognition result of a vessel, an organ name, or the like.

Further, the image output unit 13 transmits the above-mentioned composite image data and related text data to the projector 22. The image output unit 13 may output the vessel-specific image generated by the image processing unit 12 to the projector 22 instead of the composite image data and the related text data. The projector 22 is set to match the position of the vascular tube appearing in the affected area in the surgical field at the time of surgery, based on the position information reflected in the captured image of the vascular tube included in the composite image data transmitted from the image output unit 13. Project a vascular specific image on (that is, overlap the vascular). For example, a single color green is used as the color information of the vessel specific image. In the case of a single color, it is preferable not to use yellow and red because yellow is difficult to distinguish in the surgical field and red is difficult to distinguish from the colors of organs and blood. Here, green is used. As the color information of the vessel specific image, a color having a complementary color relationship with the background color of the captured image may be used so that a doctor or the like can easily distinguish the image instantly.

The monitor 30 displays the data of the composite image from the image output unit 13 (that is, the composite image in which the vessel specific image is superimposed on the captured image). The monitor 30 is composed of, for example, a liquid crystal display or an organic EL (Electroluminescence) display, and has a display surface for displaying an image.

FIG. 2 is a diagram showing a specific configuration of the medical image projection system 5. The medical image projection system 5 is installed, for example, in an operating room of a hospital. In the operating room, open surgery is performed on a patient hm who is lying on his back on the operating table 110. The imaging irradiation device 20 irradiates the surgical field 130 (for example, the affected part of the patient) including the opened portion of the patient hm with white light, excitation light, and projected light, and also images the surgical field 130. The medical image projection system 5 includes an imaging irradiation device 20, a camera control unit 10 and a monitor 30 shown in FIG. 1, as well as a mouse 45 that can be operated by a user such as a doctor.

The image pickup irradiation device 20 includes an optical unit 28 in addition to the camera head 21, the projector 22, and the light source 23 shown in FIG. The optical unit 28 is arranged so as to face the camera head 21 and the projector 22. The optical unit 28 transmits visible light and fluorescence from the surgical field toward the camera head 21, while reflecting the projected light emitted from the projector 22 and projecting it onto the surgical field. The optical unit 28 is adjusted so that the optical axes of visible light and fluorescence toward the camera head 21 and the optical axes of the projected light emitted from the projector 22 toward the surgical field are aligned with each other. In a state where the optical axis of the camera head 21 and the optical axis of the projector 22 are aligned, the camera control unit 10 sets the position of the vessel included in the image captured by the camera head 21 as the center of, for example, the optical axis of the camera head 21. It is represented by the coordinates to be used, and this coordinate information is transmitted to the projector 22 together with the color information of the vessel specific image. The projector 22 projects a vessel-specific image at the position of the received coordinate information. As a result, the projector 22 can map (project) the vascular tube in the surgical field with the vascular tube specific image.

The mouse 45 is an input device that accepts operations by an operator or the like on the camera control unit 10. The input device may be a keyboard, a touch pad, a touch panel, a button, a switch, or the like. For example, during surgery, the surgeon or the like displays an image captured by the camera head 21 on the monitor 30 to visually recognize the state of surgery.

Next, the operation of the medical image projection system 5 according to the first embodiment will be shown.

FIG. 3 is a flowchart showing an example of an operation procedure of the medical image projection system 5 according to the first embodiment. This operation is executed when an operator such as a doctor activates the image recognition function of the vessel to the camera control unit 10.

In FIG. 3, the image input unit 11 acquires a visible light image captured by the camera head 21 with the surgical field as a subject (S1). The image processing unit 12 analyzes a visible light image using the image recognition function of the vessel (that is, a learned model capable of realizing the image recognition function using AI described above) (S2). The image processing unit 12 determines whether or not a vessel has been detected as a result of image analysis (S3).

When a vessel is detected (S3, YES), the image output unit 13 transmits a vessel image projection instruction to the projector 22 (S4). The vascular image projection instruction includes the position information of the vascular in the captured image and the vascular specific image superimposed on the vascular. The vessel specific image is a color image in which a region representing the outer shape of the vessel included in the captured image is filled with a specific color. Further, the image output unit 13 generates a composite image in which a vascular specific image corresponding to the vascular detected and recognized by the image processing unit 12 is superimposed on the captured image input from the image input unit 11 (S5). .. The image output unit 13 outputs this composite image to the monitor 30 (S6).

On the other hand, when the image processing unit 12 does not detect the vessel (S3, NO), the image output unit 13 is a visible light image captured by the camera head 21 (that is, an captured image input from the image input unit 11). ) Is output to the monitor 30 as it is (S7). After the processing of steps S6 and S7, the camera control unit 10 ends the operation shown in FIG. The operation shown in FIG. 3 is repeated until an operator such as a doctor stops the image recognition function of the vessel with respect to the camera control unit 10.

FIG. 4 is a diagram showing an image of an organ 200 on which a vascular specific image mg is projected onto a vascular 100 detected by a vascular image recognition function. The image of the organ 200 is displayed on the monitor 30. The image of the organ 200 may be an image of an organ visually observed by a doctor in the surgical field. When the image recognition function of the vessel is stopped, the vessel specific image is not generated, so that the image of the organ 200 remains the captured image input from the image input unit 11 and does not change in particular. When the image recognition function of the vessel is activated and the vessel 100 is detected, the vessel specific image mg representing the region (outer shape) of the vessel is superimposed on the image of the organ 200. Here, the Grisson capsule, which is treated as a vessel, appears. Further, the vessel specific image mg is a colored (for example, green) image having the same contour as the outer shape of the vessel 100. The vessel specific image may be any color except yellowish and reddish colors.

As described above, the medical image projection system 5 according to the first embodiment projects an ICG fluorescence image onto an organ by projection mapping. A surgeon such as a doctor can visualize the boundary between the vascular tube and the ischemic area in real time without taking his eyes off the surgical field. In the medical image projection system 5, the camera control unit 10 is equipped with artificial intelligence (in other words, the image processing unit 12) that holds the image recognition function of the vessel, so that the pulse in the surgical field is at the same judgment level as that of a skilled doctor. The tube can be accurately detected and recognized, and can be displayed on the surgical field so that the surrounding surgeons can grasp it. Therefore, safe and secure surgical navigation is possible.

As described above, in the first embodiment, the camera control unit 10 is connected to the camera head 21 that images the surgical field. The camera control unit 10 inputs a captured image of the surgical field from the camera head 21. The camera control unit 10 recognizes the vessels reflected in the captured image based on the input captured image. The camera control unit 10 generates a composite image in which a vessel specific image (an example of information for teaching the vessel) representing the recognized vessel is superimposed on the captured image, and outputs the generated composite image to the monitor 30. ..

As a result, the medical image projection system 5 provides, for example, a recognition function equivalent to the tacit knowledge of a skilled doctor who quickly recognizes the vessels appearing in the captured image captured by the camera head 21 that captures the surgical field during surgery in real time. Can be reproduced with high accuracy. Therefore, the medical image projection system 5 can accurately notify the surgeon such as a doctor of the location of the vessel appearing on the dissected surface such as the tumor portion in the surgical field, and can support the provision of safe and secure surgery.

Further, the camera control unit 10 is connected to a projector 22 arranged so as to be projectable in the surgical field. The image output unit 13 generates a vascular image projection instruction including the position information of the vascular in the captured image GZ and the vascular specific image and sends it to the projector 22. As a result, the surgeon or the like can visually recognize the vessel in the surgical field without looking at the monitor.

Further, the image processing unit 12 recognizes the vessels by using a learned model generated based on machine learning using the teacher data of a plurality of vessels displayed in the captured image of the surgical field. As a result, the camera control unit 10 can improve the detection accuracy of the vessel by learning.

Further, the image input unit 11 inputs a visible light image as an captured image. The image processing unit 12 recognizes the vessels based on the visible light image. The trained model can detect and recognize the vessels reflected in the visible light image by the annotation work of the doctor or the like described above. As a result, the camera control unit 10 can brightly illuminate the surgical field even if the fluorescence based on the fluorescence emission of a fluorescent agent such as ICG (in other words, the affected part such as the tumor part) is not imaged. The vasculature can be accurately detected and recognized from the visible light image obtained based on the imaging. Therefore, the configuration of the medical image projection system 5 can be simplified and the cost can be reduced.

(Embodiment 2)
Next, as an example of the vascular recognition system according to the second embodiment, an endoscopic system will be illustrated and described. FIG. 5 is a diagram showing a schematic configuration example of the endoscope system 40 according to the second embodiment.

The endoscope system 40 includes an endoscope 50, a camera control unit 60 as an example of a vessel recognition device, a light source 53, and a monitor 80.

The endoscope 50 is, for example, a medical rigid endoscope. The camera control unit 60 is a captured image (for example, a still image) captured by an endoscope 50 inserted into an affected area (for example, the skin of a human body, an organ wall inside the human body, etc.) to be observed by a subject (for example, a person). Alternatively, the moving image) is subjected to predetermined image processing, and the vessels reflected in the captured image after the image processing are detected. The camera control unit 60 generates a composite image in which the detected vascular image is superimposed on the captured image after image processing, and outputs the composite image to the monitor 80. The monitor 80 displays a composite image output from the camera control unit 60.

The endoscope 50 irradiates the surgical field with white light (that is, visible light), receives the reflected light reflected by the affected part (for example, an organ) of the subject (for example, a person), and captures a visible light image. To do. The endoscope 50 irradiates infrared light to excite the fluorescent agent accumulated in the affected area or the like, and receives infrared light including fluorescence generated by the fluorescence emission of the fluorescent agent to capture a fluorescent image. The endoscope 50 includes an endoscope head 51.

The endoscope head 51 includes a visible light sensor unit 51A including a visible light image sensor 54A and an IR cut filter 56A, and an infrared light sensor unit 51B including an infrared light image sensor 54B and a visible light cut filter 56B. Including. The image sensor for visible light may be referred to as a sensor for visible light. Similarly, the infrared light image sensor may be referred to as an infrared light sensor.

The IR cut filter 56A blocks (cuts) the excitation light having the IR wavelength band that is irradiated from the light source 53 to the affected area (for example, an organ) and reflected by the affected area. The visible light image sensor 54A receives visible light that has passed through the IR cut filter 56A (that is, visible light reflected by the affected area) and captures a visible light image.

The visible light cut filter 56B irradiates the affected area (for example, an organ) from the light source 53 and blocks (cuts) the visible light reflected by the affected area. Further, the visible light cut filter 56B blocks (cuts) not only visible light but also IR light in the wavelength band of excitation light (for example, 690 nm to 820 nm). The infrared light image sensor 54B receives fluorescence that has passed through the visible light cut filter 56B (that is, fluorescence emitted based on the fluorescence emission of the fluorescent agent accumulated in the affected area), and captures a fluorescence image.

The endoscope head 51 captures, for example, a visible light image and a fluorescence image in a time-division manner. When the light source 53 emits white light, the endoscope head 51 switches the optical filter to the IR cut filter 56A. At this time, the visible light image sensor 54A receives the visible light that has passed through the IR cut filter 56A and captures a visible light image.

Further, when the light source 53 emits infrared light, the endoscope head 51 switches the optical filter to the visible light cut filter 56B. At this time, the infrared light image sensor 54B receives the fluorescence (see above) that has passed through the visible light cut filter 56B and captures a fluorescence image.

If the endoscope head 51 has a configuration in which the IR cut filter 56A and the visible light cut filter 56B can be used together, the visible light image and the fluorescent image can be captured at the same time. Further, as the image sensor constituting the visible light image sensor 25A and the infrared light image sensor 25B, for example, CMOS is used, but CCD may be used.

The light source 53 irradiates a fluorescent agent (for example, excitation light in the IR band for exciting indocyanine green (for example, light having a wavelength of 760 nm), and the light source 53 irradiates white light (for example, light having a wavelength of 700 nm or less). The light source 53 can emit IR excitation light and white light in a time-divided manner or at the same time.

The camera control unit 60 is electrically connected to the endoscope head 51, the light source 53, and the monitor 80, and controls the operation of these devices in an integrated manner. The camera control unit 60 includes an image input unit 61, an image processing unit 62, and an image output unit 63.

The image input unit 61 inputs the data of the captured image at the time of surgery captured by the endoscope head 51. In addition to the dedicated image input interface, the image input unit 61 may use an HDMI (registered trademark), USB Type-C, or the like capable of transferring video data at high speed.

The camera control unit 60 has a processor and a built-in memory, and the processor executes a program stored in the built-in memory to specifically realize the functions of the image processing unit 62 and the image output unit 63. The processor may be a GPU suitable for image processing. The image processing unit 62 may be configured by a dedicated electronic circuit designed by an MPU, a CPU, an ASIC, or the like, or an electronic circuit designed so that it can be reconfigured by an FPGA or the like, instead of the GPU.

The image processing unit 62 is equipped with artificial intelligence (AI, see the first embodiment) and has an image recognition function for detecting and recognizing a vessel reflected in the captured image input by the image input unit 61. The image processing unit 62 can realize the above-mentioned image recognition function as in the first embodiment by using the trained model equipped with artificial intelligence. The trained model is pre-generated before the start of actual operation of the endoscope system 40. Since the procedure for generating the trained model is as described with reference to the first embodiment, the description thereof will be omitted here.

Using the trained model described above, the image processing unit 62 performs image recognition processing of the vascular tube that can be reflected in the captured image input from the image input unit 61 at the time of surgery, and colors the area surrounding the detected vascular tube. To generate a vascular specific image. The vessel-specific image is superimposed on the captured image at the time of surgery by the image output unit 63. Further, the image processing unit 62 evaluates (scores) the result (for example, matching probability) of the detected vascular image recognition process, and updates the trained model by machine learning in real time using the evaluation result. May be good.

The image output unit 63 generates a composite image in which the vessel-specific image generated by the image processing unit 62 is superimposed on the captured image at the time of surgery input from the image input unit 61. The image output unit 63 transmits the composite image data to the monitor 30. This composite image data further includes position information in the composite image of the vessel and color information for teaching the region of the vessel, in addition to the captured image at the time of surgery and the vessel specific image. The image output unit 63 may transmit text data related to the vessel in addition to the above-mentioned composite image data. The text data may include, for example, an evaluation value (score value) of an image recognition result of a vessel, an organ name, or the like.

The monitor 80 displays the data of the composite image from the image output unit 63 (that is, the composite image in which the vessel specific image is superimposed on the captured image). The monitor 80 is composed of, for example, a liquid crystal display or an organic EL display, and has a display surface for displaying an image.

FIG. 6 is a diagram showing an outline of the endoscope system 40. The endoscope system 40 includes an endoscope 50, a camera control unit 60, a monitor 80, and a light source 53.

The camera control unit 60 performs image processing on the captured image input from the endoscope 50 via the transmission cable, and generates the captured image after the image processing. The camera control unit 60 is connected to the first excitation light source 325 and the second excitation light source 327, respectively, via the light source drive cable 314. The camera control unit 60 generates a control signal for driving the first excitation light source in the light source drive circuit 365 via the light source drive cable and outputs the control signal to the first excitation light source 325. Further, the camera control unit 60 generates a control signal for driving the second excitation light source 327 in the light source drive circuit 365 via the light source drive cable and outputs the control signal to the second excitation light source 327.

The monitor 80 displays the captured image output from the camera control unit 60. The monitor 80 has a display device such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), for example. In the endoscope system 40 according to the second embodiment, the monitor 80 uses a visible light image captured based on the irradiated visible light and a fluorescence image captured based on the fluorescence generated by the irradiated excitation light. And are displayed.

The light source 53 includes a first excitation light source 325, a second excitation light source 327, a combiner prism 335, an optical fiber 329, an insertion portion 331, and a phosphor 333 as main constituent members.

The first excitation light source 325 is configured by using, for example, a semiconductor laser such as a laser diode (LaserDiode) capable of irradiating a laser beam having a half-value width of about 1/10 of the light from an LED (Light Emission Diode). .. The first excitation light source 325 irradiates and outputs light in a narrow band blue region (for example, light having a wavelength in the wavelength band of 380 to 450 nm) for exciting the phosphor 333 to generate pseudo white light.

The second excitation light source 327 is configured by using, for example, a semiconductor laser such as a laser diode capable of irradiating a laser beam having a half-value width of about 1/10 of that of the light from the LED. The second excitation light source 327 irradiates and outputs light in a narrow wavelength band different from the wavelength band of the light emitted from the first excitation light source 325. That is, the second excitation light source 327 is light in the infrared region (for example, a wavelength band of 690 to 820 nm) for exciting a fluorescent agent to be administered to the affected area of the subject in advance before endoscopic surgery or endoscopy. Light having the wavelength of) is irradiated and output.

The combined wave prism 335 guides the light emitted from the first excitation light source 325 and the light emitted from the second excitation light source 327 to the same optical fiber 329, respectively. The combined wave prism 335 multiplexes the light in the blue region and the light in the infrared region.

The light source 53 causes the light in the blue region and the light in the infrared region multiplexed by the combiner prism 335 to be incident from the light incident end of the optical fiber 329. A condensing lens 341 is provided between the combined wave prism 335 and the optical fiber 329.

The optical fiber 329 can be, for example, one optical fiber wire. Further, as the optical fiber 329, a bundle fiber in which a plurality of optical fiber strands are bundled may be used.

The insertion portion 331 inserts the optical fiber 329. In addition, a transmission cable 323 is inserted through the insertion portion 331. The insertion portion 331 is, for example, an insertion portion of the endoscope 50 to be inserted into the body cavity of the subject. The insertion portion 331 is a tubular rigid member.

An imaging window is arranged on the tip surface of the insertion portion 331. The imaging window is formed by including an optical material such as optical glass or optical plastic, and allows light from a subject (for example, a subject or an affected portion in the subject) to be incident. Further, an illumination window is arranged on the tip surface of the insertion portion 331. The illumination window is formed by including an optical material such as optical glass or optical plastic, and emits illumination light from the light emitting end of the optical fiber 329.

Next, the operation of the endoscope system 40 according to the second embodiment will be shown.

FIG. 7 is a flowchart showing an example of an operating procedure of the endoscope system 40 according to the second embodiment. This operation is executed when an operator such as a doctor activates the image recognition function of the vessel to the camera control unit 60.

In FIG. 7, the image input unit 61 acquires a visible light image captured by the endoscope head 51 with the surgical field as a subject (S11). The image processing unit 62 analyzes the visible light image using the image recognition function of the vessel (that is, a learned model capable of realizing the image recognition function using the AI described above) (S12). The image processing unit 62 determines whether or not a vessel has been detected as a result of image analysis (S13).

When a vessel is detected (S13, YES), the image output unit 63 adds a vessel-specific image corresponding to the vessel detected and recognized by the image processing unit 62 to the captured image input from the image input unit 61. A superimposed composite image is generated (S14). The image output unit 63 outputs the composite image to the monitor 80 (S15).

On the other hand, when the image processing unit 62 does not detect the vessel (S13, NO), the image output unit 63 is input from the visible light image (that is, the image input unit 61) captured by the endoscope head 51. The captured image) is output to the monitor 80 as it is (S16). After this, the camera control unit 10 ends the operation shown in FIG. 7. The operation shown in FIG. 7 is repeated until an operator such as a doctor stops the image recognition function of the vessel with respect to the camera control unit 10.

As described above, the endoscope system 40 according to the second embodiment displays the location of the vessel on the monitor 80 by connecting the image captured in the subject captured by the endoscope 50 to the camera control unit 60. It can be displayed accurately. Further, since the camera control unit 60 can be connected to the endoscope system 40 existing in many hospitals, the endoscope system 40 can be used in many hospital facilities and its versatility is increased.

(Modification 1 of Embodiment 1)
When the medical image projection system 5 according to the first embodiment is used, for example, if the head of an operator such as a doctor is reflected in the captured image captured by the camera head 21, the part to be treated is not displayed on the monitor 30. In some cases. Therefore, in the first modification of the first embodiment, when it is detected that the head of an operator such as a doctor has been imaged, the medical image projection system 5 issues an alarm to the operator such as a doctor. You may let us know. For example, the camera control unit 10 has a built-in speaker and outputs an alarm sound from the speaker. Since the alarm sound is a sound for notifying a surgeon such as a doctor of the situation where the part such as the tumor part to be treated is not reflected, it may be a sound that calls attention such as "pip". It may be a pleasant sound such as "melody". Further, the alarm sound may be a voice explaining the situation.

Further, the camera control unit 10 outputs a default alarm image instead of or outputs an alarm sound, and displays a default alarm image on the monitor 30, so that a site such as a tumor to be treated by a surgeon such as a doctor is not shown. May be informed. The alarm image may be, for example, a text for explaining a situation in which a mark indicating an abnormality, a tumor portion, or the like is not shown. In this case, the surgeon such as a doctor simply shifts his or her head, and the image projected on the monitor 30 returns to the state in which the part to be treated is accurately projected. Instead of the surgeon such as a doctor shifting the head, a nurse or the like who assists the surgery may change the posture of the camera head 21. Further, in the video, the case where the medical instrument used for the operation covers the part to be operated is the same as the case where the operator's head is covered. In this case, the surgeon simply shifts the medical instrument, and the image returns to the state in which the part to be treated is projected. Instead of the surgeon shifting the medical instrument, a nurse or the like who assists the surgery may change the posture of the camera head 21.

In addition, the head of a surgeon such as a doctor or a medical instrument is covered, and the part to be treated is not shown in the image. In addition, the enlargement ratio of the image captured by the camera head 21 is low, and it is difficult to show the organ in detail. In some cases. In this case, the camera control unit 10 changes the magnifying power of the camera head 21 according to an operation instruction of an operator such as a doctor or automatically, takes an image at the changed magnifying power, and acquires an captured image. The camera control unit 10 may execute the image recognition function of the vessel on the captured image at the changed magnification and detect the vessel.

In addition, the field of view imaged by the camera head 21 is cloudy, and a clear image may not be obtained. In this case, the camera control unit 10 may perform image processing on the image captured by the camera head 21. For example, the camera control unit 10 adjusts the white balance of the captured image and changes the color tone of the captured image so as to suppress the white color. Further, the camera control unit 10 performs a process of reducing the white component in the frequency spectrum of the captured image. The camera control unit 10 may execute the image recognition function of the vessel on the image in which the white color is suppressed and become clear, and detect the vessel.

(Modification 2 of Embodiment 1)
When the camera control unit 10 executes the image recognition function of the vessel, the load of the processing executed by the camera control unit 10 becomes heavy. Further, in the medical image projection system 5 according to the first embodiment, the vascular projection image is projected on the site corresponding to the surgical field of the laparotomized subject, which imposes a considerable burden on the patient. .. In addition, since the part to be treated is in a mapped state, an operator such as a doctor may have an illusion of the actual part by continuing to look at the mapped image. Therefore, it is desirable that the image recognition function of the vessel by the camera control unit 10 is stopped except when necessary for the treatment.

Therefore, in the second modification of the first embodiment, the camera control unit 10 determines the current treatment scene (situation) by analyzing the captured image captured by the camera head 21, and determines the determined scene. Based on this, the image recognition function of the vessel is activated or stopped. FIG. 8 is a diagram showing the registered contents of the scene determination table Tb1 representing a scene when performing a laparotomy according to the second modification of the first embodiment.

In the scene judgment table Tb1, No. 1 to NO. Nine scenes are registered. Specifically, the scene No. 1 is a scene of "incision". Scene No. 2 is a scene of "securing blood vessels". Scene No. Reference numeral 3 denotes a scene of "peeling the liver (hepatic coronary mesentery, etc.)". Scene No. 4 is a scene of "cholecystectomy". Scene No. 5 is a scene of "preparation before excision". Scene No. Reference numeral 6 is a scene of "marking the excised part". Scene No. 7 is a scene of "during excision". Scene No. 8 is a scene of "during excision (rest)". Scene No. 9 is a scene "after excision".

Information for determining the scene includes procedures, treatments, and instruments used. For example, the scene No. In 1, the procedure is "incision". The procedure is "cutting the skin (laparotomy)". The instrument used is a "female". In this case, since the organ (for example, the liver) in which the vessel exists does not appear, the camera control unit 10 stops the image recognition function of the vessel. On the other hand, the scene No. In 7, the procedure is "during excision". The procedure is "remove the liver to remove the tumor". The instrument used is "Cusa". In this case, since the organ in which the vessel exists appears, the camera control unit 10 activates the image recognition function of the vessel.

In the camera control unit 10, the nurse who assists the treatment sets the scene number for the camera control unit 10. The scene may be determined by instructing and operating. Further, the camera control units 10 and 60 may automatically determine the scene by the equipment used in the image captured by the camera head 21 or the endoscope head 51. In this case, the camera control units 10 and 60 accumulate image data obtained by capturing images of many used devices, perform machine learning by deep learning on these image data, and determine a scene from the used devices included in the captured images. Generate a trained model in advance. The camera control units 10 and 60 may input image data captured by the camera head 21 and the endoscope head 51 into the trained model and determine the scene from the equipment used.

In this way, the camera control units 10 and 60 can determine the scene and activate or stop the image recognition function of the vessel according to the scene. Therefore, the camera control units 10 and 60 can execute the image recognition function of the vascular tube only when necessary during the treatment.

The camera control unit 10 may activate the image recognition function of the vascular tube when the image captured by the camera head 21 temporarily does not include the medical device. The image recognition function of the vessel may be stopped if it is not necessary. As a result, the activation / stop of the image recognition function of the vessel can be switched more finely.

Further, the camera control units 10 and 60 may activate or stop the image recognition function of the vessel other than the determination of the scene. For example, the camera control unit 10 may detect the movement of the camera head 21 and stop the image recognition function of the vessel when the camera head 21 is moving.

As described above, in the second modification of the first embodiment, when the image processing unit 12 detects the medical device for removing the tumor portion in the input captured image, the image recognition of the vessel using the captured image is performed. Invokes the function (that is, starts the recognition process). As a result, the camera control unit 10 can display the vascular specific image at an appropriate timing according to the start of the operation.

Further, the image processing unit 12 stops the image recognition function of the vessel when the medical device is no longer detected from the input captured image. As a result, the camera control unit 10 can erase the display of the vessel specific image at the end of the operation at a timing when it is no longer needed. Therefore, the load on the camera control unit 10 can be reduced.

(Modification 3 of Embodiment 1)
In the third modification of the first embodiment, the camera control unit 10 changes the display mode of the vascular specific image according to the surgical situation. Specifically, the camera control unit 10 changes the display of the vascular specific image according to the situation of vascular discovery and treatment. For example, when the camera control unit 10 first detects a vessel included in an image captured by the camera head 21 by the image recognition function of the vessel, the camera control unit 10 obtains a vessel specific image suitable for "first discovery (undecided)". The composite image superimposed on the captured image is displayed on the monitor 30.

FIG. 9 is a diagram showing a screen of a monitor 30 in which a vessel specific image according to a modification 3 of the first embodiment is superimposed and displayed on an organ. The vascular specific image is a circular marker mk1 having a diameter centered on the position of the vascular and surrounding the vascular. The circular marker mk1 is lit or blinks. At the stage of the first discovery, the camera control unit 10 draws a circular marker mk1 with a thin purple line. After that, when the camera control unit 10 repeatedly detects the vessels included in the captured image by the image recognition function of the vessels, the camera control unit 10 may draw the circular marker mk1 with a thicker green line according to the number of detections.

Further, when the treatment by the operator is performed (treated), for example, when the tumor portion is removed, the camera control unit 10 deforms the outer shape of the marker mk1 from a circle to a quadrangle and draws it in the same color or another color. You may. Here, the camera control unit 10 determines that "treated", for example, after the first discovery of the vessel, confirms that the image captured by the camera head 21 includes forceps, an electric knife, a cusa, and the like. After that, when it is confirmed that these medical devices are no longer included, it may be determined that the treatment has been completed.

Further, the camera control unit 10 may determine "treated" by using artificial intelligence. The camera control unit 10 accumulates images of organs including many treated organs, uses these images as teacher data, performs machine learning by deep learning on the treated organs, and learns to detect the treated organs from the captured images. Generate a completed model in advance. The camera control unit 10 may input the data of the image captured by the camera head 21 into the trained model to determine "treated".

Further, when there is bloodshed, the camera control unit 10 may draw in the same color or another color so that the outer shape of the marker mk1 is deformed from a circle to a triangle, or the marker mk1 is blinked. Here, the determination of "bloodshed" may use artificial intelligence in the same manner as the determination of "treated". The camera control unit 10 accumulates images of many bloody organs, uses these images as teacher data, performs machine learning by deep learning on the bloody organs, and prepares a trained model for detecting the bloody organs from the captured images in advance. Generate in. The camera control unit 10 may input data of an image captured by the camera head into the trained model to determine "bloodshed".

Further, the camera control unit 10 determines "bloodshed", for example, after confirming that the captured image contains an electric knife, and then confirming that the red component in the miscellaneous image image has increased sharply, "bloodshed". You may judge that.

Further, the camera control unit 10 may change the display form of the marker mk1 according to the result of time. For example, immediately after the camera control unit 10 discovers the vessel, the marker mk1 is displayed in a conspicuous manner, for example, by increasing the brightness. When the first time elapses after the discovery, the camera control unit 10 displays the marker mk1 at a reduced brightness so as to gradually become inconspicuous. At this time, the brightness of the marker mk1 may be lowered to the extent that the marker mk1 almost disappears. Further, after the lapse of the first time and the lapse of the second time, the camera control unit 10 raises the brightness of the marker mk1 so as to be conspicuous again so as not to forget the existence position of the vessel. In this way, the camera control unit 10 can change the display mode of the marker mk1 in consideration of the state of the operator. When making the marker mk1 stand out, the camera control unit 10 may blink the marker mk1.

In addition, during surgery, when the surgeon turns over or shifts an organ such as the affected area, the vascular tube may temporarily disappear from the field of view, and the marker mk1 indicating that the vascular tube is temporarily present may disappear. When a vessel has already been detected, the camera control unit 10 has a rectangular frame image (for example, a yellow frame image) indicating that the vessel has been detected in the screen frame (for example, the upper left and right upper and lower ends of the screen) of the monitor 30. ) Wg may always be displayed. The rectangular frame image wg is displayed by lighting or blinking. The camera control unit 10 may display a rectangular frame image only when the marker mk1 indicating that there is a vessel temporarily disappears. As a result, even if the marker mk1 indicating that there is a vessel temporarily disappears, a rectangular image is displayed on the screen frame of the monitor 30, so that a surgeon such as a doctor can check the pulse. It can be confirmed that the tube has been detected.

Further, the camera control unit 10 may change the line width of the rectangular frame image depending on the surgical situation or the passage of time. For example, the camera control unit 10 may display a wide rectangular frame image when a surgeon such as a doctor excises a tumor portion using an electric knife. Further, the camera control unit 10 may change the color of the rectangular frame image from yellow to another color such as green. Further, the camera control unit 10 may gradually narrow the width of the rectangular frame image with the passage of time.

As described above, in the modified example 3 of the first embodiment, when the image processing unit 12 detects the vessel, the image output unit 13 has a rectangular frame image wg (colored frame) indicating that the vessel is detected. The rectangular frame image wg is displayed on the monitor 30 so that the image) blinks. As a result, a surgeon such as a doctor can notice the existence of a vessel.

Further, in the image output unit 13, when the vessel is detected by the image processing unit 12, a circular marker mk1 (colored circular image having a predetermined diameter) centered on the position in the captured image GZ where the vessel is detected is generated. A circular marker mk1 is displayed on the monitor 30 so as to blink. This makes it easier for an operator such as a doctor to grasp the position of the vessel.

(Modification 4 of Embodiment 1)
When the vascular specific image and the ICG fluorescence image (that is, the fluorescence image captured by the camera head 21 based on the fluorescence emission of ICG) are overlapped and displayed on the monitor 30, in these overlapping regions, a doctor or the like It is difficult for the surgeon to determine the position of the vessel. FIG. 10 is a diagram showing a screen of a monitor 30 in which the ICG fluorescence image fg and the vessel specific image mg according to the modified example 4 of the first embodiment partially overlap and are superimposed on the captured image GZ including an organ. Is. Here, the vascular specific image mg is a circular image painted in green. Although the ICG fluorescence image fg is displayed in blue, it is mixed with the color of the organ behind it (red) and expressed in magenta (magenta).

The camera control unit 10 displays the captured image so that an image having a high degree of urgency or importance can be preferentially discriminated. As an example, when the camera control unit 10 displays the captured image GZ (visible light image) by the camera head 21, the ICG fluorescence image fg, and the vessel specific image mg on the monitor 30, these are displayed on the respective layers. For example, when the vascular specific image mg is preferentially displayed, the camera control unit 10 displays the vascular specific image mg on the uppermost layer, displays the ICG fluorescence image fg on the intermediate layer, and displays the captured image GZ ( Visible light image) is displayed on the lowest layer. When the vascular identification image and the ICG fluorescence image do not overlap, that is, when the vascular identification image is displayed superimposed on the visible light image, the camera control unit 10 sets the color of the vascular identification image on the back side thereof. Set so that there is a complementary color relationship with the color of the visible light image (image of the organ). This makes it easier for the operator or the like to visually recognize the vascular specific image. The color of the vessel-specific image may be set so as to have a complementary color relationship with the background color (black in this case) around the captured image. Red and inconspicuous yellow, which are difficult to distinguish from blood color, are not used as the color of the vascular specific image.

In the fourth modification of the first embodiment, the camera control unit 10 displays the vessel specific image mg on the uppermost layer and the ICG fluorescence image fg on the intermediate layer. As a result, the vessel specific image mg is drawn on the upper layer of the ICG fluorescence image fg, so that even in the region where the vessel specific image mg and the ICG fluorescence image fg overlap, the surgeon such as a doctor can identify the vessel. Image mg, that is, the position of the vessel is easy to see.

The camera control unit 10 changes the color of the vascular specific image and the ICG fluorescent image in the region where the vascular specific image and the ICG fluorescent image overlap, without dividing the layer. Image processing such as blinking the tube specific image or not displaying the ICG fluorescent image around the vessel may be performed. When changing the color of the vessel specific image, the camera control unit 10 may change the color to have a complementary color with the color of the ICG fluorescence image fg. This makes it easier to discriminate the vascular specific image mg. The color of the vessel-specific image may be changed to a color that has a complementary color to the background color of the captured image GZ.

As described above, in the modified example 4 of the first embodiment, the camera head 21 can image the fluorescence based on the fluorescence emission of the fluorescent agent accumulated in the tumor portion in the surgical field. In the image output unit 13, both the ICG fluorescence image fg (fluorescence image of the tumor portion based on fluorescence imaging) and the vessel are detected in the captured image of the surgical field, and a part of the ICG fluorescence image fg and the vessel are detected. When a part of the specific image mg (a part of a circular region having a predetermined diameter centered on the position of the vessel) overlaps, the position of the vessel 100 is centered by using the complementary color of the background color of the captured image GZ. The vascular specific image mg is displayed on the monitor 30 so as to blink. As a result, the camera control unit 10 can make the vasculature in the captured image stand out by displaying the vascular specific image using complementary colors.

(Modification 5 of Embodiment 1)
FIG. 11 shows the ICG fluorescence image fg and the vessel identification displayed on the monitor 30 when the ICG fluorescence image fg and the vessel identification image mg1 according to the modified example 5 of the first embodiment overlap with the organ included in the captured image GZ. It is a figure which shows the image mg1 schematically. Here, the vascular specific image mg1 is a green image that substantially matches the outer shape of the vascular. The ICG fluorescence image fg is a bluish-purple image. Further, the cusa detection image cg that identifies the position of the cusa, which is a medical instrument, is a blue circular image. The camera control unit 10 can detect a cusa, which is a medical device. Here, it is assumed that the cusa detection image cg is arranged so as to cover the vessel specific image mg1.

The camera control unit 10 performs the image processing shown in the mathematical formula (1) on the ICG fluorescence image fg, the cusa detection image cg, and the vessel specific image mg1, and the image after the image processing is the captured image GZ including the organ. Is superimposed on the image and displayed on the monitor 30. Further, in this image processing, the camera control unit 10 performs contour extraction on the cucusa detection image cg to obtain contour extraction image cgs of the cucumber detection image cg.

ICG fluorescence image fg-Cusa detection image cg + Vessel specific image mg1 + Cusa detection image cg contour extraction image csg …… (1)

On the monitor 30, the ICG fluorescence image fg is hollowed out in the region where the ICG fluorescence image fg and the Cusser detection image cg overlap. A vascular specific image mg1 appears in the hollowed out area. Therefore, the operator such as a doctor can accurately grasp the position of the vessel without the vessel specific image mg1 overlapping the ICG fluorescence image fg. Further, by adding the contour extraction image csg to the ICG fluorescence image fg, the region of the ICG fluorescence image fg can also be discriminated.

In the fifth modification of the first embodiment, the camera head 21 can image the fluorescence based on the fluorescence emission of the fluorescent agent accumulated in the tumor portion in the surgical field. When both the ICG fluorescence image fg and the vessel are detected in the captured image of the surgical field, and a part of the ICG fluorescence image fg and a part of the vessel specific image mg1 overlap with each other, the image output unit 13 determines. A vessel specific image mg1 (an example of an image of a predetermined color) is displayed in a region excluding a part of the ICG fluorescence image fg around the position of the vessel. This makes it easier for the surgeon or the like to find the position of the vessel even if the ICG fluorescence image and the vessel specific image overlap.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modification examples, modification examples, replacement examples, addition examples, deletion examples, and equal examples within the scope of claims. It is understood that it naturally belongs to the technical scope of the present disclosure. In addition, each component in the various embodiments described above may be arbitrarily combined as long as the gist of the invention is not deviated.

For example, in the first embodiment, the medical image projection system 5 superimposes a vasa vasorum specific image of a vasa vasorum detected in an organ such as an affected organ on an image of this organ and displays it on a monitor 30, and also displays the surgical field. The case of projecting toward is shown. The medical image projection system 5 may be a system that simply projects the vasa vasorum specific image of the vasa vasorum toward the surgical field without superimposing it on the captured image of the organ and displaying it on the monitor 30.

This application is based on the Japanese patent application (Japanese Patent Application No. 2019-060805) filed on March 27, 2019, and the content of which is incorporated as a reference in this application.

This disclosure reproduces the vascular recognition function equivalent to the implicit knowledge of a skilled doctor who quickly recognizes the vascular appearing in the surgical field in real time and with high accuracy, and notifies the location of the vascular appearing on the dissection surface. It is useful as a vascular recognition device, a vascular recognition method, and a vascular recognition system that support the provision of safe and secure surgery.

5 Medical image projection system 10, 60 Camera control unit 11, 61 Image input unit 12, 62 Image processing unit 13, 63 Image output unit 21 Camera head 22 Projector 23, 53 Light source 24A Visible light sensor unit 24B Infrared light sensor unit 25A , 54A Visible light image sensor 25B, 54B Infrared light image sensor 26A, 56A IR cut filter 26B, 56B Visible light cut filter 30, 80 Monitor 40 Endoscope system 51 Endoscope head 100 Vascular

Claims (12)

1. It is a vessel recognition device connected to an imaging device that images the surgical field.
   An image input unit that inputs an image captured from the surgical field from the imaging device,
   An image processing unit that recognizes the vessels reflected in the captured image based on the input captured image, and
   It includes an image output unit that generates a composite image in which the recognized information for teaching the vessel is superimposed on the captured image and outputs the generated composite image to a monitor.
   Vascular recognition device.
2. It is connected to a projector arranged so that it can be projected in the surgical field.
   The image output unit generates a vascular image projection instruction including the position information of the vascular in the captured image and the information for teaching the vascular, and sends it to the projector.
   The vessel recognition device according to claim 1.
3. The image processing unit recognizes the vessels by using a trained model generated based on machine learning using teacher data of a plurality of vessels displayed in the captured image of the surgical field.
   The vessel recognition device according to claim 1.
4. The image input unit inputs a visible light image as the captured image, and receives the image.
   The image processing unit recognizes the vessel based on the visible light image.
   The vessel recognition device according to claim 3.
5. When the image processing unit detects a medical device for removing the tumor portion in the input captured image, the image processing unit starts the recognition process of the vessel using the captured image.
   The vessel recognition device according to claim 1.
6. The image processing unit stops the recognition process when the medical device is no longer detected from the input captured image.
   The vessel recognition device according to claim 5.
7. When the image processing unit recognizes the vessel, the image output unit causes the monitor to display the colored frame image so that the colored frame image indicating that the vessel is recognized blinks.
   The vessel recognition device according to claim 1.
8. When the vessel is recognized by the image processing unit, the image output unit is colored so that a colored circle image having a predetermined diameter centered on the position in the captured image in which the vessel is recognized blinks. Display the image on the monitor,
   The vessel recognition device according to claim 1.
9. The imaging device can image fluorescence based on the fluorescence emission of a fluorescent agent accumulated in the tumor portion in the surgical field.
   In the image output unit, both the fluorescence image of the tumor portion based on the imaging of the fluorescence and the vessel are recognized in the captured image of the surgical field, and a part of the fluorescence image and the position of the vessel are recognized. When a part of a circular region having a predetermined diameter centered on the image overlaps, the circular region image centered on the position of the vessel blinks by using the complementary color of the background color of the captured image. Display the image on the monitor,
   The vessel recognition device according to claim 1.
10. The imaging device can image fluorescence based on the fluorescence emission of a fluorescent agent accumulated in the tumor portion in the surgical field.
    In the image output unit, both the fluorescence image of the tumor portion based on the imaging of the fluorescence and the vessel are recognized in the captured image of the surgical field, and a part of the fluorescence image and the position of the vessel are recognized. When a part of a circular region having a predetermined diameter centered on is overlapped, an image of a predetermined color is displayed in a region excluding a part of the fluorescent image around the position of the vessel.
    The vessel recognition device according to claim 1.
11. It is a vascular recognition method executed by a vascular recognition device connected to an imaging device that images the surgical field.
    The captured image of the surgical field from the imaging device is input, and
    Based on the input image, the vessel reflected in the image is recognized.
    A composite image in which the recognized information for teaching the vessel is superimposed on the captured image is generated.
    Output the generated composite image to the monitor.
    Vascular recognition method.
12. It is a vascular recognition system in which an imaging device that images the surgical field and a vascular recognition device are connected to each other.
    The vessel recognition device is
    The captured image of the surgical field from the imaging device is input, and
    Based on the input image, the vessel reflected in the image is recognized.
    A composite image in which the recognized information teaching the vessel is superimposed on the captured image is generated, and the generated composite image is output to the monitor.
    Vascular recognition system.

Images