CN116097287A - Computer program, learning model generation method, operation support device, and information processing method - Google Patents
Computer program, learning model generation method, operation support device, and information processing method Download PDFInfo
- Publication number
- CN116097287A CN116097287A CN202180058418.6A CN202180058418A CN116097287A CN 116097287 A CN116097287 A CN 116097287A CN 202180058418 A CN202180058418 A CN 202180058418A CN 116097287 A CN116097287 A CN 116097287A
- Authority
- CN
- China
- Prior art keywords
- connective tissue
- learning model
- loose connective
- field image
- computer program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000004590 computer program Methods 0.000 title claims abstract description 38
- 230000010365 information processing Effects 0.000 title claims abstract description 7
- 238000003672 processing method Methods 0.000 title claims abstract description 7
- 210000002808 connective tissue Anatomy 0.000 claims abstract description 280
- 230000008569 process Effects 0.000 claims abstract description 26
- 210000001519 tissue Anatomy 0.000 claims description 59
- 230000003902 lesion Effects 0.000 claims description 41
- 238000012549 training Methods 0.000 claims description 20
- 238000011156 evaluation Methods 0.000 claims description 19
- 238000002674 endoscopic surgery Methods 0.000 claims description 16
- 238000012935 Averaging Methods 0.000 claims description 7
- 230000000717 retained effect Effects 0.000 claims description 5
- 230000003068 static effect Effects 0.000 claims 2
- 238000002372 labelling Methods 0.000 claims 1
- 230000011218 segmentation Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 35
- 238000002357 laparoscopic surgery Methods 0.000 description 35
- 238000003860 storage Methods 0.000 description 24
- 210000005036 nerve Anatomy 0.000 description 23
- 238000009826 distribution Methods 0.000 description 21
- 238000004891 communication Methods 0.000 description 20
- 238000012545 processing Methods 0.000 description 19
- 238000003384 imaging method Methods 0.000 description 16
- 238000010336 energy treatment Methods 0.000 description 13
- 238000011176 pooling Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 10
- 238000001356 surgical procedure Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 9
- 210000000056 organ Anatomy 0.000 description 9
- 238000013528 artificial neural network Methods 0.000 description 8
- 239000003086 colorant Substances 0.000 description 8
- 238000003780 insertion Methods 0.000 description 8
- 230000037431 insertion Effects 0.000 description 8
- 230000000740 bleeding effect Effects 0.000 description 7
- 239000000835 fiber Substances 0.000 description 7
- 238000005452 bending Methods 0.000 description 6
- 239000011159 matrix material Substances 0.000 description 6
- 230000015654 memory Effects 0.000 description 6
- 210000000944 nerve tissue Anatomy 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 201000011510 cancer Diseases 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 238000003709 image segmentation Methods 0.000 description 5
- 238000004040 coloring Methods 0.000 description 4
- 238000005520 cutting process Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 210000004204 blood vessel Anatomy 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000002271 resection Methods 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000001464 adherent effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 210000004177 elastic tissue Anatomy 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 239000007788 liquid Substances 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000010287 polarization Effects 0.000 description 2
- 238000010079 rubber tapping Methods 0.000 description 2
- 102000008186 Collagen Human genes 0.000 description 1
- 108010035532 Collagen Proteins 0.000 description 1
- 208000029836 Inguinal Hernia Diseases 0.000 description 1
- 210000000683 abdominal cavity Anatomy 0.000 description 1
- 210000003815 abdominal wall Anatomy 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 210000000577 adipose tissue Anatomy 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000002512 chemotherapy Methods 0.000 description 1
- 229920001436 collagen Polymers 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 229940079593 drug Drugs 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000000981 epithelium Anatomy 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 230000012447 hatching Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 210000002429 large intestine Anatomy 0.000 description 1
- 210000003041 ligament Anatomy 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 210000001365 lymphatic vessel Anatomy 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 238000010827 pathological analysis Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 210000002307 prostate Anatomy 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 210000002784 stomach Anatomy 0.000 description 1
- 210000002435 tendon Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00045—Display arrangement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/776—Validation; Performance evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
- A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
- A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/031—Recognition of patterns in medical or anatomical images of internal organs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments
Abstract
Provided are a computer program, a learning model generation method, a surgical assistance device, and an information processing method. Causing the computer to execute the following process: an operation field image obtained by photographing an operation field of an endoscopic operation is acquired, and the acquired operation field image is input into a learning model for identifying a loose connective tissue portion included in the operation field image, the learning model learning to output information on the loose connective tissue when the operation field image is input.
Description
Technical Field
The present invention relates to a computer program, a learning model generation method, an operation support device, and an information processing method.
Background
In laparoscopic surgery, for example, surgery is performed to remove a lesion such as a malignant tumor formed in a patient. At this time, the inside of the patient is photographed by a laparoscope, and the obtained observation image is displayed on a monitor (for example, see patent document 1). The operator performs laparoscopic surgery using various surgical tools while viewing the observation image displayed on the monitor. For example, an operator uses forceps to spread out tissue including a lesion in an appropriate direction, and exposes loose connective tissue existing between the tissue including the lesion and the tissue to be preserved. The operator uses an energy treatment tool such as an electrosurgical knife to remove the exposed loose connective tissue and separate the tissue including the lesion from the tissue to be preserved.
Prior art literature
Patent literature
Patent document 1: japanese patent laid-open publication No. 2005-287839
Disclosure of Invention
Problems to be solved by the invention
However, since blood vessels, nerves, or a liquid matrix, and various cells are present around the fibers constituting the loose connective tissue, it is not necessarily easy for an operator to find the loose connective tissue from the operation field image.
The present invention aims to provide a computer program capable of outputting a recognition result of a loose connective tissue portion from an operation field image, a learning model generation method, an operation support device, and an information processing method.
Solution for solving the problem
The computer program according to an aspect of the present invention is for causing a computer to execute: and acquiring an operation field image obtained by shooting an operation field of the microscopic operation, and inputting the acquired operation field image into a learning model so as to output the recognition result of the loose connective tissue part contained in the operation field image, wherein the learning model learns to output the recognition result of the loose connective tissue under the condition that the operation field image is input.
In the method for generating a learning model according to one aspect of the present invention, training data including an operation field image obtained by photographing an operation field of a microscopic operation and forward-solved data representing a loose connective tissue portion in the operation field image are acquired using a computer, and a learning model that outputs a result of recognizing loose connective tissue when the operation field image is input is generated from a set of the acquired training data.
An operation support device according to an aspect of the present invention includes: an acquisition unit that acquires an operation field image obtained by capturing an operation field of an endoscopic operation; a recognition unit that recognizes a loose connective tissue portion included in the surgical field image acquired by the acquisition unit using a learning model that, when the surgical field image is input, outputs a recognition result of the loose connective tissue; and an output unit that outputs auxiliary information related to the endoscopic surgery based on the recognition result of the recognition unit.
In the information processing method according to an aspect of the present invention, a computer is used to acquire an operation field image obtained by photographing an operation field of an endoscopic surgery, and a loose connective tissue portion included in the acquired operation field image is identified using a learning model that learns to output a result of identifying loose connective tissue when the operation field image is input, and to output auxiliary information related to the endoscopic surgery based on the result of identifying.
Effects of the invention
According to the present application, the result of identifying the loose connective tissue portion can be output from the operation field image.
Drawings
Fig. 1 is a schematic diagram illustrating a schematic configuration of a laparoscopic surgery support system according to embodiment 1.
Fig. 2 is a block diagram illustrating an internal configuration of the surgical assist device.
Fig. 3 is a schematic diagram showing an example of an operation field image.
Fig. 4 is a schematic diagram showing a configuration example of the learning model.
Fig. 5 is a schematic diagram showing the recognition result of the learning model.
Fig. 6 is a flowchart illustrating the generation steps of the learning model.
Fig. 7 is a flowchart illustrating the steps performed for surgical assistance.
Fig. 8 is a schematic diagram showing a display example in the display device.
Fig. 9 is a schematic diagram showing a display example of an identification image in embodiment 2.
Fig. 10 is an explanatory diagram illustrating the configuration of the softmax layer of the learning model in embodiment 3.
Fig. 11 is a schematic diagram showing a display example in embodiment 3.
Fig. 12 is a schematic diagram showing a display example in embodiment 4.
Fig. 13 is a flowchart showing a display switching step in embodiment 4.
Fig. 14 is a flowchart showing a display switching step in embodiment 5.
Fig. 15 is a block diagram illustrating an internal configuration of the surgical assist device according to embodiment 6.
Fig. 16 is a flowchart illustrating steps of processing performed by the surgical assist device according to embodiment 6.
Fig. 17 is an explanatory diagram illustrating a method of analyzing the operation result.
Fig. 18 is a diagram showing an example of an evaluation coefficient table.
Fig. 19 is a diagram showing an example of the calculation result of the score.
Fig. 20 is a flowchart showing steps of a process performed by the surgical assist device of embodiment 7.
Fig. 21 is a block diagram illustrating an internal configuration of the surgical assist device according to embodiment 8.
Fig. 22 is a schematic diagram showing a display example in embodiment 8.
Fig. 23 is a schematic diagram showing a display example of the recognition result corresponding to the certainty factor.
Fig. 24 is a schematic view showing a configuration example of a user interface provided in the surgical assist device.
Detailed Description
The mode of applying the present invention to an auxiliary system for laparoscopic surgery will be specifically described below with reference to the drawings. The present invention is not limited to laparoscopic surgery, and can be applied to all endoscopic surgery using imaging devices such as thoracoscopes, digestive tract endoscopes, cystoscopes, arthroscopes, robotic-assisted surgery, spinal endoscopes, surgical microscopes, neuroendoscopes, and endoscopes.
(embodiment 1)
Fig. 1 is a schematic diagram illustrating a schematic configuration of a laparoscopic surgery support system according to embodiment 1. In laparoscopic surgery, instead of performing open surgery, a plurality of tapping devices called puncture devices (troca) 10 are attached to the abdominal wall of a patient, and tools such as a laparoscope 11, an energy treatment device 12, and forceps 13 are inserted into the patient from the tapping devices provided in the puncture devices 10. The operator performs a treatment such as cutting out an affected part using the energy treatment tool 12 while observing an image (operation field image) of the patient's body captured by the laparoscope 11 in real time. Surgical instruments such as the laparoscope 11, the energy treatment instrument 12, and the forceps 13 are held by an operator, a robot, or the like. The operator is a medical worker related to laparoscopic surgery, including a doctor who performs surgery, an assistant, a nurse, a doctor who monitors surgery, and the like.
The laparoscope 11 includes an insertion part 11A inserted into a patient, a photographing device 11B built in a front end portion of the insertion part 11A, an operation part 11C provided in a rear end portion of the insertion part 11A, and a universal cord 11D for connecting to a Camera Control Unit (CCU) 110 or a light source device 120.
The insertion portion 11A of the laparoscope 11 is formed of a rigid tube. A bending portion is provided at the distal end portion of the rigid tube. The bending mechanism in the bending section is a known mechanism incorporated in a general laparoscope, and is configured to bend in four directions, for example, up, down, left, and right, by pulling an operation wire that is linked to the operation of the operation section 11C. The laparoscope 11 is not limited to the above-described soft scope having a curved portion, and may be a hard scope having no curved portion, or may be an imaging device having no curved portion or hard tube.
The imaging device 11B includes a driving circuit including a solid-state imaging element such as CMOS (Complementary Metal Oxide Semiconductor), a Timing Generator (TG), an analog signal processing circuit (AFE), and the like. The driving circuit of the imaging device 11B captures signals of the respective colors of RGB output from the solid-state imaging element in synchronization with the clock signal output from the TG, and performs necessary processing such as noise removal, amplification, a/D conversion, and the like in the AFE to generate digital image data. The driving circuit of the photographing device 11B transmits the generated image data to the CCU110 through the universal cord 11D.
The operation unit 11C includes a corner lever, a remote switch, and the like, which are operated by an operator. The angle lever is an operating tool that receives an operation for bending the bending portion. Instead of the angle lever, a bending operation knob, a joystick, or the like may be provided. The remote switch includes, for example, a change-over switch for changing the observation image to a moving image display or a still image display, a zoom switch for enlarging or reducing the observation image, and the like. The remote switch may be assigned a predetermined specific function or may be assigned a function set by an operator.
Further, a vibrator constituted by a linear resonant actuator, a piezoelectric actuator, or the like may be incorporated in the operation unit 11C. When an event to be notified to the operator who operates the laparoscope 11 occurs, the CCU110 may also notify the operator of the occurrence of the event by vibrating the operation unit 11C by operating the vibrator incorporated in the operation unit 11C.
A transmission cable for transmitting a control signal output from the CCU110 to the imaging device 11B or image data output from the imaging device 11B, a light guide for guiding illumination light emitted from the light source device 120 to a front end portion of the insertion portion 11A, and the like are disposed inside the insertion portion 11A, the operation portion 11C, and the universal cord 11D of the laparoscope 11. The illumination light emitted from the light source device 120 is guided to the distal end portion of the insertion portion 11A by the light guide, and is irradiated to the surgical field through the illumination lens provided at the distal end portion of the insertion portion 11A. In the present embodiment, the light source device 120 is described as a separate device, but the light source device 120 may be built in the CCU 110.
The CCU110 includes a control circuit that controls the operation of the imaging device 11B included in the laparoscope 11, an image processing circuit that processes image data from the imaging device 11B input through the universal cord 11D, and the like. The control circuit includes CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like, and outputs control signals to the imaging device 11B in response to the operation of various switches provided in the CCU110 or the operation of the operation unit 11C provided in the laparoscope 11, thereby performing control such as imaging start, imaging stop, and zooming. The control circuit is not limited to CPU, ROM, RAM, and may include GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and the like. The image processing circuit includes DSP (Digital Signal Processor), an image memory, and the like, and performs appropriate processing such as color separation, color interpolation, gain correction, white balance adjustment, and gamma correction on the image data input through the universal cord 11D. CCU110 generates a frame image for a moving image from the processed image data, and sequentially outputs the generated frame images to operation support device 200 described later. The frame rate of the frame image is, for example, 30FPS (Frames Per Second).
CCU110 may generate image data that meets predetermined criteria such as NTSC (National Television System Committee), PAL (Phase Alternating Line), DICOM (Digital Imaging and COMmunication in Medicine), etc. The CCU110 can display the operation field image (video) on the display screen of the display device 130 in real time by outputting the generated video data to the display device 130. The display device 130 is a monitor provided with a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like. The CCU110 may output the generated video data to the video recording device 140, and may store the video data in the video recording device 140. The video recorder 140 includes a storage device such as HDD (Hard Disk Drive), and the HDD stores the video data outputted from the CCU110 together with an identifier for identifying each operation, date and time of the operation, operation place, patient name, operator name, and the like.
The operation support device 200 generates support information related to the laparoscopic surgery based on image data (i.e., image data of an operation field image obtained by photographing an operation field) input from the CCU 110. Specifically, the surgical assistance device 200 performs the following processing: the loose connective tissue portions contained in the surgical field image are identified, and the identified loose connective tissue portions are displayed on the display device 130 in a distinguishable manner. Here, the loose connective tissue portion included in the operation field image represents a collection of pixels corresponding to loose connective tissue within the operation field image.
In the present embodiment, the description has been given of the configuration in which the identification process of loose connective tissue is performed in the operation support device 200, but the configuration may be such that the same function as the operation support device 200 is provided in the CCU110 and the identification process of loose connective tissue is performed in the CCU 110.
Hereinafter, the internal configuration of the surgical assist device 200, and the identification process and the display process executed by the surgical assist device 200 will be described.
Fig. 2 is a block diagram illustrating an internal configuration of the operation support device 200. The surgical assistance device 200 is a dedicated or general-purpose computer including a control unit 201, a storage unit 202, an operation unit 203, an input unit 204, an output unit 205, a communication unit 206, and the like. The surgical auxiliary device 200 may be a computer provided in an operating room or a computer provided outside the operating room. The surgical assist device 200 may be a server installed in a hospital for performing laparoscopic surgery, or may be a server installed outside the hospital. The surgical assistance device 200 may also be used to assist in tele-surgery.
The control unit 201 includes, for example, a CPU, a ROM, and a RAM. A ROM provided in the control unit 201 stores a control program or the like for controlling operations of the hardware units provided in the surgical assist device 200. The CPU in the control unit 201 executes a control program stored in the ROM and various computer programs stored in a storage unit 202 described later, and controls the operations of the hardware units to thereby function as the whole of the surgical operation support device of the present application. The RAM provided in the control unit 201 temporarily stores data and the like used in the process of performing the operation.
In the present embodiment, the control unit 201 includes a CPU, a ROM, and a RAM, but the control unit 201 may be configured as any type, for example, an arithmetic circuit or a control circuit including one or more GPUs, FPGAs, quantum processors, volatile or nonvolatile memories, and the like. The control unit 201 may have a function such as a clock that outputs date and time information, a timer that measures an elapsed time from when the measurement start instruction is provided to when the measurement end instruction is provided, and a counter that counts the number of times.
The storage unit 202 includes a storage device using a hard disk, a flash memory, or the like. The storage unit 202 stores a computer program executed by the control unit 201, various data acquired from the outside, various data generated inside the device, and the like.
The computer program stored in the storage unit 202 includes: an identification processing program PG1 that causes the control section 201 to execute processing for identifying loose connective tissue portions contained in the surgical field image; a display processing program PG2 that causes the control section 201 to execute processing for displaying auxiliary information based on the recognition result on the display device 130; and a learning processing program PG3 for generating a learning model 300. The recognition processing program PG1 and the display processing program PG2 need not be separate computer programs, and may be implemented as a single computer program. These programs are provided, for example, by a non-transitory storage medium M that stores computer programs readable. The storage medium M is a portable memory such as a CD-ROM, a USB memory, or a SD (Secure Digital) card. The control unit 201 reads a desired computer program from the storage medium M using a reading device not shown in the figure, and stores the read computer program in the storage unit 202. Alternatively, the above-described computer program may be provided by communication using the communication section 206.
The learning model 300 used in the recognition processing program PG1 is stored in the storage unit 202. The learning model 300 is a learning model that learns to input to the surgical field image and outputs information on loose connective tissue portions included in the surgical field image. The learning model 300 is described by its definition information. The definition information of the learning model 300 includes parameters such as information of layers included in the learning model 300, information of nodes constituting each layer, weighting between nodes, and deviation. The learning model 300 stored in the storage unit 202 is a learning model obtained by learning using a predetermined learning algorithm, using a surgical field image obtained by capturing a surgical field and positive solution data representing loose connective tissue portions in the surgical field image as training data. The constitution of the learning model 300 and the generation step of the learning model 300 will be described in detail later.
The operation unit 203 includes operation devices such as a keyboard, a mouse, a touch panel, a non-contact panel, a stylus pen, and a microphone-based voice input device. The operation unit 203 receives an operation performed by an operator or the like, and outputs information related to the received operation to the control unit 201. The control section 201 executes appropriate processing based on the operation information input from the operation section 203. In the present embodiment, the operation support device 200 is provided with the operation unit 203, but may be configured to receive an operation by various devices such as the CCU110 connected to the outside.
The input unit 204 includes a connection interface for connecting input devices. In the present embodiment, the input device connected to the input unit 204 is the CCU110. Image data of the surgical field image photographed by the laparoscope 11 and processed by the CCU110 is input to the input unit 204. The input unit 204 outputs the input image data to the control unit 201. The control unit 201 may store the image data acquired from the input unit 204 in the storage unit 202.
The output unit 205 includes a connection interface for connecting output devices. In the present embodiment, the output device connected to the output unit 205 is the display apparatus 130. When information such as the recognition result of the learning model 300 is generated, which should be notified to the operator, the control unit 201 outputs the generated information from the output unit 205 to the display device 130, and displays the information on the display device 130.
The communication unit 206 includes a communication interface for transmitting and receiving various data. The communication interface provided in the communication unit 206 is a communication interface conforming to a wired or wireless communication standard used in ethernet (registered trademark) or WiFi (registered trademark). When data to be transmitted is input from the control section 201, the communication unit 206 transmits the data to be transmitted to a specified destination. When receiving data transmitted from an external device, the communication unit 206 outputs the received data to the control unit 201.
Next, an operation field image input to the operation support device 200 will be described.
Fig. 3 is a schematic diagram showing an example of an operation field image. The surgical field image in the present embodiment is an image obtained by photographing the inside of the abdominal cavity of the patient with the laparoscope 11. The surgical field image need not be an original image output by the imaging device 11B of the laparoscope 11, but may be an image (frame image) processed by the CCU110 or the like. The surgical field image may be an image output from the CCU110 to the display device 130, or may be an image processed by an image processing device (not shown) detachably attached to the laparoscope 11. The surgical field image may be a recorded video image already stored in the recording device 140.
Fig. 3 shows an example of an image of a surgical field obtained by photographing a case of a laparoscopic surgery. The surgical field shown in fig. 3 includes tissue NG including a lesion such as a malignant tumor, tissue ORG constituting an organ, and loose connective tissue LCT filling the space between these tissues. In the present embodiment, the tissue NG is a site to be removed from the body, and the tissue ORG is a site to be retained in the body. In the example of fig. 3, the loose connective tissue LCT is exposed by grasping the tissue NG with forceps 13 and expanding upward in the figure. Wherein connective tissue refers to tissue that contains elastic fibers, collagen fibers, adipose tissue, fine mesh tissue, etc., and fills in between the tissues. Loose connective tissue (loose connective tissue) LCT has the function of preserving organs and epithelium, is present between many organs or tissues, and is one of connective tissue with proteinaceous fibers. The more elastic fibers are called dense connective tissue (ligaments, tendons, etc.), and are distinguished from loose connective tissue. Loose connective tissue LCTs are mostly visually recognized as fibrous in surgery. The direction of the fibers is not certain, and a mesh may be formed as a whole. There is a liquid matrix and a variety of cells between the fibers. When a large amount of loose connective tissue is found in an operation, particularly when the separation or separation between organs is performed, the operation is safely performed by properly treating the loose connective tissue. In the example of fig. 3, loose connective tissue LCT is indicated by dashed lines.
In laparoscopic surgery, for example, surgery is performed to remove a lesion such as a malignant tumor formed in a patient. At this time, the operator holds the tissue NG including the lesion with the forceps 13 and spreads it in an appropriate direction, thereby exposing loose connective tissue LCT existing between the tissue NG including the lesion and the remaining tissue ORG. The operator uses the energy treatment tool 12 to excise the exposed loose connective tissue LCT, thereby peeling the tissue NG including the lesion from the tissue ORG to be preserved.
Further, from the viewpoint of ease of excision of the loose connective tissue LCT, it is preferable that the loose connective tissue LCT of the excision subject has stretchability. In addition, a space for moving the energy treatment tool 12 or forceps 13 is preferably provided on the inner side of the loose connective tissue LCT to be resected. Furthermore, it is preferred that the loose connective tissue LCT of the resected subject remains in a stressed state. The example of fig. 3 shows a situation where there is a space SP inside the loose connective tissue LCT and at least a part remains in a stressed state.
Around the fibers that make up the loose connective tissue LCT, there are blood vessels, nerves, or fluid matrix, various cells, and it is not necessarily easy for the operator to find the loose connective tissue LCT from the surgical field image. Accordingly, the surgical assistance device 200 of the present embodiment recognizes a loose connective tissue portion from an operation field image using the learning model 300, and outputs assistance information related to laparoscopic surgery according to the recognition result.
Next, a configuration example of the learning model 300 used in the operation support device 200 will be described.
Fig. 4 is a schematic diagram showing an exemplary configuration of the learning model 300. The learning model 300 is a learning model for image segmentation, and is constructed from a neural network having a convolutional layer, such as SegNet. Fig. 4 shows an exemplary structure of SegNet, but the structure is not limited to SegNet, and learning model 300 may be constructed using any neural Network capable of image segmentation, such as FCN (Fully Convolutional Network), U-Net (U-Shaped Network), PSPNet (Pyramid Scene Parsing Network), and the like. Instead of the neural network for image segmentation, the learning model 300 may be constructed using a neural network for object detection such as YOLO (You Only Look Once) and SSD (Single Shot Multi-Boox Detector).
In the present embodiment, the input image input to the learning model 300 is a surgical field image obtained from the laparoscope 11. The learning model 300 learns to output information related to loose connective tissue (for example, a probability indicating whether or not each pixel belongs to loose connective tissue) for the input of the operation field image.
The learning model 300 in the present embodiment includes, for example, an encoder 310, a decoder 320, and a softmax layer 330. The encoder 310 is configured by alternately disposing a convolution layer and a pooling layer. The convolution layers are multilayered into 2 to 3 layers. In the example of fig. 4, the convolutional layers are shown as unshaded and the pooling layers are shown as shaded.
In the convolution layer, convolution operations of input data and filters of respectively determined sizes (for example, 3×3, 5×5, etc.) are performed. That is, the input value input to the position corresponding to each element of the filter is multiplied by the weight coefficient set in the filter in advance for each element, and the linear sum of the multiplication values of each of these elements is calculated. The output in the convolutional layer is obtained by adding the calculated linear sum to the set deviation. In addition, the result of the convolution operation may also be converted by an activation function. As the activation function, reLU (Rectified Linear Unit) can be used, for example. The output of the convolution layer represents a feature map obtained by extracting features of the input data.
In the pooling layer, local statistics of the feature map output from the convolution layer connected to the upper layer on the input side are calculated. Specifically, a window of a predetermined size (for example, 2×2, 3×3) corresponding to the position of the upper layer is set, and local statistics are calculated from input values in the window. As the statistic, for example, a maximum value can be employed. The size of the feature map output from the pooling layer is scaled down (downsampled) according to the size of the window. The example of fig. 4 shows a feature map in which the input image of 224 pixels×224 pixels is sequentially downsampled to 112×112, 56×56, 28×28, …, 1×1 by sequentially repeating the operation in the convolutional layer and the operation in the pooling layer in the encoder 310.
The output of the encoder 310 (a 1 x 1 feature map in the example of fig. 4) is input to the decoder 320. The decoder 320 is constructed by alternately disposing deconvolution layers and anti-pooling layers. The deconvolution layer is multilayered into 2 to 3 layers. In the example of fig. 4, the deconvolution layer is shown as non-hatched and the deconvolution layer is shown as hatched.
In the deconvolution layer, deconvolution operation is performed on the input feature map. The deconvolution operation is an operation for restoring a feature map before a convolution operation, based on the assumption that the input feature map is a result obtained by the convolution operation using a specific filter. In this operation, when a specific filter is represented by a matrix, a product of the transposed matrix of the matrix and the input feature map is calculated to generate a feature map for output. The operation result of the deconvolution layer may be converted by the activation function such as ReLU described above.
The anti-pooling layer provided by the decoder 320 corresponds one-to-one with the pooling layer provided by the encoder 310, and the corresponding pairs have substantially the same size. The anti-pooling layer re-increases (upsamples) the size of the feature map downsampled in the pooling layer of the encoder 310. The example of fig. 4 shows a feature map of sequentially up-sampling 1×1, 7×7, 14×14, …, 224×224 by sequentially repeating the operations in the convolutional layer and the operations in the pooling layer in the decoder 320.
The output of the decoder 320 (in the example of fig. 4, a 224 x 224 feature map) is input to the softmax layer 330. The softmax layer 330 outputs probabilities of tags identifying locations in locations (pixels) by applying a softmax function to input values from the deconvolution layer connected to the input side. In this embodiment, a label for identifying loose connective tissue is set, and whether loose connective tissue is included in units of pixels may be identified. By extracting pixels whose probability of the label output from the softmax layer 330 is a threshold or more (for example, 50% or more), an image representing a loose connective tissue portion (hereinafter referred to as a recognition image) is obtained. The threshold value may be stored in the storage unit 202 in advance. Further, the operation unit 203 may receive a change in the threshold value and store the changed threshold value in the storage unit 202. In this case, the control unit 201 may determine whether or not each pixel belongs to loose connective tissue using the changed threshold value.
In the example of fig. 4, the 224-pixel×224-pixel image is used as the input image to the learning model 300, but the size of the input image is not limited to the above-described size, and may be appropriately set according to the processing capability of the surgical assist device 200, the size of the surgical field image obtained from the laparoscope 11, and the like. The input image to the learning model 300 need not be the entire surgical field image obtained from the laparoscope 11, but may be a partial image generated by cutting out the region of interest of the surgical field image. Since the region of interest including the processing target is located in the vicinity of the center of the surgical field image, for example, a partial image obtained by cutting the vicinity of the center of the surgical field image into a rectangular shape so as to be about half the original size may be used. By reducing the size of the image input to the learning model 300, the recognition accuracy can be improved while the processing speed is improved.
The learning model 300 may also be configured to identify a portion or all of the fibrous loose connective tissue as an aggregate. That is, the learning model 300 may be configured to identify each of the loose connective tissues as one aggregate, or may be configured to identify a predetermined number (e.g., 10) or more of the loose connective tissues as one aggregate.
Fig. 5 is a schematic diagram showing the recognition result of the learning model 300. In the example of fig. 5, the loose connective tissue portion identified using the learning model 300 is shown by a thick solid line, and the portion of the organ or tissue other than the loose connective tissue portion is shown by a broken line as a reference. The control unit 201 of the surgical assist device 200 generates a loose connective tissue identification image, and displays the identified loose connective tissue portion in a distinguishable manner. The identification image is an image of the same size as the surgical field image, and is an image in which a specific color is assigned to a pixel identified as loose connective tissue. The color assigned to the pixels of loose connective tissue is preferably a color that does not exist inside the human body so as to be distinguished from organs, blood vessels, and the like. The color that does not exist in the human body is, for example, a cold color system (blue system) such as blue or bluish. Further, information indicating transparency is added to each pixel constituting the identification image, an opaque value is set to a pixel identified as loose connective tissue, and a transparent value is set to other pixels. By displaying the thus generated identification image superimposed on the operation field image, the loose connective tissue portion can be displayed as a structure having a specific color on the operation field image. The control unit 201 may adjust parameters such as hue, brightness, chroma, and transparency in order to display the recognized image of loose connective tissue easily.
The operation of the surgical assist device 200 will be described below.
The surgical assist device 200 generates the learning model 300 in a learning phase before the start of the operation, for example. As a preparation stage for generating the learning model 300, in the present embodiment, annotation is performed by manually dividing loose connective tissue portions with respect to an operation field image obtained from the laparoscope 11. Note that, the operation field image recorded in the video recording device 140 may be used for the annotation.
In the annotation, an operator (expert such as doctor) displays the surgical field images in time series on the display device 130, and at the same time, finds loose connective tissue existing between the tissue including the lesion (the portion to be removed) and the organ (the portion to be retained) in a state of easy excision. Specifically, the tissue including the lesion is developed, and loose connective tissue in an exposed state is found. The preferred connective tissue for annotation is, for example, a stretch-forming portion. Alternatively, loose connective tissue, preferably carrying out annotation, is kept in tension. Alternatively, a space is preferably provided on the back side of the loose connective tissue to which the annotation is applied, and a space is provided for the energy treatment tool 12 or forceps 13 to move. When the operator finds loose connective tissue that is in a condition of easy excision, the operator selects a portion conforming to the loose connective tissue in pixel units in the surgical field image by using a mouse, a stylus, or the like provided in the operation unit 203. Alternatively, a pattern of loose connective tissue suitable for learning may be selected, and the number of data may be increased by a perspective conversion or mirroring process or the like. Moreover, if learning progresses, the recognition result of the learning model 300 may be stolen to increase the data number. By performing the annotation as described above to generate the learning model 300, the learning model 300 is configured to identify the fibrous portion at a stage when loose connective tissue having elasticity is transferred from a pre-stressed state to a stressed state.
In this embodiment, about 4000 surgical field images are annotated, and by increasing the number of data, about 20000 sets of training data consisting of a group of the surgical field images and forward solution data representing loose connective tissue portions are finally prepared. The training data is stored in a storage device (e.g., the storage portion 202 of the surgical assistance device 200).
Fig. 6 is a flowchart illustrating the steps of generating the learning model 300. The control unit 201 of the surgical assist device 200 reads the learning process program PG3 from the storage unit 202, and generates the learning model 300 by executing the following steps. In addition, at a stage before the start of learning, it is assumed that initial values are given to definition information describing the learning model 300.
The control unit 201 first accesses the storage unit 202 and selects a set of training data for learning (step S101). The control unit 201 inputs the surgical field image included in the selected training data to the learning model 300 (step S102), and executes the operation performed by the learning model 300 (step S103). That is, the control unit 201 generates a feature map from the input surgical field image, and performs an operation performed by the encoder 310 that sequentially downsamples the generated feature map, an operation performed by the decoder 320 that sequentially upsamples the feature map input from the encoder 310, and an operation performed by the softmax layer 330 that identifies each pixel of the feature map that is finally obtained from the decoder 320.
The control unit 201 acquires the calculation result from the learning model 300, and evaluates the acquired calculation result (step S104). For example, the control unit 201 may evaluate the operation result by calculating the similarity between the image data of the loose connective tissue portion obtained as the operation result and the forward solution data included in the training data. The similarity is calculated, for example, by Jaccard coefficients. When the loose connective tissue portion extracted from the learning model 300 is set as a and the loose connective tissue portion included in the forward solution data is set as B, jaccard coefficients are provided by a n B/a u b×100 (%). Instead of Jaccard coefficients, either the Dice coefficients or the Simpson coefficients may be calculated, or the similarity may be calculated using other existing methods.
The control unit 201 determines whether or not learning is completed based on the evaluation of the calculation result (step S105). When the similarity equal to or greater than the preset threshold is obtained, the control section 201 can determine that the learning is ended.
When it is determined that the learning is not completed (S105: no), the control unit 201 sequentially updates the weight coefficients and the deviation in each layer of the learning model 300 from the output side to the input side of the learning model 300 by using the reverse error propagation method (step S106). After updating the weight coefficient and the deviation of each layer, the control unit 201 returns the process to step S101, and executes the process from step S101 to step S105 again.
When it is determined in step S105 that the learning is completed (yes in S105), the control unit 201 ends the processing performed in the present flowchart, since the learning model 300 in which the learning is completed is obtained.
In the present embodiment, the learning model 300 is generated in the surgical assist device 200, but the learning model 300 may be generated in an external computer. The surgical assistance device 200 may acquire the learning model 300 generated by an external computer by using communication or the like, and store the acquired learning model 300 in the storage unit 202.
The surgical assistance device 200 performs surgical assistance in the operation phase after the learning model 300 is generated. Fig. 7 is a flowchart illustrating the steps performed for surgical assistance. The control unit 201 of the surgical assist device 200 reads out and executes the identification processing program PG1 and the display processing program PG2 from the storage unit 202, thereby executing the following steps. When the laparoscopic surgery starts, an operation field image obtained by photographing an operation field with the photographing device 11B of the laparoscope 11 is output to the CCU110 via the universal cord 11D at any time. The control unit 201 of the surgical assist device 200 acquires the surgical field image output from the CCU110 via the input unit 204 (step S121). The control section 201 performs the following processing every time an operation field image is acquired.
The control unit 201 inputs the acquired operation field image into the learning model 300, performs an operation using the learning model 300 (step S122), and identifies a loose connective tissue portion included in the operation region image (step S123). That is, the control unit 201 generates a feature map from the input surgical field image, and performs an operation performed by the encoder 310 that sequentially downsamples the generated feature map, an operation performed by the decoder 320 that sequentially upsamples the feature map input from the encoder 310, and an operation performed by the softmax layer 330 that identifies each pixel of the feature map that is finally obtained from the decoder 320. In addition, the control unit 201 recognizes a pixel whose probability of the label output from the softmax layer 330 is a threshold or more (for example, 50% or more) as a loose connective tissue portion.
The control unit 201 generates an identification image of loose connective tissue, and displays the loose connective tissue portion identified using the learning model 300 in a distinguishable manner (step S124). As described above, the control unit 201 may assign colors (for example, colors of a cold color system (blue system) such as blue or light blue) that are not present in the human body to the pixels identified as loose connective tissue, and set the transparency of the background to the pixels other than the loose connective tissue.
The control unit 201 outputs the identification image of loose connective tissue generated in step S124 to the display device 130 together with the surgical field image acquired in step S121 from the output unit 205, and superimposes the identification image on the surgical field image and displays it on the display device (step S125). Thus, the loose connective tissue portions identified using the learning model 300 are displayed as structures having a specific color on the surgical field image. The control unit 201 may display a message indicating that loose connective tissue represented by a specific color is a site to be excised on the display device 130.
Fig. 8 is a schematic diagram showing a display example in the display device 130. For ease of drawing production, in the display example of fig. 8, the loose connective tissue portions identified using the learning model 300 are indicated by thick solid lines. In fact, since the portion conforming to the loose connective tissue portion is coated in a pixel unit with a color that does not exist inside the human body such as blue or bluish, an operator can clearly recognize the loose connective tissue and grasp a portion to be excised by viewing the display screen of the display device 130.
In the present embodiment, the recognition result of the loose connective tissue of the learning model 300 is displayed on the display device 130, but a display device different from the display device 130 may be provided, and the surgical field image may be displayed on the display device 130, so that the recognition result of the learning model 300 may be displayed on another display device. The operation field image may be displayed in an area within the screen of the display device 130, and the recognition result of the learning model 300 may be displayed in another area within the same screen.
In the present embodiment, the loose connective tissue portion identified by the learning model 300 is displayed on the display device 130, but a line intersecting the identified loose connective tissue portion may be displayed on the display device 130 as a recommended line to be excised.
In the present embodiment, since the loose connective tissue portion is recognized by the learning model 300, the line to be excised can be estimated, and therefore, when the surgical robot is connected to the surgical assist device 200, a control signal indicating excision of the loose connective tissue can be output to the surgical robot.
The surgical assistance device 200 may set a recommended range to be excised in the identified loose connective tissue portion, and display the recommended range by changing the display mode (color, hue, brightness, chroma, transparency, etc.) of the set recommended range on the display device 130. That is, the transparency may also be changed to display the loose connective tissue portion of the recommended range, and not to display the loose connective tissue portion outside the recommended range. In addition, the display color may be changed in the loose connective tissue portion outside the recommended range and the loose connective tissue portion outside the recommended range.
The recommended range is appropriately set. For example, in the case where loose connective tissue between a lesion tissue (a site that should be removed by endoscopic surgery) and a normal tissue (a site that should be preserved by endoscopic surgery) is identified, the surgical assist device 200 may divide a range including the loose connective tissue into three ranges of a range close to the lesion tissue, a range close to the normal tissue, and a range therebetween, and set any one of the divided three ranges as the recommended range. The surgical assist device 200 may calculate the length in the longitudinal direction of loose connective tissue (the direction in which the fibers extend) in the surgical field image, for example, and divide the calculated length into three portions. The number of the division ranges is not limited to three, but may be two or more.
In addition, the surgical assistance device 200 may set the recommended range according to the degree of progress of the lesion. For example, in a case where the lesion has a high degree of progress, the surgical assistance device 200 may set a range close to the normal tissue as the recommended range in order to obtain a large margin. In contrast, in the case where the degree of progress of the lesion is low, in order to reduce the resection range, the surgical assistance device 200 may also set the range near the lesion tissue as the recommended range. Further, information on the degree of progress of the lesion may be input in advance through the operation section 203 or the communication section 206.
The surgical assistance device 200 may set a recommended range according to the operator. For example, when the operator prefers to cut the range near the lesion tissue, the surgical assistance device 200 may set the range near the lesion tissue as the recommended range. In contrast, when the operator prefers to cut the range close to the normal tissue, the surgical assistance device 200 may set the range close to the normal tissue as the recommended range. The range of resection favored by the operator can be set in advance by the operation unit 203 or the communication unit 206.
As described above, in the present embodiment, since the structure of loose connective tissue can be identified using the learning model 300 and the loose connective tissue can be displayed in a distinguishable manner in units of pixels, visual assistance in laparoscopic surgery can be performed. The image generated from the operation support device 200 is used not only for operation support but also for education support of a nursing doctor or the like, and is also used for evaluation of laparoscopic surgery. For example, by comparing the image recorded in the intra-operative video recording device 140 with the image generated by the operation support device 200, it is possible to evaluate the laparoscopic surgery by judging whether or not the site excised by the laparoscopic surgery is appropriate.
(embodiment 2)
In embodiment 2, a description will be given of a configuration in which a display mode is changed according to the certainty when loose connective tissue is recognized.
Note that, since the overall configuration of the laparoscopic surgery support system, the internal configuration of the surgery support device 200, and the like are the same as those of embodiment 1, the description thereof is omitted.
The surgical assistance device 200 according to the aforementioned embodiment 1 generates an identification image representing a loose connective tissue portion by assigning a specific color (for example, a color of a cool color system) and transparency 1 (opaque) to pixels having a probability of a threshold value or more (for example, 50% or more) and assigning transparency 0 (completely transparent) to pixels having a probability of less than the threshold value by referring to the probability output from the softmax layer 330 of the learning model 300. The surgical assist device 200 can uniformly display (overlay) the loose connective tissue portion by outputting such an identification image and overlaying it on the surgical field image.
In contrast, the surgical assistance device 200 according to embodiment 2 sets a specific color (for example, a color of a cool color system) for each pixel of the identification image, sets transparency for each pixel according to the probability (certainty) of being output from the softmax layer 330 of the learning model 300, and generates an identification image of the loose connective tissue portion. Specifically, the surgical assist device 200 sets the transparency of each pixel such that the higher the certainty factor, the lower the transparency, and the lower the certainty factor, the higher the transparency. For example, the transparency at a certainty factor of X% can be set to X/100. The operation support device 200 outputs the generated identification image and displays the generated identification image superimposed on the operation field image, thereby enabling display (soft map display) according to the degree of certainty.
Fig. 9 is a schematic diagram showing a display example of a recognition image in embodiment 2. For ease of drawing production, the transparency is shown by the concentration in the example of fig. 9. That is, in the example of fig. 9, the identification image is represented by increasing the concentration of the loose connective tissue portion with high certainty and decreasing the concentration of the loose connective tissue portion with low certainty.
In embodiment 2, since a loose connective tissue portion with relatively high certainty can be clearly displayed, useful information can be accurately presented to an operator when a pulling operation, a peeling operation, or the like is performed.
In embodiment 2, the transparency is changed according to the certainty factor, but the color, hue, chroma, brightness, and the like may be changed according to the certainty factor.
Embodiment 3
In embodiment 3, a description will be given of a configuration in which a site to be removed by a laparoscopic surgery and a site to be retained by a laparoscopic surgery are identified and displayed together with a loose connective tissue portion included in an operation field image.
Note that, since the overall configuration of the laparoscopic surgery support system, the internal configuration of the surgery support device 200, and the like are the same as those of embodiment 1, the description thereof is omitted.
Fig. 10 is an explanatory diagram illustrating the configuration of the softmax layer 330 of the learning model 300 in embodiment 3. In fig. 10, the softmax layer 330 is represented as one-dimensional for simplicity. The softmax layer 330 of the learning model 300 outputs probabilities of labels identifying locations in the pixels of the feature map as described in embodiment 1. In embodiment 3, a tag for identifying loose connective tissue, a tag for identifying a site to be removed (tissue NG in the example of fig. 3), and a tag for identifying a site to be reserved (tissue ORG in the example of fig. 3) are set. If the probability of identifying the tag of loose connective tissue is equal to or greater than the threshold value, the control unit 201 of the surgical assist device 200 recognizes the pixel as loose connective tissue. Similarly, if the probability of the label identifying the portion to be removed is equal to or greater than the threshold value, the control unit 201 recognizes the pixel as the portion to be removed, and if the probability of the label identifying the portion to be reserved is equal to or greater than the threshold value, the control unit 201 recognizes the pixel as the portion to be reserved.
The learning model 300 for obtaining such a recognition result is generated by learning using a lot of training data prepared in advance. In embodiment 3, a set of forward solution data obtained by dividing a tissue portion including a lesion such as a malignant tumor, a tissue portion constituting an organ, and a loose connective tissue portion joined between these tissues can be used as training data. The method for generating the learning model 300 is the same as that of embodiment 1, and therefore, the description thereof is omitted.
Fig. 11 is a schematic diagram showing a display example in embodiment 3. The operation support device 200 according to embodiment 3 recognizes a loose connective tissue portion included in an operation field image, a region to be removed by a laparoscopic surgery (a tissue portion including a lesion such as a malignant tumor), and a region to be preserved by a laparoscopic surgery (a tissue portion constituting an organ) by using the learning model 300, and therefore displays these on the display device 130 in a distinguishable manner. In the example shown in fig. 11, the loose connective tissue portion identified using the learning model 300 is indicated by a thick solid line, and the region that should be removed by laparoscopic surgery (the region on the upper side of the loose connective tissue portion) and the region that should remain (the lower side of the loose connective tissue portion) are indicated by different hatching, respectively. In practice, the portion conforming to the loose connective tissue is colored in a pixel unit with a color which does not exist in the human body such as blue or light blue, and the region to be removed by the laparoscopic surgery and the region to be preserved are respectively colored with different colors and displayed. By viewing the display screen of the display device 130, the operator can clearly identify loose connective tissue to be joined to the site to be removed by laparoscopic surgery and the site to be preserved.
Instead of coloring the entire region to be removed by laparoscopic surgery and the region to be retained, only the outline of the region may be colored and displayed. Instead of coloring both the area removed by laparoscopic surgery and the area to be reserved, only one of the areas may be displayed by coloring.
As described above, in embodiment 3, at least one of the region to be removed by the laparoscopic surgery and the region to be reserved is displayed in a distinguishable manner together with the loose connective tissue portion recognized by the learning model 300, so that the operator can be clearly presented with information of the loose connective tissue portion to be excised.
In embodiment 4, a description will be given of a structure in which a loose connective tissue portion is displayed at a timing instructed by an operator.
The operation support device 200 according to embodiment 4 switches between display and non-display of the loose connective tissue portion according to the switch operation of the operator. That is, the surgical assistance device 200 displays the loose connective tissue portion on the display device 130 only when a specific switch (hereinafter referred to as a change-over switch) is operated. Here, the switch for switching between the display and non-display of the loose connective tissue portion may be a switch provided in the operation unit 11C of the laparoscope 11, or may be a foot switch not shown in the figure. The CCU110 is notified of operation information indicating that the change-over switch is operated, for example, via the universal cord 11D, and the surgical auxiliary device 200 is notified via the CCU 110.
Fig. 12 is a schematic diagram showing a display example in embodiment 4. Fig. 12A shows a display example in the case where the change-over switch is operated, and fig. 12B shows a display example in the case where the change-over switch is not operated. In embodiment 4, only in the case where the changeover switch is operated, as shown in fig. 12A, the loose connective tissue portion is displayed on the display device 130. The loose connective tissue portion may be displayed by coloring a portion conforming to the loose connective tissue in a color that does not exist in the human body such as blue or light blue in pixel units, as in embodiment 1.
Fig. 13 is a flowchart showing a display switching procedure in embodiment 4. The control unit 201 of the surgical assist device 200 determines whether or not the changeover switch is operated based on the operation information notified from the CCU110 (step S401).
When it is determined that the changeover switch is operated (yes in step S401), the control unit 201 displays the loose connective tissue portion (step S402). In order to display the loose connective tissue portion identified using the learning model 300 in a distinguishable manner, the control unit 201 assigns a specific color to the pixels corresponding to the portion, and generates an identification image in which transparency is set to be transparent to the background of pixels other than the loose connective tissue. The control unit 201 outputs the generated identification image to the display device 130 together with the surgical field image, and displays the loose connective tissue portion by superimposing the identification image on the surgical field image.
If it is determined that the selector switch is not operated (no in step S401), the control unit 201 sets the loose connective tissue portion to be not displayed (step S403). The control unit 201 may set the transparency of the background transparent to the pixels of the recognized loose connective tissue portion so that the loose connective tissue portion is not displayed. The control unit 201 outputs the generated identification image to the display device 130 together with the surgical field image, and can display no loose connective tissue portion by displaying the identification image superimposed on the surgical field image. Instead of changing the transparency, the loose connective tissue portion may be configured not to be displayed, or the output of the identification image may be stopped.
As described above, in embodiment 4, the loose connective tissue portion can be displayed at a timing desired by the operator, and can be set to be not displayed at other timings.
In the present embodiment, the display is made when the change-over switch is operated and not made when the change-over switch is not operated, but may be made not to be displayed when the change-over switch is operated and not to be operated. A switch for switching the loose connective tissue portions between displayed and not displayed may also be provided in CCU110. Further, the display and non-display of the loose connective tissue portion may be switched by the operation of the touch panel provided in the display device 130 or the operation unit 203 provided in the operation support device 200. In the present embodiment, the display is switched using a physical switch, but the display may be switched according to a voice instruction from an operator. Therefore, a voice input unit such as a microphone may be provided in the operation unit 11C of the laparoscope 11 or the CCU110.
Embodiment 5
In embodiment 5, a configuration in which the display and non-display of the loose connective tissue portion are switched according to the case of laparoscopic surgery will be described.
Depending on the speed of operation of the surgical assistance device 200, time lags may occur in the identification process of loose connective tissue. Therefore, when the loose connective tissue portion identified using the learning model 300 is displayed while the forceps 13 is used to expand a portion including a lesion or the like or while the energy treatment tool 12 is moved, there is a possibility that a deviation may occur between the displayed position of the loose connective tissue portion and the actual position of the loose connective tissue. Therefore, the operation support device 200 according to embodiment 5 displays a loose connective tissue portion when the target site of the laparoscopic surgery is stopped, and does not display a loose connective tissue portion when the energy treatment tool 12 starts to move.
Fig. 14 is a flowchart showing a display switching procedure in embodiment 5. In this flowchart, a display switching procedure when the energy treatment tool 12 is used to excise loose connective tissue after the forceps 13 is used to expand a portion including a lesion and the like and expose the loose connective tissue will be described.
The control unit 201 of the surgical assist device 200 determines whether or not the expansion of the target site is stopped based on the surgical field images sequentially input from the input unit 204 (step S501). For example, the control unit 201 may determine whether or not the expansion of the target portion is stopped by generating an optical flow from the surgical field images sequentially input from the input unit 204. If the expansion of the target site is not stopped (S501: NO), the loose connective tissue may move with the expansion of the target site, and thus stand by until the expansion of the target site is stopped. In the present embodiment, it is determined whether or not the expansion of the target portion is stopped, but it may be determined whether or not the surgical tool (for example, forceps 13) for expanding the target portion is stopped. Further, the entire surgical field image may be not stopped, but it may be determined whether or not a predetermined region (for example, a region near the center of the surgical field image) is stopped.
When it is determined that the expansion of the target site is stopped (yes in step S501), the control unit 201 displays the loose connective tissue portion (step S502). When the expansion of the target site is stopped, it is considered that loose connective tissue is stationary, and even if the recognized loose connective tissue is displayed, the possibility of the display position shift is low. In order to display the loose connective tissue portion identified using the learning model 300 in a discriminable manner, the control unit 201 assigns a specific color to the pixels corresponding to the portion, and generates an identification image in which transparency is set to be transparent to the background for the pixels other than the loose connective tissue portion. The control unit 201 outputs the generated identification image to the display device 130 together with the surgical field image, and displays the loose connective tissue portion by displaying the identification image superimposed on the surgical field image.
Next, the control unit 201 determines whether or not the treatment is started (step S503). As in step S501, the control unit 201 generates an optical flow, and determines whether or not the energy treatment tool 12 as the surgical tool starts to move, thereby determining whether or not the treatment has started. When the treatment is not started (S503: no), the control unit 201 waits until the treatment is started, that is, until the energy treatment tool 12 starts to move while continuing to display the loose connective tissue portion.
When it is determined that the treatment has started (yes in step S503), the control unit 201 sets the loose connective tissue portion to be not displayed (step S504). In order to set the loose connective tissue portion to be not displayed, the control unit 201 may set the transparency of the background transparent to the pixels of the recognized loose connective tissue portion. The control unit 201 outputs the generated identification image to the display device 130 together with the surgical field image, and can display no loose connective tissue portion by displaying the identification image superimposed on the surgical field image. Instead of changing the transparency, the loose connective tissue portion may be configured not to be displayed, or the output of the identification image may be stopped.
As described above, in embodiment 5, since the loose connective tissue portion is displayed until the expansion of the target site is stopped until the treatment is started, an image that does not cause discomfort due to time lag can be provided to the operator.
In embodiment 6, a description will be given of a configuration in which the surgical assist device 200 includes a plurality of learning models.
Fig. 15 is a block diagram illustrating an internal configuration of the surgical assist device 200 according to embodiment 6. The surgical assistance device 200 according to embodiment 6 includes a first learning model 410 and a second learning model 420. Other components of the surgical assist device 200 and the overall configuration of the system including the surgical assist device 200 are the same as those of embodiments 1 to 5, and therefore, the description thereof will be omitted. In the present embodiment, the description has been given of a configuration in which the surgical assist device 200 includes two types of learning models, but a configuration in which three or more types of learning models are provided may be employed.
In embodiment 6, the first learning model 410 is a learning model for identifying loose connective tissue existing in a range near a site (lesion tissue) removed by endoscopic surgery, and the second learning model 420 is a learning model for identifying loose connective tissue existing in a range near a site (normal tissue) that should be preserved by endoscopic surgery. Hereinafter, loose connective tissue existing in a region near a site (lesion tissue) removed by endoscopic surgery is also referred to as "outer loose connective tissue", and loose connective tissue existing in a region near a site (normal tissue) to be preserved by endoscopic surgery is also referred to as "inner loose connective tissue".
The first learning model 410 and the second learning model 420 are similar to the learning model 300 described in embodiment 1, and a learning model for image segmentation such as SegNet or a learning model for object detection such as YOLO is used. The first learning model 410 is generated by performing machine learning according to a predetermined algorithm using a plurality of sets of data sets including an operation field image and positive solution data obtained by selecting, in units of pixels, a portion of the operation field image that corresponds to loose connective tissue near a lesion tissue. Similarly, the second learning model 420 is generated by performing machine learning according to a predetermined algorithm using a plurality of sets of data sets including an operation field image and positive solution data obtained by selecting, in units of pixels, a portion of the operation field image that corresponds to loose connective tissue near normal tissue. Since the learning step is the same as that of embodiment 1, the description thereof will be omitted.
Fig. 16 is a flowchart illustrating steps of processing performed by the surgical assist device 200 according to embodiment 6. The control unit 201 of the surgical assist device 200 acquires information on the degree of progress of the lesion (step S601). In the case where the degree of progress of the lesion is determined in advance by a diagnostic method such as preoperative image diagnosis or pathological diagnosis, the control unit 201 can receive information on the degree of progress of the lesion through the operation unit 203 or the communication unit 206 before the laparoscopic surgery starts. The degree of progress of the lesion may be determined by an operator who confirms the surgical field image displayed on the display device 130. In this case, after the laparoscopic surgery is started, the control unit 201 can receive information on the degree of progress of the lesion through the operation unit 203 or the communication unit 206. In addition, a learning model for determining the degree of progress of the lesion may be separately prepared, and the control unit 201 may determine the degree of progress of the lesion by performing an operation based on the learning model.
The control unit 201 determines whether the degree of progress is high based on the information acquired in step S601 (step S602). For example, the control unit 201 compares a value indicating the degree of progress with a threshold value, and determines that the degree of progress is high when the value is equal to or higher than the threshold value, and determines that the degree of progress is low when the value is lower than the threshold value. When the degree of progress is determined to be high (yes in S602), the control unit 201 selects the second learning model 420 to obtain a large margin (step S603), and when the degree of progress is determined to be low (no in S602), selects the first learning model 410 to reduce the ablation range (step S604).
The control unit 201 performs the steps S605 to S609 similar to those of embodiment 1, recognizes loose connective tissue using the selected learning model (the first learning model 410 or the second learning model 420), superimposes the recognized image of the loose connective tissue on the operation field image, and displays the superimposed image on the display device 130.
In the case where the degree of progress of the lesion is high, since the loose connective tissue on the inner side (near the normal tissue) is identified and displayed on the display device 130, the operator can obtain a larger margin of the lesion tissue by cutting off the loose connective tissue on the inner side displayed on the display device 130. In addition, in the case where the degree of progress of the lesion is low, since the loose connective tissue on the outside (near the lesion tissue) is identified and displayed on the display device 130, the operator can reduce the resection range by resecting the loose connective tissue on the outside displayed on the display device 130.
In embodiment 6, the control unit 201 is configured to select the first learning model 410 or the second learning model 420 according to the degree of progress of the lesion, but may be configured to accept the selection of the learning model by the operator. That is, the control unit 201 may accept a selection of the first learning model 410 when the operator wishes to remove the loose connective tissue on the outside, and accept a selection of the second learning model 420 when the operator wishes to remove the loose connective tissue on the inside.
The first learning model 410 may be a learning model obtained by learning using training data annotated by a first doctor, and the second learning model 420 may be a learning model obtained by learning using training data annotated by a second doctor. The operation support device 200 can arbitrarily receive the operator's selection of the learning model through the operation unit 203 or the communication unit 206.
The first learning model 410 and the second learning model 420 may be learning models selected according to patient attributes. For example, the first learning model 410 may be a learning model selected for patients receiving chemotherapy and the second learning model 420 may be a learning model selected for obese patients. Other learning models selected based on the patient's age, sex, height, weight, etc. attributes may also be included. The surgical assistance device 200 may select a learning model (for example, the first learning model 410, the second learning model 420, or the like) corresponding to the attribute of the patient with reference to the patient information input from the outside, such as an electronic medical record.
The first learning model 410 and the second learning model 420 may be learning models selected according to whether bleeding is present or not. For example, the first learning model 410 may be a learning model selected without bleeding, and the second learning model 420 may be a learning model selected at the time of bleeding. In this case, the surgical assistance device 200 may select the first learning model 410 when bleeding is not detected, and select the second learning model 420 when bleeding is detected. In addition, a known method is used for detecting bleeding. For example, whether bleeding is present or not can be determined by detecting the expansion of a red region in an operation field image by image analysis.
The first learning model 410 and the second learning model 420 may be learning models selected according to the state of loose connective tissue. For example, the first learning model 410 may be a learning model that identifies loose connective tissue in a state covered by adherent tissue, and the second learning model 420 may be a learning model that identifies loose connective tissue in a state not covered by adherent tissue. The presence or absence of the healed tissue is determined by, for example, an operator. When the operator determines that there is adhered tissue and inputs information about the adhered tissue, the surgical assistance device 200 selects the first learning model 410. In addition, when the operator determines that there is no adhesion tissue and inputs information about the adhesion, the surgical assistance device 200 selects the second learning model 420.
The first learning model 410 and the second learning model 420 may be learning models selected according to the surgical field. For example, the first learning model 410 may be a learning model selected in the case where the operation region is the stomach, and the second learning model 420 may be a learning model selected in the case where the operation region is the large intestine. In addition, other learning models may be included that are selected when inguinal hernias, prostates, lungs, and the like are used as the surgical field. The operation support device 200 may select a learning model (for example, the first learning model 410, the second learning model 420, etc.) corresponding to the operation region with reference to patient information input from the outside, such as an electronic medical record.
The first learning model 410 and the second learning model 420 may be learning models selected according to the types of the laparoscope 11 and the imaging device 11B. For example, the first learning model 410 may be a learning model selected when using the laparoscope 11 manufactured by company a, and the second learning model 420 may be a learning model selected when using the laparoscope 11 manufactured by company B, which is different from company a. The surgical assistance device 200 may select the first learning model 410 or the second learning model 420 based on the device information input as the prior information.
The first learning model 410 may be a learning model constructed by SegNet, for example, and the second learning model 420 may be a learning model constructed by U-Net, for example, that learns with a different algorithm. In this case, the operation support device 200 may receive the operator's selection of the learning model through the operation unit 203 or the communication unit 206.
Embodiment 7
In embodiment 7, a description will be given of a configuration in which an optimal learning model is selected based on an inputted operation field image.
As in embodiment 6, the surgical assistance device 200 according to embodiment 7 includes a first learning model 410 and a second learning model 420. The first learning model 410 is, for example, a learning model constructed from SegNet, and the second learning model 420 is, for example, a learning model constructed from U-Net. The combination of the neural networks constructing the first learning model 410 and the second learning model 420 is not limited to the above, and any neural network may be used.
Alternatively, the first learning model 410 and the second learning model 420 may be learning models that internally constitute different. For example, the first learning model 410 and the second learning model 420 may be learning models constructed using the same neural network, but the kinds and layers of layers, the number of nodes, the connection relationship of the nodes, and the like are different.
In addition, the first learning model 410 and the second learning model 420 may be learning models that are learned using different training data. For example, the first learning model 410 may be a learning model obtained by learning using training data including forward solution data of a first expert annotation, and the second learning model 420 may be a learning model obtained by learning using training data including forward solution data of a second expert annotation different from the first expert. The first learning model 410 may be a learning model obtained by learning using training data including an operation field image captured by a medical institution and annotation data (forward solution data) for the operation field image, and the second learning model 420 may be a learning model obtained by learning using training data including an operation field image captured by another medical institution and annotation data (forward solution data) for the operation field image.
When the surgical field image is input, the surgical assistance device 200 executes an operation based on the first learning model 410 and an operation based on the second learning model 420 in the control unit 201. In order to execute these operations in parallel, the control unit 201 may further include a plurality of operation units (for example, a plurality of GPUs). The control unit 201 analyzes the operation result based on the first learning model 410 and the operation result based on the second learning model 420, and selects a learning model (the first learning model 410 or the second learning model 420) most suitable for recognizing loose connective tissue based on the analysis result.
Fig. 17 is an explanatory diagram illustrating a method of analyzing the operation result. From each learning model for identifying loose connective tissue, a probability (certainty) indicating whether each pixel corresponds to loose connective tissue is output as an operation result. When the pixel numbers are summed up for each degree of certainty, the distribution shown in fig. 17A to 17C can be obtained, for example. The horizontal axis of each graph shown in fig. 17A to 17C represents certainty, and the vertical axis represents the number of pixels (the proportion of the entire image). Ideally, each pixel is classified as having a certainty factor of 1 (in the case where the probability of loose connective tissue is 100%) or 0 (in the case where the probability of loose connective tissue is 0), and therefore when the distribution of certainty factor is studied based on the calculation result obtained from the ideal learning model, the two polarization distributions as shown in fig. 17A are obtained.
When the calculation results are acquired from the first learning model 410 and the second learning model 420, the control unit 201 of the surgical assist device 200 adds up the pixel numbers for each degree of certainty, and selects a learning model having a distribution close to an ideal distribution. For example, in the case where the distribution obtained from the operation result of the first learning model 410 is the distribution shown in fig. 17B, and the distribution obtained from the operation result of the second learning model 420 is the distribution shown in fig. 18C, since the latter is closer to the ideal distribution, the control section 201 selects the second learning model 420.
The control unit 201 evaluates each distribution by using an evaluation coefficient whose evaluation value becomes higher as the certainty factor approaches 1 or 0, for example, to determine whether the ideal distribution is approached. Fig. 18 is a diagram showing an example of an evaluation coefficient table. Such an evaluation coefficient table is prepared in advance in the storage section 202. In the example of fig. 18, the evaluation coefficient is set to a higher value as the certainty factor approaches 1 or 0.
When obtaining the result of the total of the number of pixels for each degree of certainty, the control unit 201 calculates a score indicating whether the distribution is good or bad by multiplying the result by an evaluation coefficient. Fig. 19 is a diagram showing an example of the calculation result of the score. Fig. 19A to 19C show the results of calculating scores for the respective distributions shown in fig. 17A to 17C. The score calculated from the ideal distribution is highest. The total score is 84 when the distribution obtained from the operation result of the first learning model 410 is calculated, and 188 when the distribution obtained from the operation result of the second learning model 420 is calculated. In this case, since the score of the second learning model 420 is higher than that of the first learning model 410, the control section 201 selects the second learning model 420 as an appropriate learning model.
Fig. 20 is a flowchart showing steps of a process executed by the surgical assist device 200 according to embodiment 7. When the operation field image is acquired (step S701), the control unit 201 executes the operation based on the first learning model 410 (step S702), and acquires the operation result of the first learning model 410 (step S703). The control unit 201 sums up the pixel numbers for each degree of certainty for the first learning model 410 (step S704), multiplies the pixel numbers by the evaluation coefficients, and calculates a score (first score) of the distribution (step S705).
Similarly, the control unit 201 performs an operation based on the second learning model 420 on the surgical field image acquired in step S701 (step S706), and acquires the operation result of the second learning model 420 (step S707). The control unit 201 sums up the pixel numbers for each degree of certainty to the second learning model 420 (step S708), multiplies the pixel numbers by the evaluation coefficients, and calculates a score (second score) of the distribution (step S709).
In the present flowchart, the steps are performed on the second learning model 420 (S706 to S709) after the operations on the first learning model 410 are performed (S702 to S705) for convenience, but these steps may be performed sequentially or simultaneously in parallel.
The control unit 201 compares the first score and the second score, and determines whether the first score is equal to or greater than the second score (step S710).
When determining that the first score is equal to or greater than the second score (yes in step S710), the control unit 201 selects the first learning model 410 as an appropriate learning model (step S711). Thereafter, the control section 201 performs the identification process of loose connective tissue using the selected first learning model 410.
If it is determined that the first score is smaller than the second score (S710: no), the control unit 201 selects the second learning model 420 as an appropriate learning model (step S712). Thereafter, the control section 201 performs the identification process of loose connective tissue using the selected second learning model 420.
As described above, in embodiment 7, a more appropriate learning model can be selected to perform the process of recognizing loose connective tissue.
The surgical assistance device 200 may also perform a process of recognizing loose connective tissue using the result of the operation of the first learning model 410 in the foreground and perform the operation based on the second learning model 420 in the background. The control unit 201 may perform evaluation of the first learning model 410 and the second learning model 420 at regular timings, and may switch the learning model for identifying loose connective tissue based on the evaluation result. The control unit 201 may evaluate the first learning model 410 and the second learning model 420 at a timing when an instruction is given by an operator or the like, and may switch the learning model for identifying loose connective tissue based on the evaluation result.
In embodiment 7, the method using the evaluation coefficients is described as the evaluation method of the first learning model 410 and the second learning model 420, but the method using the evaluation coefficients is not limited to the method, and various statistical indexes may be used for evaluation. For example, the control unit 201 may determine the variance or standard deviation of the distribution, and determine the two polarizations of the distribution when the variance or standard deviation is high. The control unit 201 may evaluate the calculation result of each model by obtaining the kurtosis or skewness of the graph by taking the value of the ratio (%) of 100 pixels as the value of the vertical axis of the graph. The control unit 201 may evaluate the calculation result of each model using the mode, the percentage, or the like.
In embodiment 8, a description will be given of a constitution for identifying loose connective tissue and nerves.
Fig. 21 is a block diagram illustrating an internal configuration of the surgical assist device 200 according to embodiment 8. The surgical assistance device 200 according to embodiment 8 includes a learning model 500 for identifying nerves in addition to the learning model 300 for identifying loose connective tissue. Other components of the operation support device 200 and the overall configuration of the system including the operation support device 200 are the same as those of embodiments 1 to 7, and therefore, the description thereof will be omitted.
In the learning model 500, similarly to the learning model 300 described in embodiment 1, a learning model for image segmentation such as SegNet, a learning model for object detection such as YOLO, or the like is used to learn as an input to an operation field image, and information on nerves (for example, a probability indicating whether each pixel belongs to a nerve) is output.
The learning model 500 is generated by performing machine learning according to a predetermined algorithm using a plurality of sets of data sets including an operation field image and positive solution data obtained by selecting a portion conforming to a nerve in pixel units in the operation field image in training data. Since the learning step is the same as that of embodiment 1, the description thereof will be omitted.
When the operation field image is acquired from the input unit 204, the control unit 201 of the operation support device 200 inputs the acquired operation field image into the learning model 500, and performs an operation based on the learning model 500. The control unit 201 recognizes pixels whose probability of a label output from the softmax layer of the learning model 500 is a threshold or more (for example, 50% or more) as a nerve portion.
The control section 201 may also perform an operation based on the learning model 300 in parallel with identifying the nerve portion to identify the loose connective tissue portion. The surgical assist device 200 may be configured to include a plurality of computation units (e.g., GPUs) for independently executing the computation based on the learning model 300 and the computation based on the learning model 500.
Fig. 22 is a schematic diagram showing a display example in embodiment 8. Fig. 22 shows an example in which the recognition result of the loose connective tissue portion of the learning model 300 and the recognition result of the nerve portion of the learning model 500 are displayed superimposed on the operation field image. The control unit 201 of the surgical assist device 200 can display the loose connective tissue portion and the nerve portion by generating an identification image representing the loose connective tissue portion and an identification image representing the nerve portion, respectively, and displaying the generated two identification images superimposed on the surgical field image. In this case, it is preferable to assign a specific color (for example, blue color) to the loose connective tissue portion and assign another color (for example, green color) to the nerve portion for display. Further, the loose connective tissue portion and the nerve portion may be arbitrarily switched and displayed by receiving a selection operation by the operation unit 203.
The control unit 201 may select the recognition result of the higher certainty factor when the recognition result of the loose connective tissue of the learning model 300 and the recognition result of the nerve of the learning model 500 are repeated, and may output information based on the selected recognition result. The certainty of the recognition result of the learning model 300 is calculated from the probability output from the softmax layer 330. For example, the control unit 201 may calculate the certainty factor by obtaining an average value of probability values for each pixel identified as loose connective tissue. The certainty of the recognition result of the learning model 500 is the same. For example, when the result of identifying the structure included in the operation field image by the learning model 300 is that the structure is identified as loose connective tissue with a degree of certainty of 95% and the result of identifying the same structure by the learning model 500 is that the structure is nerve with a degree of certainty of 60%, the control unit 201 may present the operator with a recognition result that the structure is loose connective tissue.
The control unit 201 of the surgical assist device 200 may display the recognition result in a display manner corresponding to each degree of certainty when the recognition result of the loose connective tissue of the learning model 300 and the recognition result of the nerve of the learning model 500 overlap (that is, when the same pixel is recognized as both the loose connective tissue and the nerve). Fig. 23 is a schematic diagram showing a display example of the recognition result corresponding to the certainty factor. In the example of fig. 23, the loose connective portion and the nerve portion are shown enlarged. For example, when a specific structure included in the surgical field image is recognized as loose connective tissue with 90% certainty, and the same structure is not recognized as nerve tissue (when the certainty is less than 50%), the control unit 201 colors a pixel corresponding to the structure with, for example, a blue color (black in the drawing), and presents the same to the operator. Similarly, when the structure included in the surgical field image is recognized as a nerve tissue with a degree of certainty of 90% and the same structure is not recognized as loose connective tissue by the learning model 300 (when the degree of certainty is less than 50%), the control unit 201 colors the structure with a color of, for example, a green system (white in the drawing) and presents the structure to the operator. On the other hand, when the structure included in the surgical field image is recognized as loose connective tissue with a degree of certainty of 60% and the same structure is recognized as nerve tissue with a degree of certainty of 60%, the control unit 201 colors the structure with, for example, an intermediate color (gray in the drawing) of a blue color and a green color, and presents the structure to the operator. The control unit 201 may determine the color to color the pixel corresponding to the structure based on the certainty factor of the loose connective tissue and the certainty factor of the nerve. For example, when the display color of the loose connective tissue portion is (0, B) and the display color of the nerve tissue portion is (0, G, 0), the control unit 201 may determine the display color of the pixel having the certainty factor of loose connective tissue X and the certainty factor of nerve tissue Y as (0, g×y/(x+y), b×x/(x+y), for example.
As described above, in embodiment 7, it is possible to identify loose connective tissue and nerves that are difficult for the operator to distinguish by the operation support device 200, and to present the identification result to the operator.
In the present embodiment, the description has been given of the constitution for identifying loose connective tissue and nerves, but other structures may be identified instead of nerves. Here, among other structures identified together with loose connective tissue, a structure similar to loose connective tissue such as lymphatic vessel may be selected.
In the embodiment, when the recognition result is repeated, the display is performed with the display color corresponding to the certainty factor, but the higher certainty factor may be preferentially displayed. For example, when the structure included in the surgical field image is recognized as loose connective tissue with a degree of certainty of 95% and the same structure is recognized as nerve tissue with a degree of certainty of 60%, the control unit 201 may recognize the structure as loose connective tissue, color the structure with a blue color, and present the structure to the operator.
Embodiment 9
In embodiment 9, a user interface provided in the surgical assist device 200 will be described.
Fig. 24 is a schematic diagram showing a configuration example of a user interface provided in the surgical assist device 200. Fig. 24 shows an example in which the display area 131 for displaying the identification image of loose connective tissue and a user interface for controlling the display manner of the identification image are juxtaposed. The user interface shown in fig. 24 includes a model selecting unit 132, a display method selecting unit 133, a threshold setting unit 134, a transparency setting unit 135, an averaging instructing unit 136, and a display color selecting unit 137. Various buttons and sliders provided in these user interfaces are operated by the operation unit 203 provided in the surgical assist device 200.
The model selecting unit 132 includes a selection button for selecting a neural network for constructing the learning model 300. The example of FIG. 24 shows a state in which "U-Net" is selected. The model selecting unit 132 may further include a selection button for accepting selection of either one of the first learning model 410 and the second learning model 420 described in embodiment 6. The model selection unit 132 may highlight the recommended model according to the degree of progress of the lesion, the operator in charge of the laparoscopic surgery, the state of loose connective tissue, the operation area, and the like.
The display method selecting unit 133 includes a selection button for receiving a selection of either one of the overlapping display and the soft map display. As described in embodiment 2, the superimposed display is a display method in which loose connective tissue portions are uniformly displayed in the same color, and the soft map display is a display method in which transparency is changed according to certainty. The example of fig. 24 shows a state in which the "overlap" display is selected.
The threshold setting unit 134 includes a slider that sets a threshold for determining whether or not the pixel of interest is loose connective tissue. The slider is configured such that the threshold value becomes smaller when sliding to the left (loose connective tissue is easily recognized), and the threshold value becomes larger when sliding to the right (loose connective tissue is difficult to recognize).
The transparency setting unit 135 includes a slider for changing the transparency of loose connective tissue. The slider is configured to have a lower transparency when slid to the left and a higher transparency when slid to the right.
The averaging instruction unit 136 includes an instruction button for turning on or off the averaging of the display color. When the averaging of the display colors is turned on, the control unit 201 averages the display color of the operation field image of the background and the display color set for the loose connective tissue, and displays the averaged color as the display color of the loose connective tissue portion. For example, when the display color set for the loose connective tissue portion is (0, B1) and the display color of the loose connective tissue portion in the operation field image of the background is (R2, G2, B2), the control unit 201 may color and display the loose connective tissue portion with the color of (R2/2, G2/2, (b1+b2)/2). Alternatively, the weighting coefficients W1 and W2 may also be introduced, and the identified loose connective tissue portions may be colored and displayed with a color of (w2×r2, w2×g2, w1×b1+w2×b2).
The display color selecting unit 137 includes a slider and a palette for changing the display color of the loose connective tissue portion. The display color selecting unit 137 may set the color designated by the slider as the display color of the loose connective tissue portion, or may set the color selected by the palette as the display color of the loose connective tissue portion. The display color selecting unit 137 may further include a default button for restoring the display color changed by the user to a default display color (for example, a color of a cool color system).
When receiving the instruction to change the display mode, the control unit 201 of the surgical assist device 200 may change the display mode of the identification image of the loose connective tissue portion displayed on the display area 131 according to the instruction to change, by the model selecting unit 132, the display method selecting unit 133, the threshold setting unit 134, the transparency setting unit 135, the averaging instructing unit 136, and the display color selecting unit 137.
In the example of fig. 24, the model selecting unit 132, the display method selecting unit 133, the threshold setting unit 134, the transparency setting unit 135, the averaging instructing unit 136, and the display color selecting unit 137 are provided, but the user interface for controlling the display mode of the identification image is not limited to these. For example, the user interface may include a selection unit that accepts whether or not estimation by the learning model 300 (or the learning models 410, 420, 500) is possible. The user interface may further include a setting unit for setting the estimated start time.
The control unit 201 may also receive a change in the display mode through the user interface shown in fig. 24, and when the display mode is changed from the default setting, notify the operator that the display mode is changed at an appropriate timing. For example, when the surgical assist device 200 is started or the operation is started, the control unit 201 may compare the default setting value of the display mode with the current setting value of the display mode, and if there is a difference between the two setting values, display the difference on the display device 130 or notify the portable terminal carried by the operator.
It should be understood that the embodiments of the present disclosure are illustrative in all respects, and not restrictive. The scope of the invention is indicated by the claims rather than by the foregoing meanings, and is intended to include all changes which come within the meaning and range of equivalency of the claims.
Reference numerals illustrate:
10: puncture outfit
11: laparoscope
12: energy treatment device
13: pliers with pliers body
110: camera Control Unit (CCU)
120: light source device
130: display device
140: video recording device
200: surgical auxiliary device
201: control unit
202: storage unit
203: operation part
204: input unit
205: output unit
206: communication unit
300: learning model
PG1: recognition processing program
PG2: display processing program
PG3: learning processing program
Claims (21)
1. A computer program for causing a computer to execute:
acquiring an operation field image obtained by shooting an operation field of an operation under a microscope;
the acquired operation field image is input into a learning model for recognizing a loose connective tissue portion contained in the operation field image, the learning model learning to output information on the loose connective tissue when the operation field image is input.
2. The computer program according to claim 1, for causing the computer to execute the following process:
displaying the loose connective tissue portions identified using the learning model on the surgical field image in a discriminable manner.
3. A computer program according to claim 2, for causing the computer to execute the following process:
the loose connective tissue portions are colored and displayed with the color of the cold color system.
4. A computer program according to claim 2, for causing the computer to execute the following process:
Averaging the display color set for the loose connective tissue portion and the display color of the loose connective tissue portion in the surgical field image;
the identified loose connective tissue portions are colored with the averaged color and displayed.
5. A computer program according to any one of claims 2 to 4, for causing the computer to perform the following process:
and changing the display mode of the loose connective tissue part according to the certainty factor of the identification result of the loose connective tissue part.
6. A computer program according to any one of claims 1 to 5,
the loose connective tissue is fibrous tissue that binds between the site that should be removed by the endoscopic procedure and the site that should be retained by the endoscopic procedure.
7. A computer program according to any one of claims 1 to 6,
the loose connective tissue is composed of a plurality of fibrous tissues,
the computer program is for causing the computer to execute: based on the information output from the learning model, a part or all of the plurality of fibrous tissues is identified as an aggregate.
8. A computer program according to any one of claims 1 to 7, for causing the computer to perform the following process:
segmenting the identified loose connective tissue portion into a plurality of ranges;
loose connective tissue portions within any of the plurality of ranges after segmentation are selectively displayed.
9. The computer program according to any one of claims 1 to 8, characterized in that,
the learning model includes a first learning model for identifying loose connective tissue portions present in a range near a site that should be removed by the endoscopic surgery and a second learning model for identifying loose connective tissue portions present in a range near a site that should be preserved by the endoscopic surgery,
the computer program is for causing the computer to execute:
either one of the first learning model and the second learning model is selected according to the degree of progress of the lesion.
10. The computer program according to any one of claims 1 to 9, characterized in that,
the learning model includes a first learning model for identifying loose connective tissue portions present in a range near a site that should be removed by the endoscopic surgery and a second learning model for identifying loose connective tissue portions present in a range near a site that should be preserved by the endoscopic surgery,
The computer program is for causing the computer to execute:
either one of the first learning model and the second learning model is selected according to an operator.
11. The computer program according to any one of claims 1 to 10, characterized in that,
the learning model includes a plurality of learning models for identifying portions of loose connective tissue,
the computer program is for causing the computer to execute:
one learning model is selected from the plurality of learning models according to the attribute of the patient, the presence or absence of the adhesion tissue covering the loose connective tissue, the surgical area including the loose connective tissue, or the kind of photographing device photographing the loose connective tissue.
12. The computer program according to any one of claims 1 to 11, characterized in that,
the learning model includes a plurality of learning models for identifying portions of loose connective tissue,
the computer program is for causing the computer to execute:
evaluating each learning model based on information output from each learning model when the surgical field image is input,
the loose connective tissue portions contained in the surgical field image are identified using a learning model selected according to the evaluation result.
13. The computer program according to any one of claims 1 to 12, characterized in that,
the learning model learns in such a manner that the loose connective tissue portion is identified at a stage in which loose connective tissue having elasticity is transferred from a state before stress to a state after stress.
14. The computer program according to any one of claims 1 to 13, characterized in that,
the learning model learns in such a manner that information on at least one of two sites bound by the loose connective tissue is outputted together with information on the loose connective tissue in the case where an operation field image is inputted,
the computer program is for causing the computer to execute:
the identified sites are displayed in a different manner than the loose connective tissue portions.
15. A computer program according to any one of claims 1 to 14, for causing the computer to perform the following process:
judging whether a specific part in the surgical field is in a static state or not;
and switching the display and non-display of the loose connective tissue part according to the judging result.
16. A computer program according to any one of claims 1 to 15, for causing the computer to perform the following process:
Judging whether the surgical tool contained in the surgical field is in a static state or not;
and switching the display and non-display of the loose connective tissue part according to the judging result.
17. A computer program according to any one of claims 1 to 16, for causing the computer to perform the following process:
inputting the acquired operation field image into a learning model for identifying a structural body part contained in the operation field image, wherein the learning model is learned to output information related to a structural body different from loose connective tissue when the operation field image is input,
and displaying the recognition result in a display mode corresponding to the recognition certainty of the loose connective tissue part and the recognition certainty of the structural body part.
18. A method for generating a learning model is characterized in that,
the computer is used to make a computer-readable medium,
obtaining training data comprising an image of an operative field obtained by photographing the operative field of the endoscopic surgery and forward solution data representing loose connective tissue portions within the operative field image,
according to the acquired training data set, a learning model is generated which outputs information related to loose connective tissue when an operation field image is input.
19. The method for generating a learning model as claimed in claim 18, wherein,
the forward solution data is data obtained by labeling a portion having elasticity, a portion having a space on the inner side, or a fibrous portion kept in a stressed state in loose connective tissue appearing in the surgical field image with a forward solution label.
20. An operation support device is characterized by comprising:
an acquisition unit that acquires an operation field image obtained by capturing an operation field of an endoscopic operation;
a recognition unit that recognizes a loose connective tissue portion included in the surgical field image acquired by the acquisition unit, using a learning model that learns to output information on the loose connective tissue when the surgical field image is input; and
and an output unit that outputs auxiliary information related to the endoscopic surgery based on the identification result of the identification unit.
21. An information processing method, characterized in that,
the computer is used to make a computer-readable medium,
acquiring an operation field image obtained by shooting an operation field of an operation under a lens,
identifying a loose connective tissue portion contained in the acquired surgical field image using a learning model that learns to output information related to loose connective tissue in the case that the surgical field image is input,
And outputting auxiliary information related to the microscopic operation according to the identification result.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2020-129614 | 2020-07-30 | ||
| JP2020129614 | 2020-07-30 | ||
| PCT/JP2021/028009 WO2022025151A1 (en) | 2020-07-30 | 2021-07-29 | Computer program, method for generating learning model, surgery assistance device, and information processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116097287A true CN116097287A (en) | 2023-05-09 |
Family
ID=80035756
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202180058418.6A Pending CN116097287A (en) | 2020-07-30 | 2021-07-29 | Computer program, learning model generation method, operation support device, and information processing method |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US20230281968A1 (en) |
| JP (2) | JP7031925B1 (en) |
| CN (1) | CN116097287A (en) |
| WO (1) | WO2022025151A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPWO2023199923A1 (en) * | 2022-04-11 | 2023-10-19 | ||
| WO2024053697A1 (en) * | 2022-09-09 | 2024-03-14 | 慶應義塾 | Surgery assistance program, surgery assistance device, and surgery assistance method |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3743247B2 (en) * | 2000-02-22 | 2006-02-08 | 富士電機システムズ株式会社 | Prediction device using neural network |
| JP2006325638A (en) * | 2005-05-23 | 2006-12-07 | Konica Minolta Medical & Graphic Inc | Method of detecting abnormal shadow candidate and medical image processing system |
| JP5797124B2 (en) * | 2012-01-31 | 2015-10-21 | 富士フイルム株式会社 | Surgery support device, surgery support method, and surgery support program |
| JP6029960B2 (en) * | 2012-12-10 | 2016-11-24 | 株式会社日立製作所 | Medical image display apparatus and medical image diagnostic apparatus equipped with the same |
| JP6460095B2 (en) * | 2014-03-28 | 2019-01-30 | 日本電気株式会社 | Learning model selection system, learning model selection method and program |
| JP2018067266A (en) * | 2016-10-21 | 2018-04-26 | 富士レビオ株式会社 | Program for forecasting onset risk or recurrence risk of disease |
| EP3566212A4 (en) * | 2017-01-06 | 2020-08-19 | Intuitive Surgical Operations Inc. | System and method for registration and coordinated manipulation of augmented reality image components |
| US11010610B2 (en) * | 2017-06-13 | 2021-05-18 | Google Llc | Augmented reality microscope for pathology |
| US20180360342A1 (en) * | 2017-06-16 | 2018-12-20 | Biosense Webster (Israel) Ltd. | Renal ablation and visualization system and method with composite anatomical display image |
| US20190069957A1 (en) * | 2017-09-06 | 2019-03-07 | Verily Life Sciences Llc | Surgical recognition system |
| WO2019054045A1 (en) * | 2017-09-15 | 2019-03-21 | 富士フイルム株式会社 | Medical image processing device, medical image processing method, and medical image processing program |
| CN107563123A (en) * | 2017-09-27 | 2018-01-09 | 百度在线网络技术（北京）有限公司 | Method and apparatus for marking medical image |
| JP7190842B2 (en) * | 2017-11-02 | 2022-12-16 | キヤノン株式会社 | Information processing device, control method and program for information processing device |
| EP3528263A1 (en) * | 2018-02-15 | 2019-08-21 | Siemens Healthcare GmbH | Providing a trained virtual tissue atlas and a synthetic image |
| JP2019162339A (en) * | 2018-03-20 | 2019-09-26 | ソニー株式会社 | Surgery supporting system and display method |
| CN108814717B (en) * | 2018-06-29 | 2020-10-27 | 微创（上海）医疗机器人有限公司 | Surgical robot system |
| WO2020110278A1 (en) * | 2018-11-30 | 2020-06-04 | オリンパス株式会社 | Information processing system, endoscope system, trained model, information storage medium, and information processing method |
| WO2021152784A1 (en) * | 2020-01-30 | 2021-08-05 | 株式会社インキュビット | Surgery assistance system |
-
2021
- 2021-07-29 CN CN202180058418.6A patent/CN116097287A/en active Pending
- 2021-07-29 WO PCT/JP2021/028009 patent/WO2022025151A1/en active Application Filing
- 2021-07-29 US US18/018,219 patent/US20230281968A1/en active Pending
- 2021-07-29 JP JP2021544875A patent/JP7031925B1/en active Active
-
2022
- 2022-02-16 JP JP2022022311A patent/JP7194889B2/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| JP7194889B2 (en) | 2022-12-23 |
| JP7031925B1 (en) | 2022-03-08 |
| JP2022069464A (en) | 2022-05-11 |
| US20230281968A1 (en) | 2023-09-07 |
| WO2022025151A1 (en) | 2022-02-03 |
| JPWO2022025151A1 (en) | 2022-02-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3463032B1 (en) | Image-based fusion of endoscopic image and ultrasound images | |
| JP2021100555A (en) | Medical image processing device, endoscope system, diagnosis support method and program | |
| JP7166430B2 (en) | Medical image processing device, processor device, endoscope system, operating method and program for medical image processing device | |
| JP7194889B2 (en) | Computer program, learning model generation method, surgery support device, and information processing method | |
| JP7289373B2 (en) | Medical image processing device, endoscope system, diagnosis support method and program | |
| JP7153974B2 (en) | COMPUTER PROGRAM, LEARNING MODEL GENERATION METHOD, AND IMAGE PROCESSING DEVICE | |
| JP2024041891A (en) | Computer program, learning model generation method, and support device | |
| JP7387859B2 (en) | Medical image processing device, processor device, endoscope system, operating method and program for medical image processing device | |
| JP7146318B1 (en) | Computer program, learning model generation method, and surgery support device | |
| WO2019088008A1 (en) | Image processing apparatus, image processing method, program, and endoscope system | |
| EP4026479A1 (en) | Medical image processing device, endoscope system, medical image processing method, and program | |
| JP7311936B1 (en) | COMPUTER PROGRAM, LEARNING MODEL GENERATION METHOD, AND INFORMATION PROCESSING DEVICE | |
| JP4615842B2 (en) | Endoscope system and endoscope image processing apparatus | |
| WO2022250031A1 (en) | Information processing device, information processing method, and computer program | |
| CN117956939A (en) | Computer program, learning model generation method, and information processing device | |
| US20220414885A1 (en) | Endoscope system, medical image processing device, and operation method therefor | |
| JP7427766B2 (en) | Image selection support device, image selection support method, and image selection support program | |
| JP7470779B2 (en) | ENDOSCOPE SYSTEM, CONTROL METHOD, AND CONTROL PROGRAM | |
| EP4111938A1 (en) | Endoscope system, medical image processing device, and operation method therefor | |
| JP2023007331A (en) | Endoscope system, medical image processing device, and operation method therefor | |
| CN115023171A (en) | Medical image data generation device for study, medical image data generation method for study, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |