WO2021176665A1

WO2021176665A1 - Surgery support system, surgery support method, and program

Info

Publication number: WO2021176665A1
Application number: PCT/JP2020/009495
Authority: WO
Inventors: 耀子鎌戸; 恵令奈下地; 翔平辺見; 久保　允則
Original assignee: オリンパス株式会社
Priority date: 2020-03-05
Filing date: 2020-03-05
Publication date: 2021-09-10

Abstract

The purpose of the present invention is to provide a technology for presenting an appropriate surgical plan to a doctor during an endoscopic examination.　The surgery support system comprises: a model creation unit (40b) that creates an organ model of a luminal organ of a patient on the basis of an image captured by an imaging unit of an endoscope inside the luminal organ of the patient and spatial layout information on the imaging unit; and a surgical plan creating unit (40e) that creates, on the basis of at least surgical constraints, a surgical plan including information specifying the location of a surgery target polyp in the organ model among polyps in the luminal organ specified from the image, and updates the surgical plan in accordance with the updating of constraints during surgery.

Description

Surgery support system, surgery support method, and program

The disclosure of this specification relates to a surgery support system, a surgery support method, and a program.

Endoscopy is an examination method that is inserted into luminal organs such as the esophagus, stomach, large intestine, trachea, and bronchi, and the inside of the lumen is observed using images obtained by endoscopy. Gender has come to be widely recognized. For example, by performing colon magnifying endoscopy using the technique described in Patent Document 1, the surface structure of the colon polyp indicates whether the colon polyp is a tumor or a non-tumor, and if it is a tumor, It is possible to classify whether it is a benign tumor or a malignant tumor. As described above, according to the endoscopy, the inside of the lumen can be directly observed, so that the lesion can be detected and treated at an early stage.

Japanese Unexamined Patent Publication No. 2016-08737

By the way, when performing surgery such as polyp removal in endoscopy, there are various restrictions such as restrictions from a medical point of view such as the amount of bleeding and restrictions from an economic point of view such as examination costs. For this reason, not all polyps found by endoscopy can be treated, and it is necessary to selectively treat high-priority polyps within the given constraints.

However, it is not easy to properly determine the polyp to be treated during the advanced medical work of endoscopy. In particular, since the polyp to be treated may change from moment to moment depending on the situation during endoscopy, a technique for notifying a doctor in real time of which polyp should be treated and assisting the doctor's decision is desired.

From the above circumstances, an object of one aspect of the present invention is to provide a technique for presenting an appropriate surgical plan to a doctor during endoscopy.

The surgical support system according to one aspect of the present invention is based on the image captured by the imaging unit of the endoscope in the lumen organ of the patient and the spatial arrangement information of the tip of the insertion portion of the endoscope. The model-creating unit that creates an organ model of the luminal organ, and the polyp that is the target of the surgery among the polyps in the luminal organ identified from the image, at least based on the constraints of surgery. It is provided with an operation plan creation unit that creates an operation plan including target polyp information for specifying a position on an organ model and updates the operation plan in accordance with the update of the constraint condition during the operation.

The surgical support method according to one aspect of the present invention is based on an image captured by an imaging unit of an endoscope in a patient's luminal organ and spatial arrangement information at the tip of the insertion portion of the endoscope. , An organ model of the luminal organ of the patient, and the organ of the polyp to be operated on among the polyps in the luminal organ identified from the image, at least based on surgical constraints. A surgical plan including target polyp information for specifying a position on a model is created, and the surgical plan is updated as the constraint conditions are updated during the surgery.

The program according to one aspect of the present invention is based on the image captured by the imaging unit of the endoscope in the tract organ of the patient and the spatial arrangement information of the tip of the insertion portion of the endoscope. An organ model of the luminal organ of the patient is created, and on the organ model of the polyp to be operated on among the polyps in the luminal organ identified from the image, at least based on the surgical constraints. An operation plan including the target polyp information for specifying the position in the operation plan is created, and a process of updating the operation plan is executed in accordance with the update of the constraint condition during the operation.

According to the above aspect, an appropriate surgical plan can be presented to the doctor during the endoscopy.

It is a figure which illustrated the structure of the operation support system which concerns on one Embodiment. It is a block diagram which illustrated the structure of the operation support system which concerns on one Embodiment. It is a block diagram which illustrates the functional structure of the processor which concerns on one Embodiment. It is a figure for demonstrating the trained model. It is a flowchart of the process performed by the operation support system which concerns on one Embodiment. It is a figure which showed the display example of the model display area. It is a figure which showed the display example of the operation plan selection screen. It is a figure which showed another display example of the operation plan selection screen. It is a figure which showed how the operation plan displayed in the model display area is updated.

FIG. 1 is a diagram illustrating the configuration of the surgery support system according to one embodiment. FIG. 2 is a block diagram illustrating the configuration of the surgery support system according to the embodiment. The surgery support system 1 shown in FIGS. 1 and 2 is a system that creates a surgery plan in consideration of the constraints of surgery and presents it to a doctor during endoscopy of the luminal organ of patient Pa. Hereinafter, the configuration of the surgery support system 1 will be described with reference to FIGS. 1 and 2.

As shown in FIG. 1, the surgery support system 1 includes an endoscope 10, an image processing device 20, a light source device 30, a surgery support device 40, a display device 50, a magnetic field generator 60, and an input device 70. It has. The endoscope 10 is not particularly limited, but is, for example, a flexible endoscope for the large intestine. Hereinafter, the case where colonoscopy is performed using a flexible endoscope will be described as an example, but the scope of application of the surgical support system 1 is not limited to colonoscopy. It can also be applied to endoscopy of other luminal organs such as the esophagus and stomach. Further, since it is only necessary to obtain the endoscopic image acquired while moving in the lumen and its coordinate information, the endoscopy may be performed not only by a flexible endoscope but also by a rigid endoscope. good.

The endoscope 10 is provided at an operation unit operated by a doctor, a flexible insertion portion inserted into a lumen, a universal cord extending from the operation portion, and an end portion of the universal cord. It includes a connector portion that is detachably connected to the processing device 20 and the light source device 30. The doctor can bend the insertion portion in an arbitrary direction by operating the operation portion, whereby the inside of the luminal organ can be freely observed by the image captured by the image sensor 11. The image sensor 11 is, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary MOS) image sensor, or the like, and is an example of an image pickup unit of the surgery support system 1. The image sensor 11 may be provided in the endoscope 10. The image sensor 11 may be provided, for example, at the tip of the insertion portion, or may be provided on the base end side of the insertion portion, that is, near the operation portion.

A magnetic sensor 12 is further provided at the insertion portion of the endoscope 10, and the signal line from the magnetic sensor 12 is connected to the surgery support device 40. The magnetic sensor 12 is arranged near the tip of the insertion portion, and detects the position and orientation of the tip of the insertion portion, that is, the spatial arrangement of the tip of the insertion portion by detecting the magnetic field generated by the magnetic field generator 60. .. The magnetic sensor 12 is, for example, a 6-axis sensor composed of two cylindrical coils whose central axes are orthogonal to each other, detects the position coordinates of the tip of the insertion portion and the Euler angles, and outputs them to the surgical support device 40.

Although FIGS. 1 and 2 show an example in which the magnetic field generator 60 generates a predetermined magnetic field and the magnetic sensor 12 provided in the endoscope 10 detects the magnetic field generated by the magnetic field generator 60. A magnetic field generator is provided in the endoscope 10, and the magnetic field generated by the endoscope 10 is detected by a magnetic sensor placed at a predetermined position to detect the spatial arrangement of the tip of the insertion portion of the endoscope 10. You may.

The image processing device 20 is a video processor that processes an image captured by the endoscope 10. The image processing device 20 converts, for example, a signal from the endoscope 10 into a video signal and outputs the signal to the display device 50. As a result, the display device 50 displays the live image based on the video signal from the image processing device 20. Further, the image processing device 20 may control the light source device 30 based on, for example, a video signal, or may perform processing related to automatic dimming control. Further, the image processing device 20 is also connected to the surgery support device 40, and outputs the image processed by the image processing device 20 to the surgery support device 40.

The light source device 30 is a device that supplies illumination light to the endoscope 10 via a light guide. The illumination light supplied by the light source device 30 is not particularly limited, but may be, for example, white light used for normal light observation of an endoscope, NBI (Narrow Band Imaging) observation, AFI (Auto-Fluorescence Imaging). Special light used for special light observation such as observation may be used. Further, the light source device 30 may supply the endoscope 10 by arbitrarily switching between white light and special light at the option of a doctor, for example.

The surgery support device 40 is a device that creates an organ model and a surgery plan by processing the information obtained during the endoscopy in real time. The surgery support device 40 may display the created organ model and the surgery plan on the display device 50 during the surgery. The surgical plan includes information for identifying the position of the polyp to be operated on the organ model among the polyps in the luminal organ identified from the endoscopic image.

The surgical support device 40 may be a general-purpose computer such as a personal computer, a tablet, or other mobile device, or may be a computer for a specific purpose, a workstation, or a mainframe computer. Further, the surgery support device 40 may be a single device, a set of a plurality of devices, or may be configured as a distributed computing system. The surgical support device 40 is configured to execute a variety of software programs, including software programs that execute all or part of the processes and algorithms disclosed herein. As shown in FIG. 2, the surgical support device 40 as an example includes a processor 41 configured to process non-image information such as images and spatial arrangement information received as inputs for various algorithms and software programs. ..

The processor 41 may include hardware, which may include, for example, at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processor 41 can include, for example, one or more circuit devices (eg, ICs) or one or more circuit elements (eg, resistors, capacitors) on a circuit board. The processor 41 may be a CPU (central processing unit). Further, various types of processors including GPU (Graphics processing unit) and DSP (Digital Signal Processor) may be used for the processor 41. The processor 41 may be a hardware circuit having an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). The processor 41 can include an amplifier circuit, a filter circuit, and the like for processing an analog signal.

The surgery support device 40 as an example includes a storage device 42 as shown in FIG. The software program and / or computer-executable instruction executed by the processor 41 is stored in a computer-readable storage medium such as the storage device 42. The "computer-readable storage medium" used in the present specification refers to a non-temporary computer-readable storage medium.

The surgery support device 40 can include one or more storage devices 42. The storage device 42 may include a memory and / or other storage device. The memory may be, for example, a computer's random access memory (RAM). The memory may be a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The storage device 42 may include, for example, a register, a magnetic storage device such as a hard disk device, an optical storage device such as an optical disk device, an internal or external hard disk drive, a server, a solid state storage device, a CD-ROM, a DVD, or other optical device. Alternatively, it may be a magnetic disk storage device or another storage device. The storage device 42 is an example of a storage unit of the surgery support system 1.

The computer-executable instructions include, for example, instructions and data that cause the surgical support device 40 to realize a certain function or group of functions, and the processor 41 is determined by executing these computer-executable instructions by the processor 41. Demonstrate function. The computer-executable instruction may be a set of instructions constituting a software program, or may be an instruction for the hardware circuit of the processor 41 to directly process.

As shown in FIG. 2, the surgery support device 40 as an example further includes a display interface 43, an image capture device 44, a spatial arrangement detection device 45, a drive circuit 46, and an input interface 47.

The display interface 43 displays the organ model and the surgical plan generated by the processor 41 on the display device 50 based on the image captured from the image acquisition device 44 and the spatial arrangement information generated by the spatial arrangement detection device 45. Output. As a result, the display device 50 displays the operation plan on the organ model together with the live image.

The image capture device 44 is a device that captures an endoscope image imaged by the endoscope 10 and subjected to predetermined processing by the image processing device 20 at regular intervals. The image capturing device 44 may acquire, for example, 30 endoscopic images per second from the image processing device 20 in the same manner as the frame rate. Further, unlike the frame rate, endoscopic images may be acquired at a cycle longer than the frame rate, for example, 3 images per second.

The spatial arrangement detection device 45 controls the drive circuit 46 that drives the magnetic field generator 60 to generate a predetermined magnetic field in the magnetic field generator 60. The spatial arrangement detection device 45 detects the magnetic field by the magnetic sensor 12, and from the detection signal of the detected magnetic field, the position coordinates (x, y, z) and the direction (Euler angles (ψ, θ, φ)) data, that is, spatial arrangement information, is generated in real time.

The input interface 47 is connected to an input device 70 such as a keyboard, mouse, touch panel, voice input device, and foot pedal. The operation signal corresponding to the operation on the input device 70 is input to the processor 41 via the input interface 47.

FIG. 3 is a block diagram illustrating the functional configuration of the processor according to the embodiment. FIG. 4 is a diagram for explaining the trained model. In the operation support device 40 described above, the processor 41 executes the software program, so that the processor 41 realizes the function shown in FIG. Hereinafter, the functions of the processor 41 will be described with reference to FIGS. 3 and 4.

As shown in FIG. 3, the processor 41 includes an acquisition unit 40a, a model creation unit 40b, a removal priority calculation unit 40c, a constraint condition management unit 40d, an operation plan creation unit 40e, and a display control unit 40f. ing.

The acquisition unit 40a acquires the endoscopic image captured by the image sensor 11 in the patient's luminal organ and the spatial arrangement information at the tip of the insertion unit. Specifically, the acquisition unit 40a acquires an endoscopic image from the image processing device 20 via the image acquisition device 44, and acquires spatial arrangement information from the spatial arrangement detection device 45.

The model creation unit 40b creates an organ model of the luminal organ of the patient Pa based on the endoscopic image and the spatial arrangement information acquired by the acquisition unit 40a. The organ model is not particularly limited, but may show, for example, the three-dimensional structure of a luminal organ. In this example, the vascular organ is the large intestine and the organ model is the large intestine model. Specifically, the model creation unit 40b uses Visual SLAM (Simultaneus Localization and Mapping) from, for example, an endoscopic image acquired at a fixed cycle and spatial arrangement information at the time of acquiring each endoscopic image. An organ model may be created. Visual SLAM is a technology that simultaneously estimates the three-dimensional information of a plurality of feature points and the position and orientation of the camera from an image taken by the camera. In the surgery support system 1, since the position and posture of the camera correspond to the position and posture of the tip of the insertion portion, the position and posture of the camera can be processed as known by the spatial arrangement information. Therefore, the three-dimensional information can be calculated at high speed and with high accuracy.

The removal priority calculation unit 40c calculates the malignancy and progression rate of the polyp in the luminal organ detected from the endoscopic image, and the removal priority of the polyp is based on the calculated malignancy and progression rate of the polyp. Calculate the degree. The rate of progression of a polyp is the rate at which the malignancy of the polyp worsens, for example, the rate at which the polyp becomes cancerous.

Specifically, the removal priority calculation unit 40c detects a polyp by analyzing an endoscopic image, and calculates the malignancy and progression rate of the detected polyp. More specifically, the removal priority calculation unit 40c has learned, for example, the malignancy and progression rate of the polyp with respect to the surface shape of the polyp identified from the polyp image, as shown in FIG. May be used to estimate the location of the polyp in the endoscopic image, the malignancy of the polyp, and the rate of progression of the polyp. The trained model PTM may be created in advance for each patient's race, age, gender, lifestyle (whether smoking, drinking, etc.), medical history, etc., and may be stored in the storage device 42, and may be removed. The priority calculation unit 40c may appropriately select a trained model to be used based on the patient information of the patient Pa. The patient information includes race, age, gender, lifestyle (whether smoking, drinking, etc.), medical history, and the like.

Further, the removal priority calculation unit 40c uses a trained model in which the malignancy of the polyp with respect to the surface shape of the polyp is learned instead of the trained model PTM shown in FIG. The location and grade of the polyp may be estimated. In that case, the removal priority calculation unit 40c further determines the polyp in the luminal organ based on the endoscopic image and the past examination result of the luminal organ of the patient Pa stored in the storage device 42. The speed of travel may be estimated. The rate of polyp progression may be estimated, for example, by comparing the size of the polyp detected from the endoscopic image with the size of the corresponding polyp in past test results. It may also be estimated by comparing the malignancy in addition to or instead of the size. The trained model that learned the malignancy is also created in advance for each patient's race, age, gender, lifestyle (whether smoking, drinking, etc.), medical history, etc., and is stored in the storage device 42. Alternatively, the removal priority calculation unit 40c may appropriately select a trained model to be used based on the patient information of the patient Pa.

The constraint condition management unit 40d manages the constraint conditions of the surgery performed during the endoscopy. Constraint conditions The constraints managed by the management unit 40d include, for example, allowable operation cost, allowable operation time, allowable bleeding amount, difficulty of operation, skill level of surgeon (doctor), and patient information. At least one of is included. Further, since these constraint conditions include those that fluctuate during the operation, the constraint condition management unit 40d manages the constraint conditions while updating them as needed during the endoscopy.

For example, the permissible surgical cost varies depending on the number and size of polyps removed, the number of disposable treatment tools used, and so on. Therefore, the constraint condition management unit 40d subtracts the cost determined according to the progress of the surgery from the permissible surgery cost. In addition, the amount of bleeding at the time of polyp removal is predictable to some extent, and a surgical plan described later is created based on the expected amount of bleeding, but the amount of bleeding that occurs in actual surgery may differ from the expected amount. Therefore, the constraint condition management unit 40d subtracts the actual bleeding amount from the allowable bleeding amount according to the progress of the operation.

The surgery plan creation unit 40e creates a surgery plan at least based on the constraint conditions of the surgery managed by the constraint condition management unit 40d. Specifically, the operation plan creation unit 40e creates an operation plan based on the constraint conditions managed by the constraint condition management unit 40d and the removal priority calculated by the removal priority calculation unit 40c. The surgery plan created by the surgery plan creation unit 40e includes information for identifying the position of the polyp to be operated on the organ model among the polyps in the luminal organ identified from the endoscopic image. It has been done.

As described above, the constraint conditions of the operation managed by the constraint condition management unit 40d change with the progress of the operation. Therefore, the operation plan creation unit 40e prepares the operation plan according to the update of the constraint conditions during the operation. Update. Specifically, the surgery plan creation unit 40e may create a new surgery plan each time the constraint condition is updated. The operation plan creation unit 40e only needs to be able to create an operation plan that reflects the latest constraint conditions, and therefore the constraint conditions and the operation plan may be updated asynchronously.

The display control unit 40f causes the display unit 80 to display at least the organ model created by the model creation unit 40b and the surgery plan created by the surgery plan creation unit 40e. The display unit 80 is, for example, a display device 50. Specifically, the display control unit 40f indicates the position of the polyp to be operated on by displaying a predetermined mark on the organ model. In order to discriminately display the malignancy and the rate of progression of the polyp, the display control unit 40f may display a predetermined mark in a size corresponding to one of the malignancy and the rate of progression of the polyp, and display the predetermined mark. It may be displayed in a color corresponding to either the malignancy of the polyp or the rate of progression. Further, when there are a plurality of polyps to be operated on, a predetermined mark may be displayed with an order in which the operation should be performed.

In addition to the polyp information that specifies the position of the polyp that is the target of surgery (hereinafter referred to as the target polyp information), the display control unit 40f has the polyp information that identifies the position of the polyp that is not the target of surgery (hereinafter, the non-target polyp information). May be displayed on the display unit 80. Further, the operation planning unit 40e may display the current position information for specifying the current position of the tip of the insertion portion on the organ model on the display unit 80. The display control unit 40f displays these auxiliary information (organ model, polyp information, current position information, etc.) side by side next to, for example, a live image. The auxiliary information may be displayed by switching to the live image according to the operation of the doctor.

The display device 50, which is an example of the display unit 80, is, for example, a liquid crystal display, a plasma display, an organic EL display, a CRT display, an LED matrix panel, electronic paper, a projector, or the like, and is another type of display device. May be good. Further, the display device 50 may display an image three-dimensionally. The display method of the 3D image by the display device 50 is not particularly limited, and any display method can be adopted. For example, it may be a naked eye type, or a method of displaying a stereoscopic image in combination with eyeglasses worn by the operator.

According to the surgery support system 1 configured as described above, the polyp to be the target of the surgery is specified under the given constraints of the surgery, and the information for specifying the position of the polyp to be the target is included. Can make a surgical plan. Therefore, by presenting the created surgery plan to the doctor, the doctor can judge the surgery target with reference to the surgery plan, and make an appropriate decision within the limited time during the surgery. Is possible. In addition, the surgical plan is updated in substantially real time as the endoscopy constraints are updated. For this reason, doctors can perform surgery according to the patient's ever-changing conditions during endoscopy, and can provide patients with optimal treatment under given constraints. Become.

FIG. 5 is a flowchart of processing performed by the surgery support system according to the embodiment. FIG. 6 is a diagram showing a display example of the model display area. FIG. 7 is a diagram showing a display example of the operation plan selection screen. FIG. 8 is a diagram showing another display example of the operation plan selection screen. FIG. 9 is a diagram showing how the surgical plan displayed in the model display area is updated. Hereinafter, the surgical support method for supporting the surgery performed during the endoscopy performed by the surgical support system 1 will be specifically described with reference to FIGS. 5 to 9.

When the examination is started, the surgery support system 1 first accepts the input of the initial setting (step S1). Here, the surgery support device 40 sets various information necessary for subsequent processing. Specifically, the processor 41 of the surgery support device 40 extracts the information of the patient Pa from a predetermined database according to the input of the doctor and sets it as the basic information. The basic information includes, for example, the patient Pa's name, race, age, gender, lifestyle (whether smoking, drinking, etc.), medical history, and the like. It may also include information on colonoscopy and other examinations received in the past. Further, the processor 41 receives the input of the constraint condition of the operation and sets the constraint condition as the constraint condition at the start of the endoscopy. Surgical constraints include permissible surgical costs, permissible surgery time, permissible bleeding volume, surgeon skill level, and patient information. Further, the processor 41 sets the reference position and the reference posture by detecting the operation of the doctor with respect to the endoscope 10. For example, when the doctor performs a predetermined operation with the tip of the endoscope 10 aligned with the anus of the patient Pa laid on the bed 2, the processor 41 sets the detected position and posture as the reference position and the reference. Register as a posture. As a result, the spatial arrangement information generated by the surgical support device 40 thereafter is created as the spatial arrangement information on the three-dimensional Cartesian coordinate system defined by the reference position and the reference posture.

When the initial setting is completed, the doctor starts inserting the endoscope 10 into the large intestine, advances the endoscope 10 in the order of the anus, the rectum, the colon, and the cecum, and reaches the endoscope 10 to the innermost part of the large intestine. Let me. The period during which the endoscope 10 is inserted from the anus to the innermost part of the large intestine is referred to as an insertion period, and is distinguished from the subsequent withdrawal period. The inspection period consists of an insertion period and a withdrawal period. The withdrawal period is a period during which the endoscope 10 is withdrawn from the innermost part toward the anus, and the doctor mainly performs a polyp removal operation while observing the inside of the large intestine in detail during the withdrawal period.

The surgery support system 1 acquires an endoscopic image and spatial arrangement information (step S2), and creates an organ model based on the acquired information (step S3). Further, the surgical support system 1 calculates the malignancy and the rate of progression of the polyp detected from the endoscopic image (step S4). Then, during the insertion period (step S5YES), the surgical support system 1 displays auxiliary information including the organ model and polyp information together with the live image (step S6). The operation during the insertion period will be described in more detail as follows.

In step S2, the image processing device 20 performs predetermined image processing on the endoscope image captured by the endoscope 10 and outputs it to the surgery support device 40. After that, the processor 41 of the surgery support device 40 acquires the endoscopic image captured by the image capture device 44 from the image processing device 20 at, for example, 30 fps. Further, the processor 41 acquires the spatial arrangement information created by the spatial arrangement detecting device 45 based on the detection result by the magnetic sensor 12 in synchronization with the acquisition of the endoscopic image. In this way, the surgery support system 1 periodically acquires the endoscopic image and the spatial arrangement information corresponding to the endoscopic image.

In step S3, the processor 41 creates a large intestine model of patient Pa using the endoscopic image and spatial arrangement information acquired in step S2. In the surgery support device 40, the processor 41 extracts a plurality of feature points from continuous endoscopic images obtained at 30 fps, and calculates the coordinate information of the feature points by a method such as Visual SLAM. Then, by using the coordinate information of these feature points, a large intestine model showing the three-dimensional structure of the large intestine of patient Pa is created. The processor 41 uses the spatial arrangement information indicating the position and orientation of the tip of the insertion portion acquired by using the magnetic sensor 12, and is faster than the case where the coordinate information of the feature point is calculated only from the image information. Moreover, it is possible to calculate the coordinate information of the feature points with high accuracy.

In step S4, the processor 41 selects a trained model based on the basic information set in step S1, and inputs an endoscopic image to the selected trained model. Then, the processor 41 detects the position of the polyp when the polyp is present in the endoscopic image by using the trained model, and the endoscopic image according to the malignancy and the progress rate of the detected polyp. To classify. The classified endoscopic images are used as information for calculating the removal priority at the time of creating the surgical plan in step S8.

In step S6, the processor 41 causes the display device 50 to display auxiliary information including the organ model and polyp information. Specifically, as shown in FIG. 6, the processor 41 displays the organ model M1 created in step S3 on the model display area 52 next to the live image display area 51 for displaying the live image L1 on the display device 50. , And further, the display device 50 displays the polyp information detected in step S4 on the organ model M1. The polyp information may be any information that identifies the position of the polyp in the large intestine on the organ model M1. For example, as in the polyp information P shown in FIG. 6, a predetermined value displayed at the position of the polyp on the organ model M1. Mark (black circle in this example) may be used. The polyp information P may be displayed in an aspect (for example, size, color, etc.) according to the malignancy of the polyp, and may be displayed in an aspect (for example, size, color, etc.) according to the degree of progression of the polyp. Etc.) may be displayed. Further, the current position information C displayed on the organ model M1 is information for specifying the current position of the tip of the insertion portion on the organ model M1. The current position information C is created based on the latest spatial arrangement information.

During the insertion period, the processes of steps S2 to S6 are repeatedly executed, so that the auxiliary information displayed on the display device 50 together with the live image is updated at any time. FIG. 6 shows how the auxiliary information is updated during the insertion period. The organ model M1 is an organ model created when the endoscope 10 is inserted halfway into the descending colon, and three polyp information is displayed on the organ model M1. The organ model M2 is an organ model created when the endoscope 10 is inserted halfway through the transverse colon, and the same three polyp information as the organ model M1 is displayed on the organ model M2. The organ model M3 is an organ model created when the endoscope 10 is inserted into the cecum, and five polyp information is displayed on the organ model M3.

When the insertion period ends (step S5NO), the surgery support system 1 acquires the constraint condition (step S7) and creates an operation plan under the acquired constraint condition (step S8). The end of the insertion period may be determined based on the movement locus of the tip of the insertion portion calculated from the history of the spatial arrangement information. For example, when the processor 41 detects that the image sensor 11 is returning the route taken in the past, it may determine that the insertion period has ended.

In step S7, the processor 41 first acquires the constraint condition at the start of endoscopy set in step S1. Next, the processor 41 updates the constraints that have changed from the start of the endoscopy to the present. The processor 41 sets, for example, a time calculated by subtracting the elapsed time from the start of endoscopy to the present from the allowable operation time as a new allowable operation time. In addition, processor 41 adds constraints that have become apparent from the start of endoscopy to the present. The processor 41 identifies, for example, the difficulty of the removal operation of each polyp and the amount of bleeding caused by the removal from the position, size, malignancy, etc. of the polyp detected during the insertion period, and adds it as a constraint condition. The amount of bleeding may be estimated using a learned model in which the amount of bleeding with respect to the surface shape of the polyp has been learned.

In step S8, the processor 41 first calculates the removal priority of each polyp based on the malignancy and progression rate of the polyps calculated in step S4. In the calculation of the removal priority, for example, the relationship between the malignancy level of the polyp and the numerical value indicating the removal priority, which is stored in the storage device 42 in advance, and the numerical value indicating the progress rate level of the polyp and the removal priority are shown. The relationship with may be used. The processor 41 averages, for example, a numerical value indicating the removal priority derived from the malignancy of the polyp and a numerical value indicating the removal priority derived from the progress rate of the polyp, thereby giving an overall removal priority to each polyp. The degree may be calculated. Further, the processor 41 may calculate a plurality of removal priorities for each polyp by changing the weighting of the malignancy and the progression rate.

After that, the processor 41 creates an operation plan based on the calculated removal priority and the constraint condition acquired in step S7. When the processor 41 calculates a plurality of removal priorities for each polyp by changing the weighting, a plurality of surgical plans may be created. In that case, the processor 41 may present a plurality of operation plans to the doctor by displaying the operation plan selection screen 50a as shown in FIGS. 7 and 8 on the display device 50, and executes the operation plan to the doctor. You may choose the surgical plan to be used.

The surgical plan OP1 shown in FIG. 7 is a surgical plan in which the malignancy is relatively prioritized over the rate of progression, and is formed in the cecum, rectal sigmoid, and lower colon judged to be relatively high in malignancy. It is shown that a total of three polyps were presented as surgical targets. In addition, the surgical plan OP2 shown in FIG. 8 is a surgical plan in which the progression rate is relatively prioritized over the malignancy, and can be formed in the cecum, transverse colon, descending colon, and lower colon, which are judged to have a relatively high progression rate. It is shown that a total of four polyps were presented as surgical targets. The target polyp information T indicated by a black circle on the organ model M3 indicates the position of the polyp to be operated on, and the non-target polyp information U indicated by a white circle on the organ model M3 indicates the position of the polyp not to be operated on. There is. In addition, the number assigned to the side of the target polyp information T indicates the surgical order, and basically, the direction in which the endoscope moves in the large intestine, that is, the direction from the innermost part of the large intestine toward the anus. Numbered in order along.

The processor 41 may switch a plurality of surgery plans and display them on the display device 50 in response to the operation of the buttons (buttons 53 and 54) on the surgery plan selection screen 50a, for example, and the doctor presses the surgery start button 55. By pressing, the surgical plan to be performed in the endoscopy may be confirmed.

When the doctor selects the surgery plan on the surgery plan selection screen 50a (step S9YES) and the withdrawal period is started, the processor 41 closes the surgery plan selection screen 50a and displays the organ on the display device 50 and the model display area 52. The model M3 and the selected surgical plan are displayed (step S10). Hereinafter, the case where the operation plan OP1 is selected will be described as an example.

During the extraction period, the surgery support system 1 acquires the endoscopic image and the spatial arrangement information (step S11), updates the constraint conditions (step S12), and updates the surgery plan (step S13). The operation during the pull-out period will be described in more detail as follows.

In step S11, the image processing device 20 performs predetermined image processing on the endoscope image captured by the endoscope 10 and outputs it to the surgery support device 40. After that, the processor 41 acquires an endoscopic image captured by the image capturing device 44 from the image processing device 20 at, for example, 30 fps. Further, the processor 41 acquires the spatial arrangement information created based on the detection result by the magnetic sensor 12 in synchronization with the acquisition of the endoscopic image. In this way, the surgery support system 1 periodically acquires the endoscopic image and the spatial arrangement information corresponding to the endoscopic image. The process of step S11 is the same as the process of step S2 during the insertion period.

In step S12, the constraint condition acquired in step S7 is updated. Specifically, the processor 41 updates the permissible operation time based on the start of endoscopy, that is, the elapsed time from the start of the operation to the present, as in step S7. Processor 41 also updates the permissible surgical costs based on the price of polyp removal performed during the surgery. The price for removing the polyp may be manually input to the surgical support device 40 by a doctor or a nurse each time the polyp is removed, and the size of the removed polyp is detected from the endoscopic image. It may be automatically specified by. The processor 41 also updates the permissible amount of bleeding based on the amount of bleeding from the luminal organs estimated from the images captured during surgery.

In step S13, the processor 41 recreates the surgical plan based on the removal priority of each polyp and the constraints updated in step S12, and newly creates the surgical plan displayed in the model display area 52. Update to plan.

During the withdrawal period, the processes of steps S10 to S13 are repeatedly executed, so that the surgical plan displayed on the display device 50 together with the live image is updated at any time. FIG. 9 shows how the surgery plan is updated in the order of surgery plan OP1, surgery plan OP1a, surgery plan OP1b or surgery plan OP1c during the withdrawal period.

The surgical plan OP1 is a surgical plan immediately after the start of the withdrawal period, and includes three target polyp information T displayed on the organ model M3. The surgical plan OP1a is a surgical plan immediately after the polyp PO formed in the cecum is excised with the snare 15. In the operation plan OP1a, the target polyp information T corresponding to the polyp PO formed in the cecum is updated to the treated information F, and the remaining two target polyp information T are given the updated processing order. The surgical plan OP1b is a surgical plan after a while has passed after the removal of the polyp PO formed in the cecum, and is a surgical plan for a case in which bleeding occurs more than expected after the removal of the polyp PO. The surgical plan OP1c is a surgical plan after a while has passed after the removal of the polyp PO formed in the cecum, and is a surgical plan for the case where the amount of bleeding is less than expected after the removal of the polyp PO. The surgical plan OP1b shows that the polyps formed in the rectal sigmoid region were excluded from the surgical target as a result of the sharp decrease in the allowable amount of bleeding. In addition, the surgical plan OP1c shows that a polyp formed in the descending colon was added to the surgical target as a result of having a margin in the allowable amount of bleeding.

When the extraction period ends and the endoscope 10 is removed from the large intestine (step S14YES), the surgery support system 1 ends the process shown in FIG.

As described above, when the operation support system 1 performs the process shown in FIG. 5, the operation plan including the information identifying the polyp to be treated is presented to the doctor during the endoscopy, so that the doctor treats it. It is possible to perform surgery efficiently while appropriately determining the polyp to be used. In addition, since the surgery plan is updated to reflect the situation during endoscopy, it is possible to respond appropriately to sudden changes in the situation.

The above-described embodiments show specific examples for facilitating the understanding of the invention, and the embodiments of the present invention are not limited thereto. The surgical support system, surgical support method, and program can be variously modified and modified without departing from the description of the claims.

In the above-described embodiment, an example of creating an organ model and calculating the malignancy and progression rate of the polyp during the insertion period has been shown, but these processes may also be performed during the extraction period. That is, the constraints and removal priorities may be updated during the withdrawal period, and the surgical plan may be updated based on the updated constraints and removal priorities.

Further, in the above-described embodiment, an example is shown in which an organ model showing a three-dimensional structure of a luminal organ is created by using three-dimensional coordinate information of feature points extracted from an endoscopic image. Not limited to 3D models. The organ model may be, for example, a two-dimensional model that gives a bird's-eye view of the luminal organ. Further, the organ model is not limited to a model showing a three-dimensional structure such as the above-mentioned three-dimensional model and two-dimensional model. It may be a planar model such as a model in which a luminal organ is projected on a plane or a model showing a cross-sectional shape of the luminal organ.

Further, in the above-described embodiment, an example of detecting the spatial arrangement including the position and orientation of the image sensor 11 by using the magnetic sensor is shown, but the means for detecting the spatial arrangement of the image sensor 11 is not limited to the magnetic sensor. For example, the spatial arrangement of the image pickup element 11 may be detected by using a sensor that detects the shape of the endoscope 10 and a sensor that detects the insertion amount of the endoscope 10. The spatial arrangement of the tip of the insertion portion may be estimated by measuring the traction amount of the operation wire inserted through the insertion portion. Further, the spatial arrangement of the tip of the insertion portion may be estimated from the operation history of the operation portion for pulling and relaxing the operation wire. Further, the spatial arrangement of the tip of the insertion portion may be estimated by combining the traction amount and the operation history with the detection information of the gyro sensor provided at the tip of the insertion portion. Further, the spatial arrangement of the tip of the insertion portion may be estimated from the information obtained from the device other than the endoscope 10, and the device other than the endoscope 10 may be, for example, a medical imaging device.

1 ... Surgery support system, 10 ... Endoscope, 11 ... Imaging element, 12 ... Magnetic sensor, 40 ... Surgery support device 40a ... Acquisition unit, 40b ... Model creation Unit, 40c ... Removal priority calculation unit, 40d ... Constraint condition management unit, 40e ... Surgical plan creation unit, 40f ... Display control unit, 41 ... Processor, 42 ... Storage device , 45 ... Spatial layout detection device, 50 ... Display device, 80 ... Display unit, C ... Current position information, M1 to M3 ... Organ model, OP1, OP1a, OP1b, OP1c, OP2・・・ Surgery plan, P ・・・ Polyp position information, Pa ・・・ Patient, PTM ・・・ Learned model, T ・・・ Target polyp information, U ・・・ Non-target polyp information

Claims

A model for creating an organ model of the patient's tract organ based on the image captured by the imaging unit of the endoscope in the patient's tract organ and the spatial arrangement information of the tip of the insertion part of the endoscope. With the creation department
Create a surgical plan containing information that identifies the position of the polyp to be operated on among the polyps in the luminal organ identified from the image on the organ model, at least based on the surgical constraints. , A surgery support system comprising:, a surgery plan creation unit that updates the surgery plan with the update of the constraint condition during the surgery.
In the surgical support system according to claim 1, further
A removal priority calculation unit that calculates the removal priority of the polyp based on the malignancy of the polyp in the tract organ and the progress rate of the polyp in the luminal organ is provided.
The surgery support system is characterized in that the surgery plan creation unit creates the surgery plan based on the constraint condition and the removal priority.
In the surgical support system according to claim 2, further
It is provided with a storage unit that stores a learned model in which the malignancy and progression rate of the polyp with respect to the surface shape of the polyp are learned.
The removal priority calculation unit is a surgical support system characterized by estimating the malignancy and the rate of progression of polyps in the luminal organ using the learned model based on the image.
In the surgical support system according to claim 2, further
It is provided with a storage unit that stores a learned model in which the malignancy of the polyp with respect to the surface shape of the polyp is learned and the past test results of the luminal organ of the patient.
The removal priority calculation unit
Based on the image, the malignancy of the polyp in the luminal organ was estimated using the trained model.
A surgical support system comprising estimating the progress rate of a polyp in the luminal organ based on the image and the past test result.
In the surgical support system according to any one of claims 1 to 4.
The constraint condition includes at least one of acceptable operation cost, allowable operation time, allowable bleeding amount, operation difficulty, operator skill level, and patient information. ..
In the surgical support system according to claim 5.
The constraints include said acceptable surgical costs.
A surgical support system characterized in that the acceptable surgical costs are renewed based on the price of polyp removal performed during the surgery.
In the surgical support system according to claim 5 or 6.
The constraints include the permissible operating time.
A surgical support system characterized in that the allowable surgical time is updated based on the elapsed time from the start of the surgery.
In the surgical support system according to any one of claims 5 to 7.
The constraints include the permissible amount of bleeding.
A surgical support system characterized in that the allowable amount of bleeding is updated based on the amount of bleeding from the luminal organ estimated based on an image captured by the imaging unit during the operation.
In the surgical support system according to any one of claims 1 to 8.
The surgical support system is characterized in that the model creating unit creates the organ model based on the image and the spatial arrangement information obtained at least during the insertion period of the endoscope into the luminal organ. ..
In the surgical support system according to any one of claims 1 to 9, further
A surgery support system including at least the organ model and a display control unit for displaying the surgery plan on the display unit.
In the surgical support system according to claim 10.
The display control unit is a surgical support system further characterized in that the display unit displays current position information that specifies the current position of the tip of the insertion unit on the organ model.
In the surgical support system according to claim 10 or 11.
The display control unit is a surgical support system further characterized in that the display unit displays polyp information that identifies the position of a polyp in the luminal organ on the organ model.
In the surgical support system according to any one of claims 10 to 12.
The display control unit is a surgery support system characterized in that at least the organ model and the surgery plan are displayed on the display unit side by side or switched with a live image captured by the imaging unit.
In the surgical support system according to any one of claims 1 to 13, further
A surgical support system including the endoscope including the imaging unit and a sensor for detecting the spatial arrangement of the tip of the insertion unit.
In the surgical support system according to claim 14.
The endoscope is an endoscope for the large intestine.
A surgical support system characterized in that the luminal organ is the large intestine.
The computer
Based on the image captured by the imaging unit of the endoscope in the patient's tract organ and the spatial arrangement information of the tip of the insertion part of the endoscope, an organ model of the patient's tract organ is created.
Create a surgical plan containing information that identifies the position of the polyp to be operated on in the organ model among the polyps in the luminal organ identified from the image, at least based on the constraints of the operation. ,
A surgical support method comprising updating the surgical plan in accordance with the updating of the constraints during the surgery.
On the computer
Based on the image captured by the imaging unit of the endoscope in the patient's tract organ and the spatial arrangement information of the tip of the insertion part of the endoscope, an organ model of the patient's tract organ is created.
Create a surgical plan containing information that identifies the position of the polyp to be operated on in the organ model among the polyps in the luminal organ identified from the image, at least based on the constraints of the operation. ,
A program characterized in that a process of updating an operation plan is executed in association with an update of the constraint condition during the operation.