JP7495216B2 - Endoscopic surgery support device, endoscopic surgery support method, and program - Google Patents

Description

This disclosure relates to an endoscopic surgery support device, an endoscopic surgery support method, and a program.

Conventionally, methods for controlling the position and orientation of a camera in a laparoscopic surgery navigation system have been studied (see Non-Patent Document 1). In the system of Non-Patent Document 1, a pseudo camera consisting of a CCD camera attached to the tip of a forceps is used as the laparoscopic camera. The position and orientation of the tip of the pseudo camera are calculated on a PC by measuring markers on each face of a regular hexagonal base attached to this pseudo camera with an external camera.

Fuminami, Masatomo, and 5 others, "Study on Camera Position and Orientation Control Method for Laparoscopic Surgery Navigation System", [online], Proceedings of the Special Project Student Poster Session of the Information and Systems Society, 2018, IEICE (Institute of Electronics, Information and Communication Engineers), ISS-P-034, [Retrieved August 9, 2019], Internet <URL: https://www.ieice.org/~iss/jpn/Publications/issposter_2018/data/pdf/ISS-P-034.pdf>

When the system of Non-Patent Document 1 is used for surgical navigation, in order to derive the position of the laparoscopic camera corresponding to the pseudo camera, in addition to the laparoscopic camera, an external camera, for example, installed on the ceiling, is required to capture the state of the surgery. Therefore, the system for performing surgical navigation becomes large-scale. In addition, since the external camera is installed, for example, on the ceiling of the operating room, a coordinate system based on the operating room (also called the operating room coordinate system) is used. On the other hand, since the laparoscopic camera is operated inside the patient's body based on the port installed in the patient, a coordinate system based on the patient (patient coordinate system) is used. Therefore, when referring to an image using the operating room coordinate system and an image using the patient coordinate system in intraoperative navigation, mutual alignment is required. Therefore, in surgical navigation, a process of registering the coordinates in advance is required. Also, re-registration is required according to the patient's movements during surgery.

The present disclosure has been made in consideration of the above circumstances, and provides an endoscopic surgery support device, an endoscopic surgery support method, and a program that can simplify the system configuration and easily perform surgical navigation.

One aspect of the present disclosure is an endoscopic surgery support device that supports endoscopic surgery, comprising a processing unit, the processing unit having a function of acquiring 3D data of a subject and information on the shape of a surgical instrument , setting a virtual port on the body surface of the subject in the 3D data, acquiring information on the detected insertion distance of the surgical instrument inserted into the subject from a real port corresponding to the virtual port, and displaying, on a display unit, information indicating the surgical instrument and the tip position and tip orientation of the surgical instrument on a first image visualized from the 3D data based on at least the position of the virtual port, the detected insertion distance of the surgical instrument , and the shape of the surgical instrument.

This disclosure allows for a simplified system configuration and easy implementation of surgical navigation.

FIG. 1 is a block diagram showing an example of the hardware configuration of a medical image processing apparatus according to a first embodiment; A block diagram showing an example of the functional configuration of a medical image processing apparatus. FIG. 1 is a diagram showing an example of a positional relationship between an organ including a target, a surgical instrument used to treat the target, and a port into which the surgical instrument is inserted, and a subject coordinate system. A diagram showing an example of a port coordinate system based on a port FIG. 13 is a diagram showing an example of the positional relationship between the tip position, tip direction, imaging range, and forceps of an endoscope. FIG. 13 is a diagram showing an example of a display in which a volume rendering image and various information are superimposed on each other; FIG. 13 is a diagram showing a display example in which a virtual endoscopic image and various information are superimposed on each other; Flowchart showing an example of the operation of a medical image processing apparatus Flowchart showing an example of a procedure for port position simulation 1 is a flowchart showing an example of an operation for calculating a port position score by a medical image processing device. FIG. 13 is a diagram showing an example of a working area determined based on a virtual port position;

Embodiments of the present disclosure are described below with reference to the drawings.

First Embodiment
1 is a block diagram showing an example of the configuration of a medical image processing apparatus 100 according to the first embodiment. The medical image processing apparatus 100 includes an acquisition unit 110, a UI 120, a display 130, a processor 140, and a memory 150. The medical image processing apparatus 100 supports laparoscopic surgery (including robotic surgery) by image processing. The medical image processing apparatus 100 performs surgical navigation. The surgical navigation includes, for example, preoperative simulation for making a plan before surgery (preoperative plan) and intraoperative navigation for providing support during surgery.

The medical image processing device 100 is connected to a CT device 200. The medical image processing device 100 acquires volume data from the CT device 200 and processes the acquired volume data. The medical image processing device 100 may be configured by a PC and software installed on the PC.

The CT device 200 irradiates the subject with X-rays and captures an image (CT image) by utilizing the difference in X-ray absorption by tissues in the body. The subject may include a living body, a human body, an animal, etc. The CT device 200 generates volume data including information on any location inside the subject. The CT device 200 transmits the volume data as a CT image to the medical image processing device 100 via a wired or wireless line. When capturing a CT image, imaging conditions related to CT imaging and contrast conditions related to the administration of a contrast agent may be taken into consideration.

Information from a surgical instrument 300 is input to the medical image processing device 100. The surgical instrument 300 is an instrument used in surgery, and includes an endoscope (camera), forceps, etc. The medical image processing device 100 acquires from the surgical instrument 300, for example, insertion distance information, orientation information, and position information of the surgical instrument 300 (described later), and camera images captured by the surgical instrument 300, and performs processing.

The acquisition unit 110 in the medical image processing device 100 includes, for example, a communication port, an external device connection port, and a connection port to an embedded device. The acquisition unit 110 acquires volume data obtained by the CT device 200. The acquired volume data may be sent immediately to the processor 140 for various processing, or may be stored in the memory 150 and then sent to the processor 140 when necessary for various processing. The volume data may also be acquired via a recording medium or recording media. The volume data may also be acquired in the form of intermediate data, compressed data, or sinograms. The volume data may also be acquired from information from a sensor device attached to the medical image processing device 100. In this way, the acquisition unit 110 has the function of acquiring various data such as volume data.

The UI 120 may include, for example, a touch panel, a pointing device, a keyboard, or a microphone. The UI 120 accepts any input operation from a user of the medical image processing device 100. The user may include an operator, a doctor, a nurse, a radiologist, a student, etc.

The UI 120 accepts various operations. For example, it accepts operations such as specifying a region of interest (ROI) and setting brightness conditions in volume data or an image based on the volume data (e.g., a three-dimensional image or a two-dimensional image described below). The region of interest may include regions of various tissues (e.g., blood vessels, bronchi, organs, bones, and the brain). The tissues may include diseased tissue, normal tissue, tumor tissue, and the like.

The display 130 may include, for example, an LCD, and displays various information. The various information may include three-dimensional images and two-dimensional images obtained from volume data. The three-dimensional images may include volume rendering images, surface rendering images, virtual endoscopic images, virtual ultrasound images, CPR images, etc. The volume rendering images may include RaySum images, MIP images, MinIP images, average value images, raycast images, etc. The two-dimensional images may include axial images, sagittal images, coronal images, MPR images, etc.

The memory 150 includes various primary storage devices such as ROM and RAM. The memory 150 may include secondary storage devices such as HDD and SSD. The memory 150 may include tertiary storage devices such as USB memory and SD card. The memory 150 stores various information and programs. The various information may include volume data acquired by the acquisition unit 110, images generated by the processor 140, setting information set by the processor 140, and various programs. The memory 150 is an example of a non-transitory recording medium on which a program is recorded.

The processor 140 may include a CPU, a DSP, or a GPU. The processor 140 functions as a processing unit 160 that performs various processes and controls by executing a medical image processing program stored in the memory 150.

FIG. 2 is a block diagram showing an example of the functional configuration of the processing unit 160. The processing unit 160 includes an area processing unit 161, a transformation processing unit 162, a model setting unit 163, a port information processing unit 164, a surgical instrument information processing unit 165, an image generation unit 166, and a display control unit 167. Each unit included in the processing unit 160 may be realized as different functions by a single piece of hardware, or may be realized as different functions by multiple pieces of hardware. Each unit included in the processing unit 160 may also be realized by dedicated hardware components.

The region processing unit 161 acquires volume data of the subject, for example, via the acquisition unit 110. The region processing unit 161 extracts any region included in the volume data. The region processing unit 161 may automatically specify a region of interest and extract the region of interest, for example, based on pixel values of the volume data. The region processing unit 161 may manually specify a region of interest and extract the region of interest, for example, via the UI 120. The region of interest may include regions of organs, bones, blood vessels, affected areas (for example, diseased tissue or tumor tissue), etc.

The region of interest may be extracted by segmenting not only a single tissue but also the tissue surrounding that tissue. For example, if the organ of interest is the liver, it may include not only the liver itself, but also blood vessels (e.g., hepatic artery, hepatic vein, portal vein) that are connected to the liver or run within or around the liver, and bones around the liver (e.g., spine, ribs). The liver itself, blood vessels within or around the liver, and bones around the liver may be segmented and obtained as separate tissues.

The model setting unit 163 sets a model of the tissue. The model may be set based on the region of interest and the volume data. The model represents the tissue represented by the volume data in a simpler form than the volume data. Therefore, the amount of data of the model is smaller than the amount of data of the volume data corresponding to the model. The model is subject to deformation processing and deformation operations that imitate various procedures in surgery, for example. The model may be, for example, a bone deformation model. In this case, the model assumes a skeleton in simple finite elements, and the bones are deformed by moving the vertices of the finite elements. The deformation of the tissue can be expressed by following the deformation of the bones. The model may include an organ model that imitates an organ (for example, a liver). The model may have a shape similar to a simple polygon (for example, a triangle), or may have other shapes. The model may be, for example, the contour line of the volume data showing the organ. The model may be a three-dimensional model or a two-dimensional model. Note that bones may be expressed by deformation of the volume data rather than deformation of the model. This is because bones have a small degree of freedom in deformation and can be expressed by affine transformation of the volume data.

The model setting unit 163 may acquire a model by generating a model based on the volume data. Alternatively, multiple model templates may be determined in advance and stored in the memory 150 or an external server. The model setting unit 163 may acquire a model by acquiring one model template from multiple model templates prepared in advance from the memory 150 or an external server in accordance with the volume data.

The model setting unit 163 sets the position of the target in the subject's tissue contained in the volume data. Alternatively, the model setting unit 163 sets the position of the target in a model that mimics the tissue. The target is set in any tissue (e.g., an organ). The model setting unit 163 may specify the target position via U120. In addition, the position of a target (e.g., an affected area) that was previously treated on the subject may be stored in the memory 150. The model setting unit 163 may acquire the target position from the memory 150 and set it. The model setting unit 163 may set the target position according to the surgical procedure. The surgical procedure indicates the method of surgical operation on the subject. The target position may include the position of the target area.

The port information processing unit 164 sets the position of a port to be provided on the subject's body surface in order to insert the surgical instrument 300 into the subject's interior.

Note that a port that is placed (planned) on the subject's body surface in the surgical plan is also called a virtual port. A port that is actually placed (drilled) on the subject's body surface corresponding to a virtual port is also called a real port. There may be some error between the position of a virtual port and the position of a real port that is actually drilled in relation to this virtual port. Ports also include a camera port into which an endoscope is inserted, a forceps port into which forceps is inserted, etc. For example, port types may be combined and described, such as a virtual port as a camera port, and a real port as a virtual camera port or forceps port as a real forceps port.

The port information processing unit 164 determines and sets the position of the virtual port on the body surface of the subject in the 3D data based on the volume data. The port information processing unit 164 may determine the position of the virtual port based on port position simulation, port position score calculation, port position adjustment, etc., which will be described later. Details of an example of determining this virtual port will be described later. The position of the virtual port may be a standard port position determined according to the surgical procedure. The port information processing unit 164 may acquire information on the standard port position from the memory 150, or may acquire it from an external device via the acquisition unit 110.

The 3D data may be, for example, volume data of a subject in an untensioned state, which will be described later, or volume data of a subject in a virtual pneumoperitoneum state, which will be described later, after a pneumoperitoneum simulation has been performed. The 3D data may also be surface data generated from the volume data. The 3D data may also include the above-mentioned model. The 3D data may also include segmentation information of organs involved in the surgery.

The deformation processing unit 162 performs processing related to deformation of the subject to be operated on. For example, as the processing related to deformation, a pneumoperitoneum simulation may be performed to virtually infuse the subject. The specific method of the pneumoperitoneum simulation may be a known method, for example, the method described in Reference Non-Patent Document 1. That is, the deformation processing unit 162 may perform a pneumoperitoneum simulation based on volume data in a non-pneumoperitoneum state, and generate volume data in a virtual pneumoperitoneum state. By using the pneumoperitoneum simulation, the user can assume that the subject is insufflated and observe the virtually insufflated state without actually insufflating the subject. Note that, among the pneumoperitoneum states, the pneumoperitoneum state estimated by the pneumoperitoneum simulation may be referred to as a virtual pneumoperitoneum state, and the actual pneumoperitoneum state may be referred to as an actual pneumoperitoneum state.

(Reference Non-Patent Document 1) Takayuki Kitasaka, Kensaku Mori, Yuichiro Hayashi, Yasuhito Suenaga, Makoto Hashizume, and Jun-ichiro Toriwaki, “Virtual Pneumoperitoneum for Generating Virtual Laparoscopic Views Based on Volumetric Deformation”, MICCAI (Medical Image Computing and Computer-Assisted Intervention), 2004, P559-P567

The pneumoperitoneum simulation may be a large deformation simulation using the finite element method. In this case, the deformation processing unit 162 may segment the body surface including the subcutaneous fat of the subject and the abdominal internal organs of the subject via the region processing unit 161. Then, the deformation processing unit 162 may model the body surface into two-layer finite elements of skin and body fat, and model the abdominal internal organs into finite elements via the model setting unit 163. The deformation processing unit 162 may optionally segment, for example, the lungs and bones and add them to the model. In addition, a gas region may be provided between the body surface and the abdominal internal organs, and the gas region (pneumoperitoneum space) may expand (inflate) in response to virtual gas injection. Note that the pneumoperitoneum simulation does not have to be performed.

In addition, various deformation operations may be performed by the user on tissues such as organs within the subject, simulating various procedures performed by the surgeon during surgery. Deformation operations may include lifting, turning, cutting, and the like of the organ. In response to this, the deformation processing unit 162 may deform a model corresponding to tissues such as organs within the subject as a process related to deformation. For example, an organ may be pulled, pushed, or cut by forceps (e.g., grasping forceps, dissection forceps, electric scalpel), and this state may be simulated by deformation of the model. When the model is deformed, the target in the model may also be deformed.

Deformation by a deformation operation is performed on a model, and may be a large deformation simulation using the finite element method. For example, the movement of an organ due to a change in body position may be simulated. In this case, the elastic force acting on the contact points of the organ or lesion, the rigidity of the organ or lesion, and other physical characteristics may be taken into account. The amount of calculation required for deformation processing on a model is reduced compared to deformation processing on volume data. This is because the number of elements in the deformation simulation is reduced. Note that deformation processing may be performed directly on the volume data without performing deformation processing on the model.

The surgical instrument information processing unit 165 acquires operation information of the surgical instrument 300 input via the UI 120. The operation information of the surgical instrument 300 may include information such as the type of operation (e.g., movement, rotation), operation position, and operation speed. The surgical instrument information processing unit 165 may acquire information from the surgical instrument 300 (e.g., operation information, other information) directly from the surgical instrument 300 or other devices (e.g., various sensors), or may acquire information via the acquisition unit 110.

The surgical instrument information processing unit 165 acquires at least a portion of information related to the surgical instrument 300 (surgical instrument information). The surgical instrument information may include information related to the endoscope 20 and information related to the forceps 30. Information related to the endoscope 20 may be acquired from an endoscopic device including the endoscope 20, for example, via the acquisition unit 110. Information related to the forceps 30 may be acquired from, for example, the memory 150 or an external device. The endoscope 20 is inserted from a real camera port to capture images of the inside of the subject. The forceps 30 is inserted from a real forceps port to be used for various procedures inside the subject.

The surgical instrument information may include the orientation of the surgical instrument 300 (i.e., the extension direction of the surgical instrument 300, the insertion direction of the surgical instrument 300) and the insertion distance of the surgical instrument 300. The orientation of the surgical instrument 300 here may be the main orientation of the entire surgical instrument 300. The orientation of the surgical instrument 300 may include the direction of inclination of the surgical instrument 300 with respect to the direction of gravity, and may include a clockwise or counterclockwise inclination around the extension direction of the surgical instrument 300 as the center of rotation, that is, the direction of twisting. The insertion distance corresponds to, for example, the distance between the actual port into which the surgical instrument 300 is inserted and the tip position of the surgical instrument 300. For example, the surgical instrument 300 may be provided with a scale indicating the insertion distance of the surgical instrument 300. The surgical instrument information processing unit 165 may electronically read the scale and obtain the insertion distance of the surgical instrument 300. In this case, for example, a real encoder (reading device) may be attached to the trocar, and an encoding marker may be attached to the surgical instrument 300. Additionally, the surgical instrument information processing unit 165 may obtain the insertion distance of the surgical instrument 300 by having the user read the scale and input the insertion distance via the UI 120.

The surgical instrument information processing unit 165 may also obtain information on the angle (i.e. orientation) of the surgical instrument 300 from an angle sensor. The angle sensor may be attached to the surgical instrument 300 or the trocar, for example. The angle of the trocar corresponds to the angle of the surgical instrument 300. The detection value by the angle sensor will be described later. The surgical instrument information processing unit 165 derives the orientation of the surgical instrument 300 based on the detection value of the angle sensor.

The surgical instrument information processing unit 165 may also calculate the orientation of the surgical instrument 300. For example, the orientation of the surgical instrument 300 may be calculated as a predetermined orientation. The predetermined orientation may be, for example, a direction in which the affected area is approached from a port that is considered standard depending on the surgical procedure. The predetermined orientation may also be a direction set by the user. The orientation of the surgical instrument 300 may also be calculated as a direction from a virtual port acquired from the port information processing unit 164 toward a target acquired from the model setting unit 163. The orientation of the surgical instrument 300 may also be calculated following a change in the position of the target acquired from the model setting unit 163. The orientation of the surgical instrument 300 may also be calculated as a change in a predetermined angle with respect to the direction toward the target acquired from the model setting unit 163.

The surgical instrument information may include the orientation (extension direction) of the endoscope 20 and the insertion distance of the endoscope 20. The surgical instrument information may include the orientation of the forceps 30 and the insertion distance of the forceps 30.

The surgical instrument information processing unit 165 derives the tip position (position of the tip) and tip direction (direction of the tip) of the surgical instrument 300. For example, the surgical instrument information processing unit 165 may calculate the tip position of the endoscope 20 based on at least the position of the virtual camera port and the insertion distance of the endoscope 20. Also, for example, when the endoscope 20 has a shape that extends linearly, the tip position of the endoscope 20 may be calculated based on the position of the virtual camera port, the insertion distance of the endoscope 20, and the direction of the endoscope 20. Also, for example, when the endoscope 20 has a non-linear shape (for example, a bent shape or a curved shape), the tip position of the endoscope 20 may be calculated based on the position of the virtual camera port, the insertion distance of the endoscope 20, and the shape of the endoscope 20. Information on the shape of the endoscope 20 may be stored in, for example, the memory 150, or may be acquired from an external device via the acquisition unit 110. For example, when the endoscope 20 has a shape that extends linearly (for example, a rigid endoscope), the direction of the entire endoscope 20 may be calculated as the tip direction of the endoscope 20. For example, if the endoscope 20 has a non-linear shape, the tip orientation of the endoscope 20 may be calculated based on the overall orientation of the endoscope 20 and the shape of the endoscope 20.

The surgical instrument information processing unit 165 may calculate the tip position of the forceps 30 based on at least the position of the virtual forceps port and the insertion distance of the forceps 30. For example, when the forceps 30 has a shape that extends linearly, the tip position of the forceps 30 may be calculated based on the position of the virtual forceps port, the insertion distance of the forceps 30, and the orientation of the forceps 30. For example, when the forceps 30 has a non-linear shape (for example, a bent shape or a curved shape), the tip position of the forceps 30 may be calculated based on the position of the virtual forceps port, the insertion distance of the forceps 30, and the shape of the forceps 30. Information on the shape of the forceps 30 may be stored in the memory 150, for example, or may be acquired from an external device via the acquisition unit 110. For example, when the forceps 30 has a shape that extends linearly, the orientation of the forceps 30 as a whole may be calculated as the tip orientation of the forceps 30. For example, if the forceps 30 have a non-linear shape, the tip orientation of the forceps 30 may be calculated based on the overall orientation of the forceps 30 and the shape of the forceps 30. For example, if the forceps 30 have a non-linear shape, the tip orientation of the forceps 30 may be calculated based on the overall orientation of the forceps 30 and the shape of the forceps 30.

The image generating unit 166 generates various images. The image generating unit 166 generates three-dimensional images or two-dimensional images based on at least a portion of the acquired volume data (e.g., an area extracted from the volume data). The image generating unit 166 may generate three-dimensional images or two-dimensional images based on volume data (e.g., volume data in a virtual pneumoperitoneum state) deformed by the transformation processing unit 162. For example, the image generating unit 166 may generate a volume rendering image or a virtual endoscopic image that represents the state in which the direction of the endoscope 20 is viewed from the position of the endoscope 20.

The display control unit 167 causes various data, information, and images to be displayed on the display 130. The display control unit 167 causes an image (e.g., a rendering image) generated by the image generation unit 166 to be displayed. The display control unit 167 also causes various information to be superimposed and displayed together with this image. The superimposed information may include information on the distance from the tip of the surgical instrument 300 to the target, the insertion distance of the surgical instrument 300, and the like. The display control unit 167 may also adjust the brightness of the rendering image. The brightness adjustment may include, for example, adjustment of at least one of the window width (WW: Window Width) and the window level (WL: Window Level).

Figure 3 is a diagram showing an example of the positional relationship between an organ including a target 40, a surgical instrument 300 used to treat the target 40, and a port PT into which the surgical instrument 300 is inserted. In Figure 3, the surgical instrument 300 includes an endoscope 20 and forceps 30. A virtual port and a real port are assumed as the port PT. The real port of the port PT is a port that is drilled relative to the virtual port of the port PT. It includes the port PT, a camera port PT1, and a forceps port PT2. In Figure 3, the subject is insufflated. In Figure 3, the organ is a liver 10. The liver 10 includes a target 40 that is the subject of surgery. In Figure 3, bones 15 (e.g., spine, ribs) around the liver 10 are also shown.

A virtual port of camera port PT1 is set on the body surface 70 of the subject in a virtual pneumoperitoneum state. A real port corresponding to the virtual port of camera port PT1 is placed on the insufflated body surface 70. A trocar 60 is placed in the real port of camera port PT1 to keep the inside of the subject airtight. An endoscope 20 is inserted into the trocar 60 toward the inside of the subject. The endoscope 20 can be operated with the position of the real port of camera port PT1 as the base point. An image sensor is placed at the tip of the endoscope 20, and an image can be captured by the image sensor. The distance from the real port of camera port PT1 to the tip of the endoscope 20 is the insertion distance ID1 of the endoscope 20.

In addition, a virtual port of the forceps port PT2 is set on the body surface 70 of the subject in a virtual pneumoperitoneum state. A real port corresponding to the virtual port of the forceps port PT2 is placed on the insufflated body surface 70. A trocar 60 is placed on the real port of the forceps port PT2 to keep the inside of the subject airtight. The forceps 30 is inserted into this trocar 60 toward the inside of the subject. The forceps 30 can be operated with the position of the real port of the forceps port PT2 as the base point. The distance from the real port of the forceps port PT2 to the tip of the forceps 30 is the insertion distance ID2 of the forceps 30.

The forceps 30, for example, advances toward the target 40 and into the subject's body, and is used for various treatments (e.g., grasping, resection, dissection, suturing) on the target 40. The endoscope 20 is operated, for example, so that the target 40 and the forceps 30 are included in the imaging range 23. Thus, the medical image processing device 100 visualizes an image corresponding to the endoscopic image from the endoscope 20, making it possible to visually confirm the treatment of the target 40 by the forceps 30, and can assist, for example, the surgeon in performing endoscopic surgery while checking the positional relationship between the forceps 30 and the target 40.

Figure 3 also shows the x, y, and z directions of the subject coordinate system (patient coordinate system) based on the subject. The subject coordinate system is a Cartesian coordinate system. The x direction may be along the left-right direction based on the subject. The y direction may be the front-back direction based on the subject (thickness direction of the subject). The z direction may be the up-down direction based on the subject (body axis direction of the subject). The x, y, and z directions may be the three directions defined by DICOM (Digital Imaging and COmmunications in Medicine). For example, values of the subject coordinate system are used for 3D data of the subject, internal regions of the subject, positions of virtual ports, etc.

Figure 4 is a diagram showing an example of a port coordinate system based on the port PT. The port coordinate system is shown in a spherical coordinate system (polar coordinate system), and a radius vector r, a first angle θ, and a second angle φ are shown. The origin of the port coordinate system is the position of the port PT. The radius vector r is parallel to the extension direction of the endoscope 20. The first angle θ is the angle between a specific axis (e.g., the y-axis of the subject coordinate system) and the radius vector r. The second angle φ is the angle between another axis (e.g., the z-axis of the subject coordinate system) on a plane perpendicular to the specific axis and the projection of the radius vector r onto this plane. The extension direction of the endoscope 20 is the depth direction of the endoscopic image (real endoscopic image) and the virtual endoscopic image.

For example, the insertion distance and orientation of the surgical instrument 300 based on the port position are expressed as values in the port coordinate system. The insertion distance of the surgical instrument 300 may be expressed as a coordinate of the radial coordinate r. The orientation of the surgical instrument 300 may be detected, for example, as the values of the first angle θ and the second angle φ. The values of the first angle θ and the second angle φ may be values detected by an angle sensor. Furthermore, the surgical instrument information processing unit 165 may calculate and obtain the first angle θ and the second angle φ based on the values detected by the angle sensor. In this case, for example, the angle sensor may be an acceleration sensor, and may detect the tilt angles in two directions perpendicular to the direction of gravity. Furthermore, the angle sensor may be a three-axis angle sensor.

When the insertion distance and orientation of the surgical instrument 300 using the port coordinate system are used to process 3D data of the subject, it is necessary to convert the values in the port coordinate system to values in the subject coordinate system. Here, conversion from the port coordinate system (spherical coordinate system) to the subject coordinate system (Cartesian coordinate system) requires less calculation and can be performed relatively easily compared to conversion from the subject coordinate system to the operating room coordinate system (conversion in a different Cartesian coordinate system). Note that even if there is some movement of the subject (e.g., the patient's movements during the examination, heartbeat), the change in the orientation of the surgical instrument 300 is smaller than the detected position value, and angle alignment is easier than position alignment.

Figure 5 is a diagram showing an example of the positional relationship between the tip position 21, tip direction 22, imaging range 23, and forceps 30 of the endoscope 20. The tip position 21, tip direction 22, and imaging range 23 of the endoscope 20 are included in the information about the endoscope 20. The position of the forceps 30 is included in the information about the forceps 30. The tip direction 22 of the endoscope 20 coincides with the direction of the radius vector r, and may coincide with the direction of the optical axis of the endoscope 20 (camera). The imaging range 23 of the endoscope 20 may be determined based on, for example, the tip position 21, tip direction 22, and angle of view of the endoscope 20. The imaging range 23 of the endoscope 20 coincides with the image range of the virtual endoscopic image.

The tip position 21 of the endoscope 20 can also be considered as the surgeon's viewpoint for observing the subject. The tip direction 22 of the endoscope 20 can also be considered as the direction of the surgeon's line of sight starting from the viewpoint. The field of view of the endoscope 20 can also be considered as the surgeon's field of view starting from the viewpoint. The imaging range 23 of the endoscope 20 can also be considered as the surgeon's field of view.

Figure 6 is a diagram showing a display example in which a volume rendering image G1 and various information are superimposed. Note that the display object shown in Figure 6 is virtual information, and for example, a virtual port is displayed as the port PT, virtual forceps is displayed as the forceps 30, and a virtual endoscope is displayed as the endoscope 20.

In the volume rendering image G1, a portion of the subject is visualized. In FIG. 6, various information is displayed together with the volume rendering image G1. This various information may include information about the endoscope 20, information about the forceps 30, information about the virtual port of the camera port PT1, information about the virtual port of the forceps port PT2, information about the target 40, various distance information, and the like. Note that the display of some of the various information may be omitted.

The information about the endoscope 20 may include information about the position, orientation, imaging range, size, shape, etc. of the endoscope 20. The information about the forceps 30 may include information about the position, orientation, size, shape, etc. of the forceps 30. In FIG. 6, the endoscope 20 and the forceps 30 are shown as straight lines for simplification, but are not limited to this and may be shown based on the orientation, size, shape, etc. of the endoscope 20 and the forceps 30.

The virtual port information of the camera port PT1 may include information such as the position, size, shape, etc. of the virtual port of the camera port PT1. The virtual port information of the forceps port PT2 may include information such as the position, size, shape, etc. of the virtual port of the camera port PT1. The target 40 information may include information such as the position, size, shape, etc. of the target 40. In FIG. 6, the target is simplified and shown as a circle, but is not limited to this and may be shown based on the orientation, size, shape, etc. of the target 40.

The various distance information may include information on the insertion distance ID1 of the endoscope 20, information on the insertion distance ID2 of the forceps 30, information on the distance d1 from the tip of the forceps 30 to the target 40, etc. The distance d1 can be calculated, for example, based on the orientation of the forceps 30, from the difference between the position of the tip of the forceps 30 and the position of the target 40.

Figure 7 shows an example of a display in which a virtual endoscopic image G2 and various information are superimposed. In the explanation of Figure 7, the explanation of matters similar to those explained in Figure 6 will be omitted or simplified.

In the virtual endoscopic image G2 in FIG. 7, a portion of the subject is visualized. In FIG. 7, various information is displayed together with the virtual endoscopic image G2. This various information may include information about one or more forceps 30 (e.g., forceps 31 and forceps 32), information about the target 40, various distance information, and the like. Note that display of some of the various information may be omitted.

The various distance information may include the insertion distance ID1 of the endoscope 20, the insertion distance ID21 of the forceps 31, the insertion distance ID22 of the forceps 32, the distance d11 from the tip (tip position) of the forceps 31 to the target 40, the distance d1 from the tip of the forceps 32 to the target 40, etc. Note that, instead of the information on the distance from the tip of the forceps 30 to the target 40, the distance information may include and display information on the distance from the virtual port of port PT to the target 40, or information on the distance obtained by subtracting the insertion distance ID2 of the forceps 30 from the distance from the virtual port of port PT to the target 40.

The display examples in Figures 6 and 7 allow the surgeon to easily confirm the distance taking into account the depth direction of the image, which is difficult to recognize when checking a two-dimensional image. For example, the surgeon can confirm information such as the remaining distance to the target 40 in meters.

Figure 8 is a flowchart showing an example of the operation of the medical image processing device 100. Note that steps S11 to S15 are performed, for example, before surgery, and steps S16 to S20 are performed, for example, during surgery.

First, volume data of the subject (e.g., a patient) is acquired (S11). A pneumoperitoneum simulation is performed (S12). Segmentation is performed to extract regions of organs, bones, and blood vessels (S13). A model (e.g., a liver organ model) is generated based on the volume data (S14). Port position simulation, port position score calculation, port position adjustment, etc. are performed to acquire information (e.g., position information) of the forceps port PT2 as a virtual port, the camera port PT1 as a virtual port, and the target 40 (S15).

Through the UI 120, operation information of the forceps 30 and operation information of the endoscope 20 are acquired (S16). The direction and insertion distance ID1 of the endoscope 20 are acquired (S17). The direction and insertion distance ID2 of the forceps 30 are acquired (S17). The tip position and tip direction of the endoscope 20 are acquired (S18). The tip position and tip direction of the forceps 30 are acquired (S18). The volume data of the virtual pneumoperitoneum state obtained by the processing of S12 is rendered to generate a volume rendering image G1 (S19). Various information is displayed together with the volume rendering image G1 (S19). This various information may include the various information shown in FIG. 6, information indicating the endoscope 20, information indicating the forceps 30, the tip position and tip direction of the endoscope 20, the tip position and tip direction of the forceps 30, etc. In addition, a virtual endoscopic image G2 is generated based on the volume data of the virtual pneumoperitoneum state by the processing of S12 (S20). Various information is displayed together with the virtual endoscopic image G2 (S20). This various information may include the various information shown in FIG. 7, information indicating the forceps 30, the tip position and tip direction of the forceps 30, etc.

After displaying S20, the processing unit 160 may proceed to S16 and repeatedly execute S16 to S20. This allows the surgeon to move the endoscope 20 and forceps 30 during surgery as appropriate according to the treatment, while checking images that correspond to this movement (e.g., volume rendering image G1, virtual endoscopic image G2).

The accuracy of the organ segmentation, bone segmentation, and blood vessel segmentation in S13 does not need to be high. For example, some errors in the contours of organs, bones, and blood vessels are sufficient as long as they do not interfere with the intraoperative simulation.

In this way, the medical image processing device 100 can derive the distance between the tip position of the surgical instrument 300 and the target 40 based on information such as the position of the set virtual port and the insertion distance ID of the surgical instrument 300, without using a three-dimensional position sensor. Therefore, although it is difficult for the surgeon to grasp the sense of depth in a two-dimensional endoscopic image, it becomes easier for the surgeon to grasp the positional relationship between the tip position of the surgical instrument 300 and the target 40. Therefore, for example, it is possible to grasp how close the surgeon is to the target 40. Furthermore, the surgeon can be confident that his or her procedure is correct thanks to the surgical assistance provided by such images.

Therefore, in surgical navigation, the medical image processing device 100 eliminates the need to align the operating room coordinate system based on an external camera or the sensor coordinate system based on a three-dimensional position sensor with the subject coordinate system based on the subject, thereby reducing the amount of calculations.

In addition, even if measurement of the 3D position of the actual port is omitted, for example in the abdomen and lungs where organs are greatly deformed, they will move even if their 3D positions are acquired, so the importance of precise 3D coordinates is low and the impact on surgical navigation is small. In addition, the surgeon can more easily intuitively grasp the positional relationship between the surgical instrument 300 and the target 40, etc., from the 2D positions in an image that visualizes 3D data, rather than information based on the 3D coordinates from a 3D position sensor. This is because the virtual endoscopic image G2 is an image based on the position of the endoscope 20, and is an image that is close to the endoscopic image referred to by the surgeon.

In this way, the medical image processing device 100 can perform surgical navigation using a simple mechanism, improving surgical safety and shortening surgical time.

Next, we will explain in detail an example of how to determine the location of a virtual port.

The port information processing unit 164 executes a port position simulation. The port position simulation is a simulation for determining whether or not a desired surgery is possible on a subject by a user operating the UI 120. In the port position simulation, the user may operate the surgical instrument 300 inserted from the position of each virtual port (virtual port position) in a virtual space while imagining the surgery, and determine whether or not the forceps 30 can access the target 40 that is to be the surgical subject. In other words, in the port position simulation, while manually operating the surgical instrument 300, it may be determined whether or not the surgical instrument 300 can access the target 40 without any problems. The port information processing unit 164 may obtain planning information for the virtual port positions by the port position simulation.

In the port position simulation, whether the above access is possible may be determined based on the subject's volume data, a combination of the acquired multiple virtual port positions, movement information regarding the possible operations of the surgical instrument 300 (e.g., the position and range in which it can be operated), the surgical procedure, the volume data of the virtual pneumoperitoneum state, and the like. The movement information of the surgical instrument 300 may include shape information regarding the shape of the surgical instrument 300 and movement information regarding the operation. This shape information may include at least some information such as the length, weight, and shape of the surgical instrument 300. This movement information may include at least some information such as the insertable distance of the surgical instrument 300 into the subject, the insertable angle with respect to the subject, and the like. In the case of robotic surgery, the movement information of the surgical instrument 300 may correspond to the kinematics of the end effector. The port information processing unit 164 may acquire the movement information of the surgical instrument 300 from the memory 150, or may acquire it from an external device via the acquisition unit 110. The port information processing unit 164 may acquire the surgical procedure information from the memory 150, or may acquire it from an external device via the acquisition unit 110.

The port information processing unit 164 may determine whether the target 40 is accessible at each virtual port position while changing the multiple virtual port positions on the subject's body surface, and may sequentially perform port position simulations. The port information processing unit 164 may specify information on a final preferred (e.g., optimal) combination of port positions in response to user input via the UI 120. In this way, the port information processing unit 164 may plan multiple virtual port positions.

The port information processing unit 164 may calculate a port position score indicating the appropriateness of surgery using multiple virtual port positions provided on the subject's body surface. In other words, the port position score based on a combination of multiple virtual port positions indicates the value of the combination of multiple virtual port positions for performing surgery. The port position score may be calculated based on the combination of multiple virtual port positions, the movement information of the surgical instrument 300, the surgical procedure, volume data of the virtual pneumoperitoneum state, etc. The port position score is calculated for each virtual port position.

The port information processing unit 164 may adjust the port position based on the port position score. In this case, the port information processing unit 164 may adjust the port position based on the amount of change in the port position score associated with the movement of the port position.

In this way, the port information processing unit 164 may calculate multiple virtual port positions according to the port position simulation. The port information processing unit 164 may also calculate multiple virtual port positions based on the port position score.

Figure 9 is a flowchart showing an example of the port position simulation procedure.

First, the port information processing unit 164 acquires volume data including the subject, for example, via the acquisition unit 110 (S111). The port information processing unit 164 acquires movement information of the surgical instrument 300, for example, via the acquisition unit 110 (S112). The deformation processing unit 162 executes a pneumoperitoneum simulation (S113) and generates volume data of the subject in a virtual pneumoperitoneum state.

The port information processing unit 164 acquires information on the surgical procedure (S114). The port information processing unit 164 acquires and sets multiple virtual port positions (initial positions) according to the acquired surgical procedure (S114). In this case, the port information processing unit 164 may set multiple virtual port positions in three-dimensional coordinates.

The port information processing unit 164 sets the position of the target 40 (target area) (S115).

The port information processing unit 164 determines whether each of the forceps 30 inserted from each virtual port can access the target 40 based on the multiple virtual port positions and the position of the target area acquired in S114 (S116). Whether each of the forceps 30 can access the target 40 may correspond to whether each of the forceps 30 can reach all positions in the target area. In other words, it indicates whether surgery according to the acquired surgical procedure is possible using the forceps 30 (multiple forceps 30, if necessary), and if access is possible, it indicates that surgery is possible.

If at least one of the forceps 30 cannot access at least a part of the target region, the port information processing unit 164 moves the position of at least one virtual port included in the multiple virtual ports along the body surface of the subject (S117). In this case, the port information processing unit 164 may move the virtual port position based on user input via the UI 120. The virtual port to be moved includes at least the virtual forceps port into which the forceps 30 that was unable to access at least a part of the target region is inserted. After the virtual port is moved, the process proceeds to S116.

When each forceps 30 is able to access the target area, the processing unit 160 ends the port position simulation process of FIG. 9.

In this way, the medical image processing apparatus 100 can determine whether surgery can be performed using the acquired multiple virtual port positions, depending on whether the target area is accessible using the acquired multiple virtual port positions by performing a port position simulation. If the target area is inaccessible using the multiple virtual port positions, at least a portion of the virtual port positions may be changed via the UI 120, and whether the target area is accessible using the changed multiple virtual port positions may be determined again. The port information processing unit 164 may set a combination of multiple virtual port positions that are accessible to the target area to the positions of the multiple virtual ports. In this way, the medical image processing apparatus 100 allows the user to manually adjust the virtual port positions and plan the virtual port positions.

The port position simulation may be performed as part of the process of S15 in FIG. 8. In this case, processes that overlap with the process of FIG. 8 (e.g., S111, S113) can be omitted.

Next, we will explain an example of how to calculate the port position score.

The multiple virtual port positions may be determined, for example, according to a surgical procedure, and may be assumed to be placed at any position on the subject's body surface. Therefore, various combinations of multiple virtual port positions may be assumed. One surgical instrument 300 can be inserted into the subject from one virtual port. Therefore, multiple surgical instruments 300 can be inserted into the subject from multiple virtual ports.

The range that one forceps 30 can reach within the subject via a virtual port is the working area (individual working area WA1 (see FIG. 11)) in which work (treatment) can be performed by one forceps 30. Therefore, the area where the individual working areas WA1 of multiple forceps 30 overlap is the working area (total working area WA2 (see FIG. 11)) that multiple forceps 30 can reach simultaneously within the subject via multiple virtual ports. In treatment according to a surgical procedure, a predetermined number (e.g., three) of forceps 30 need to operate simultaneously, so the total working area WA2 that the predetermined number of forceps 30 can reach simultaneously is taken into account.

In addition, since the positions in the subject that the forceps 30 can reach vary depending on the movement information of the forceps 30, this movement information is taken into account in deriving the virtual port position, which is the position where the forceps 30 is inserted into the subject.In addition, since the position in the subject of the overall working area WA2 that should be secured varies depending on the surgical procedure, this position is taken into account in deriving the virtual port position that corresponds to the position of the overall working area WA2.

The port information processing unit 164 may calculate a port position score for each combination of multiple virtual port positions. The port information processing unit 164 may plan a combination of virtual port positions that, among the combinations of multiple virtual port positions, results in a port position score that satisfies a specified condition (e.g., the maximum port score). In other words, the multiple virtual port positions included in the planned combination of virtual port positions may be planned as the multiple port positions to be drilled.

In the case of robotic surgery, the relationship between the virtual port position and the operation of the movable part of the surgical support robot may satisfy the relationship described in, for example, reference non-patent documents 2 and 3.
(Reference Non-Patent Document 2): Mitsuhiro Hayashibe, Naoki Suzuki, Makoto Hashizume, Kozo Konishi, Asaki Hattori, “Robotic surgery setup simulation with the integration of inverse-kinematics computation and medical imaging”, computer methods and programs in biomedicine, 2006, pp. 63-72
(Reference Non-Patent Document 3) Pal Johan From, “On the Kinematics of Robotic-assisted Minimally Invasive Surgery”, Modeling Identication and Control, Vol. 34, No. 2, 2013, P69-P82

Figure 10 is a flowchart showing an example of the operation of the medical image processing device 100 when calculating the port position score.

Before the processing of FIG. 10, similar to S111 to S114 of the port position simulation shown in FIG. 9, the acquisition of subject volume data, acquisition of movement information of the surgical instrument 300, execution of a pneumoperitoneum simulation, and acquisition of surgical procedure information are performed in advance. Note that the initial value of the port position score is 0. The port position score is an evaluation function (evaluation value) that indicates the value of a combination of port positions. Note that the variable i is an example of task identification information, and the variable j is an example of port identification information.

The port information processing unit 164 generates a work list works, which is a list of work work_i using each surgical instrument 300 according to the surgical procedure (S121). The work work_i includes information for performing work using each forceps 30 in a surgical procedure according to the surgical procedure. The work work_i may include, for example, grasping, resection, suturing, etc. The work may include a solo work using a single forceps 30 and a collaborative work using multiple forceps 30.

The port information processing unit 164 plans a minimum region least_region_i, which is the minimum region required to perform a task work_i included in the task list works, based on the volume data of the surgical procedure and the virtual pneumoperitoneum state (S122). The minimum region may be defined as a three-dimensional region in the subject. The port information processing unit 164 generates a minimum region list Least_regions, which is a list of the minimum regions least_region_i (S122).

The port information processing unit 164 plans a recommended region effective_region_i, which is a region recommended for performing a task work_i included in the task list works, based on the surgical procedure, the movement information of the surgical instrument 300, and the volume data of the virtual pneumoperitoneum state (S123). The port information processing unit 164 generates a recommended region list effective_regions, which is a list of the recommended regions effective_region_i (S123). The recommended region may include a minimum space (minimum region) for performing the task, as well as a space recommended for the operation of the forceps 30, for example.

The port information processing unit 164 acquires information on the port position list ports, which is a list of multiple virtual port positions port_j (S124). The virtual port positions are defined by three-dimensional coordinates (x, y, z) based on the subject. The port information processing unit 164 may, for example, accept user input via the UI 120 and acquire the port position list ports including one or more virtual port positions specified by the user. The port information processing unit 164 may acquire the port position list ports stored as a template in the memory 150.

The port information processing unit 164 plans a port working area region_i, which is an area in which each forceps 30 can work via each virtual port position port_j, for each work work_i based on the surgical procedure, the movement information of the forceps 30, the volume data of the virtual pneumoperitoneum state, and multiple virtual port positions (S125). The port working area may be defined as a three-dimensional area. The port information processing unit 164 generates a port working area list regions, which is a list of the port working areas region_i (S125).

For each work work_i, the port information processing unit 164 subtracts the port working area region_i from the minimum area least_region_i to calculate a subtraction area (subtraction value) (S126). The port information processing unit 164 determines whether the subtraction area is not an empty area (the subtraction value is a negative value) (S126). Whether the subtraction area is not an empty area indicates whether there is an area not covered by the port working area region_i (an area that the forceps 30 cannot reach via the virtual port) in at least a part of the minimum area least_region_i.

If the subtraction region is an empty region, the port information processing unit 164 calculates a volume value Volume_i, which is the product of the recommended region effective_region_i and the port work region region_i (S127). Then, the port information processing unit 164 sums the volume values Volume_i calculated for each work work_i to calculate a total value Volume_sum. The port information processing unit 164 sets the total value Volume_sum to the port position score (S127).

In other words, when the subtraction area is an empty area, it is preferable that there is no area in the minimum area that is not covered by the port working area, and this port position list ports (combination of virtual port positions port_j) is selected, so a value for each work work_i is added to the port position score to make this port position list easier to select. Also, by planning the port position score based on the volume value Volume_i, the larger the minimum area or port working area, the larger the port position score becomes, and this port position list ports becomes easier to select. Therefore, the port information processing unit 164 can easily select a combination of virtual port positions that has a large minimum area or port working area and makes each procedure in the surgery easier.

On the other hand, if the subtraction area is not an empty area, the port information processing unit 164 sets the port position score for the port position list ports to a value of 0 (S128). In other words, since there is at least a part of the minimum area that is not covered by the port work area and there is a possibility that the target work work_i cannot be completed, it is undesirable for this port position list Posts to be selected. Therefore, the port information processing unit 164 sets the port position score to a value of 0 so that this port position list Posts is less likely to be selected, and excludes it from the selection candidates. In this case, the port information processing unit 164 sets the overall port position score to a value of 0 even if the area becomes empty when another work work_i is performed using the same port position list ports.

The port information processing unit 164 may repeat the steps in FIG. 10 for all work_i and calculate the port position score taking into account all work_i.

In this way, by deriving the port position score, the medical image processing device 100 can grasp how appropriate a combination of virtual port positions is when performing surgery using multiple virtual port positions placed on the subject's body surface. The individual working area WA1 and the overall working area WA2 are determined by the placement positions of the multiple virtual ports. Even in this case, the medical image processing device 100 can derive a combination of multiple virtual port positions that results in a port position score equal to or greater than a threshold value th1 (e.g., the maximum) by taking into account the score (port position score) for each combination of multiple virtual port positions, and can set a virtual port position that makes it easy to perform surgery.

In addition, by appropriately securing the working area based on the port position score, the user can secure a wide field of view inside the subject that cannot be directly observed during surgery, and a wide port working area can be secured, making it easier to deal with unforeseen circumstances.

In addition, in laparoscopic surgery, the position of the real port that is drilled in correspondence with the virtual port remains unchanged, but the forceps 30 inserted into the real port can be moved within a certain range. Therefore, in laparoscopic surgery, the difficulty of each procedure in the surgery changes depending on the planned virtual port position, so planning the virtual port position is important.

Figure 11 is a diagram showing an example of a working area determined based on a virtual port position. An individual working area WA1 is an individual working area corresponding to each virtual port position port_j. An individual working area WA1 may be an area within the subject PS that an individual end effector can reach. An area where each individual working area WA1 overlaps is a total working area WA2. The total working area WA2 may correspond to the port working area region_i. By using the port position score, the medical image processing apparatus 100 can optimize each virtual port position and derive suitable individual working area WA1 and total working area WA2.

Next, we will explain the details of port position adjustment.

The port information processing unit 164 acquires information on multiple virtual port positions (candidate positions) based on, for example, a template stored in the memory 150 or a user instruction via the UI 120. The port information processing unit 164 calculates a port position score when multiple virtual port positions are used based on a combination of the acquired multiple virtual port positions.

The port information processing unit 164 may adjust the position of the virtual port based on the port position score. In this case, the port information processing unit 164 may adjust the virtual port position based on the port position score for the multiple virtual port positions acquired and the port position score when at least one of the multiple virtual port positions is changed. In this case, the port information processing unit 164 may take into account minute movements and differentiation of the port position along each direction (x direction, y direction, z direction) in three-dimensional space.

For example, the port information processing unit 164 may calculate a port position score F(ports) for multiple virtual port positions according to (Equation 1) and calculate a differential value F' of F.

F(port_j(x+Δx, y, z)) - F(port_j(x, y, z))
F(port_j(x, y+Δy, z)) - F(port_j(x, y, z)) ... (Equation 1)
F(port_j(x, y, z+Δz)) - F((port_j(x, y, z))

In other words, the port information processing unit 164 calculates the port position score F for the virtual port position F(port_j(x+Δx, y, z)), calculates the port position score F for the virtual port position F(port_j(x, y, z)), and calculates the difference between them. This difference value indicates the change in the port position score F for a small change in the x direction at the virtual port position F(port_j(x, y, z)), that is, it indicates the differential value F' of F in the x direction.

The port information processing unit 164 also calculates the port position score F for the virtual port position F(port_j(x, y+Δy, z)), calculates the port position score F for the virtual port position F(port_j(x, y, z)), and calculates the difference between them. This difference value indicates the change in the port position score F for a small change in the y direction at the virtual port position F(port_j(x, y, z)), that is, indicates the differential value F' of F in the y direction.

The port information processing unit 164 also calculates the port position score F for the virtual port position F(port_j(x, y, z+Δz)), calculates the port position score F for the virtual port position F(port_j(x, y, z)), and calculates the difference between them. This difference value indicates the change in the port position score F for a small change in the z direction at the virtual port position F(port_j(x, y, z)), that is, indicates the differential value F' of F in the z direction.

The port information processing unit 164 calculates the maximum value of the port position score based on the differential value F' in each direction. In this case, the port information processing unit 164 may calculate the virtual port position at which the port position score is maximum according to the steepest descent method based on the differential value F'. The port information processing unit 164 may adjust the virtual port position so that the calculated virtual port position becomes the port position to be drilled, thereby optimizing the virtual port position. Note that the virtual port position does not have to be the one at which the port position score is maximum, and may be, for example, a position at which the port position score is equal to or greater than the threshold value th2, as long as the port position score is improved (higher).

The port information processing unit 164 may apply such adjustment of the virtual port position to the adjustment of other virtual port positions included in the combination of multiple virtual port positions, or to the adjustment of virtual port positions in other combinations of multiple virtual port positions. This allows the port information processing unit 164 to plan multiple virtual ports with their respective virtual port positions adjusted (e.g. optimized) as the port positions to be drilled. In this way, the medical image processing device 100 can automatically adjust the virtual port positions and set the positions of the virtual ports.

In addition, in the case of multiple port positions, an error of about a predetermined length (e.g., 25 mm) may occur between the virtual port position (planned drilling position) and the real port position (actual drilling position), and it is considered that the planning accuracy of the virtual port position is sufficient if it is at most 3 mm. Therefore, the port information processing unit 164 may set virtual port position candidates for each predetermined length on the body surface of the subject, determine a combination of multiple virtual port positions by exhaustively selecting the virtual port position candidates, and calculate the port position score for each combination of multiple virtual port positions. In other words, the virtual port position candidates may be arranged in a grid of a predetermined length (e.g., 3 mm) on the body surface of the subject. In addition, when the number of ports assumed on the body surface (e.g., the number of intersections of the grid) is n and the number of ports included in the combination of virtual port positions is m, the port information processing unit 164 may select and combine m virtual port positions from the n virtual port position candidates in order, and calculate the port position score for each combination. In this way, if the grid is not too fine, such as a 3 mm spacing, the calculation load on the port information processing unit 164 can be prevented from becoming excessive, and the port position scores for all combinations can be calculated.

The port information processing unit 164 may adjust the multiple virtual port positions according to a known method. The port information processing unit 164 may plan the port position to be drilled at multiple virtual port positions included in the combination of virtual port positions after adjustment. Known methods for adjusting port positions may include the techniques described in the following reference non-patent documents 4 and 5 and reference patent document 1.

(Reference Non-Patent Document 4) Shaun Selha, Pierre Dupont, Robert Howe, David Torchiana, “Dexterity optimization by port placement in robot-assisted minimally invasive surgery”, SPIE International Symposium on Intelligent Systems and Advanced Manufacturing, Newton, MA, 28-31, 2001
(Reference Non-Patent Document 5) Zhi Li, Dejan Milutinovic, Jacob Rosen, “Design of a Multi-Arm Surgical Robotic System for Dexterous Manipulation”, Journal of Mechanisms and Robotics, 2016
(Reference Patent Document 1) U.S. Patent Application Publication No. 2007/0249911

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or revised examples within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

For example, the endoscopic surgery in which the preoperative simulation and intraoperative navigation are performed may be laparoscopic surgery, arthroscopic surgery, bronchoscopy surgery, colonoscopic surgery, or other endoscopic surgery. In addition, the endoscopic surgery may be surgery in which the surgeon directly operates forceps, or robotic surgery using a surgical robot.

In addition, the pneumoperitoneum simulation may be omitted. For example, in arthroscopic surgery and bronchoscopic surgery, the pneumoperitoneum simulation may be omitted.

The surgical instrument information processing unit 165 may acquire an endoscopic image from an endoscopic device including an endoscope 20 via the acquisition unit 110. Then, the position of the surgical instrument 300 reflected in the endoscopic image, that is, the relative position of the surgical instrument 300 with respect to the endoscopic image, may be recognized by image recognition or the like. The surgical instrument information processing unit 165 may correct (adjust) the acquired position of the surgical instrument 300 based on the position of the surgical instrument 300 reflected in the endoscopic image. The surgical instrument information processing unit 165 may also correct (adjust) the tip position of the endoscope 20 based on the position of the surgical instrument 300 reflected in the endoscopic image. Therefore, the image range of the virtual endoscopic image G2 with the tip position of the endoscope 20 where the image sensor is placed as the viewpoint can be adjusted using the endoscopic image. As a result, even if the derivation accuracy of the position and orientation (angle) of the surgical instrument 300 is insufficient, the endoscopic image of the endoscope 20 is used to improve the accuracy of the position of the surgical instrument 300 in the volume rendering image G1 and the generation accuracy of the virtual endoscopic image G2.

The surgical instrument information processing unit 165 may correct the position of the surgical instrument 300 in the volume rendering image G1 and the tip position of the endoscope 20, which is the viewpoint of the virtual endoscopic image G2, based on the position of the surgical instrument 300 in the endoscopic image of the endoscope 20, for example, in accordance with the methods of Reference Patent Document 2 and Reference Non-Patent Documents 6 and 7 below.
(Reference Patent Document 2: Japanese Patent No. 4869189)
(Reference non-patent literature 6: Toshiya Nakaguchi and 5 others, "Implementation of an automatic magnification tracking system for laparoscopic surgery", [online], Japan Society for Medical and Biological Engineering, Vol. 43, No. 4, 2005, pp. 685-693, [searched on August 9, 2019], Internet <URL: https://www.jstage.jst.go.jp/article/jsmbe/43/4/43_4_685/_pdf/-char/en>)
(Reference non-patent literature 7: Kyojiro Nanbu and 3 others, "Minimally invasive advanced surgery support system using endoscopes, etc." [online], Toshiba Review, Vol. 56, No. 9, 2001, [searched on August 9, 2019], Internet <URL: https://www.toshiba.co.jp/tech/review/2001/09/56_09pdf/a09.pdf#search=%27%E3%80%8C%E5%86%85%E8%A6%96%E9%8F%A1%E3%81%AA%E3%81%A9%E3%81%A B%E3%82%88%E3%82%8B%E4%BD%8E%E4%BE%B5%E8%A5%B2%E9%AB%98%E5%BA%A6%E6%89%8B%E8%A1%93%E6%94%AF%E6%8F%B4%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%80%8D%E5%8D%97%E9%83%A8+%E6%9D%B1%E8%8A%9D%E3%83%AC%E3%83%93%E3%83%A5%E3%83%BC2001%27＞）

In addition, various sensors may measure (actually measure) the position of the port, the position of the surgical instrument 300, and the angle of the surgical instrument 300 during the actual surgery. For example, a three-dimensional position sensor or an angle sensor may be attached to the surgical instrument 300 or the trocar 60 to obtain the actual measurement value of the three-dimensional position sensor or the angle sensor. The surgical instrument information processing unit 165 may obtain the actual measurement value obtained by the various sensors. The display control unit 167 may display the information of the actual measurement value. The surgeon may check the information of the displayed actual measurement value and manually adjust the position, orientation, and virtual port position of the virtual surgical instrument 300 via the UI 120. Alternatively, the processing unit 160 may automatically adjust the position, orientation, and virtual port position of the virtual surgical instrument 300 based on the above-mentioned actual measurement value so that it matches, for example, the position, orientation, and real port position of the actually measured surgical instrument 300. In this case, a volume rendering image G1 or a virtual endoscopic image G2 may be generated and displayed based on the adjusted virtual port position. This allows the medical image processing device 100 to match the position, orientation, and port of the surgical instrument 300 based on the virtual calculation results to the actual state during surgery, further improving the accuracy of intraoperative navigation. In addition, only when highly accurate information is required (for example, when the surgical instrument 300 is located near an important target), the actual port position may be actually measured using a three-dimensional position sensor. This makes it possible to reduce the frequency of actual measurement to reduce the processing load or to delay the timing of actual measurement. In addition, the accuracy of the three-dimensional position sensor and angle sensor may be low. The surgical instrument information processing unit 165 may adjust the position, orientation, and virtual port position of the virtual surgical instrument 300 according to the accuracy of the measured actual measurement value.

The position of the actual port position may also be partially measured. Information obtained by partial measurement may include, for example, only the X coordinate of the three-dimensional position, information obtained from a camera in the operating room, and the distance from a specific land port measured with a tape measure. Furthermore, the accuracy of the partially measured actual measurement information may be low. The surgical instrument information processing unit 165 may adjust the position, orientation, and virtual port position of the virtual surgical instrument 300 according to the accuracy and degree of freedom of the measured actual measurement value.

In addition, similar to the target 40, the area or position of a dangerous area (e.g., blood vessels that should not be cut, other organs that do not include the target 40) that the surgeon should pay attention to during surgery may be set. The processing unit 160 may perform the same processing on the dangerous area as that on the target 40. For example, the processing unit 160 may set a dangerous area in the volume data, and superimpose information based on the distance between the surgical instrument 300 and the dangerous area on an image in which the 3D data of the subject is visualized. In this case, although it is difficult for the surgeon to grasp the sense of depth in, for example, a two-dimensional endoscopic image, it becomes easier for the surgeon to grasp the position of the surgical instrument 300 and the positional relationship between the tip position of the surgical instrument 300 and the dangerous area. Therefore, for example, it is possible to grasp how close the surgeon is to the dangerous area.

Although it has been shown that the surgical instrument 300 and the port are not provided with a three-dimensional position sensor or marker for identifying the position, the three-dimensional position sensor or marker may be provided as an auxiliary. For example, a three-dimensional position detection sensor or marker may be attached to the trocar 60, and the surgical instrument 300 information processing unit 165 may acquire the port position and the position and direction of the surgical instrument 300 using the three-dimensional position detection sensor or marker. The method of acquiring the port position and the direction of the surgical instrument 300 using the marker may be the same as the method shown in Non-Patent Document 1. As a result, when the derivation accuracy of the position and direction of the surgical instrument 300 and the tip position and tip direction of the surgical instrument 300 is low, the medical image processing device 100 can auxiliary acquire information on the port position and the position and direction of the surgical instrument 300 using the three-dimensional position detection sensor or marker. In addition, by arranging the three-dimensional position sensor or marker on the trocar 60, for example, the base part of the forceps 30 becomes easier to see, making it easier for the surgeon to perform surgery.

In addition, although an example has been given of converting values in the port coordinate system based on the detection values of the angle sensor into values in the subject coordinate system, this is not limiting. For example, the surgical instrument information processing unit 165 may convert values in the port coordinate system based on the detection values of the angle sensor into values in a coordinate system (orthogonal coordinate system) based on the bed on which the subject is placed. In addition, the processing unit 160 may generate images and derive distance information using the tilt angle of the surgical instrument 300 relative to the direction of gravity, or may calculate the orientation of the surgical instrument 300 in the subject coordinate system based on the tilt angle of the surgical instrument 300, and generate images and derive distance information based on the orientation of the surgical instrument 300 in the subject coordinate system.

In addition, although an example has been given in which angle information is derived for both the endoscope 20 and the forceps 30, the derivation of the angle information for the forceps 30 may be omitted. This is because the surgeon can determine in which direction the forceps 30 is facing from the position and orientation of the forceps 30 in the endoscopic image captured by the endoscope 20 or the virtual endoscopic image G2. In other words, the angle of the forceps 30 can be simulated from the relative position and relative orientation of the forceps 30 to the endoscopic image captured by the endoscope 20 or the virtual endoscopic image G2. Therefore, even if there is no information on the orientation of the forceps 30, the surgeon can determine the positional relationship between the tip position of the forceps 30 and the position of the target 40.

The medical image processing device 100 may include at least a processor 140 and a memory 150. The acquisition unit 110, the UI 120, and the display 130 may be external to the medical image processing device 100.

Also, an example has been given in which the volume data as the captured CT image is transmitted from the CT device 200 to the medical image processing device 100. Alternatively, the volume data may be transmitted to a server on a network (e.g., an image data server (PACS) (not shown)) etc. so that it is temporarily accumulated and stored. In this case, the acquisition unit 110 of the medical image processing device 100 may acquire the volume data from the server etc. via a wired or wireless line when necessary, or may acquire the volume data via any storage medium (not shown).

Also, an example has been given in which volume data as a captured CT image is transmitted from the CT device 200 to the medical image processing device 100 via the acquisition unit 110. This also includes cases in which the CT device 200 and the medical image processing device 100 are essentially combined into a single product. It also includes cases in which the medical image processing device 100 is used as a console for the CT device 200.

Although the above example shows an example in which an image is captured by the CT device 200 and volume data including information about the inside of the subject is generated, images may be captured by other devices and volume data may be generated. Examples of other devices include an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, an angiography device, or other modality devices. The PET device may also be used in combination with other modality devices.

It can also be expressed as a medical image processing method in which the operations in the medical image processing device 100 are specified. It can also be expressed as a program for causing a computer to execute each step of the medical image processing method.

(Summary of the above embodiment)
One aspect of the above embodiment is a medical image processing device 100 (an example of an endoscopic surgery support device) that supports endoscopic surgery, and includes a processing unit 160. The processing unit 160 acquires 3D data of a subject (for example, volume data in an untensioned state, volume data in a virtual pneumoperitoneum state, and a three-dimensional model). The processing unit 160 sets a virtual port on a body surface 70 of the subject in the 3D data. The processing unit 160 acquires an insertion distance ID of a surgical instrument 300 inserted into the subject from a real port corresponding to the virtual port. The processing unit 160 superimposes information indicating the surgical instrument 300 on a first image that visualizes 3D data based on at least the position of the virtual port and the insertion distance ID of the surgical instrument 300, and displays the superimposed information on the display 130 (an example of a display unit). The real port corresponding to the virtual port may be, for example, a real port that is installed as planned with respect to the virtual port, or a real port that is installed at a position shifted from the planned position of the virtual port.

As a result, the medical image processing device 100 does not need to install a high-precision three-dimensional position sensor in the operating room to derive the position of the surgical instrument 300, and the system configuration for surgical navigation can be simplified. This three-dimensional position sensor corresponds to the external camera installed on the ceiling of the operating room in Non-Patent Document 1, for example. Furthermore, since the three-dimensional position sensor is no longer necessary, there is no operating room coordinate system in which the three-dimensional position sensor is installed. This eliminates the need for time-consuming measurement of the positional relationship between the subject coordinate system and the operating room coordinate system before surgery, and eliminates the need for a process of registering the subject coordinate system and the operating room coordinate system. Furthermore, it also eliminates the need for a process of re-registration according to the patient's movements during surgery.

The 3D data may also include 3D data of a virtual pneumoperitoneum state in which a pneumoperitoneum simulation has been performed. This eliminates the need for the medical image processing device 100 to perform a process of registering the subject coordinate system and the operating room coordinate system before surgery, even for subjects who require pneumoperitoneum in actual surgery. In addition, it also eliminates the need for a process of re-registration during surgery in response to the patient's movements.

The processing unit 160 may also acquire the insertion direction of the surgical instrument 300 inside the subject, and display information indicating the surgical instrument 300 based on the insertion direction of the surgical instrument 300. This allows the medical image processing device 100 to display the direction (orientation) of the surgical instrument 300 in the first image. This allows the surgeon to intuitively grasp, for example, the orientation of the surgical instrument 300 relative to the field of view of the endoscope 20. Furthermore, the accuracy of surgical navigation can be improved.

The processing unit 160 may also calculate the insertion direction of the surgical instrument 300 inside the subject, and display information indicating the surgical instrument 300 based on the insertion direction of the surgical instrument 300. This allows the medical image processing device 100 to display the calculated and estimated direction (orientation) of the surgical instrument 300 in the first image. Therefore, the surgeon can intuitively grasp, for example, the orientation of the surgical instrument 300 relative to the field of view of the endoscope 20. This also improves the accuracy of surgical navigation. Furthermore, an angle sensor is no longer required, and the system configuration in surgical navigation can be simplified. Furthermore, since the angle sensor is no longer required, the process of registering the angles in the subject coordinate system and the operating room coordinate system before surgery is no longer necessary. Furthermore, the process of re-registration according to the patient's movement during surgery is no longer necessary.

In addition, there may be two or more surgical instruments 300, at least one of which may include an endoscope 20. There may be two or more virtual ports, at least one of which may include a virtual camera port into which the endoscope 20 is inserted on the body surface 70 of the subject. The processing unit 160 may acquire an insertion distance ID1 of the endoscope 20 inserted into the subject from a real camera port corresponding to the virtual camera port. The processing unit 160 may calculate the tip position 21 of the endoscope 20 based on at least the position of the virtual camera port and the insertion distance ID1 of the endoscope 20. The first image may be a virtual endoscopic image G2 with the tip position 21 of the endoscope 20 as the viewpoint. As a result, the medical image processing device 100 can derive the tip position of the endoscope 20 with a simple system configuration and a small amount of calculation, and can visualize the virtual endoscopic image G2 based on this position. This can improve the accuracy of surgical navigation.

The processing unit 160 may also acquire a real endoscopic image, which is an image captured by the endoscope 20. The processing unit 160 may adjust the calculated tip position of the endoscope 20 or the position of the surgical instrument 300 in the virtual endoscopic image G2 based on the image of the surgical instrument 300 included in the real endoscopic image. The image of the surgical instrument 300 may indicate relative information of the surgical instrument 300 in the real endoscopic image (e.g., the position in the real endoscopic image). This allows the medical image processing device 100 to adjust the tip position 21 of the endoscope 20 to change the field of view of the virtual endoscope, or adjust the position of the surgical instrument 300 shown in the virtual endoscopic image G2, so that the contents of the real endoscopic image and the virtual endoscopic image G2 become closer (e.g., match). Thus, the surgeon can grasp the position of the surgical instrument 300 closer to the real space.

The processing unit 160 may also set the target 40 in the 3D data and display information based on the distance (e.g., distance d1) between the surgical instrument 300 and the target 40 superimposed on the first image. This allows the surgeon to perform various procedures on the target 40 while grasping the distance from the surgical instrument 300 that the surgeon can operate during surgery to the target 40.

The processing unit 160 may also set risk areas in the 3D data and display information based on the distance between the surgical instrument 300 and the risk areas superimposed on the first image. This allows the surgeon to grasp the distance from the surgical instrument 300 that the surgeon can operate during surgery to the risk areas. Therefore, the surgeon can perform various treatments on the target 40 while avoiding the risk areas, for example.

The processing unit 160 may also acquire measurement information (e.g., measurement position) of the real port, and display information indicating the surgical instrument 300 superimposed on the first image based on the measurement information of the real port and the position of the virtual port corresponding to the real port. As a result, even if there is a discrepancy between the measurement position of the real port and the set position of the virtual port, the medical image processing device 100 can provide a first image based on the actual position of the real port, for example, by adjusting the position of the virtual port so as to reduce this discrepancy.

One aspect of the above embodiment is a method for supporting laparoscopic surgery, comprising the steps of acquiring 3D data of a subject, setting a virtual port on the body surface of the subject in the 3D data, acquiring an insertion distance of a surgical instrument 300 inserted into the subject from a real port corresponding to the virtual port, and displaying information indicating the surgical instrument 300 on a display unit on a first image that visualizes the 3D data based on at least the position of the virtual port and the insertion distance of the surgical instrument 300.

One aspect of this embodiment is a program for causing a computer to execute the above-mentioned laparoscopic surgery support method.

The present disclosure is useful for endoscopic surgery support devices, endoscopic surgery support methods, and programs that can simplify the system configuration and easily perform surgical navigation.

10 liver 15 bone 21 endoscope tip position 22 endoscope tip direction 23 imaging range 30, 31, 32 forceps 40 target 60 trocar 70 body surface 100 medical image processing device 110 acquisition unit 120 UI
130 Display 140 Processor 150 Memory 160 Processing section 161 Area processing section 162 Transformation processing section 163 Model setting section 164 Port information processing section 165 Surgical instrument information processing section 166 Image generating section 167 Display control section 200 CT device 300 Surgical instrument PT Port PT1 Camera port PT2 Forceps port ID, ID1, ID2, ID21, ID22 Insertion distance

Claims (11)

An endoscopic surgery support device for supporting endoscopic surgery,
A processing unit is provided,
The processing unit includes:
Acquire 3D data of the subject and information on the shape of the surgical instrument ;
setting a virtual port on a body surface of the subject in the 3D data;
acquiring information indicating an insertion distance of the surgical instrument inserted into the subject from a real port corresponding to the virtual port;
and displaying, on a display unit, information indicating the surgical instrument and a tip position and tip direction of the surgical instrument superimposed on a first image in which the 3D data is visualized based on at least the position of the virtual port, the detected insertion distance of the surgical instrument, and the shape of the surgical instrument.
Laparoscopic surgery support device. The 3D data includes 3D data of a virtual pneumoperitoneum state in which a pneumoperitoneum simulation has been performed.
The endoscopic surgery support device according to claim 1 . The processing unit includes:
acquiring information indicating an insertion direction of the surgical instrument inside the subject;
displaying information indicating the surgical instrument based on the detected insertion direction of the surgical instrument;
The endoscopic surgery support device according to claim 1 or 2. The processing unit includes:
calculating an insertion direction of the surgical instrument inside the subject;
displaying information indicating the surgical instrument based on an insertion direction of the surgical instrument;
The endoscopic surgery support device according to claim 1 or 2. the surgical instruments are two or more, at least one of which includes an endoscope;
There are two or more virtual ports, at least one of which includes a virtual camera port through which the endoscope is inserted on a body surface of the subject;
The processing unit includes:
acquiring an insertion distance of the endoscope inserted into the subject from a real camera port corresponding to the virtual camera port;
Calculating a tip position of the endoscope based on at least the position of the virtual camera port and the insertion distance of the endoscope;
the first image is a virtual endoscopic image having a viewpoint at the tip position of the endoscope;
The endoscopic surgery support device according to any one of claims 1 to 4. The processing unit includes:
An actual endoscopic image is acquired, which is an image captured by the endoscope;
adjusting the calculated tip position of the endoscope or the position of the surgical instrument in the virtual endoscopic image based on the image of the surgical instrument included in the actual endoscopic image;
The endoscopic surgery support device according to claim 5 . The processing unit includes:
Setting a target in the 3D data;
and displaying information based on a distance between the surgical instrument and the target on the first image in a superimposed manner.
The endoscopic surgery support device according to any one of claims 1 to 6. The processing unit includes:
A dangerous area is set in the 3D data,
and displaying information indicating a distance between the surgical instrument and the dangerous site on the first image in a superimposed manner.
The endoscopic surgery support device according to any one of claims 1 to 7. The processing unit includes:
Acquire measurement information on the actual port;
displaying information indicating the surgical instrument on the first image in a superimposed manner based on measurement information of the real port and a position of the virtual port corresponding to the real port;
The endoscopic surgery support device according to any one of claims 1 to 8. A processor acquires 3D data of a subject and information on the shape of a surgical tool ;
The processor sets a virtual port on a body surface of the subject in the 3D data;
acquiring information indicating an insertion distance of the surgical instrument inserted into the subject from a real port corresponding to the virtual port by the processor;
The processor causes a display unit to display information indicating the surgical instrument and a tip position and tip orientation of the surgical instrument superimposed on a first image in which the 3D data is visualized based on at least the position of the virtual port, the detected insertion distance of the surgical instrument, and the shape of the surgical instrument;
The present invention relates to a method for supporting laparoscopic surgery. A program for causing a computer to execute the laparoscopic surgery support method according to claim 10.

Images