JP7444569B2 - Arthroscopic surgery support device, arthroscopic surgery support method, and program - Google Patents

Description

The present disclosure relates to an arthroscopic surgery support device, an arthroscopic surgery support method, and a program that support arthroscopic surgery.

Conventionally, in surgical simulation, medical image display devices have been known that perform processing on a 3D medical image of a subject in response to operational inputs to a display screen, and display the processed 3D medical image on a display screen. (See Patent Document 1). The processing for the three-dimensional medical image here includes transformation processing.

JP2014-54358 Publication

The medical image display device of Patent Document 1 has room for improvement in visualization of an observation target (target) in a subject in intraoperative navigation and preoperative simulation.

The present disclosure has been made in view of the above circumstances, and provides an arthroscopic surgery support device, an arthroscopic surgery support method, and a program that can improve the visualization of an observation target in a subject.

One aspect of the present disclosure is an arthroscopic surgery support device that supports arthroscopic surgery, including an acquisition unit and a processing unit, the acquisition unit having a function of acquiring volume data of a subject, The processing unit sets a model representing the tissue included in the volume data, sets a first field of view for visualizing the volume data, and a position of a target in the model, and transforms the model. obtain first operation information for deforming the model, calculate changes in the position and orientation of the target before and after deforming the model based on the first operation information, and calculate the calculated position and orientation of the target. the second field of view, such that the position and orientation of the target after the change with respect to the first field of view and the position and orientation of the target before the change with respect to the second field of view are equivalent. The present invention is an arthroscopic surgery support device having a function of calculating the volume data and visualizing the volume data based on the second visual field.

According to the present disclosure, it is possible to improve the ease of observation environment for a target in a subject.

A block diagram showing an example of the hardware configuration of a medical image processing device in the first embodiment Block diagram showing an example of functional configuration of a medical image processing device Diagram illustrating an overview of processing by a medical image processing device Diagram showing an example of model generation Diagram showing a modified example of the model Schematic diagram showing an example of the positional relationship between the target and the actual camera position Schematic diagram showing actual camera positions and organ deformations Schematic diagram showing an example of virtual camera position Diagram showing an example of displaying an image in which the body surface of organs that do not contain targets or organs that contain targets are excluded from rendering. Diagram showing an example display of an image in which an organ that includes a target and a moved organ that does not include the target are rendered. Flowchart showing an example of the operation of a medical image processing device

Embodiments of the present disclosure will be described below with reference to the drawings.

(How one aspect of the present disclosure was obtained)
In preoperative simulation and intraoperative navigation, an object to be observed (target), which is a part of the subject, may be focused on and observed in detail. For example, in arthroscopic surgery, if a target is located on the back side of an organ when viewed from the camera position, it is assumed that the organ is deformed so that the target can be confirmed from the camera position. In this case, when the medical image display device of Patent Document 1 is used, the organ portion in the volume data is transformed and a three-dimensional medical image is displayed on the display by volume rendering or the like. Therefore, the volume data must be transformed, resulting in an enormous amount of calculation. Furthermore, it is necessary to appropriately set stiffness for each voxel in the volume data, and to take into account slippage at organ boundaries, body fluid flow, and air. Furthermore, when an organ is repeatedly deformed little by little in anticipation of a surgical process, it is necessary to maintain consistency of volume data related to the deformation process. Therefore, it becomes necessary to manually make fine adjustments to the volume data and the region of interest. Therefore, for example, it is difficult to simulate in real time that an operator manipulates an organ using forceps in arthroscopic surgery.

Furthermore, since the rendered image that has been transformed and rendered through non-rigid transformation processing of volume data is checked as is in preoperative simulation and intraoperative navigation, the accuracy of the transformation processing is required to be highly accurate. In particular, it is required that organ boundaries do not collapse.

In addition, when deforming an organ, in order to make it easier to observe the organ including the target, other organs (for example, the small intestine in the abdominal region) around this organ are removed in detail (segmentation). It is possible to visualize the target, but this is a roundabout idea. It is also possible to perform a simulation in which only the organ including the target is deformed, but in this case, there is a possibility that a plurality of organs may overlap in the virtual space, and the visualization accuracy of the object may deteriorate.

Further, when surface data is transformed instead of volume data, processing speed related to transformation can be improved, but the accuracy of surface rendering may be insufficient. This is because if the segmentation of each organ to generate surface data is inaccurate, for example, if there are a large number of small blood vessels, it is difficult to accurately generate the surface of the small blood vessels or intricate organs. For example, blood vessels that connect to the liver and blood vessels that do not connect to the segment may be visualized more obscurely.

In the following embodiments, an arthroscopic surgery support device, an arthroscopic surgery support method, and a program that can improve the visualization of an observation target in a subject will be described.

(First embodiment)
FIG. 1 is a block diagram showing a configuration example of a medical image processing apparatus 100 according to the first embodiment. The medical image processing apparatus 100 includes an acquisition unit 110, a UI 120, a display 130, a processor 140, and a memory 150. The medical image processing apparatus 100 supports arthroscopic surgery (including robotic surgery) by image processing. The medical image processing apparatus 100 performs, for example, intraoperative navigation to support the surgeon during surgery, or preoperative simulation to help the surgeon formulate a surgical plan before surgery.

A CT device 200 is connected to the medical image processing device 100 . The medical image processing apparatus 100 acquires volume data from the CT apparatus 200 and processes the acquired volume data. The medical image processing apparatus 100 may be configured by a PC and software installed in the PC.

The CT apparatus 200 irradiates a subject with X-rays and captures an image (CT image) by utilizing differences in the absorption of X-rays by tissues within the body. The subject may include a living body, a human body, an animal, and the like. The CT apparatus 200 generates volume data that includes information about any location inside the subject. The CT apparatus 200 transmits volume data as a CT image to the medical image processing apparatus 100 via a wired line or a wireless line. When taking a CT image, imaging conditions regarding CT imaging and contrast conditions regarding administration of a contrast agent may be taken into consideration.

The acquisition unit 110 in the medical image processing apparatus 100 includes, for example, a communication port, an external device connection port, and a connection port to an embedded device, and acquires volume data obtained by the CT apparatus 200. The acquired volume data may be immediately sent to the processor 140 for various processing, or may be stored in the memory 150 and then sent to the processor 140 for various processing when necessary. Further, the volume data may be acquired via a recording medium or a recording medium. Further, the volume data may be obtained in the form of intermediate data, compressed data, or sinogram. Further, the volume data may be obtained from information from a sensor device attached to the medical image processing apparatus 100. In this way, the acquisition unit 110 has a function of acquiring various data such as volume data.

UI 120 may include, for example, a touch panel, a pointing device, a keyboard, or a microphone. The UI 120 accepts any input operation from the user of the medical image processing apparatus 100. Users may include surgeons, doctors, nurses, radiology technicians, students, and the like.

The UI 120 accepts various operations. For example, operations such as specifying a region of interest (ROI) and setting brightness conditions in volume data or images based on volume data (for example, three-dimensional images and two-dimensional images described later) are accepted. Regions of interest may include regions of various tissues (eg, blood vessels, bronchi, organs, organs, bones, brain). Tissues may include diseased tissue, normal tissue, tumor tissue, and the like. Further, the UI 120 may accept various operations (for example, transformation operations) on a model generated based on volume data.

Display 130 may include, for example, an LCD, and displays various information. The various information may include three-dimensional images and two-dimensional images obtained from volume data. Three-dimensional images may include volume rendering images, surface rendering images, virtual endoscopic images, virtual ultrasound images, CPR images, and the like. Volume rendered images may include RaySum images, MIP images, MinIP images, average images, raycast images, and the like. Two-dimensional images may include axial images, sagittal images, coronal images, MPR images, and the like.

The memory 150 includes various types of ROM and RAM primary storage devices. The memory 150 may include a secondary storage device such as an HDD or an SSD. The memory 150 may include a tertiary storage device such as a USB memory or an SD card. Memory 150 stores various information and programs. The various information may include volume data acquired by the acquisition unit 110, images generated by the processor 140, setting information set by the processor 140, and various programs. Memory 150 is an example of a non-transitory recording medium in which programs are recorded.

Processor 140 may include a CPU, DSP, or GPU. The processor 140 functions as a processing unit 160 that performs various processes and controls by executing a medical image processing program stored in the memory 150.

FIG. 2 is a block diagram showing an example of the functional configuration of the processing unit 160. The processing section 160 includes a region processing section 161, a transformation processing section 162, a model setting section 163, an information setting section 164, an image generation section 165, and a display control section 166. Note that each unit included in the processing unit 160 may be realized as different functions by one piece of hardware, or may be realized as different functions by a plurality of pieces of hardware. Further, each unit included in the processing unit 160 may be realized by a dedicated hardware component.

The region processing unit 161 acquires volume data of the subject via the acquisition unit 110, for example. The area processing unit 161 extracts an arbitrary area included in the volume data. The region processing unit 161 may automatically specify a region of interest and extract the region of interest, for example, based on the pixel values of the volume data. The region processing unit 161 may manually specify a region of interest and set the region of interest, for example, via the UI 120. The region of interest may include areas such as organs, bones, blood vessels, diseased areas (eg, diseased tissue or tumor tissue), and the like.

Furthermore, the region of interest may include not only a single tissue but also surrounding tissues, and may be segmented and extracted. For example, if the organ as the region of interest is the liver, not only the liver itself, but also the blood vessels that connect to the liver or run within or around the liver (e.g., hepatic artery, hepatic vein, portal vein), and the bones around the liver ( For example, the spine, ribs) may be included. Further, the liver body, the blood vessels in or around the liver, and the bones around the liver may be obtained by segmentation as separate tissues.

The model setting unit 163 sets a model of the organization. The model may be established based on the region of interest and volume data. The model represents the tissue represented in the volume data in a manner that is simpler than the volume data. Therefore, the amount of data of the model is smaller than the amount of volume data corresponding to the model. The model is subject to transformation processing and transformation operations. The model may be, for example, a bone deformation model. In this case, the model assumes a skeleton made of simple finite elements, and the bones are deformed by moving the vertices of the finite elements. Tissue deformation can be expressed by following bone deformation. The model may include an organ model that simulates an organ (eg, liver). The model may have a shape similar to a simple polygon (eg, a triangle) or may have other shapes. The model may be, for example, a contour line of volume data representing an organ. The model may be a three-dimensional model or a two-dimensional model. Note that bones may be expressed not by deformation of a model but by deformation of volume data. This is because bones have a small degree of freedom of deformation and can be expressed by affine transformation of volume data.

The model setting unit 163 may obtain the model by generating the model based on the volume data. Further, a plurality of model templates may be determined in advance and held in the memory 150 or an external server. The model setting unit 163 may acquire a model by acquiring one model template from a plurality of model templates prepared in advance from the memory 150 or an external server in accordance with the volume data.

The information setting unit 164 inputs or obtains various information and sets it. For example, the information setting unit 164 may set information (camera information) regarding an endoscope (camera) used in arthroscopic surgery. The camera is inserted through the camera port and images the inside of the subject. The camera information may include information such as the camera position with respect to the subject, camera direction, camera view angle, and the like. The camera orientation is the direction in which the camera captures an image, and may coincide with the direction of the optical axis of the camera. The imaging range of the camera may be determined according to the camera position, camera orientation, and camera angle of view. In other words, the camera position can also be said to be the operator's viewpoint for observing the subject. The camera direction can also be said to be the direction of the operator's line of sight starting from the viewpoint. The camera angle of view can also be said to be the operator's viewing angle based on the viewpoint. The imaging range can also be said to be the operator's visual field. The field of view may be determined depending on the viewpoint, the direction of the line of sight, and the viewing angle. The field of view serves as a reference for visualizing volume data. Note that the camera position, camera direction, camera angle of view, and imaging range here are information assuming the camera actually placed, so the actual camera position (real viewpoint), real camera orientation (actual line of sight direction) , also referred to as an actual camera field of view (actual viewing angle), and an actual imaging range (actual field of view). The actual camera information here is also referred to as actual camera information. Note that in intraoperative navigation, the information setting unit 164 may acquire and set camera information detected by an arbitrary sensor.

The information setting unit 164 sets forceps information regarding the forceps (real forceps) operated by the operator. The forceps information may include information on the position and orientation of the forceps.

The information setting unit 164 may set information on the positions of various ports to be drilled into the subject. The various ports include, for example, a forceps port into which forceps are inserted and a camera port into which a camera is inserted. Note that since the camera is inserted through the camera port, the degree of freedom of movement is small. Furthermore, since the forceps are inserted through the forceps port, the degree of freedom of movement is small.

The information setting unit 164 simulates a target in an organization and sets target information regarding a target in a model corresponding to the organization. The target information may include information on the position and orientation of the target. The target is set within or near an arbitrary tissue (for example, an organ). Once the target position is determined, the direction of the target (target orientation) is determined. The target orientation is a parameter used to determine the change in the orientation of the target during deformation, and the initial value can be set arbitrarily. The target location can be tangential to the location of the target within the tissue or at a point on the contour of the tissue that is the shortest distance from the target location within the tissue. The information setting unit 164 may specify the target position via U120. Furthermore, the memory 150 may hold the position of a target (for example, an affected area) that has been treated on a subject in the past. The information setting unit 164 may acquire the target position from the memory 150 and set it. The information setting unit 164 may calculate and set the target orientation based on the target position in the tissue.

The deformation processing unit 162 performs processing related to deformation of the subject to be operated on. Tissues such as organs within a subject can be subjected to various deformation operations by a user to imitate various treatments performed by a surgeon during surgery. The deformation operation may include lifting the organ, flipping it over, cutting it, and the like. Correspondingly, the deformation processing unit 162 may deform a model corresponding to a tissue such as an organ within the subject as a process related to deformation. For example, organs may be pulled, pushed, or cut by forceps (eg, grasping forceps, dissection forceps, electrocautery), and this may be simulated by deforming the model. These deformations are used, for example, to make a target that is invisible or difficult to see by the surgeon because it is located inside or behind an organ visible (capable of being imaged) from the surgeon's viewpoint (corresponding to the camera position). can be carried out. That is, this can be carried out to make a target that is located inside or on the back side of the model and thus invisible or difficult to see to the operator visible from the operator's viewpoint. For example, the deformation creates a situation in which there are no obstacles (such as bones) between the operator's viewpoint and the target. As the model deforms, the targets in the model may also deform.

Deformation by the deformation operation is performed on the model, and may be performed by large deformation simulation using the finite element method. For example, the movement of an organ due to a change in body position may be simulated. In this case, the elastic force applied to the contact point of the organ or lesion, the rigidity of the organ or lesion, and other physical characteristics may be taken into consideration. The amount of calculation required for deforming a model is reduced compared to deforming processing for volume data. This is because the number of elements in the deformation simulation is reduced.

In the model settings, determine which part of the organ to be modeled is connected to other parts of the subject, the root part of the blood vessel connected to the organ (the part that connects to the organ) is difficult to move, and the tip part of the blood vessel is Information such as "easy to move", "bones are difficult to move", "other tissues move avoiding bones", etc. can be set. Information on which parts of the tissue are easy to move and which parts are difficult to move may be individually set in detail, or may be set as uniform ease of movement, for example. Furthermore, there may be one or more directions in which the tissue bends due to deformation. For example, the deformation processing unit 162 provides information that if the target exists on the back side of the vein when viewed from the camera position, it is difficult to deform the model so that the target can be visually recognized, and arthroscopic surgery is difficult. can.

The model may be displayed on display 130. The transformation processing unit 162 may transform the model by acquiring information on transformation operations for the model via the UI 120. In this case, the amount of deformation of each part of the model may be determined according to the amount of deformation operation for the model. The information on the deformation operation may include information on pressing the organ with forceps, for example. The information for pushing the organ may include information such as the position at which the organ is pushed, the force at which the organ is pushed, and the direction in which the organ is pushed. The force pushing the organ acts as a factor that deforms the movable point of the organ in the direction of pushing the organ.

The deformation processing unit 162 may limit the deformation of the model of the organ including the target, depending on the positional relationship between the organ including the target and the organ not including the target. For example, if there is a bone that is difficult to deform in a part of the periphery of the organ, the bone may be assumed not to deform, and the direction in which the bone exists may be restricted so that the organ does not deform. Restriction information regarding restrictions on model deformation (for example, information on positions where deformation is restricted in the model) may be set as information regarding the model and held in the memory 150.

Note that the shape of the model may be inaccurate compared to the actual tissue. For example, the contours of organs may be imprecise. Furthermore, the degree of deformation of the model may be inaccurate compared to the deformation of the actual tissue; for example, the outline and shape of an organ before and after deformation may be inaccurate. For example, an inaccurate contour at the edge of the liver may cause part of the stomach to extend into the liver. This is because even if the deformation of the model has some error compared to the actual deformation, it is sufficient that the target can be easily recognized in the visualization of the volume data corresponding to the model.

The deformation processing unit 162 calculates changes in target position and target orientation before and after deforming the model. In this case, the target position and target orientation after deformation may be calculated based on the target position and target orientation before deformation, taking into account the deformation of the model.

The transformation processing unit 162 calculates a virtual camera position, camera orientation, camera view angle, and imaging range for visualizing volume data. The camera position, camera direction, camera angle of view, and imaging range here are not information assuming an actually placed camera, but are information derived virtually, so the virtual camera position (virtual viewpoint), Also referred to as virtual camera direction (virtual line of sight direction), virtual camera view angle (virtual viewing angle), and virtual imaging range (virtual field of view). The virtual camera information here is also referred to as virtual camera information.

For example, the deformation processing unit 162 determines the virtual camera position and virtual camera orientation for visualizing the target based on the real camera position, the real camera orientation, and the target position and target orientation in the transformed (post-deformed) organ. Calculate. In this case, an angle A1 (see FIG. 3) formed between the target orientation of the deformed organ including the target and the actual camera orientation is calculated. Then, the virtual camera position is determined so that the angle A2 (see FIG. 3) formed between the target direction of the undeformed (before deformed) organ and the virtual camera direction is the same as the calculated angle A1.

The deformation processing unit 162 may change the position and orientation of the forceps according to the deformation of the model. In this case, the real forceps position and real forceps orientation may be changed to the virtual forceps position and virtual forceps orientation in the same manner as the change from the real camera position and real camera orientation to the virtual camera position and virtual camera orientation. For example, the virtual forceps position and the virtual forceps orientation may be calculated based on the real forceps position, the real forceps orientation, and the target position and target orientation in the deformed organ. In this case, the angle between the target orientation of the organ after deformation including the target and the real forceps orientation is calculated, and the angle between the target orientation of the organ before deformation and the virtual forceps orientation is the same as the calculated angle. As such, the virtual forceps position may be determined. Note that the actual forceps may be those intended for use in actual surgery. The virtual forceps may be forceps virtually derived by calculation.

Further, the deformation processing unit 162 may perform a pneumoperitoneum simulation in which the subject is virtually insufflated as a process related to deformation. A specific method for the pneumoperitoneum simulation may be a known method, for example, the method described in Reference Non-Patent Document 1. That is, the deformation processing unit 162 may perform a pneumoperitoneum simulation based on volume data in a non-pneumoperitoneum state to generate volume data in a virtual pneumoperitoneum state. Through the pneumoperitoneum simulation, the user can assume that the subject is in a pneumoperitoneum state and virtually observe the pneumoperitoneum state without actually performing pneumoperitoneum on the subject. Note that among the pneumoperitoneum states, the pneumoperitoneum state estimated by the pneumoperitoneum simulation may be referred to as a virtual pneumoperitoneum state, and the actual pneumoperitoneum state may be referred to as an actual pneumoperitoneum state.

(Reference Non-Patent Document 1) Takayuki Kitasaka, Kensaku Mori, Yuichiro Hayashi, Yasuhito Suenaga, Makoto Hashizume, and Jun-ichiro Toriwaki, “Virtual Pneumoperitoneum for Generating Virtual Laparoscopic Views Based on Volumetric Deformation”, MICCAI (Medical Image Computing and Computer- Assisted Intervention), 2004, P559-P567

The pneumoperitoneum simulation may be a large deformation simulation using the finite element method. In this case, the deformation processing unit 162 may segment the subject's body surface including subcutaneous fat and the subject's abdominal viscera. Then, the deformation processing unit 162 may model the body surface into a two-layer finite element of skin and body fat, and model the abdominal internal organs into a finite element. The deformation processing unit 162 may optionally segment, for example, lungs and bones, and add them to the model. Further, a gas region may be provided between the body surface and the abdominal internal organs, and the gas region (pneumoperitoneum space) may be expanded (inflated) in response to virtual gas injection.

The image generation unit 165 generates various images. The image generation unit 165 generates a three-dimensional image or a two-dimensional image based on at least a portion of the acquired volume data (for example, an area extracted from the volume data). The image generation unit 165 may generate a three-dimensional image or a two-dimensional image based on the volume data transformed by the transformation processing unit 162 (for example, volume data of a virtual pneumoperitoneum state). In this case, the image generation unit 165 may generate a three-dimensional image or a two-dimensional image by visualizing (for example, rendering) the volume data so as to represent the state viewed from the virtual camera position toward the virtual camera.

The display control unit 166 causes the display 130 to display various data, information, and images. The image includes an image generated by the image generation unit 165 (for example, a rendered image). Further, the display control unit 166 may adjust the brightness of the rendered image. The brightness adjustment may include, for example, adjusting at least one of window width (WW) and window level (WL).

FIG. 3 is a diagram illustrating an overview of processing by the medical image processing apparatus 100.

First, the acquisition unit 110 acquires volume data of the subject. The region processing unit 161 extracts the region of the organ TL included in the subject as necessary. The model setting unit 163 generates a model ML based on the region of the organ TL. The organ TL includes the target TG (see process A; the organ TL will be explained without distinguishing it from the model ML). This model ML is the model MLA before transformation. In the state before the model ML is deformed, when the organ TL corresponding to the model MLA is visualized as viewed from the real camera position 21A toward the real camera direction 22A, the target TG cannot be visually recognized.

The deformation processing unit 162 performs deformation simulation on the model MLA to obtain a deformed model MLB (see process B). In the deformation simulation, deformation information is obtained. For example, the deformation information may indicate the deformation of the bones of the model ML before and after the deformation. Then, the deformation processing unit 162 calculates the target position 41B and target orientation 42B after deformation. The target orientation 42 (42A, 42B) is a tangential direction at the target position 41 (41A, 41B) of the organ. In the state after the model ML is deformed, when the organ corresponding to the model MLB is visualized as viewed from the real camera position 21A toward the real camera direction 22A, the target TG below the organ TL can be visually recognized.

The information setting unit 164 acquires real camera information including the real camera position 21A and the real camera orientation 22A. The deformation processing unit 162 derives the positional relationship between the deformed target and the real camera based on the deformed target position 41B and target orientation 42B, as well as the real camera position 21A and real camera orientation 22A. The deformation processing unit 162 determines the virtual camera position 21B based on the positional relationship between the deformed target and the real camera, that is, based on the deformed target position 41B and target orientation 42B, as well as the real camera position 21A and real camera orientation 22A. and calculate the virtual camera orientation 22B (see process C)). In other words, the deformation processing unit 162 sets the virtual camera position 21B and the virtual camera orientation 22B so that the model MLA before deformation has the angle and distance at which the target TG would have been visible in the model MLB after the deformation. Calculate. This angle is the angle A1 formed by the real camera orientation 22A and the target orientation 42B after transformation, and the angle A2 formed between the virtual camera orientation 22B and the target orientation 42A before transformation, which are the same angle. The image generation unit 165 generates an image visualized using the virtual camera position 21B and the virtual camera orientation 22B based on the volume data before transformation. The display control unit 166 causes the display 130 to display the generated image. In other words, the original volume data before transformation is visualized based on the virtual camera information.

As a result, the medical image processing apparatus 100 moves from the virtual camera position 21B to the target TG while maintaining the angular relationship between the real camera and the target TG after deformation and the angular relationship between the virtual camera and the target TG before deformation. Organs containing TG are visible. Therefore, the user can fully observe the target TG. Therefore, compared to the case where, for example, the target TG is visualized and observed by transforming the volume data of the organ, the amount of calculation by the processor 140 and the user's effort can be reduced. Such visualization showing the approach to the target TG is sufficient for intraoperative navigation and preoperative simulation.

FIG. 4 is a diagram showing an example of generation of model ML.

The model ML shown in FIG. 4 is a liver bone deformation model. Liver 10 includes a portal vein 12 and a vein 14. The region processing unit 161 separates and extracts a region including the main body of the liver 10, the portal vein 12, and the vein 14 from the subject as the region of the liver 10. The main body of the liver 10 is connected to the portal vein 12 and vein 14. The portal vein 12 and vein 14 may be treated as fixed points outside the liver 10. This information is used, for example, when calculating bone deformation using the finite element method. Note that the main body of the liver 10, the portal vein 12, and the vein 14 may be extracted as separate regions, and these three regions may be combined to form the region of the liver 10.

FIG. 5 is a diagram showing a modification of the model ML. The transformation processing unit 162 receives a transformation operation via the UI 120, for example, and transforms the model ML. For example, when an operation is received via the UI 120 to move a part of the model MLA before deformation in the direction of arrow α (upper left direction in FIG. 5), the part of model MLA before deformation is deformed in the direction of arrow α in response to detection of this movement, and the Generate model MLB. Liver 10A shows a liver before deformation corresponding to model MLA before deformation. Liver 10B shows a liver corresponding to model MLB after deformation. Therefore, the outline of the liver 10 changes before and after the deformation. Note that here, the model ML of the liver 10 is deformed, and the volume data of the liver 10 is not deformed, so that the amount of calculations related to the deformation can be reduced.

FIG. 6 is a schematic diagram showing an example of the positional relationship between the target TG and the actual camera position 21A. Here, laparoscopic surgery is given as an example of arthroscopic surgery, but other arthroscopic surgery may be used. Note that FIGS. 6 to 10 illustrate that the contour of the model ML matches the contour of the liver 10. Furthermore, the model ML may or may not be displayed on the display 130. In FIGS. 6 to 10, model ML is exemplified as an organ model of the liver 10.

FIG. 6 shows the camera position (actual camera position 21A) and camera orientation (actual camera orientation 22A) before deformation, as well as the position and orientation of the forceps 30. In FIG. 6, when viewed from the real camera position 21A, the target TG in the model MLA corresponding to the liver 10 before deformation is hidden behind other parts of the liver 10. Therefore, even when looking at the real camera direction 22A from the real camera position 21A, the target TG cannot be confirmed. Note that the actual camera position 21A and the position of the forceps 30 do not match and are different.

FIG. 7 is a schematic diagram showing a modified example of the actual camera position 21A and organs.

The model MLA before deformation is deformed so that the target position 41 is visible from the camera position 21, that is, so that there are no obstacles between the camera position 21 and the target position 41. The model ML may be transformed in response to a transformation operation via the UI 120. In the modified model MLB, the target position 41B is visible from the real camera position 21A, that is, there is no obstacle between the real camera position 21A and the target position 41B. The obstacle here may be another location in the liver 10 that includes the target TG, or another tissue (eg, bone).

Further, in FIG. 7, as the model ML corresponding to the liver 10 is deformed, the target TG included in the model ML is deformed, that is, the target position 41 and target orientation 42 change. The deformation processing unit 162 may calculate movement information and rotation information of the target TG before and after deformation, based on deformation information related to the deformation of the model ML. For example, a user's surgical operation (for example, pulling an organ, flipping an organ, flipping an organ, or resecting an organ) may be indicated by movement or rotation of the target TG due to deformation in calculations. In other words, the deformation of an organ caused by a surgical operation may be expressed in a pseudo manner by a combination of movement and rotation using calculations. The deformation information of the model ML and the movement information and rotation information of the target TG may be held in the memory 150.

FIG. 8 is a schematic diagram showing an example of the virtual camera position 21B.

The deformation processing unit 162 acquires movement information and rotation information of the target TG. Then, the back calculation results of the movement information and rotation information of the target TG are applied to the model ML before deformation, and the virtual camera position 21B and virtual camera orientation 22B are calculated. In other words, the deformation processing unit 162 calculates the angle and distance relationship of the deformed target position 41B and target orientation 42B with respect to the real camera position 21A and real camera orientation 22A, and the undeformed target position with respect to the virtual camera position 21B and virtual camera orientation 22B. The position of the real camera position 21A and the real camera orientation 22A are changed to the virtual camera position 21B and the virtual camera position 22A so that the angle and distance relationship between the position 41A and the target orientation 42A are equal, that is, the angles A1 and A2 are equal. Change the camera orientation to 23B. In this case, the deformation processing unit 162 may change the position and orientation of the forceps 30 in the same way as changing the camera position and camera orientation. That is, the position and orientation of the actual forceps 30 (real forceps 30A) may be changed to the position and orientation of the virtual forceps 30 (virtual forceps 30B).

Next, an example of displaying images by the medical image processing apparatus 100 will be described.

As shown in FIG. 9, the display control unit 166, for example, hides organs that do not contain the target TG (for example, bones) and the surface of the organ that contains the target TG, displays internal blood vessels, and creates a rendered image. You may display it. Thereby, the medical image processing apparatus 100 can suppress the presence of an obstacle in front of the image (for example, a virtual endoscopic image) seen from the virtual camera position 21B to which the viewpoint has been moved, and The organs around the TG can be easily observed.

Further, the image generation unit 165 may render an organ that does not include the target TG (for example, a bone or another organ that may become an obstacle to surgery) separately from an organ that includes the target TG. As shown in FIG. 10, the display control unit 166 may combine and display a separately rendered image of an organ that does not include the target TG and an image of the organ that includes the target TG.

In this case, the deformation processing unit 162 may move the organ that does not include the separately rendered target TG as the virtual camera position 21B moves from the real camera position 21A. For example, in the same way as when deriving the virtual camera position 21B and the virtual camera orientation 22B, the movement information and rotation information of the target TG before and after deformation of the model ML are calculated backward to calculate the position and orientation of the organ that does not include the target TG. You may do so. In this case, the relationship between the angle and distance of the target position 41B and the target orientation 42B after deformation with respect to the position and orientation of the organ that does not include the target TG before movement, and the deformation with respect to the position and orientation of the organ that does not include the target TG after movement. The position and orientation of the organ that does not include the target TG before the movement is changed to the position and orientation of the organ that does not include the target TG after the movement so that the angle and distance relationship between the previous target position 41A and the target orientation 42A are equal. You can change the direction. FIG. 10 shows an example of an image rendered separately from the liver 10 by moving the bone 15 to become a bone 15B.

FIG. 11 is a flowchart showing an example of the operation of the medical image processing apparatus 100.

First, volume data of a subject (for example, a patient) is acquired (S11). A pneumoperitoneum simulation is performed (S12). Organ segmentation for extracting organs and bone segmentation for extracting bones are performed (S13). A model ML is obtained based on the volume data (S14). The forceps port, camera port, and target information (for example, target position 41 and target orientation 42) are acquired (S15). Information on the position (real forceps position) and orientation (real forceps orientation) of the real forceps and the position (real camera position 21A) and orientation (real camera orientation 22A) of the real camera is acquired (S16). Note that in S16, the information on the actual camera position 21A and the actual camera orientation 22A may be acquired by acquiring the actual imaging range (actual field of view).

Information about pressing the organ including the target TG with the forceps 30 is acquired (S17). The model ML is deformed based on the information about pressing the organ, and changes in the target position 41 and target orientation 42 before and after the deformation of the model ML are calculated (S18). Here, the target position 41B and target orientation 42B after deformation may be calculated. Based on changes in the target position 41 and target orientation 42, the real forceps position and real forceps orientation, and the real camera position 21A and real camera orientation 22A, the position (virtual forceps position) and orientation (virtual forceps direction), the position (virtual camera position 21B), and direction (virtual camera orientation 22B) of the virtual camera are calculated (S19). Based on the virtual camera position 21B and the virtual camera orientation 22B, an image is generated by rendering the volume data by viewing the virtual camera orientation 22B from the virtual camera position 21B (S20). Note that in S19 and S20, the calculation of the virtual camera position 21B and the virtual camera orientation 22B may be the calculation of a virtual imaging range (virtual field of view), and image generation may be performed based on the virtual imaging range. .

Note that, after rendering in S20, the processing unit 160 may input the real forceps position, real forceps orientation, real camera position, and real camera orientation in S16 via the UI 120, and perform the processing from S17 onwards. In other words, the processes from S16 to S20 may be repeated. As a result, even if the user freely moves and operates the forceps 30 and camera, the medical image processing apparatus 100 renders the target TG in a state that is easy to see while reflecting the position and orientation of the operated forceps 30 and camera. can be visualized.

Note that, after the rendering in S20, the processing unit 160 may input information for pressing the organ in S17 via the UI 120, and perform the processing from S18 onwards. In other words, the processes of S17 to S20 may be repeated. As a result, even if the user moves the forceps 30 and changes the way the organ is pushed (for example, the position at which the organ is pushed, the force at which it is pushed, and the direction in which it is pushed), the medical image processing apparatus 100 can maintain the position of the operated forceps 30 or the camera. The target TG can be rendered and visualized in an easy-to-see state while reflecting the direction and direction.

Note that the accuracy of the organ segmentation and bone segmentation in S13 does not need to be high accuracy. For example, even if there is some error in the outline of an organ or bone, it is sufficient that the state of the target TG in the organ can be easily recognized.

According to the process shown in FIG. 11, the medical image processing apparatus 100 can reduce the amount of calculations related to transformation by transforming the model ML without transforming volume data with a large number of pixels. Furthermore, a state in which the target TG can be visually recognized by deforming the organ by deforming the model ML can be easily reproduced without deforming the volume data to be visualized. Therefore, the user checks whether it is possible to approach the target TG from the actual camera position, whether there are any obstacles on the way to the approach, whether the angle is such that the approach with the forceps 30 is possible, etc. can. Furthermore, if the target TG cannot be visually recognized even after deforming the model ML, the medical image processing apparatus 100 can recommend performing open surgery instead of arthroscopic surgery. In other words, it can provide information for determining whether arthroscopic surgery is possible or open surgery is necessary.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that those skilled in the art can come up with various changes or modifications within the scope of the claims, and these naturally fall within the technical scope of the present disclosure. Understood.

For example, the arthroscopic surgery in which preoperative simulation and intraoperative navigation are performed may be laparoscopic surgery, arthroscopic surgery, bronchoscopy surgery, colonoscopic surgery, or other arthroscopic surgery. Furthermore, arthroscopic surgery may be a surgery performed by the surgeon directly operating forceps, or a robotic surgery using a surgical robot.

Tissue modification may also include tissue ablation. The deformation by cutting here is performed on the model ML, but it may also be applied to volume data. This is because the amount of calculation involved in excision is not enormous even if volume data is used. Specifically, upon acquiring a cutting operation via the UI 120, for example, the deformation processing unit 162 may transform the volume data by cutting the volume data in accordance with the cutting operation. Further, the area processing unit 161 sets the portion of the volume data to be excised as a non-masked area that is not to be displayed, and the display control unit 166 causes a part of the volume data included in the non-masked area to be hidden. It's okay.

Furthermore, tissue deformation may be expressed by moving grid points arranged at equal intervals with respect to volume data. In this case, a change in the orientation of the target can be expressed by a change in the orientation of line segments drawn between grid points. For example, this implementation method is described in Reference Patent Document 1.
(Reference Patent Document 1: US Patent No. 8311300)

Regarding the movement of the viewpoint, while the camera position (viewpoint) can be moved with respect to the target TG before and after transformation, the distance between the target TG and the camera position may not be kept constant and the distance may change. . For example, when an organ including the target TG is visualized using a virtual endoscopic image (perspective projection image), the distance between the viewpoint and the target TG may or may not be maintained in the virtual endoscopic image. Note that in the case of parallel projection, it is difficult to estimate the distance between the target TG and the viewpoint. If the distance is not maintained, for example, zooming in or zooming out can be expressed that is different from the distance between the actual camera position and the target TG.

Furthermore, after the real camera position relative to the target TG, the virtual camera position facing the real camera, and the movement to the virtual camera orientation, the transformation processing unit 162 further changes at least one of the virtual camera position and the virtual camera orientation via the UI 120. Good too. Thereby, the medical image processing apparatus 100 can additionally perform manual viewpoint movement in addition to automatic viewpoint movement based on the positional relationship between the actual camera position and the target TG in the model ML after deformation. Therefore, for example, by fine-tuning the result of automatic viewpoint movement by manual viewpoint movement and checking the image generated based on the manually moved viewpoint, it is possible to make the state of the target TG even easier to see. can.

In addition, although the processing unit 160 has been shown to visualize the area around the target TG by moving it from the real camera position to the virtual camera position when the target TG is difficult or impossible to see from the real camera position, the processing unit 160 is not limited to this. do not have. For example, although the target TG is visible from the real camera position, the processing unit 160 may move the target TG to a virtual camera position where it is easier to see and visualize the surroundings of the target TG as seen from the virtual camera position. good. Thereby, the medical image processing apparatus 100 can make it easier to observe the state of the target TG even when the target TG is visible.

Although the liver is mainly illustrated as an organ to which the above embodiment is applied, other organs may be used. For example, it may be the lungs, stomach, intestines, pancreas, or other organs.

Furthermore, the medical image processing apparatus 100 only needs to include at least a processor 140 and a memory 150. The acquisition unit 110, the UI 120, and the display 130 may be externally attached to the medical image processing apparatus 100.

In addition, the volume data as a captured CT image has been illustrated as being transmitted from the CT apparatus 200 to the medical image processing apparatus 100. Alternatively, the volume data may be sent to a server on a network (for example, an image data server (PACS) (not shown)) and stored so that the volume data is temporarily stored. In this case, the acquisition unit 110 of the medical image processing apparatus 100 may acquire the volume data from a server or the like via a wired line or wireless line, or via an arbitrary storage medium (not shown) when necessary. You may.

Further, the volume data as a captured CT image is transmitted from the CT apparatus 200 to the medical image processing apparatus 100 via the acquisition unit 110, as an example. This also includes the case where the CT apparatus 200 and the medical image processing apparatus 100 are substantially combined into one product. It also includes a case where the medical image processing apparatus 100 is treated as a console of the CT apparatus 200.

Moreover, although the CT apparatus 200 captures images and generates volume data including information inside the object, it is also possible to capture images using other devices and generate volume data. Other devices include Magnetic Resonance Imaging (MRI) devices, Positron Emission Tomography (PET) devices, Angiography devices, or other modality devices. Additionally, the PET device may be used in combination with other modality devices.

Further, the operation of the medical image processing apparatus 100 can be expressed as a defined medical image processing method. Further, it can be expressed as a program for causing a computer to execute each step of the medical image processing method.

(Summary of the above embodiment)
One aspect of the embodiment described above is a medical image processing device 100 (an example of an arthroscopic surgery support device) that supports arthroscopic surgery, and includes an acquisition unit 110 and a processing unit 160. The acquisition unit 110 has a function of acquiring volume data of a subject. The processing unit 160 sets a model ML that represents the tissue included in the volume data. The processing unit 160 sets a real field of view (an example of a first field of view) for visualizing volume data and a position of the target TG in the model ML. The processing unit 160 acquires information on a deformation operation (an example of first operation information) for deforming the model ML. The processing unit 160 may calculate changes in the position and orientation of the target TG before and after deforming the model ML based on the information on the deformation operation. The processing unit 160 uses the calculated changes in the position and orientation of the target TG to determine the position and orientation of the target TG after the change with respect to the real field of view and the position and orientation of the target TG before the change with respect to the virtual field of view (an example of the second field of view). The virtual field of view may be calculated so that the position and orientation are equivalent. The processing unit 160 may visualize the volume data based on the virtual field of view.

Thereby, the medical image processing apparatus 100 can make the positional relationship between the target TG and the real camera after deformation the same as the positional relationship between the target TG and the virtual camera before deformation, without deforming the tissue. , a virtual field of view can be determined and visualization of volume data based on the virtual field of view can be provided. Therefore, for example, even if the target is located on the back side of an organ when looking at the real line of sight from the real field of view, instead of deforming the organ and visualizing it, change the field of view and visualize the undeformed organ. It is possible to express the same state as deforming and visualizing an organ. Furthermore, since the volume data is not transformed, the amount of calculation can be reduced. Furthermore, it is no longer necessary to appropriately set stiffness for each voxel of the volume data and take into account slippage at the boundaries of organs, flow of body fluids, and air. Furthermore, when an organ is repeatedly deformed little by little assuming a surgical process, there is no need to maintain consistency of volume data related to the deformation process. Therefore, manual fine-tuning of volume data and regions of interest is also not required. Therefore, for example, a simulation in which an operator manipulates an organ using the forceps 30 in arthroscopic surgery can be performed in real time. In particular, it is ensured that organ boundaries are not broken.

Furthermore, since the visualized organs are not deformed, a plurality of organs do not overlap in the virtual space due to deformation of the organs. Additionally, since the visualized organs are not deformed, for example, even if the segmentation of each organ is incorrect, blood vessels that are connected to the outline of the liver and those that are not can be clearly visualized, reducing the accuracy of surface rendering. can be restrained from doing so. In this way, visualization of the observation target in the subject can be improved.

Further, the processing unit 160 may classify the tissue including the target TG in the subject and set the model ML based on the classified tissue. Thereby, the medical image processing apparatus 100 can visualize the volume data seen from the virtual field of view using the model ML only for the tissue including the target TG, making it easier to observe the tissue including the target TG. Even in this case, the medical image processing apparatus 100 does not deform the organs to be visualized, so deformed organs and non-deformed organs do not coexist, and multiple organs overlap in virtual space due to organ deformation. can be prevented. Furthermore, it becomes easier to perform deformation operations on models corresponding to organs closely related to the target TG.

Further, the processing unit 160 may display information indicating the outline of the tissue on the display 130 (an example of a display unit). Then, in accordance with the deformation of the model ML, the contour may be deformed and information indicating the contour may be displayed. Although some organs have unclear outlines, the medical image processing apparatus 100 can clarify what shape the tissue has by displaying information indicating the outlines. Note that the model ML for the transformation operation may also be displayed on the display 130. The contour may be generated by polygons.

Further, the processing unit 160 may hide the bones of the subject in the visualization of volume data based on the virtual field of view. Thereby, the medical image processing apparatus 100 can exclude information not directly related to the organ including the target from the display target, and can make the target easier to see.

Further, in visualizing volume data based on a virtual visual field, the processing unit 160 may display bones of the subject included in an image visualized when visualizing volume data based on a real visual field. Thereby, the medical image processing apparatus 100 can visualize the target TG and the surrounding bones, and can support arthroscopic surgery by the surgeon. Note that the processing unit 160 may visualize bones without extracting them. For example, since the CT value of bones is large compared to other tissues, bones can be visualized by visualizing voxels with CT values equal to or higher than a threshold value.

Further, the processing unit 160 may display the forceps 30 inserted into the subject toward the target TG along with the volume data. Thereby, the medical image processing apparatus 100 can visualize the positional relationship between the target TG and the forceps 30 for performing various measures on the target TG, and can support arthroscopic surgery by the surgeon.

Furthermore, the processing unit 160 may acquire information on a transformation operation (an example of second operation information) for operating the distance between the target position 41A before the model is transformed and the virtual viewpoint corresponding to the virtual field of view. Then, the volume data may be visualized based on the information on this transformation operation. Thereby, the medical image processing apparatus 100 can adjust the virtual viewpoint by manually moving the viewpoint, and can visualize the target TG at a distance and angle from the virtual viewpoint desired by the surgeon. Therefore, it is possible to easily obtain an image including the target TG desired by the follower.

One aspect of the above embodiment includes the steps of acquiring volume data of a subject, setting a model representing tissue included in the volume data, a first field of view for visualizing the volume data, and a step of setting the position of the target; a step of obtaining first operation information for deforming the model; and a step of determining changes in the position and orientation of the target before and after deforming the model based on the first operation information. calculating, and using the calculated changes in the position and orientation of the target, the position and orientation of the target after the change with respect to the first visual field and the position and orientation of the target before the change with respect to the second visual field are made equal to each other; This is an arthroscopic surgery support method comprising the steps of calculating a second visual field and visualizing volume data based on the second visual field.

One aspect of the present embodiment is a program for causing a computer to execute the arthroscopic surgery support method described above.

The present disclosure is useful for an arthroscopic surgery support device, an arthroscopic surgery support method, a program, etc. that can improve visualization of an observation target in a subject.

10, 10A, 10B Liver 12 Portal vein 14 Vein 15, 15B Bone 21 Camera position 21A Real camera position 21B Virtual camera position 22 Camera orientation 22A Real camera orientation 22B Virtual camera orientation 30 Forceps 30A Real forceps 30B Virtual forceps 41, 41A, 41B Target positions 42, 42A, 42B Target orientation 100 Medical image processing device 110 Acquisition unit 120 UI
130 Display 140 Processor 150 Memory 160 Processing unit 161 Region processing unit 162 Transformation processing unit 163 Model setting unit 164 Information setting unit 165 Image generation unit 166 Display control unit 200 CT device ML, MLA, MLB Model TG Target

Claims (9)

An arthroscopic surgery support device that supports arthroscopic surgery,
Comprising an acquisition unit and a processing unit,
The acquisition unit has a function of acquiring volume data of the subject,
The processing unit includes:
Setting a model representing the tissue included in the volume data,
setting a first field of view for visualizing the volume data and a position of a target in the model;
obtaining first operation information for transforming the model;
Based on the first operation information, calculate changes in the position and orientation of the target before and after deforming the model,
Using the calculated changes in the position and orientation of the target, determine the position and orientation of the target after the change with respect to the first visual field and the position and orientation of the target before the change with respect to the second visual field. Calculate the second visual field so that
having a function of visualizing the volume data based on the second field of view;
Arthroscopic surgery support device. The arthroscopic surgery support device according to claim 1, wherein the processing unit classifies a tissue including the target in the subject and sets the model based on the divided tissue. The processing unit includes:
displaying information indicating the outline of the tissue on a display unit;
deforming the contour in accordance with deformation of the model and displaying information indicating the contour;
The arthroscopic surgery support device according to claim 2. The processing unit hides the bones of the subject in the visualization of the volume data;
The arthroscopic surgery support device according to any one of claims 1 to 3. In the visualization of the volume data, the processing unit displays bones of the subject included in an image visualized when the volume data is visualized based on the first field of view.
The arthroscopic surgery support device according to any one of claims 1 to 3. The processing unit displays forceps inserted toward the target inside the subject, along with the volume data.
The arthroscopic surgery support device according to any one of claims 1 to 5. The processing unit includes:
obtaining second operation information for manipulating a distance between the position of the target before deforming the model and a viewpoint corresponding to the second field of view;
visualizing the volume data based on the second operation information;
The arthroscopic surgery support device according to any one of claims 1 to 6. obtaining volume data of the subject;
setting a model representing the tissue included in the volume data;
setting a first field of view for visualizing the volume data and a position of a target in the model;
obtaining first operation information for deforming the model;
calculating changes in the position and orientation of the target before and after deforming the model based on the first operation information;
Using the calculated changes in the position and orientation of the target, determine the position and orientation of the target after the change with respect to the first visual field and the position and orientation of the target before the change with respect to the second visual field. calculating the second visual field so that the
Visualizing the volume data based on the second field of view;
Arthroscopic surgery support method with. A program that causes a computer to execute the arthroscopic surgery support method according to claim 8.

Images