JP2013031660A - Method and apparatus for processing medical image, and robotic surgery system using image guidance - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and apparatus for processing medical image, and robotic surgery system using image guidance.SOLUTION: The method of processing a medical image, includes: acquiring medical images captured using a plurality of different medical image capturing apparatuses with respect to a predetermined organ; extracting surface information of the predetermined organ, which is included in each of the medical images, from each of the medical images; mapping each of the medical images using the extracted surface information; and generating a synthesis image in which the medical images have been matched, based on the mapping result.

Description

  The present invention relates to a method and apparatus for processing medical images, and more particularly, to a robotic surgery system using image guidance.

  In robotic surgery, unlike open surgery, a doctor cannot directly see the surgical site in the patient's body with the naked eye, but grasps the surgical site only through the screen displayed on the monitor. Doctors who perform robotic surgery perform surgery by grasping the surgical site through computed tomography (CT), magnetic resonance imaging (MRI), ultrasound images, etc. before surgery, There is a limit that depends heavily on the experience of the doctor. In addition, in order to acquire a surgical site image, a method has been tried in which a laparoscope is inserted into a patient's body and robot surgery is performed while viewing the displayed actual image in the body. However, the only images that can be acquired with an endoscope such as a laparoscope are only images of the external surface of the organ tissue in the body. When the site is inside an organ, it is difficult to accurately grasp the exact position and form of the surgical site.

US Pat. No. 5,704,897

  An object of the present invention is to provide a method and apparatus for processing medical images. It is another object of the present invention to provide a computer-readable recording medium that records a program for causing the computer to execute the method. Furthermore, it is to provide a robotic surgery system using image guidance based on the processed medical image. However, the object of the present invention is not limited to the object described above.

  According to an aspect of the present invention, a medical image processing method includes a step of acquiring a medical image of a predetermined organ imaged using a plurality of different medical image imaging devices, and each of the acquired medical images Respectively, extracting the surface information of the predetermined organ included in each of the acquired medical images, mapping the medical images using the extracted surface information, Generating a composite image in which the medical images are aligned based on a mapping result.

  According to another aspect, a computer-readable recording medium recording a program for causing the medical image processing method to be executed by a computer is provided.

  According to still another aspect, the medical video processing device includes a video acquisition unit that acquires a medical video of a predetermined organ imaged using a plurality of different medical video imaging devices, and the acquired medical video The medical image is mapped by using a surface information extraction unit that extracts surface information of the predetermined organ included in each of the acquired medical images, and the extracted surface information. A video mapping unit; and a composite video generation unit configured to generate a composite video in which the medical video is matched based on the mapping result.

  According to still another aspect, in a robotic surgery system that guides an image of a surgical site and performs robotic surgery with a surgical robot, an endoscope that captures a medical image of a predetermined organ in a subject An ultrasonic device, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, and a positron emission tomography (PET) device for imaging medical images of the device and the predetermined organ A non-endoscopic device including at least one of the medical images, and the medical images captured using the plurality of medical image capturing devices, and each of the acquired medical images from the acquired medical images. And extracting the surface information of the predetermined organ contained in the map, mapping the medical image using each of the extracted surface information, A medical image processing device that generates a combined image in which the medical images are aligned, a display device that displays the generated combined image, a surgical robot that performs a robotic operation by a user input, Is provided.

  According to the present invention, the use of a marker at the time of video alignment is troublesome and inconvenient by matching a medical video in real time based on only information included in the medical video without using an artificial marker. Can be reduced. In particular, in robotic surgery, it is possible to reduce a decrease in the accuracy of image matching due to interference between the marker of the metal component and the surgical robot.

  Then, by generating a composite image in which the endoscopic image and the non-endoscopic image are aligned in real time, a more accurate patient diagnosis image is provided to the doctor, and the robot operation system performs more accurate image guidance. Can be. Further, in the case of robotic surgery, by providing an accurate medical image of the surgical site in this way, it is possible to accurately grasp the site to be operated and the site to be stored, so that the surgical performance is improved. Furthermore, when robot surgery is automated in the future, information that can accurately control the robot is provided.

It is a block diagram of the robotic surgery system by one Embodiment of this invention. It is a block diagram of the robotic surgery system by one Embodiment of this invention. It is drawing which shows the relative position of the endoscope apparatus and ultrasound apparatus about the bladder by one Embodiment of this invention. 1 is a configuration diagram of a medical image processing apparatus according to an embodiment of the present invention. 4 is a diagram illustrating a process of extracting first surface information after a first surface model is generated by a first extraction unit according to an exemplary embodiment of the present invention. 4 is a diagram illustrating a process of extracting second surface information after a second surface model is generated by a second extraction unit according to an exemplary embodiment of the present invention. 2 is a diagram separately showing only an arrangement of an endoscope device and an ultrasonic device in the robotic surgical system of FIG. 1B according to an embodiment of the present invention. 4 is a diagram illustrating an example of information included in a three-dimensional ultrasound image used when a synthesized image is generated by a synthesized image generation unit according to an exemplary embodiment of the present invention. 3 is a view illustrating a composite image according to an exemplary embodiment of the present invention. 4 is a flowchart of a method for processing medical images according to an embodiment of the present invention. 10 is a detailed flowchart of a method for processing the medical image of FIG. 9. 4 is a flowchart of a process of extracting first surface information according to an embodiment of the present invention. 5 is a flowchart of a process of extracting second surface information according to an exemplary embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1A is a configuration diagram of a robotic surgery system 1 according to an embodiment of the present invention. Referring to FIG. 1A, the robot surgery system 1 includes a first medical image photographing device 11, a second medical image photographing device 21, a medical image processing device 30, a surgical robot 40, and a display device 50. In FIG. 1A, only hardware components related to the present embodiment are described in order to prevent the features of the present embodiment from becoming unclear. However, those skilled in the art will understand that other general-purpose hardware components are included in addition to the hardware components shown in FIG. 1A.

  Referring to FIG. 1A, the robotic surgical system 1 is shown to include only the first and second medical image capturing apparatuses 11 and 21, but the present embodiment is not limited thereto. One additional medical imaging device may be provided.

  Types of medical imaging equipment include endoscopic equipment, ultrasound equipment, computed tomography (CT) equipment, magnetic resonance imaging (MRI) equipment, positron emission tomography (Positron Emission Tomography: There are various types such as PET) devices. Hereinafter, medical video imaging apparatuses other than the medical video imaging apparatus that captures endoscopic video are referred to as non-endoscopic apparatuses. That is, the ultrasonic apparatus, the computer tomography apparatus, the magnetic resonance imaging apparatus, and the positron emission tomography apparatus are non-endoscopic apparatuses.

  Hereinafter, for convenience of explanation, the first medical image photographing apparatus 11 of the robotic surgery system 1 according to the present embodiment corresponds to an endoscope apparatus such as a laparoscopic apparatus, and the second medical image photographing apparatus 21 is Although described as corresponding to an ultrasound device such as a trans-rectal ultrasound (TRUS) device, the present embodiment is not limited to this. That is, each of the first and second medical imaging apparatuses 11 and 21 is a medical imaging apparatus such as the endoscope apparatus, ultrasonic apparatus, computed tomography apparatus, magnetic resonance imaging apparatus, or positron emission tomography apparatus. Any one of them may be used.

  FIG. 1B is a block diagram of a robotic surgery system 100 according to an embodiment of the present invention. Referring to FIG. 1B, as described above, the robot surgical system 100 includes the endoscope device 10, the ultrasonic device 20, the medical image processing device 30, the surgical robot 40, and the display device 50.

  In FIG. 1B, as in FIG. 1A, only hardware components related to the present embodiment are described in order to prevent the features of the present embodiment from becoming unclear.

  The robotic surgery system 100 is a system in which an arm of a surgical robot 40 is inserted into a small hole formed in a patient's body, and a doctor operates the patient by controlling the movement of the surgical robot 40 outside the patient's body. .

  Recently, da Vinci (registered trademark) manufactured by Intuitive Surgical, Inc. of the United States has been generally used as such a surgical robot 40. More specifically, da Vinci (registered trademark) is a robot that is directly inserted into a patient's body and can move like a doctor's hand, so that the doctor operates directly on the surgical site. It is a robot that operates on. In the present embodiment, for convenience of explanation, da Vinci (registered trademark) is merely given as an example of the surgical robot 40, and the surgical robot 40 can be operated through the movement of the robot in the patient's body. The apparatus may be used.

  When a doctor intends to perform an operation using the operation robot 40 in the robot operation system 100, the doctor proceeds with the operation with reference to a medical image in the patient's body displayed on the display device 50. That is, in the robotic surgery system 100, a doctor inserts a special lens into a patient's body, and proceeds with surgery after securing a visual field through images of nerves, blood vessels, and organs that cannot be seen with the naked eye.

  In the robotic surgery system 100, unlike the laparotomy, the doctor cannot directly see the surgical site in the patient's body with the naked eye and can grasp the surgical site only through the screen displayed on the display device 50. Accurate video is required.

  In particular, robotic surgery is commonly used for operations such as prostate cancer, rectal cancer, esophageal cancer, and bladder cancer that cause severe side effects and complications when peripheral nerves and blood vessels are damaged during surgery. It is important that an elaborate and detailed image of the surgical site is displayed on the display device 50.

  Conventionally, a method of performing an operation by recalling a site to be operated from the brain of a doctor who stores a diagnostic image by showing a CT, MRI, ultrasound, PET image or the like to the doctor before the operation has been used. However, this is highly dependent on the experience of the doctor, making accurate surgery difficult.

  In the case of conventional robotic surgery, a laparoscope was inserted into the patient's body, and the robotic surgery was performed while viewing the displayed actual image of the body. However, the only images that can be acquired with a laparoscope for the surgical site are only images of the external surface of the organ tissue in the body. Therefore, when the surgical site is hidden behind the organ and cannot be seen, or when the surgical site is inside the organ, it is difficult to obtain an actual image of the surgical site with a laparoscope. This was especially the case for prostate surgery.

  To describe prostate surgery, the prostate has a narrow surgical site and is connected to the urethra. And when the prostate is removed, the neurovascular bundle near the prostate must be preserved. This is because this neurovascular bundle plays an important role in urine function, sexual function and the like. However, since the laparoscope provides only an image of the tissue on the outer surface, it is somewhat difficult to precisely and precisely grasp the position and form of the prostate with only the laparoscope.

  In order to improve such an existing method, a transrectal ultrasound (TRUS) apparatus has been conventionally used. However, there is a limitation of an ultrasound image that is not an actual image of a surgical site.

  Conventionally, the position and direction of a transrectal ultrasound device is detected in real time using an optical or magnetic field marker to acquire a 3D ultrasound image and use it for surgery. did. However, when the magnetic field method is used, the marker position measurement becomes inaccurate due to the interference between a metallic substance such as a surgical robot and the magnetic field. When using the optical method, the movement range of the surgical robot overlaps with the range of activity of the surgical robot at the time of position measurement, and the movement of the surgical robot is restricted.

  Furthermore, conventionally, a method of rotating a transrectal ultrasound apparatus using a tandem robot other than a surgical robot and then acquiring a three-dimensional prostate image has been used. However, this method requires the use of additional robots, which limits the movement of the surgical robot due to interference by the tandem robot. In addition, since the positions of the surgical robot and the tandem robot have to be corrected, the ultrasonic image cannot be used at the time of actual surgery.

  That is, as described above, when performing robotic surgery, particularly when performing robotic surgery that should not damage peripheral nerves and blood vessels, such as robotic surgery for the prostate, an accurate image of the surgical site has been conventionally used. It was difficult to obtain and there was a limit to patient safety.

  However, in the robotic surgery system 100 according to the present embodiment, a surgical site hidden in an organ or the like can be obtained by using a composite image in which different medical images captured by a plurality of different medical image capturing apparatuses are aligned in real time. An accurate image is also provided for a surgical site located inside an organ. Therefore, according to the present embodiment, it is possible to guarantee the performance of the robot operation and the safety of the patient.

  However, in the present embodiment, a medical image for a doctor who performs robotic surgery under the robotic surgery system 100, that is, medical image guidance will be described, but the medical image processing apparatus 30 according to the present embodiment generates the medical image. The composite image is not necessarily limited to that provided only by the robotic surgery system 100. That is, the medical image provided in the present embodiment is similarly provided in other systems for simply diagnosing or diagnosing a patient who is not robotic surgery.

  Hereinafter, a process of processing a medical image in the robotic surgery system 100 according to the present embodiment will be described in detail.

  Here, for the convenience of explanation, the case where the surgical site according to the present embodiment is the prostate is taken as an example. As described above, when the surgical site is the prostate, the medical image processing process according to the present embodiment will be described as using a bladder that is a specific organ located around the prostate. That is, the specific organ may be an organ at a surgical site to be treated by the surgical robot 40 or another organ around the surgical site.

  Further, those skilled in the art will understand that the present embodiment is that the surgical site is another part of the patient, or medical image processing is performed using other organs.

  Referring to FIG. 1B again, under the robotic surgical system 100, the endoscope apparatus 10 acquires an endoscopic image of a patient's organ, for example, the bladder. Therefore, the endoscopic image includes an image of the bladder and its surroundings. In the present embodiment, the endoscope apparatus 10 corresponds to a laparoscopic apparatus, but is not limited thereto.

  Then, the ultrasonic device 20 acquires an ultrasonic image of the patient's bladder and its surroundings. Therefore, the ultrasound image includes images of the bladder and the periphery inside and outside thereof. That is, unlike the endoscopic image, the ultrasonic image includes information about the tissue inside the bladder. In the present embodiment, the ultrasound device 20 corresponds to a transrectal ultrasound device, but is not limited thereto.

  In FIG. 1B, the endoscope apparatus 10 and the ultrasonic apparatus 20 capture medical images at different positions. That is, in the robotic surgery system 100, medical images are taken by controlling the movement and position of the endoscope apparatus 10 and the ultrasonic apparatus 20 respectively. At this time, the robotic surgery system 100 stores the imaging positions of the endoscope device 10 and the ultrasonic device 20 under control, for example, virtual coordinates on the robotic surgery table, etc. Continue to save (not shown).

  FIG. 2 is a view showing the relative positions of the endoscope apparatus 10 and the ultrasonic apparatus 20 with respect to the bladder according to an embodiment of the present invention. Referring to FIG. 2, when the endoscope apparatus 10 is a laparoscopic apparatus, an endoscope image is acquired from above the bladder relative to the bladder. When the ultrasound device 20 is a transrectal ultrasound device, an ultrasound image is acquired from below the bladder relative to the bladder. However, the positions of the endoscope apparatus 10 and the ultrasonic apparatus 20 are exemplary and may be changed depending on the robot operation environment.

  Referring to FIG. 1B again, the medical image processing apparatus 30 generates a composite image by matching the endoscope image and the ultrasound image acquired from the endoscope apparatus 10 and the ultrasound apparatus 20. The operation and function of the medical image processing apparatus 30 according to the present embodiment will be described in more detail as follows.

  FIG. 3 is a block diagram of the medical image processing apparatus 30 according to an embodiment of the present invention. Referring to FIG. 3, the medical video processing apparatus 30 includes a detection unit 31, a video acquisition unit 32, a surface information extraction unit 33, a video mapping unit 34, and a composite video generation unit 35. The video acquisition unit 32 includes an endoscopic video acquisition unit 321 and a non-endoscopic video acquisition unit 322. The surface information extraction unit 33 includes a first extraction unit 331 and a second extraction unit 332, and video mapping. The unit 34 includes a comparison unit 341 and a position matching unit 342.

  The medical image processing apparatus 30 corresponds to a processor. The processor may be realized by an array of a large number of logic gates, or may be realized by a combination of a general-purpose microprocessor and a memory in which a program executed by the microprocessor is stored. Further, those skilled in the art will understand that the present invention may be realized by other forms of hardware.

  The detection unit 31 detects the current position of each medical video imaging apparatus stored in the storage unit (not shown) on the robotic surgery system 100 described above when trying to generate a composite video.

  The image acquisition unit 32 acquires medical images of an organ imaged using a plurality of different medical image capturing devices, that is, an endoscopic image and a non-endoscopic image. The surface information extraction unit 33 extracts the surface information of the organs included in each acquired medical image from each acquired medical image. In particular, the surface information extraction unit 33 extracts, as surface information, information representing at least one of the position and form of the surface of a predetermined organ from each acquired medical image.

  The video mapping unit 34 maps the medical video using each extracted surface information. In particular, the surface information extraction unit 33 maps the medical image by using the extracted surface information to match the positions of the medical image capturing devices.

  Hereinafter, the process of processing an endoscopic image will be described in more detail first, and then the process of processing a non-endoscopic image such as an ultrasound image, a CT image, and an MR (Magnetic Resonance) image will be described in further detail. Explained.

  The endoscope video acquisition unit 321 acquires an endoscope video imaged using the endoscope apparatus 10 (FIG. 1B).

  The first extraction unit 331 extracts first surface information representing at least one of the position and form of the surface of the organ from the endoscopic image captured by the endoscope apparatus 10 (FIG. 1B). That is, in this embodiment, the 1st extraction part 331 extracts the 1st surface information about the bladder which appeared in the endoscopic image | video.

  As the method, the first extraction unit 331 generates disparity space images by acquiring distance information between the bladder and surrounding external tissues and the endoscope apparatus 10 (FIG. 1B). According to one embodiment, the first extraction unit 331 acquires a parallax space image using the endoscope apparatus 10 (FIG. 1B) provided with two stereo cameras. At this time, according to another embodiment, the first extraction unit 331 uses the endoscope apparatus 10 (FIG. 1B) further including a projector that emits at least one of structured light and pattern light. Generate parallax space video. In this case, the endoscopic image acquisition unit 321 also acquires information about structured light or pattern light reflected from the bladder and surrounding external tissues. That is, the first extraction unit 331 calculates the distance from the endoscope apparatus 10 (FIG. 1B) to the bladder and surrounding external tissues using information about the acquired structured light or pattern light. At this time, the first extraction unit 331 generates a distance image such as a parallax space image based on the calculated distance.

  Next, the first extraction unit 331 generates a three-dimensional first surface model corresponding to the endoscopic image by using the acquired distance information, that is, the calculated distance or the generated distance image. .

  Finally, the first extraction unit 331 extracts first surface information representing at least one of the position and shape of the bladder surface from the first surface model generated in this way.

  FIG. 4 is a diagram illustrating a process of extracting first surface information after the first surface model is generated by the first extraction unit 331 (FIG. 3) according to an embodiment of the present invention. 4 is an endoscopic image acquired by the endoscope apparatus 10 (FIG. 1B), and is an image in which structured light or pattern light is irradiated to the actual bladder and its periphery.

  An image 402 in FIG. 4 is a parallax space image, and corresponds to a kind of distance image generated using at least one of structured light and pattern light. However, as described above, the first extraction unit 331 (FIG. 3) can also generate a parallax space image without using structured light and pattern light.

  A video 403 in FIG. 4 is a first surface model generated by the first extraction unit 331 (FIG. 3) through the above process. The first extraction unit 331 (FIG. 3) extracts first surface information 404 about the form and position of the bladder surface from the first surface model of the image 403.

  Referring to FIG. 3 again, the non-endoscopic image acquisition unit 322 acquires a non-endoscopic image captured using a non-endoscopic device such as the ultrasound device 20 (FIG. 1B). The second extraction unit 332 extracts second surface information representing at least one of the position and form of the surface of the organ from the video imaged by the non-endoscopic device. That is, in this embodiment, the 2nd extraction part 332 extracts the 2nd surface information about the bladder which appeared in the non-endoscopic image | video.

  As the method, first, the second extraction unit 332 acquires information about the boundary line representing the surface of the bladder from the non-endoscopic video imaged by the non-endoscopic device. At this time, the information about the boundary line is acquired by applying at least one of line detection and edge detection to the non-endoscopic image.

  When the non-endoscopic image is an ultrasonic image, the property that the ultrasonic wave has high echogenicity with respect to the surface tissue of the organ is used due to the characteristics of the ultrasonic wave. That is, the second extraction unit 332 acquires information about the boundary line by using the property that the surface tissue of the organ appears as a relatively bright line in the ultrasound image.

  When the non-endoscopic image is an MR image, the second extraction unit 332 uses line detection or edge detection based on the point that a difference in image brightness occurs in the MR image due to the difference in the molecular composition ratio of the tissue. In this way, information about the boundary line is acquired.

  Similarly, when the non-endoscopic video is a CT video, the second extraction unit 332 uses line detection or edge detection based on the point that a video brightness difference occurs in the CT video due to a tissue density difference. In this way, information about the boundary line is acquired.

  Next, the second extraction unit 332 generates a three-dimensional second surface model corresponding to the surface of the organ (bladder) using the acquired boundary line information. At this time, the second extraction unit 332 generates a three-dimensional second surface model by rendering the boundary line three-dimensionally based on the acquired boundary line information.

  Finally, the second extraction unit 332 extracts second surface information representing at least one of the position and shape of the surface of the bladder from the second surface model generated in this way.

  FIG. 5 is a diagram illustrating a process of extracting second surface information after the second surface model is generated by the second extraction unit 332 (FIG. 3) according to an embodiment of the present invention. Referring to FIG. 5, the process of extracting the second surface information 505 from the ultrasound image 501, the process of extracting the second surface information 505 from the MR image 502, and the second surface information 505 from the CT image 503. The process is shown. Depending on what medical imaging device is used in the environment of the robotic surgical system 100 (FIG. 1B), the second extraction unit 332 (FIG. 3) performs the second surface information 505 according to the corresponding process among the respective processes. To extract.

  In the case of the ultrasound image 501, the second extraction unit 332 (FIG. 3) extracts the boundary line corresponding to the surface of the bladder from each image using the characteristics of the ultrasound image described above. Then, the second surface model 504 is generated by rendering each boundary line three-dimensionally.

  In the case of the MR image 502, the second extraction unit 332 (FIG. 3) extracts the boundary line corresponding to the rectal surface from each image using the above-described MR image characteristics. Then, the second surface model 504 is generated by rendering each boundary line three-dimensionally.

  In the case of the CT image 503, the second extraction unit 332 (FIG. 3) extracts the boundary line corresponding to the rectal surface from each image using the characteristics of the CT image described above. Then, the second surface model 504 is generated by rendering each boundary line three-dimensionally.

  The second extraction unit 332 (FIG. 3) extracts second surface information 505 representing at least one of the shape and position of the surface of the organ based on the boundary line information appearing in the second surface model 504.

  That is, the second extraction unit 332 (FIG. 3) determines the ultrasonic image 501, the MR image 502, and the CT image 503 depending on what medical image capturing device is used in the environment of the robotic surgery system 100 (FIG. 1B). The second surface information 505 is extracted by a corresponding process among the processes.

  Referring to FIG. 3 again, the video mapping unit 34 maps the medical video using the extracted surface information. The video mapping unit 34 maps the medical video by matching the positions of the respective medical video imaging apparatuses using the extracted first surface information and second surface information. When the endoscope apparatus 10 (FIG. 1B) and the ultrasonic apparatus 20 (FIG. 1B) are used as shown in FIG. 1B, the position of the endoscope apparatus 10 (FIG. 1B) and the surface information are used. The endoscopic image and the ultrasonic image are mapped by matching the position of the ultrasonic device 20 (FIG. 1B).

  The mapping process will be described in more detail. The video mapping unit 34 includes the comparison unit 341 and the position matching unit 342 as described above.

  The comparison unit 341 compares the extracted surface information. That is, the comparison unit 341 compares the first surface information and the second surface information. The reason is that the extracted first surface information and second surface information are information on the surface of the same part of the organ (bladder). Therefore, the comparison unit 341 compares how the extracted first surface information and second surface information correspond to the surface of the same part of the organ. At this time, the comparison unit 341 performs comparison using an ICP (Iterative Closest Point) algorithm, which is a known algorithm.

  The position matching unit 342 matches the position of the medical image apparatus detected by the detection unit 31 based on the comparison result.

  Eventually, the video mapping unit 34 maps the medical video based on the matching result of the process.

  FIG. 6 is a diagram separately showing only the arrangement of the endoscope apparatus 10 and the ultrasonic apparatus 20 in the robotic surgical system 100 of FIG. 1B according to an embodiment of the present invention. Referring to FIG. 6, the endoscope apparatus 10 corresponds to a laparoscopic apparatus, and the ultrasonic apparatus 20 corresponds to a transrectal ultrasonic apparatus.

On the robotic surgery system 100 (FIG. 1B), the virtual position of the endoscope apparatus 10 is based on the X _camera coordinate system. The virtual position of the ultrasonic device 20 is based on the _XUS coordinate system. That is, since the position of the endoscope apparatus 10 and the position of the ultrasonic apparatus 20 use different coordinate systems, the mutual positions are independent.

However, the position of the endoscope apparatus 10 and the position of the ultrasonic apparatus 20 are matched according to the same reference. For this reason, according to the present embodiment, the video mapping unit 34 uses the extracted first surface information and second surface information as a reference. If it demonstrates in detail, 1st surface information and 2nd surface information are the information about the surface of the same site | part of an organ (bladder). Therefore, the X _camera coordinate system and the X _US coordinate system are matched based on the first surface information extracted from the endoscopic image and the second surface information extracted from the ultrasound image. As a result, the position of the endoscope apparatus 10 and the position of the ultrasonic apparatus 20 are also matched.

  Referring to FIG. 3 again, the composite video generation unit 35 generates a composite video in which the medical video is matched based on the mapping result. The generated composite image may be a three-dimensional medical image of the organ and its surroundings. More specifically, the generated composite image includes an image of an external tissue about the organ and the periphery included in the image captured by the endoscope apparatus 10 (FIG. 1B), and a non-endoscopic apparatus 20 (FIG. 1B). 3 is an image in which the organs and the images of the internal and external tissues in the periphery included in the image captured by the above are simultaneously expressed in three dimensions. The composite video corresponds to a kind of augmented video.

  The composite video generated by the composite video generation unit 35 is generated by being synthesized so that the positions of the organs appearing in the endoscopic video and the non-endoscopic video are the same.

  An endoscopic image itself corresponds to an actual three-dimensional image of an organ and its surroundings. However, it is difficult for the endoscopic image to know information such as the form and position of the tissue inside and outside the organ.

  In general, such a non-endoscopic image corresponds to a set of images obtained by photographing an organ in a cross section. However, non-endoscopic images such as ultrasound images, CT images, and MR images include a kind of fluoroscopic information about the forms and positions of organs and surrounding internal and external tissues. Therefore, the acquired non-endoscopic image includes not only the external tissue of the organ but also information on the form and position of the internal tissue. Therefore, when an endoscopic image and a non-endoscopic image are combined, information about the actual organ and surrounding tissues inside and outside can be accurately known and provided to the doctor. Doctors can proceed with more elaborate and detailed surgery.

  Here, non-endoscopic images such as ultrasound images, CT images, and MR images may be two-dimensional images depending on the types of medical image capturing apparatuses 11 and 21 (FIG. 1A) that capture the images. 3D video may be used. When the acquired non-endoscopic image is a plurality of two-dimensional non-endoscopic images as shown in FIG. 5, the synthesized image generating unit 35 performs a two-dimensional non-endoscopic image through a known method such as volume rendering. An endoscopic video is generated into a three-dimensional non-endoscopic video and then used for composition.

  FIG. 7 is a diagram illustrating an example of information included in a three-dimensional ultrasound image used when the composite image generation unit 35 generates a composite image according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example when a transrectal ultrasonic device is used. Referring to FIG. 7, as described above, the ultrasound image includes a kind of fluoroscopic information about the form and position of the tissue inside and outside the organ such as the bladder. Therefore, in the three-dimensional ultrasound image, the form and position of the outer surface 701 of the bladder, the position of the prostate 702 inside the bladder, and the position of the nerve bundle 703 around the bladder are three-dimensionally expressed. Here, the form and position of the outer surface 701 of the bladder are information included in the second surface information. Even if FIG. 7 is not a three-dimensional ultrasound image, those skilled in the art will understand that such information (information of 701, 702, and 703) is included through the three-dimensional ultrasound image. Let's go.

  Referring to FIG. 3 again, the composite video generation unit 35 generates a composite video by matching the endoscopic video and the non-endoscopic video.

  Referring to FIGS. 1A and 1B again, the display device 50 displays the composite video generated by the composite video generation unit 35. The robotic surgery system 1, 100 performs image guidance by providing the synthesized video to a doctor who performs robotic surgery through the display device 50. The display device 50 is a device for displaying visual information to report information to the user, such as a general monitor, an LCD (Liquid Crystal Display) screen, an LED (Light Emitting Diode) screen, a scale display device, etc. including.

  On the other hand, referring to FIG. 3 again, the position matching unit 342 uses the first surface information and the second surface information to determine the position of the endoscope apparatus 10 (FIG. 1B) and the ultrasonic apparatus 20 (FIG. 1B). As long as the location continues to be matched in real time, different images of the same organ acquired with different devices at different locations will continue to be mapped. Accordingly, the composite video generation unit 35 continues to match the endoscopic video and the ultrasonic video, and continuously generates the composite video, so that the display device 50 can be used for the endoscope device 10 (FIG. 1B) and Regardless of the movement of the ultrasonic device 20 (FIG. 1B), the composite image can be displayed in real time.

  According to one embodiment, the display device 50 can display the generated composite video as it is, but depending on the usage environment of the robotic surgery system 1, 100, only a part of the region of interest included in the video information included in the composite video. Controlled to be displayed. That is, when the endoscopic image and the non-endoscopic image are combined, the display device 50 displays the position of the prostate 801 that is a part of the region of interest included in the non-endoscopic image and the nerve bundle 802. Only the position of is controlled to be displayed. Further, when the information about the positions of the prostate 801 and the nerve bundle 802 is preprocessed, the display device 50 has a specific part in the composite image and the prostate 801 and the nerve bundle 802 depending on the use environment of the robotic surgical system 1100. You can display the corresponding information together.

  FIG. 8 is a view illustrating a composite image according to an exemplary embodiment of the present invention. Referring to FIG. 8, the synthesized image shows that an ultrasound image having information on the internal and external tissue positions of the bladder, such as the prostate and nerve bundles, is synthesized with the bladder and surrounding endoscopic images. It is a drawing.

  FIG. 9 is a flowchart of a method for processing medical images according to an embodiment of the present invention. Referring to FIG. 9, the medical image processing method according to the present embodiment includes steps processed in time series by the medical image processing apparatus 30 of the robotic surgical system 1, 100 shown in FIGS. 1A, 1B, and 3. Is done. Therefore, even if the contents are omitted below, the contents described with respect to the drawings are also applied to the medical image processing method according to the present embodiment.

  In step 901, the video acquisition unit 32 acquires a medical video about an organ imaged using a plurality of different medical video imaging devices. In step 902, the surface information extraction unit 33 extracts the surface information of the organ included in each acquired medical image from each acquired medical image.

  In step 903, the video mapping unit 34 maps the medical video using each extracted surface information. In step 904, the composite video generation unit 35 generates a composite video in which the medical video is matched based on the mapping result.

  FIG. 10 is a detailed flowchart of a method for processing the medical image of FIG. Similarly, the contents described with reference to FIGS. 1A, 1B, and 3 are also applied to the medical image processing method according to the present embodiment, even if the contents are omitted below.

  In step 1001, the endoscope video acquisition unit 321 acquires an endoscope video imaged using the endoscope device 10 (FIG. 1B). In step 1002, the first extraction unit 331 extracts first surface information representing at least one of the position and form of the surface of the organ from the endoscopic image captured by the endoscope apparatus 10 (FIG. 1B). To do.

  In step 1003, the non-endoscopic video acquisition unit 322 acquires a non-endoscopic video imaged using a non-endoscopic device such as the ultrasound device 20 (FIG. 1B). In step 1004, the second extraction unit 332 extracts second surface information representing at least one of the position and form of the surface of the organ from the video imaged by the non-endoscopic device.

  Here, the start of step 1001 and step 1003 may be advanced in parallel at the same time, or any one step may be started first and advanced. That is, the progress of step 1001 and step 1002 and the progress of step 1003 and step 1004 are independently performed without any influence.

  In step 1005, the video mapping unit 34 maps the medical video using the extracted surface information. In step 1006, the composite video generation unit 35 generates a composite video in which the medical video is matched based on the mapping result.

  FIG. 11 is a flowchart of a process of extracting first surface information according to an embodiment of the present invention. In step 1101, the first extraction unit 331 acquires distance information between the external tissue and the endoscope apparatus 10 (FIG. 1B) about the organ and the periphery.

  In step 1102, the first extraction unit 331 generates a three-dimensional first surface model corresponding to the endoscope image using the acquired distance information. In step 1103, the first extraction unit 331 extracts first surface information from the generated first surface model.

  FIG. 12 is a flowchart of a process of extracting second surface information according to an embodiment of the present invention. In step 1201, the second extraction unit 332 acquires information about the boundary line representing the surface of the organ from the video imaged by the non-endoscopic device 20 (FIG. 1B).

  In step 1202, the second extraction unit 332 generates a three-dimensional second surface model corresponding to the surface of the organ using the acquired boundary line information. In step 1203, the second extraction unit 332 extracts second surface information from the generated second surface model.

  On the other hand, the above-described embodiment of the present invention can be created by a program executed by a computer, and is embodied by a general-purpose digital computer that operates the program using a computer-readable recording medium. The data structure used in the above-described embodiment of the present invention is recorded on a computer-readable recording medium through a plurality of means. The computer-readable recording medium is a magnetic recording medium (for example, ROM (Read Only Memory), floppy (registered trademark) disk, hard disk, etc.), or an optical reading medium (for example, CD-ROM, DVD, etc.). Such a recording medium is included.

  So far, the present invention has been described with a focus on preferred embodiments thereof. Those skilled in the art will appreciate that the present invention can be embodied in modified forms without departing from the essential characteristics of the invention. Accordingly, the disclosed embodiments should be considered in an illustrative, not a limiting sense. The scope of the present invention is shown not in the above description but in the claims, and all differences within the equivalent scope should be construed as being included in the present invention.

  The present invention is applicable to the technical field related to medical devices.

DESCRIPTION OF SYMBOLS 1 Robotic surgery system 10 Endoscope apparatus 11 1st medical imaging | video imaging device 20 Ultrasound apparatus 21 2nd medical imaging | photography apparatus 30 Medical imaging | video processing apparatus 31 Detection part 32 Image acquisition part 33 Surface information extraction part 34 Image mapping part 35 Synthesis Image generation unit 40 Surgical robot 50 Display device 321 Endoscope image acquisition unit 322 Non-endoscope image acquisition unit 331 First extraction unit 332 Second extraction unit 341 Comparison unit 342 Position matching unit

Claims (26)

Obtaining a medical image of a predetermined organ imaged using a plurality of different medical image capturing devices;
Extracting each surface information of the predetermined organ included in each of the acquired medical images from each of the acquired medical images;
Mapping the medical image using each of the extracted surface information;
Generating a composite image in which the medical images are matched based on the mapping result.   2. The extraction step according to claim 1, wherein information representing at least one of a position and a form of a surface of the predetermined organ is extracted as the surface information from each of the acquired medical images. Medical video processing method.   2. The mapping according to claim 1, wherein the mapping step maps the medical image by matching each position of the medical image capturing device using the extracted surface information. Medical video processing method. Further comprising the step of detecting the position of each of the medical imaging devices;
The mapping step includes:
Comparing each extracted surface information;
Matching the detected position of the medical imaging device based on the comparison result, and
The medical image processing method according to claim 1, wherein in the mapping step, the medical image is mapped based on the matching result.   The medical image processing according to claim 4, wherein the comparing step compares how the extracted surface information corresponds to the surface of the same part of the predetermined organ. Method.   The plurality of medical imaging apparatuses include at least one of an endoscope apparatus, an ultrasound apparatus, a computed tomography (CT) apparatus, a magnetic resonance imaging (MRI) apparatus, and a positron emission tomography (PET) apparatus. The medical image processing method according to claim 1, further comprising a non-endoscopic device.   The generated composite image is included in the image of the external tissue about the predetermined organ and the periphery included in the image captured by the endoscope device and the image captured by the non-endoscopic device. The medical image processing method according to claim 6, wherein the predetermined organ and the image of the internal and external tissues about the periphery are images that are simultaneously expressed three-dimensionally. The extracting step includes:
Extracting first surface information representing at least one of the position and form of the surface of the predetermined organ from an endoscopic image captured by the endoscopic device;
The method further comprises the step of: extracting second surface information representing at least one of the position and form of the surface of the predetermined organ from an image photographed by the non-endoscopic device. A medical image processing method according to claim 1. Extracting the first surface information comprises:
Obtaining distance information between an external tissue and the endoscopic device for the predetermined organ and surroundings;
Using the obtained distance information to generate a three-dimensional first surface model corresponding to the endoscopic image,
The medical image processing method according to claim 8, wherein the first surface information is extracted from the generated first surface model. Extracting the second surface information comprises:
Obtaining information about a boundary line representing the surface of the predetermined organ from an image captured by the non-endoscopic device;
Using the acquired boundary information to generate a three-dimensional second surface model corresponding to the surface of the predetermined organ,
The medical image processing method according to claim 8, wherein the second surface information is extracted from the generated second surface model. The step of acquiring the boundary line information includes:
The medical image according to claim 10, wherein the boundary line is obtained by applying at least one of line detection and edge detection to an image captured by the non-endoscopic device. Processing method.   The medical image processing method according to claim 1, wherein the predetermined organ is a surgical site to be treated by a surgical robot or an organ around the surgical site.   A computer-readable recording medium recording a program for causing a computer to execute the method according to claim 1. A video acquisition unit for acquiring a medical image of a predetermined organ imaged using a plurality of different medical video imaging devices;
A surface information extracting unit that extracts surface information of the predetermined organ included in each of the acquired medical images from each of the acquired medical images;
A video mapping unit that maps the medical video using each of the extracted surface information;
A medical video processing apparatus comprising: a composite video generation unit that generates a composite video in which the medical video is matched based on the mapping result.   The surface information extraction unit extracts information representing at least one of the position and form of the surface of the predetermined organ as the surface information from each of the acquired medical images. The medical image processing apparatus described.   The image mapping unit according to claim 14, wherein the image mapping unit maps the medical image by matching each position of the medical image capturing device using the extracted surface information. Medical video processing device. A detection unit for detecting the position of each of the medical imaging devices;
The video mapping unit
A comparison unit for comparing each of the extracted surface information;
The medical image processing apparatus according to claim 14, further comprising: a position matching unit configured to match the position of the detected medical image apparatus based on the comparison result.   The plurality of medical imaging apparatuses include at least one of an endoscope apparatus, an ultrasound apparatus, a computed tomography (CT) apparatus, a magnetic resonance imaging (MRI) apparatus, and a positron emission tomography (PET) apparatus. The medical image processing apparatus according to claim 14, further comprising a non-endoscopic apparatus.   The generated composite image is included in the image of the external tissue about the predetermined organ and the periphery included in the image captured by the endoscope device and the image captured by the non-endoscopic device. 19. The medical image processing apparatus according to claim 18, wherein the predetermined organ and the image of the internal and external tissues about the periphery are images that are simultaneously expressed three-dimensionally. The surface information extraction unit
A first extraction unit that extracts first surface information representing at least one of a position and a form of a surface of the predetermined organ from an endoscope image captured by the endoscope device;
A second extraction unit that extracts second surface information representing at least one of a position and a form of the surface of the predetermined organ from an image captured by the non-endoscopic device. The medical image processing apparatus according to claim 18.   The endoscopic device is a laparoscopic device, and when the non-endoscopic device includes the ultrasound device, the ultrasound device is a transrectal ultrasound (TRUS) device. The medical image processing apparatus according to claim 18.   The medical image processing apparatus according to claim 14, wherein the predetermined organ is a surgical site to be treated by a surgical robot or an organ around the surgical site. In a robotic surgery system that guides video about the surgical site and performs robotic surgery with a surgical robot,
An endoscopic device that captures medical images of a predetermined organ in a subject;
Non-inclusive of at least one of an ultrasound device, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, and a positron emission tomography (PET) device that images a medical image of the predetermined organ. An endoscopic device;
The medical images captured using the plurality of medical image capturing devices are acquired, and surface information of the predetermined organ included in each of the acquired medical images is obtained from each of the acquired medical images. A medical video processing device that extracts and maps the medical video using each of the extracted surface information, and generates a synthesized video in which the medical video is matched based on the mapping result;
A display device for displaying the generated composite video;
And a surgical robot for performing robotic surgery in response to user input.   24. The medical image processing apparatus extracts information representing at least one of a position and a form of a surface of the predetermined organ as the surface information from each of the acquired medical images. The described system.   The medical image processing device maps the medical image by using the extracted surface information to match the positions of the endoscope device and the non-endoscopic device. 24. The system of claim 23.   The system according to claim 23, wherein the predetermined organ is a surgical site to be treated by the surgical robot or an organ around the surgical site.