JP2011510705A - Imaging system for common bile duct surgery - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform an inadvertent damage to a living body structure even by an experienced surgeon as a result of laparoscopic surgery requiring extra training and limitations in use of this technique.
A method and apparatus for imaging CBD features of a patient introduces a fluorescent contrast agent into the CBD. The light source sends visible and fluorescent light through the laparoscope to the patient's abdominal cavity. A camera attached to or integrated with the laparoscope detects the visual light image and the fluorescence image. The visual light image and the fluorescent image signal are processed so that the fluorescent light emission image signal and the visual image signal are combined into a single display signal. The system adjusts the display characteristics of the fluorescent image, such as color, so that the contrast between the fluorescent image and the visual light image is good. As a result, the surgeon can easily distinguish these two images. The display signal is sent to a video monitor, where the surgeon views the visual light image and the fluorescent image as a single overlaid image.

Description

  The present invention relates to a surgical imaging system. In particular, it relates to an imaging system for gallbladder surgery.

The methods described herein are approaches that can be implemented, but are not necessarily methods previously conceived or pursued. Thus, unless specifically stated otherwise, any method described herein should not be considered prior art simply because they are included herein.
Currently, gallbladder surgery is performed using laparoscopic techniques. During this type of surgery, the surgeon inserts several tubes called trocars or ports into the abdominal cavity. A 10 mm diameter optical scope (laparoscope) is inserted into one of the ports. The laparoscope is connected to a video camera so that surgeons and surgical teams can see the interior of the abdominal cavity on the screen. An elongated instrument is inserted into the other port to hold, incise, and excise the tissue.

  Laparoscopic surgery requires extra training to work with new instruments and operate using a two-dimensional view of the surgical field. As a result of the limitations in use of this technique, even experienced surgeons will inadvertently damage the structure of the body at a higher rate than in open surgery. The most serious complication of gallbladder surgery occurs when the surgeon inadvertently injures or resects the common bile duct (CBD). This complication occurs in 1/200 surgery (0.5%) in the United States. Thus, out of nearly 800,000 laparoscopic gallbladder surgeries performed annually in the United States, approximately 4,000 patients suffer from CBD injury.

The present invention is shown by way of example and not by way of limitation. In the accompanying drawings, like reference numerals refer to similar elements.
FIG. 2 shows a surgical instrument used during laparoscopic surgery according to one embodiment of the present invention. It is a figure which shows the anatomical structure of a bile duct. It is a figure which shows an intraoperative cholangiogram (IOC). It is a block diagram which shows the add-on structure of the common bile duct imaging system by one Example of this invention. 1 is a block diagram showing a stand-alone configuration of a general common bile duct imaging system according to a position example of the present invention. FIG. FIG. 2 shows an optical layout for a laparoscopic illumination system according to one embodiment of the present invention. FIG. 3 shows an optical layout for a laparoscopic camera system according to one embodiment of the present invention. 1 is a block diagram illustrating the implementation of a prior art surgical instrument for injecting fluid into the gallbladder. FIG. 3 is a block diagram illustrating overlaying a visual light image with a fluorescent emission image on a single display according to one embodiment of the present invention. FIG. 11 is a block diagram that illustrates a computer system on which an embodiment may be implemented. 1 shows a laparoscope with an integrated light source, according to one embodiment of the invention.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
In the discussion that follows, like numerals refer to like parts from several perspectives when referring to the drawings.

In the present specification, examples are described according to the following outline:
1.0 General overview
2.0 System structure overview
3.0 Example technologies and processes
3.1 Common bile duct imaging system
3.2 Optical layout of imaging system
4.0 Common bile duct fluorescence and display
5.0 Implementation Mechanism-Hardware Overview
1.0 General Overview The embodiments of the present invention summarized above are described in more detail, along with some alternative embodiments of the present invention. The embodiments of the present invention described below are described with respect to common common bile duct (CBD) laparoscopic surgery, although other applications of laparoscopic surgery may be performed in alternative embodiments of the present invention. Similar operations can be performed on those that can be performed and performed in common common bile duct laparoscopic surgery.

  In one example, a fluorescent contrast agent is introduced into the CBD by direct injection into the gallbladder, gallbladder duct, CBD, or intravenous injection and excretion of the contrast agent into the bile by the liver. The light source illuminates the light path of the laparoscope. The light source sends both visible and infrared (IR) light (also known as fluorescence excitation light) through the laparoscope to the patient's abdominal cavity. Fluorescent contrast agents are excited by narrow band light energy and produce light emission in a certain wavelength band. The laparoscopic camera assembly can be communicatively connected to the camera controller via an electronic cable or wirelessly via Bluetooth (or any wireless technology) or wireless LAN. The camera assembly includes a visible light detection camera and an infrared detection camera. A camera connected to the laparoscope detects a visible light image and a fluorescence emission image.

  The visible and fluorescent image signals from the camera assembly are combined into a single display signal so that the two images are properly aligned and overlaid (or combined). It is combined with. This system allows the surgeon to easily distinguish between the two images by adjusting the display characteristics of the fluorescent image, such as color, to adjust the contrast of the visual light image.

  The display signal is sent to a video monitor where the surgeon views the visible light image and the fluorescent image as a single overlaid (or combined) image. The surgeon can display the fluorescent image in the desired color and instruct the system to properly contrast the fluorescent image with the visible image.

  The overlay image can be switched by the user via a switch or software control. This system can operate multiple displays with different combinations of images. A sensor can be included in the camera housing to allow the user to know which orientation is ground or sky. This allows the surgeon to select the camera orientation to sky or ground and display them. This is very useful in NOTES type surgery (described later).

  This system can record a combined visual and fluorescent image on an external or internal digital recording device such as a CD, DVD, optical disc, hard disk or flash memory. The system has an Ethernet connection that allows an Internet or intranet connection so that recording to a server is possible or records can be transmitted over the Internet or an intranet.

2.0 System Structure Overview Referring to FIG. 1, during gallbladder surgery, a surgeon inserts several tubes 101 (referred to as trocars or ports) into the abdominal cavity as described above. A 10 mm diameter optical scope (laparoscope 102) is inserted into one of the ports. The laparoscope is connected to a video camera to allow surgeons and surgical teams to see the interior of the abdominal cavity on a video screen. An elongated instrument 103, 104, 105 holds, incises, and excises tissue through other ports.
FIG. 2 shows an anatomical view of the bile duct. The common bile duct (CBD) 201 carries bile for digestion from the liver 202 to the intestine 203. The gallbladder 204 is a lateral sac that stores bile and is squeezed into the CBD 201 during a meal. The gallbladder 204 is connected to the CBD 201 by a gallbladder duct 205. The gallbladder duct 205 must be clearly identified, iterated or ligated by the surgeon and then cut with scissors. If the surgeon mistakes CBD 201 for gallbladder duct 205, CBD injury occurs. If a surgeon uses electrocautery energy to coagulate bleeding near the CBD 201, the CBD 201 may be injured.

  Currently, the only way to view CBD is to look at intraoperative cholangiogram (IOC). This requires using an overhead or portable fluoroscopic device to place the catheter into the gallbladder duct 205 during surgery, inject X-ray contrast fluid, and see the X-ray contour formed by the dye It is. This shows the shape and course of CBD 201 and the bile tree. As shown in FIG. 3, the IOC image 301 is displayed on a black and white screen and can be printed or saved. IOC execution is not considered a standard practice. Because of the cost, time and trouble of doing this, the IOC is not performed in every surgery. In addition, X-ray exposure is a problem for patients and surgical staff. Patients are exposed to X-ray radiation for patient-specific surgical procedures. However, operating room staff are exposed each time this type of procedure is performed. Nevertheless, a study found that when surgeons performed IOC, their patients had half the CBD injury compared to patients who did not perform IOC.

  It is clear that in order to safely perform gallbladder surgery, the surgeon needs to see CBD. However, it is present 1-3 mm below the adipose tissue and peritoneum, so it cannot be seen directly. The safest and most useful way to view CBD is to give the surgeon “live”, in real time (or near real time), CBD location and course images during surgery (essentially in real time IOC) Is to provide. This allows the surgeon to always be aware of the location of the CBD, avoiding him accidentally or intentionally damaging the CBD. This has not been done in the past because there was no reliable and simple way to visualize CBD deep from the visible surface during laparoscopic surgery. In one example, the bile duct is imaged during gallbladder surgery and the CBD image is shown to the surgeon on a real-time display. This device can reduce the CBD damage rate by at least 50%. This helps nearly 2,000 patients per year (in the US) to avoid suffering from CBD injury and immediately identify and avoid CBD, thus facilitating all processing It can be made.

3.0 Example Technology and Process
3.1 Common Bile Duct Imaging System Examples can be constructed as either an add-on to the laparoscopic system or an integrated stand-alone system. In this example, the surgeon can see the CBD in its correct position during gallbladder surgery.
FIG. 4 illustrates one embodiment of an add-on configuration that is integrated into a current laparoscopic light source and video system. The fluorescence imaging module 404 introduces IR light 407 via the existing fiber optic illumination system 403 and the light channel of the laparoscope 406. An infrared detection coupler 405 is added to the laparoscope 406 between the laparoscope and the visible light camera 409. Infrared detection coupler 405 includes a camera that can detect IR signals from fluorescent labels or tissue autofluorescence. The infrared detection coupler 405 is communicatively connected to the fluorescence imaging module 404. The connector may be an electronic cable, fiber optic cable, wireless transfer system or any combination and / or quantity. The connecting cables can be disposable or reusable, and if the connecting cables come into contact with a sterile surgical field, they need to be sterilized. Alternatively, the IR image can be transmitted to a remote IR camera (which can be located near the fluorescence imaging module 404) via an optical path. This is necessary if the dimensions of the IR camera are not compatible with the size specification of the coupler 405.
During surgery, the light source 403 transmits visible light through the optical fiber of the laparoscope 406. The fluorescence imaging module 404 transmits infrared light (fluorescence excitation light) through the optical fiber of the laparoscope 406 at the same time. The camera controller 402 receives a visible light image signal (actual field of view of the surgical area using visible light) from an existing camera 409 mounted on the laparoscope 406. The camera controller 402 processes the visible light image signal into the visible image display signal and sends the visible image display signal to the fluorescence imaging module 404.

  The fluorescence imaging module 404 receives an IR (fluorescence) light image signal from the infrared detection coupler 405. This processes the fluorescent image signal along with the visible image display signal received from the camera controller 402 to produce a video output signal 408 that includes a real-time overlay (or combination) of the fluorescent image signal and the visible image display signal. Generate. The fluorescence imaging module 404 processes the fluorescence image signal using a computer system or a dedicated microprocessor to create a CBD satisfactory and natural graphic display. The fluorescence imaging module 404 can overlay two images in proper alignment using any known technique for combining a fluorescence image signal and a visible image display signal into a single display signal. . This can include simple reliability for general focus points. Here, two cameras are aligned before surgery and two image signals are in a simple manner, or in two image signals in order to properly align the two images Combined using software that automatically detects potential reference points.

  The video output signal 408 is sent to the video monitor 401 where the surgeon views the visible light image and the fluorescent image as a single superimposed image. The surgeon can display the fluorescent image in a desired color, shape or texture and instruct the fluorescent imaging module 404 so that the fluorescent image has an appropriate contrast with the visible image.

Alternatively, if an add-on system is required to replace one of the components of the current laparoscopic system with either the camera head 409, camera controller 402 or light source 403 There is also. In this case, the system interfaces with the remaining components at either the input or output of these devices.
FIG. 5 is a stand-alone CBD imaging system that integrates an infrared source, infrared detection system, visible light source and visible image detection system into a complete and stand-alone laparoscopic imaging system with improved optical capabilities. An example is shown. In this embodiment, both the visible light camera and the infrared camera are incorporated into a single camera housing 505 that is attached to or integrated with the laparoscope 504. The light source 503 transfers visible light and infrared light to the optical fiber in the laparoscope 504. Alternatively, the visible light source and infrared source can be integrated into the laparoscope 504 itself or to a trocar that eliminates the fiber optic cable required to connect the camera controller 502 to the laparoscope 504. As a result, the laparoscopic assembly can be made lighter and easier to handle.

  During normal surgery, the camera controller 502 instructs the light source 503 to illuminate the light path of the laparoscope 504. The light source 503 sends both visible and infrared light to the laparoscope 504 (upon request). The camera controller 504 receives a signal from the camera in the laparoscope 504. The camera assembly 505 is communicatively connected to the camera controller 504 via electronic cable, fiber optic cable, blue suce Bluetooth (or any wireless technology), or via a wireless local area network or a combination thereof. be able to. The camera assembly 505 includes a visible light detection camera and an infrared detection camera. The camera in the laparoscope 504 detects a visible light image and a fluorescence emission image. Alternatively, the camera assembly 505 can include other types of detectors that can achieve visible and fluorescent image detection as a camera.

  Visible light image and fluorescent image signals from camera assembly 505 are processed by camera controller 504. As with the fluorescence imaging module described above, the camera controller 504 can combine any fluorescence image signal and visible image signal into a single display signal to overlay the two images with proper alignment. Known techniques can also be used. In this case, since the two cameras are in an integral camera assembly 505, the parallax of these cameras is negligible and their alignment can be done at the factory. These two image signals are then combined in a simple manner.

The camera can also be aligned to a common focus before surgery. Alternatively, software that automatically detects common reference points in the two image signals can be used to properly align the two images.
The camera controller 504 adjusts the display characteristics of the fluorescent image, such as color, so that it makes a good contrast with the visible light image so that the surgeon can easily distinguish between the two images. adjust.

The display signal is sent to the video monitor 501, where the surgeon views the visible light image and the fluorescent image as a single overlaid image. The surgeon can instruct the camera controller 504 to display the fluorescent image in the desired color, shape, and texture so that the fluorescent image has an appropriate contrast to the visible image.
Alternatively, the ultraviolet, visible (range) or infrared fluorescence detector can be the same CCD device used for the detection of visible light. A single CCD can also be used to detect infrared and visible light. The CCD can be controlled by a controller circuit that allows detection of the emitted light signal, either simultaneously or alternating with visible light (interlace detection). This detection may require the use of passive or active filters and switching mechanisms. The visible light and fluorescence emission light signals from the CCD are then transferred to an electronic circuit that separates the fluorescence emission signal and the visible light signal for digital processing.

  Current laparoscopic cameras typically have three (red, blue and green) CCD chips. In order to detect both visible and fluorescent emission light signals, a three-chip device can be used as described in the previous paragraph. Another alternative is to use a four-chip, five-chip or more laparoscopic camera. Such a camera includes a 3CCD chip for visible light detection and a CCD added for fluorescent emission detection. Another dedicated CCD added for the detection of this fluorescent emission will be optimized to detect infrared, near-infrared, visible light or ultraviolet wavelength light. For this detection, active or passive filtering and switching controllers will be required. The advantage of this is that the fluorescence detector can be activated and filtered separately from the visible light controller.

  In the case of a single-chip, 3-chip, 4-chip, 5-chip or more chip laparoscopic camera system, this entire assembly is permanently attached via a retractable housing at the visual end. Laparoscope, endoscope, thoracoscope or by designing to be a component part of the endoscope head or by placing it at the tip of the endoscope in a compact and “tip tip” configuration It will be attached to any of the cystoscopes. In all of these cases, a separate ring, separator or beam splitting box is not required. All optical operations will be performed within the camera housing. The camera will be connected to its controller box via an electronic cable or wirelessly via Bluetooth (or other wireless technology) or a wireless local area network.

  FIG. 11 shows an embodiment of a laparoscope 1101 to which an integrated light source 1102 and a camera 1103 are attached. The light source 1102 can be a standard white light bulb, filtered light, lamp, LED, laser, or the like. The light source 1102 can be powered by an electrical cord, a built-in battery, or inductive coupling. Inside the body of the light source 1102, the lens system shapes the light beam into a wide angle or a narrow angle depending on the user's choice.

The light source may be essentially cylindrical and have a plurality of planes around its circumference. Since this shape is flat, it has resistance to rolling and can prevent the light source 1102 from rolling off the table when it is not attached to the laparoscope 1101.
3.2 Imaging System Optical Layout Referring to FIG. 6, in one embodiment, light from a broadband light source 602 (which is used in a visual camera) is used for navigation and targeting of CBD 603, 604. Combined with light. The purpose is to identify the CBD during gallbladder surgery or removal of the gallbladder so that it is not physically damaged. The light used to identify the CBD can be single color or multicolor consisting of a number of single color light sources. This may be random polarization, linear polarization or circular polarization. The light source may be coherent or non-coherent. The light source may be a constant wave or a pulse wave.

  The light source is in the housing 601 (light box). As the light sources are combined using a beam splitter or combiner 605, they are directed to an optical fiber 606 (which can be a bundle of fibers). The optical fiber 606 has one end connected to the optical box 606 and the other end connected to a laparoscopic connector. The laparoscopic connector has a fiber bundle attached to it. As the optical signals enter the fiber, they pass through the laparoscope and exit to the patient. The light signal illuminates the abdominal cavity, more specifically the gallbladder, CBD and nearby organs. This is the excitation path. A number of light sources can be used in the light box 601. The light box 601 allows for the excitation of multiple fluorescent dyes or autofluorescent tissues with simultaneous or rapid transitions. If the wavelengths of two or more light sources overlap, the overlapping light sources must be triggered alternately in order for the associated camera to detect the appropriate fluorescent image. If there is no overlap, the light sources can be illuminated simultaneously.

  Organic materials have optical properties that are unique to that individual material. The system uses these unique characteristics to identify the CBD. In this case, the system detects CBD bile fluorescence, CBD bile with fluorescent dye added to the bile or autofluorescence of the tissue itself. Organic materials absorb photons and emit photons at longer wavelengths (the wavelength at which Stoke is shifted). This release is called fluorescence emission, which can be collected by a laparoscope. In the laparoscope shaft, there is a series lens in the length direction of the tube. Referring to FIG. 7, light is collected by the first lens and relayed to the other end of the device. From here, the light exits and can be accessed by the detector. In this case, the light is split by the beam splitter 704 and directed to a special filter that blocks all light except the fluorescence emission 703. In this case, these emissions are in the form of an image. The laparoscope also collects visible light as an image that can be viewed by the human eye or preferably by a camera. Visible light is split by a beam splitter 704 and directed to a camera that detects a visual image 702. Individual chips can be cameras, and the number of cameras can vary depending on the application. A focusing assembly (not shown) can be placed in front of the cameras 702, 703 to correct any beam distortion that occurs in the optical path. The light is split into a specific camera by a beam splitter.

The compact housing 701 is optional and can be used in the standalone embodiment described in FIG. 5 above. If the add-on embodiment described in FIG. 4 is used, the compact enclosure 701 cannot be executed.
The two images (visible and fluorescent images, ie navigation and targeting images) are superimposed on each other in real time so that the surgeon can see the CBD and not damage it.

4.0 Common common bile duct fluorescence and display
The CBD imaging system includes the following steps:
1. Injection of fluorescent contrast material into CBD.
2. Excitation of the phosphor using a light source that can be in the ultraviolet, infrared or visible range.
3. Detect fluorescence.
4. Process the fluorescence image to remove artifacts and scatter.
5. Display live images of surgery and fluorescent images with a clear CBD position displayed to the surgeon on the monitor in real time.
The fluorescent agent injected into the common bile duct of the contrast agent may be any agent that fluoresces in the ultraviolet, visible, or infrared (IR) region. The drug can be Indocyanine Green (ICG), fluorescein, methylene blue, isosulfan blue, or any new fluorescent or color-based visualization medium and optically active substance that becomes a label. If the fluorescent agent is excreted in bile (such as ICG), the fluorescent agent can be administered intravenously before or during surgery. In the case of ICG, a management kit with a biomarker, a biocompatible solution for injection, and the necessary tubing and instructions is provided. The time of intravenous administration of ICG is 40-60 minutes before the start of surgery. The ICG can be injected as part of a chemical “cocktail” that can optimize, enhance or change the optical properties of the ICG.
With reference to FIGS. 2 and 8, alternatively, the fluorescent contrast agent can be injected into the CBD by direct injection into the gallbladder 204, gallbladder duct 205 or CBD 201. Injecting the drug into the gallbladder away from the CBD is easy, has the advantage that it does not require any conventional incision and is safe. A specialized device 801 exists (as described in US Pat. No. 5,224,931) to inject fluid into the CBD next to the gallbladder 802.

  As another alternative, the novel laparoscopic instrument can be used specifically to inject fluorescent contrast material into the gallbladder. Such a device can have a 5 mm diameter shaft, a jaw holding the gallbladder, and a channel for injecting fluorescent material. This injection tube can be separate from the jaw, or when grasping the gallbladder, the fluorescent material can be injected directly into the gallbladder without spillage (like snake bites) The jaw can be traversed.

  The fluorescent substance can also be introduced into the CBD via the gallbladder duct 205 (duct connecting the gallbladder to the CBD) instead of via the gallbladder 204. To do this, the cystic duct is freely dissected in a standard manner relative to a standard surgical cholangiogram (IOC). An IOC catheter is fixedly placed in the gallbladder duct and contrast agent is injected into the gallbladder duct and then into the CBD. This last example would result in CBD imaging. However, this requires successful analysis of previous gallbladder ducts, so if that analysis is not so much, before CBD is imaged, the patient is in some of the risky treatments. Will be exposed.

Fluorescent contrast agent excitation agents from light energy sources can be excited by light of different wavelengths, including ultraviolet (UV), visible, and infrared (IR) wavelengths. The energy source can be one or more broad spectrum lamps, one or more lasers, or one or more light emitting diodes (LEDs). Here, the light source is a narrow band energy source. Typically, a narrow wavelength band in the ultraviolet, infrared, or visible region is used to excite specific fluorescent molecules. The narrowband energy source can be part of the laparoscopic light source or can be placed in a separate housing in various ways used to direct light directly to the tissue.

As described above, the narrowband energy source is coupled to the laparoscopic light source by the optical coupling box, thus coupling visible light and narrowband light within the existing fiber optic cable connected to the laparoscope. Alternatively, a narrowband light source projects light onto tissue via a completely separate illumination system, such as a second laparoscope, a special optical probe, or via one or more optically active trocars be able to. A narrow band light source can generate one or more narrow wavelengths of light energy, and its intensity and wavelength can be adjusted by the user.
If no fluorescent agent is used, a narrow band energy source can be used to change the type of visible light projected onto the surgical area. One or multiple wavelengths of light (with or without white light) can be used to illuminate the surgical area. This effect can be used to enhance the contrast, depth and valve color of various tissues, depending on their optical reflection properties, absorption properties and autofluorescence. When projecting light from one or more separate light sources (instead of or in addition to a laparoscope), the user can change the color of the light to improve depth perception and achieve various shadowing effects, Intensity and spatial distribution can be controlled and varied. This requires a special electronic controller box and the user can use a joystick, switch or knob to control the lighting elements described above. Changing the spatial distribution of the light source to use a combination of various colors and light intensities helps the surgeon to perceive depth and tissue color.

  In one embodiment, the ICG injection combined with the light generated from the infrared laser / LED light source of the light box (above) allows the existing laparoscopic examination without the need to add an infrared light detection camera to the laparoscope Generate enough fluorescence that the camera system can image. Thus, this embodiment would add an ICG injection / instrument delivery and infrared laser / LED light box as described above without the need to add any camera system. The surgeon will be able to view the surgical area using existing laparoscopic cameras and monitors, and also view a fluorescent image of the CBD within the surgical area.

Contrast Agent Detection Fluorescent contrast agents are excited by narrow band light energy and produce light emission in a certain wavelength band. This released energy can be captured by an endoscope, laparoscope, thoracoscope, cystoscope, surgical microscope or a second optical probe introduced into the body cavity for this purpose. The light energy passing through the capture device described above is then separated by a beam demultiplexer or other optical filtering device directed to the detector, if necessary. Methods for detecting fluorescent contrast agents have been described above. The filter can be used to filter undesired wavelengths of light from the collected light energy to improve the detection of fluorescent material. The filter can be static or variable and can be controlled by an electronic controller.

Fluorescence Image Processing Once detected and converted to a digital signal, the fluorescence emission signal passes through a microprocessor or computer to extract a critical tissue image. This process uses software algorithms to enhance the image, change the size, shape and surface pattern of the image, change the color of the image, and / or change the image to computer generated graphics. All these parameters can be user adjustable, can be adapted, or can be set to a predetermined combination of choices to accommodate different user preferences.

Live Surgical Image and Fluorescent Image Display In FIG. 9, a series of images showing a visible light image and a fluorescent image are shown. The digital output from the processed fluorescence signal is digitally combined with the visible light image to create a seamless overlay of both images. Image combining and / or overlaying can be performed by computer or microprocessor software. Parameters for the overlay and presence of each image layer are user selectable.

  The combined visible / fluorescent image is displayed on an existing standard laparoscopic CRT, display, video monitor, flat panel display, projector, or head mounted display. The combined digital image is output in a format compatible with standard monitors in today's market. The overlay image can be switched on or off by the user by a switch or software control (alternatively it can also be voice driven). The overlay image provides an image in a manner that allows the surgeon to see the position of the CBD via fluorescence emission from any normal viewing angle while working. Some surgeons may prefer to display two monitors with and without an overlay image. This system can handle multiple displays with different combinations of images. This system displays an overlay image with a visible light image shown in picture-in-picture mode where one image can be shown as the main image and the other image as a smaller image on the sub-picture display You can also.

A visible light image 901 of the tissue for the common bile duct and artery is shown without overlay. Fluorescence image 902 of the common bile duct and artery is also shown without overlay. The two types of images alone do not convey enough information to the surgeon. The combination of the two images allows the surgeon to draw what is under the tissue and the tissue itself. The normal common bile duct image and the improved common bile duct image are displayed together with the 903 surgical image in a natural overlaid manner. Thus, the surgeon can see the CBD even though the tissue is on top. The surgeon can now use the overlaid image to avoid damaging the CBD.
When the image overlay is activated, the visible light image can change color and / or intensity to enhance the fluorescent image. The user can change the fluorescent image to any desired color. Since the fluorescence image shows only the fluorescence in the main body cavity, the image quality can be easily improved.

  Software for image processing allows the user to configure and control the CBD visualization system before, during, and after surgery. Control can be performed by a computer keyboard, special keypad, touch screen, treadle, voice control, head-up display, and the like. This control can be provided to a surgeon in a sterile room, such as a plastic cover, as a foot pedal on the floor or used by a traveling nurse in a non-sterile setting.

  Computers used for digital processing of images and control of image detection are visible and fluorescent connected to external or internal digital recording devices such as CDs, DVDs, optical discs, hard disks or flash memory. Software and hardware for recording images can be included. The ability to print statically combined images on photographic paper can be included in the system. The system can provide an Ethernet connection to allow an Internet or intranet connection so that records can be made to a server or transmitted over the Internet or intranet for training purposes.

Application of NOTES
NOTES® (Natural Orifice Translumenal Endoscopic Surgery) was developed several years ago to address the following concepts:
1) Realize the benefits of non-invasive surgery by reducing recovery time;
2) experience less physical discomfort associated with conventional treatment; and
3) Virtually no visible scars from this type of surgery.
All of these benefits encouraged research and research to advance and develop new equipment and techniques for physicians and researchers to use during NOTES procedures.
As a specific example, in natural orifice surgery, the gallbladder may be removed through the mouth. The doctor inserts a tube into the esophagus, makes a small incision in the stomach or digestive tract to gain access to the abdominal cavity, and removes the organ by the same route. Some surgeries may be performed via the rectum, vagina, urethra or bladder as well.

One of the main problems with NOTES surgery is spatial orientation and visualization. This is because the visible axis adopted by the flexible endoscope changes while being inserted into the peritoneal cavity. Furthermore, the quality of the visible image of an endoscope is usually inferior to that of a standard laparoscopic system.
During NOTES gallbladder surgery, the surgeon can use a descending approach to remove the gallbladder by incising the gallbladder toward the single tissue where this critical tubule structure is located. In this regard, if the surgeon can clearly see the position of the common common bile duct, he can safely ligate or clip the pedicle and finish the operation in less time and less effort be able to. Alternatively, visualization and manipulation are limited by the current NOTES system, so visualization of the common common bile duct during gallbladder and arterial incisions would be useful during NOTES gallbladder surgery. In any case, clear visualization of the common common bile duct makes NOTES gallbladder surgery faster and safer for patients.

  In one embodiment, the common bile duct vision system operates in the same manner as the laparoscopy application described above. The fluorescence excitation light is introduced into the fiber optic system of the flexible endoscope. A beam splitter collar attached to the endoscope and a separate fluorescence excitation camera system will be used to capture the fluorescence image. The fluorescent image will be processed and presented to the surgical team in both images: overlay mode, picture-in-picture, or side-by-side format (all described above). In the fully integrated NOTES platform, the fluorescence excitation light source and camera are integrated into the endoscopic instrument system. In NOTES applications, ICG or other fluorescent or color labels will be introduced into the common common bile duct either by intravenous injection prior to surgery or by direct injection into the gallbladder during surgery. Direct injection could be done with existing endoscopic needle catheters, percutaneous needles or newly designed devices or catheters for injection.

  In one embodiment, the displayed image is magnified in a two-dimensional plane. This embodiment displays a three-dimensional image to the surgeon. The surgeon or assistant can rotate the image using a remote control or using a command device such as a laparoscope, endoscope, thoracoscope, cystoscope, etc.

5.0 Hardware Overview FIG. 10 is a block diagram that illustrates a computer system 1000 upon which an embodiment of the invention may be implemented. Computer system 1000 includes a bus 1002 or other communication mechanism for communicating information, and a processor 1004 coupled with bus 1002 for processing information. Computer system 1000 also includes main memory 1006, such as random access memory (RAM) or other dynamic storage device coupled to bus 1002 for storing information and instructions executed by processor 1004. Main memory 1006 can also be used to store temporary numeric variables or other intermediate information during execution of instructions executed by processor 1004. Computer system 1000 further includes a read only memory (ROM) 1008 or other static storage device coupled to bus 1002 for storing static information and instructions for processor 1004. A storage device 1010, such as a magnetic disk or optical disk, is provided and is coupled to the bus 1002 for storing information and instructions.

  Computer system 1000 is coupled to display 1012 via bus 1002, such as a cathode ray tube (CRT), projection, head mounted display, or flat panel display for displaying information to a computer user. be able to. An input device 1014 that includes alphanumeric characters and other keys is coupled to the bus 1002 for communicating information and command selections to the processor 1004. Another type of user input device is a cursor control 1016, such as a mouse, trackball or cursor direction key, that communicates direction information and command selections to the processor 1004 and controls cursor movement on the display 1012. is there. The input device typically has two axes, a first axis (eg, x) and a second axis (eg, y) that allow the device to define a position in the plane. Have a degree of freedom.

  The invention is also related to the use of computer system 1000 for performing the techniques described herein. According to one embodiment of the invention, these techniques are performed by computer system 1000 in response to processor 1004 executing one or more sequences of one or more instructions contained in main memory 1006. The Such instructions can be read into main memory 1006 from a medium that can be read by another machine, such as storage device 1010. Execution of the sequence of instructions contained in main memory 1006 causes processor 1004 to execute the process steps described herein. In alternative embodiments, hardwired circuitry can be used to implement the present invention or in combination with software instructions. Thus, embodiments of the invention are not limited to a specific combination of hardware circuitry and software.

  As used herein, the term “machine-readable medium” refers to any medium that provides data using the machine for surgery in a particular manner. In an embodiment performed using computer system 1000, various machine-readable media are involved, for example, in providing instructions to processor 1004 for execution. Such a medium may take many forms, including but not limited to storage media and transmission media. Storage media includes both non-volatile media and volatile media. Non-volatile media includes optical or magnetic disks, such as storage device 1010. Volatile media includes dynamic memory, such as main memory 1006. Transmission media includes coaxial cable, including wire with bus 1002, copper wire and fiber optic. Common forms of machine-readable media are, for example, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, CD / DVD, any other optical media, punch cards , Paper tape, any other physical media having a hole pattern, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip, or cartridge, or any other medium that can be read by a computer.

  Various forms of machine-readable media are involved in having the processor 1004 execute one or more sequences of one or more instructions. For example, the instructions can be initially held on a magnetic disk of a remote computer. A remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1000 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. The infrared detector can receive data held in the infrared signal, and appropriate circuitry can place the data on the bus. Bus 1002 holds data in main memory 1006, from which processor 1004 retrieves and executes instructions. The instructions received by main memory 1006 may optionally be stored on storage device 1010 either before or after execution by processor 1004.

  Computer system 1000 also includes a communication interface 1018 that is coupled to bus 1002. Communication interface 1018 provides a bi-directional data communication coupling to network link 1020 that is connected to local network 1022. For example, the communication interface 1018 may be an integrated services digital network (ISDN) card or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, communication interface 1018 may be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link can also be implemented. In any type of implementation, communication interface 1018 sends and receives electrical, electromagnetic, and optical signals that hold digital data streams representing various types of information.

  The network link 1020 typically provides data communication to other data devices via one or more networks. For example, the network link 1020 can provide a connection to the host computer 1024 via the local network 1022 or to a data instrument operated by an Internet service provider (ISP) 1026. ISP 1026, in turn, provides a data communication service, now commonly referred to as the “Internet” 1028, through a global packet data communication network. Both the local network 1022 and the Internet 1028 use electromagnetic or optical signals that hold digital data streams.

The computer system 1000 can send messages and receive data including program codes via the network, network link 1020 and communication interface 1018. In the specific example of the Internet, the server 1030 can transmit the requested code to the application program via the Internet 1028, ISP 1026, local network 1022, and communication interface 1018.
The received code can be executed by processor 1004 as it is received and / or stored in storage device 1010 or in other non-volatile storage for later execution. . In this aspect, the computer system 1000 can obtain the application code in the form of a carrier wave.

  In the foregoing description, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, what indicates the invention and what is intended as the invention is the only specific form in which such claims may be patented, including future amendments. And exclusively, it is only the claims of this application. Any definitions expressly set forth herein for terms contained in such claims shall be governed by the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense.

401 video monitor
402 Camera Controller
403 light source
404 Fluorescence Imaging Module
405 Infrared detector coupler
407 Infrared light output
408 Video output with overlay image
409 Existing camera
501 video monitor
502 Camera controller with IR detector
503 Light source of white light and IR light
601 Box with light source
602 Broadband UV-VS-IR light source
603 Laser light source
604 solid state light source (eg LED)
605 Beam splitter or combiner
606 Optical fiber that sends light to the patient directly or through a laparoscope
701 Compact housing including navigation and targeting camera and visual camera
702 visual camera (shown in RGB, but BGR is also possible)
703 Navigation and targeting camera that can also be a fluorescence detection camera
901 Normal image of the tissue about the common bile duct
902 IR laser image of bile duct subsurface
903 merged images
1002 bus
1004 processor
1006 Main memory
1008 ROM
1010 Enclosure
1012 display
1014 Input device
1015 Cursor control
1018 Communication interface
1022 Local network
1024 hosts
1030 servers

Claims (39)

A method for imaging features of a patient's common bile duct (CBD) comprising the following steps:
Introducing at least one fluorescent agent into the CBD of the patient;
Receiving visual image information of the patient's CBD from at least one visual image sensor attached to the laparoscope;
Receiving fluorescence image data of the patient's CBD from at least one fluorescence image sensor attached to the laparoscope;
Processing the visual image data and the fluorescent image data;
Creating composite image data by combining the visual image data and the fluorescent image data; and transmitting the combined image data to a first display.   The method of claim 1, wherein the creating step excludes the visual image data or the fluorescent image data from the combined image in response to a user command.   The method of claim 1, wherein the transmitting step transmits the visual image data simultaneously to a second display.   The method of claim 1, wherein the transmitting step transmits the visual image data simultaneously to the first display in a picture-in-picture manner.   The method of claim 1, wherein the creating step enhances the characteristics of the fluorescent image data so that imaging of the fluorescent image data is highlighted. Capturing the visual image data and the fluorescence image data through the laparoscope inserted into a patient;
Transferring visual light and fluorescence excitation light through the laparoscope; and
The method of claim 1, further comprising: allowing a user to adjust illumination characteristics of the visible light and fluorescence excitation light used to capture the visual image data and / or the fluorescence image data.   The method of claim 1, further comprising storing the combined image data on a storage device. A device that images the features of the patient's common bile duct (CBD),
A module for introducing at least one fluorescent agent into the CBD of the patient;
A module for receiving visual image information of the patient's CBD from at least one visual image sensor attached to the laparoscope;
A module for receiving fluorescence image data of the patient's CBD from at least one fluorescence image sensor attached to the laparoscope;
A module for processing the visual image data and the fluorescent image data;
An apparatus comprising: a module for creating composite image data by combining the visual image data and the fluorescent image data; and a module for transmitting the combined image data to a first display.   9. The apparatus of claim 8, wherein the creating module excludes the visual image data or the fluorescent image data from the combined image in response to a user command.   The apparatus of claim 8, wherein the module for transmitting transmits the visual image data simultaneously to a second display.   9. The apparatus of claim 8, wherein the module for transmitting transmits the visual image data simultaneously to the first display in a picture-in-picture.   The apparatus of claim 8, wherein the creating module enhances the properties of the fluorescent image data so that imaging of the fluorescent image data is highlighted. A module for capturing the visual image data and the fluorescence image data via the laparoscope inserted into a patient;
A module for transferring visual light and fluorescence excitation light to the patient's abdominal cavity via the laparoscope; and the visual light and fluorescence used by a user to capture the visual image data and / or the fluorescent image data The apparatus of claim 8, further comprising: a module for enabling adjustment of illumination characteristics of the excitation light.   14. The apparatus of claim 13, further comprising a module for storing image data coupled to a storage device. A computer readable medium carrying a sequence of one or more instructions for imaging a feature of a patient's common bile duct (CBD), wherein execution of the one or more of the sequences of instructions by one or more processors is performed. The following steps on one or more said processors:
Introducing at least one fluorescent agent into the CBD of the patient;
Receiving visual image information of the patient's CBD from at least one visual image sensor attached to the laparoscope;
Receiving fluorescence image data of the patient's CBD from at least one fluorescence image sensor attached to the laparoscope;
Processing the visual image data and the fluorescent image data;
A computer readable medium for performing the steps of creating composite image data by combining the visual image data and the fluorescent image data; and transmitting the combined image data to a first display.   The computer readable medium of claim 15, wherein the creating step excludes the visual image data or the fluorescent image data from the combined image in response to a user command.   The computer-readable medium of claim 15, wherein the transmitting step simultaneously transmits the visual image data to a second display.   The computer-readable medium of claim 15, wherein the transmitting step transmits the visual image data simultaneously to the first display in a picture-in-picture manner.   The computer-readable medium of claim 15, wherein the creating step enhances the properties of the fluorescent image data so that imaging of the fluorescent image data is highlighted. Capturing the visual image data and the fluorescence image data through the laparoscope inserted into a patient;
Transferring visual light and fluorescence excitation light through the laparoscope; and
The computer readable medium of claim 15, further comprising enabling a user to adjust illumination characteristics of the visible light and fluorescence excitation light used to capture the visual image data and / or the fluorescent image data. .   21. The computer readable medium of claim 20, further comprising storing the combined image data on a storage device. A system that images the features of the patient's common bile duct (CBD) during surgery:
At least one sensor for detecting a visual light image mounted on the laparoscope;
At least one sensor for detecting a fluorescence image mounted on the laparoscope;
A receiver for receiving visual image information from the at least one sensor for detecting a visual light image and receiving fluorescent image data from the at least one sensor for detecting a fluorescent emission image;
A processor for receiving visual image data and fluorescent image data from the receiver, extracting CBD image data from the fluorescent image data, and combining the visual image data and the CBD image data into display data And a display processor communicatively connected to the processor for processing display data received from the processor for a display of a display device communicatively connected. A wireless transmitter mounted on the laparoscope and communicatively connected to the at least one sensor for detecting a visual light image and the at least one sensor for detecting a fluorescence emission image In addition,
The wireless transmitter transfers visual image information received from the at least one sensor for detecting a visual light image and received from the at least one sensor for detecting a fluorescence emission image. 23. The system of claim 22, wherein the system transfers fluorescent image data.   The receiver receives visual image information from the at least one sensor for detecting a visual light image and detects a fluorescent emission image by any combination of at least one cable or at least one fiber optic cable 23. The system of claim 22, wherein fluorescence system data is received from the at least one sensor for performing.   24. The system of claim 23, wherein the receiver receives visual image data and fluorescence image data from the wireless transmitter.   23. The system of claim 22, wherein the receiver is communicatively connected to the at least one sensor for detecting a visual light image and the at least one sensor for detecting a fluorescent emission image.   23. The system of claim 22, further comprising a storage device for storing display data received from the processor.   23. The system of claim 22, further comprising a light source that emits visible light and fluorescence excitation light through the laparoscope to the abdominal cavity of the patient.   30. The system of claim 28, wherein the light source is any combination of at least one laser, at least one filtered light, at least one lamp, or at least one LED.   30. The system of claim 28, wherein the properties of the visible light and / or the fluorescence excitation light of the light source are adjustable by a user.   23. The system of claim 22, wherein the processor adjusts display characteristics of the CBD image data to distinguish the CBD image data from the visual image data of the display data. A system that images the features of the patient's common bile duct (CBD) during surgery:
A fluorescence imaging module comprising at least one sensor for detecting a visual light image configured to be mounted on a laparoscope;
A receiver for receiving fluorescent image data from the fluorescent image module and receiving visual image information from an existing visual image sensor attached to the laparoscope;
A processor for receiving visual image data and fluorescent image data from the receiver, wherein the processor extracts CBD image data from the fluorescent image data and overlays the CBD image data on the visual image data; and communication connection A display processor communicatively coupled to the processor for processing the overlayed display data received from the processor for a display of a display device being operated. A wireless transmitter mounted on the laparoscope and connected to the fluorescent image module by communication;
The wireless transmitter transfers visual image information received from the fluorescent image module;
33. The system of claim 32.   35. The system of claim 32, wherein the receiver receives visual image information from the fluorescent image module by any combination of at least one cable or at least one fiber optic cable.   34. The system of claim 33, wherein the receiver receives fluorescent image data from the wireless transmitter.   33. The system of claim 32, further comprising a storage device that stores data received from the processor and overlaid.   The processor of claim 32, wherein the processor adjusts display characteristics of the CBD image data to distinguish the CBD image data from the visual image data when the CBD image data is overlaid on the visual image data. system.   33. The system of claim 32, further comprising a light source that emits visible light and fluorescence excitation light through the laparoscope to the abdominal cavity of the patient.   40. The system of claim 38, wherein the light source is any combination of at least one laser, at least one filtered light, at least one lamp, or at least one LED.

Images