WO2013163391A1 - Surgical structured light system - Google Patents

Abstract

A Surgical Structured Light (SSL) system is disclosed that provides real-time, dynamic 3D visual information of the surgical environment, allowing registration of pre- and intra-operative imaging, online metric measurements of tissue, and improved navigation and safety within the surgical field.

Description

SURGICAL STRUCTURED LIGHT SYSTEM

RELATED APPLICATIONS

[0001] This application claims priority to the benefit of the filing date of U.S.

Provisional Patent Application No. 61/638,466 filed April 25, 2012 which is hereby incorporated by reference in its entirety.

BACKGROUND OF THE DISCLOSED SUBJECT MATTER Field of the Disclosed Subject Matter

[0002] The disclosed subject matter relates to improved 3D imaging. Particularly, the present subject matter described here is a Surgical Structured Light (SSL) system that includes a real-time 3D sensor that measures and models the surgical site during a procedure. The present subject matter provides real-time, dynamic 3D visual information of the surgical environment using a standard laparoscopic setup. This capability allows registration of pre- and intra-operative imaging, online metric measurements of tissue, and improved navigation and safety within the surgical field.

[0003] The systems and techniques disclosed herein are suitable for a myriad of applications and embodiments wherever depth perception can be a critical parameter such as minimally invasive surgeries, forensic applications, military applications, and law

enforcement.

Description of Related Art

[0004] Standard laparoscopes provide a 2D image of a surgical site, leaving the surgeon without depth perception and making surgery more difficult. Although current laparoscope technology can provide perceptual depth information, whereby the surgeon can view stereo cameras and the human brain fuses the images to perceive depth, currently no methods exist to computationally provide this information for the purposes of measurements and automation.

[0005] Surgeons routinely use a telescope and camera system to provide an image of the surgical site during minimally invasive surgery in the abdomen and in the chest. This delivers a crisp, high-resolution 2D image, but the lack of depth perception imposes the following limitations and constraints on the surgical team: limits and slows the surgeon's technical performance; fails to inform the surgeon about the spatial relationships of various organs and their components (i.e., blood vessels, ureters, bronchial tubes, etc.); prevents registration of intra-operative real-time images with pre-operative images; inadequate 3D geometric visualization with which to do intra-operative planning; prevents accurate intraoperative measurements; and creates a very long learning curve for surgeons.

[0006] Since 1990, surgeons have developed minimally invasive surgery (MIS) techniques by using an endoscope (laparoscope or thoracoscope) to visualize the surgical site in the abdomen or chest. These scopes, when combined with a high resolution camera, provide a crisp, bright image of the surgical site and eliminate the need for a large incision through which to view and complete the operation. However, this technology delivers a 2- dimensional (2D) image of a 3-dimension (3D) surgical site, and, because of the insertion site for the scope, the angle of view of the surgical site is severely limited. Although the surgeon can sometimes complete MIS with this technology, the loss of depth perception and ability to view the surgical site from multiple angles slows the surgeon, lengthens the learning curve, and, in some complex cases, prevents the surgeon from completing the operation without making a large incision.

[0007] The benefits of minimally invasive surgery are well established. Despite that, many surgeons have not learned to use these techniques, in large part because of the long learning curve associated with a 2D image. Technology that would routinely deliver 3D information to surgeons would not only increase the availability of minimally invasive surgery to more patients by shortening the learning curve, it would also lead to better surgical procedures.

[0008] Minimally invasive surgeries, such as laparoscopy and arthroscopy, have become standard of care surgical techniques because they require smaller incisions compared to traditional open surgery. Central to these surgical techniques are endoscopes with video cameras that allow surgeons to visualize interior body structures, as requisite for performing surgery.

[0009] The most common laparoscopic procedure in the United States is the cholecystectomy, which is the surgical removal of the gall bladder. Overall, the efficacy of laparoscopy varies slightly with the type of surgical procedure; cholecystectomy is typically the standard against which other procedures are judged. Recently, laparoscopic surgery has been proven to reduce the risk of hospital-acquired infections, which also shows that the shift from open surgery to minimally invasive methods is critically important for patients in the long term.

[0010] The laparoscopic surgery market focuses on the instrumentation used in the surgical procedure. Laparoscopic surgical products consist of three broad segments:

visualization, access, and resection instruments. Visualization consists of laparoscopes and cameras, access will include trocars, suction/irrigation, and insufflation systems, and resection will include scissors or forceps and direct energy tools such as ultrasonic and electrocautery scalpels and vessel sealers. The visualization segment of laparoscopic surgery products consists of laparoscopes and surgical cameras. Laparoscopes consist of a telescopic rod-lens system, a fiber optic cable, and a light source. [0011] In laparoscopic surgery, the components that are required at first to perform the procedure are the visualization instruments. A visualization instrument consists of the telescopic rod lens, a video camera, fiber optic cables, and a cold light source. Typically, the rod lens, fiber optic cable, and light source are combined in an integral medical instrument collectively known as the laparoscope, while the video camera remains a distinct, but inseparable element of the visualization system.

[0012] There is strong evidence that the outcomes of MIS are better than traditional open surgery for most abdominal and thoracic operations: reduced complications, much quicker recovery, and equal or better treatment of the underlying condition. The unmet need is an imaging system that would provide additional information with which the surgeon could more efficiently and successfully complete surgery using MIS techniques.

[0013] Current visualization techniques are wanting in their ability to provide depth information that can be crucial to surgeons. There is an urgent need for a system that allows surgeons to visualize internal body structured in three dimensions for more accurate and safe surgical procedures.

[0014] The presently disclosed subject matter meets the need for 3D imaging for minimally invasive complex intra-abdominal and intra-thoracic operations and significantly improve the experience of the surgeon and improves outcomes for patients.

SUMMARY OF THE DISCLOSED SUBJECT MATTER

[0015] The present subject matter described here is a Surgical Structured Light (SSL) system that includes a real-time 3D sensor that measures and models the surgical site during a procedure. The present subject matter provides real-time, dynamic 3D visual information of the surgical environment using a standard laparoscopic setup. This capability allows registration of pre- and intra-operative imaging, online metric measurements of tissue, and improved navigation and safety within the surgical field. By adding this 3D component, minimally-invasive surgery can be easier to learn and more widely used.

[0016] The presently disclosed subject matter provides a new imaging technology that provides 1) real-time 3D reconstructions of the surgery site, 2) unique, novel views not previously available using endoscopic imaging, and 3) metrology of various dimensions inside of a surgical site. The current gold standard for imaging during MIS provides only a 2D image by using a high definition (HD) video camera and a rod lens rigid endoscope.

[0017] The present disclosure alleviates the burden of conventional imaging systems which require the surgeon to view the 3D image in a console set away from the patient. The present subject matter uses modern computation capabilities to generate an image with which the surgeon can precisely measure the size of or distance between structures in the surgical field, visualize the relationship between various structures, and register the intra-operative image with preoperative images. These additional data points should improve the efficiency, safety, and overall quality of MIS.

[0018] Laparoscopy is a minimally invasive surgical technique used in a wide range of abdomen operations. The surgeon performs the procedure under the skin with the assistance of a camera system. Standard laparoscopes utilize 2D imaging techniques, which limit depth perception and can restrict or slow surgeon performance, create a long learning curve for new surgeons, and create problems during surgery. This technology gives the standard laparoscope real-time, dynamic 3D imaging capabilities of the surgical scene for routine and complex laparoscopic procedures. The added dimension can significantly improve surgeon experience and potentially improve outcomes for patients.

[0019] Further, by minimizing depth perception issues, 3D imaging can provide shorter surgery times and improved navigation and safety, thus promoting wider use of minimally invasive surgery. [0020] In an exemplary embodiment, the imaging system comprises a first camera; a second camera; a light source, the light source producing light at a frequency invisible to a human eye; a dispersion unit, the dispersion unit projecting a predetermined pattern of light from the invisible light source; an instrument, the instrument projecting the predetermined pattern of invisible light onto a target area; a band pass filter, the band pass filter directing visible light to the first camera and the predetermined pattern of invisible light to the second camera; wherein the second camera images the target area and predetermined pattern of invisible light, and computes a three-dimensional image.

[0021] Additionally, the first camera displays a visible image to an operator, and the light source produces a continuous ray of infrared light. In some embodiments, the three- dimensional image is computed from a single image of the target area. Advantageously, the second camera can be configured to take real time intra-operative images of the target area, and compare pre-operative images with intra-operative images.

[0022] In some embodiments, a RGB color is assigned to select locations of the second camera image, and a processor measures at least one length (e.g., depth) of the target area. In some instances the instrument is a laparoscope which includes a beam splitter disposed at an end thereof. The laparoscope can include first and second optical channels, wherein the first optical channel projects the predetermined pattern of light onto the target area, and the second optical channel receives all available light from the target area.

[0023] In accordance with another aspect of the disclosed subject matter, a method for creating a three dimensional image comprises providing a first camera; providing a second camera; generating a ray of light at a frequency invisible to a human eye; dispersing the ray of invisible light through a dispersion unit; projecting a predetermined pattern of invisible light onto a target area; receiving the predetermined pattern of invisible light and visible light from the target area; directing visible light to the first camera and the predetermined pattern of invisible light to the second camera; imaging the target area and predetermined pattern of invisible light; and computing a three-dimensional measurement.

[0024] In some embodiments, the ray of invisible light is infrared light and the three- dimensional measurement is computed from a single image of the target area. Additionally, the second camera can be configured to capture real time intra-operative images of the target area, and a processor measures at least one length of the target area.

[0025] In another exemplary embodiment, an imaging system comprising a first camera; a second camera; a light source, the light source producing light at a frequency visible to a human eye; a dispersion unit, the dispersion unit projecting a predetermined pattern of light of a known range of frequencies from the light source; an instrument, the instrument projecting the predetermined pattern of light of a known range of frequencies onto a target area; a beam splitter, the beam splitter directing a first ray of light towards the first camera and a second ray of light towards the second camera; a notch filter, the notch filter filtering the second ray of light to the predetermined pattern of light of a known range of frequencies; wherein the second camera images the target area and predetermined pattern of light to compute a three-dimensional image.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] A detailed description of various aspects, features, and embodiments of the subject matter described herein is provided with reference to the accompanying drawing, which is briefly described below. The drawings are illustrative and are not necessarily drawn to scale, with some components and features being exaggerated for clarity. The drawings illustrate various aspects and features of the present subject matter and may illustrate one or more embodiment(s) or example(s) of the present subject matter in whole or in part. [0027] Figure 1 provides a schematic view of one embodiment of the disclosed subject matter.

DETAILED DESCRIPTION

[0028] Structured light projects a known pattern of light into a scene and measures this pattern with a camera. By knowing the structure of the pattern of light received by the camera, it is possible to compute 3D information. However, implementing this in-vivo during laparoscopy is difficult. The appearance of the surgical site must be maintained while implementing a Surgical Structured Light (SSL) system. Projecting a pattern of visible light onto the surgical site during laparoscopy, which is allowed to be perceived by the surgeon, could undesirably alter the surgeon's interpretation of the image and confuse or distract the surgeon. By analyzing how the light pattern is distorted by the 3D environment, structure can be accurately deduced in real-time using computational techniques.

[0029] The present subject matter provides for a functional SSL for use during laparoscopy, that maintains the view of the surgical site without distractions. In one exemplary embodiment, and in order to minimize distraction, a frequency of light is used that is not perceived by the human eye, such as infrared (IR) light. Using this method, dense 3D information is captured from a "single shot", meaning only one image of the projected pattern is necessary to estimate 3D measurements. In one embodiment, an IR laser light source is diffracted with a special grating with a known pattern and an IR camera images the scene with the projected pattern and allows for a processor within the user interface to compute a depth image for presentation to the surgeon. In this exemplary embodiment, because the pattern is in the IR wavelength, the human eye cannot see the pattern and the view of the scene containing the projected pattern with a white light camera is uninterrupted.

Additionally, an RGB color can be assigned to each 3D point in order to recover a photorealistic 3D reconstruction in real-time. [0030] This capability allows for registration of pre- and intra- operative imaging, online metric measurements of tissue, and improved navigation and safety within the surgical field. By adding this 3D component minimally-invasive surgery would be easier to learn and more widely used.

[0031] The present disclosure provides a laparoscope with twin parallel optical channels to both transmit and receive light information to and from the scene and allow for triangulation. A beam splitter is placed at the tip of the device. One optical channel projects an IR pattern which is obtained from a laser point source dispersed from a special optical grating. The other optical channel receives all light from the scene, and a bandpass filter sends the white-light to a standard color camera and the IR light to an IR camera. The white light camera displays the "normal" image to the surgeon, while the IR camera uses the projected IR pattern to reconstruct the scene in 3D, in real-time. Further, real-time texture mapping of the geometry using the 2D viewing textures can be performed since the images are already registered (i.e. the system can identify which particular pixel(s) imaged a 3D location, thereby allowing for the 3D image to be combined with the 2D image).

[0032] In another exemplary embodiment, the system can employ high-frequency white light "bursts" which would not be seen by the human eye, but could still be imaged by fast cameras, and would not distract the surgeon. The frequency of such bursts can be varied, as so desired, and is bounded only by the specifications of the particular camera model chosen for a given application.

[0033] In yet another exemplary embodiment, the system can employ a

predetermined band of visible light to create the projected pattern instead of the invisible light embodiment (e.g. IR) as described above. In this configuration a notch filter is included which serves to remove this patterned band of visible light from the physician's view. [0034] The imaging systems and techniques disclosed herein meet the need for 3D imaging for minimally invasive complex intra-abdominal and intrathoracic operations and significantly improve the experience of the surgeon and potentially improve outcomes for patients. This disclosure allows for live, dynamic 3D measurement capabilities of a surgical scene in real-time. It has been proven to work better than stereo imaging processing and has a large host of applications. This technology can be used for surgery, but can additionally be made smaller to work in other parts of the body, with or without a laparoscope, and applied to different types of medical procedures where 3D information "on-the fly" is helpful.

[0035] The disclosed subject matter provides a new approach to manufacturing and designing endoscopes that overcomes current limitations of visualization techniques used in minimally invasive surgeries by employing a system that visualizes internal body parts in three dimensions. A pattern of structured light, either in the infrared spectrum or a reduced band of visible light in conjunction with a notch filter, is projected on the tissue of interest by the endoscope. The pattern is detected by the accompanying camera system and subsequently converted to three-dimensional images that can be viewed by the surgeon. This technology can be used for visualizing internal structures in all areas of medicine and beyond, where depth perception can be a critical parameter, including in all minimally invasive surgeries, forensic applications, military applications, and law enforcement.

[0036] In one embodiment, the Surgical Structured Light system of the present disclosure may be constructed using a DLP Pico Projector (Texas Instruments/DigiKey), coupled with an endoscope for insertion. In other embodiments, a small diameter flash light is used. In some embodiments, the flashlight includes a cone at the end to make it into a point source. In some embodiments, a transparency is provided with a pattern to project. In some embodiments, the pattern is placed at the end of an endoscope. In some embodiments, a pattern is projected and an high-dynamic range (HDR) camera is used to view it so that the pattern is visible to the HDR sensor for reconstruction, but can be easily filtered out to a lower bit-depth image for viewing by the surgeon so the surgeon's view is not obstructed by the pattern. In some embodiments, a camera and projector combination is used that both images the scene and projects a pattern at a frequency that is not detectable to the human eye, again so the surgeon surgeon's view is not obstructed by the pattern.

[0037] Additional embodiments of the present subject matter use a genetic algorithm to evolve the best pattern to project. In some embodiments, the genetic algorithm uses an HDR grayscale camera to: take the desired image without any projected patterns; design the ideal pattern (high frequency random noise, vertical stripes, etc); start out half-way between the ideal image and the ideal pattern; evolve new patterns (pixel values can range b/w 0-255); the fitness function simply deconvolves and then creates an 8-bit image and weights error from the ideal image after deconvolution along with how close the evolved pattern is to the ideal pattern. The final result will image a projected pattern on a scene in HDR format, deconvolve with the pattern, reduce to 8-bit format, and be able to perform structured light on the HDR image, all while retaining the ability to show the filtered image without the pattern included therein.

[0038] Laparoscopy, part of the larger class of surgical procedures known as endoscopy, is minimally invasive surgery through small incisions on the abdomen. It is often cheaper and less stressful for minor surgeries than laparotomies, which use significantly larger incisions. Typical procedures entail surgical tools, an imaging device, and a light source being inserted through the small incisions into a working space created by

pressurizing the abdomen with C0₂.

[0039] Laparoscopy is widely used to conduct surgeries for hernia, abnormal growth removal, biopsy, organ removal (gallbladder, appendix, ovaries, etc.), etc. as well as diagnostic procedures to check for infertility, abnormal growths, chronic pelvic pain, chronic pelvic inflammatory disease, endometriosis, ectopic pregnancy, tube/ovary torsion, etc.

[0040] Standard laparoscopes, either (1) telescopic rod lens systems connected to a video camera or (2) digital systems with lens replaced with a charge-coupled device, generally provide 2D imagining of the surgical site. Inherently, these images lack depth perception information and can slow surgical performance.

[0041] Conceptually, the structured light techniques disclosed herein project a known pattern of light into a scene and measure the pattern with a camera. 3D information can be computed by looking at the structural changes in the light seen by the camera.

[0042] In accordance with an aspect of the disclosure, the Surgical Structure Light

(SSL) system disclosed herein adds 3D imaging capability to laparoscopic procedures by adding a structured light source and detector to the standard laparoscope.

[0043] In an exemplary embodiment, infrared (IR) light is chosen, for the SSL platform because patterned visible light could interfere with the surgeon's view of the surgery site on the 2D image. A laparoscope with twin parallel optical channels provides light (visible and structured IR) transmission and reception to and from the surgical site while allowing for triangulation. The IR light is delivered via a dispersion unit containing a special optical grating to produce the pattern to which the IR camera compares the captured IR light in order to generate the 3D image.

[0044] All light received back from the surgical site (i.e., visible light, and invisible structured IR light) is split by a band pass filter to its respective visible or IR camera. The IR system color-codes its output (to denote depth) in real-time to display a 3D rendition of the surgical scene alongside the standard 2D image. Furthermore, real time texture can be added to the 2D image since the 3D images are already registered. [0045] Additional embodiments of the present subject matter use high-frequency white light "bursts" instead of IR light for 3D imaging. The bursts are not be detectable by the human eye, i.e. no surgeon distraction, but can be imaged by high-speed cameras.

Additionally or alternatively, the system can employ a predetermined band of visible light to create the projected pattern and a notch filter which serves to remove this patterned band of visible light from the physician's view.

[0046] The present subject matter has many applications including: Laparoscope adaptation for laparoscopic procedures; endoscope adaptation for 3D imaging during other endoscopic procedures; when configured on a smaller scale (e.g. no laparoscope) the SSL system can be used in medical monitoring applications where real time 3D imaging is helpful; 3D sensing for other applications such as video gaming systems, robotics, filming, interactive control, vehicle sensors, etc.

[0047] The present subject matter leads to better surgical procedures by enabling the following: faster operations; prevention of inadvertent injuries of structures adjacent to the surgical site; precise registration of intra-operative images with preoperative images; intraoperative planning; and real-time, accurate intraoperative measurements.

[0048] In one embodiment, the improved 3D laparoscope of the present disclosure comprises a lens splitter, an IR light source (e.g. laser or LED) with a dispersion unit (e.g. grating for laser, or mask for LED) and an imaging head. The system of the present disclosure may project any of several patterns that maximize information content/retrieval. In other embodiments, the system of the present disclosure can use any of several wavelengths of light to create the projected pattern (including ultra-high speed white light projections, as well as a band of visible light in conjunction with a notch filter) that are not perceived by the surgeon's eye. [0049] One embodiment of the present disclosure comprises software including registration and modeling software to take the 3D point clouds, build meshes from these clouds, and texture map the 2D imagery onto the models. This software is operated by processors housed within the user interface which includes a display for the surgeon's viewing of the surgical site.

[0050] The system of the present disclosure may be used in in-vivo animal environments and provides optimal: 1) accuracy of reconstructions, 2) frame rate for reconstruction of models, and 3) interface with the surgeon. The present disclosure is useful to healthcare providing institutions (hospitals, specialty surgical centers, etc.) providing endoscopic treatments. It is also useful in other fields including surveillance equipment (forensic, law enforcement, and military applications) and manufacturing applications.

[0051] The present disclosure provides for real-time, dynamic 3D visual information of the surgical environment using a standard laparoscopic setup. Although current laparoscope technology can provide perceptual depth information, these technologies require the surgeon to view stereo cameras and mentally compile a variety of images together so as to fuse a complex virtual image in his/her brain in order to perceive depth information. Often this is a time consuming and error prone endeavor which can distract the surgeon from safely operating the surgical tools, as well as lead to undue surgeon fatigue.

[0052] Prior to the present disclosure no methods existed to computationally provide this information for the purposes of measurements and automation. Attempts had been made to use stereo vision with computer vision techniques to estimate dense 3D information, however the inevitable existence of texture-less areas and significant lighting challenges make this an incredibly difficult, if not impossible, endeavor to perform robustly in a commercial environment. The system disclosed herein overcomes these drawbacks and provides for a 3D image generation with greater spatial density, and thus more accurate modeling of the surgical site.

[0053] The present disclosure solves the aforementioned shortcomings of

conventional laparoscopic imaging systems by using a technique called Structured Light (SL), whereby a known pattern of light is projected into the scene and then imaged with a camera system. By analyzing how the light pattern is distorted by the 3D environment, anatomical structure can be accurately deduced in a real-time sense using computational techniques. One concern may be that the projected light could potentially distract the surgeon by disturbing the visual scene, and this has been the approach taken in the past. The present subject matter alleviates this by projecting light which is not perceived by the surgeon (e.g. in the infrared (IR) wavelength; high frequency white light bursts; or band of visible light coupled with a notch filter) thereby making the projected information invisible to the human eye. There is precedence for using IR light in-vivo, however these procedures were directed towards fluorescence techniques for visualizing tumors, and did not provide the structured light reconstruction of 3D images, as disclosed herein.

[0054] One embodiment of the present disclosure includes a hardware setup that begins with a dual, parallel optical channel laparoscope. Some existing laparoscopes use this dual channel technology for perceptual stereo, however the device disclosed herein includes a removable head at the imaging end which allows for 3D reconstruction and metrology functions to these existing dual channel laparoscopes. The system 10 of the present disclosure, as shown in Fig. 1, provides a removable head piece 110 which takes the place of the cameras at the tip of conventional dual channel laparoscopes. In the exemplary embodiment illustrated, the removable head piece 110 is positioned external to the body. A light source 600 is provided for generating light at a non-visible frequency (e.g., IR light) which is passed through a dispersion unit 500 that is configured with a predetermined structure to diffract the IR light into a known pattern 500'. In some embodiments the light source 600 can be a laser and the corresponding dispersion unit 500 includes a dispersion grating. In other embodiment the light source can be a light emitting diode (LED) and the corresponding dispersion unit is configured as a dispersion mask. A first channel 102 of the laparoscope is configured to project this non-visible patterned light into the body and thus serves as a Projection Channel. A second channel 104 is configured for imaging light from the body and thus serves as an Imaging Channel. Using the Projection Channel 102, a predetermined IR light pattern is projected through the laparoscope into a target area at the surgical scene 20. In addition to this projected IR light, the visible light (the source of which can be ambient operating room lighting and/or a directed beam, e.g., flashlight) for inspection by the physician is captured by the system disclosed herein. Therefore, the system and methods disclosed herein employs a first camera 300 for capturing the patterned invisible or undetected light, and second camera 400 for capturing the visible light (e.g., IR and white light color cameras, respectively). These cameras are in communication with the user interface 700 which houses a processor(s) for computing and displaying 3D images and measurements to the surgeon.

[0055] To facilitate the precise placement of both cameras compactly, a beam splitter

120 is positioned at the tip of the laparoscope 110. A beam splitter 120 is an optical device which is capable of splitting a beam of light into a plurality of discrete rays. Therefore, all of the light which exits the body through the Imaging Channel 104 first goes through the beam splitter 120. The beam splitter 120 splits the light into two separate beams. One beam 120' goes to a standard, high-definition white light camera 400 which is used for normal viewing by the surgeon. The white light camera 400 is not sensitive to IR light and so the projected pattern remains invisible to the surgeon viewing the procedure via the images presented by the white light camera 400. The other beam 120" is passed through an IR band-pass filter 200 which is designed to filter out all visible wavelengths of light and only let pass-through non- visible IR light. This causes only the patterned IR light to pass to the IR camera 300, which is then able to view the distorted view of the IR pattern which was passed through the

Projection Channel 102.

[0056] In another exemplary embodiment, the IR light source 600 of Fig. 1 is substituted with a light source which produces a ray of visible light. Additionally, the band pass filter 200 is substituted with a notch filter. Notch filters, also commonly referred to as band-stop or band-rejection filters, can transmit most wavelengths with little intensity loss while attenuating light within a specific wavelength range (the stop band) to a very low level. Accordingly, notch filters are effectively the inverse of bandpass filters, which offer high in- band transmission and high out-of-band rejection so as to only transmit light within a small wavelength range. The notch filter creates a restricted region over a given range wavelengths. In an exemplary embodiment, the notch filter of the presently disclosed SSL system is configured to restrict wavelengths within the blue spectrum, however alternative portions of the spectrum can be restricted if so desired. In some embodiments the notch filter is positioned within the beam splitter 120 wherein the light is both split and filtered

simultaneously. In other embodiments, the notch filter is disposed between the beam splitter 120 and the camera 300. The use of a visible light source in combination with a notch filter is advantageous in that it affords the same quality and resolution of imagining, while alleviating the need for specialized, and costly, equipment such as an IR camera.

[0057] By knowing the positions and orientations of all cameras and light projectors, it is possible to reconstruct the 3D information in the scene densely using triangulation techniques in computer software. In addition, because there is a white light color camera 400, it is also possible to assign RGB color information to each 3D scene point, thereby producing a photo-realistic 3D reconstruction. Additionally, the texture of the target site 20 can be automatically registered with the depth data since the same optical channel 104 is used for both depth and texture. Accordingly, the system disclosed herein provides a Surgical Structured Light (SSL).

[0058] In accordance with another aspect of the disclosed subject matter, a processor executing software can be provided which encompasses both the algorithms required to reconstruct the scene with the observed imagery as well as the user interface (UI) display 700 which is presented to the surgeon. The UI is such that the introduction of 3D information can be seamlessly blended into the surgeon's procedural routines without effort. In one embodiment, 2D imagery is combined with the dense 3D information, which can take the form of a side-by-side view of 2D and 3D imagery. Additionally, a button (not shown) on the laparoscope 110 which can switch between the views on-demand, or a blending of the 2D video onto the 3D structure to view both images simultaneously, if so desired. As would be understood by one of ordinary skill, the UI design will differ depending on the particular application of the disclosed structured light system (e.g., assisted navigation may require a different presentation than a virtual ruler).

[0059] There are various uses of such a 3D imaging system. Although the disclosed

SSL system is not a perceptual system, it is possible to provide real-time 3D measurements of individual points in the scene 20. This can be useful during a procedure when the surgeon needs to know (quickly) the precise distance between two locations in the body. Therefore, the SSL would be a seamless means of performing such a measurement because the software has 3D locations of all visible points in the scene. As such, SSL effectively provides a virtual ruler without the injection of any physical tools into the body. This could be especially useful if the surgeon is performing a biopsy at a location pre-determined from pre-operative imaging techniques to a location which is known metrically, but isn't visibly obvious in the image. [0060] Another useful application is in registering the intra-operative 3D video available via the SSL system to pre-operative images. This is a powerful tool for the surgeon as she tries to identify structure in- vivo. It requires accurate geometry, and the SSL system provides dense 3D information from which this can be extracted. Because it is possible to compute photo -realistic 3D information, one could also potentially provide novel viewpoints during fine-scaled manipulations, for example, in the case where a side-view is more useful in navigating a tool to a site, and this is currently impossible since all views must come from the point-of-view of the camera. Another possible use is to perform very accurate small surface reconstructions of anatomy. Lesions, tumors and other surface anomalies may not be identifiable with the human eye, and the reconstructions provided for via the SSL system may be able to recover very dense surface models that can be used to identify abnormalities in- vivo.

[0061] There are numerous additional applications for this type of real-time dense 3D information is obtainable. Surgical robotics is an obvious domain for application of the SSL system, whereby more automation may be possible with the existence of better 3D measurement capabilities. The present subject matter overcomes the limitations currently enforced by using 2D information from cameras.

[0062] In some embodiments, the system of the present disclosure integrates the software for 3D reconstruction and User Interface with the device. The system of the present disclosure can be used to perform surgical mockup experiments to evaluate the system in advance of animal and human use.

[0063] The present disclosure includes design of optical system to allow dense 3D reconstruction, and an intuitive and simple user interface to allow the surgeon to access the 3D data as needed. The system of the present disclosure may be combined with a surgeon education program educating surgeons on the new technology. [0064] While the disclosed subject matter is described herein in terms of certain preferred embodiments, those skilled in the art will recognize that various modifications and improvements may be made to the disclosed subject matter without departing from the scope thereof. Moreover, although individual features of one embodiment of the disclosed subject matter may be discussed herein or shown in the drawings of the one embodiment and not in other embodiments, it should be apparent that individual features of one embodiment may be combined with one or more features of another embodiment or features from a plurality of embodiments.

[0065] In addition to the specific embodiments claimed below, the disclosed subject matter is also directed to other embodiments having any other possible combination of the dependent features claimed below and those disclosed above. As such, the particular features presented in the dependent claims and disclosed above can be combined with each other in other manners within the scope of the disclosed subject matter such that the disclosed subject matter should be recognized as also specifically directed to other embodiments having any other possible combinations. Thus, the foregoing description of specific embodiments of the disclosed subject matter has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosed subject matter to those embodiments disclosed.

It will be apparent to those skilled in the art that various modifications and variations can be made in the method and system of the disclosed subject matter without departing from the spirit or scope of the disclosed subject matter. Thus, it is intended that the disclosed subject matter include modifications and variations that are within the scope of the appended claims and their equivalents.

Claims

CLAIMS What is claimed is:

1. An imaging system comprising: a first camera; a second camera; a light source, the light source producing light at a frequency invisible to a human eye; a dispersion unit, the dispersion unit projecting a predetermined pattern of light from the invisible light source; an instrument, the instrument projecting the predetermined pattern of invisible light onto a target area; a band pass filter, the band pass filter directing visible light to the first camera and the predetermined pattern of invisible light to the second camera; wherein the second camera images the target area and predetermined pattern of invisible light to compute a three-dimensional image.

2. The system of claim 1, further comprising a user interface, wherein the user interface configured to display two dimensional and three dimensional images.

3. The system of claim 1, wherein the light source produces infrared light.

4. The system of claim 1, wherein the light source produces a continuous ray of light.

5. The system of claim 1, wherein the three-dimensional image is computed from a single image of the target area.

6. The system of claim 1, wherein the second camera takes real time intra-operative images of the target area.

7. The system of claim 6, wherein the system compares pre-operative images with intraoperative images.

8. The system of claim 1, wherein a RGB color is assigned to select locations of the second camera image.

9. The system of claim 1, wherein a processor measures at least one length of the target area.

10. The system of claim 9, wherein the processor calculates a depth of the target area.

11. The system of claim 1, wherein instrument is a laparoscope.

12. The system of claim 11, wherein the laparoscope includes a beam splitter disposed at an end thereof.

13. The system of claim 11, wherein the laparoscope includes first and second optical channels.

14. The system of claim 13, wherein the first optical channel projects the predetermined pattern of light onto the target area.

15. The system of claim 13, wherein the second optical channel receives all available light from the target area.

16. A method for creating a three dimensional image comprising: providing a first camera; providing a second camera; generating a ray of light at a frequency invisible to a human eye; dispersing the ray of invisible light through a dispersion unit; projecting a predetermined pattern of invisible light onto a target area; receiving the predetermined pattern of invisible light and visible light from the target area; directing visible light to the first camera and the predetermined pattern of invisible light to the second camera; imaging the target area and predetermined pattern of invisible light; and computing a three-dimensional measurement.

17. The method of claim 16, wherein the ray of invisible light is infrared light.

18. The method of claim 16, wherein the second camera takes real time intra-operative images of the target area.

19. The method of claim 16, wherein a processor measures at least one length of the target area.

20. An imaging system comprising: a first camera; a second camera; a light source, the light source producing light at a frequency visible to a human eye; a dispersion unit, the dispersion unit projecting a predetermined pattern of light from the light source; an instrument, the instrument projecting the predetermined pattern of light onto a target area; a beam splitter, the beam splitter directing a first ray of light towards the first camera and a second ray of light towards the second camera; a notch filter, the notch filter filtering the second ray of light to the predetermined pattern of light; wherein the second camera images the target area and predetermined pattern of light to compute a three-dimensional image.