JP2011504782A - Elimination of irradiation crosstalk during imaging using multiple imaging devices - Google Patents

Abstract

  The system includes a plurality of scanning devices and light receivers that allow display of a plurality of images of the site using output signals generated in response to light from the light receivers. In order to avoid crosstalk caused by a receiver that receives light emitted by multiple scanning devices, light of different wavelength bands is applied to different scanning devices to filter the received light or to simultaneously It can be applied to one scanning device and multiplexed either between planes or between pixels, or the light supplied to each scanning device can be modulated and the received light can be demodulated, so that a single An image is generated in response to light from the scanning device. Expensive elements such as laser light sources, photodetectors, controllers, and processors can be shared by multiple imaging devices to minimize the cost of the imaging system.

Description

  The present invention relates to removal of irradiation crosstalk during imaging using a number of imaging devices.

  In minimally invasive therapeutic procedures, many of the instruments used are designed to pass through the channels in the flexible endoscope, i.e. fit within the lumen and advance to the distal end of the flexible endoscope. The An endoscope can provide an image to observe while a physician uses the instrument to perform its intended function. The general idea in designing therapeutic instruments currently used for such treatment is to be compatible with the available flexible endoscopes, i.e. the cross-sectional dimensions of the instrument are substantially flexible endoscopes. It must be smaller than the mirror and configured to be usable when passing through a working channel, or lumen, contained within the flexible endoscope. This limitation on the dimensions of instruments that can be used for minimally invasive treatments tends to limit the types of instruments that can be used, making it even more difficult to work with such instruments. Various types of diagnostic or therapeutic devices that could otherwise be used to treat patients undergoing minimally invasive therapy are free of dimensional constraints and other problems associated with the use of the device, while endoscopy If fit within the mirror's working channel, it would be useful for such treatment.

  Thus, various types of instruments or other types of elements can be used for minimally invasive treatments without requiring the dimensions to be small enough to pass through a conventional endoscope or other small guide conduit. It is desirable to develop a different approach. While such instruments may be used to perform functions at internal sites imaged separately by an endoscope, the approach generally involves passing the instrument percutaneously into the patient's body and then to the desired site for use. Requires a separate incision for the instrument so that it can be advanced. Since an instrument can be inserted into an internal site using a catheter or conduit, it would be practical to provide an alternative approach to imaging the path followed by the catheter or conduit. The new approach should place more emphasis on the use of instruments, conduits, and / or catheters within the patient's body rather than imaging using conventional endoscopes at the site. In order to broaden the use of instruments, catheters, and conduits, it is preferable to implement different approaches to imaging internal sites either at the distal end of the device or just proximal to the distal end. The imaging that is required to provide the field of view in which the device is used should be provided by means other than conventional endoscopes. It should be possible to image behind and at the distal end of the device. Further, it should be possible to provide a stereoscopic image of the site in which the instrument or other device is used without using an endoscope. Stereoscopic images can be particularly useful because they provide more information about depth when using an instrument at a site.

  It may also be desirable to generate multiple images of different locations on one or more instruments or elements. This is because by using multiple images, the limited field of view obtained from only a single image and position can be expanded. These images can also be used to observe parts of the site that would be disturbed when viewed from only a single location, as well as the perspective provided by images made at different sites It is also desirable to observe the field of view of the site. A more desirable function is to use images made with different wavelength bands of light to expand the information provided by such information relative to that provided by a single such image alone. .

  In order to minimize costs and provide a more effective operation, light sources and other elements used by a plurality of different imaging probes provided on instruments and / or other devices to generate an image of the site It is also desirable to be able to share or multiple shares. Clearly, sharing a base station that includes one or more light sources and image processing capabilities with multiple imaging devices located on one or more instruments or other elements is more cost effective. In some examples, it may be desirable to share light of the same wavelength band generated by a single light source shared by multiple imaging devices. Images can be generated either continuously or in parallel by the imaging device. In other applications, it may be desirable to image light within a different wavelength band from a plurality of different light sources to a plurality of imaging probes located at the distal end of an instrument or other element to image an internal site.

  The advantages of providing a system capable of imaging from multiple locations on one or more instruments or elements as well as using the same base station are clearly not limited to medical applications. There are many other applications and environments that use imaging technology, from the distal end of one or more instruments, such as robotic end effectors, and multiple on one or more instruments or elements that share processing with a light source. Similar benefits can be obtained by imaging the site from the location.

  One concern that occurs when multiple imaging devices are used to image a region at the same time, for example, when generating a three-dimensional (3D) image, can cause significant image noise due to illumination crosstalk. That is. In this case, the two images used to generate the 3D image are obtained by two imaging devices that scan approximately the same region of the site, but at a known distance from each other. When using a scanning optical fiber to generate each image, the scanning optical fiber scans the illumination spot on the surface of the site. Reflections from these spots are captured to produce a series of pixels for each image derived from the output signal of each scanning device. However, reflections from spots illuminated by one scanning optical fiber may be detected by other imaging devices, creating crosstalk problems that cause low quality images. Therefore, it is important to minimize crosstalk between different imaging devices that are scanning the overlapping region.

US Pat. No. 6,975,898

M. Brown and D. G. Lowe, entitled "Recognizing Panoramas," published in the Proceedings of the Ninth IEEE International Conference on Computer Vision (2003) The online "Atlas of Gastroenterological Endoscopy," A. Freytag, T. Deist (2003)

  Exemplary systems and methods for generating multiple different images of a site with multiple imaging devices while preventing crosstalk in the images are disclosed below. The system includes a plurality of imaging devices including a plurality of scanning devices and a plurality of light receivers. Each light receiver receives light from a region of the region illuminated by one of the plurality of scanning devices in association with one of the plurality of scanning devices. In addition, each scanning device is coupled to the distal end of an optical fiber that is used to transmit light to the scanning device, so that light is emitted by the scanning device to illuminate the site. The light receiver thus receives light from the part and uses it to generate an image of the part. At least one light source is included to supply light to the scanning device via a plurality of optical fibers. A plurality of light beams generated by the light received by the plurality of light receivers by preventing light emitted by one of the plurality of scanning devices from interfering with light emitted by any other plurality of scanning devices. Means are provided for imaging the site so as to prevent crosstalk between the two images.

  The system further includes an optical switch that is controlled to direct light from at least one light source at a time through the optical fiber to one of the selected plurality of scanning devices. In at least one exemplary embodiment, the means for imaging to prevent crosstalk comprises a controller connected to the optical switch. The controller controls the optical switch so that only one image of the site is allowed to be captured simultaneously by multiple imaging devices. In this method, the image of the part is time-division multiplexed every frame (frame-by-frame basis). The plurality of scanning devices scan the site with light emitted in the desired scan pattern, and then resume scanning another scan interval followed by a retrace interval. The controller selectively enables the optical switch to supply light to the first scanning device that scans the region of the site while the second scanning device is in the blanking period. The controller then selectively enables the optical switch to supply light to the second scanning device that scans the region of the site while the first scanning device is in the blanking period.

  In different exemplary embodiments, the means for imaging to prevent crosstalk includes a controller that controls the optical switch, so that light from at least one light source is simultaneously captured in the region of the region and then the region. To a single scanning device that is scanning long enough to scan a spot corresponding to a single pixel of the image. Since the image of the part is thus pixel-multiplexed, the image is captured pixel by pixel for only one pixel of each image being captured simultaneously.

  The at least one light source may include a plurality of light sources, one or more of the plurality of light sources being used by only one of the plurality of scanning devices. One or more light sources used by one scanning device generate light in one or more wavelength bands, which are used by any other scanning device that illuminates a common part of the site It is different from the wavelength band of light generated by any other light source.

  In a further alternative embodiment, the means for imaging to prevent crosstalk is for filtering light received by the plurality of receivers receiving light from a common portion of the site illuminated by the plurality of scanning devices. Includes a plurality of optical filters used. Of the plurality of optical filters, certain optical filters pass light in one or more wavelength bands emitted by the scanning device associated with the particular light receiver that received the light, but wavelengths emitted by different scanning devices. The light of the belt does not pass. The plurality of optical filters can have different polarizations. In this case, the light emitted by each scanning device has a specific polarization, which is consistent with that of the optical filter used to filter the light received by the receiver associated with the scanning device. Yes. That is, only light received from a site generated by a scanning device associated with a specific light receiver is used to generate an image of the site based on the signal output by the specific light receiver.

  In yet another exemplary embodiment, the means for imaging to prevent crosstalk is in response to light received by the plurality of light receivers and a light modulator that modulates light provided differently to each scanning device. And a demodulator that demodulates the generated output signal. The demodulator separates the output signal based on each different scanning device that has generated light reflected from the site and received by multiple light receivers, so that only the light emitted by the scanning device with which the particular light receiver is associated. Are used to generate the image. Optical modulators modulate light using either amplitude modulation (AM) or frequency modulation (FM) schemes.

  The method includes steps that are generally consistent with the function of the elements of the system.

  Other features of the technology are directed to exemplary systems and methods for imaging a site. The system is shared by a plurality of scanning devices that emit light for illuminating the site and a plurality of light receivers that receive light from the site. The system includes at least one display that can be used to display an image generated by imaging the site. The scan controller includes one or more light sources that control the plurality of scanning devices and generate light that is supplied to the plurality of scanning devices to illuminate the site. The scan controller also includes one or more detectors that detect light received from the site with a plurality of light receivers. That is, one or more light sources are shared between a plurality of scanning devices, and one or more detectors are shared between a plurality of light receivers. The functional interface connects a plurality of scanning devices and a plurality of light receivers to the scanning controller. A computing device is also connected to the at least one display, the scan controller, and the functional interface to control the system so that an image of the site is emitted on the at least one display by the plurality of scanning devices. Without degradation due to interference between the light received by the instrument.

  In one exemplary embodiment, the scan controller includes a different light source for each of the two or more scanning devices, such that light in different wavelength bands is emitted by the two or more scanning devices.

  In other exemplary embodiments, the scan controller operates in one of three modes. Each of the three modes supplies light from one or more light sources to a single selected scanning device that is simultaneously scanning a region of the site, and scans light from one or more light sources simultaneously. Ensuring that light emitted from only one scanning device of a plurality of scanning devices simultaneously illuminates any region of the site while supplying the device, and light of different characteristics to the plurality of scanning devices The means for supplying is used to selectively supply light from one or more light sources to a plurality of scanning devices, thereby allowing each scanning device to irradiate any other scanning device that irradiates the same region of the site. Providing light having different characteristics from the supplied light.

  Corresponding methods include the claims consistent with the functions performed by the elements of the system.

  A number of concepts have been briefly introduced so far by providing means for solving the present invention, which will be further described in detail in the following description of embodiments for carrying out the invention. However, the purpose of embodiments for practicing the invention is not to identify key or essential features of the claimed subject matter, but also to help determine the scope of the claimed subject matter. Absent. In practice, it is desirable to use non-standard means to provide an enhanced and / or composite view of the site where one or more instruments or other elements are used.

  The various features and attendant advantages of one or more exemplary embodiments and modifications thereof will be more readily appreciated by reference to the following detailed description when taken in conjunction with the accompanying drawings. It will be.

Functional blocks describing components of an exemplary system having a single base station suitable for imaging using a variety of probes and including a functional interface that can provide various alternative functions associated with the probe, depending on the preferred embodiment. FIG. FIG. 3 is a schematic diagram of an exemplary approach that provides for continuous switching of light from a single light source such that light is continuously supplied to a plurality of different probes for imaging purposes. FIG. 3 is a schematic diagram of an exemplary configuration for providing parallel illumination to multiple probes using light of the same wavelength from light sources of three different wavelengths. FIG. 2 is a schematic diagram illustrating an exemplary configuration for splitting optical signals of different wavelengths among a plurality of different probes used to image a site. 2 is a more detailed functional block diagram of an exemplary system for imaging a site with a scanning device disposed at the distal end of a plurality of instruments, catheters, and / or conduits. FIG. Fig. 6 is a schematic diagram showing how separable signals for two scanning illuminators are used in a synchronous plane sequential method for imaging a site. Fig. 4 is a schematic diagram showing how separable signals for two scanning illuminators are used in an asynchronous or synchronous manner for imaging a site. FIG. 6 is a schematic diagram illustrating how an internal site is imaged in multiple perspective views using a plurality of spaced apart instruments or detectors disposed on the distal end of the instrument. FIG. 6 is a cross-sectional view of the distal portion of an exemplary instrument showing how light entering the side window is transmitted through the multimode optical fiber to the proximal end of the instrument. FIG. 6 is a schematic diagram illustrating how a plurality of spaced instruments or a detector disposed on the distal end of the instrument is used to image an internal site that includes an area that is otherwise invisible. FIG. 6 is a schematic view of the distal end of an exemplary forceps instrument illustrating the placement of the scanning illumination device and the collection fiber. How the exemplary parent device is used to image the anterior, while a child device capable of distal imaging advances through the side of the parent device and proceeds through the lumen and into the bile duct and main pancreatic duct It is the schematic of a part of stomach and duodenum which shows this. It is the schematic of the stomach which shows how ROI (region of interest) along a stomach wall is imaged using the parent endoscope which has an imaging capability in front, and the child instrument which has an imaging capability. FIG. 11B is a schematic view of an exemplary image of an ROI along the stomach wall displayed to a user of the parent endoscope and child device of FIG. 11A. FIG. 4 is a cross-sectional view of an exemplary scanning fiber distal end used during imaging of a distal end portion of an instrument, catheter, or conduit. FIG. 5 is a schematic side cross-sectional view of a conduit used to carry a forceps device with distal imaging and capable of distal imaging and to take multiple images of an internal site. FIG. 7 illustrates a central front field of view SFE having a plurality of side fields of view SFE (scanning fiber endoscope) and a trajectory for transporting the instrument to the distal end of the configuration. FIG. 6 illustrates a central front view SFE having multiple side views SFE and multiple conduits with optional side ports for delivering one or more instruments capable of imaging to an internal site. Since a plurality of scanning devices are mounted around the outer periphery, any two or more pairs of opposing scanning devices are used for stereoscopic observation, and depth information of a site where one or more instruments are used 1 is a schematic view of a distal end of a conduit that can deliver one or more instruments to a site that can provide an image comprising: FIG. 5 is a schematic elevation view of the distal portion of an exemplary embodiment of an array of confocal imagers showing details of one confocal imager. FIG. 16B is a schematic elevation view of a distal surface or distal end of an instrument that includes the confocal imaging device arrangement of FIG. 16A. FIG. 5 is a cutaway view of the distal end of an alternative exemplary embodiment of an array of imagers that uses a common lens set to collect light on the site and receive light from the site for all confocal imagers in the array. is there. FIG. 4 is a schematic diagram of the distal end of an exemplary 2 × 2 array of confocal imagers at time A, showing no overlap at points in the scanning region of the four confocal imagers on the surface of the site. The time B (or array displacement after FIG. 18A shows that vertical misalignment causes overlapping scan areas and can produce an image that can be stitched more easily to form an entire image of the site. FIG. 2 is a schematic view of the distal end of an exemplary 2 × 2 array of confocal imaging devices (after they occur). FIG. 4 is a diagram illustrating four exemplary overlapping images of pancreatic cancer. It is a figure explaining the example (simulation) result which produces | generates the whole image of the site | part which can identify pancreatic cancer easily by splicing four images of FIG. 19A. FIG. 6 is a schematic diagram of an existing instrument or tissue stapling instrument illustrating how an imaging device is coupled to an existing instrument with a sheath to allow imaging of the site where the instrument is used. It is sectional drawing of the example of FIG. 20A. FIG. 20B is a cross-sectional view of an alternative exemplary embodiment illustrating how two imaging devices can be coupled to the existing instrument described in FIG. 20A by a sheath.

(Figures and disclosed embodiments are not limiting)
Exemplary embodiments are illustrated in the referenced figures of the accompanying drawings. The purpose of the embodiments and the accompanying drawings disclosed herein are illustrative rather than limiting. The examples shown in the accompanying drawings and discussed herein are not subject to any limitation on the present technology and the appended claims.

(Outline of imaging system using one base station for various probes)
FIG. 1 illustrates an exemplary system that includes a single base station 20 that is used to image with multiple probes, where the multiple probes are used in the site being imaged. It can be an imaging device placed on an instrument or other element. The base station 20 includes a computer 22 and may be a general purpose personal computer or a dedicated computing device designed for a specific purpose to support a system for imaging with multiple probes. The computer 22 is connected to a keyboard 24 that is used for text input and control operations by the user, and a mouse, trackball, foot pedal or the like for controlling and selecting the position of the cursor on the image display when inputting to the computer 22. It is connected to a pointing and / or input device 26 which may be another type of device. Also, a first monitor 28 and a second monitor 30 are connected to the computer 22 and are used to generate by a plurality of SFE probes 36 (designated as probes A, B, C, and D). An image generated in response to the output signal can be displayed. It will be appreciated that the system is not limited to only four such probes, but may include more or fewer SFE probes, or other types of imaging devices may be used.

  Computer 22 communicates with the SFE scanning device / controller and light source / detector box 32 via one or more optical fibers 38. Further details of the configuration of box 32 are discussed below. The SFE scanning device / controller and light source / detector also communicate with the functional interface 34, through which signals are transmitted to and from the plurality of SFE probes. The functional interface 34 is controlled by the computer 22 and enables one of at least four alternative functions to be performed, depending on the specific configuration used in the imaging system, as described in detail below. These alternative functions use a function controller that continuously switches the red, green, and blue (RGB) laser light generated by the SFE light source in box 32 between multiple SFE probes used in the system. Including that. Continuous switching is performed using, for example, a MEMS (or galvanometer controlled) mirror switch, as described below in connection with FIG. In continuous switching mode, all of the light received from the site being imaged using multiple SFE probes extends from the distal end of the SFE probe and is grouped together for RGB group light detection in box 32. Can be transmitted via a collecting fiber.

  As described in detail in FIG. 3, a functional interface 34 can be used instead to perform a parallel probe illumination function with multiple beam splitters, which will be discussed below. In this parallel probe irradiation mode, light of the same wavelength band is used for all SFE probes, and frame sequential or pixel sequential time division multiplexing is applied to provide light to each of a plurality of SFE probes. FIG. 3 illustrates an exemplary configuration showing how this mode is implemented, as discussed in detail below.

  A third alternative function provided by the functional interface 34 is to split the optical signal. Because this mode of operation uses a variety of probes, separate RGB illumination fibers include different wavelength bands. Next, the optical signal received from the site is simultaneously split into separate wavelength bands before being detected. Further details are provided in connection with the example configuration shown in FIG. As a further alternative (details not shown), the light supplied to one or more specific SFE probes can be polarized such that the polarization is in a specific direction or mode. The light received from the SFE probe can then be filtered using an optical filter that passes only light having a polarization that matches the polarization of the light supplied to one or more specific SFE probes, and different SFEs. The light received by the probe can be separated before detection based on its polarization characteristics. In this method, only the light supplied to the specific one or more SFE probes is used for imaging the site.

  Finally, this function performed by the functional interface 34 can include modulation of light supplied from one or more light sources to each different scanning device, thereby being supplied to each of the different scanning devices. The light is modulated separately from the light supplied to any other scanning device. In addition, the light received by one or more receivers associated with a particular scanning device is detected and an output signal is generated, and the output signal is also demodulated by matched demodulation, so that the light modulated with different demodulation is Will be removed. The modulation / demodulation applied by the functional interface 34 can be either AM demodulation or FM demodulation, facilitating the demodulation function at a specific carrier frequency between output signals generated by detecting light from different receivers. So that crosstalk between different channels of the imaging device is avoided.

  Referring now to FIG. 2, an exemplary continuous switching configuration 40 is described in which RGB light (or more generally light of the same wavelength band) from a light source (not shown) is input optical fiber. Is transmitted through 42 and radiated along path 44 toward lens 46. Since the lens 46 collects the light on the MEMS mirror 48 and the mirror is coupled to a rotation driver (not shown) by the rotation axis 50, the light is directed to the subsequent reflectors 52a, 52b, 52c, and 52d. Will be directed sequentially. The light is continuously reflected by each reflector toward the lens 54, and the light is collected into one of the optical fibers 56. In this example, there are four optical fibers 56, each transmitting light entering from one of the lenses 54 to one of the probes A, B, C, or D. At the time shown in FIG. 2, the light is reflected into the optical fiber, the optical fiber having its distal end coupled to the SFE, which is placed on one of the instruments or other elements placed at the site. Yes. That is, only one probe is activated at a time determined by controlling the rotation of the MEMS mirror switch. Note that it may be necessary to adjust the white balance before using each probe to compensate for variations in coupling efficiency within the continuous switching configuration 40. Alternatively, a galvanometer control mirror can be used in place of the MEMS mirror 48.

FIG. 3 illustrates how the exemplary configuration 60 can simultaneously supply light of the same wavelength to multiple probes 76 (identified by AX). In this example, red light 62 from a light source (not shown) enters from the left and is partially reflected by a total reflection interference (FTIR) box beam splitter 68. Unreflected red light continues to enter the left (of the beam splitter) and in turn is partially reflected as well. This process is repeated for each probe, finally reaching the mirror or prism 70 for the final probe (ie probe X) and reflecting all the remaining light downwards in the direction of the dichroic longpass beam splitter 72. Change direction. The dichroic long pass beam splitter 72 transmits red light but is not reflected by other dichroic long pass beam splitters in the path of green light 64 (entering from the left as shown in this figure). Are selected to reflect. Each preceding dichroic long-pass beam splitter in this path reflects part of the green light downward, while transmitting red light reflected downward from above. Thus, it will be apparent that the dichroic long pass beam splitter 72 has the characteristics λ _cut > λ _green and λ _cut <λ _red . Similarly, incoming blue light 66 (from the left in the figure) reflects a series of dichroism selected to partially reflect blue light but transmit red and green light reflected downward from above. Each of the long pass beam splitters 74 is partially reflected downward. The combined RGB light is transmitted toward lenses 78a and 78b, which collect the RGB light into a single mode optical fiber that is coupled to probes 76 (A-X). That is, these probes receive RGB light from three light sources simultaneously. It will also be appreciated that additional or fewer different wavelength bands of light can be fed to more or fewer probes simultaneously as well.

  An exemplary configuration 80 for splitting optical signals in different wavelength bands is illustrated in FIG. In this example, RGB light 82 containing different wavelength bands is directed toward the diffraction grating or prism 84, and the diffraction grating or prism passes through the different wavelength bands along different paths in a PMT (photomultiplier tube) detection port. Reflects towards 86. RGB light 82 is red light over a wavelength range of 635 nm to 670 nm (as can be generated using a laser diode (not shown)), green light (argon ions) with wavelengths of 514 nm, 532 nm, and 543 nm. Laser, which can be generated using a dual 1064 nm laser, or a helium neon laser), and blue light with a wavelength of 440-450 nm, or 468-478 nm (eg, Nichia® blue laser) Generated using a diode). Therefore, blue light having a wavelength of 440 nm is received by the PMT detection port 88, while blue light having a wavelength of 450 nm is received by the PMT detection port 90. Similarly, green light with a wavelength of 532 nm and 543 nm is received at PMT detection ports 92 and 94, respectively, while red light with a wavelength of 635nm and 650nm is received at PMT detection ports 96 and 98, respectively. How to use this approach to transmit light in multiple wavelength bands using a single optical fiber (eg, an optical fiber having a distal end positioned to receive light from a site), and then transmit It will be clear whether the emitted light is split into optically different wavelength bands and input to different channels. The light thus split can also include (or alternatively) invisible light, such as infrared light or ultraviolet light. The optical frequency of the light emitted from the laser diode can be adjusted by changing environmental conditions such as the temperature of the laser diode. For example, cooling a laser diode below the room temperature can typically shift the optical frequency by more than 10 nm, providing at least two laser wavelengths for each active laser diode.

(More detailed exemplary system)
FIG. 5 illustrates an exemplary system 100 that can be used to provide imaging of a site at multiple locations located at the distal end of one or more instruments or other elements. In the present system, one or more light sources 102 (i.e., numbers 1-N) provide an optical signal, the optical signal is transmitted through one or more optical fibers 104, and the optical fibers are one or more. Having a distal end supported by an instrument or other element (not shown). The light supplied to each scanning irradiation device by the light source 102 can be in the same wavelength band or different wavelength bands, and can be controlled to be supplied to the scanning irradiation device simultaneously or sequentially. One or more instruments or other elements are located at the site to be imaged, where, for example, instruments or other elements are used, so that light transmitted via the optical fiber 104 is used to scan the area. Can be used. Although the initial application of the system provides imaging on a medical instrument or element placed at an internal site within a patient's body, the purpose of the system 100 is not limited to medical applications.

  A modulator 106 is provided in the exemplary system of FIG. 5 and is used to modulate the light source 102 based on signals supplied by the scan controller 110 in response to commands from the computer 118. The modulator operates as an optical switch to allow surface-to-pixel multiplexing by one or more scanning devices. Direct modulation of a laser diode light source is an exemplary method of multiplexing between different scanning devices. A laser diode can be directly modulated by switching its wavelength from the ultraviolet to the infrared through the visible spectrum and switching its power at a pixel sampling rate, for example, exceeding 20 million samplings / second (greater than 20 MHz). . In the spectral range from ultraviolet to blue, laser diodes can be directly modulated at speeds exceeding 50 MHz, and suitable laser diodes are available from Nichia (Japan). Recently, ROHM (registered trademark) (Kyoto, Japan) announced a green GaN-based laser diode that generates light having a wavelength of 532 nm and has a modulation speed comparable to that of a blue laser diode. Instead, the concept of doubling the frequency of infrared laser diodes to achieve a wavelength of approximately 1064 nm was prototyped by a company developing lasers for HDTV laser projection displays, which modulates green light above 50 MHz. I need that. These companies are Novalux (R) (Sunnyvale, CA), Corning (R) (Corning, NY), and Osram (R) Opto Semiconductors (Regensburg, Germany). Finally, red laser diodes that generate light with a wavelength of approximately 630-670 nm can be directly modulated above 50 MHz, from many manufacturers such as Sony (R) and Sanyo (R) (Japan). Available. The high modulation speed (greater than 50 MHz) of the laser diode light source allows optical switching or multiplexing at the pixel speed.

  Current prototypes of scanning fiber endoscopes that display red, green, and blue (RGB) images of 500 scan lines at 30 Hz require a pixel sampling rate of approximately 20 million samples / second. An exemplary front-field endoscope having a sub-millimeter scanning illumination device and a resonant oscillating single optical fiber with a distal projection lens system and a ring of collection fibers surrounding the scanning fiber is illustrated in FIG. Are discussed in detail below. In order to provide a pixel speed that multiplexes between two scanning fiber endoscopes using the same RGB laser wavelength, the pixel is twice the normal speed between the two devices, ie approximately 40 million samples / second. Must be sampled at. Faster pixel modulation is required for more scanning imaging devices to eliminate crosstalk. Instead, the modulation rate of each light source can be sufficiently greater than the pixel sampling rate of a single imaging device. For example, a constant modulation rate faster than 50 MHz can be used for the carrier frequency of the laser light source, but variations in amplitude or AM (amplitude modulation) can occur when this beam is scanned across the tissue. The amplitude of this carrier can be used to generate an image signal for the spatially varying absorption and / or backscattering characteristics of the tissue illuminated by the scanning laser light. As shown in FIG. 5, after detecting an optical signal using a high bandwidth optical detector 108 (such as a photomultiplier tube), the AM signal is demodulated at a modulator stage 106 (or at another demodulator stage 107). can do. That is, each imaging device can have its own carrier frequency specific channel or band, which is similar to providing different channels or stations in wireless transmission and reception technologies. Lasers that emit light from ultraviolet to infrared wavelengths can be modulated at speeds that exceed the pixel speed, so many cycles of laser irradiation can be included in one image pixel, from another probe that images the same area. The AM signal can be detected without crosstalk.

  In FIG. 5, a computer 118 is also used to generate an image based on electrical signals received from the optical detector 108 to perform scan calibration, colorimetry, and brightness control of the light source 102. The computer 118 can also generate a control signal that can be applied to bend the tip of a catheter, endoscope, or other instrument introduced at the site to be imaged, into a branched lumen or other It is easy to introduce the device to the area around the corner through the passage. The scan controller 110 also generates a scan actuator drive signal that is applied to each scan actuator (drive) 112 located at the distal end of one or more instruments or other elements to provide an optical fiber or A mirror MEMS scanning device (not shown) is driven to scan the site with light emitted in a desired scanning pattern such as raster scanning, helical scanning, and Lissajous pattern scanning.

  A temperature control 114 is connected to the scan controller 110 and receives temperature signals from each temperature sensor 116 located in the scanning illumination device so that the scan controller corrects the temperature measured at the site as needed or images. The temperature of the device can be adjusted. In some applications, a single temperature sensor 116 can adequately monitor the temperature of the site, which is a temperature correction for each scanning device used to image the site based on the sensed temperature. This is because it can be applied. However, in other applications, the temperature of each imaging device is often monitored and controlled individually using a real-time control loop.

  Light received from the site being scanned is transmitted through an optical fiber and input to an optical detector 108 that can be optionally synchronized with the control of the light source 102 using the signal input from the modulator 106. The The intent of providing such synchronization is an input signal corresponding to light directed to the site by a particular one of the various scanning illuminators that may have a different wavelength band than the light supplied by one different scanning illuminator Only to ensure that the optical fiber supplies. In this method, the electrical output signal from the optical detector corresponds only to the light received from the site when the site is illuminated only by a particular scanning illumination device. The optical detector can include a PMT, photodiode, phototransistor, CCD array, or other light sensitive device. Although the photodetector can be placed distal to the imaging device, from a dimensional consideration and cost, there are significant advantages to placing the photodetector proximal and shared between the imaging devices. In general, using a base station that includes expensive elements such as laser light sources, processing power, photodetectors, etc. shared by multiple imaging devices results in a more cost effective system. As described in this specification, even if a light source is shared by multiple imaging probes that are imaging the same part, if the light is from a plurality of different scanning devices, the light received from the part by the imaging device Crosstalk that causes interference can still be prevented.

  Under the control of the user interface 120, the computer 118 uses the electrical signal received from the optical detector 108 to display an image of the site at the display 1 monitor 28 (and / or an optional display 2 touch screen or other monitor 30. ) Can be generated on. Multiple images can be displayed on a single display device, or the user can selectively switch between images displayed on each display device. An image can be reconstructed from the output signal generated by detecting light received by the one or more imaging devices using dedicated electronics or software techniques. Each imaging device can have its own remapping file that is used to control image reconstruction, and the reconstruction can be switched instantaneously between imaging devices. Data that generates these images and other relevant data collected during imaging can be stored in data storage 122 including, for example, local or remote hard disk drives or optical storage media for later retrieval, use, and processing.

In general, it is desirable for a plurality of scanning illumination devices to share a light source and other component system 100. Therefore, to avoid problems that can occur when a site is illuminated simultaneously by multiple scanning illuminators, use multiplexing or other techniques that separate the signals for each different probe or scanning illuminator in time Is desirable. FIG. 6 shows that in the case of two different illumination light sources, the light received from the detected site and used to generate the image is synchronized with the illumination light source of the site and is not mixed with the reflected light from the site. A timing diagram and configuration 130 illustrating how the different scanning illumination devices 140 and 142 are activated to irradiate the site at different times will be described. 6 is controlled accordingly, so that the irradiator generates a scanning beam during a series of time intervals 132 and then returns to a rest state during the time interval 134. When the interval 134 starts, the scanning irradiation apparatus B that has been turned off and has been in the dormant state performs scanning during the time interval 136, and then returns to the dormant state during the time interval 138. That is, only one scanning irradiation device at a time actively scans the site. The lower part of FIG. 6 illustrates the scanning light 144 emitted from the distal end 140 of the scanning illumination device A at time T _X , the maximum helical spiral scan that the scanning light 148 produces at the end of the time interval 132. (Full helical scanning spiral scan) 150 is the central portion 146 of the helix. At time T _X , the scanning irradiation apparatus B is near the center of the rest period, that is, does not irradiate any part at that time.

  An alternative approach to controlling the scanning illumination devices A and B so that the scanning illumination devices A and B generate separable optical signals 166 and 168 (which are asynchronous or synchronizable) is shown in the exemplary configuration 160 of FIG. Explained. In this approach, the helical scanning device 162 comprises a light pulse 174 while simultaneously generating a helical scan 170 of the site. Similarly, the helical scanning device 164 comprises a light pulse 176 and at the same time generates a helical scan 172. The signals that activate the light sources used to generate the respective helical scans by these two scanning illuminators can be in pixel order synchronized with the detection of light from the site. This approach reduces any potential phototoxicity by dispersing the light exposure of the tissue over time. The pulse order of the light pulses used in each scanning irradiation device is shown at the bottom of the figure.

(Advantages of imaging parts from multiple positions)
FIG. 8A illustrates an example 180 illustrating the use of the new approach for laparoscopic surgery, where it is provided by imaging from multiple spaced locations on the distal ends of multiple medical instruments. The multi-viewpoint image is useful when more effectively observing a site where the medical instrument is used. Tissue 180 provided in the image of the site being illuminated in this example by a single central scanner illumination device 184 included at the distal end of endoscope 182 so that the details of the site are more apparent. As a result, shading improves tissue perspective. The FOV (field of view) of the central scanning device irradiation device irradiates the tissue 186 with light of the desired scanning pattern. The light reflected from the tissue is received by a plurality of optical fibers 188 arranged in a ring around the central scanner irradiator and transmitted proximally to a detector (not shown in this figure). And used to generate an image of the part. The forceps instrument 190 includes a return optical fiber 192 at its center and similarly receives light from the tissue 186 irradiated by the central scanning device irradiation device in the endoscope. The light reflected from the site enters between the distal end of this return optical fiber of the forceps instrument and the open ends of the forceps grips 194a and 194b. A third location for imaging the site is located on the dissecting instrument 196 and includes a window 198 that receives the light reflected from the tissue 186, which is reflected by the return optical fiber 192 and in the endoscope. The angle is different from both of the optical fiber rings. The light that has passed through the window 198 in the cutting instrument passes proximally through the multimode optical fiber 200 (only one shown), as illustrated in a cross-sectional view of a portion of the cutting instrument 196 in FIG. 8B. Is transmitted. Light enters window 198 from the side of the cutting instrument and is internally reflected many times at the interface between the cleaved distal end of multimode optical fiber 200 and the air, polymer, or metal interface in the cutting instrument.

  The advantages of using multiple scanning illuminators to image a site and detecting light from multiple different locations on the distal end of the instrument or element are illustrated in example 210 shown in FIG. 9A. In this example, endoscope 182 extends through abdominal wall 212. A central scanning illuminator 184 in the endoscope scans a portion of tissue 214 near the distal end of the endoscope with the desired scanning pattern of light, and the ring of optical fibers 188 reflects from the portion of tissue 214. Receives and transmits transmitted light. The tissue of the internal site forms a ridge or ridge, so that the other part 216 of the tissue is removed from the FOV of the scanning irradiation device of the endoscope 182, and the other part 218 of the tissue is hidden by the protruding shape of the ridge or ridge of the tissue , In the deep shadow relative to the FOV irradiation of the central scanning irradiation device of the endoscope 182. However, in this example, the two forceps instruments 190a and 190b also extend through the abdominal wall to the opposite side of the endoscope 182. Each of these forceps instruments includes a central scanning irradiator 193 that irradiates tissue, from different directions and different positions on either side of the central scanning irradiator in endoscope 182. Thus, light reflected from the tissue portion 216 is received at the distal end of the return optical fiber 192 in the left forceps device 190a, while light reflected from the tissue portion 218 is received from the right forceps device of the endoscope. Received by return optical fiber 192 in 190b. Effective use of multiple scanning illuminators at different locations that illuminate the site from different angles greatly improves the visibility of the site in the image generated by the system, i.e., for one instrument or element Effectively broadens the imaging FOV provided by only a single scanning illuminator that detects light only at the distal end.

  The tissue 214 in FIG. 9A is an object of interest, such as a tumor 220, and interference from light from the right forceps device 190b that illuminates a portion 218 of the tissue is expected. For example, this light interference may be due to more light absorption than the surrounding tissue, which can be detected by the right forceps device 190b or other device capable of imaging. In this example, the increased absorption contrast can be detected from the backscattered optical signal to the right forceps instrument 190b that illuminates a portion 218 of tissue. Alternatively, the increased absorption contrast can be detected from the side scattered optical signal to the endoscope 182 or can be detected from the transmitted optical signal through the tissue portion 216 to the left forceps instrument 190a. it can. In this example, imaging capable instruments share optical signals and provide the user with enhanced shadows from various perspectives, allowing imaging in both reflective and transmissive states within the same region of the body. In these limited cases where the illuminated areas of the field of view do not overlap directly, no method is employed to reduce crosstalk.

  Details of the distal end of forceps instrument 190 are illustrated in FIG. 9B. As shown here, the distal end of the scanning irradiator 193 that receives and collects light from the site and the return optical fiber 192 is disposed between the grippers 194a and 194b. That is, the forceps device can image a site where the forceps device is used to grip tissue or other material.

  Another medical example 230 is provided in FIG. 10 and schematically illustrates a parent endoscope 232 designed to descend into the patient's esophagus and into the stomach 234 and then into the duodenum 236. The parent endoscope 232 includes a front-field scanning illuminator and a corresponding optical fiber that receives light emitted in the front FOV at the distal end 238 of the parent endoscope. A lateral field scanning illuminator 240 is also provided at the distal end of the parent endoscope to scan the side generally perpendicular to the longitudinal axis of the parent endoscope. In this view, the return optical fiber that receives the light reflected from the tissue at the side of the distal end of the parent endoscope illuminated by the lateral field scanning illumination device is not visible. The parent endoscope uses its imaging capabilities to allow the operator to advance the distal end of the parent endoscope into the duodenum to locate the opening 250 from the duodenum into the bile duct 246 and one main pancreatic duct 248. Support.

  A side port 241 is disposed in the vicinity of the lateral field scanning illumination device 240 through which a child endoscope 242 with forceps instruments including grippers 244a and 244b extends. Located on the distal end of the child endoscope 242 and between the two graspers (not visible in this figure) is a forward-field scanning illumination device, generally as shown for the forceps instrument 190 of FIG. 9B. It is a simple configuration. Using the FOV of this anterior field scanning illuminator, the forceps instrument at the distal end of the child endoscope in either the bile duct or the main pancreatic duct for an operator to collect a tissue sample or for some other purpose Can help you move forward. As a result, the combined imaging capabilities of the parent endoscope and child endoscope greatly enhance the accomplishment of this nature of work by providing a more complete imaging capability than that achieved by a single imaging device alone. make it easier.

FIG. 11A illustrates a parent endoscope 262 having a front field scanning illumination device 266 with a FOV 268 at its distal end and a return optical fiber that receives light from tissue and other subjects within the FOV of the front field scanning illumination device. An example 260 is described. In this example, the parent endoscope 262 is advanced into the patient's stomach 264. A child endoscope 270 having a forceps instrument 272 at its distal end also includes a front-field scanning irradiator having a FOV 274 directed toward a ROI (region of interest) 276. That is, the forceps device can easily image the ROI and selectively collect tissue samples anywhere. The front visual field scanning irradiation device on the parent endoscope 262 and the front visual field scanning irradiation device on the child endoscope 270 image the stomach wall at different distances from the ROI. The front-field scanning illumination device on the child endoscope 270 can have light that is more highly collected nearer the ROI 276 than the parent endoscope 262, and a return optical fiber (not visible in this figure). Can receive reflected light and produce an image with higher resolution but shallower depth of focus (DOF) than that generated in response to light received from the return optical fiber in the parent endoscope . Therefore, by providing these two scanning irradiation apparatuses provided with FOV and DOF having different characteristics, it is possible to enhance the capacity of the entire system and execute a reliable operation. Alternatively, the child endoscope 270 can be illuminated with light that emits a fluorescent signal from the site, which is generally much weaker than backscattered laser illumination. Fluorescent signals can be used to form ROI diagnostic images, and tissue health information can be collected using optical inspection of various modes of tissue. Simply positioning the child endoscope 270 closer to the ROI 276 than the parent endoscope 262 significantly increases the collection efficiency of the optical signal. The reason is that if R is a separation distance between the distal end of the child endoscope and the ROI, the intensity is attenuated by (1 / R) ² . The child endoscope 270 can also provide stereoscopic depth enhancement observation or deeper tissue imaging of the ROI using optional biomarker enhancement of infrared frequency light and tissue specific image contrast mechanisms.

  FIG. 11B illustrates an image of the ROI 276 displayed to the user using signals from the parent endoscope 262 and the child endoscope 270. In the parent endoscopic image 274, a gastric mucosal fistula of the mucous membrane covering the inside of the stomach, that is, a pleat 278 is displayed at a low resolution by simple color imaging of backscattered light. Within this ROI image is an insertion image provided by a child endoscope extended to the closest separation distance R, and an enlarged image of the gastric mucosa 278 is obtained. In addition, high contrast fluorescent labeling of Helicobacter pylori or other that is highlighted by a locally applied fluorescent dye (eg, acriflavine hydrochloride) and is not visible in the parent endoscopic image Provides biologically specific cells of interest. Once the child endoscope has advanced from the parent endoscope, the ROI 276 image of the parent endoscope is no longer an occluded image. The child endoscope image can be minimized and stitched to the disturbed portion of the parent endoscope image using the following technique.

(Exemplary scanning irradiation device and return optical fiber)
Although other designs can be used for the scanning irradiation device, an example of a scanning fiber irradiation device and imaging device 300 is illustrated in FIG. The scanning fiber illuminator and imaging device 300 includes a flexible single mode optical fiber 304 that passes through a tube of patterned piezoelectric material 306 that serves to drive and move the distal end 310 of the optical fiber in a desired scanning pattern. The distal end 310 extends distally beyond the patterned piezoelectric material tube and is cantilevered thereon, near the distal end of the instrument or other element on which the scanning fiber illuminator is attached or supported It is in. The patterned piezoelectric material tube is held in place by a piezoelectric material mounting collar 308. A quadrant electrode 314 is stretched over the patterned piezoelectric material tube and is selectively activated by an applied voltage to generate biaxial motion of the distal end 310 of the optical fiber 304. A wire 316 transmits a voltage signal to each quadrant electrode to activate the piezoelectric material for each axis of motion, and simultaneously transmits a temperature control signal to a temperature control (not shown). In this exemplary embodiment, the two axes on which the distal end of the optical fiber is driven are generally orthogonal to each other. Although an amplified sine wave applied to one axis of a patterned piezoelectric material tube and a cosine wave applied to the other axis can generate a circular scan, those skilled in the art will recognize that the distal end of optical fiber 304 It will be appreciated that a variety of different scan patterns can be generated by moving 310 appropriately. Appropriate modulation of the amplitude of the voltage signal applied to the quadrant electrode creates a two-dimensional pattern that fills the desired area and can be imaged with light emitted from the distal end 310 of the optical fiber. Some examples of the various scan patterns that can be implemented include linear scans, raster scans, sinusoidal scans, toroidal scans, spiral scans, and propeller scans. In some exemplary embodiments, the distal end of the optical fiber is driven to move at approximately its own resonant frequency (or close to the resonant frequency), thereby providing greater scanning amplitude for a constant applied drive signal. Make it possible.

  Other types of scanning instruments that can alternatively be used to image at the distal end of an instrument or other element optically scan the interior site with light to produce an image of the interior site that is used instead A MEMS scanning device (not shown) having a scanning beam used for the purpose. An example of a MEMS scanning device for imaging is shown in commonly-assigned US Pat. No. 6,057,086, the disclosure and specification of which are hereby incorporated by reference. One skilled in the art will appreciate that the reflective mirror can be driven to scan the site with light transmitted to the distal end of the instrument or other element.

  Light emitted from the distal end 310 travels through the lenses 318, 320, and 322 and is directed to a site in front of the scanning fiber illuminator as it moves in the desired scanning pattern. The diameter of the entire scanning fiber irradiation apparatus is generally 1.0 mm or less. The light reflected or scattered by the part illuminated by the scanning light is then detected and used to provide an imaging function. In the present exemplary embodiment, an annular ring 302 of return optical fiber is disposed around the distal end of the scanning fiber illuminator and has an outer diameter generally less than 2.0 mm. Light from the site passes into the distal end 324 of the return optical fiber and is transmitted proximally to the detector in the base station as described above. The output signal generated by the detector is then used to generate an image of the site proximate to the distal end of the scanning fiber illuminator. As described above, the side field irradiator can use a reflective surface or a mirror (not shown), and can then easily image the site on one or more sides of the scanning fiber illuminator.

  Providing a device capable of imaging for use on a site and multiple sites for imaging on multiple devices has obvious advantages over the case of a single site for imaging on the device. is there. An exemplary configuration 340 is illustrated in FIG. In this example, the catheter or conduit 342 is hollow and passes the forceps device 346 through an internal lumen formed in the catheter or conduit. A flexible cable 348 extends through the interior center of the forceps instrument, and a scanning irradiation device 350 disposed at the proximal end of the forceps instrument (not shown) and the distal end of the forceps instrument (between the grippers 356a and 356b) Transmit light and other signals between them. In addition, the distal end of a return optical fiber (not shown separately) that receives light from a portion irradiated in the FOV 352 of the scanning irradiation device 350 is centrally disposed between the grippers. The light emitted by the scanning irradiation device 350 is directed to the tissue 354a along a part of the body cavity 344 in which the present configuration is inserted. The FOV of the scanning irradiator 350 is directed forward relative to the forceps instrument and limits the wall of the body cavity 344 that can be seen in the resulting image.

  However, the catheter or conduit 342 also includes scanning irradiation devices 360 and 366. The flexible cables 358 and 364 extend along the opposite surface of the catheter and conduit. The distal end of the flexible cable 358 is coupled to the scanning illumination device 360 while the distal end of the flexible cable 364 is coupled to the scanning illumination device 366. Within these flexible cables are included optical fibers that carry light and other signals bidirectionally between the scanning illuminator and the proximal end of the flexible cable. The scanning illuminator uses light from a proximal light source (not shown) to emit light in a desired scanning pattern with a FOV 362 directed to the side of the body cavity 344, and tissue 354b disposed therein Irradiate. Similarly, the scanning irradiator 366 emits light in a desired scanning pattern having an FOV 368 directed to irradiate tissue 354c disposed on opposing walls of the body cavity. Light received from tissue 354b and 354c is transmitted through return optical fibers in flexible cables 358 and 364, respectively, and used to generate images of these different positions, allowing the user to more effectively manipulate forceps instrument 346. To obtain a tissue sample from the desired ROI. Using multiple images of the interior surface of the body cavity provides clearly more visual information than using only a single image of a single part of the body cavity.

  Two other exemplary configurations 370 and 390 are illustrated in FIGS. 14A and 14B, respectively, illustrating another example of how imaging can be performed on a number of instruments or other elements. In FIG. 14A, the exemplary configuration 370 has a central SFE 372 having a front imaging capability at its distal end 376 and having a plurality of SFEs 374a, 374b, and 374c with lateral imaging capability arranged around the central SFE. Including. SFEs 374a, 374b, and 374c include side ports 378a, 378b, and 378c, respectively, through which light is emitted in the desired scanning pattern, so that FOV 382a, which is directed radially differently around the central SFE, 382b and 382c are provided. Each of these lateral fields of view SFE also includes a return optical fiber (not shown) that transmits light received from a portion of the illuminated area within the respective FOV. The central SFE 372 includes a return optical fiber (not shown) that scans the front FOV 380 with light of a desired scanning pattern and receives light from a portion of the site illuminated by the light in the FOV 380. That is, the combined imaging capabilities of the four SFEs provide a very good range distal and surrounding the distal end of this configuration. A guide wire or track 384 also extends down at least one side of the central SFE 372 and is used to advance any of a number of additional instruments or other elements toward the distal end of the configuration 370. be able to. The additional instrument or other element may have imaging capabilities and may include a scanning illumination device, or may include only the scanning illumination device or return optical fiber, or neither.

  The exemplary configuration 390 of FIG. 14B is similar to the configuration 370, but includes conduits 392a and 392b, and is not capable of imaging in this exemplary embodiment, eg, transporting liquid to or removing liquid from the site, or The difference is that other functions are executed. The exemplary configuration 390 further includes elliptical conduits 394a and 394b disposed oppositely about the central SFE 372. These elliptical conduits can optionally each include a side port (such as side port 396 shown on elliptical conduit 394a). The side port can perform a desired operation, such as removing a tissue sample from the site such that other instruments or elements that advance through the interior of the elliptical conduit face outward toward the site. The small, generally cylindrical, exemplary configurations 370 and 390 are well protected from around the object (ie, tissue), but their shape advances along the guide wire 384 or through the elliptical conduit. It also limits the size of instruments that can be.

(Exemplary Multiple Scanner Stereoscope Conduit)
An instrument or conduit that includes at least two disparate scanning devices can be used to provide a stereoscopic view of the site, which can provide useful depth information, which greatly enhances the user's understanding of the site. And as a result of the depth information, the instrument can be used more effectively at the site. FIG. 15 illustrates an exemplary embodiment of a conduit 400 having a central lumen 406 through which one or more instruments are advanced to a site where the one or more instruments are used. In the illustrated example, a plurality of imaging devices 404 are arranged around the outer periphery of the conduit 400. In general, any two of the imaging devices located on opposite sides (not essential) of the conduit 400 can be selectively activated to generate a stereoscopic image of the site. In the example shown in FIG. 15, the imaging devices 402a and 402b are activated to scan a region (not shown in the figure) having two spaced fields of view 408 and 410. Using images generated by receiving light from sites illuminated in two different fields of view, a stereo image of the site is provided, just like two images obtained by two separate eyes. Using different imaging devices 404, the orientation of the stereoscopic image relative to the conduit 400 has been changed in response to the viewer's congestion angle, or the rotation of the conduit relative to tissue has been corrected or extended. Avoid the instrument interfering with the image. Imaging devices 402a, 402b, and other imaging devices 404 can be confocal imaging devices (as described below in connection with FIGS. 16A, 16B, and 17), or alternatively adjacent. An imaging device 404 with a received optical fiber can be used to receive light from sites illuminated by different fields of view 408 and 410. The received light is transmitted proximally through an optical fiber to an optical sensor (not shown), thereby corresponding electrical signals used to generate an image that is used to form a stereoscopic image of the site. Is generated.

  16A and 16B illustrate an exemplary embodiment showing an array 420 of nine confocal imaging devices. However, it should be noted that more or fewer confocal imaging devices can be used for the instrument or other elements. In general, the FOV of a confocal imaging device is relatively small, and using only one such device to image a site where one or more instruments or other elements are used limits its convenience. However, by combining images generated by a plurality of such confocal imaging devices to generate an overall image with a wider FOV than any one of the confocal imaging devices, the user views the image and can see one or more. Instruments or other elements can be easily used at the site.

  FIG. 16A describes only the arrangement of the three confocal imaging devices 422a, 422b, and 422c and shows details of the confocal imaging device 422b. In the exemplary embodiment, each confocal imaging device includes at least one lens 424 at its distal end, which is used to collect light emitted by the confocal imaging device when scanning a site, such as tissue 430. Light and collect light received from the site and transmit it proximally through optical fiber 436. Because the distal end of the optical fiber 436 is cantilevered from the scan driver, light from a light source (not shown) is transmitted from the proximal end of the optical fiber 436 that passes through the scan driver 434. Scan driver 434 is capable of driving a cantilevered portion of optical fiber 436 to vibrate in two orthogonal directions at or near its resonant frequency when excited by a drive signal supplied through lead 438. A piezoelectric device having The scanning driver itself is cantilevered from a cylindrical mount 432 in the confocal imaging device. The confocal imaging devices 422a, 422b, and 422c respectively scan the regions 428a, 428b, and 428c on the tissue 430 (or other type of site being imaged) into focused scan spots 426a, 426b, and 426c. Scan by. The light returned from the focused scan spot is generally free of crosstalk with the light from other confocal imaging devices because it is generated by the light collected on different regions of the site and its specific This is because the light from the confocal imaging device returns to the center of the cantilevered optical fiber and is condensed, and substantially no light from the other confocal imaging device exists. The scanning of regions 428a, 428b, and 428c can be as desired, such as a helical scan, raster scan, Lissajous scan, or other suitable region scan pattern generated by applying an appropriate drive signal through wire 438 to the scan driver. This is performed using the scanning pattern. Each of the images corresponding to the regions scanned by each confocal imaging device can be combined with the entire image of the site to facilitate use at the site of the instrument or other element.

  FIG. 16B illustrates the distal surface or end of the instrument 450 that includes the array 420 and shows nine lenses 424 used by each of the confocal imaging devices comprising the array in this exemplary embodiment. . Stereo non-confocal imagers 452a and 452b can optionally be provided on each side of array 420, and similarly, stereo non-confocal imagers 454a and 454b are located at the top and bottom of the distal end of the instrument. . Alternatively, a tip-curved anchor can be embedded in one or both positions of the stereo nonconfocal imaging device pair to curve or deform the distal end of the instrument in the desired direction. With only one pair of tip-curved anchors, the instrument must be rotated about its long axis to bend in different planes. The instrument 450 includes at least one track 456 disposed on its outer peripheral surface and is configured to introduce other instruments or elements into the site where the instrument 450 is advanced. The track 456 is generally T-shaped and extends longitudinally along the instrument 450 from near the proximal end of the instrument to near its distal end.

  Yet another exemplary embodiment of a confocal array is a confocal array that is similar to array 420 but includes arrays that use common lenses 466, 468, and 470 to point to different spots on the site. Collects the light emitted by all of the focal imaging devices, collects the light from those spots being scanned, collects the core of each cantilevered fiber optic core containing each confocal imaging device Return to the distal end. FIG. 17 illustrates an instrument 460 that uses this approach and shows the distal end of a cantilevered optical fiber 436 being deformed in the direction of the desired scan pattern. Instrument 460 includes a track (not shown) similar to that of instrument 450 to allow other instruments or elements to be introduced to the site where instrument 460 has been advanced. In addition to scanning by each confocal imager over the area that the FOV of each confocal imager is responsible for, the instrument 460 includes a lens barrel 464 to which lenses 466, 468 and 470 are attached, and an array of confocal imagers. To change the depth of the confocal scan by providing a relative movement along the longitudinal z-axis of the instrument (as indicated by the arrow) between the attached more proximal housing 462 It is configured. That is, when the array of confocal imaging devices scans each region on the site to provide a 3D scan of the tissue (or other material containing the site), the depth of the confocal scan of the tissue (in this figure (Not shown) can be changed. A lens or fused lens set, such as a gradient index lens, collects light toward all confocal imaging devices in the array along a generally parallel channel, while the focal plane of the array is a linear driver (not shown). ) To adjust along the z-axis. If the relative movement of the lens set is not given, approximately the same result can be achieved instead by one of various other methods. Specifically, although not shown, the scanning optical fibers of various confocal imaging devices can be shifted from the lens to a distance z so that light beams from different confocal imaging devices are collected at different z-axis positions. And / or their direction can be adjusted and / or each can be imaged using different wavelengths of light. In any of these alternative approaches, two or more depth planar images are obtained while simultaneously operating the array of confocal imagers.

(Overlapping of confocal images generated by an array of confocal imaging devices)
FIG. 18A illustrates an exemplary arrangement 500 that includes four confocal imagers 502a, 502b, 502c, and 502d, which are generally similar to the confocal imagers described above. These four confocal imaging devices emit the light collected by the lens 504 and scan the regions 506a, 506b, 506c, and 506d on the site 508, respectively. As shown in FIG. 18A, at some time A, there is no overlap between these four scanned regions on the site. Scanning of these areas can occur in 1/30 second, which is necessary to obtain a complete four images of the region corresponding to the scanned area in the exemplary embodiment of the confocal imager. It ’s a great time. However, due to the relative movement between the array 500 and the site 508, a vertical displacement of the array occurs at time B as shown in the example of FIG. 18B. This displacement can occur because the user is holding the array and imaging a larger area of tissue, or the array moves due to unavoidable movement, or movement relative to the arrangement of sites. For example, if the site is within the patient's body, the site may move relative to the array due to patient breathing, muscle contraction or movement of the body, beating, or other physiological causes. The task of generating an entire image of the region based on the combination of the four images of the region is that the original scanned regions 506a-506d at time A overlap at least somewhat with the scanned regions 506a′-506d ′ at time B. There must be. This overlap between adjacent images can be accentuated by the user intentionally panning the distal end of the array 500 over the site, so the resulting duplicate images can be stitched together to create an overall image of the site. Any suitable software (below) can be used to form.

(Exemplary software for stitching overlapping images)
Non-Patent Document 1 discloses a technique for generating an entire stereoscopic image by connecting a plurality of overlapping images. This technique can be easily used in connection with the joining of overlapping images of various parts of the site generated by a plurality of imaging devices as described above. The AUTOSTICH ™ software for performing this task is available on the website worldwideweb. cs. ubc. ca / ~ mbrown / autoost / autostich. You can download it from html (where worldwideweb is replaced by “www”). This software can be applied to most multiple digital images where at least some of the adjacent images overlap, producing a complete image of up to 360x180 degrees or as wide as the input image range. This software is referenced as the only example of another commercially available software program that can be used to stitch overlapping images together to produce a fully connected image of the site.

  FIG. 19A illustrates four exemplary overlapping endoscopic images 600a, 600b, 600c, and 600d (obtained from the images of Non-Patent Document 2) of pancreatic cancer, and by four scanning devices such as the scanning device described above. Four generated overlapping images are shown. These overlapping images can be joined together using the stitching software as described above to generate a whole image 602 as shown in FIG. 19B. Many other image examples illustrating the capabilities of such software are provided on the above web for AUTOSTITCH ™ software.

(Example explaining the addition of an imaging device to an existing instrument)
One advantage of the small imaging device disclosed above is that it is easy to combine with existing instruments or other elements to allow imaging of sites that could not be realized with larger imaging devices. . 20A-20C illustrate an exemplary approach 700 that can be used to add an imaging device to an existing instrument. In this example, the existing instrument is a medical stapler 702, or may be a linear incision instrument for an endoscope such as Model i60 ™ from Power Medical Interventions ™. As shown in FIG. 20A, the medical stapler includes a movable jaw 704 that pivots about a pivot axis 708 toward a fixed jaw 706. To couple the imaging device 712 to an existing medical stapler, a sleeve 710 is placed through the imaging device and its optical fiber (not shown separately). Since the sleeve 710 can be formed of a heat-shrinkable tube, after the sheath is placed through both the imaging device and the proximal portion of the medical stapler, the sheath is heated to contract around both the medical stapler and the imaging device. , Thereby coupling the imaging device 712 to the medical stapler 702. A distal end 714 extending beyond the sheath 710 and a portion of the imaging device 702 can be coupled to the fixation jaw 706 using a biocompatible adhesive such as cyanoacrylate or other suitable adhesive. In this exemplary configuration, the distal end 714 is tilted slightly upward to face the FOV 716 of the imaging device distal to the medical stapler (or linear cutter). This arrangement is even more useful when the existing instrument is an endoscopic linear cutter because the FOV images the site in the direction in which the linear cutter is propelled to perform the incision operation.

  FIG. 20B illustrates a cross-sectional view of the present exemplary embodiment showing how the sheath is deflated to couple the imaging device 712 to an existing medical stapler (or linear endoscope cutter). An alternative exemplary embodiment 720 shown in FIG. 20C illustrates how two imaging devices 712 are similarly coupled on either side of an existing medical stapler (or linear cutter device) 702, After slipping over both the imaging device and the existing instrument, a sheath 722 is used that is heated to tightly shrink around the configuration. If the existing instrument is a medical stapler, the distal end of the imaging device is closer to the fixed and movable jaws so that the stapled area is optionally visible in a stereoscopic field in the image generated by the imaging device. This exemplary embodiment is particularly useful when placed in position.

  Although the ideas disclosed herein have been described in connection with preferred forms of implementing them and modifications thereto, many other modifications are possible within the scope of the following claims. Those skilled in the art will appreciate. Therefore, the scope of these ideas is not limited at all by the above description, and is completely determined by referring to the following claims.

Claims (25)

A system for generating a plurality of various images of a site generated by a plurality of imaging devices while avoiding crosstalk in an image,
(A) a plurality of imaging devices including a plurality of scanning devices and a plurality of light receivers, wherein each of the light receivers is associated with one of the plurality of scanning devices and irradiated by the one of the plurality of scanning devices; The light is emitted by the scanning device because it receives light from the region of the region and each scanning device is coupled to a distal end of an optical fiber that is used to transmit the light to the scanning device. A plurality of imaging devices that illuminate the site and the light receiver receives light from the site for use in generating an image of the site;
(B) at least one light source for supplying light to the scanning device via a plurality of optical fibers;
(C) when generating a plurality of images of the site using light received from the site by the plurality of light receivers, the light emitted by one of the plurality of scanning devices And a means for imaging the site so as to prevent crosstalk between the plurality of images by preventing interference with light emitted by the plurality of scanning devices.   2. The optical switch according to claim 1, further comprising an optical switch controlled to simultaneously direct light from the at least one light source to a selected one of the plurality of scanning devices via an optical fiber. The system described.   The means for imaging the site comprises a controller connected to the optical switch, the controller allowing the light to only allow one image of the site to be captured simultaneously by the plurality of imaging devices. 3. The system according to claim 2, wherein a switch is controlled so that the image of the part is time-division multiplexed every frame.   The plurality of scanning devices scan the part with light emitted in a desired scanning pattern, and then a blanking period follows to resume another scan, and the controller includes a second scanning device for the blanking period. The optical switch selectively enables light to be supplied to a first scanning device that scans the region of the region, while the first scanning device is in a retrace period, 4. The system of claim 3, wherein the optical switch selectively enables light to be supplied to a second scanning device that scans the region of the site.   The means for imaging the site is such that light from at least one light source scans the region of the site simultaneously and at a length sufficient to scan a spot corresponding to a single pixel being captured at that time. A controller that controls the optical switch to be fed to only one scanning device of the plurality of scanning devices, whereby an image of the region is obtained for each of the images for which the image is being captured simultaneously 3. The system of claim 2, wherein only one pixel is multiplexed pixel by pixel so that it is captured pixel by pixel.   The at least one light source comprises a plurality of light sources, wherein one or more light sources of the plurality of light sources are used by only one of the plurality of scanning devices to generate light in one or more wavelength bands; The wavelength band is a wavelength of light generated by any other light source among the plurality of light sources used by any other scanning device among the plurality of scanning devices for irradiating a common part of the part. The system of claim 1, wherein the system is different from a band.   The means for imaging the part is a plurality used for filtering the light received by a plurality of light receivers receiving light from the common part of the part irradiated by the plurality of scanning devices. A specific optical filter of the plurality of optical filters that passes light of one or more wavelength bands emitted by the scanning device associated with a specific light receiver while any other The system according to claim 6, wherein light of another wavelength band emitted by the scanning device is blocked.   The plurality of optical filters have different polarizations, and the light emitted by each of the scanning devices is a polarization of an optical filter used to filter the light received by the receiver associated with the scanning device. Only light received from the site emitted by the scanning device associated with a particular receiver and having the particular polarization corresponding to the light source based on the output signal generated in response to the light. The system of claim 7, wherein the system is used to generate an image of a region. The means for imaging the part is:
(A) an optical modulator that modulates light supplied to each of the scanning devices differently;
(B) Each of the different scanning devices that demodulates an output signal generated in response to the light received by the plurality of light receivers and generates the light reflected from the portion and received by the plurality of light receivers. A demodulator for separating the output signal, wherein a specific light receiver uses only the light emitted by the associated scanning device to generate an image. The system according to claim 1. The light modulator is
(A) amplitude modulation;
10. The system according to claim 9, wherein the light is modulated using one type of modulation selected from the group consisting of (b) frequency modulation. A method for avoiding crosstalk in an image generated by a plurality of imaging devices used to image a region and generate a plurality of images,
(A) a step of transmitting light from at least one light source to the plurality of imaging devices, wherein the plurality of imaging devices include a plurality of scanning devices and a plurality of light receivers, and each of the scanning devices includes the part Coupled to an optical fiber that transmits light from the at least one light source for use in irradiating each of the light receivers in association with a particular scanning device and receiving light from the site;
(B) emitting the light from each of the scanning devices to irradiate at least a portion of the site;
(C) in response to the light received from the site by the plurality of light receivers, generating an output signal indicative of the light received for use in generating the plurality of images;
(D) when generating the image of the site using the light received by each of the light receivers associated with the one scanning device, the light emitted by any one of the plurality of scanning devices is arbitrary; Crosstalk between the plurality of images of the portion generated using the light received by the plurality of light receivers by avoiding interference with light emitted by the other plurality of scanning devices Controlling the imaging of the part so as to prevent.   The controlling step includes the step of selectively supplying light to only one of the scanning devices at the same time so that only one image is captured simultaneously by the plurality of imaging devices, wherein the image of the site is frame by frame The method according to claim 11, wherein the method is time division multiplexed. The irradiating step comprises:
(A) scanning a part of the site with each of the scanning devices with light emitted in a desired scanning pattern;
And (b) a step of returning to a start position during a blanking period and restarting another scan by the scanning device after the scanning device completes scanning with a desired scanning pattern. The method described in 1.   The controlling step provides light to a single imaging device at a time sufficient to scan a spot corresponding to a single pixel of an image of the site being captured, thereby providing the site. 14. The method of claim 13, comprising pixel multiplexing pixel by pixel by simultaneously capturing only one pixel of the image.   Further comprising providing light from one or more light sources to only one of the plurality of scanning devices, the one or more light sources operating in one or more wavelength bands, 12. The method of claim 11, wherein the wavelength band of any other light source that supplies light to any other plurality of scanning devices is different.   The controlling step includes filtering light arriving at the receiver, wherein only light generated by a scanning device associated with a particular receiver generates one of the plurality of images of the site. 16. A method according to claim 15, characterized in that it is used by said specific receiver. The controlling step includes
(A) modulating light provided at different modulation frequencies to each of the scanning devices;
(B) demodulating an output signal generated in response to light received by the plurality of light receivers, to each of the different scanning devices that radiate the light reflected from the portion and received by the plurality of light receivers. Based on, separating the output signal, generating only one of the plurality of images using only light from the desired scanning device received by a receiver associated with the desired scanning device; 12. The method of claim 11, comprising the steps of: The modulating step includes
(A) amplitude modulation;
18. The method of claim 17, comprising modulating the light with one type of modulation selected from the group consisting of (b) frequency modulation. A system for imaging a site, the system being shared by a plurality of scanning devices that emit light used to illuminate the site and a plurality of receivers that receive light from the site;
(A) at least one display that can be used to display an image generated by imaging the site;
(B) The scan controller detects light received from the site by one or more light sources that generate light provided to the plurality of scanning devices to illuminate the site and the plurality of light receivers. A scanning controller for controlling the plurality of scanning devices, including one or more detectors, wherein the one or more light sources are shared between the plurality of scanning devices, and the one or more detectors Is a scan controller shared between the plurality of light receivers;
(C) a functional interface for connecting the plurality of scanning devices and the plurality of light receivers to the scanning controller;
(D) connected to the at least one display, the scan controller, and the functional interface to control the system and between the light emitted by the plurality of scanning devices and received by the plurality of light receivers; And an arithmetic unit that generates the image of the part on the at least one display without deterioration due to interference.   The scanning controller includes two or more different light sources for the plurality of scanning devices, and light in different wavelength bands is emitted by the two or more scanning devices. system. The scan controller operates in one of three modes, each of the three modes:
(A) providing light only from one or more light sources to a single selected scanning device that is simultaneously scanning the region of the site;
(B) While simultaneously supplying light from the one or more light sources to the plurality of scanning devices, the light emitted from only one scanning device of the plurality of scanning devices ensures that any region of the site is Irradiating at the same time,
(C) selectively supplying light from the one or more light sources to the plurality of scanning devices using means for supplying light of different characteristics to the plurality of scanning devices, The apparatus of claim 19, comprising: supplying light having characteristics different from the light supplied to any other scanning device that irradiates the same region of the site. system. By a plurality of imaging devices including a plurality of scanning devices that emit light used to illuminate a site and a plurality of light receivers that receive light from the site used to generate an image of the site Using a common base station for imaging the site, comprising:
(A) selectively using the common base station to supply light to the plurality of scanning devices by one or more light sources in the common base station shared by the plurality of scanning devices; Controlling to prevent crosstalk between the light received by the plurality of light receivers from the site;
(B) responding to light received by the plurality of light receivers by generating an output signal using one or more detectors in the common base station shared by the plurality of light receivers. When,
(C) processing the output signal from the detector in the common base station to generate a plurality of images of the site;
(D) allowing a user to display the plurality of images of the site. The step of not causing the crosstalk includes:
(A) supplying light only from the one or more light sources to a selected scanning device that is simultaneously scanning the region of the site;
(B) While simultaneously supplying light from the one or more light sources to a plurality of light sources, the light emitted from only one scanning device of the plurality of scanning devices reliably passes through any region of the part for the time Step to irradiate
(C) The light having different characteristics is supplied to each of the scanning devices so that light having characteristics different from the light supplied to any other scanning device that irradiates the same region of the part is supplied 23. The method of claim 22, comprising a step selected from a group of steps consisting of selectively supplying from one or more light sources to the plurality of scanning devices. The step of selectively supplying comprises:
(A) supplying light of different wavelength bands to different scanning devices;
And (b) modulating the light supplied to the plurality of scanning devices at a different modulation frequency for different scanning devices. 24. The method according to 23. Supplying only the selected scanning device includes supplying the light to only one scanning device during a period of time, the period comprising:
(A) the time required to scan the pixels of the generated image with the light illuminating the site emitted by the selected scanning device;
24. (b) corresponding to one of the time required to scan the entire image generated by the light illuminating the part emitted by the selected scanning device. The method described.

Images