WO2018058013A1 - Endoscopic fluorescence imaging - Google Patents

Abstract

Systems and methods are configured for combined fluorescence imaging and white light imaging of tissue such as during surgical endoscopic procedures. A chip-on-tip type endoscope can be equipped with both white light and blue light LEDs. A single camera or dual cameras are configured with backside illuminated CMOS image sensor(s) to receive and process the white light and fluorescence images. The light sources, image sensors and image processing circuitry are configured to synchronously emit light and record pixels for visible white light and fluorescence frames alternately. Global or quasi-global shuttering can be used on the image sensor(s). A modified color filter array and other filters can be provided to enhance fluorescence imaging capabilities.

Description

ENDOSCOPIC FLUORESCENCE IMAGING

REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of and incorporates by reference each of the following provisional applications:

U.S. Prov. Ser. No. 62/399,429 filed September 25, 2016;

U.S. Prov. Ser. No. 62/399,436 filed September 25, 2016;

U.S. Prov. Ser. No. 62/399,712 filed September 26, 2016;

U.S. Prov. Ser. No. 62/405,915 filed October 8, 2016;

U.S. Prov. Ser. No. 62/423,213 filed November 17, 2016;

U.S. Prov. Ser. No. 62/424,381 filed November 18, 2016;

U.S. Prov. Ser. No. 62/428,018 filed November 30, 2016;

U.S. Prov. Ser. No. 62/429,368 filed December 2, 2016;

U.S. Prov. Ser. No. 62/485,454 filed April 14, 2017;

U.S. Prov. Ser. No. 62/485,641 filed April 14, 2017;

U.S. Prov. Ser. No. 62/502,670 filed May 6, 2017;

U.S. Prov. Ser. No. 62/550, 188 filed August 25, 2017;

U.S. Prov. Ser. No. 62/550,560 filed August 25, 2017;

U.S. Prov. Ser. No. 62/550,581 filed August 26, 2017; and

U.S. Prov. Ser. No. 62/558,818 filed September 14, 2017.

All of the above-referenced provisional patent applications are collectively referenced herein as "the commonly assigned incorporated applications."

FIELD

[0002] This patent specification generally relates mainly to a medical device for use in tissue examinations. More particularly, some embodiments relate to devices and methods for fluorescence imaging in medical applications such as visually detecting tissues such as tumors, nerves, and vessels during surgical procedures.

BACKGROUND

[0003] Endoscopic fluorescence imaging systems can be used to detect tissue such as tumors and vessels during surgical procedures. Infrared dyes can be used as tagging dyes for marking tissue. Some endoscopic fluorescence imaging systems are capable of acquiring high resolution images in the normal white light visible spectrum, while simultaneously acquiring and overlaying the infrared signal on top of normal visible spectrum images in order to provide a contrast to a surgeon while operating. Some systems are designed to detect unbound intravascular indocyanine green (ICG), an FDA approved NIR (near infrared) fluorescent dye. ICG is typically intravenously administered in high doses, and imaging is performed 30-60 minutes after injection. The intravascular fluorescent load achieved with this approach is high, and approved clinical imaging devices have adequate sensitivity for these applications. Examples of such systems include a fluorescent module incorporated into operating microscopes (OPM I Pentero Infrared 800, Carl Zeiss), as well at the SPY® and Pinpoint® systems (Novadaq), and the FluoBeam® 800 (Fluoptics) hand-held unit. While these systems may have adequate sensitivity for intravascular imaging, they may lack practical use in other applications such as targeted tumor-specific NIR fluorescence due to low sensitivity. In simultaneous visible and NIR capture imaging systems, one camera captures the image in the visible spectrum and second camera captures the fluorescent image. This is achieved by splitting the incident light from the field into two channels using a beam-splitter. One beam transmits the NIR fluorescent light to one of the cameras, while the other beam of visible light passes through the beam splitter into the second camera. See, e.g., US Patent 8,961 ,403 B2.

[0004] US Pat. No. 9,407,838 discusses a system suitable for lab analysis or high-end surgical applications that can simultaneously record a visible light image and an infrared light image from fluorescent dye. The discussed system

simultaneously uses a laser for emitting excitation light for an infrared or near- infrared fluorophore and a visible light source. The discussed system also includes a notch beam splitter, a notch filter, a synchronization module, an image sensor for simultaneously detecting both visible light and infrared light, an image processing unit for image subtraction after capture, an image displaying unit, and light- conducting channels.

[0005] The subject matter described or claimed in this patent specification is not limited to embodiments that solve any specific disadvantages or that operate only in environments such as those described above. Rather, the above background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.

SUMMARY

[0006] According to some embodiments, an endoscope distal tip for multi-band imaging of internal patient tissue comprises: a multi-pixel imaging structure configured to receive a first light that is in a first wavelength range and second light that is in a second wavelength range different from the first wavelength range; where said imaging structure comprises a multi-pixel, backside-illuminated light sensor array and a readout circuit that are electrically and physically integrated into a single stack; wherein said readout circuit is configured to provide first image information for respective pixels in response to said first light in the first wavelength range received by the sensor during first time intervals, and second image information for respective pixels in response to said second light in the second wavelength range received by the sensor during second time intervals that are interspersed in time with said first time intervals; and an output from said distal tip providing said image information.

[0007] According to some embodiments, the stack preferably has a diagonal dimension of no more than 3 mm along an image plane of the sensor, more preferably said dimension of no more than 2 mm, and most preferably said dimension is no more than 1 .5 mm.

[0008] According to some embodiments, the second wavelength range matches fluorescence from cancerous tissue in a patient's bladder.

[0009] According to some embodiments, the second wavelength range matches fluorescence from cancerous tissue in a patient's bladder induced at least in part with an agent introduced into the patient that preferentially causes said cancerous tissue to fluoresce.

[0010] According to some embodiments, said second wavelength range represents pink-red color. [0011] According to some embodiments, said second wavelength is at approximately 610 nm.

[0012] According to some embodiments, said second wavelength range matches fluorescence from nerve tissue.

[0013] According to some embodiments, said second wavelength range matches fluorescence from nerve tissue in a patient induced at least in part with an agent introduced into the patient that preferentially causes said nerve tissue to fluoresce.

[0014] According to some embodiments, said second wavelength range represents dark green color.

[0015] According to some embodiments, said second wavelength is at approximately 510 nm.

[0016] According to some embodiments, the endoscope tip can be attached at a distal end of a cannula.

[0017] According to some embodiments, the tip is deflected relative to a long axis of said cannula.

[0018] According to some embodiments, the endoscope tip includes a control selectively varying deflection of the tip relative to a long axis of the cannula through the cannula.

[0019] According to some embodiments, the endoscope includes a handle secured to a proximal end of the cannula and a display screen secured to and integral with the handle, and an electrical connection through the cannula between the readout circuit and the display screen configured to deliver image information from the sensor to the display screen.

[0020] According to some embodiments, the endoscope includes a coupling between the cannula and the handle configured to attach the cannula to the handle and detach the cannula from the handle by hand, without tools, and wherein the cannula and tip are a single-use unit discarded after use on a patient.

[0021] According to some embodiments, the readout circuit in the stack is configured to provide the first image information and the second image information in respective alternating first and second time intervals. [0022] According to some embodiments, at least some of the second time intervals are longer that a least some of the first time intervals.

[0023] According to some embodiments, the readout circuit is configured as a global shutter that essentially concurrently reads the sensor pixels providing the image information.

[0024] According to some embodiments, the readout circuit is configured as a quasi-global shutter that concurrently reads only subsets of the sensor pixels providing the image information.

[0025] According to some embodiments, the endoscope tip includes a source of excitation light promoting fluorescence from said tissue.

[0026] According to some embodiments, the endoscope tip includes a filter keeping at least some of the excitation light from reaching said sensor.

[0027] According to some embodiments, an endoscope distal tip for multi-band imaging of internal patient tissue comprises: a first multi-pixel imaging structure configured to receive a first light that is in a first wavelength range; a second multi- pixel imaging structure configured to receive a second light that is in a second wavelength range different from the first wavelength range; wherein the first imaging structure comprises a first multi-pixel, backside-illuminated light sensor array and a first readout circuit that are electrically and physically integrated into a respective first single stack, and the second imaging structure comprises a second multi-pixel, backside-illuminated light sensor array and a second readout circuit that are electrically and physically integrated into a respective second single stack; the first readout circuit provides image information in response to said first light received by the first sensor; and the second readout circuit provides second image information in response to said second light received by the second sensor; and the distal tip provides an output containing said first and second image information.

[0028] According to some embodiments, the first sensor and the second sensor in the endoscope tip have respective fields of view that overlap, and the first light comes from all tissue in the field of view but the second light comes only from selected tissue in the field of view. [0029] According to some embodiments, the second light illuminating the second sensor has a wavelength range matching fluorescing cancerous tissue in a patient's bladder.

[0030] According to some embodiments, the second light illuminating the second sensor has a wavelength range matching fluorescing nerve tissue in a patient.

[0031] According to some embodiments, each of the first and second stacks has a diagonal dimension no greater than 1 .5 mm.

[0032] According to some embodiments, the first and second stack are side-by- side.

[0033] According to some embodiments, a method of endoscopic multi-band imaging of internal patient tissue comprises: imaging internal patient tissue that is in a field of view with a backside-illuminated multi-pixel sensor configured to receive a first light that is in a first wavelength range and produce first image information and to receive a second light that is in a second wavelength range different from the first wavelength range and produce second image information; wherein the first light represents all tissue in the field of view but the second light preferentially represents only selected tissue in the field of view; reading out the first and the second image information with a readout circuit that is electrically and physically integrated with the sensor into a single stack; wherein said readout circuit is configured to read out the first image information in a sequence of first time intervals and the read out the second image information in a second time intervals that are interspersed in time with the first time intervals; and providing an output from said distal tip containing image information read out by the readout circuit.

[0034] According to some embodiments of the method, the selected tissue is cancerous tissue in a patient's bladder.

[0035] According to some embodiments of the method, the selected tissue is nerves in the patient.

[0036] According to some embodiments of the method, the second light is preferentially fluorescence from selected tissue in the field of view. [0037] According to some embodiments of the method, the first light and the second light impinge on a sensor area that preferably has a diameter or diagonal dimension no greater than 2 mm, and more preferably has a diameter or diagonal dimension no greater than 1 .5 mm.

[0038] According to some embodiments, an endoscope for multi-band imaging of internal patient tissue comprises: a cannula, a tip secured at a distal portion of the cannula, a handle secured to a proximal portion of the cannula, and a display screen mounted to the handle; wherein said tip comprises a multi-pixel imaging structure configured to receive a first light that is in a first wavelength range and second light that is in a second wavelength range different from the first wavelength range; said imaging structure comprises a multi-pixel, backside- illuminated light sensor array and a readout circuit that are electrically and physically integrated into a single stack; wherein said readout circuit is configured to provide first image information for respective pixels in response to said first light in the first wavelength range received by the sensor during first time intervals, and second image information for respective pixels in response to said second light in the second wavelength range received by the sensor during second time intervals that are interspersed in time with said first time intervals; and a connection from the readout circuit to the display screen configured to provide the first and second image information from the readout circuit to the display screen for display thereon and image of the tissue imaged in response to the first light and an image of the tissue imaged in response to the second light.

[0039] According to some embodiments, the sensor in said endoscope preferably has a diagonal dimension no greater than 2 mm, and more preferably no greater than 1 .5 mm.

[0040] According to some embodiments, the endoscope cannula is bent at a distal portion to thereby point the tip in a direction angled relative to a long axis of the cannula.

[0041] According to some embodiments, the endoscope includes a connector configured for tool-free mounting of the cannula to the handle and removing the cannula from the handle. [0042] According to some embodiments, the endoscope includes an array filter that comprises a repeating rectangular pattern of red, green, blue, and green filters (RGBR) through which light passes to become said second light.

[0043] As used herein, the grammatical conjunctions "and", "or" and "and/or" are all intended to indicate that one or more of the cases, object or subjects they connect may occur or be present. In this way, as used herein the term "or" in all cases indicates an "inclusive or" meaning rather than an "exclusive or" meaning.

[0044] As used herein the terms "surgical" or "surgery" refer to any physical intervention on a patient's tissues, and does not necessarily involve cutting a patient's tissues or closure of a previously sustained wound.

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To further clarify the above and other advantages and features of the subject matter of this patent specification, specific examples of embodiments thereof are illustrated in the appended drawings. It should be appreciated that these drawings depict only illustrative embodiments and are therefore not to be considered limiting of the scope of this patent specification or the appended claims. The subject matter hereof will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0046] FIGs. 1 A and 1 B are diagrams illustrating aspects of a conventional CMOS image sensor;

[0047] FIG. 2 is a diagram showing aspects of a distal tip assembly of multicolor band endoscope, according to some embodiments;

[0048] FIGs. 3A and 3B are diagrams illustrating aspects of a CMOS image sensor system having improved visualization of tissue using combined

fluorescence and white light endoscopy, according to some embodiments;

[0049] FIGs. 4A and 4B are schematic diagrams illustrating aspects of a stacked CMOS image sensor system having improved visualization of tissue using combined fluorescence and white light endoscopy, according to some

embodiments; [0050] FIG. 5. is a diagram illustrating aspects of a filter configured to enhancing capture of fluorescence light during combined fluorescence and white light endoscopy imaging, according to some embodiments;

[0051] FIGs. 6A and 6B are charts illustrating some aspects of the timing of illuminating light sources and sensor exposures for combined fluorescence and white light endoscopy imaging, according to some embodiments;

[0052] FIG. 7 is a cross sectional diagram illustrating aspects of a camera module configured for fluorescence endoscopy, according to some embodiments;

[0053] FIG. 8 is a diagram showing aspects of a distal tip assembly having dual camera modules, according to some other embodiments;

[0054] FIG. 9 is a perspective view of a handheld multi-color band endoscope, according to some embodiments; and

[0055] FIG. 10 is a block diagram illustrating aspects of a multi-color band endoscope, according to some embodiments.

DETAILED DESCRIPTION

[0056] A detailed description of examples of preferred embodiments is provided below. While several embodiments are described, it should be

understood that the new subject matter described in this patent specification is not limited to any one embodiment or combination of embodiments described herein, but instead encompasses numerous alternatives, modifications, and equivalents. In addition, while numerous specific details are set forth in the following description in order to provide a thorough understanding, some embodiments can be practiced without some or all of these details. Moreover, for the purpose of clarity, certain technical material that is known in the related art has not been described in detail in order to avoid unnecessarily obscuring the new subject matter described herein. It should be clear that individual features of one or several of the specific embodiments described herein can be used in combination with features of other described embodiments or with other features. Further, like reference numbers and designations in the various drawings indicate like elements. [0057] FIGs. 1 A and 1 B are diagrams illustrating aspects of a conventional CMOS image sensor. A color image is captured using a color filter array (CFA), where the sensor pixels are masked with red (R), green (G) and blue (B) filters. A 2x2 portion 100 of a common conventional CFA known as a Bayer filter is shown in FIG. 1 A. As shown in FIG. 1 B, each color band image is captured by a separate sub-group of pixels of the sensor pixel array. A full-color image, with intensities of all three primary colors represented by each pixel, is then synthesized by merging the separate sub pixel group using a demosaicing algorithm. As shown in FIG. 1A, the standard Bayer filter CFA has 2G 1 R 1 B or "RGBG". With this arrangement, more pixels are allocated to G, because the human eye is most sensitive to green (around 550 nm) and most natural images have high green content.

[0058] In medical imaging, fluorescence techniques allow visualization of features that are invisible or are not easily visible under conventional white light. According to some embodiments, a video endoscope system is provided for fluorescence endoscopy. A protocol is also described to combine the fluorescence endoscopy with white light endoscopy. According to some embodiments, dual color band or multi-color band imaging systems are described herein that provide improved visualization of tissue using combined fluorescence and white light endoscopy.

[0059] FIG. 2 is a diagram showing aspects of a distal tip assembly of a multi- band or multi-color band endoscope, according to some embodiments. According to some embodiments, the endoscope 200 is disposable, or partially disposable after single-use. Endoscopes with disposable cannulas that include a distal tip with a camera are discussed in co-owned pending U.S. Applications Nos. 14/913,867, filed February 23, 2016, and 15/371 ,858, filed December 7, 2016, which are incorporated herein by reference. Other examples of endoscopes with disposable cannulas are discussed in U.S. Patent 9,622,646, also incorporated herein by reference. The distal tip assembly 204 includes a lens/sensor barrel 210 that includes a camera module having a color sensor with a color filter array, and electronics and circuitry as will be described in further detail, infra. Surrounding the lens/sensor barrel 210 are blue LEDs 232 and white LEDs 234. The blue LEDs 232 are configured to emit excitation light suitable for fluorescence endoscopy. In some examples, the blue LEDs 232 are configured to emit light at about 410 nm (violet-blue). The white LEDs 234 are configured to emit white light suitable for visible white light endoscopy. Below the lens/sensor barrel 210 is a port 220 that is configured to provide fluid (flowing either into or out of the patient) and/or provide an opening through which a tool or other device can pass (e.g. a needle).

According to some embodiments, the port 220 is greater than 1 .0 mm in its smallest dimension. According to some embodiments, the LEDs 232 and 234 are approximately 0.75mm x 0.25mm. The outer wall 202, according to some embodiments, is at least 0.25mm thickness and is appropriately rounded to avoid any sharp edges. According to some embodiments, the lens/sensor barrel 210 and the LEDs 232 and 234 can be recessed so that a dome shaped cover can be placed so as to further ensure smoothness of the distal tip. According to some embodiments, the outer diameter of the tip assembly is 12 Fr to 15 Fr (4mm to 5mm), and preferably is no more than 6 mm as the outside diameter of outer wall 202. Note that although FIG. 2 shows a total of six LEDs (three white and three blue), in general, other numbers of LEDs may be provided according to factors such as desired lighting quality, endoscope size, and LED characteristics such as size and brightness. In some embodiments four or fewer LEDs can be provided and in some embodiments 10 or more LEDs can be provided. Furthermore, the number of white and blue LEDs does not have to be equal, but also will depend on various factors. Other light sources can be substituted, such as optic fibers that deliver light generated elsewhere.

[0060] FIGs. 3A and 3B are diagrams illustrating aspects of a CMOS image sensor system having improved visualization of tissue using combined

fluorescence and white light endoscopy, according to some embodiments. In FIG. 3A a 2x2 portion 300 is shown of an RGBR color filter array. A color filter array having the shown pattern will have red resolution roughly 2 times greater than that of blue or green. As can be seen in FIG. 3B, when the RGBR color filter layer 302 is used, the sensor array 304 has enhanced sensitivity to fluorescence light coming from the tissues of interest during an endoscopic procedure. According to some embodiments, the sensor array 304 and RGBR color filter 302 are used in the camera module of endoscope 200 shown in FIG. 2. [0061] According to some embodiments, endoscope 200 can be used for differential imaging of a patient's bladder to determine the presence and characteristics of cancerous tissue. In this example of use, the color filter array 302 is configured to enhance sensitivity to pink-red light (about 610 nm wavelength) by selecting a pattern that preferentially passes pink-red to sensor array 304. The image provided by pink-red light preferentially shows cancerous tissue emitting fluorescent light in that color, typically due to the appropriate dye or other agent introduced in the patient at an appropriate time before imaging. Such dyes or other agents are known in the art of fluorescence medical imaging.

[0062] According to some embodiments, endoscope 200 can be used for differential imaging of a patient's nerves to assist in surgery where it may be desirable to identify nerve tissue so as to avoid damaging such tissue or to perform procedures on such tissue. In this example of use, the color filter array 302 is configured to enhance sensitivity to dark green light (about 510 nm wavelength) by selecting a pattern that preferentially passes dark green to sensor array 304. A filter such as in FIG. 3A but with a pattern RGBG can be used for the purpose. The image provided by dark green light preferentially shows nerve tissue emitting fluorescent light in that color, typically due to the appropriate dye or other agent introduced in the patient at an appropriate time before imaging. Such dyes or other agents are known in the art of fluorescence medical imaging.

[0063] According to some embodiments, light sources 234 can provide light other than white light, sources 232 can provide different color(s) excitation light, and color filter 302 can preferentially pass light in different color(s) to sensor 304, to thereby preferentially image selected tissue with light of a first wavelength range incident on sensor 304 and other selected tissue with light of a second wavelength range, different from the fist range, incident on sensor 304.

[0064] FIGs. 3A and 3B illustrate one example of a filter array 302 but other examples of filter arrays that enhance sensing different colors at sensor 304 are possible. While using a white light image and a color image such as pink-red, or dark green are given as examples, other combinations of images may be useful for imaging patients' tissue, and fluorescent images in two or more colors may be useful. Typically, sensor array 304 is read in alternating or at least interspersed time intervals, for example alternating white image and color image time intervals. The resulting white and color images typically are shown composited, in geometric registration, so that the position of selected tissue such as cancerous tissue or nerve tissue can be visualized relative to surrounding anatomy, but separate white and color images can be displayed instead or in addition.

[0065] FIGs. 4A and 4B are schematic diagrams illustrating aspects of a stacked CMOS image sensor system having improved visualization of tissue using combined fluorescence and white light endoscopy, according to some

embodiments. FIG. 4A depicts RGBR color filter 302 formed within an upper portion of sensor array 304, which includes backside-illuminated (BSI) CMOS photo diodes. The sensor array 304 is bonded directly to a logic circuit chip 430 that contains an image signal processor (ISP). FIG. 4B is a side view of the BIS CMOS image sensor (CIS) stack 400 containing the RGBR color filter array 302 and logic circuit chip 430. Using a 3D stacked image sensor 400 consisting of a BSI image sensor die face-to-face stacked on a logic die containing an image signal processor (ISP) has a number of advantages for combined fluorescence and white light endoscopy applications, one of which is the ability to use global shuttering and/or quasi-global shuttering. The stacked image sensor and ISP arrangement shown in FIGs 4A and 4B allows for readout of the entire area of the pixel array simultaneously. The image can be captured simultaneously using all pixels. According to some embodiments, the image sensor/ISP architecture includes a memory structure and additional MOS transistors to provide additional functionality and increased read-out speed. Another benefit of using a stacked image sensor in endoscopy applications is that the fast read-out times allow for higher quality moving image capabilities which is desirable for imaging during surgical procedures. Furthermore, the use of stacked arrangement such as shown in FIGs. 4A and 4B allows for a more compact sensor chip since the circuit section of the chip does not take up additional sensor chip surface area. Having a compact sensor chip is beneficial in chip-on-tip endoscopy applications since it is desirable to reduce the frontal area occupied by the camera module.

[0066] According to some embodiments, stack 400 can be a stacked back- illuminated structure similar to a stacked BSI CMOS device from Sony in the family of devices offered under the trade designation Exmor RS sensors. Preferably, stack 400 is an integrated, single-chip device comprising individual photo-diodes, each under a respective color filter through which light from tissue passes before impinging on the photo-diode, metal wiring under the photodiodes, and image processing circuitry under the metal wiring to read the electrical output of the photo-diodes and convert it to image information to be sent for further processing and/or display. One example of such stack is designated Sony Exmor RS sensor stack IMX 398, which comprises a 12 MP sensor (4608x3456) and has a 6.4 mm diagonal dimension. The stack comprises a sensor layer and a logic layer that is behind the sensor layer and contains readout circuits to read out the electrical signals that the sensor pixels provide in response to light. For use in endoscope 200, the diagonal dimension of sensor stack 400 preferably is no more than 3 mm, more preferably no more than 2 mm, and even more preferably no more than 1 .5 mm. This can be accomplished by reducing the number of pixels in the sensor, for example to a number sufficient to provide lower spatial resolution such as 400x400, or by shrinking the dimensions of sensor elements such as photo-diodes, or both.

[0067] According to some embodiments, the stacked arrangement can include one or more additional filters 440 to further enhance combined fluorescence and white light endoscopy imaging. For example, filters 440 often include an infrared (IR) filter configured to suppress IR sensitivity, or a different color filter configured to suppress sensitivity to a different color as required for a specific tissue or specific use of endoscope 200. Filters 440 can also include a band pass filter that is configured to allow passage of wavelengths around 600 nm, or some other wavelength(s) that coincide with fluorescent light coming from the tissues of interest. Further examples of possible filters are shown and described with respect to FIG. 5, infra. According to some embodiments, the stacked arrangement can include further layers, such as a DRAM layer 450 (shown in FIG. 4B) that can be configured to store signals read at high speed temporarily, enabling even shorter read-out times. A stack 400 with a built-in DRAM layer can be similar in structure to, but preferably smaller than, a device offered by Sony under the designation Exmor RS IMX 400. In this case, the stack comprises a sensor layer, a DRAM layer, and a logic (readout) layer, stacked in the direction of the light that illuminates the sensor layer. A built-in DRAM layer for temporary storage of images need not be used when the read-out circuitry in stack 400 is sufficiently fast to send the image information from sensor array 304 essentially in real time to processing circuitry elsewhere, such as in handle 940 (FIG. 9) or in display 950 (FIG. 9), where the output of stack 400 can be converted into images for display and/or storage. According to some embodiments, the stacked CMOS image sensor system(s) shown in FIGs. 4A and 4B are used in the camera module of endoscope 200 shown in FIG. 2.

[0068] FIG. 5. is a diagram illustrating aspects of a filter configured to enhance capture of fluorescence light during combined fluorescence and white light endoscopy imaging, according to some embodiments. Shown in the main plot are curves 510, 512 and 514 which are pixel response curves for respective colors in an example of a CMOS sensor array. One or more filters, such as filter(s) 440 shown in FIG. 4B, can be configured to allow passage of light in region 520.

According to some embodiments, wavelengths below about 420nm (region 524) and above 700nm (region 526) are heavily filtered out. According to some embodiments, the filter(s) can be further configured selectively to reduce light having wavelengths in region 522. Note that in practice the precise filtering characteristics and wavelengths blocked and/or passed will depend on the materials and structures used for filtering.

[0069] The reduction in region 522 has been found to be useful in reducing the amount of blue light "background" or "noise" that is recorded by the "red pixels" (the pixels associated with "R" in the color filter array). As can be seen in FIG. 5, a typical CMOS pixel response curve for red pixels, curve 514, has some non-zero response in the blue region (region 522). In particular, the cross-hatched area 516 represents that the red pixels will pick up light in the region near the fluorescence excitation source light (e.g. from 350nm to 450nm). Since the excitation source light in fluorescence endoscopy can be several times brighter than the resultant fluorescence light, the excitation light can show up as background or noise in the red pixel image. It has been found that by selectively suppressing the blue light in the region 522, and especially near the wavelengths of the source light, the background noise can be greatly reduced in the red pixel image. The amount of reduction in 522 can be tailored to the particular characteristics of the sensor and image processing used, as well as the particular surgical application(s) the endoscope is directed to. The design of filter 440 should also take into account the properties of the color filter array 302 (shown in FIGs. 3B, 4A and 4B). For example, in many cases the color filter array 302 will heavily filter out wavelengths below 420nm. In such cases the filter 440 can be configured to heavily reduce infrared light greater than 700nm, and selectively reduce blue light in the region 522 to reduce blue light background from the red pixels. According to some embodiments, some or all of filtering aspects of filter 440 described herein can be integrated into the design of the color filter array 302. For example, for

suppressing the blue light background recorded by the red pixels in the region 522, the described blue light suppression can be partially or fully accomplished by color filter array 302. Specifically, the red filter portions of filter 302 can be configured to suppress more light in region 522 than red filters in typical CFA designs (e.g. curve portion 516). According to some embodiments, filter 440 can be a liquid crystal tunable filter (LCTF) that can be selectively switched between being clear or nearly clear so it is essentially transparent and being in a state in which it blocks or significantly attenuates some wavelengths, as is described in further detail with respect to FIG. 7, infra.

[0070] FIGs. 6A and 6B are charts illustrating some aspects of the timing of illuminating light sources and sensor exposures for combined fluorescence and white light endoscopy imaging, according to some embodiments. Four time intervals 610, 612, 614 and 616 are shown wherein white light sources (e.g. White LEDs 234 in FIG. 2) and blue light sources (e.g. Blue LEDs 232 in FIG. 2) are alternately activated. During the T-White intervals 610 and 614 the white light sources are energized while the global shutter allows passage of white light and are read out of the entire sensor array (e.g. of sensor 304 in FIGs 4A and 4B). Image processing is pre-formed for the white light frames. Using all of the pixels the white light frames are processed, for example, using ISP circuitry 430 and possibly other processors. During the T-Blue intervals 612 and 616 the blue light sources are energized while the global shutter is set to filter out or significantly attenuate selected wavelengths such as in the blue region and are read out of the R pixels from the sensor array (e.g. of sensor 304 in FIGs 4A and 4B). Image processing is pre-formed for the red color band frames. Using all of the "R" pixels, the red light frames are processed, for example, using ISP circuitry 430 and possibly other processors. Thus, illumination light (white or blue) is synchronized with the appropriate global shuttering and image processing. The alternately captured white light video and sub-band video (R pixels) present different information of the object being imaged. The white light band image presents a normal view of the object, while the sub-band light image presents additional information about the object. According to some embodiments, the two video types are overlaid by the system processor. In medical applications, displaying the composite video allows clinicians to locate a tissue of interest (e.g. a diseased tissue) because the sub-band image precisely "highlights" the tissue of interest within an ordinary white light background image. This type of overlying or compositing is particularly useful in endoscopic surgical procedures.

[0071] According to some embodiments, the time periods can alternate between white and a color but in other embodiments the periods can be

interspersed in a different sequence, for example in the sequence T-white, T-blue, T-blue, T-white, T-blue, T-white, etc. In addition, two or more colors can be included in the same sequence of time intervals, for example to form a sequence T-white, T-blue, T-red, T-white, etc.

[0072] According to some embodiments, the image processing can be configured differently for the white and red color band frames to further enhance visualization. For example, during the T-Blue intervals (612 and 616) the relative weights of the three-color channels are manipulated to enhance the visibility of the fluorescent tissue. The red channels can be boosted while the blue channels can be suppressed, by adjusting their relative gains. This will enhance the fluorescent tissue while reducing blue reflectance from the excitation light source (the blue LEDs).

[0073] According to some embodiments, in addition to alternating the white and blue light illumination, the duration of the blue (and/or other color(s)) and white light intervals can be manipulated to provide enhanced imaging for certain applications. For example, in some cases the fluorescence image may be too weak compared to the white image. In such cases the T-White and T-Blue interval timing can be configured such that the T-Blue intervals 612 and 616 are longer than the T-White intervals 610 and 614. In some examples, the T-Blue intervals can be 2 or more times longer than the T-White intervals so that the fluorescence image in enhanced over the white image.

[0074] According to some embodiments, quasi-global shuttering can be used in some cases instead of full global shuttering. Especially in case of a lower cost, smaller sized pixel array, quasi global shuttering can be used wherein the pixels in an entire column (or row) are read out in series, but all of the columns (or rows) are read out simultaneously. In this case each column (or row) has a dedicated analog to digital converter (ADC). For example, if the array size is 400x400 pixels (and the pixel size is about 2.2 um x 2.2 um to 3.0 um x 3.0 um) with a pixel read-out time of 5 u sees, each column (or row) can be read-out in about 2 ms, which leaves plenty of time for exposure (30+ ms for exposure at 30 frames per second). Note that in such quasi-global shuttering embodiments, the sensor readout in the T-Blue intervals (612 and 616) will include all of the pixel data, and the non-red pixel data can simply be ignored when forming the red color band frames.

[0075] FIG. 7 is a cross sectional diagram illustrating aspects of a camera module configured for fluorescence endoscopy, according to some embodiments. The cross section shows a camera module and includes the lens/sensor barrel 210 which is also visible in FIG. 2. The outer body of the module is housed by camera module holder 702. The lens cover 710 overlies iris 712. One or more lens components are housed within holder 702, such as lens components 720 and 722 in this example shown. A BIS CMOS image sensor (CIS) stack 400 is shown bonded to circuit board 740 in this example. According to some embodiments, the CMOS sensor stack 400 includes an RGBR color filter array such as filter layer 302 shown in FIGs. 3B, 4A and 4B. One or more filters 440 are shown overlaid on the sensor stack 400. According to some embodiments, the filter(s) 440 can include a tunable filter, such as liquid crystal tunable filter (LCTF), that uses electronically controlled liquid crystal (LC) elements to transmit a selectable wavelength of light and exclude or suppress others. The LCTF filter can be dynamically controlled by a system processor to be in one of two states that are synchronized with the timing of illumination, global shutter (or quasi-global) capture mode, and image processing, as shown in FIGs. 6A and 6B. During the T-White intervals, (610 and 614 in FIGs. 6A and 6B) the LCTF filter is set to a "clear" state to allow passage of all colors of the spectrum. During the T-Blue intervals (612 and 616 in FIGs. 6A and 6B) the LCTF filter is set to a "blue" state which blocks most of all of the blue light from entering the sensor 304. In this way, the captured fluorescence light signal can be significantly enhanced by greatly reducing the undesirable background blue light or other spurious fluorescence or phosphorescence light while the white light images are unaffected.

[0076] FIG. 8 is a diagram showing aspects of a distal tip assembly having dual camera modules, according to some other embodiments. According to some embodiments, the endoscope portion 800 is disposable, or partially disposable after single-use, as in the case of endoscope portion 204 shown in FIG. 2. The distal tip assembly in this case includes two camera modules: a white camera module 810 and a blue camera module 812. The white camera module 810 includes a CMOS sensor having an ordinary color filter array (as shown in FIGs. 1A and 1 B), while the blue camera module 812 is specially configured to image the fluorescent light. According to some embodiments, the blue camera module 812 has no color filter array, but rather has a pass band filter with a narrow pass band corresponding to the fluorescent light (e.g. 600nm or 610nm). According to some embodiments, a single stacked CIS chip structure 850 can be used for both camera modules 810 and 812. In such cases, the stacked CIS structure is configured to selectively read out the portion of the sensor structure belonging to the white camera module and blue camera module during the appropriate intervals. Surrounding the camera modules 810 and 812 are a number of blue LEDs 832 and white LEDs 834. The blue LEDs 832 are configured to emit excitation light suitable for fluorescence endoscopy. In some examples, the blue LEDs 832 are configured to emit light at about 410 nm. The white LEDs 834 are configured to emit white light suitable for visible white light endoscopy. Also visible is a port 820 that is configured to provide fluid (flowing either into or out of the patient), and/or provide an opening through which a tool or other device can pass (e.g. a needle). According to some embodiments, the port 820 has an inner diameter of 1 .0 mm, and the outer diameter of the entire tip is less than 6mm on its largest dimension.

[0077] According to some embodiments, the blue camera module 812 is optimized to maximize sensitivity to the fluorescence light, while minimizing interference from other light sources. Since in this example the CMOS sensor does not have a CFA which would cause loss of incoming light, a filter can be used to block the undesirable blue light from entering the image sensor. According to some embodiments, a combined filter or separate filter can also be used to block undesired spurious fluorescence not originated from the targeted tissue. A combined filter or separate filter can also be used to block undesired spurious phosphorescence light from entering the blue camera. According to some embodiments, a band pass filter can be used that blocks all the wavelengths below about 580nm and in the infrared band. One or more filters can be placed either in front of the camera lens (e.g. between the outermost element and the iris, or outside the iris. Alternatively, or in addition, the filter(s) can form part of the sensor lens stack (such as shown with filter(s) 440 in FIG. 7).

[0078] FIG. 9 is a perspective view of a handheld multi-color band endoscope, according to some embodiments. The endoscope 200 includes an elongated cannula 920 with a distal tip 204 for inserting into a hollow organ or cavity of the body. Note that further details of tip 204 are shown in FIG. 2, supra. According to some embodiments, tip 204 includes a camera module and white and blue LEDs, as is shown in more detail in FIG. 2, and/or other colors. According to some embodiments, tip 204 can include dual camera modules such as shown in FIG. 8, supra. According to some embodiments, the distal end of the cannula 920 can also be slightly bent.

[0079]The endoscope 200 includes a handle portion 940 that is sized and shaped for easy grasping by the endoscope operator (e.g. doctor or other medical professional). According to some embodiments, the cannula 920 includes a fluid channel which is fluidly connected to a proximal fluid port (not shown) and port 220 (shown in FIG. 2). According to some embodiments, the channel within the cannula 920 can also be used as working channel via a proximal opening (not shown). According to some embodiments, a re-usable portions of endoscope 200 is removably mounted to enable some portions of endoscope 200 to be re-used while other portions intended to be disposed of after single-use. For example, endoscope 200 can be configured with a connector between the handle portion 940 so it can be detached from cannula 920, as discussed for the endoscope shown in FIGs. 1 and 2 of said pending U.S. application Ser. No. 14/913,867. In this case the cannula 920, including the distal tip 204 can be disposed of after a single-use while the handle portion 940 (including electronics and a re-chargeable battery) and display module 950 can be re-used many times. Other configurations are possible. For example, in some embodiments the display module 950 includes much or all of the electronics and a re-chargeable battery, and the display module is configured to be removable from the handle portion 940. In this case the handle portion 940 and cannula 920 can be configured and intended to be disposed of after a single use while the display module 950 can be re-used many times.

Display 950 can be removably mounted on the upper side of handle portion 940 as shown. By making some portions of the device 200 all single-use, stringent decontamination and disinfection procedures, as well as the risk of cross- contamination and hospital acquired diseases, can be significantly lessened or avoided. Endoscope tip 204 alternatively can be used as a part of a non- disposable endoscope cannula, if made such that it can be suitably

decontaminated between patients.

[0080] A distal portion of cannula 920 can be bent so tip 204 can point to the tissue to be imaged, as in the case of the tip portion of the cannula in FIGs. 1 and 2 of said application Ser. No. 14/913,867. When the tip portion is bent in this manner, the professional using endoscope 200 can rotate it around the long axis of cannula 920 such that tip 204 points to the desired tissue.

[0081] If desirable to change the orientation of tip 204 relative to the long axis of cannula 920, endoscope 200 can be provided with a facility to deflect tip 204 relative to cannula 920 before and/or during a patient examination. For example, a distal portion of cannula 920 can be made of a material that can be bent and retains its shape after being bent, so that a desired angle between tip 204 and the long axis of cannula 920 can be set before introducing cannula 920 into a patient. If desirable to change the angle while cannula 920 is in a patient, endoscope 200 can be provided with a tip deflecting mechanism, for example of the type discussed in U.S. Patent 9,549,666 B2, which is hereby incorporated by reference. Deflection of tip 204 relative to the long axis of cannula 920 can be arranged so it can be in a single plane, e.g., left-right deflection using two cables to pull tip 204 to the left or to the right (or up-down), or it can be to any angle, using more cables or an arrangement similar to that in said U.S. Patent 9,549,666 B2.

[0082] Endoscope portions such as 200 and 800 can be used in endoscopes having other configurations, for example in endoscopes that have straight cannulas, have different handles, and different image displays, and in endoscopes in which the cannula is re-usable rather than being a single-use, disposable cannula.

[0083] FIG. 10 is a block diagram illustrating aspects of a multi-color band endoscope, according to some embodiments. As shown, the LEDs (e.g. 232 and 234 shown in FIG. 2) and BIS CMOS image sensor (CIS) stack 400 (also shown in FIGs. 4B and 7) are located in the distal tip assembly 204. The camera controls, LED controls and image processing 1010 is handled elsewhere in the endoscope, such as in the handle and/or display module. In cases where a portion of the endoscope is configured for single-use and other portions are re-usable, the components 1010 can be located in the re-usable portion(s). Note that a field programmable gate array (FPGA) 1020 can be provided to handle some preprocessing such as demosaicing, gain modification, etc. The control bus 1012 can use, for example, IC2 protocol. The external interfaces 1014 can be used, for example, to receive user input such as from buttons and/or touch screen displays. Such input could be used, for example, to make image adjustments, lighting adjustments and/or select either an "all-white" or "all-blue" rather than the default combined white mode and blue mode overlaid image.

[0084] Although the foregoing has been described in some detail for purposes of clarity, it will be apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways of implementing both the processes and apparatuses described herein. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the body of work described herein is not to be limited to the details given herein, which may be modified within the scope and equivalents of the appended claims.

Claims

CLAIMS What it claimed is:

1 . An endoscope distal tip for multi-band imaging of internal patient tissue, comprising:

a multi-pixel imaging structure configured to receive a first light that is in a first wavelength range and second light that is in a second wavelength range different from the first wavelength range;

said imaging structure comprising a multi-pixel, backside-illuminated light sensor array and a readout circuit that are electrically and physically integrated into a single stack;

wherein said readout circuit is configured to provide:

first image information for respective pixels in response to said first light in the first wavelength range received by the sensor during first time intervals; and

second image information for respective pixels in response to said second light in the second wavelength range received by the sensor during second time intervals that are interspersed in time with said first time intervals; and

an output from said distal tip providing said image information.

2. The endoscope tip of claim 1 , in which the stack has a diagonal dimension of no more than 3 mm along an image plane of the sensor.

3. The endoscope tip of claim 2, in which said dimension of no more than 2 mm.

4. The endoscope tip of claim 3, in which said dimension of no more than 1 .5 mm.

5. The endoscope tip of claim 1 , in which said second wavelength range

matches fluorescence from cancerous tissue in a patient's bladder.

6. The endoscope tip of claim 1 , in which said second wavelength range matches fluorescence from cancerous tissue in a patient's bladder induced at least in part with an agent introduced into the patient that preferentially causes said cancerous tissue to fluoresce.

7. The endoscope tip of claims 1 , in which said second wavelength range represents pink-red color.

8. The endoscope tip of claims 1 , in which said second wavelength is at

approximately 610 nm.

9. The endoscope tip of claim 1 , in which said second wavelength range

matches fluorescence from nerve tissue.

10. The endoscope tip of claim 1 , in which said second wavelength range

matches fluorescence from nerve tissue in a patient induced at least in part with an agent introduced into the patient that preferentially causes said nerve tissue to fluoresce.

1 1 . The endoscope tip of claims 1 , in which said second wavelength range represents dark green color.

12. The endoscope tip of claims 1 , in which said second wavelength is at

approximately 510 nm.

13. The endoscope tip of claim 1 , further including a cannula having a distant end to which said tip is secured.

14. The endoscope tip of claim 13, in which the tip is deflected relative to a long axis of said cannula.

15. The endoscope tip of claim 13, including control selectively varying

deflection of the tip relative to a long axis of the cannula through the cannula.

16. The endoscope tip of claims 13, further including a handle secured to a proximal end of the cannula and a display screen secured to and integral with the handle, and including an electrical connection through the cannula between the readout circuit and the display screen configured to deliver image information from the sensor to the display screen.

17. The endoscope tip of claims 16, including a coupling between the cannula and the handle configured to attach the cannula to the handle and detach the cannula from the handle by hand, without tools, and wherein the cannula and tip are a single-use unit discarded after use on a patient.

18. The endoscope tip of claims 1 , in which the readout circuit in the stack is configured to provide the first image information and the second image information in respective alternating first and second time intervals.

19. The endoscope tip of claims 1 , in which at least some of the second time intervals are longer that a least some of the first time intervals.

20. The endoscope tip of claims 1 , in which the readout circuit is configured as a global shutter that essentially concurrently reads the sensor pixels providing the image information.

21 . The endoscope tip of claims 1 , in which the readout circuit is configured as a quasi-global shutter that concurrently reads only subsets of the sensor pixels providing the image information.

22. The endoscope tip of claims 1 , further including a source of excitation light promoting fluorescence from said tissue.

23. The endoscope tip of claims 22, further including a filter keeping at least some of the excitation light from reaching said sensor.

24. An endoscope distal tip for multi-band imaging of internal patient tissue, comprising: a first multi-pixel imaging structure configured to receive a first light that is in a first wavelength range;

a second multi-pixel imaging structure configured to receive a second light that is in a second wavelength range different from the first wavelength range;

wherein:

the first imaging structure comprises a first multi-pixel, backside- illuminated light sensor array and a first readout circuit that are electrically and physically integrated into a respective first single stack; and

the second imaging structure comprises a second multi-pixel, back- illuminated light sensor array and a second readout circuit that are electrically and physically integrated into a respective second single stack;

the first readout circuit provides image information in response to said first light received by the first sensor; and

the second readout circuit provides second image information in response to said second light received by the second sensor; and

an output from said distal tip providing said first and second image

information.

25. The endoscope tip of claim 24, in which the first sensor and the second sensor have respective fields of view that overlap, and the first light comes from all tissue in the field of view but the second light comes only from selected tissue in the field of view.

26. The endoscope tip of claim 25, in which the second light has a wavelength range matching fluorescing cancerous tissue in a patient's bladder.

27. The endoscope tip of claim 25, in which the second light has a wavelength range matching fluorescing nerve tissue in a patient.

28. The endoscope tip of claim 24, in which each of the stacks has a diagonal dimension no greater than 1 .5 mm.

29. The endoscope of claim 28, in which the first and second stack are side-by- side.

30. A method of endoscopic multi-band imaging of internal patient tissue,

comprising:

imaging internal patient tissue that is in a field of view with a backside- illuminated multi-pixel sensor configured to receive a first light that is in a first wavelength range and produce first image information and to receive a second light that is in a second wavelength range different from the first wavelength range and produce second image information;

wherein the first light represents all tissue in the field of view but the second light preferentially represents only selected tissue in the field of view; reading out the first and the second image information with a readout circuit that is electrically and physically integrated with the sensor into a single stack;

wherein said readout circuit is configured to read out the first image

information in a sequence of first time intervals and the read out the second image information in a second time intervals that are interspersed in time with the first time intervals; and

an output from said distal tip providing image information read out by the readout circuit.

31 . The method of claim 30, in which the selected tissue is cancerous tissue in a patient's bladder.

32. The method of claim 30, in which the selected tissue is nerves in the

patient.

33. The method of claim 30, in which the second light is preferentially fluorescence from selected tissue in the field of view.

34. The method of claim 30, including collecting the first light and the second light at a sensor area that has a diameter no greater than 2 mm.

35. The method of claim 30, including collecting the first light and the second light at a sensor area that has a diameter no greater than 1 .5 mm.

36. An endoscope for multi-band imaging of internal patient tissue, comprising: a cannula, a tip secured at a distal portion of the cannula, a handle secured to a proximal portion of the cannula, and a display screen mounted to the handle;

said tip comprising a multi-pixel imaging structure configured to receive a first light that is in a first wavelength range and second light that is in a second wavelength range different from the first wavelength range; said imaging structure comprising a multi-pixel, backside-illuminated light sensor array and a readout circuit that are electrically and physically integrated into a single stack;

wherein said readout circuit is configured to provide:

first image information for respective pixels in response to said first light in the first wavelength range received by the sensor during first time intervals; and

second image information for respective pixels in response to said second light in the second wavelength range received by the sensor during second time intervals that are interspersed in time with said first time intervals;

a connection from the readout circuit to the display screen configured to provide the first and second image information from the readout circuit to the display screen for display thereon of an image of tissue imaged in response to the first light and an image of tissue imaged in response to the second light.

37. The endoscope of claim 36, in which each of the sensor has a diagonal dimension no greater than 2 mm.

38. The endoscope of claim 36, in which the sensor has a diagonal dimension no greater than 1 .5 mm.

39. The endoscope of claim 36, in which the cannula is bent at a distal portion to thereby point the tip in a direction angled relative to a long axis of the cannula.

40. The endoscope of claim 36, further including a connector configured for tool-free mounting of the cannula to the handle and removing the cannula from the handle.

41 . The endoscope of claim 36, further including an array filter that comprises a repeating rectangular pattern of red, green, blue, and green filters (RGBR) through which light passes to become said second light.