WO2013025530A1 - An image capture unit in a surgical instrument - Google Patents

An image capture unit in a surgical instrument Download PDF

Info

Publication number
WO2013025530A1
WO2013025530A1 PCT/US2012/050394 US2012050394W WO2013025530A1 WO 2013025530 A1 WO2013025530 A1 WO 2013025530A1 US 2012050394 W US2012050394 W US 2012050394W WO 2013025530 A1 WO2013025530 A1 WO 2013025530A1
Authority
WO
WIPO (PCT)
Prior art keywords
image capture
image
sensor
received light
light
Prior art date
Application number
PCT/US2012/050394
Other languages
English (en)
French (fr)
Inventor
Ian M. Mcdowall
John D. Stern
Original Assignee
Intuitive Surgical Operations, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/209,121 external-priority patent/US8684914B2/en
Priority claimed from US13/209,041 external-priority patent/US8764633B2/en
Priority claimed from US13/208,982 external-priority patent/US8672838B2/en
Priority claimed from US13/209,084 external-priority patent/US8734328B2/en
Priority claimed from US13/209,158 external-priority patent/US8784301B2/en
Priority to KR1020227012984A priority Critical patent/KR102529996B1/ko
Priority to KR1020197010909A priority patent/KR102079689B1/ko
Priority to CN202011093468.1A priority patent/CN112220438B/zh
Application filed by Intuitive Surgical Operations, Inc. filed Critical Intuitive Surgical Operations, Inc.
Priority to KR1020217020786A priority patent/KR102390141B1/ko
Priority to KR1020207029014A priority patent/KR102274896B1/ko
Priority to KR1020207004427A priority patent/KR102166302B1/ko
Priority to EP12824330.0A priority patent/EP2741702B1/en
Priority to CN201280050585.7A priority patent/CN103889353B/zh
Priority to KR1020147006103A priority patent/KR101971211B1/ko
Priority to JP2014525180A priority patent/JP6258201B2/ja
Priority to KR1020237015005A priority patent/KR20230062898A/ko
Publication of WO2013025530A1 publication Critical patent/WO2013025530A1/en

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000095—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope for image enhancement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00064—Constructional details of the endoscope body
    • A61B1/00071—Insertion part of the endoscope body
    • A61B1/0008—Insertion part of the endoscope body characterised by distal tip features
    • A61B1/00096—Optical elements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00163—Optical arrangements
    • A61B1/00193—Optical arrangements adapted for stereoscopic vision
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/05—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by the image sensor, e.g. camera, being in the distal end portion
    • A61B1/051—Details of CCD assembly
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B18/00—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B2018/00982—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body combined with or comprising means for visual or photographic inspections inside the body, e.g. endoscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • A61B2090/3618—Image-producing devices, e.g. surgical cameras with a mirror
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras

Definitions

  • aspects of this invention are related generally to endoscopic imaging and are more particularly related to capturing light from a common front end optical structure in a plurality of image capture sensors.
  • da Vinci ® Surgical Systems commercialized by Intuitive Surgical, Inc., Sunnyvale, California, are minimally invasive teleoperated surgical systems that offer patients many benefits, such as reduced trauma to the body, faster recovery, and shorter hospital stay.
  • One key component of a da Vinci ® Surgical System e.g., the model IS3000, da Vinci® Si HD
  • Such electronic stereoscopic imaging systems may output high definition video images to the surgeon, and may allow features such as zoom to provide a "magnified" view that allows the surgeon to identify specific tissue types and characteristics, as well as to work with increased precision.
  • an image capture system is coupled to a proximal end (away from the surgical site) of a stereoscopic endoscope.
  • some stereoscopic endoscopes have included image capture components in the distal end (nearest the surgical site) of the endoscope.
  • Figs. 1A to ID are examples of image capture sensor configurations in a distal end of a stereoscopic endoscope from U.S. Patent No. 4,873,572 (filed Feb. 24, 1988).
  • a distal end 100A of an endoscope includes a plate-like package 113A with a center line coinciding with a longitudinal axis 133A of the endoscope.
  • Two charge coupled devices (CCDs) 114A1 and 114A2 are mounted on opposing surfaces of package 113A.
  • Two objective lenses 115A1 and 115A2 are symmetrically arranged on both sides of longitudinal axis 133A of the endoscope.
  • Mirrors 116A1, 116A2 are symmetrically arranged on the optical axis of the respective objective lenses 115A1, 115A2.
  • a distal end 100B of an endoscope includes two objective lenses 115B1, 115B2 arranged the same as objective lenses 115A1 and 115A2 in Fig. 1A.
  • Mirrors 116B1 and 116B2 are mounted with the mirror surfaces parallel to and removed from longitudinal axis 133B of the endoscope.
  • Light reflected from an object external to the endoscope passes through objective lenses 115B1, 115B2 and is reflected by mirrors 116B1, 116B2 to refracting prisms 117B1, 117B2.
  • the optical path from prisms 117B1, 117B2 is to the imaging surfaces of CCDs 114B1, 114B2.
  • CCDs 114B1 and 114B2 are mounted so that the imaging surfaces of CCDs 114B1, 114B2 intersect at right angles with the optical axis of the optical path from prisms 117B1, 117B2, respectively.
  • CCD 114B1 and 114B2 are each mounted with the imaging surface inclined at a
  • two objective lenses 115C1 and 115C2 are eccentric, for example, to the upper side from the center axis of the lens.
  • Reflecting prisms 117C1 and 117C2 are arranged on the optical axes of the respective objective lenses 115C1 and 115C2.
  • the centers of prisms 115C1 and 115C2 are positioned at a same height as the respective objective lenses 115C1 and 115C2, but are somewhat displaced in the horizontal direction.
  • Prism 117C1 is somewhat displaced to the left from objective lens 115C1 and the prism 117C2 is somewhat displaced to the right from objective lens 115C2.
  • each of prisms 117C1 and 117C2 is reflected by the respective slopes of prism 118C to form an image on the imaging surface of CCD 114C fitted to package 113C.
  • Video signals from CCD 114C are transmitted to a video processor external to the endoscope.
  • a distal end 100D of an endoscope includes two eccentric objective
  • lenses 115D1, 115D2 arranged the same as objective lenses 115C1 and 115C2 in Fig. 1C.
  • the positions of prisms 117D1 and 117D2 are displaced forward and rearward in comparison to prisms 117C1 and 117C2 in Fig. 1C.
  • the light from prisms 117D1 and 117D2 is reflected respectively by mirrors 118D1 and 118D2 to form respective images on CCDs 114D1 and 114D2 mounted adjacently on package 113D, which is parallel to longitudinal axis of the endoscope.
  • One mirror 118D1 is concave and so forms an image on CCD 114D1 for a somewhat shorter optical path length than the optical path length for the image on CCD 114D2.
  • the left optical channel has a shorter optical path length than the right optical channel.
  • Video signals from CCDs 114D1 and 114D2 are transmitted to a video processor external to the endoscope.
  • Figs. 1 A to ID illustrate a few ways of capturing a stereo image in the constrained space of an endoscope tip. But since a small outer diameter of the endoscope distal end is desirable, the configurations in these figures also illustrate how difficult it is to capture high quality stereoscopic images in small outer diameter distal-end image capture systems due to many problems.
  • Figs. 1C and ID require the image to lie flat in the plane of the optics. Either the CCD or the optical components cannot lie on the mid-plane of a round endoscope tip thus these configurations require either very small optical components (and a small inter-pupillary distance) or a very small CCD which limits the imaging quality as the area is small thus restricting the number of pixels and/or the pixel size. Also, in the configuration of Fig. ID, the optical path lengths have different lengths, and so the optical components for each channel must be different.
  • An image capture unit with coplanar image capture sensors overcomes shortcomings of the prior art cameras used in the distal end of an endoscope and provides many new capabilities.
  • Each of the coplanar image capture sensors in a channel of the endoscope has a common front end optical structure, e.g., the lens assembly in the image capture unit is the same for each of the sensors.
  • the common optical and coplanar configuration of the image capture sensors eliminates the need for calibration of lens artifacts. Re-registration of different images captured in independent channels of the endoscope is not required.
  • the images captured in a channel of the endoscope are temporally registered.
  • the images are also spatially registered relative to each other.
  • Visible images of a scene or a visible image of a scene and one or more fluorescence images in the scene are acquired by the image capture unit.
  • a controller enhances the acquired images.
  • the enhanced images are displayed on a stereoscopic display, in one aspect.
  • the enhanced images may include (a) visible images with enhanced feature definition, in which a particular feature in the scene is emphasized to the operator of a minimally invasive surgical system, for example; (b) images having increased image apparent resolution; (c) images having increased dynamic range; (d) images displayed in a way based on a pixel color component vector having three or more color components; and (e) images having extended depth of field.
  • an image capture unit includes a first image capture sensor with a first sensor surface and a second image capture sensor with a second sensor surface.
  • the first and second sensor surfaces are coplanar.
  • the first surface is in a first plane and the second surface is in a second plane.
  • the first and second planes are parallel, and are separated by a known distance.
  • a beam splitter in the image capture unit, is positioned to receive light. The beam splitter directs a first portion of the received light to the first sensor surface and passes a second portion of the received light through the beam splitter.
  • a reflective unit in the image capture unit, is positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor.
  • the first and second image capture sensors are different areas on an image capture sensor chip.
  • the first and second image capture sensors are two separate image capture sensor chips mounted on a common platform.
  • the first and second image capture sensors are two separate imaging areas on a single image capture sensor chip.
  • a distal end of an endoscope includes the first and second image capture sensors, a prism assembly including the beam splitter, and the reflective unit.
  • a stereoscopic endoscope includes a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies. The first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality. Each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different prism assembly, and a different reflective unit in the plurality.
  • the beam splitter is included in a prism assembly that also includes a surface positioned to direct the first portion of light received from the beam splitter onto the first sensor surface. This surface is positioned so that no other light hits the surface.
  • the reflective unit includes a reflective surface positioned to reflect the second portion of the received light onto the surface of the second image capture sensor.
  • the prism assembly and the reflective unit are included in a single integral structure.
  • the prism assembly includes a distal face through which the received light enters the prism assembly.
  • the image capture unit has a first optical path length from the distal face to the first sensor surface that is about equal to a second optical path length from the distal face to the second sensor surface.
  • the first and second optical path lengths have different lengths and the difference in length of the two optical path lengths is configured to provide a difference in focus between the images acquired by the first image capture sensor and the second image capture sensor.
  • the prism assembly includes a beam splitter configured to reflect the first portion of the light received by the prism assembly and configured to transmit the second portion of the received light.
  • the first portion of the received light is a first percentage of the received light
  • the second portion of the received light is a second percentage of the received light.
  • the beam splitter is configured so that the first and second percentages are about equal.
  • the beam splitter is configured so that the first and second percentages are not equal.
  • the beam splitter may be implemented in many ways, including but not limited to, thin metallic coatings, dielectric coatings, dichroic coatings, or a pattern of reflective tiles on an otherwise transparent interface.
  • the first and second image capture sensors can both be color image capture sensors or alternatively, one of the image sensors is a color image sensor and the other of the image captures sensors is a monochrome image capture sensor.
  • a first image from a first portion of light received from a common front end optical system is captured by the first image capture sensor.
  • a second image from a second portion of the light received from the common front end optical system is captured by the second image capture sensor.
  • the first and second image capture sensors are coplanar and the first and second images are registered spatially relative to each other upon being captured.
  • the configuration of the beam splitter is varied and the illumination source may be varied.
  • the light received from the lens assembly enters the prism assembly through a distal face.
  • the beam splitter is configured to reflect a first portion of the received light on a basis of a polarization state of the received light, and is configured to transmit a second portion of the received light on the basis of the polarization state of the received light.
  • a first optical path length from the distal face to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • a controller is coupled to the first and second image capture sensors.
  • the controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image increasing the saliency of a feature in the image based on polarization differences in the received light.
  • the beam splitter is configured to reflect a first percentage of the received light, and is configured to transmit a second percentage of the received light.
  • a first optical path length from a distal face of the prism assembly to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • the first and second image capture sensors are color image capture sensors.
  • the controller is coupled to the first and second image capture sensors. The controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor.
  • the controller generated image has enhanced spatial resolution.
  • the controller generated image has enhanced dynamic range.
  • the first percentage is about fifty percent of the received light and the second percentage is about fifty percent of the received light.
  • the beam splitter and reflective surface of the prism assembly are positioned to offset an image captured by the first image capture sensor from an image captured by the second image capture sensor with the first optical path length remaining about equal to the second optical path length.
  • the controller samples a first pixel in a first image captured by the first image capture sensor and samples a second pixel captured by the second image capture sensor and corresponding to the first pixel. Using information from the two sampled pixels, the controller generates a pixel in an image having increased color performance in comparison to the images captured by the first and second image capture sensors.
  • the controller may perform this process using groups of pixels instead of single pixels.
  • the first percentage is selected based on the dynamic range of the first image capture sensor, e.g., so that an image captured by the first image capture sensor is not clipped due to the dynamic range of the first image capture sensor.
  • the first percentage is about N% of the received light and the second percentage is about M% of the received light.
  • N and M are positive numbers.
  • One hundred percent minus N% is about equal to M%.
  • the controller samples a pixel in an image captured by the first image capture sensor and samples a corresponding pixel in an image captured by the second image capture sensor.
  • the controller uses the information from the sampled pixels to generate a pixel in an output image.
  • the output image has an increased dynamic range relative to an image captured by a single image capture sensor.
  • the controller may perform this process using groups of pixels instead of single pixels.
  • the light received by the beam splitter includes a plurality of color components.
  • the beam splitter is configured to reflect one color component of the plurality of color components and is configured to transmit other color components in the plurality of color components.
  • the first optical path length from the distal face of the prism assembly to the first sensor surface is about equal to the second optical path length from the distal face to the second sensor surface.
  • the first image capture sensor is a monochrome image capture sensor
  • the second image capture sensor is an image capture sensor having a color filter array for the other color components in the plurality of color components.
  • the controller has full spatial resolution in the one of the plurality of color components and has reduced spatial resolution in the other of the plurality of color components.
  • the controller generates an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor.
  • the beam splitter includes a plurality of notch filters.
  • a notch filter is a filter with a spectrally narrow band in which the filter is reflective, and a broader pass band, which may be on one side or both sides of the reflective band.
  • the plurality of notch filters reflects a first set of light components as a first portion of the received light and passes a second set of light components as a second portion of the received light.
  • the first optical path length from the distal face of the prism assembly to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • the system includes an illuminator that generates output light including a plurality of color components.
  • the controller is configured to receive a demosaiced image of a first image captured by the first image capture sensor, and configured to receive a demosaiced image of a second image captured by the second image capture sensor.
  • the controller generates an N-element color component vector for a pixel in an output image from a color component vector of a corresponding pixel in the first demosaiced image and a color component vector of a corresponding pixel in the demosaiced second image, where N is at least three.
  • the beam splitter reflects a first portion of the received light and transmits a second portion of the received light.
  • a first optical path length from a distal face of the prism assembly to the first sensor surface is smaller than a second optical path length from the distal face to the second sensor surface.
  • the first image capture sensor captures an image focused at a first object distance
  • the second image capture sensor captures an image focused at a second object distance.
  • the controller is coupled to the first and second image capture sensors to receive the first and second images. The controller automatically shifts an output image between the first image and the second image as the object to endoscope distance changes without physically moving the optics inside the endoscope.
  • the controller is configured to sample a region of pixels in the first image and to sample a corresponding region of pixels in the second image to generate pixels in an output image having increased apparent depth of field in comparison to the first and second images individually.
  • the controller combines a first image captured by the first image capture sensor with a second image captured by the second image capture sensor, and generates a third image that automatically stays in focus during physical motion of the endoscope relative to the tissue being viewed. This is accomplished by the controller processing regions of the first and second images and comparing their sharpness. The controller creates the third image from the pixels in the sharper of the two images at each of the regions. The third image is thus constructed from the sharpest portions of the two images.
  • the controller retrieves a first image captured by the first image capture sensor and a second image captured by the second image capture sensor, and generates a channel depth map based on the relative sharpness of pixel regions acquired from the first and second image capture sensors.
  • the depth map may be used by the system in various ways. One way is for the controller to generate a three-dimensional surface of a scene and then to project (by executing software) and texture map the first and second images back on the three-dimensional surface to generate a textured virtual image surface.
  • the controller generates a new virtual image for a virtual camera point from the channel depth map and the textured image surface. More than one virtual camera position and corresponding image may be created if desired.
  • real-time images of a scene are generated as the virtual camera position is swept back and forth from a left eye position to a right eye position, i.e., swept back and forth across an interocular separation.
  • an image from a virtual camera view point is generated, the image is displayed on a non-stereo display unit.
  • the view point is moved to the next virtual camera position, and an image from that view point is generated and displayed.
  • the displayed scene rocks back and forth over time and so gives depth cues to a person viewing the display without requiring a stereoscopic viewer.
  • an apparatus in yet another aspect, includes a first image capture sensor and a second image capture sensor.
  • the first capture image sensor has a first sensor surface, while the second image capture sensor has a second sensor surface.
  • the apparatus also includes a first lens assembly and a second lens assembly.
  • a reflective unit is positioned to receive light that passes through the first lens assembly and is positioned to receive light that passes through the second lens assembly.
  • the reflective unit reflects the received light from the first lens assembly unto the first sensor surface.
  • the reflective unit also reflects the received light from the second lens assembly unto the second sensor surface.
  • a first optical path length from the first lens assembly to the first sensor surface is about equal to a second optical path length from the second lens assembly to the second sensor surface.
  • the first and second image capture sensor surfaces of this apparatus are coplanar.
  • the first sensor surface is in a first plane.
  • the second sensor surface is in a second plane, and the first and second planes are substantially parallel and are separated by a known distance. In both cases the first optical path length and the second optical path length are about equal.
  • Figs. 1 A to ID are prior art examples of image capture sensor configurations in a distal end of a stereoscopic endoscope.
  • FIG. 2 is a block diagram of a minimally invasive surgical system that includes a plurality of image capture units in a distal end of a stereoscopic endoscope.
  • Fig. 3A is a block diagram of a distal end of a stereoscopic endoscope that includes a plurality of image capture units.
  • Fig. 3B is a block diagram of a distal end of a monoscopic endoscope that includes an image capture unit.
  • Fig. 4A is an illustration of a portion of an image capture unit that includes a lens assembly and a sensor assembly having coplanar image capture sensors.
  • Fig. 4B is an example of an implementation of the structure in Fig. 4A using a pentaprism as the prism assembly.
  • Fig. 5A is a schematic illustration of a distal end of a stereoscopic endoscope including an illumination channel, which provides unpolarized light from an illuminator, and left and right stereoscopic optical channels that each includes an image capture unit having a lens assembly and a sensor assembly.
  • FIG. 5B is a schematic illustration of a distal end of a stereoscopic endoscope including an illumination channel that provides unpolarized light from an illuminator, a polarizer that polarizes the illumination from the illumination channel, and left and right stereoscopic optical channels that each includes an image capture unit having a lens assembly and a sensor assembly.
  • Fig. 6A is a schematic illustration of a distal end of a stereoscopic endoscope, an illumination channel that provides light from an illuminator, and left and right stereoscopic optical channels that each includes an image capture unit having a lens assembly and a sensor assembly.
  • Each sensor assembly includes a beam splitter with a coated surface that reflects a first percentage of the received light and that passes a second percentage of the received light through the coated surface.
  • Fig. 6B is a schematic illustration of the offset for a block of pixels in the first image and a corresponding block of pixels in the second image.
  • Fig. 7A is a schematic illustration of a distal end of a stereoscopic endoscope including an illumination channel that provides light from one of a plurality of illuminators, and left and right stereoscopic optical channels that each includes an image capture unit having a lens assembly and a sensor assembly.
  • Figs. 7B to 7E are block illustrations of different illuminators that can be coupled to the stereoscopic endoscope of Fig. 7A as well as to any of the other endoscopes and devices described herein.
  • Fig. 7F is a graphical representation of a filter that includes a plurality of notch filters.
  • Fig. 7G is a graphical representation of a spectrum of light that is received by an image capture unit.
  • Fig. 7H is a graphical representation of another filter that includes a plurality of notch filters.
  • Fig. 8A is a schematic illustration of a distal end of a stereoscopic endoscope including an illumination channel that provides light from an illuminator, and left and right stereoscopic optical channels that each includes an image capture unit having a lens assembly and a sensor assembly. Each sensor assembly has two different optical path lengths.
  • Figs. 8B and 8C are illustrations of two images captured in a sensor assembly with different depths of field and different focus.
  • Fig. 8D is an illustration of one combination of the two images of Figs. 8B and 8C combined to form an in focus image with extended depth of field.
  • Fig. 8E shows the sharpness of the images acquired by the first and second image capture sensors versus object distance.
  • Fig. 8F is a process flow diagram of a method to generate a virtual image at a virtual camera view point from images captured by an image capture unit.
  • Fig. 9 is an illustration of a portion of an image capture unit that includes a lens assembly and a sensor assembly having image capture sensors that are not coplanar.
  • Fig. 10 is a schematic illustration of a distal end of a stereoscopic endoscope with a surface that directs light received from two lens assemblies to coplanar image sensors.
  • the first digit of a reference number indicates the figure in which the element with that reference number first appeared for single digit figure numbers.
  • the first two digits of a reference number indicates the figure in which the element with that reference number first appeared for double digit figure numbers.
  • electronic stereoscopic imaging includes the use of two imaging channels (i.e., one channel for left side images and another channel for right side images).
  • a stereoscopic optical path includes two channels (e.g., channels for left and right images) for transporting light from an object such as tissue to be imaged. The light transported in each channel represents a different view (stereoscopic left or right) of a scene in the surgical field.
  • Each one of the stereoscopic channels may include one, two, or more optical paths, and so light transported along a single stereoscopic channel can form one or more images.
  • one left side image may be captured from light traveling along a first optical path
  • a second left side image may be captured from light traveling along a second optical path.
  • an illumination channel includes a path providing illumination to tissue from an illumination source located away from an image capture unit (e.g., away from the distal end of an endoscope), or an illumination source located at or near the image capture unit (e.g., one or more light emitting diodes (LEDs) at or near the distal end of an endoscope).
  • an illumination source located away from an image capture unit (e.g., away from the distal end of an endoscope)
  • LEDs light emitting diodes
  • white light is visible white light that is made up of three (or more) visible color components, e.g., a red visible color component, a green visible color component, and a blue visible color component. If the visible color components are provided by an illuminator, the visible color components are referred to as visible color illumination components.
  • White light may also refer to a more continuous spectrum in the visible spectrum as one might see from a heated tungsten filament or xenon lamp, for example.
  • a visible image includes a visible color component.
  • a non-visible image is an image that does not include any of the visible color components.
  • a non-visible image is an image formed by light outside the range typically considered visible.
  • images captured as the result of fluorescence are referred to as acquired fluorescence images.
  • Fluorescence imaging modalities Fluorescence may result from natural tissue fluorescence, or the use of, for example, injectable dyes, fluorescent proteins, or fluorescent tagged antibodies. Fluorescence may result from, for example, excitation by laser or other energy source. In such configurations, it is understood that a notch filter is used to block the excitation wavelength that enters the endoscope.
  • Fluorescence images can provide vital in vivo patient information that is critical for surgery, such as pathology information (e.g., fluorescing tumors) or anatomic information (e.g., fluorescing tagged tendons).
  • the angle of incidence is the angle between a light ray incident on a surface and the line perpendicular to the surface at the point of incidence.
  • images are processed digitally and may be re-oriented or mirrored by changing the way in which the image is indexed. Re-orientation or mirroring may also be accomplished in the order in which the image sensor is read.
  • image capture units 225L, 225R are located at a distal end of a stereoscopic endoscope 202 in a minimally invasive surgical system 200, e.g., a da Vinci ® minimally invasive teleoperated surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.
  • a minimally invasive surgical system 200 e.g., a da Vinci ® minimally invasive teleoperated surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.
  • the distal direction is towards tissue 203 and the proximal direction is away from tissue 203.
  • One image capture unit 225L captures left side images for a stereoscopic image, sometimes referred to as left side stereoscopic images.
  • a second image capture unit 225R captures right side images for a stereoscopic image, sometimes referred to as right side stereoscopic images.
  • each image capture unit includes a lens assembly and a sensor assembly.
  • the lens assembly is sometimes referred to as a front end optical system.
  • the sensor assembly includes a pair of coplanar image capture sensors, in one aspect, a folded optical path that transmits light from the lens assembly to one of the coplanar image capture sensors, and another folded optical path that transmits light from the lens assembly to the other coplanar image capture sensor.
  • the same lens assembly in the image capture unit is used for both image capture sensors so that the image capture sensors are said to have a common front end optical structure.
  • the combination of the shared lens assembly and the coplanar configuration of the image capture sensors eliminates the need for calibration to compensate for lens artifacts.
  • a pair of images captured in a channel of endoscope 202 may also be temporally registered to each other.
  • image capture units 225L, 225R are used in a minimally invasive surgical system that includes multiple viewing modes: a normal mode, and one or more enhanced modes.
  • a person switches between the viewing modes by using display mode switch 252 that typically is in a user interface 262 presented on a surgeon's control console 250, sometimes referred to as surgeon's console 250.
  • the enhanced images are displayed on stereoscopic display 251.
  • the enhanced images may include (a) visible images with enhanced feature definition, in which a particular feature in the scene is emphasized to the operator of minimally invasive surgical system 200; (b) images having increased image apparent resolution; (c) images having increased dynamic range; (d) images displayed in a way based on a pixel color component vector having three or more color components; and (e) images having extended depth of field. [0075] Prior to considering image capture units 225L, 225R and the enhanced modes of operation in further detail, minimally invasive surgical system 200 is described.
  • System 200 is illustrative only and is not intended to limit the application of image capture units 225L, 225R to this specific system.
  • Image capture units 225L, 225R could be implemented in various other devices such as stereoscopic microscopes, monoscopic endoscopes, microscopes, and also could be used for replacement of existing endoscopic cameras.
  • Minimally invasive surgical system 200 for example, a da Vinci ® Surgical System, includes image capture units 225L, 225R.
  • a surgeon at surgeon's console 250 remotely manipulates endoscope 202 that is mounted on a robotic manipulator arm (not shown).
  • endoscope 202 that is mounted on a robotic manipulator arm (not shown).
  • Further information regarding minimally invasive surgical systems may be found for example in U.S. Patent Application No. 11/762,165 (filed Jun. 23, 2007; disclosing Minimally Invasive Surgical System), U.S. Patent No. US 6,837,883 B2 (filed Oct. 5, 2001; disclosing Arm Cart for Telerobotic Surgical System), and U.S. Patent
  • An illuminator 210 is coupled to stereoscopic endoscope 202.
  • Illuminator 210 includes at least a white light source and optionally may include one or more fluorescence excitation sources.
  • Illuminator 210 is used in conjunction with at least one illumination channel in stereoscopic endoscope 202 to illuminate tissue 203.
  • illuminator 210 may be replaced by an illumination source at the distal tip, or near the distal tip, of endoscope 202.
  • Such distal tip illumination may be provided by LEDs, for example, or other illumination sources.
  • illuminator 210 provides white light illumination that illuminates tissue 203 in white light.
  • illuminator 210 can also provide non- visible light that excites fluorescence and as well as a subset of the visible color components that make-up white light.
  • three (or more) visible color components make up white light, e.g., white light includes a first visible color component, a second visible color component, and a third visible color component.
  • Each of the three visible color components is a different visible color component, e.g., a red color component, a green color component, and a blue color component. Additional color components may also be used such as cyan to improve color fidelity of the system.
  • a fluorescence excitation source in illuminator 210 provides a fluorescence excitation illumination component that excites fluorescence in tissue 203.
  • a fluorescence excitation illumination component that excites fluorescence in tissue 203.
  • narrow band light from the fluorescence excitation source is used to excite tissue-specific near infrared emitting fluorophores so that fluorescence images of specific features within tissue 203 are acquired by image capture units 225L, 225R.
  • illuminator 210 Light from illuminator 210 is directed onto an illumination channel 226 that couples illuminator 210 to the illumination channel in endoscope 202.
  • the illumination channel in stereoscopic endoscope 202 directs the light to tissue 203.
  • an illumination source such as LEDs or other sources, is provided at, or near the distal tip on endoscope 202.
  • the illumination channels can be implemented with a fiber optic bundle, a single stiff or flexible rod, or an optical fiber.
  • Each one of image capture units 225R, 225L in endoscope 202 include, in one aspect, a single lens assembly for passing light received from tissue 203 to a sensor assembly.
  • Light from tissue 203 may include visible spectrum light components reflected from a white light illumination source and fluorescence light (visible or non-visible) that originates at tissue 203, for example as the result of receiving energy from a fluorescence excitation illumination source.
  • the reflected white light components are used to capture an image or images that a viewer would expect to see in the normal visible light spectrum.
  • Image capture unit 225L is coupled to stereoscopic display 251 in surgeon's console 250 via a left camera control unit (CCU) 230L.
  • Image capture unit 225R is coupled to stereoscopic display 251 in surgeon' s console 250 via a right camera control unit (CCU) 230R.
  • Camera control units 230L, 230R receive signals from a system process module 263 that controls gains, controls capturing images, controls transferring captures images to dual image enhancement
  • System process module 263 represents the various controllers including the vision system controllers in system 200.
  • Camera control units 230L, 230R may be separate units, or may be combined in a single dual controller unit.
  • Display mode select switch 252 provides a signal to a user interface 262 that in turn passes the selected display mode to system process module 263.
  • Various vision system controllers within system process module 263 configure illuminator 210 to produce the desired illumination, configure left and right camera control units 230L and 230R to acquire the desired images, and configure any other elements needed to process the acquired images so that the surgeon is presented the requested images in display 251.
  • Color correction modules 241L and 241R are in some embodiments each part of dual image enhancement modules 240L and 240R, respectively (described in more detail below). Color correction modules 241L and 240L transform the color of the acquired images to a new desired color balance as determined by system process module 263. As shown in Fig. 2, user interface 262, system process module 263, and image enhancement modules 240L, 240R are grouped as a central controller 260 for descriptive purposes.
  • Optional image processing module 264 receives video from central controller 260 and processes images from color correction modules 241L and 241R prior to display on stereoscopic display 251 in surgeons console 250.
  • Optional image processing module 264 is equivalent to image processing modules in prior art minimally invasive surgical systems and so is not considered in further detail.
  • Fig. 3A is a block diagram of a distal end of stereoscopic endoscope 302A that includes image capture units 325L and 325R and an illumination channel 305.
  • Each image capture unit 325R, 325L includes a lens assembly 301R, 301L and a sensor assembly 320R, 320L.
  • Sensor assembly 320R, 320L is positioned to receive light that passes through lens assembly 301R, 301L.
  • Each image capture unit 320R, 320L includes a prism assembly 330R, 330L, a reflective assembly 340R, 340L and coplanar image capture sensors (310R, 315R), (310L, 315L), in one aspect.
  • Stereoscopic endoscope 302A is an example of stereoscopic endoscopic 202.
  • each stereoscopic channel in a distal end of stereoscopic endoscope 302A has the same component configuration.
  • image capture unit 325L for the left stereoscopic channel
  • image capture unit 325R for the right stereoscopic channel
  • the distal direction is towards tissue 303 and the proximal direction is away from tissue 303.
  • illumination channels 305 in endoscope 302A illuminates tissue 303 in this example. While it not shown in Fig. 3A, one or more surgical instruments within the field of view of endoscope 302A may also be illuminated via light from illumination channel 305.
  • illumination channel 305 The use of an illumination channel in an endoscope is illustrative only and is not intended to be limiting in the various examples presented in this description. The illumination may be provided by an illumination source in the endoscope or by some other apparatus that is internal or external to the endoscope.
  • Lenses 304L and 304R in lens assembly 301L and 301R may include one of more optical components that direct the received light to sensor assembly 320L and sensor assembly 320R, respectively.
  • lens assembly 301L and 301R are folded to reduce the longitudinal length of image capture units 325L and 325R.
  • each of beam splitters 331L and 331R is implemented as a buried coated surface 331L, 331R. As explained more completely below, the coating or coatings on each of coated surface 331L, 331R are selected to provide a particular functionality. Coated surface 331L, 331R reflects a first portion of the light received from lens assembly 301L, 301R and transmits a second portion of the received light.
  • the coated surface distinguishes between differences in polarization of the light received from lens assembly 301L, 301R.
  • the coated surface includes notch filters that also reflect portions of the received light and transmit other portions of the received light.
  • beam splitter 331L directs a first portion of the received light onto a first image capture sensor 310L, e.g., onto a surface 311L of image capture sensor 310L, in image capture unit 325L and transmits a second portion of the received light through beam splitter 331L.
  • beam splitter 331R directs a first portion of the received light onto a first image capture sensor 310R, e.g., onto a surface 311R of image capture sensor 310R, in image capture unit 325R and transmits a second portion of the received light through beam splitter 331R.
  • lens 350L and 350R focus the received light to account for the optical path length to image capture sensors 315L and 315R.
  • Lenses 350L and 350R are optional.
  • the light from lenses 350L and 350R is received by reflective assemblies 340L and 340R, respectively.
  • Reflective unit 340L directs the received light onto a second image capture sensor 315L, e.g., directs the received light onto a surface 316L of image capture sensor 315L, in image capture unit 325L.
  • reflective unit 340R directs the received light onto a second image capture sensor 315R, e.g., directs the received light onto a surface 316R of image capture sensor 315R, in image capture unit 325R.
  • the light is directed onto a surface of an image capture sensor and so for brevity it is said that the light is directed onto the image capture sensor.
  • Each of reflective assemblies 340L and 340R includes a reflective surface 341L, 341R, e.g., a mirror surface, which reflects the received light.
  • apparatuses 340L and 340R are each implemented as a prism with one face having a reflective coating, or are each implemented using total internal reflection on the hypotenuse of the prism.
  • an angle ⁇ formed by the intersection of a plane including reflective surface 341R and a plane including surface 311R of image capture sensor 310R and surface 316R of image capture sensor 315R is a forty-five degree angle and so the prism is referred to as a forty-five degree prism.
  • Surface 341R of a forty-five degree prism exhibits total internal reflection when the medium proximal to surface 314R is air and so surface 341R is a reflective surface.
  • Image capture sensors 310L and 315L are coplanar, i.e., top sensor surfaces 311L and 316L are effectively in the same plane. Bottom surfaces of sensors 310L and 315L are on a plane defined by a first surface of platform 312. Similarly, image capture sensors 310R and 315R are coplanar, e.g., top surfaces 311R and 316R are effectively in the same plane. Bottom surfaces of sensors 310R and 315R are on a plane defined by a second surface of platform 312. Platform 312 may be composed of two planar parts, e.g., two ceramic parts bonded along axis 390. The first surface of platform 312 is opposite and removed from the second surface of platform 312.
  • a first semiconductor die 317R including two image capture
  • sensors 310R, 315R is mounted on a first ceramic platform.
  • a second semiconductor die 317L including two image capture sensors 310L, 315L is mounted on a second ceramic platform.
  • the two ceramic platforms are then bonded together to form platform 312. Wires to the two dies 317R, 317L are passed through a channel or channels in platform 312.
  • the use of two image capture sensors in a die is illustrative only and is not intended to be limiting. In some aspects, the two image sensors are in separate dies. (See Fig. 9.)
  • the two image capture sensors may be part of one large image capture sensor in the die, and pixels of the image capture sensor between image capture sensor 31 OR and image capture sensor 315R, for example, are ignored.
  • platform 312 may not be used and the two sets of image capture sensors are included in a single structure configured to provide the necessary connections to power, control, and video cables.
  • the use of the two coplanar image capture sensors in an image capture unit as shown in Fig. 3A is only illustrative and is not intended to be limiting.
  • more than two coplanar image capture sensors could be used, e.g., multiple beam splitters could be used in a line and the reflective unit placed at the proximal end of the line.
  • the coplanar configuration of the image capture sensors eliminates the need for calibration to compensate for lens artifacts and re-registration of different images captured by image captures sensors 310R/315R (first pair) and 310L/315L (second pair).
  • the spatial relation between the two image capture sensors within a given pair is constant, and since the image captures sensors within a given pair share a common lens assembly, i.e., a common front end optical structure, spatial registration of a pair of images captured by the two image capture sensors remains constant over time and during changing optical conditions, such as changing focus.
  • Fig. 3B is similar to Fig. 3A, except endoscope 302B is a monoscopic endoscope, and the illumination is provided by an illuminator external to and not connected to endoscope 302B.
  • the illuminator is mounted on endoscope 302B.
  • elements 301, 304, 320, 330, 331, 340, 341, 350 310, 311, 315, 316, and 312B of image capture unit 325 are the same as elements 301L, 304L, 320L, 330L, 331L,
  • An illumination source external to the endoscope may be used for the various described endoscope embodiments and aspects rather than, or in addition to, an illumination channel inside or mounted on the endoscope.
  • FIG. 4A is an illustration of a portion of an image capture unit 425A that includes a lens assembly 401A and a sensor assembly 420A.
  • Sensor assembly 420A includes a prism assembly 430A, a reflective unit 440A, and coplanar image capture sensors 41 OA, 415 A, in one aspect.
  • assembly 401 A includes a plurality of optical elements including an optical element that defines an optical stop 470A, sometimes referred to as stop 470A.
  • a stop is a plane in space having an aperture or a place in an optical path where the chief ray has a height of zero.
  • Light passing through stop 470A is received by a beam splitter 431 A in prism assembly 430A of image capture unit 425 A.
  • a stereoscopic apparatus would include two image capture units as illustrated in Fig. 4A.
  • the left and right stereoscopic channels with image capture units are symmetric, and so only a single channel and image capture unit is described to avoid duplicative description.
  • the other channel including the image capture unit is symmetric across a plane intersecting the longitudinal axis of the endoscope with the channel illustrated in Fig. 4A.
  • Beam splitter 431 A is positioned to receive light that passes through stop 470A. Beam splitter 431 A is configured to direct a first portion of the received light to a first image capture sensor 41 OA, and to pass a second portion of the received light through the beam splitter to reflective unit 440A.
  • beam splitter 431 A in prism assembly 430A is implemented as a coated first surface.
  • beam splitter 431 A is sometimes referred to as coated first surface 431 A. Coated first surface 431 A separates the received light into the two portions.
  • Coated first surface 431 A reflects the first portion of the received light to a second
  • Coated first surface 431 A transmits the second portion of the received light through coated first surface 431 A to reflective unit 440A.
  • the angle of incidence of the light to coated surface 431 A is less than forty-five degrees.
  • Second surface 432A is positioned so that no light other than the light reflected by coated first surface 431 A hits second surface 432A.
  • second surface 432A is a reflective surface that is implemented as one of a coated surface and a total internal reflection surface.
  • Reflective unit 440A includes a third surface 441 A that reflects the light received from beam splitter 431 A onto a second image capture sensor 415 A.
  • third surface 441 A is a reflective surface implemented, for example, as one of a coated surface and a total internal reflection surface.
  • Image capture sensors 41 OA and 415 A are coplanar.
  • a first optical path length from stop 470A to coated first surface 431 A to second surface 432A to image capture sensor 41 OA is about equal to a second optical path length from stop 470A through coated first surface 431 A to coated surface 441 A to image capture sensor 415 A.
  • the first and second optical path lengths are not equal.
  • the unequal optical path lengths can be implemented by adjusting the spatial relationship between a surface in reflective unit and the beam splitter in the prism assembly, as described more completely below.
  • image capture sensor 41 OA has a top sensor surface in a first plane and image capture sensor 415 A has a top sensor surface in a second plane, where the first and second planes are parallel and are separated by a known distance (See Fig. 9).
  • the optical path lengths to the two image capture sensors can be either about equal or unequal.
  • the top sensor surface of an image capture sensor is the surface of the image capture sensor that receives light from at least one optical component in the sensor assembly.
  • Optional optics, as illustrated by lenses 350 (Fig. 3) may be used in the second optical path to sensor 415A, for example.
  • optical path lengths are equal within the tolerances associated with manufacturing and mounting the various optical path elements.
  • definition of the optical path lengths as starting at stop 470A is illustrative and is not intended to be limiting.
  • the optical path lengths could also be variously defined, such as with respect to a distal face of prism assembly 430A through which the received light enters prism assembly 430A, with respect to a first element in the lens assembly, or with respect to coated first surface 431 A.
  • Fig. 4A the separation of prism assembly 430A, reflective unit 440A, and image capture sensors 41 OA, 415 A is not to scale.
  • prism assembly 430A and reflective unit 440A are mostly solid glass and there is a small gap between the glass structures and image capture
  • Fig. 4B is an example of an implementation of the structure in Fig. 4A using prisms.
  • Fig. 4B is an illustration of an image capture unit 425B in distal end of a stereoscopic endoscope.
  • Image capture unit 425B includes a shared lens assembly 40 IB and a sensor assembly 420B.
  • Sensor assembly 420B includes a prism assembly 430B, a reflective unit 440B, and coplanar image capture sensors 410B, 415B, in one aspect.
  • Sensor assembly 420B is positioned to receive light that passes through lens assembly 401B.
  • the other image capture unit (not shown) includes components equivalent to those shown in Fig. 4B.
  • the two image capture units are symmetric across a plane that intersects the longitudinal axis of the stereoscopic endoscope in a way equivalent to that shown in Fig. 3A.
  • Element 485 is optional and is illustrative of various features, either singly or in combination, that may be incorporated into the various lens assembly embodiments at various positions.
  • element 485 represents a window for notch filters to block fluorescence excitation wavelengths.
  • element 485 represents a focusing element.
  • element 485 represents a plano-plano element.
  • element 485 represents a structure that includes liquid crystal cells that function as a variable focus lens when a control voltage is applied to the liquid crystal cells.
  • the control voltage dynamically changes the refractive index profile of the material the light passes through.
  • the liquid crystal cells can adjust the focus to a desired focal distance from infinity to ten centimeters with good optical performance.
  • a liquid crystal cell having these performance characteristics is available as the Lens VectorTM AutoFocus element from Lens Vector, Inc. of Mountain View, CA, US. (LENS VECTOR is a trademark of LensVector, Inc. in the United States.)
  • element 485 represents one or more filters to preprocess the light that is received by the other elements in the lens assembly.
  • the filter or filters can be attached to the distal end surface of the endoscope, or the filter or filters may be inserted between aperture element 470B and prismatic structure 460.
  • prismatic structure 460 is an integral prismatic structure (two prisms or more are joined without any gaps between the prisms) that includes a prism assembly 430B with beam splitter 43 IB and a reflective unit 440B with a reflective surface 44 IB.
  • prismatic structure 460 is designed with a material with a particular refractive index and prism geometry to achieve the desired image separation onto the two image capture sensors. The design of prismatic structure 460 starts out as a multi-face prism and is then optimized to accommodate the sensor spacing, and the desired maximal outer tube diameter of the stereoscopic endoscope, which restricts the prism, geometry.
  • prism assembly 430B includes a pentaprism with at least two coated surfaces 43 IB and 432B.
  • surface 432B may not be coated and may use total internal reflection.
  • a beam splitter, i.e., coated first surface 43 IB separates the received light into the two portions.
  • Coated first surface 43 IB reflects the first portion of the received light to second surface 432B that in turn directs, i.e., reflects, the light onto first image capture sensor 410A.
  • Coated first surface 43 IB transmits the second portion of the received light through surface 431 A to reflective unit 440B.
  • Prism 450 is optional and is provided to facilitate handling and mounting of prismatic structure 460.
  • the pentaprism is a reflecting prism used to deviate a beam of light by ninety degrees.
  • the light beam reflects inside the pentaprism twice, which allows the transmission of the light beam through a right angle without producing an inverted or reversed image.
  • a pentaprism has a periphery made up of five sides.
  • a pentaprism is a constant deviation prism in that the pentaprism deviates the optical path through the same angle irrespective of the orientation of the pentaprism to the optical path.
  • a pentaprism may be bonded or otherwise jointed to or positioned with respect to other prisms or optics.
  • coated first surface 43 IB is a multi-layer buried coated surface.
  • the surface is coated with a multilayer coating, e.g., a multi-layer dichroic coating, metallic coating, or a rugate coating, which has the desired reflective and transmission properties, as discussed more completely below.
  • the angle of incidence of the light with surface 43 IB is less than forty-five degrees, in one aspect, although this is not a requirement.
  • Second surface 432B is positioned so that no light other than the light reflected by coated first surface 43 IB hits second surface 432B.
  • Distal end surface 433, sometimes referred to as a distal face, is the end of prismatic structure 460 and of prism assembly 430B through which the light received from lens assembly 40 IB enters.
  • first, second, and third surfaces 43 IB, 432B and 441B are positioned so that a first optical path length from distal end surface 433 to image capture sensor 410B is about equal to a second optical path length from distal end surface 433 to image capture sensor 415B.
  • surface 44 IB is positioned so that the two optical path lengths are not equal by a small amount.
  • Reflective unit 440B in one aspect, is a prism with a surface 44 IB having a reflective coating or surface 44 IB is used at such an angle to afford reflection via total internal reflection.
  • an angle formed by the intersection of a plane including reflective surface 44 IB and a plane including a top sensor surface of image capture sensor 415B is a forty-five degree angle and so the prism is referred to as a forty-five degree prism.
  • prismatic structure 460 is an integral structure formed by gluing three parts together using known techniques. As noted previously, the three parts are designed with materials, refractive index, and prism geometry to achieve the desired image separation on image capture sensors 410B and 415B. The use of three parts is illustrative only and is not intended to be limiting. For example, prismatic structure 460 could be an integral structure formed using two parts.
  • Image capture sensors 410B and 415B are coplanar and in one aspect are mounted on a platform on which the coplanar image sensors for the other lens assembly are also mounted (See Fig. 3A). In some aspects, the two image capture sensors are included in a single structure.
  • Stop 470B is placed or formed near or inside prismatic structure 460 to reduce the size of prismatic structure 460 and to reduce the angle of incidence of corner rays onto image capture sensors 410B, 415B. Stop 470B may be positioned either distally (as shown) or proximally of element 485
  • middle element 483 in lens assembly 401B may be a prism.
  • this prism is positioned to create a folded optical path to create an endoscope with, for example, a field of view oriented at 30 degrees to the endoscope's longitudinal axis. This fold happens in a plane perpendicular to the plane of image capture sensors 410B, 4I5B.
  • FIGs. 2, 3 A, 3B, 4A, and 4B various aspects of an image capture unit are shown and described.
  • the image capture unit includes a lens assembly that receives light from an object to be imaged.
  • the front end optical path through the lens assembly may be straight or may be folded to provide various angled fields of view for the endoscope.
  • Light from the lens assembly travels to a sensor assembly.
  • Various optional optical components may be placed in the lens assembly, or as design limits allow, such optical components may be inserted in an optical path or paths of the sensor assembly.
  • the sensor assembly includes two or more coplanar image capture sensors. These coplanar image capture sensors are arranged lengthwise along the endoscope. For example, with two coplanar image capture sensors, one of the image capture sensors is positioned relatively closer to the lens assembly (towards the endoscope's distal end), and one of the image capture sensors is positioned relatively farther away from the lens assembly (away from the endoscope's distal end). Thus, the image capture sensors are in a plane generally parallel to the endoscope's centerline longitudinal axis. In one aspect the image capture sensors are both formed as image capture regions (e.g., CMOS image capture regions; alternatively, CCD regions may be used) on the same semiconductor substrate.
  • image capture regions e.g., CMOS image capture regions; alternatively, CCD regions may be used
  • This single substrate therefore has a length at least about twice its width, and so it can be positioned within the endoscope for space use efficiency, since the sensor regions are arranged lengthwise in the endoscope.
  • the image capture sensors may be individually formed and positioned on a support platform or substrate that is arranged in the endoscope in a way similar to the single sensor substrate arrangement.
  • the optical paths in the sensor assembly are folded so that light that is incident on the image capture sensors is generally perpendicular to light received from the lens assembly.
  • the optical paths in the sensor assembly are arranged to be substantially equal length so that the lens assembly affects the images captured on the optical sensors in substantially the same way.
  • Various combinations of reflective surfaces either coated surfaces or total internal reflection surfaces, may be used to define the image capture assembly optical path geometries.
  • prisms are used to simplify reflective surface alignment requirements for the image capture sensors.
  • a single image capture unit is used in an endoscope, and so the endoscope has a monoscopic capability.
  • two image capture units are used in an endoscope to provide stereoscopic capability.
  • One image capture unit provides left stereoscopic image capability and the other image capture unit provides right stereoscopic image capability.
  • the image capture sensors in the two image capture units are oriented back-to-back so that the sensors are generally towards the center of the endoscope. This arrangement allows an efficient use of lateral space in an endoscope, since the front end objective paths for the two image capture units are positioned to provide good stereoscopic separation and the image capture sensor circuitry can be consolidated.
  • the back-to-back image capture sensors are supported by a single platform or substrate to provide further space savings and better optical alignment (e.g., the two image capture units can be aligned with one another prior to incorporation into the endoscope).
  • the back-to-back image capture sensors may be positioned generally along the endoscope's centerline longitudinal axis, although in some aspects they may be offset from the endoscope's centerline longitudinal axis due to, for example, other distal end endoscope features that may be used.
  • two image capture units may be positioned so that the coplanar image capture sensors are facing one another, or so that the coplanar image capture sensors for one image capture unit are coplanar with the coplanar image capture sensors of the other image capture sensor (i.e., all image capture sensors are coplanar, in some instances on the same substrate).
  • Various other orientations of the image capture sensor planes between the two image capture units may be used. It should also be understood that although the one or more image capture units are generally described as being at the endoscope's distal end, in some aspects the one or more image capture units may be at the endoscope's proximal end.
  • the compact dual imaging image capture unit optical geometry allows two relatively large image sensors to be placed in an endoscope for each desired optical channel, and the image capture unit feature that splits the incoming light into two or more beams allows many different visual display features to be presented to a person using an imaging system. For example, if two image capture units are used, two sets of precisely aligned stereoscopic images may be captured, with one stereoscopic image set having the characteristics captured from the first light beam and the other stereoscopic image set having the characteristics captured from the second light beam.
  • the imaging of the scene is enhanced by using prismatic structure 460 and coplanar image capture sensors 410B, 415B to gather additional information about the way the illumination interacts with tissue and other objects in the field of view.
  • the polarization state of the illumination light is perturbed differently by the tissue surface, subsurface structures in the tissue, and surgical instruments.
  • the illumination can be predominantly polarized or predominantly unpolarized.
  • predominantly polarized means polarized to the extent that is required for differentiation, e.g., ideally better than 1 ,000: 1 although lower contrast ratios may suffice.
  • predominantly unpolarized means unpolarized to the extent that is required.
  • FIG. 5A is a schematic illustration of a distal end of a stereoscopic endoscope 502A with an image capture unit 525L, 525R and an illumination channel 505, which provides unpolarized light from an illuminator. As indicated by arrow 535, the distal direction is towards tissue 503 and the proximal direction is away from tissue 503.
  • Each image capture unit 525R, 525L includes a lens assembly 501R, 501L and a sensor assembly 520R, 520L.
  • Sensor assembly 520R, 520L is positioned to receive light passed through lens assembly 501R, 501L.
  • Each sensor assembly 520R, 520L includes a prism assembly 530R, 530L, a reflective unit 540R, 540L, and coplanar image capture sensors (510R, 515R), (510L, 515L), in one aspect.
  • Stereoscopic endoscope 502A is sometimes referred to as endoscope 502A.
  • Some of the illumination from illumination channel 505 is reflected at the surface of tissue 503. While it is not shown in Fig. 5A, some of the illumination from illumination channel 505 also may be reflected by surgical instruments within the field of view of endoscope 502A. Some tissues impart a degree of polarization in the reflected light from those tissues.
  • endoscope 502A are described.
  • endoscope 502A are equivalent to those in the right channel due to the symmetry of endoscope 502A, and so in the following description, the left channel reference numeral is included within parentheses following the description of an element in the right channel. This indicates that the description is also applicable to the corresponding element in the left channel.
  • lens element 504R in lens assembly 50 IR (50 IR) to stop 570R (570L).
  • the elements in lens assembly 401B are an example of lens element 504R (504L).
  • an optional quarter wave plate 580R is inserted in the optical path between stop 570R (570L) and the distal end surface of image capture unit 525R (525L).
  • quarter wave plate 580R is not included in the optical path.
  • Beam splitter 53 IR (531L) is a polarization based beam splitter, in this aspect. Beam splitter 53 IR (531L) is configured to reflect a first portion of the received light on a basis of a polarization state of the received light and is configured to transmit a second portion of the received light on the basis of a polarization state of the received light. In one aspect, beam splitter 531R (531L) is a coated surface.
  • unpolarized light entering prism assembly 530R is evenly divided between first image capture sensor 510R (510L) and second image capture sensor 515R (515L) by beam splitter 531R (531L).
  • Light linearly polarized entering prism assembly 530R (530L) is reflected or transmitted by beam splitter 531R (531L) according to the relative orientation of the polarization of the light and the coating orientation of surface 531R (531L).
  • all the polarized light is directed to either first image capture sensor 51 OR (510L) or second image capture sensor 515R (515L).
  • the angle of incidence of the light to coated surface 531R (531L) is less than forty-five degrees.
  • Coated first surface 531R (531L) reflects the first portion of the received light to a second surface 532R (532L) that in turn directs, e.g., reflects, the light onto first image capture sensor 510R (510L).
  • Second surface 532R (532L) can be either a coated surface or a total internal reflection surface. Second surface 532R (532L) is positioned so that no light other than the light reflected by coated first surface 531R (531L) hits second surface 532R (532L).
  • Coated first surface 531R (531L) transmits the second portion of the received light through surface 531R (531L) to reflective unit 540R (540L). Specifically, the light transmitted through coated first surface 531R (531L) is received by a third surface 541R (541L) of reflective unit 540R (540L) that in turn directs, e.g., reflects, the light onto second image capture sensor 515R (515L).
  • Surface 541R (541L) can be either a coated surface or a total internal reflection surface.
  • prism assembly 530R (530L) and reflective unit 540R (540L) are included in a prismatic structure.
  • the prismatic structure in this aspect, is equivalent to prismatic structure 460 (Fig. 4B) with buried coated surface 43 IB having a buried multi-layer polarization selective layer, as described more completely below.
  • prismatic structure 460 is applicable to the prismatic structure used in the aspect of Fig. 5A.
  • Coated first surface 531R (531L) is, for example, a buried multi-layer polarization beam splitter. Such layers are known to those knowledgeable in the field, and are commonly used in polarizing beam splitter cubes. These dielectric film based coatings are usually buried and such coatings are used in one aspect.
  • a polarization based beam splitter may be constructed with materials from Moxtek® Incorporated of Orem Utah, a Polatechno Co. Ltd of Japan Company, as a PBF02 polarizing beam splitter. (MOXTEK is a registered U.S. trademark of Moxtek® Incorporated of Orem Utah.)
  • Image capture sensors 51 OR (510L) and 515R (515L) are coplanar.
  • a first optical path length from stop 570R (570L) to coated first surface 531R (531L) to second surface 532R (532L) to image capture sensor 51 OR (510L) is about equal to a second optical path length from stop 570R (570L) through coated first surface 531R (531L) to third surface 541R (541L) to image capture sensor 515R (515L).
  • the definition of the optical path lengths as starting at stop 570R (570L) is illustrative and is not intended to be limiting.
  • optical path lengths could also be variously defined, such as with respect to a distal face of prism assembly 530R (530L) through which the received light enters prism assembly 530R (530L), with respect to a first element in lens assembly 501R (501L), or with respect to coated first surface 531R (531L).
  • Coplanar image capture sensors 51 OR (510L) and 515R (515L) have a common optical path length through the front end optical structure in lens assembly 501R (501L) and about the same optical path length to each image capture sensor in sensor assembly 520R (520L).
  • One image capture sensor 510R (510L) captures an image comprised of light reflected by polarization beam splitter 531R (531L).
  • Other image capture sensor 515R (515L) captures an image comprised of light transmitted by polarization beam splitter 531R (531L)
  • Image capture unit 525R images two polarization states (either orthogonal linear states, or with quarter wave plate 580R (580L), left and right circularly polarized states).
  • the images acquired by image capture unit 525R (525L) provide information based on the preferential polarization imparted by the structures in the tissue itself.
  • Image capture unit 525R (525L) can only capture the relative strength of two orthogonal components of the polarization state of the received light (not the entire polarization nature of the light.) However, this is sufficient to provide useful information.
  • light is preferentially polarized when the light is specularly reflected from a surface.
  • the degree of the polarization depends on the illumination angle and the surface reflectance properties of the surface.
  • dual image enhancement module 240R processes the images captured by image capture sensors 51 OR (510L) and 515R (515L) (Fig. 5 A) to reduce the reflection from surgical instruments.
  • Information from pixels in the second image captured by image capture sensor 515R (515L) is used to modify pixels in the first image captured by image capture sensor 51 OR (510L), e.g., a percentage of the pixel value in the second image is subtracted from the corresponding pixel value in the first image, to further reduce the brightness of pixels representing the surgical instrument.
  • the percentage is empirically determined. This is possible because the two images have the same front end optical structure and are registered spatially relative to each other and are registered temporally relative to each other.
  • dual image enhancement module 240R processes the pixels in the second image captured by image capture sensor 515R (515L) from the received polarized light, and may, for example, false color the pixels.
  • the false colored image is then combined with the first image from image capture sensor 510R (515L) and the combined image is sent to stereoscopic display 251 for viewing by the surgeon.
  • "false color” is a good thing; it enables the particular pixels of interest to be visually salient relative to the background.
  • dual image enhancement module 240R processes the pixels in the second image captured by image capture sensor 515R (515L) from the received polarized light, and then determines corresponding pixels in the first image captured by image capture sensor 510R (510L).
  • Dual image enhancement module 240R changes the properties of the corresponding pixels in the first image, e.g., makes them more transparent and then combines the first and second images.
  • the combined image is sent to stereoscopic display 251 for viewing by the surgeon. Since tissue that did not polarize the incident light is now more transparent, this allows the surgeon to more clearly identify, the tissue that polarized the incident light and so provides additional cues for tissue differentiation.
  • the combined image generated by dual image enhancement module 240R increases the saliency of a feature in the image based on polarization differences in the received light, e.g., makes the feature more visible by making overlying tissue more transparent and by reducing other specular reflections.
  • the image capture sensors in an image capture unit are coplanar, the sets of image capture sensors in the two image capture units are in a fixed constant relationship to each other. Also, lens assemblies for each of the image capture units are equivalent. Thus, there is no need for active registration when processing the various images to assure that the images viewed by the surgeon are aligned and form proper stereoscopic images. This reduces the processing required of the images relative to images that are captured without these characteristics.
  • FIG. 5B is a schematic illustration of a distal end of a stereoscopic endoscope 502B with an image capture unit 525R, 525L and an illumination channel 505 that provides light from an illuminator. As indicated by arrow 535, the distal direction is towards tissue 503 and the proximal direction is away from tissue 503.
  • Each image capture unit 525R, 525L includes a lens assembly 501R, 501L and a sensor assembly 520R and 520L. Sensor assembly 520R and 520L is positioned to receive light passed through lens assembly 501R, 501L.
  • Each sensor assembly 520R, 520L includes a prism assembly 530R, 530L, a reflective unit 540R, 540L, and coplanar image capture sensors (510R, 515R), (510L, 515L), in one aspect.
  • stereoscopic endoscope 502A is sometimes referred to as endoscope 502A.
  • a polarizer 581 that polarizes the illumination is provided in the illumination optical path length.
  • polarizer 581 is shown at the distal end of stereoscopic endoscope 502A, but this illustrative only and is not intended to be limiting to this location.
  • Polarizer 581 represents a polarizer in the path of the illumination and can be placed at an appropriate place in that path.
  • illumination channel 505 can deliver light from a polarized source, or polarized light source can be located at location 581.
  • endoscope 502B are equivalent to those in the right channel, and so in the following description, the left channel reference numeral is included within parentheses following the description of an element in the right channel. This indicates that the description is also applicable to the corresponding element in the left channel.
  • lens element 504R 504L
  • stop 570R 570L
  • the elements in lens assembly 401B are an example of lens element 504R (504L).
  • optional quarter wave plate 580R 580L
  • Fig. 5A between stop 570R (570L) and the distal end surface of sensor assembly 520R (520L) has been removed.
  • the quarter wave plate is included in the lens assembly.
  • polarization beam splitter 531R (531L) in prism assembly 530R (530L) is configured to reflect a first portion of the received light on a basis of a polarization state of the received light and is configured to transmit a second portion of the received light on the basis of a polarization state of the received light.
  • polarization beam splitter 531R is configured to direct one polarization state onto first image capture sensor 510R (510L) and is configured to transmit, e.g., pass, the remaining light to reflective unit 540R (540L) that in turn directs the transmitted light onto image capture sensor 515R (515L).
  • prism assembly 530R (530L), polarization beam splitter 531R (531L), reflective unit 540R (540L) and image capture sensors 510R (510L) and 515R (515L) is not repeated and the above description with respect to Fig. 5 A is incorporated herein by reference.
  • Some of the polarized illumination from polarizer 581 reflects from the surface of tissue 503.
  • the light reflected on the surface has approximately the same polarization as the polarized illumination, e.g., the top surface reflection retains a greater portion of the original polarization state of the light hitting tissue 503.
  • the light which has a polarization different from the polarization of the illumination e.g., light that is de-polarized, must have interacted with the sub-surface of tissue 503.
  • the illumination is linearly polarized
  • the light is progressively depolarized as it enters and reflects back from subsurface features 503-1 to 503-4 of tissue 503.
  • the reflected light from the subsurface features 503-1 to 503-4 is essentially depolarized.
  • image capture sensor 515R captures an image from light primarily reflected from the surface of tissue 503 and approximately fifty percent of the light from subsurface features 503-1 to 503-4.
  • Image capture sensor 51 OR (510L) captures an image from light that was primarily reflected from subsurface features 503-1 to 503-4 and does not capture a significant portion of the light from surface 503.
  • Sensor assemblies 520R (520L) can only capture the relative strength of two orthogonal components of the polarization state of the received light (not the entire polarization nature of the light.) However, this is sufficient to provide useful information.
  • dual image enhancement module 240R processes the pixels in the first image captured by image capture sensor 51 OR (510L) from the received light, and then determines corresponding pixels in the second image captured by image capture sensor 515R (515L).
  • Dual image enhancement module 240R can, under the control of input from user interface 262, adjust what is shown on stereoscopic display 251. For example to see just surface 503, image enhancement module 240R (240L) subtracts the image captured by image capture sensor 51 OR (510L) from the image captured by image capture sensor 515R (515L).
  • image enhancement module 240R scales the image captured by image capture sensor 51 OR (510L) to achieve similar brightness to the image being displayed on stereoscopic display 251.
  • the resulting image is sent to stereoscopic display 251 for viewing by the surgeon.
  • the enhanced image generated by dual image enhancement module 240R increases the saliency of a feature in the image based on polarization differences in the received light.
  • a surgeon has the ability to preferentially see "through” the top surface of tissue 503 as the reflection from that top layer can be suppressed, as previously described, by making the top layer more transparent and by reducing specular reflections. Additionally, just the top surface of tissue 503 may be seen as described above.
  • tissue features this technique may accentuate may include endometriosis. This technique also may enhance the ability to differentiate nerves which have polarization signatures.
  • the combined image generated by dual image enhancement module 240R (240L) increases the saliency of a feature in the image based on polarization differences in the received light.
  • Image capture units 625R, 625L eliminate the problems of the prior art by providing both higher apparent resolution and higher dynamic range as described more completely below.
  • FIG. 6A is a schematic illustration of a distal end of a stereoscopic endoscope 602 with an image capture unit 625L, 625R and an illumination channel 505 that provides light from an illuminator. As indicated by arrow 635, the distal direction is towards tissue 603 and the proximal direction is away from tissue 603.
  • Each image capture unit 625R, 625L includes a lens assembly 601R, 601L and a sensor assembly 620R and 620L.
  • Sensor assembly 620R, 620L is positioned to receive light that passes through lens assembly 601R, 601L.
  • Each sensor assembly 620R, 620L includes a prism
  • Stereoscopic endoscope 602 is sometimes referred to as endoscope 602.
  • endoscope 602 are described.
  • the optical paths through the left channel of stereoscopic endoscope 602 are equivalent those in the right channel due to the symmetry of endoscope 602, and so in the following description, the left channel reference numeral is included within parentheses following the description of an element in the right channel. This indicates that the description is also applicable to the corresponding element in the left channel.
  • elements with the same reference numeral as elements in Figs. 5A and 5B are the same or equivalent elements to those previously described with respect of Figs. 5 A and 5B. To avoid repetition, elements with the same reference numeral are not described again in detail with respect to Fig. 6A.
  • the light from tissue 503 passes through lens element 504R (504L) and stop 570R (570L) in lens assembly 501R (501L).
  • the elements in lens assembly 40 IB are an example of lens element 504R (504L).
  • the light received by sensor assembly 620L (620R) enters prism assembly 630R.
  • Prism assembly 630R (630L) includes a beam splitter 631R (631L), e.g., a coated first surface, that reflects a first percentage of the light received by prism assembly 630R (630L) and that passes a second percentage of the received light through the coated first surface, e.g., through beam splitter 631R (631L) to reflective unit 540R (540L).
  • the light reflected by coated first surface 631R (631L) is received by a second surface 632R (632L) that in turn directs, e.g., reflects, the light onto first image capture sensor 610R (610L).
  • Second surface 632R (632L) is, for example, one of a coated surface, and a total internal reflection surface.
  • the light transmitted through coated first surface 631R (631L) is received by a third surface 541R (541L) of reflective unit 540R (540L) that in turn directs, e.g., reflects, the light onto second image capture sensor 615R (615L).
  • Second surface 632R (632L) is positioned so that no light other than the light reflected by coated first surface 631R (631L) hits second surface 632R (632L).
  • the angle of incidence of the light to coated first surface 631R (631L) is less than forty-five degrees.
  • prism assembly 630R (630L) and reflective unit 540R (540L) are included in a prismatic structure with a pentaprism that includes prism assembly 630R (630L).
  • the prismatic structure in this aspect is equivalent to prismatic structure 460 (Fig. 4B) with buried coated
  • prismatic structure 460 is applicable to the prismatic structure used in the aspect of Fig. 6A.
  • prism assembly 630R includes a beam splitter implemented as a coated first surface 531R (531L) that separates the received light into the two portions— (i) a first percentage of the received light sometimes referred to as a first portion, and (ii) a second percentage of the received light, sometimes referred to as a second portion.
  • Image capture sensors 61 OR (610L) and 615R (615L) are coplanar.
  • a first optical path length from stop 570R (570L) to coated first surface 531R (531L) to second surface 632R (632L) to image capture sensor 61 OR (610L) is about equal to a second optical path length from stop 570R (570L) through coated first surface 631R (631L) to a third surface 541R (541L) to image capture sensor 615R (615L).
  • the definition of the optical path lengths as starting at stop 570R (570L) is illustrative and is not intended to be limiting.
  • optical path lengths could also be variously defined, such as with respect to a distal face of prism assembly 630R (630L) through which the received light enters prism assembly 630R (630L), with respect to a first element in lens assembly 601R (601L), or with respect to coated first surface 631R (631L).
  • coplanar image capture sensors 61 OR (610L) and 615R (615L) have a common optical path length through the front end optical structure in lens assembly 601R (601L) and about the same optical path length to each image capture sensor in sensor assembly 620R (620L).
  • One image capture sensor 61 OR (610L) captures an image from the first portion of the light received by sensor assembly 620R (620L).
  • Other image capture sensor 615R (615L) captures an image from the second portion of the light received by sensor assembly 620R (620L).
  • each of image capture sensors 61 OR (610L) and 615R (615L) is a color sensor with a color filter array.
  • the color filter array is removed from one of the color sensors and the sensor functions as a monochrome sensor.
  • the color filter array of the other color sensor is modified for the number of color components received by that sensor.
  • coated first surface 631R (631L) of prism assembly 630R (630L) is configured to reflect and is configured to transmit about equal portions of the light received by prism assembly 630R (630L) from lens assembly 601R (601L), i.e., the first and second percentages are about equal.
  • the beam splitter 631R (631L) reflects and transmits about equal portion of light, the beam splitter is referred to as a balanced beam splitter.
  • Each of image capture sensors 610R (610L) and 615R (615L) is a color sensor with a color filter array in this aspect.
  • the color filter array is a Bayer color filter array. Thus, the two Bayer pattern image capture sensors are looking through the same optics at the same scene.
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 610R (610L) and 615R (615L) have a common front end optical structure and about the same optical path length to each sensor.
  • each point in space in the scene is represented by two pixels— one pixel in the color image captured by image capture sensor 61 OR (610L) and one pixel in the color image captured by image capture sensor 615R (615L).
  • each output pixel sent to stereoscopic display 251 from dual image enhancement module 240R (240L) in central controller 260 is based on sampling of two pixels, one from each of the images captured by Bayer pattern image capture sensors 610R (610L) and 615R (615L).
  • the sampling of two input pixels by dual image enhancement module 240R allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260.
  • the apparent resolution in the image viewed on stereoscopic display 251 is greater than the resolution in an image viewed on stereoscopic display 251 based on an image captured by a single image capture sensor.
  • the resolution of an image sent to stereoscopic display 251 can be higher than that of the image from a single image capture sensor and so is said to have greater apparent resolution.
  • coated first surface 631R (631L) of prism assembly 630R (630L) is still configured to reflect and transmit about equal portions of the light received by prism assembly 630R, i.e., the first and second percentages are about equal.
  • Each of image capture sensors 610R (610L) and 615R (615L) is a color sensor with a Bayer color filter array.
  • coplanar image capture sensors 61 OR (610L) and 615R (615L) have a common front end optical structure and about the same optical path length to each sensor.
  • surfaces 631R (631L) and 632R (632L) of prism assembly 630R (630L) are tilted slightly, without changing the total optical path length, so that the image captured by image capture sensor 61 OR (610L) is offset from the image captured by image capture sensor 615R (615L) by one-half pixel.
  • Fig. 6B is a schematic illustration of the offset for a block of pixels 691 in the first image and a corresponding block of pixels 695 in the second image.
  • Each pixel in block of pixels 691 is represented by a square having a dashed line perimeter and the color of the pixel is given by the first letter of the color followed by the number 2 at the center of the square.
  • Each pixel in block of pixels 695 is represented by a square having a solid line perimeter and the color of the pixel is given by the first letter of the color followed by the number 1 at the center of the square.
  • the Bayer color filter array is a red-green- blue-green (RGBG) array.
  • dual image enhancement module 240R (240L) in central controller 260 interpolates the pixels in the two images to create a color image with higher spatial resolution for the image sent to stereoscopic display 251.
  • the interpolation is equivalent to that done in a three CCD color camera, except unlike the three CCD color camera, two color images are used.
  • the larger number of pixels available for the sampling enhances the apparent resolution and provides improved signal-to- noise performance.
  • coated first surface 631R (631L) of prism assembly 630R (630L) is configured to reflect a first color component in the light received by prism assembly 630R from lens assembly 601R (601L) and is configured to transmit other color components in the received light.
  • image capture sensors 610R (610L) and 615R (615L) are not each a color sensor with a Bayer color filter array.
  • image capture sensor 610R is a monochrome sensor, e.g., a color sensor with the color filter array removed.
  • the light received by sensor assembly 620R (620L) is taken as having a plurality of visible color components— a first visible color component, a second visible color component, and a third visible component.
  • Image capture sensor 615R (615L) is a color sensor with a color filter array for two of the three visible color components, e.g., the number of visible color components in the plurality of visible color components minus one.
  • coplanar image capture sensors 61 OR (610L) and 615R (615L) have a common front end optical structure and about the same optical path length to each sensor.
  • prism assembly 630R (630L) and reflective unit 540R (540L) are arranged so that the images captured by both image capture sensors 61 OR (610L) and 615R (615L) are the same scene, but the first image is monochrome and the second image is multi-color.
  • the monochrome image represents the green color component image
  • the multi-color image is a red color component and a blue color component image.
  • color filter array of image capture sensor 615R (615L) is a red and blue checkerboard pattern.
  • the ratio of green pixels to red and blue pixels is the same as in a Bayer color filter array, but there are twice as many pixels of each color compared to a prior art single color sensor.
  • the resolution of the green pixels is generated using full spatial sampling by dual image enhancement module 240R (240L) in central controller 260, while the resolution of the red and blue pixels is generated using reduced spatial sampling, relative to the green pixels, by dual image enhancement module 240R (240L).
  • the spatial sampling for the red and blue pixels in this aspect is the same as the spatial sampling for the green pixels in the prior art single color sensor. This enhanced spatial sampling also provides enhanced apparent resolution and improved signal to noise ratio.
  • a scene has a brightness range from 0 to 400. If this scene is imaged to a sensor having a dynamic range from 0 to 100, values over 100 are clipped. For example if the brightness value for a pixel should be 400, the brightness value captured for that pixel is 100. Brightness information is lost for parts of the scene having a brightness larger than 100, because all brightness values greater than 100 are clipped and set to 100. If the gain for the image capture sensor is adjusted to 0.25, the high brightness values are not clipped, but all scene brightness values lower than four are mapped to zero. For this case, the high brightness values are not clipped, but information in the darker parts of the scene is lost.
  • Image capture unit 625R (625L) provides a solution to this problem by preserving both the high brightness information and more of the low brightness information relative to the prior art systems.
  • a brightness range of 0 to 400 is illustrative only. In an actual surgical scene, the brightness variation may be many orders of magnitude. Thus, the dynamic range of the surgical scene is greater than the dynamic range of the image capture sensor. Thus, a single image capture sensor cannot capture the dynamic range of the surgical scene.
  • Coated first surface 631R (631L), e.g., the beam splitter, of prism assembly 630R (630L) in sensor assembly 620R (620L) is configured to reflect and is configured to transmit different portions of the light received by prism assembly 630R (630L), i.e., the first and second percentages defined above are different.
  • Beam splitter 631R (631L) is a dynamic range adjusted beam splitter in this aspect.
  • a dynamic range adjusted beam splitter reflects M of the received light and passes N of the received light, where M is different from N .
  • M and N are positive numbers.
  • M plus N is equal to about one hundred percent. The equality may not be exact due to light losses and due to tolerances of the various parts of sensor assembly 620R (620L).
  • each of image capture sensors 610R (610L) and 615R (615L) is a color sensor with a color filter array.
  • the color filter array is a Bayer color filter array in one aspect.
  • the two Bayer pattern image capture sensors 610R (610L) and 615R (615L) are looking through the same optics at the same scene.
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 61 OR (610L) and 615R (615L) have a common front end optical structure and about the same optical path length to each sensor.
  • the selection of properties of the coating for first surface 631R (631L) considers the brightness of the image captured and/or the brightness of a portion of the image captured, the possible dynamic ranges of image capture sensors 610R (610L) and 615R (615L), and the capabilities of the imaging pipeline.
  • the imaging pipeline is the portion of central control system 260 (Fig. 2) that processes the captured images and generates output images for stereoscopic display 251.
  • the imaging pipeline may be part of CCU 230R (230L) or part of dual image enhancement module 240R (240L) depending on the implementation chosen.
  • a maximum brightness Bmax of the region is empirically determined.
  • the region of importance can be either the complete scene or a portion of the complete scene.
  • the dynamic range of image capture sensor 61 OR (610L) is then determined, e.g., 0 to Slmax.
  • Fraction M of the received light that is reflected by the coating on surface 631R (631L) is selected as:
  • the percentage of the received light M% that is reflected by the coating on first surface 631R (631L) is selected so that dynamic range of light incident on first image capture sensor 61 OR (610L) is not clipped. Thus, the brightness of pixels in high brightness regions of the image captured by first image capture sensor 61 OR (610L) are not clipped in this aspect.
  • the percentage of the received light transmitted by the coating on surface 631R (631L) is N% , where N% is about equal to one hundred minus M% .
  • the gain of second image capture sensor 615R (615L) is adjusted so the dynamic range of sensor 615R (615L) is from zero to about N% times maximum brightness Bmax.
  • Image capture sensor 61 OR (610L) has a dynamic range of 0 to 100.
  • the coating on first surface 631R (631L) is selected to reflect about 25% of the received light and to transmit about 75% of the received light.
  • the gain for image capture sensor 615R (615L) is adjusted to have a dynamic gain of 0 to 300.
  • the two images captured by sensors 610R (610L) and 615R (615L) are acquired essentially simultaneously.
  • the light received by sensor assembly 620R (620L) is split into two images with a brightness range on sensor 610R (610L) being 0 to 100 (1/4 *400) and a brightness range on sensor 615R (615L) being 0 to 300 (3/4 * 400). Pixels in the high brightness regions of the scene are not saturated in the image captured by sensor 615R (615L).
  • the image captured by sensor 61 OR (610L) preserves and accurately images the darker portions of the scene. Note that in this example scene regions with a brightness of one may be lost if the image capture sensors do not save values less than one. However, this is only one part out of four hundred so is unlikely to be noticed by the surgeon viewing the image on stereoscopic display 251.
  • each point in space in the scene is represented by two pixels— one pixel in the color image captured by image capture sensor 61 OR (610L) and one pixel in the color image captured by image capture sensor 615R (615L).
  • the two pixels have different brightness values that are a linear scale of each other if the brightness value on neither pixel has been clipped.
  • Each output pixel to stereoscopic display 251 from dual image enhancement module 240R (240L) in central controller 260 is based on sampling of two pixels, one from each of the images captured by Bayer pattern image capture sensors 610R (610L) and 615R (615L).
  • the sampling of input two pixels by dual image enhancement module 240R (240L) allows imaging of smaller features than is possible if only an image from a single image capture sensor is processed by central controller 260.
  • a tone mapping process in dual image enhancement module 240R (240L) maps regions of the acquired images to preserve contrast and to match the dynamic output range of stereoscopic display 251.
  • the tone mapping maps the brightness from the captured images to the displayable range while preserving the image contrast and color appearance important to best represent the original scene content.
  • the tone mapping is similar to prior art techniques except that it is enhanced by sampling two pixel values for the same point in space and time instead of one pixel value sequentially.
  • tone mapping For pixels that are not saturated, the brightness value in the image captured by sensor 615R (615L) is three times the brightness value in the image captured by sensor 61 OR (610L) for the same point in space.
  • the tone mapping process utilizes this relationship in determining the brightness for a pixel in an image output to stereoscopic display 251. Based on information on the surrounding pixels, tone mapping works on individual groups/regions of pixels to maintain the contrast and thus reduces the noise level and the dynamic range.
  • the prior art brightness range 0 to 100 has been extended to a brightness range of 1 to 400 for a gain in overall dynamic range performance. Combining the two images and taking advantage of the double sampling further enables reductions in noise in the middle brightness regions along with the increased dynamic range overall.
  • the image capture sensors in a stereoscopic channel are coplanar.
  • a first of the image capture sensors could have a top sensor surface in a first plane
  • a second of the image capture sensors could have a top sensor surface in a second plane.
  • the first and second planes are parallel to each other and are separated by a known distance.
  • the spacing between the beam splitter and the reflective surface in the reflective unit is adjusted to compensate for the known distance so that the first optical path length to the first image capture sensor is about equal to the second optical path length to the second image capture sensor.
  • RGB red, green, and blue
  • Most cameras use a Bayer color filter array image capture sensor with a RGB based color filter array.
  • the three color primary model is also used in most liquid crystal displays (LCDs) and plasma displays.
  • the three color primary model actually restricts the range of hues, which can be captured by a camera and presented in a display. Using additional color primaries enables the capture and display of more perceptually accurate images.
  • Sharp Electronics Corporation sells light emitting diode (LED) backlit LCD televisions with quad pixel technology. Sharp adds yellow to the conventional red, green and blue color filter array, enabling richer colors to be displayed.
  • Fig. 7A is a schematic illustration of a distal end of a stereoscopic endoscope 702 with image capture units 725L, 725R and an illumination channel 705 that provides light from, for example, one of illuminator 710B (Fig. 7B), illuminator 7 IOC (Fig. 7C), illuminator 710D (Fig. 7D) and
  • Each image capture unit 725R, 725L includes a lens assembly 701R, 701L and a sensor assembly 720R, 720L. Sensor assembly 720R, 720L is positioned to receive light that passes through lens assembly 701R, 701L.
  • Each sensor assembly 720R, 720L includes a prism assembly 730R, 730L, a reflective unit 740R, 740L, and coplanar image capture sensors (710R, 715R), (710L, 715L), in one aspect.
  • Stereoscopic endoscope 702 is sometimes referred to as endoscope 702.
  • endoscope 702 are described.
  • the optical paths through the left channel of stereoscopic endoscope 702 are equivalent those in the right channel due to the symmetry of endoscope 702, and so in the following description, the left channel reference numeral is included within parentheses following the description of an element in the right channel. This indicates that the description is also applicable to the corresponding element in the left channel.
  • elements with the same reference numeral as elements in Figs. 5A and 5B are the same or equivalent elements to those previously described with respect of Figs. 5 A and 5B. To avoid repetition, elements with the same reference numeral are not described again in detail with respect to Fig. 7A.
  • FIG. 7A The basic structure of image capture units 725L, 725R in each of the different aspects described below is presented in Fig. 7A.
  • the spatial relationships and alignments between prism assembly 730R (730L), reflective unit 540R (540L), image capture sensors 71 OR (710L), and image capture sensors 715R (715L) are the same for each of the different aspects considered with respect to Fig. 7 A.
  • the coatings on coated first surface 731R (731L) of prism assembly 730R (730L), the type of image capture sensors, and the filters used on the image capture sensors may change in different as aspects, as described more completely below.
  • the reflected light from tissue 703 passes through lens element 704R (704L) and stop 570R (570L) in lens assembly 701R (701L).
  • the elements in lens assembly 40 IB are an example of lens element 704R (704L).
  • the light that passes through lens assembly 701R (701L) is received by prism assembly 730R (730L) and enter beam splitter 731R (731L).
  • Beam splitter 731R (731L) is implemented, in this aspect, as a coated first surface that reflects a first portion of the received light, and passes a second portion of the received light to reflective unit 540R (540L).
  • the light reflected by coated first surface 731R (731L) is received by a second surface 732R (732L) in prism assembly 730R (730L) that in turn directs, e.g., reflects, the light onto first image capture sensor 71 OR (710L).
  • Surface 732R (732L) can be either a coated surface or a total internal reflection surface.
  • the light transmitted through coated first surface 731R (731L) is received by a third surface 541R (541L) of reflective unit 540R (540L) that in turn directs, e.g., reflects, the light onto second image capture sensor 715R (715L).
  • Second surface 732R (732L) is positioned so that no light other than the light reflected by coated first surface 731R (731L) hits second surface 732R (732L).
  • the angle of incidence of the light to coated first surface 731R (731L) is less than forty-five degrees.
  • prism assembly 730R (730L) and reflective unit 740R (740L) are included in a prismatic structure with a pentaprism that includes prism assembly 730R (730L).
  • the prismatic structure in this aspect is equivalent to prismatic structure 460 (Fig. 4B) with buried coated
  • prismatic structure 460 is applicable to the prismatic structure used in this aspect.
  • the first portion reflected by coated first surface 731R (731L) includes first selected wavelengths of a plurality of color components in the received light
  • the second portion transmitted by coated first surface 731R (731L) includes second selected wavelengths of the plurality of color components in the received light.
  • the first selected wavelengths of the plurality of color components in the received light are different from the second selected wavelengths of the plurality of color components in the received light.
  • coated first surface 731R (731L) comprises a plurality of notch filters. The notch filters separate the received light into the first portion and the second portion.
  • Image capture sensors 71 OR (710L) and 715R (715L) are coplanar.
  • a first optical path length from stop 570R (570L) to coated first surface 731R (731L) to second surface 732R (732L) to image capture sensor 71 OR (710L) is about equal to a second optical path length from stop 570R (570L) through coated first surface 731R (731L) to a third surface 741R (741L) to image capture sensor 715R (715L).
  • the definition of the optical path lengths as starting at stop 570R (570L) is illustrative and is not intended to be limiting.
  • optical path lengths could also be variously defined, such as with respect to a distal face of prism assembly 730R (730L) through which the received light enters prism assembly 630R (630L), with respect to a first element in lens assembly 701R (701L), or with respect to coated first surface 731R (731L).
  • coplanar image capture sensors 71 OR (710L) and 715R (715L) have a common front end optical structure and about the same optical path length to each sensor.
  • a first image capture sensor 71 OR (710L) captures an image from the first portion of the light received by sensor
  • each of image capture sensors 71 OR (710L) and 715R (715L) is a color sensor with a standard color filter array.
  • the color filter array on one of the color sensors and the sensor function is selected based on the light that is received by endoscope 702.
  • Controller 260 processes the first and second images captured by image capture sensors 71 OR (710L) and 715R (715L), respectively, and generates an output image for stereoscopic display 251. Controller 260 generates an N color component vector for each pixel in the output image, where N ranges from three to six.
  • the color component vector for a pixel in the output image is generated from a color component vector of a corresponding pixel (or some combination of pixels) in the first image and a color component vector of a corresponding pixel in the second image.
  • each pixel in the output image represents a point in space in the scene captured by the image capture sensors.
  • the pixels in the captured images corresponding to the pixel in the output image are the pixels representing the same point in space.
  • a six color component illuminator 710B (Fig. 7B) is coupled to illumination channel 705 (Fig. 7A).
  • Illuminator 710B includes six laser illumination sources 710B 1 to 710B 6 that generate red color component Rl, red color component R2, green color component Gl, green color component G2, blue color component Bl, and blue color component B2, respectively.
  • the wavelengths of red color components Rl and R2, green color components Gl and G2, and blue color components Bl and B2 are typically 2 to 5 nanometer (nm) wide instead of about 60 nm per color component from a conventional light emitting diode based illuminator.
  • coated first surface of 731R (731L) of prism assembly 730R (730L) includes a plurality of notch filters.
  • a first notch filter reflects red color component Rl in the received light and transmits red color component R2 in the received light.
  • a second notch filter reflects green color component Gl in the received light and transmits green color component G2 in the received light.
  • a third notch filter reflects blue color component Bl in the received light and transmits blue color component B2 in the received light.
  • the coating design would implement the notches. See for example, the "Stopline” and "Quad-Notch" product lines of Semrock Products, Rochester, New York.
  • coated first surface 731R (731L) reflects red color component Rl in the received light, green color component Gl in the received light, and blue color component Bl in the received light to second surface 732R (732L).
  • Second surface of 732R (732L) reflects red color component Rl, green color component Gl, and blue color component Bl received from coated first surface 731R (731L) onto image capture sensor 71 OR (710L).
  • Coated first surface of 731 R (731L) transmits red color component R2 in the received light, green color component G2 in the received light, and blue color component B2 in the received light to third surface 541R (541L) in reflective unit 540R (540L).
  • Third surface of 541R (541L) reflects red color component R2, green color component G2, and blue color component B2 received from coated first surface 731R (731L) onto image capture sensor 715R (715L).
  • Each of image capture sensors 71 OR (710L) and 715R (715L) is a color sensor with a color filter array in this aspect.
  • the color filter array is a Bayer red, green, green, blue (RGGB) color filter array.
  • RGGB RGGB
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 710R (710L) and 715R (715L) have a common front end optical structure and substantially the same optical path length to each sensor.
  • the color component vector for each pixel output to stereoscopic display 251 from dual image enhancement module 240R (240L) in central controller 260 is derived from the six primary color components for this pixel. This permits driving a stereoscopic display with for example, a four element vector for each pixel without any loss in resolution.
  • Dual image enhancement module 240R (240L) samples the two captured images and applies a color correction matrix to generate the pixel vector required by stereoscopic display 251.
  • the color component vector sent to stereoscopic display 251 can have more than three color components to more accurately represent the spectral content of the scene.
  • the number of color components used in the color component vector sometimes referred to as a color vector, would match the number of color components used by the display.
  • Sharp has shown five color component displays.
  • the fluorescence when fluorescence is excited, the fluorescence may be in the visible spectrum.
  • Fluorescein may be excited by 490 nm blue light and fluoresces mainly in the 520 to 530 nm range.
  • the notch filter for the blue wavelengths is configured to reflect color component Bl and to pass other blue wavelengths so that the fluorescence from the Fluorescein is passed to image capture sensor 715R (715L).
  • the notch filters and the illumination components from the illuminator for the red and green color components are selected so that the reflected color components Rl, Bl, and Gl are captured by first image capture sensor 710R (710L).
  • the remaining colors are delivered to second image capture sensor 715R (715L).
  • the excitation laser component Bl may saturate the blue pixels of first image capture sensor 71 OR (710L), but this is acceptable as the fluorescence from the Fluorescein is captured by second image capture sensor 715R (715L)
  • a color correction module 241R (241L) in dual image enhancement module 240R (240L) of central controller 260 uses the three color pixels from the first image and the three color pixels from the second image to generate a five color component output vector for a Sharp five color display, e.g., generated a red, red complement, yellow, green, blue (RR'YGB) color component.
  • the color correction matrix is set to the desired mapping from the six element acquisition color space to the display' s five element color space.
  • color correction module 241R (241L) in dual image enhancement module 240R (240L) of central controller 260 is coupled to the first and second image capture sensors 710R (710L) and 715R (715L).
  • Camera control unit 230R (230L) demosaics the first image captured by the first image capture sensor 71 OR (710L), and demosaics the second image captured by the second image capture sensor 715R (715L).
  • Each pixel in the demosaiced images has a three element color vector. The two demosaiced imaged are received by color correction module 241R (241L).
  • the three element color vectors are combined from the two demosaiced images to create an N-element color component vector for each pixel, where N is at least three and in this example N is six.
  • the color correction matrix in color correction module 241R (241L) then generates an M-element color component vector for a pixel in an output image from the N-element color component vector of the corresponding pixel. Note that when M is three, the processes described above that utilize the two captured images to enhance resolution, etc. can also be applied.
  • a white broadband illuminator 7 IOC (Fig. 7C) is coupled to illumination channel 705 (Fig. 7A).
  • illuminator 710C use a Xenon lamp 710C1 with an elliptic back reflector and a band pass filter coating to create broadband white illumination light with little infrared content.
  • a Xenon lamp is illustrative only and is not intended to be limiting.
  • a high pressure mercury arc lamp, other arc lamps, or other broadband light sources may be used.
  • coated first surface of 731R (731L) of prism assembly 730R (730L) includes a filter 790F (Fig. 7F) that includes a plurality of notch filters.
  • a first notch filter 79 IF reflects a first part Rl of a red color component R in the received light and transmits a complement Rl' of first part Rl of red color component R in the received light.
  • a second notch filter 792F reflects a first part Gl of a green color component G in the received light and transmits a complement Gl' of first part Gl of green color component G in the received light.
  • a third notch filter 793F reflects a first part Bl of a blue color component B in the received light and transmits a complement Bl' of first part Bl of blue color component B in the received light.
  • the use of three notch filters is for ease of discussion and is not intended to be limiting to three separate filters.
  • the three separate notch filters can be viewed as a single multi-notch filter with the reflect, transmit properties described here.
  • coated first surface of 731 R (731L) reflects red color component Rl in the received light, green color component Gl in the received light, and blue color component Bl in the received light onto second surface 732R (732L).
  • Coated surface of 732R (732L) reflects all wavelengths received from coated first surface 731R (731L) onto image capture sensor 710R (710L).
  • Coated first surface of 731 R (731L) transmits red color component Rl' in the received light, green color component Gl' in the received light, and blue color component Bl' in the received light to third surface 541R (541L) in reflective unit 540R (540L).
  • Coated surface of 541R (541L) reflects all wavelengths received from coated first surface 731R (731L) onto image capture sensor 715R (715L).
  • Each of image capture sensors 71 OR (710L) and 715R (715L) is a color sensor with a color filter array in this aspect.
  • the color filter array is a Bayer RGGB color filter array.
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 71 OR (710L) and 715R (715L) have a common front end optical structure and substantially the same optical path length to each sensor.
  • the color component vector for each output pixel to stereoscopic display 251 from dual image enhancement module 240R (240L) in central controller 260 is derived from the six primary color components for this pixel. This permits driving a stereoscopic display with for example, a four element color vector for each pixel without any loss in resolution.
  • Dual image enhancement module 240R (240L) samples the two images and applies a color correction matrix to generate the pixel vector required by stereoscopic display 251.
  • the coatings— the notch filter— on coated first surface 731R (731L) provide the flexibility to pick the color separation.
  • the color separation of coated first surface 731R (731L) is, of course, multiplied by the pixel RGGB pattern.
  • coated first surface 731R (731L) is separating out yellow for example, it is expected that this would excite both green and red pixels and this would then be recovered in the color correction matrix process of the imaging pipeline.
  • the coated first surface can be configured to pass blue wavelengths in a fluorescence spectrum, and reflect all other wavelengths of blue, for example.
  • the color correction matrix acts upon a six element vector per pixel (a three color component vector of a corresponding pixel in a first image captured by image capture senor 71 OR (710L) plus a three color component vector of a corresponding pixel from a second image captured by image capture sensor 715R (715L)) and produces an output color component vector with the number of color components in the display - for a "normal” LCD, this is the red, green blue (RGB)color components, and for a Sharp five color display, this is the red, red complement, yellow, green, blue (RR'YGB) color components.
  • color correction module 241R (241L) in dual image enhancement module 240R (240L) of central controller 260 is coupled to the first and second image capture sensors 71 OR (710L) and 715R (715L).
  • Camera control unit 230R (230L) demosaics the first image captured by the first image capture sensor 71 OR (710L) and generates an N-element color vector for each pixel— N typically is three.
  • Camera control unit 230R (230L) similarly demosaics the second image captured by the second image capture sensor 715R (715L) into pixels, where each pixel is presented by a second N-element color vector.
  • Dual image enhancement module 240R (240L) then processes the two N-element color vector as a 2N (typically six) color vector through color correction color correction module 241R (241L) to create an output color vector for each pixel with the desired number of color components.
  • the output color vector for a typical liquid crystal display would have three color components.
  • the output color vector would be a five element color vector.
  • a specific color correction matrix would be chosen based on optimization to achieve the best color performance by some metric. The metric could be, for example, to match the color seen by a human observer with Xenon illumination for some set of tissue types.
  • illuminator 710C (Fig. 7C) is coupled to illumination channel 705 (Fig. 7A). Illuminator 710C is the same as that described above and so that description is not repeated here. Thus, the scene is illuminated by broadband white light from illuminator 710C.
  • lens element 704R (704L) is different from the lens element considered above for the other aspects related to Fig. 7A.
  • lens element 704R (704L) is designed to correct for longitudinal color aberrations so that the various color components focus on about the same plane.
  • lens element 704R (704L) does not correct for longitudinal color aberrations.
  • lens element 704R (704L) is designed to focus different wavelengths of light at different distances from the lens, i.e., is designed to have a significant and controlled amount of longitudinal color.
  • lens groups 482 and 484 in lens assembly 401B are_designed to have a significant and controlled amount of longitudinal color. Such lens groups are known to those knowledgeable in the field and so is not considered in further detail herein.
  • coated first surface of 731R (731L) of prism assembly 730R (730L) includes filter 790F (Fig. 7F) that was previously described. To avoid repetition, the above description of filter 790F is not repeated here.
  • coated first surface of 731 R (731L) reflects red color component Rl in the received light, green color component Gl in the received light, and blue color component Bl in the received light to second surface 732R (732L).
  • Coated surface of 732R (732L) reflects all wavelengths received from coated first surface 731R (731L) onto image capture sensor 710R (710L).
  • Coated first surface of 731 R (731L) transmits red color component Rl' in the received light, green color component Gl' in the received light, and blue color component Bl' in the received light to third surface 541R (541L) in reflective unit 540R (540L).
  • Coated surface of 541R (541L) reflects all wavelengths received from coated first surface 731R (731L) onto image capture sensor 715R (715L).
  • each of image capture sensors 71 OR (710L) and 715R (715L) is a color sensor with a color filter array in this aspect.
  • the color filter array is a Bayer RGGB color filter array.
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 71 OR (710L) and 715R (715L) have a common front end optical structure and substantially the same optical path length to each sensor.
  • prism assembly 730R (730L) and reflective unit 540R (540L) are arranged so that the color images captured by both Bayer pattern image capture sensors 710R (710L) and 715R (715L) are the same scene
  • the color image captured by image capture sensor 71 OR (710L) has the same full resolution as the color image captured by image capture sensor 715R (715L).
  • each point in space in the scene is represented by two pixels with each of the two pixels having a different three element vector.
  • sensor assembly 720R (720L) has acquired a full resolution image with six primary color components— three from the first captured image and three from the second captured image— for each point in space with no loss of light due to the notch filters.
  • dual image enhancement module 240R includes a digital filter kernel for each of the first and second images.
  • the digital filter kernel processes the captured image based on the known longitudinal color aberrations to enhance the sharpness and focus of the image, i.e., to generate third and fourth images.
  • the third and fourth images are demosaiced.
  • the resulting demosaiced third image and fourth image can each be focused at a range of distances from the lens assembly and so as to provide an image having a greater depth of field than from a traditional lens design that brought all the color components into focus.
  • Digital filter kernels are known to those knowledgeable in the field. This approach works well for surfaces with smooth reflectance curves in wavelength; this is the case for most tissue.
  • the vector for each output pixel to stereoscopic display 251 from dual image enhancement module 240R (240L) in central controller 260 is derived from the six primary color components for pixels in the demosaiced third and fourth images.
  • Dual image enhancement module 240R (240L) samples the third and fourth images created by the digital filter kernel and applies a color correction matrix to generate the pixel vector required by stereoscopic display 251.
  • the resulting combined image has a depth of field that is potentially three times greater than an image obtained from the conventional lens system.
  • an illuminator 710D (Fig. 7D) is coupled to illumination channel 705 (Fig. 7A).
  • Illuminator 710D includes three laser illumination sources 710D1 to 710D3 that generate a red color component Rl, a green color component Gl, and a blue color component Bl, respectively.
  • three laser illumination sources 710D1 to 710D3 are referred to as three laser illumination modules.
  • the configuration of an illuminator with multiple different illumination sources, a mirror for source 710D1 and dichroic mirrors for sources 710D2 and 710D3 is known.
  • the wavelengths of red color component Rl, a green color component Gl, and blue color component Bl are typically 2 to 5 nanometer (nm) wide instead of about 60 nm per color component from a conventional light emitting diode based illuminator.
  • the light from the red, green and blue lasers reflects off tissue 703.
  • the intensities of the reflected color component Rl, color component Gl, and color component Bl are modulated by the reflectance function of the tissue.
  • tissue 703 may emit other light.
  • the light from one of the lasers may excite fluorescence. See for example spectrum 795G (Fig. 7G).
  • the three peaks are light reflected by tissue 703 from the red, green and blue lasers of illuminator 710D.
  • the relatively smaller and broader peak in the red region of the spectrum, is fluorescence.
  • Spectrum 795 G is presented to aid in visualization of the light that enters endoscope 702 and is not representative of any actual data.
  • Sensor assembly 720R (720L) enables capture of a color image by first image capture sensor 710R (7101.) and enables all the light not associated with the color image to be redirected to second image capture sensor 7I5L (715R).
  • sensor assembly 720R (720L) also enables imaging fluorescence in real time at nearly any wavelength in the visible or near infrared, and enables imaging natural tissue fluorescence.
  • Fluorescent markers may be stimulated by a narrow band source like a laser 710E4 in illuminator 710E that emits a wavelength in the near infrared spectrum.
  • the fluorescent emission spectrum is a range of wavelengths.
  • the shape of the fluorescent emission spectrum is not significantly dependent on the excitation wavelength in a range of excitation wavelengths for the fluorescence.
  • fluorescence is triggered by light from a laser module 710E4 in illuminator 710E.
  • antibody agents which were obtained from Medarex, Inc., were excited using a 525 nm laser.
  • illuminator 710D includes one or more light modules in addition to the three laser modules that generate red color component Rl, green color component Gl, and blue color component Bl. The number and type of additional light modules are selected based on the fluorescence excitation wavelengths for one or more fluorescences of interest.
  • the particular fluorescence excitation source selected for combination light source 210 depends on the fluorophore or fluorophores used. Excitation and emission maxima of various FDA approved fluorescent dyes used in vivo are presented in Table 1.
  • Indigo Carmine 1 ' is a U.S. registered trademark of Akorn, Inc. of Lake Forrest, 111. USA.
  • Table 2 presents examples of common protein fluorophores used in biological systems.
  • a fluorophore can be bound to an agent that in turn binds to a particular tissue of the patient.
  • another laser module 710E4 can be added to illuminator 710D to obtain illuminator 710E that provides light with the excitation maxima wavelength for that fluorophore.
  • appropriate light sources can be included in illuminator 710D to provide illumination of a color image of the scene and to excite fluorescence.
  • an appropriate light source can be selected based upon the fluorescence being utilized.
  • coated first surface of 731 R (731L) of prism assembly 730R (730L), i.e., the beam splitter includes a filter 790H (Fig. 7H) that includes a plurality of notch filters.
  • a first notch filter 791H reflects red color component Rl in the received light and transmits other wavelengths of red color component R in the received light.
  • a second notch filter 792H reflects green color component Gl in the received light and transmits other wavelengths of green color component G in the received light.
  • a third notch filter 793H reflects blue color component Bl in the received light and transmits other wavelengths of blue color component B in the received light.
  • the use of three notch filters is for ease of discussion and is not intended to be limiting to three separate filters. The three separate notch filters can be viewed as a single multi-notch filter with the reflect, transmit properties described here.
  • coated first surface of 731 R (731L) reflects red color component Rl in the received light, green color component Gl in the received light, and blue color component Bl in the received light to second surface 732R (732L).
  • Second surface of 732R (732L) reflects all wavelengths received from coated surface 731R (731L) onto image capture sensor 710R (710L).
  • Coated first surface of 731 R (731L) transmits red light other than red color component Rl in the received light, green light other than green color component Gl in the received light, and blue light other than blue color component Bl in the received light to third surface 541R (541L) in reflective unit 540R (540L).
  • Third surface of 541R (541L) reflects all wavelengths received from coated surface 731R (731L) onto image capture sensor 715R (715L).
  • image capture sensor 71 OR (710L) acquires an RGB image of the laser illuminated scene. This is a color image. All the remaining light is reflected onto image capture sensor 715R (715L). This light must be from a source other than the illumination. For example, if there is natural tissue fluorescence, the fluorescence is captured by image capture sensor 715R (715L). If image capture sensor 715R (715L) is a color image capture sensor, the fluorescence can be of any color and the fluorescence is imaged in color on image capture sensor 715R (715L). There may be slight losses in the fluorescence due to the notch filters on coated first surface of 731 R (731L). In this aspect, image capture sensor 71 OR (710L) is a color sensor with a color filter array. The color filter array is a Bayer RGGB color filter array.
  • image capture sensor 715R (715L) depends on the characteristics of the other light that is captured by image capture sensor 715R (715L).
  • the gain and other parameters of image capture sensor 715R (715L) may be independent of image capture sensor 71 OR (710L) so that image capture sensor 715R (715L) acquires the best image possible.
  • image capture sensor 715R (715L) can be a monochrome sensor.
  • image capture sensor 715R (715L) can be a color image capture sensor with a Bayer color filter array.
  • image capture sensor 715R (715L) the two image capture sensors are looking through the same optics at the same scene.
  • coplanar image capture sensors 71 OR (710L) and 715R (715L) have a common front end optical structure and substantially the same optical path length to each sensor.
  • an excitation blocking notch filter is required in the image capture unit imaging path, e.g., between stop 570R (570L) and sensor assembly 720L (720R). Irrespective of the excitation wavelength of the fluorescence, the fluorescent light can be in the visible spectrum and be imaged onto image capture sensor 715R (715L). Multiple fluorophores could be imaged simultaneously if desired and the emission spectra excite different colors on image capture sensor 715R (715L).
  • dual enhancement module 240R depends on the whether second image capture sensor 715R (715L) is a color image capture sensor or a monochrome image capture sensor. If both image capture sensors are color sensor, CCU 230R (230L) demosaics the two captured images and controller 260 generates an N-element color component vector for each pixel in the output image as described above. Three of the color components in the N-element color component vector represent a color image and the remaining color components represents fluorescence. If the second image capture sensor is monochrome, the demosaicing of the second captured image is unnecessary.
  • controller 260 is configured to generate at least a four element color component vector for a pixel in an output image from a color component vector for a corresponding pixel in the first image and a color component vector for a corresponding pixel in the second image.
  • Three of the elements in the four element color component vector are for a visible color image, and a fourth element in the four element color component vector is for a fluorescent image.
  • the image capture sensors in a stereoscopic channel are coplanar.
  • a first of the image capture sensors could have a top sensor surface in a first plane
  • a second of the image capture sensors could have a top sensor surface in a second plane.
  • the first and second planes are parallel to each other and are separated by a known distance (See Fig. 9).
  • the spacing between the beam splitter and the reflective surface in the reflective unit is adjusted to compensate for the known distance so that the first optical path length to the first image capture sensor is about equal to the second optical path length to the second image capture sensor.
  • a first image 881 (Fig. 8B) captured by sensor assembly 820R (820L) has a first focus.
  • a second image 882 (Fig. 8C) captured by sensor assembly 820R (820L) has a second focus.
  • image 881 the features in the center of the image are in focus as represented by the dark distinct lines.
  • the features in the left and right sides of image 881 are further from image capture unit 825R (825L) and are out of focus as represented by the cross-hatching.
  • image 882 the features in the center of the image are out of focus as represented by the cross-hatching.
  • the features in the left and right sides of image 882 are in focus as represented by the dark distinct lines.
  • the in focus portions of images 881 and 882 are used to generate a composite in focus image that is sent to stereoscopic display 251 (Fig. 8D).
  • This provides an image containing better depth of field than is available in either of images 881 and 882.
  • Another benefit of this approach is that information from images 881 and 882 along with a sharpness metric are used to generate an image from a perspective of virtual cameras that are sent to stereoscopic display 251 for display. This allows the surgeon to view tissue 803 from slightly different perspectives without physically moving endoscope 802.
  • FIG. 8A is a schematic illustration of a distal end of a stereoscopic endoscope 802 with image capture units 825L, 825R and an illumination channel 805 that provides light from an illuminator. As indicated by arrow 835, the distal direction is towards tissue 803 and the proximal direction is away from tissue 803.
  • Each image capture unit 825R, 825L includes a lens assembly 801R, 801L and a sensor assembly 820R, 820L.
  • Sensor assembly 820R, 820L is positioned to receive light that passes through lens assembly 801R, 801L.
  • Each sensor assembly 820R, 820L includes a prism assembly 830R, 830L, a reflective unit 840R, 840L, and coplanar image capture sensors (810R, 815R), (810L, 815L), in one aspect.
  • the configuration illustrated in Fig. 9 could also be used in the aspects described using the configuration of Fig. 8A.
  • Stereoscopic endoscope 802 is sometimes referred to as endoscope 802.
  • endoscope 802 are described.
  • the optical paths through the left channel of stereoscopic endoscope 802 are equivalent those in the right channel due to the symmetry of endoscope 802, and so in the following description, the left channel reference numeral is included within parentheses following the description of an element in the right channel. This indicates that the description is also applicable to the corresponding element in the left channel.
  • the imaging processing that is described is performed in real time so that a continuous video sequence is provided to the display.
  • the light from tissue 803 passes through lens element 804R (804L) and stop 870R (870L) in lens assembly 801R (801L).
  • the elements in lens assembly 40 IB are an example of lens element 804R (804L).
  • the light that passes through lens assembly 801R (801L) is received by image capture unit 825L (825R) and enters prism assembly 830R.
  • Prism assembly 830R (830L) includes a beam splitter 831R (831L) that reflects a first percentage of the received light. Beam splitter 831R (831L) passes a second percentage of the received light through beam splitter 831R (831L) to reflective unit 840R (840L).
  • beam splitter 831R (831L) is implemented as a coated first surface 831R (831L).
  • the light reflected by coated first surface 831R (831L) is received by a second surface 832R (832L) that in turn directs, e.g., reflects, the light onto first image capture sensor 810R (810L).
  • Surface 832R (832L) can be either a coated surface or a total internal reflection surface.
  • the light transmitted through coated first surface 831R (831L) is received by a third surface 541R (541L) of reflective unit 540R (540L) that in turn directs, e.g., reflects, the light onto second image capture sensor 815R (815L).
  • Second surface 832R (832L) is positioned so that no light other than the light reflected by coated first surface 831R (831L) hits second surface 832R (832L).
  • the angle of incidence of the light to coated surface 831R (831L) is less than forty-five degrees.
  • prism assembly 830R (830L) and reflective unit 840R (840L) are included in a prismatic structure with a pentaprism that includes prism assembly 830R (830L).
  • the prismatic structure in this aspect is equivalent to prismatic structure 460 (Fig. 4B) with buried coated
  • prismatic structure 460 is applicable to the prismatic structure used in this aspect.
  • prism assembly 830R (830L) includes a beam splitter 831R
  • Image capture sensors 810R (810L) and 815R (815L) are coplanar.
  • a first optical path length from stop 870R (870L) to coated first surface 831R (831L) to second surface 832R (832L) to image capture sensor 810R (810L) is a first length.
  • a second optical path length from stop 870R (870L) through coated first surface 831R (831L) to a third surface 841R (841L) in assembly 840R (840L) to image capture sensor 815R (815L) is a second length.
  • the first and second lengths are different, i.e., are not about equal.
  • optical path lengths as starting at stop 870R (870L) is illustrative and is not intended to be limiting.
  • the unequal optical path lengths could also be variously defined, such as with respect to a distal face of prism assembly 830R (830L) through which the received light enters prism assembly 830R (830L), with respect to a first element in lens assembly 801R (801L), or with respect to coated first surface 831R (831L).
  • the different optical path lengths can be achieved with glass plates on the two image sensor areas with different indexes of refraction or other similar means.
  • coplanar image capture sensors 810R (810L) and 815R (815L) have a common optical path length through the front end optical structure in lens assembly 801R (801L) and different optical path lengths to each sensor in sensor assembly 820R (820L).
  • First image capture sensor 810R (810L) captures an image from the first portion of the light received by sensor assembly 820R (820L).
  • Second image capture sensor 815R (815L) captures an image from the second portion of the light received by sensor assembly 820R (820L).
  • each of image capture sensors 810R (810L) and 815R (815L) is a color sensor with a color filter array.
  • the color filter array is removed from one of the color sensors and the sensor functions as a monochrome sensor.
  • the sharpness from the image captured by the monochrome sensor is transferred to the color image if the focus of the two images is not too different, e.g., the two images are focused in the vicinity of the point where sharpness curves 885, 886 of the two images intersect (See Fig. 8E).
  • coated first surface 831R (831L) of prism assembly 830R (830L) is configured to reflect and is configured to transmit about equal portions of the light received by prism assembly 830R (830L), i.e., the first and second percentages are about equal.
  • beam splitter 831R in this aspect, is the balanced beam splitter described above.
  • each of image capture sensors 810R (810L) and 815R (815L) is a color sensor with a color filter array.
  • the color filter array is a Bayer color filter array.
  • the two Bayer pattern image capture sensors 810R (810L) and 815R (815L) are looking through the same optics at the same scene.
  • a Bayer pattern image capture sensor is a single chip sensor, or part of a single chip, that includes a Bayer color filter array.
  • coplanar image capture sensors 810R (810L) and 815R (815L) have a common front end optical structure, but different optical path lengths to each sensor.
  • the light going to image capture sensor 810R (810L) is captured as a first image focused at a first object distance DA from image capture unit 825R (825L). See for example Fig. 8B.
  • the light going to image capture sensor 815R (815L) takes a slightly longer path to arrives sensor at 815R (815L) and is captured as a second image focused at a second object distance DB from the image capture unit. See for example, Fig. 8C.
  • Distance DB is larger than distance DA.
  • the difference in the path lengths is based on the front end optical design, e.g., focal length of lens element 804R (804L), and the distance for the best focus of each of the two captured images.
  • the first and second images are aligned and taken through the same lens element 804R (804L).
  • the focal plane of the two images is different.
  • the depth of field of the two images is also different.
  • the image focused at distance DA e.g., image 881
  • the image focused at distance DB e.g., image 882
  • surgeon can use user interface 262 to cause dual image enhancement module 240R (240L) to generate, from the two captured images (Figs. 8B and 8C), an in focus image (Fig. 8D) with an enhanced field of view
  • dual image enhancement module 240R 240L
  • the surgeon can show the two images on two planes of a display so the accommodation in surgeons control console display 251 , sometimes called stereoscopic display 251 , matches the surgical accommodation at the surgical site.
  • surgeon can switch between the first image with one focus and the second image with a second focus.
  • dual image enhancement module 240R (240L) generates an in focus extended depth of field image from the two images captured by image capture sensors 81 OR (810L) and 815R (815L).
  • a first sharpness metric is generated for a group of pixels, i.e., a tile, in the first image and a second sharpness metric is generated for a corresponding group of pixels in the second image.
  • the two sharpness metrics are compared to determine which sharpness metric is larger.
  • the group of pixels (or a selected pixel or pixels within the tile) with the largest sharpness metric is selected for the blended image.
  • Dual image enhancement module 240R (240L) steps through the two captured images group by group and selects one of the two groups for inclusion in the blended image that is in focus over a greater field of view than the two captured images.
  • Fig. 8E is an illustration of a sharpness profile 885 for imaged captured by image capture sensor 810R (810L) and a sharpness profile 886 for the images captured by image capture sensor 815R (815L).
  • the sharpness profile is a curve of the sharpness of an image captured by an image capture sensor versus the distance from the distal end of the endoscope.
  • sharpness profiles 885 and 886 are empirically generated based on a calibration performed using endoscope 802 and a plurality of images taken of known objects at different distances from the distal end of endoscope 802.
  • profiles 885 and 886 are stored as look-up tables in a memory.
  • the stored sharpness profiles are used by dual image enhancement module 240R (240L) to determine whether to display a first image captured by image capture sensor 81 OR (810L) or to display a second image captured by image capture sensor 815R (815L) as endoscope 802 is moved without moving the optics in image capture unit 825R (825L).
  • Dual image enhancement module 240R (240L) generates a sharpness metric, using a conventional technique know to those knowledgeable in the field, for each of the first and second images.
  • Dual image enhancement module 240R uses sharpness profiles 885 and 886 and the sharpness metrics to determine a distance between the distal end of endoscope 802 and the tissue for each of the images.
  • distance DA to tissue 803 is either about forty-two millimeters (mm) or about twenty seven millimeters (mm) based on sharpness profile 885.
  • second image has a sharpness metric of about 0.43
  • distance DB to tissue 803 is about forty-two millimeters (mm) based on sharpness profile 886.
  • second image has a sharpness metric of about 0.08
  • the tissue distance is about twenty seven millimeters (mm).
  • sharpness profile 885 is ambiguous as to tissue depth based on the sharpness metric; the tissue depth is disambiguated by second curve 886. As both images are obtained at substantially the same time, this disambiguation is not skewed by temporal effects.
  • Dual image enhancement module 240R (240L) uses the relative sharpness metric between the two images and the a-priori calibration of the curves to determine which of the two images to display as endoscope 802 is moved and the distance from the image capture unit to the tissue varies. The position of the distal end of the endoscope is known by central controller 260.
  • endoscope 802 If endoscope 802 is moved closer to tissue 803, the first captured is image is sent to display 251 by dual image enhancement module 240R (240L). If endoscope 802 is moved away from tissue 803, dual image enhancement module 240R (240L) compares the distance from tissue 803 of the distal end of endoscope 802 with the distance at which sharpness profiles 885 and 886 intersect, referred to as the profile intersection distance. If the distance from tissue 803 of the distal end of endoscope 802 is smaller than the profile intersection distance, the first captured image is transmitted to display 251. If the distance from tissue 803 of the distal end of endoscope 802 is larger than the profile intersection distance, the second captured image is transmitted to display 251.
  • endoscope 802 is moved, the surgeon is presented with an appropriate image without having to move the optics in image capture unit 825R (825L).
  • the determination of which of the two images to present to the surgeon may be determined solely on the basis of the relative sharpness metric; the calculation of tissue depth is not necessary.
  • the image pair presented in the left and right eyes of the stereo viewer would be kept consistent and the sharpness metric comparison may use the data from the pair of images captured for the left eye and the pair of images captured for the right eye during the determination to use the first or second image.
  • the first image would be chosen for both the right and left eyes or the second image would be chosen for the right and left eyes.
  • the captured two images and sharpness profiles 885, 886 are used to create a depth map.
  • Sharpness profiles 885 is divided into sharpness profile 886 to generate a channel sharpness profile of a sharpness metric versus distance e.g., the value of profile 886 at a given distance is divided by the value of profile 885 at that given distance to obtain the sharpness metric for the given distance.
  • This sharpness metric may be calculated through the use of sliding tiles to create a depth map for the entire image. Naturally, this information may be combined with more traditional stereo matching algorithms to generate a more well conditioned depth map.
  • sharpness profiles 885 and 886 are empirically generated based on a calibration performed using endoscope 802 and a plurality of images taken of known objects at different distances from the distal end of endoscope 802. Sharpness profiles are obtained for both image capture unit 825R and for image capture unit 825L. A channel sharpness profile is generated for each of the left and right channels. In one aspect, the left and right channel sharpness profiles are saved as a look-up table in a memory 899 (Fig. 8F).
  • Dual image enhancement module 240R (240L) generates a sharpness metric, using a conventional technique know to those knowledgeable in the field, for tiles or groups around corresponding pixels in each of the first and second images.
  • the sharpness metric for the pixel, or a group of pixels about the pixel, in the first captured image is divided into the sharpness metric for the corresponding pixel, or corresponding group of pixels about the corresponding pixel, in the second capture image to generate a channel sharpness metric for the pixels.
  • the channel sharpness metric for the pixels is used to locate a point on the channel sharpness profile.
  • the distance corresponding to that point on the channel sharpness profile is the depth associated with the (X, Y) position of the pixel. Performing this process for each pixel in the captured images generates a depth map for the captured images.
  • the depth map is used to generate view of a scene of tissue 803 from a perspective different from the perspective of endoscope 802, i.e., from a perspective of a virtual camera view point.
  • Central controller 260 uses a three-dimensional virtualization module 890 (Fig. 8F) to generate a stereoscopic image pair for the virtual camera view points.
  • Three-dimensional virtualization module 890 uses the first and second images captured in the right stereoscopic channel of endoscope 802, and generates a first channel depth map for the scene viewed by the right channel. Similarly, three-dimensional virtualization module 890 uses the first and second images captured in the left stereoscopic channel of endoscope 802, and generates a second channel depth map for the scene viewed by the left channel. Additional information from stereo matching in the images may also be used to improve the stability and accuracy of the combined depth map. A three-dimensional surface of the scene is generated using the first and second channel depth maps and the best focused portions of the four captured images are projected on and textured onto the three-dimensional surface to create a textured three-dimensional image surface.
  • dual image enhancement module 240R uses the channel depth map for the right channel and the textured three-dimensional image surface to generate an image for the right channel from the virtual camera view point.
  • dual image enhancement module 240L uses the channel depth map for the left channel and the textured three-dimensional image surface to generate an image for the left channel from the virtual camera view point.
  • the two images are sent to stereoscopic display 251 and the surgeon is presented a virtual three-dimensional image without moving endoscope 802 to the virtual camera view point.
  • a 3-D virtualization module 895 is executed for each channel. This can be done either in parallel or sequentially depending on the memory and processing power available in central controller 260.
  • RETRIEVE IMAGES process 891 for the right channel first and second images captured in image capture sensors 81 OR and 815R are retrieved and provided to CREATE DEPTH MAP process 892, sometimes called process 892.
  • Process 892 performs the same process repetitively for groups of pixels, e.g., tiles, in the two images.
  • a group of pixels including pixel (X, Y) are selected from each of the first and second images in the right channel.
  • X is the x-coordinate of the pixel
  • Y is the y-coordinate of the pixel.
  • the group of pixels could be a five pixel by five pixel block, with pixel (X, Y) at the center of the block.
  • a first sharpness metric for the block of pixels from the first image is generated and a second sharpness metric for the same block of pixels from the second image is generated.
  • the second sharpness metric is divided by the first sharpness metric to generate a channel sharpness metric.
  • a distance Z corresponding to the channel sharpness metric is obtained using right channel sharpness profile in memory 899.
  • Distance Z is assigned to pixel (X, Y).
  • the pixel is represented as pixel (X, Y, Z xy ).
  • a right channel depth map has been created with each pixel in the scene viewed by the right channel.
  • Each pixel has a value in the x-dimension, a value in the y-dimension, and a depth in the z-dimension.
  • the right channel depth map is stored in a memory 895, in one aspect.
  • the depths in the right channel depth map are averages values that are a function of the size of the group of pixels.
  • a large group of pixels reduces noise, but as the group becomes larger detail in the z-dimension is lost due to the averaging over the group.
  • a size for the group is empirically determined to reduce the noise to an acceptable level while maintaining the desired specificity in the depth values.
  • the placement of the pixel for which the depth is being generated at the center of block is illustrative only and is not intended to be limiting to this specific placement.
  • memory 895 includes a right channel depth map and a left channel depth map.
  • the left and right channel depth maps provide a depth mapping of the scene for the view point of endoscope 802.
  • GENERATE 3-D SURFACE process 893 in central controller 260 uses the left and right channel depth maps to generate a three dimensional surface of the scene.
  • the use of two depths maps to generate a three dimensional depth map surface is known to those knowledgeable in the field and so is not considered further herein.
  • PROJECT AND TEXTURE MAP IMAGES process 894 projects and texture maps the sharpest portions of the two acquired images in the right channel and the sharpest portions of the two acquired images in the left channel back on the three dimensional depth map surface to generate a textured three-dimensional image surface, sometimes referred to as a textured image surface.
  • Projecting and texturing using a plurality of captured images and a three dimensional surface to generate a textured image surface is known to those knowledgeable in the field and so is not considered further herein.
  • images for a stereoscopic image pair from a virtual camera view point can be generated.
  • user interface 262 permits a surgeon to specify a virtual camera view point from which the surgeon wishes to view the current scene, e.g., a different perspective from which the surgeon wishes to view the current scene.
  • RECEIVE VIRTUAL CAMERA VIEW POINT process 897 receives the virtual camera view point from user interface 262 and provides the virtual camera view point to GENERATE IMAGES process 896.
  • GENERATE IMAGES process 896 uses the right channel depth map from DEPTH MAPS 895, the textured image surface, and the virtual camera view point to generate a virtual right channel image from the virtual camera view point.
  • GENERATE IMAGES process 896 uses the left channel depth map from DEPTH MAPS 895, the textured image surface, and the virtual camera view point to generate a virtual left channel image from the virtual camera view point. Generating a virtual image using a depth map and a textured image surface for a virtual camera view point is know to those knowledgeable in the field and so is not considered in further detail.
  • TRANSMIT IMAGES process 898 Upon generation of the left and right channel virtual images for the virtual camera view point, TRANSMIT IMAGES process 898 sends the two virtual images to stereoscopic display 251 for display to the surgeon.
  • the surgeon is able to view the scene from a different view point, i.e., a virtual camera view point.
  • a virtual camera view point i.e., a virtual camera view point.
  • viewing the scene from a view point wildly different than that of endoscope 802 does not necessarily lead to a high quality image.
  • images for virtual camera points near the endoscope tip are of reasonable quality.
  • This approach may also be used for generating the images for other participants in an operating room.
  • the operating room staff near the patient likely views the surgical procedure on a non-stereo display unit, such as on a television screen or a monitor.
  • the ability to create images from virtual camera positions can enable the presentation of the scene in such a way as to give the operating room staff a sense of depth on that display.
  • This may be done by presenting, in real time, a sequence of images from virtual camera positions which sweep from the actual surgeon right eye position to the actual surgeon left eye position and back, i.e., the virtual camera positions are swept over the interocular separation between the two eye positions.
  • the sweeping of the virtual camera view point back and forth generates a sequence of images of a scene from a plurality of virtual camera view points. Specifically, at each time step, an image from a different virtual camera view point is generated and presented.
  • the actual right eye position and the actual left eye position refer to the left eye position and the right eye position of a stereoscopic viewer that is a part of the surgeons control console, in one aspect.
  • the left eye position and right eye position are separated by the interocular separation.
  • the display of the images generated by this sweeping of the virtual camera position over time gives the appearance on the display of small back and forth head motions, i.e., the displayed images provide a slight rocking back and forth of the scene.
  • This is one of the many depth cues used by the human visual system and provides a sense of the scene depth to the operating room staff without requiring the use of a stereoscopic display. Additionally, this has the advantage that the image communicates depth independent of the orientation of the operating room staff to the display and does not require any special additional equipment.
  • the virtual camera images may also be created by considering the images captured as samples of a light field and using image based rendering techniques to create the new virtual camera views.
  • the stereoscopic image for the virtual camera view point was generated using images from stereoscopic endoscope 802.
  • a stereoscopic image for the virtual camera view point is generated using a pair of images from a monoscopic endoscope, e.g., an endoscope that has only the right channel of stereoscopic endoscope 802.
  • 3-D virtualization module 890 performs process 891 to 894 for the two captured images to generate a first image for the three-dimensional image.
  • generate images process 896 To generate the second image for the three-dimensional image, generate images process 896 generates a virtual image from a view point with the proper ocular separation so that when the first image and the virtual image are viewed as a stereoscopic pair of images, the surgeon perceives a three-dimensional image.
  • This approach is sufficient to generate a stereoscopic view of the scene for a virtual camera view point when there are no sharp edges of fore-ground objects in the scene viewed by the monoscopic endoscope.
  • the scene is smooth with no sharp edges and so this process generates useable stereoscopic views.
  • there is a surgical instrument in the scene there may be a hole in stereoscopic view behind the surgical instrument, but typically this area is not of interest to the surgeon.
  • the stereoscopic view with the hole behind the surgical instrument provides the information needed by the surgeon and so is acceptable.
  • beam splitter 831R (831L) was configured as a balanced beam splitter.
  • a beam splitter 831R (831L) configured to take advantage of the fact that the illumination of a surgical site is typically accomplished with a light guide, called the illumination channel above, which is part of the endoscope.
  • the illumination is attached to and travels with the tip of the endoscope.
  • coated first surface 831R (831L) is configured to reflect and transmit different portions of the received light, i.e., the first and second percentages defined above are different.
  • Beam splitter 831R (831L) reflects M of the received light and passes N of the received light, where M is different from N .
  • M and N are positive numbers.
  • M plus N is equal to about one hundred percent. The equality may not be exact due to light losses and due to tolerances of the various parts of image capture unit 825R (825L).
  • the other aspects of prism assembly 830R (830L) are the same as described above.
  • the processing of the images captured by sensor assembly 820R (820L) is equivalent to that described above for the balanced beam splitter and so is not repeated.
  • Fig. 9 is a schematic illustration of a part of a channel with an image capture unit that has such image capture sensors.
  • the configuration of Fig. 9 can be used in any of the aspects and examples described herein and so each of the various aspects and examples are not repeated for this configuration.
  • image capture unit 925 includes a lens assembly 901 and a sensor assembly 920.
  • Sensor assembly 920 includes a prism assembly 930, a reflective unit 940, and image capture sensors 910, 915.
  • Lens assembly 901 includes a plurality of optical elements including an optical element that defines an optical stop 970, sometimes referred to as stop 970. Light passing through stop 970 is received by a beam splitter 931 in prism assembly 930 of image capture unit 925.
  • a stereoscopic apparatus would include two image capture units as illustrated in Fig. 9.
  • the left and right stereoscopic channels with image capture units are symmetric, and so only a single channel and image capture unit is described to avoid duplicative description.
  • the other channel including the image capture unit is symmetric across a plane intersecting the longitudinal axis of the endoscope with the channel illustrated in Fig. 9.
  • Beam splitter 931 is positioned to receive light that passes through stop 970. Beam splitter 931 is configured to direct a first portion of the received light to a first image capture sensor 910, and to pass a second portion of the received light through the beam splitter to reflective unit 940.
  • beam splitter 931 in prism assembly 930 is implemented as a coated first surface.
  • beam splitter 931 is sometimes referred to as coated first surface 931. Coated first surface 931 separates the received light into the two portions.
  • Coated first surface 931 reflects the first portion of the received light to a second surface 932 that in turn directs, e.g., reflects, the light onto first image capture sensor 910.
  • the second portion of the received light is passed through coated first surface 931 to reflective unit 940.
  • the angle of incidence of the light to coated surface 931 is less than forty-five degrees.
  • Second surface 932 is positioned so that no light other than the light reflected by coated first surface 931 hits second surface 932.
  • second surface 932 is a reflective surface that is implemented as one of a coated surface and a total internal reflection surface.
  • Reflective unit 940 includes a third surface 941 that reflects the light received from beam splitter 931 onto a second image capture sensor 915.
  • third surface 941 is a reflective surface implemented, for example, as one of a coated surface and a total internal reflection surface.
  • First image capture sensor 910 has a top sensor surface in a first plane 911.
  • Second image capture sensor 915 has a top sensor surface in second plane 914.
  • First plane 911 is substantially parallel to second plane 914 and is separated from second plane 914 by a known distance d.
  • about parallel or substantially parallel means that the two planes are planes within the tolerances associated with manufacturing and mounting the image captures sensors.
  • the top sensor surface of an image capture sensor is the surface of the image capture sensor that receives light from at least one optical component in the sensor assembly.
  • a first optical path length from stop 970 to coated first surface 931 to second surface 932 to image capture sensor 910 is about equal to a second optical path length from stop 970 through coated first surface 931 to coated surface 941 to image capture sensor 915.
  • image capture sensors 910 and 915 have a common optical path length through the front end optical structure in lens assembly 901 and about the same optical path length to each image capture sensor in sensor assembly 920.
  • coated surface 941 is positioned so that the first and second optical path lengths are not equal.
  • image capture sensors 910 and 915 have a common optical path length through the front end optical structure in lens assembly 901 and different optical path lengths to each image capture sensor in sensor assembly 920.
  • central controller 260 with camera control units (CCUs) 230R, 230L controls the frequency at which sets of images in each of the channels are captured.
  • the processing of the captured images described above is done in real time, in one aspect, so that the surgeon views a video sequence of stereoscopic images.
  • the above examples considered an image capture unit mounted at a distal end of an endoscope, either stereoscopic or monoscopic.
  • the novel image capture unit configuration can be used in conventional endoscope cameras as well as in for example, stereoscopic surgical microscopes. Accordingly, the description of implementing the image capture unit in an endoscope is illustrative only is not intended to be limiting.
  • Fig. 10 is a schematic illustration of a distal end of a stereoscopic endoscope 1002 with an image capture unit 1025.
  • Image capture unit 1025 includes left lens assembly 1001L, right lens assembly 1001R and a sensor assembly 1020 with coplanar right and left image capture sensors 1010R and 1010L, in this aspect.
  • stereoscopic endoscope 1002, sometimes referred to as endoscope 1002 includes an illumination channel 1005.
  • any of the various illuminators described herein including internal and external illuminators can be used with endoscope 1002.
  • right and left image capture sensors 1010R and 1010L are included in a semiconductor substrate 1017, sometime called a semiconductor die or semiconductor chip, which is mounted on a platform 1012.
  • the light from tissue 1003 passes through lens assemblies 1001R and 1001L to sensor assembly 1020.
  • the elements in lens assembly 40 IB (Fig. 4B) are an example of lens assembly 1001R and lens assembly 1001L.
  • the light that passes through lens assemblies 1001R and 1001L is received by a reflective surface 1041 of a reflective unit 1040.
  • Reflective surface 1041 directs, e.g., reflects, the received light onto a top sensor surface of first image capture sensor 1010R and onto a top sensor surface of second image capture sensor 1010L.
  • Surface 1041 can be either a coated surface or a total internal reflection surface.
  • first image capture sensor 1010R and to second image capture sensor 1010L within endoscope 1002 are substantially the same length.
  • a first optical path length from lens assembly 1001R to the top sensor surface of first image capture sensor 1010R is about equal to a second optical path length from lens assembly 1001L to the top sensor surface of second image capture sensor 1010L.
  • each stereoscopic optical channel in endoscope 1002 has about the same optical path length to an image sensor.
  • the surface of image capture sensor 1010R and the surface of image capture sensor 1010L are coplanar.
  • image capture sensor 1010R has a top sensor surface in a first plane
  • image capture sensor 1010L has a top sensor surface in a second plane, where the first and second planes are parallel and are separated by a known distance (See Fig. 9).
  • the position of the reflective surface in the reflective unit is adjusted to compensate for the known distance so that the first optical path length to the first image capture sensor is about equal to the second optical path length to the second image capture sensor.
  • a lens could be used in one or both of the optical paths to create the about equal optical path lengths.
  • the images captured by first image capture sensor 1010R and to second image capture sensor 1010L have the same focus and depth of field and are spatially registered relative to each other.
  • Image capture unit 1025 is small and compact and so can be used in applications that do not have sufficient space to accommodate the image capture units described above. Also note that this image capture unit is not symmetric about central axis 1090 of endoscope 1002. In certain applications where there are additional lumens for tools or instruments, this asymmetry is an advantage.
  • This construction also affords perfect alignment of the imaging areas for the left and right eyes in the case where the image sensor comprises an imaging area for each eye on a single silicon die or in the case where the device uses a larger imaging device and one does not necessarily use some of the pixels in the region between the two imaging areas.
  • the modules are implemented as software executing on a processor
  • the software is stored in a memory as computer readable instructions and the computer readable instructions are executed on the processor. All or part of the memory can be in a different physical location than a processor so long as the processor can be coupled to the memory.
  • Memory refers to a volatile memory, a non-volatile memory, or any combination of the two.
  • the functions of the various modules, as described herein, may be performed by one unit, or divided up among different components, each of which may be implemented in turn by any combination of hardware, software that is executed on a processor, and firmware.
  • the components When divided up among different components, the components may be centralized in one location or distributed across system 200 for distributed processing purposes.
  • the execution of the various modules results in methods that perform the processes described above for the various modules and controller 260.
  • a processor is coupled to a memory containing instructions executed by the processor. This could be accomplished within a computer system, or alternatively via a connection to another computer via modems and analog lines, or digital interfaces and a digital carrier line.
  • a computer program product comprises a computer readable medium configured to store computer readable code needed for any part of or all of the processes described herein, or in which computer readable code for any part of or all of those processes is stored.
  • Some examples of computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives, servers on a network and signals transmitted over a network representing computer readable program code.
  • a non-transitory tangible computer program product comprises a tangible computer readable medium configured to store computer readable instructions for any part of or all of the processes or in which computer readable instructions for any part of or all of the processes is stored.
  • Non-transitory tangible computer program products are CD-ROM discs, DVD discs, flash memory, ROM cards, floppy discs, magnetic tapes, computer hard drives and other physical storage mediums.
  • first and second are used as adjectives to distinguish between elements and are not intended to indicate a number of elements.
  • top, bottom, and side are used as adjectives to aid in distinguishing between elements as viewed in the drawings, and to help visualize relative relationships between the elements.
  • top and bottom surfaces are first and second surfaces that are opposite and removed from each other.
  • a side surface is a third surface that extends between the first and second surfaces.
  • Top, bottom, and side are not being used to define absolute physical positions.
  • spatially relative terms such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like— may be used to describe one element's or feature's relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the device in use or operation in addition to the position and orientation shown in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features.
  • the exemplary term “below” can encompass both positions and orientations of above and below.
  • the device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
  • descriptions of movement along and around various axes include various special device positions and orientations.
  • An apparatus can comprise: a first image capture sensor comprising a first sensor surface; a second image capture sensor comprising a second sensor surface, wherein the first and second image capture sensor surfaces are coplanar; a beam splitter positioned to receive light, configured to direct a first portion of the received light to the first sensor surface, and configured to pass a second portion of the received light through the beam splitter; and a reflective unit positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor.
  • the first and second image capture sensors can comprise different areas of an image capture sensor chip.
  • the apparatus can comprise an endoscopic comprising a distal end including the first and second image capture sensors, the beam splitter and the reflective unit.
  • the apparatus can further comprise a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, beam splitters, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the beam splitter, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different beam splitter, and a different reflective unit in the plurality.
  • the apparatus can further comprise a prism assembly including the beam splitter, wherein the beam splitter comprises a first surface configured to reflect the first portion of the received light and to pass the second portion of the received light; and/or wherein the prism assembly can further comprise a second surface configured to direct the reflected light from the first surface to the first sensor surface, wherein the second surface is positioned so that no other light hits the second surface; and/or wherein the first surface can comprise a multilayer coated surface; and/or wherein the reflective unit can further comprise a reflective surface positioned to reflect the second portion of the received light to the surface of the second image capture sensor; and/or wherein the first surface can have an angle of incidence smaller than forty five degrees; and/or wherein the prism assembly and the reflective unit comprise a single integral structure (e.g., two or three parts glued together or a pentaprism), wherein the first surface can comprise a multilayer coated surface; and/or wherein the single integral structure can comprise one of two parts glued together and three parts glued together; and
  • the beam splitter can further comprise a coated surface configured to reflect the first portion of the received light and to transmit the second portion of the received light; wherein the first and second portions of the received light can be different or substantially equal first and second percentages, respectively, of the received light.
  • both the first and second image capture sensors can comprise color image capture sensors, or one can comprise a monochrome image capture sensor.
  • An apparatus can comprise: a first image capture sensor comprising a first sensor surface; a second image capture sensor comprising a second sensor surface, wherein the first sensor surface is in a first plane, the second sensor surface is in a second plane, the first and second planes are substantially parallel and are separated by a known distance, a beam splitter positioned to receive light, configured to direct a first portion of the received light to the first sensor surface, and configured to pass a second portion of the received light through the beam splitter; and a reflective unit positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor.
  • An apparatus can comprise: a first image capture sensor comprising a first sensor surface; a second image capture sensor comprising a second sensor surface, a first lens assembly; a second lens assembly; and a reflective unit positioned to receive light that passes through the first lens assembly and to reflect the received light from the first lens assembly unto the first sensor surface, and positioned to receive light that passes through the second lens assembly and to reflect the received light from the second lens assembly unto the second sensor surface, wherein a first optical path length from the first lens assembly to the first sensor surface is about equal to a second optical path length from the second lens assembly to the second sensor surface.
  • the first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • a method can comprise: capturing, in a first image capture sensor of an image capture unit, a first image from a first portion of light received from a common front end optical system; and capturing, in a second image capture sensor of the image capture unit, a second image from a second portion of the light received from the common front end optical system, wherein the first and second image capture sensors are coplanar and the first and second images are spatially registered relative to each other upon the capturings.
  • the method can further comprise separating the received light into the first portion and the second portion by a beam splitter of the image capture unit; directing the first portion of the received light onto the first image capture sensor; and directing the second portion of the light onto the second image capture sensor.
  • An apparatus can comprise: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light, comprising: a distal face through which the received light enters the prism assembly; a beam splitter configured to reflect a first portion of the received light on a basis of a polarization state of the received light, and to transmit a second portion of the received light on the basis of the polarization state of the received light; a surface configured to direct the first portion of the received light onto the first image capture sensor; and a reflective unit positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor, wherein a first optical path length from the distal face to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • the first and second image capture sensor surfaces can be coplanar. In some aspects, the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance. In some aspects, the first and second image capture sensors can comprise different areas of an image capture sensor chip. In some aspects, the apparatus can comprise an endoscope comprising a distal end including the first and second image capture sensors, the prism assembly, and the reflective unit.
  • the apparatus can further comprise a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different prism assembly, and a different reflective unit in the plurality.
  • the beam splitter can comprise a multilayer coated surface.
  • the reflective unit can further comprise a reflective surface positioned to reflect the second portion of the received light and to the second image capture sensor.
  • the prism assembly and the reflective unit can comprise a single integral structure (e.g., two or three parts glued together and/or a pentaprism).
  • the beam splitter can have an angle of incidence smaller than forty five degrees.
  • the apparatus can further comprise a stop positioned distal to the beam splitter.
  • the apparatus can further comprise a quarter wave plate positioned between the stop and the distal face of the prism assembly.
  • the apparatus can further comprise a controller coupled to the first and second image capture sensors, wherein the controller combines a first image captured by the first image capture sensor and a second image captured by the second image capture sensor to generate an image increasing the saliency of a feature in the image based on polarization differences in the received light.
  • the apparatus can further comprise a source of illumination having a polarization, wherein the received light comprises reflected light having a polarization different from the polarization and reflected light having the polarization.
  • the apparatus can further comprise an illumination source of unpolarized illumination, wherein the received light comprises reflected unpolarized light, and reflected polarized light.
  • a method can comprise: capturing, in a first image capture sensor of an image capture unit, a first image from a first polarized portion of light received from a common front end optical system; capturing, in a second image capture sensor of the image capture unit, a second image from a second portion of the light received from the common front end optical system, wherein a first optical path length from the common front end optical system to the first sensor surface is about equal to a second optical path length from the common front end optical system to the second sensor surface.
  • the first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • the method can further comprise separating the received light into the first polarized portion and the second portion based on polarization by a beam splitter of the image capture unit; directing the first polarized portion of the received light to the first image capture sensor by the beam splitter; and directing the second portion of the light to the second image capture sensor by a reflective unit.
  • the method can further comprise combining the first image captured by the first image capture sensor and the second image captured by the second image capture sensor to generate an image increasing the saliency of a feature in the image based on polarization differences in the received light.
  • the method can further comprise illuminating a scene with polarized or unpolarized light.
  • An apparatus can comprise: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light, comprising a distal face through which the received light enters the prism assembly, a beam splitter configured to reflect a first percentage of the received light, and to transmit a second percentage of the received light, and a surface configured to direct the first percentage of the received light onto the first image capture sensor; and a reflective unit positioned to receive the second percentage of the received light and to direct the second percentage of the received light to the second image capture sensor, wherein a first optical path length from the distal face to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • first and second image capture sensors can comprise different areas of an image capture sensor chip.
  • the apparatus can comprise an endoscope comprising a distal end including the first and second image capture sensors, the prism assembly, and the reflective unit.
  • the apparatus can further comprise a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different prism assembly, and a different reflective unit in the plurality.
  • the prism assembly can comprise a pentaprism.
  • the apparatus can further comprise a stop positioned distal to the prism assembly.
  • the first and second image capture sensors can comprise color sensors.
  • the apparatus can further comprise a controller coupled to the first and second image capture sensors, wherein the controller combines information from a first image captured by the first image capture sensor and information from a second image captured by the second image capture sensor to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor: wherein the first and second percentages both about equal, and the controller generated image has enhanced spatial resolution; or wherein the first and second percentages are not about equal, and the controller generated image has enhanced dynamic range.
  • the first percentage and the second percentage can be about equal, wherein the first percentage can be about fifty percent of the received light and the second percentage can be about fifty percent of the received light, wherein the beam splitter and the surface can be positioned to offset an image captured by the first image capture sensor from an image captured by the second image capture sensor with the first optical path length remaining about equal to the second optical path length; and/or wherein the apparatus can further comprise a controller coupled to the first and second image capture sensors and configured to sample a first pixel in a first image captured by the first image capture sensor, sample a second pixel captured by the second image capture sensor and corresponding to the first pixel, and generate a pixel in an image having increased apparent resolution in comparison to the images captured by the first and second image capture sensors.
  • the first percentage and the second percentage are not equal, and the first percentage is based on a dynamic range of the first image capture sensor.
  • the first percentage is about N of the received light and the second percentage is about M of the received light, and N and M are positive numbers, one hundred percent minus N is about equal to M , and the N is selected based on a dynamic range of the first image capture sensor, wherein the apparatus can further comprise a controller coupled to the first and second image capture sensors, and configured to sample a pixel in an image captured by the first image capture sensor, sample corresponding pixels in an image captured by the second image capture sensor, and generate a pixel in an output image using the sampled pixels, wherein the output image has an increased dynamic range relative to an image captured by a single image capture sensor.
  • An apparatus can comprise: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light comprising a plurality of color components, comprising a distal face through which the received light enters the beam splitter, a beam splitter configured to reflect one color component of the plurality of color components and to transmit other color components in the plurality of color components, and a surface configured to direct the one color component onto the first image capture sensor; and a reflective unit positioned to receive the second portion of the received light and to direct the other color components to the second image capture sensor, wherein a first optical path length from the distal face to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • the first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • the first image capture sensor can comprise a monochrome image capture sensor; and the second image capture sensor can comprise an image capture sensor having a color filter array for the other color components in the plurality of color components, wherein the one of the plurality of color components is a green color component and the other of the plurality of color components are a red color component and a blue color component, wherein the color filter array can be a checkerboard for red and blue pixels; and/or wherein the apparatus can further comprise a controller coupled to the first and second image capture sensors and configured to have full spatial resolution in the one of the plurality of color components, to have reduced spatial resolution in the other of the plurality of color components, and to generate an image having improved spatial resolution and sharpness relative to an image captured by a color image capture sensor.
  • a method can comprise: capturing, in a first image capture sensor of an image capture unit, a first image from a first percentage of light received from a common front end optical system; capturing, in a second image capture sensor of the image capture unit, a second image from a second percentage the light received from the common front end optical system, wherein a first optical path length in the image capture unit to the first image capture sensor is about equal to a second optical path length in the image capture unit to the second image capture sensor; and combining information from the first image and information from the second image to generate an image having one of enhanced spatial resolution and enhanced dynamic range relative to an image captured by a single image capture sensor.
  • the first and second image capture sensor surfaces can be coplanar, or the first and second sensor surface can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • An apparatus can comprise: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light, comprising a distal face through which the received light enters the prism assembly, a beam splitter comprising a plurality of notch filters, wherein the plurality of notch filters reflects a first set of light components as a first portion of the received light and passes a second set of light components as a second portion of the received light, and a surface configured to direct the first portion of the received light onto the first image capture sensor; and a reflective unit positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor, wherein a first optical path length from the distal face to the first sensor surface is about equal to a second optical path length from the distal face to the second sensor surface.
  • first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • first and second image capture sensors can comprise different areas of an image capture sensor chip.
  • the apparatus can comprise an endoscope comprising a distal end including the first and second image capture sensors, the prism assembly, and the reflective unit.
  • the apparatus can further comprise: a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different prism assembly, and a different reflective unit in the plurality.
  • the prism assembly can comprise a pentaprism.
  • the apparatus can further comprise a stop positioned distal to the prism assembly.
  • the first and second image capture sensors can comprise color sensors; and/or the apparatus can further comprise an illuminator generating output light including a plurality of color components; and/or the apparatus can further comprise a color correction module coupled to the first and second image capture sensors, configured to receive a demosaiced image of a first image captured by the first image capture sensor, receive a demosaiced image of a second image captured by the second image capture sensor, and generate an N-element color component vector for a pixel in an output image from a color component vector of a corresponding pixel in the first image and a color component vector of a corresponding pixel in the second image, wherein N is at least three.
  • the first and second image capture sensors can comprise color sensors, each color sensor comprising a Bayer red, green, green, blue color filter array; wherein the apparatus can further comprise an illuminator comprising: a first laser illumination source generating a first red color component output illumination; a second laser illumination source generating a second red color component output illumination, wherein the second red color component output illumination is different from the first red color component output illumination; a third laser illumination source generating a first green color component output illumination; a fourth laser illumination source generating a second green color component output illumination, wherein the second green color component output illumination is different from the first green color component output illumination; a fifth laser illumination source generating a first blue color component output illumination; and a sixth laser illumination source generating a second blue color component output illumination, wherein the second blue color component output illumination is different from the first blue color component output illumination.
  • an illuminator comprising: a first laser illumination source generating a first red color component output illumination; a second laser illumination source generating a second red color component output
  • the plurality of notch filters can reflect the first red color component light, the first green color component light, and the first blue color component light as the first portion of the received light, and wherein the plurality of notch filters can pass the second red color component light, the second green color component light, and the second blue color component light as the second portion of the received light.
  • the apparatus can further comprise a controller coupled to the first and second image capture sensors, configured to receive a demosaiced image of a first image captured by the first image capture sensor, receive a demosaiced image of a second image captured by the second image capture sensor, and generate a six element color component vector for a pixel in an output image from a three color component vector of a corresponding pixel in the first image and a three color component vector of a corresponding pixel in the second image; and in some aspects can further comprise a broadband white light illuminator.
  • the plurality of notch filters can reflect a first portion of a red color component, a first portion of a green color component, and a first portion of a blue color component of the received light, and pass a second portion of the red color component, a second portion of the green color component, and a second portion of the blue color component of the received light.
  • the apparatus can further comprise a controller coupled to the first and second image capture sensors, configured to receive a demosaiced image of a first image captured by the first image capture sensor, receive a demosaiced image of a second image captured by the second image capture sensor, and generate an N element color for a pixel in an output image from a three color component vector of a corresponding pixel in the first image and a color component vector of a corresponding pixel in the second image, wherein N is an integer in a range of three to six; wherein the apparatus can further comprise a display coupled to the controller and configured to receive an N element color component vector for each pixel.
  • the apparatus can further comprise a lens system positioned distal to the prism assembly, the lens system configured to focus different wavelengths of light at different distances from the lens system.
  • the apparatus can further comprise a controller coupled to the first and second image capture sensors, configured to configured to apply a digital filter kernel to a first image captured by the first image capture sensor and to a second image captured by the second image capture sensor and generate third and fourth images, demosaic the third and fourth images, and generate an N element color component vector for a pixel in an output image from a color component vector of a corresponding pixel in the demosaiced third image and a color component vector of a corresponding pixel in the demosaiced fourth image, wherein N is an integer in a range of three to six.
  • the apparatus can further comprise an illuminator comprising: a first laser illumination source generating a red color component output illumination; a second laser illumination source generating a green color component output illumination; and a third laser illumination source generating a blue color component output illumination; wherein in some aspects the plurality of notch filters reflects red color component light, green color component light, and blue color component light as the first portion of the received light and passes fluorescence; wherein in some aspects, the apparatus can further comprise a controller coupled to the first and second image capture sensors, configured to generate at least a four element color component vector for a pixel in an output image from a color component vector for a corresponding pixel in the first image and a color component vector for a corresponding pixel in the second image, wherein three of the elements in the four element color component vector are for a visible color image, and a fourth of the elements in the four element color component vector is for a fluorescent image; wherein in some aspects, each of the first and second image capture sensors comprises a color sensor comprising
  • a method can comprise: separating, by a plurality of notch filters of an image capture unit, light received through a common from end optical system, into a first plurality of light components and a second plurality of light components; capturing, in a first image capture sensor of the image capture unit, a first image from the first plurality of light components; capturing, in a second image capture sensor of the image capture unit, a second image from the second plurality of light components, wherein a first optical path length in the image capture unit to the first image capture sensor is about equal to a second optical path length in the image capture unit to the second image capture sensor; and generating an N-element color component vector for a pixel in an output image from a color component vector of a corresponding pixel in the first image and a color component vector of a corresponding pixel in the second image, wherein N is at least three.
  • the first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively,
  • An apparatus can comprise: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light, comprising a distal face through which the received light enters the prism assembly, a beam splitter configured to reflect a first portion of the received light, and to transmit a second portion of the received light, and a surface configured to direct the first portion of the received light onto the first image capture sensor; and a reflective unit positioned to receive the second portion of the received light and to direct the second portion of the received light to the second image capture sensor, wherein a first optical path length from the distal face to the first sensor surface is smaller than a second optical path length from the distal face to the second sensor surface, the first image capture sensor captures an image focused at a first object distance, and the second image capture sensor captures an image focused at a second object distance.
  • first and second image capture sensor surfaces can be coplanar, or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • first and second image capture sensors can comprise different areas of an image capture sensor chip.
  • the apparatus can comprise an endoscope comprising a distal end including the first and second image capture sensors, the prism assembly, and the reflective unit.
  • the apparatus can further comprise a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor, a different second image capture sensor, a different prism assembly, and a different reflective unit in the plurality.
  • the prism assembly comprises a pentaprism.
  • the apparatus can further comprise a stop positioned distal to the prism assembly.
  • a first portion of the received light can be a first percentage of the received light
  • the second portion of the received light can be a second percentage of the received light
  • the first and second percentages are both about equal
  • one hundred percent minus the first percentage is about the second percentage
  • the first and second percentages are not about equal
  • the first percentage is smaller than the second percentage
  • at least one of the first and second image capture sensors comprises a color image capture sensor; in some aspects further comprising a controller coupled to the first and second image capture sensors to receive first and second images, and configured to automatically generate an in focus output image from the first and second images, in some aspects further configured to sample a first group of pixels in the first image, to sample a corresponding second group of pixels in the second image, and to select one of the first group of pixels and the second group of pixels for the output image; in other aspects further comprising a controller coupled to the first and second image capture sensors, configured to retrieve a first image captured by the
  • a method can comprise: capturing, in a first image capture sensor of an image capture unit, a first image from a portion of light received from a common front end optical system; capturing, in a second image capture sensor of the image capture unit, a second image from a second portion the light received from the common front end optical system, wherein a first optical path length in the image capture unit to the first image capture sensor smaller than a second optical path length in the image capture unit to the second image capture sensor; and combining information from the first image and information from the second image to generate an in focus image having an extended depth of field.
  • the first and second image capture sensor surfaces are coplanar, or the first and second sensor surfaces are in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • a method can comprise: capturing, in a first image capture sensor of an image capture unit, a first image from a portion of light received from a common front end optical system; capturing, in a second image capture sensor of the image capture unit, a second image from a second portion the light received from the common front end optical system, wherein a first optical path length in the image capture unit to the first image capture sensor smaller than a second optical path length in the image capture unit to the second image capture sensor; and combining information from the first image and information from the second image to generate an image from a virtual camera view point.
  • the first and second image capture sensor surfaces can be coplanar or the first and second sensor surfaces can be in first and second planes, respectively, that are substantially parallel and are separated by a known distance.
  • the method can further comprise displaying, on a non-stereo display unit, images of a scene from a plurality of virtual camera view points so that the displayed scene rocks back and forth over time.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Public Health (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Veterinary Medicine (AREA)
  • Radiology & Medical Imaging (AREA)
  • Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Biophysics (AREA)
  • Signal Processing (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Gynecology & Obstetrics (AREA)
  • Endoscopes (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Astronomy & Astrophysics (AREA)
  • Instruments For Viewing The Inside Of Hollow Bodies (AREA)
  • Studio Devices (AREA)
  • Stroboscope Apparatuses (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Structure And Mechanism Of Cameras (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
PCT/US2012/050394 2011-08-12 2012-08-10 An image capture unit in a surgical instrument WO2013025530A1 (en)

Priority Applications (11)

Application Number Priority Date Filing Date Title
KR1020237015005A KR20230062898A (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
JP2014525180A JP6258201B2 (ja) 2011-08-12 2012-08-10 外科手術器具の画像取込み装置
KR1020147006103A KR101971211B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
CN201280050585.7A CN103889353B (zh) 2011-08-12 2012-08-10 外科手术器械中的图像捕获单元
EP12824330.0A EP2741702B1 (en) 2011-08-12 2012-08-10 An image capture unit in a surgical instrument
KR1020197010909A KR102079689B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
CN202011093468.1A CN112220438B (zh) 2011-08-12 2012-08-10 外科手术器械中的图像捕获单元
KR1020227012984A KR102529996B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
KR1020217020786A KR102390141B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
KR1020207029014A KR102274896B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치
KR1020207004427A KR102166302B1 (ko) 2011-08-12 2012-08-10 수술기기 내의 이미지 캡쳐 장치

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
US13/209,041 2011-08-12
US13/209,121 US8684914B2 (en) 2011-08-12 2011-08-12 Image capture unit and an imaging pipeline with enhanced color performance in a surgical instrument and method
US13/209,084 2011-08-12
US13/209,158 US8784301B2 (en) 2011-08-12 2011-08-12 Image capture unit and method with an extended depth of field
US13/209,158 2011-08-12
US13/209,121 2011-08-12
US13/209,084 US8734328B2 (en) 2011-08-12 2011-08-12 Increased resolution and dynamic range image capture unit in a surgical instrument and method
US13/208,982 US8672838B2 (en) 2011-08-12 2011-08-12 Image capture unit in a surgical instrument
US13/208,982 2011-08-12
US13/209,041 US8764633B2 (en) 2011-08-12 2011-08-12 Feature differentiation image capture unit and method in a surgical instrument

Publications (1)

Publication Number Publication Date
WO2013025530A1 true WO2013025530A1 (en) 2013-02-21

Family

ID=47715395

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2012/050394 WO2013025530A1 (en) 2011-08-12 2012-08-10 An image capture unit in a surgical instrument

Country Status (5)

Country Link
EP (1) EP2741702B1 (ja)
JP (4) JP6258201B2 (ja)
KR (7) KR102274896B1 (ja)
CN (3) CN112220438B (ja)
WO (1) WO2013025530A1 (ja)

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014170194A1 (de) * 2013-04-17 2014-10-23 Siemens Aktiengesellschaft Verfahren und vorrichtung zur stereoskopischen darstellung von bilddaten
WO2015176298A1 (en) 2014-05-23 2015-11-26 Covidien Lp 3d laparoscopic image capture apparatus with a single image sensor
WO2016018616A1 (en) * 2014-07-30 2016-02-04 Novartis Ag Vital stain visualization in ophthalmic surgical procedures and associated devices, systems, and methods
CN105611276A (zh) * 2015-12-22 2016-05-25 广州广康医疗科技有限公司 一种与达芬奇机器人配套使用的高清3d录直播系统
US9462251B2 (en) 2014-01-02 2016-10-04 Industrial Technology Research Institute Depth map aligning method and system
EP3156761A1 (en) * 2015-10-14 2017-04-19 H4 Technologies Limited Image acquisition and processing
EP3254606A1 (en) * 2016-06-07 2017-12-13 Karl Storz GmbH & Co. KG Endoscope and imaging arrangement providing depth of field
CN108896545A (zh) * 2018-05-09 2018-11-27 歌尔股份有限公司 涂胶检测方法、装置及计算机可读存储介质
WO2019048492A1 (en) * 2017-09-08 2019-03-14 Sony Corporation IMAGING DEVICE, METHOD AND PROGRAM FOR PRODUCING IMAGES OF A SCENE
US10390688B2 (en) 2014-11-21 2019-08-27 Olympus Corporation Image pickup system
EP3557531A1 (en) * 2018-04-18 2019-10-23 Vision RT Limited Camera monitoring system for monitoring a patient in a bore based medical system
CN110709001A (zh) * 2017-06-12 2020-01-17 索尼公司 医疗系统、医疗设备和控制方法
EP3605199A1 (en) * 2018-08-01 2020-02-05 HTC Corporation Head-mounted display and imaging apparatus for displaying image thereof
WO2021176442A1 (en) * 2020-03-06 2021-09-10 270 Surgical Ltd. A multi focal endoscope
US11163169B2 (en) 2016-06-07 2021-11-02 Karl Storz Se & Co. Kg Endoscope and imaging arrangement providing improved depth of field and resolution
WO2021231508A1 (en) * 2020-05-12 2021-11-18 Covidien Lp Systems and methods for image mapping and fusion during surgical procedures
WO2022023157A1 (en) * 2020-07-30 2022-02-03 Interdigital Ce Patent Holdings, Sas Debayering with multiple camera rotations
US11307430B2 (en) 2016-06-07 2022-04-19 Karl Storz Se & Co. Kg Optical device and method for providing improved depth of field and resolution modes
US11375882B2 (en) 2017-06-07 2022-07-05 Olympus Corporation Endoscope system which generates an image of high dynamic range
US11463605B2 (en) 2016-02-12 2022-10-04 Contrast, Inc. Devices and methods for high dynamic range video
WO2022243994A1 (en) * 2021-05-19 2022-11-24 270 Surgical Ltd. Distal tip of a multi camera medical imaging device
US11637974B2 (en) 2016-02-12 2023-04-25 Contrast, Inc. Systems and methods for HDR video capture with a mobile device
DE102022105090B3 (de) 2022-03-03 2023-06-29 Schölly Fiberoptic GmbH Stereoskopische Anordnung und Operationsmikroskop mit stereoskopischer Anordnung
EP3986232A4 (en) * 2019-06-20 2023-07-05 Cilag GmbH International OFFSET LIGHTING OF A SCENE WITH MULTIPLE EMITTERS IN A LASER MAPPING IMAGING SYSTEM
US11789283B2 (en) 2019-08-22 2023-10-17 Canon Kabushiki Kaisha Imaging apparatus
EP4061199A4 (en) * 2019-11-22 2023-11-29 Convergascent LLC USE OF MULTIPLE IMAGERS AND COMPUTER-ASSISTED PHOTOGRAPHY IN ENDOSCOPY
US11910099B2 (en) 2016-08-09 2024-02-20 Contrast, Inc. Real-time HDR video for vehicle control
US11985316B2 (en) 2018-06-04 2024-05-14 Contrast, Inc. Compressed high dynamic range video
US12102488B2 (en) 2018-01-31 2024-10-01 Intuitive Surgical Operations, Inc. Display with folded optical path

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6340084B2 (ja) * 2014-03-21 2018-06-06 ハイパーメツド・イメージング・インコーポレイテツド 小型光センサ
TWI561773B (en) * 2014-08-06 2016-12-11 Delta Electronics Inc Six-primary solid state illuminator and operating method using the same
WO2016059906A1 (ja) * 2014-10-16 2016-04-21 オリンパス株式会社 内視鏡装置
DE102015217253A1 (de) * 2015-09-10 2017-03-16 Robert Bosch Gmbh Umfelderfassungseinrichtung für ein Fahrzeug und Verfahren zum Erfassen eines Bilds mittels einer Umfelderfassungseinrichtung
JP2017148432A (ja) * 2016-02-26 2017-08-31 富士フイルム株式会社 内視鏡用光源装置
US10992917B2 (en) * 2016-06-17 2021-04-27 Sony Corporation Image processing device, image processing method, program, and image processing system that use parallax information
US11782256B2 (en) * 2016-09-21 2023-10-10 Omnivision Technologies, Inc. Endoscope imager and associated method
NL2017973B1 (en) * 2016-12-09 2018-06-19 Quest Photonic Devices B V Dichroic prism assembly with four or five channels
WO2018173412A1 (ja) 2017-03-24 2018-09-27 オリンパス株式会社 内視鏡システム
CN108803228B (zh) * 2018-06-15 2021-01-08 郭洪志 一种仿生摄像头三维立体成像系统和方法
CN109259717B (zh) * 2018-08-27 2020-08-14 彭波 一种立体内窥镜及内窥镜测量方法
DE102018122816B9 (de) * 2018-09-18 2024-07-04 Carl Zeiss Meditec Ag Verfahren und Vorrichtung zum Bestimmen einer Eigenschaft eines Objekts
CN111474140A (zh) * 2020-03-23 2020-07-31 江苏大学 一种双通道正交相位显微成像采样系统
CN111766699A (zh) * 2020-07-24 2020-10-13 重庆金山科技（集团）有限公司 成像装置及方法、内窥镜系统
US20230308628A1 (en) * 2020-08-31 2023-09-28 Sony Group Corporation Medical imaging system, medical imaging device, and operation method
US11808935B2 (en) 2021-10-12 2023-11-07 Olympus Corporation Endoscope and endoscope apparatus

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4873572A (en) 1987-02-27 1989-10-10 Olympus Optical Co., Ltd. Electronic endoscope apparatus
US5743846A (en) * 1994-03-17 1998-04-28 Olympus Optical Co., Ltd. Stereoscopic endoscope objective lens system having a plurality of front lens groups and one common rear lens group
US6331181B1 (en) 1998-12-08 2001-12-18 Intuitive Surgical, Inc. Surgical robotic tools, data architecture, and use
US20030083551A1 (en) 2001-10-31 2003-05-01 Susumu Takahashi Optical observation device and 3-D image input optical system therefor
US6837883B2 (en) 1998-11-20 2005-01-04 Intuitive Surgical, Inc. Arm cart for telerobotic surgical system
WO2008079578A2 (en) * 2006-12-21 2008-07-03 Intuitive Surgical, Inc. Stereoscopic endoscope
US20080228037A1 (en) 2000-07-14 2008-09-18 Novadaq Technologies Inc. Compact fluorescence endoscopy video system
US20100165080A1 (en) * 2008-12-26 2010-07-01 Fujifilm Corporation Image capturing apparatus and endoscope
US20100245550A1 (en) 2009-03-24 2010-09-30 Olympus Corporation Fluoroscopy apparatus and fluoroscopy method

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4074306A (en) * 1975-07-28 1978-02-14 Olympus Optical Co., Ltd. Endoscope utilizing color television and fiber optics techniques
JPS62174715A (ja) * 1987-01-24 1987-07-31 Olympus Optical Co Ltd 内視鏡装置
JPH0820603B2 (ja) * 1987-02-27 1996-03-04 オリンパス光学工業株式会社 ビデオスコ−プ装置
JPH01122274A (ja) * 1987-11-06 1989-05-15 Fujitsu Ltd 撮像装置
JP3093875B2 (ja) * 1992-06-09 2000-10-03 オリンパス光学工業株式会社 立体視内視鏡
JP3628717B2 (ja) * 1994-03-17 2005-03-16 オリンパス株式会社 立体視内視鏡
JPH10118004A (ja) * 1996-10-24 1998-05-12 Fuji Photo Film Co Ltd 蛍光撮像装置
JP2000221411A (ja) * 1999-01-29 2000-08-11 Fuji Photo Optical Co Ltd 偽似カラー表示可能な電子内視鏡装置
IL135571A0 (en) * 2000-04-10 2001-05-20 Doron Adler Minimal invasive surgery imaging system
JP5259033B2 (ja) * 2001-08-03 2013-08-07 オリンパス株式会社 内視鏡システム
US20070182844A1 (en) 2003-03-09 2007-08-09 Latia Imaging Pty Ltd Optical system for producing differently focused images
JP2004313523A (ja) 2003-04-17 2004-11-11 Pentax Corp 固体撮像素子、電子内視鏡
JP2005176940A (ja) * 2003-12-16 2005-07-07 Olympus Corp 電子内視鏡
JP2005182521A (ja) 2003-12-19 2005-07-07 Matsushita Electric Ind Co Ltd 虹彩撮像カメラおよび虹彩認証システム
JP2005244549A (ja) 2004-02-26 2005-09-08 Matsushita Electric Ind Co Ltd 認証用画像撮像装置
US20060241495A1 (en) * 2005-03-23 2006-10-26 Eastman Kodak Company Wound healing monitoring and treatment
JP5086535B2 (ja) * 2005-11-21 2012-11-28 オリンパスメディカルシステムズ株式会社 ２板撮像装置
DE102006034205B4 (de) * 2006-07-25 2012-03-01 Carl Mahr Holding Gmbh Dynamische Bildaufnahme mit bildgebenden Sensoren
JP5448353B2 (ja) * 2007-05-02 2014-03-19 キヤノン株式会社 光干渉断層計を用いた画像形成方法、及び光干渉断層装置
US20090131800A1 (en) * 2007-11-15 2009-05-21 Carestream Health, Inc. Multimodal imaging system for tissue imaging
JP5631299B2 (ja) 2008-03-28 2014-11-26 コントラスト オプティカル デザイン アンド エンジニアリング，インク． 全ビーム画像スプリッタシステム
JP2010082040A (ja) 2008-09-30 2010-04-15 Fujifilm Corp 内視鏡システム
JP5358368B2 (ja) * 2009-09-18 2013-12-04 富士フイルム株式会社 内視鏡システム

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4873572A (en) 1987-02-27 1989-10-10 Olympus Optical Co., Ltd. Electronic endoscope apparatus
US5743846A (en) * 1994-03-17 1998-04-28 Olympus Optical Co., Ltd. Stereoscopic endoscope objective lens system having a plurality of front lens groups and one common rear lens group
US6837883B2 (en) 1998-11-20 2005-01-04 Intuitive Surgical, Inc. Arm cart for telerobotic surgical system
US6331181B1 (en) 1998-12-08 2001-12-18 Intuitive Surgical, Inc. Surgical robotic tools, data architecture, and use
US20080228037A1 (en) 2000-07-14 2008-09-18 Novadaq Technologies Inc. Compact fluorescence endoscopy video system
US20030083551A1 (en) 2001-10-31 2003-05-01 Susumu Takahashi Optical observation device and 3-D image input optical system therefor
WO2008079578A2 (en) * 2006-12-21 2008-07-03 Intuitive Surgical, Inc. Stereoscopic endoscope
US20100165080A1 (en) * 2008-12-26 2010-07-01 Fujifilm Corporation Image capturing apparatus and endoscope
US20100245550A1 (en) 2009-03-24 2010-09-30 Olympus Corporation Fluoroscopy apparatus and fluoroscopy method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP2741702A4

Cited By (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101772187B1 (ko) * 2013-04-17 2017-08-25 지멘스 악티엔게젤샤프트 이미지 데이터의 입체적 묘사를 위한 방법 및 디바이스
CN105208909A (zh) * 2013-04-17 2015-12-30 西门子公司 用于立体显示图像数据的方法和装置
WO2014170194A1 (de) * 2013-04-17 2014-10-23 Siemens Aktiengesellschaft Verfahren und vorrichtung zur stereoskopischen darstellung von bilddaten
US9462251B2 (en) 2014-01-02 2016-10-04 Industrial Technology Research Institute Depth map aligning method and system
WO2015176298A1 (en) 2014-05-23 2015-11-26 Covidien Lp 3d laparoscopic image capture apparatus with a single image sensor
EP3145383A4 (en) * 2014-05-23 2018-02-21 Covidien LP 3d laparoscopic image capture apparatus with a single image sensor
WO2016018616A1 (en) * 2014-07-30 2016-02-04 Novartis Ag Vital stain visualization in ophthalmic surgical procedures and associated devices, systems, and methods
US10390688B2 (en) 2014-11-21 2019-08-27 Olympus Corporation Image pickup system
EP3156761A1 (en) * 2015-10-14 2017-04-19 H4 Technologies Limited Image acquisition and processing
CN105611276B (zh) * 2015-12-22 2017-12-26 广州广康医疗科技股份有限公司 一种与达芬奇机器人配套使用的高清3d录直播系统
CN105611276A (zh) * 2015-12-22 2016-05-25 广州广康医疗科技有限公司 一种与达芬奇机器人配套使用的高清3d录直播系统
US11637974B2 (en) 2016-02-12 2023-04-25 Contrast, Inc. Systems and methods for HDR video capture with a mobile device
US11463605B2 (en) 2016-02-12 2022-10-04 Contrast, Inc. Devices and methods for high dynamic range video
US11785170B2 (en) 2016-02-12 2023-10-10 Contrast, Inc. Combined HDR/LDR video streaming
US11307430B2 (en) 2016-06-07 2022-04-19 Karl Storz Se & Co. Kg Optical device and method for providing improved depth of field and resolution modes
US10324300B2 (en) 2016-06-07 2019-06-18 Karl Storz Se & Co. Kg Endoscope and imaging arrangement providing depth of field
US11163169B2 (en) 2016-06-07 2021-11-02 Karl Storz Se & Co. Kg Endoscope and imaging arrangement providing improved depth of field and resolution
EP3254606A1 (en) * 2016-06-07 2017-12-13 Karl Storz GmbH & Co. KG Endoscope and imaging arrangement providing depth of field
US11910099B2 (en) 2016-08-09 2024-02-20 Contrast, Inc. Real-time HDR video for vehicle control
US11375882B2 (en) 2017-06-07 2022-07-05 Olympus Corporation Endoscope system which generates an image of high dynamic range
US11463629B2 (en) 2017-06-12 2022-10-04 Sony Corporation Medical system, medical apparatus, and control method
EP3639725A4 (en) * 2017-06-12 2020-06-10 Sony Corporation MEDICAL SYSTEM, MEDICAL DEVICE AND CONTROL METHOD
CN110709001A (zh) * 2017-06-12 2020-01-17 索尼公司 医疗系统、医疗设备和控制方法
CN110709001B (zh) * 2017-06-12 2022-09-23 索尼公司 医疗系统、医疗设备和控制方法
US11699215B2 (en) 2017-09-08 2023-07-11 Sony Corporation Imaging device, method and program for producing images of a scene having an extended depth of field with good contrast
WO2019048492A1 (en) * 2017-09-08 2019-03-14 Sony Corporation IMAGING DEVICE, METHOD AND PROGRAM FOR PRODUCING IMAGES OF A SCENE
US12102488B2 (en) 2018-01-31 2024-10-01 Intuitive Surgical Operations, Inc. Display with folded optical path
JP7326009B2 (ja) 2018-04-18 2023-08-15 ビジョン アールティ リミテッド ボアベースの医療システムにおいて患者を監視するためのカメラ監視システム及びカメラ監視システムを較正する方法
JP2019188137A (ja) * 2018-04-18 2019-10-31 ビジョン アールティ リミテッド ボアベースの医療システムにおいて患者を監視するためのカメラ監視システム及びカメラ監視システムを較正する方法
EP3557531A1 (en) * 2018-04-18 2019-10-23 Vision RT Limited Camera monitoring system for monitoring a patient in a bore based medical system
US11612762B2 (en) 2018-04-18 2023-03-28 Vision Rt Limited 3D stereoscopic camera monitoring system and method of calibrating a camera monitoring system for monitoring a patient in a bore of a medical system for radiation treatment
CN108896545A (zh) * 2018-05-09 2018-11-27 歌尔股份有限公司 涂胶检测方法、装置及计算机可读存储介质
US11985316B2 (en) 2018-06-04 2024-05-14 Contrast, Inc. Compressed high dynamic range video
US10884263B2 (en) 2018-08-01 2021-01-05 Htc Corporation Head-mounted display and imaging apparatus for displaying image thereof
EP3605199A1 (en) * 2018-08-01 2020-02-05 HTC Corporation Head-mounted display and imaging apparatus for displaying image thereof
EP3986232A4 (en) * 2019-06-20 2023-07-05 Cilag GmbH International OFFSET LIGHTING OF A SCENE WITH MULTIPLE EMITTERS IN A LASER MAPPING IMAGING SYSTEM
US11974860B2 (en) 2019-06-20 2024-05-07 Cilag Gmbh International Offset illumination of a scene using multiple emitters in a hyperspectral, fluorescence, and laser mapping imaging system
US11789283B2 (en) 2019-08-22 2023-10-17 Canon Kabushiki Kaisha Imaging apparatus
EP4061199A4 (en) * 2019-11-22 2023-11-29 Convergascent LLC USE OF MULTIPLE IMAGERS AND COMPUTER-ASSISTED PHOTOGRAPHY IN ENDOSCOPY
WO2021176442A1 (en) * 2020-03-06 2021-09-10 270 Surgical Ltd. A multi focal endoscope
WO2021231508A1 (en) * 2020-05-12 2021-11-18 Covidien Lp Systems and methods for image mapping and fusion during surgical procedures
WO2022023157A1 (en) * 2020-07-30 2022-02-03 Interdigital Ce Patent Holdings, Sas Debayering with multiple camera rotations
WO2022243994A1 (en) * 2021-05-19 2022-11-24 270 Surgical Ltd. Distal tip of a multi camera medical imaging device
DE102022105090B3 (de) 2022-03-03 2023-06-29 Schölly Fiberoptic GmbH Stereoskopische Anordnung und Operationsmikroskop mit stereoskopischer Anordnung

Also Published As

Publication number Publication date
JP6258201B2 (ja) 2018-01-10
KR20220057635A (ko) 2022-05-09
CN107913053A (zh) 2018-04-17
KR20200019779A (ko) 2020-02-24
KR20140050712A (ko) 2014-04-29
CN107913053B (zh) 2020-10-30
KR102390141B1 (ko) 2022-04-25
KR20210087564A (ko) 2021-07-12
JP2014524290A (ja) 2014-09-22
KR20190047066A (ko) 2019-05-07
JP7169725B2 (ja) 2022-11-11
JP2019072557A (ja) 2019-05-16
JP6480508B2 (ja) 2019-03-13
EP2741702A4 (en) 2016-02-24
KR102274896B1 (ko) 2021-07-09
CN103889353B (zh) 2017-05-03
EP2741702B1 (en) 2022-10-19
KR101971211B1 (ko) 2019-04-23
CN103889353A (zh) 2014-06-25
EP2741702A1 (en) 2014-06-18
JP2020168569A (ja) 2020-10-15
KR102166302B1 (ko) 2020-10-16
JP2017148657A (ja) 2017-08-31
KR20230062898A (ko) 2023-05-09
CN112220438A (zh) 2021-01-15
JP6740401B2 (ja) 2020-08-12
KR20200119901A (ko) 2020-10-20
KR102529996B1 (ko) 2023-05-08
CN112220438B (zh) 2024-02-06
KR102079689B1 (ko) 2020-02-20

Similar Documents

Publication Publication Date Title
JP7169725B2 (ja) 外科手術器具の画像取込み装置
US20210204803A1 (en) Image capture unit in a surgical instrument
US20210103137A1 (en) Increased resolution and dynamic range capture unit in a surgical instrument and method
US9782056B2 (en) Image capture unit and method with an extended depth of field
US9254076B2 (en) Feature differentiation image capture unit and method in a surgical instrument
US8684914B2 (en) Image capture unit and an imaging pipeline with enhanced color performance in a surgical instrument and method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12824330

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2014525180

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20147006103

Country of ref document: KR

Kind code of ref document: A