JP2017511176A - Surgical system including a general illumination device for non-white light - Google Patents

Abstract

The non-white light from the endoscope (201) of the remotely operated surgical system (200) is used to illuminate the surgical site (203). The camera (220L) captures an image of the surgical site and displays the image on the monitor (251). Non-white light illumination minimizes noise in the image of the surgical site presented on the monitor as compared to the image captured using white light illumination and displayed on the monitor.

Description

RELATED APPLICATIONS This application is priority to US patent application Ser. No. 61 / 954,512 (disclosing “Surgical System Including a Non-White Light General Illuminator” filed on Mar. 17, 2014 by Jeffrey DiCarlo et al.). And claims its benefit, which is incorporated herein by reference.

  Aspects of the present invention relate to endoscopic imaging, and more specifically to non-white light used for general illumination of teleoperated surgical systems.

  Commercialized by Sunnyvale, California's Intuitive Surgical, Inc., the da Vinci (R) surgical system provides patients with many benefits such as reduced trauma, faster recovery and shorter hospital stays A minimally invasive teleoperated surgical system. One feature of the da Vinci® surgical system is the ability to provide a two-channel (ie, left and right) video capture and display a visualized image to provide the surgeon with stereoscopic viewing.

  Such an electronic stereoscopic imaging system can output high resolution video images to the surgeon, and the surgeon not only identifies specific tissue types and characteristics, but also works with improved accuracy. Allows functions such as zoom that provide an “enlarged” view that allows you to do. However, in a typical surgical field, the image quality captured by the camera of an electronic stereoscopic imaging system is limited by the signal-to-noise ratio of the camera.

  As the camera collects light, the captured light is converted to electrons and accumulated in the wells of the image sensor. There is one well per pixel. FIG. 1 is a schematic diagram of a well 101 for a red pixel R, a well 102 for a green pixel G, and a well 103 for a blue pixel B. If the camera collects more electrons in the well, the signal is multiplied while the noise remains relatively constant, thereby increasing the signal-to-noise ratio, i.e., the signal captured in the well Increase against.

  The physical characteristic of light capture by a camera is that if the camera's pixels capture more light, the camera can better estimate the speed at which the light was captured. However, if a camera pixel collects too much light and overfills the well, the pixel signal is lost and is no longer valid. Therefore, the exposure time of the camera tries to collect light so as to fill all of the electron wells 101, 102, 103 as high as possible without overfilling any of the electron wells 101, 102, 103. Is set as follows.

  In a typical surgical site scene illuminated with white light for general viewing, red is the dominant color in the scene captured by the camera. This is because most of the reflected light has a red spectrum compared to the blue and green spectra.

  Typically, color video cameras used in teleoperated surgical systems include a color filter array. The color filter array is a mosaic of different color filters. Ideally, each different color filter passes only a portion of the visible electromagnetic spectrum corresponding to a particular color spectrum, for example, the first set of filters in the color filter array primarily transmits red light. The second set of filters mainly passes green light, and the third set of filters mainly passes blue light.

  The camera includes an image sensor having pixels that capture light passing through the color filter array. Each pixel is a well that is filled with electrons when it captures light. The set of camera pixels that captures light passing through the first set of filters is included in the first color channel of the camera. The set of camera pixels that capture light passing through the second set of filters is included in the second color channel of the camera. The set of camera pixels that captures light passing through the third set of filters is included in the third color channel of the camera.

  As known to those skilled in the art, in one example, white light illumination is a red spectrum light, a green spectrum light, and a blue spectrum that appear white to the human eye with normal color perception. It consists of a combination of light. However, due to the dominant reflection of red spectral light by the surgical site, the red pixel well 101 (FIG. 1) is typically filled much faster than either the green pixel well 102 or the blue pixel well 103. The To prevent the red pixel well 101 from overflowing, the camera exposure is set to limit the light collected so that the red pixel well 101 does not overflow.

  When the color channel well that receives most of the light just overflows, stopping the light collection results in the fact that the other color channel wells may not be sufficient as shown in FIG. Become. In the example of FIG. 1, the green well 102 and the blue well 103 are less than 50% of the total when light collection stops. The signal-to-noise ratio of these non-perfect color channels is significantly smaller than the signal-to-noise ratio of the color channel (s) just trying to overflow. Again, for the example of FIG. 1, the signal to noise ratio for the red channel is about 6, while the signal to noise ratio for each of the green and blue channels is about 3.

  The camera has a worse signal-to-noise ratio performance when all of the camera color channel wells 101, 102, 103 are not filled. Signals from wells 102 and 103 that are not perfect need to have a gain applied to the signal as part of the white balance stage of the surgical system's image processing to form an image for display. White balance adjustment is necessary to ensure that the white surface appears white on the display monitor when the camera captures an image of the white surface. White balance adjustment amplifies color channels that are not perfect (blue and green color channels in FIG. 1), for example, when the camera captures a white image, all color channels have the same value. Configured by applying digital gain. The amplification of these non-perfect well signals increases the noise of these non-perfect well color signals relative to other color signals, and further increases the noise of the final image.

  In one aspect, non-white light from an endoscope of a teleoperated surgical system is used to illuminate the surgical site. The camera captures an image of the surgical site and the image is displayed on the monitor. Non-white light illumination minimizes the noise of the surgical site image presented on the monitor relative to the image captured using white light illumination and displayed on the monitor.

  The color of the light used to illuminate the surgical site is non-white light, for example light having a purple hue, but the image displayed on the monitor does not contain this hue. For the viewer, the light that illuminates the surgical site appears white when viewed on a monitor. Only when the endoscope is removed from the patient, the light emitted from the endoscope is directly viewed as if the viewer is looking at non-white light. Non-white light is different from the combination of, for example, only two narrow spectrum light sources that are used for general illumination and used to highlight specific anatomical structures.

  In one aspect, the device includes a camera and a lighting device. The camera is configured to separate light incident on the camera into a set of pixels. Each set of pixel sets is in a different color channel of the camera. In one aspect, the camera includes a color filter array. The color filter array is configured to separate light incident on the camera into a set of pixels.

  The illuminator is configured such that the illuminator outputs non-white light, whereby each camera color channel has a substantially equal response to non-white light reflected from a perfect reflecting surface. As used herein, a perfect reflective surface is a surface that has a response to the illumination spectrum that is spectrally uniform and equally attenuated throughout the illumination spectrum. As used herein, “substantially equal” or “substantially equal” means that the response is a difference in the reflective properties of the reflective surface (eg, the reflective surface is perfectly accurate anywhere on the surface). May not be exactly equal due to the normal differences in the response of the image sensor's electron well, but the response may be within the combined tolerance of the image sensor and the surface. Means equal.

  In one aspect, the lighting device includes a plurality of color component illumination sources. The illumination source of the plurality of color components is configured such that the illumination device outputs non-white light.

  In another aspect, the apparatus also includes a controller coupled to the illumination source of the plurality of color components. The control device is configured to weight each output of the plurality of color component illumination sources such that the combination of the output of the plurality of color component illumination sources is non-white light.

  In yet another aspect, the device includes a camera, a lighting device, and a control device. The camera is configured to separate light incident on the camera into color components. Color components are captured by a camera as a set of pixels. Each set of pixel sets is in a different camera color channel. In one aspect, the camera includes a color filter array. The color filter array is configured to separate light incident on the camera into a set of pixels.

  The control device is coupled to the lighting device. The controller is configured to adjust the characteristics of the light output by the illuminator to increase the signal-to-noise ratio of one camera color channel pixel for a color image captured by the camera. .

  In one aspect, the illumination device has a plurality of color component illumination sources. The controller is coupled to a plurality of color component illumination sources. The controller is configured to adjust characteristics of at least one of the plurality of color component illumination sources to increase the signal-to-noise ratio of the pixels of one camera color channel.

  In one embodiment, the illumination source for the plurality of color components is a plurality of light emitting diodes. In another aspect, the illumination source of the plurality of color components is a plurality of laser diodes.

  In one aspect, the controller is configured to control the output of the lighting source of the plurality of color components of the lighting device such that the lighting device outputs non-white, whereby each camera color channel is a complete It has approximately equal response to non-white light reflected from the reflecting surface. In another aspect, the control device is configured to change the illumination level of at least one of the illumination sources of the plurality of color components of the lighting device such that the lighting device outputs non-white light. In another aspect, a fixed filter is used to change the illuminance level of at least one of the illumination sources of the color components of the lighting device such that the lighting device outputs non-white light. In yet another aspect, illuminance level variations that produce non-white light are controlled to adjust for unequal power loss of aging induced by the lifetime of the illumination source of the plurality of color components.

  In another aspect, the apparatus includes an image processing pipeline configured to form a high dynamic range image from a monochromatic image captured by a camera. In this aspect, the controller is configured to change the illuminance level of at least one of the illumination sources of the plurality of color components of the illumination device. Changing the illumination level can be performed using, for example, a rotating wheel or a liquid crystal device.

  The rotating wheel has a plurality of sections. Each of the plurality of sections has a color of the plurality of color illumination components, and each of the plurality of sections has a different light attenuation level.

  In one aspect, the liquid crystal device is set to an on / off pulse width modulated shutter mode and the variable ratio of on / off times includes one or more on / off cycles per camera image frame capture. In another aspect, the liquid crystal device is configured as an adjustable attenuator. In yet another aspect, the liquid crystal device is configured as a wavelength tunable filter.

  The method includes illuminating the scene with non-white light. Non-white light is configured such that each camera color channel of the camera has approximately equal response to non-white light reflected from a perfectly reflective surface. The method also includes capturing a scene image with a camera and outputting a display image based on the captured image without adjusting the white balance of the captured image.

  Another method includes capturing a color image. The captured color image includes a set of pixels. Each set of pixel sets is in a different color channel of the camera. The method also includes constructing a high dynamic range image from the set of pixels.

  Yet another method includes illuminating the site with non-white light. Non-white light is configured such that each camera color channel of the camera has a substantially equal response to non-white light reflected from the site. This method includes a step of capturing an image of a part with a camera and a step of outputting an image for display based on the captured image.

It is a figure which shows the filling amount of the prior art of the electronic well of an image sensor about the color component of the image which took in the scene illuminated with white light. 1 is a block diagram of a part of a remotely operated surgical system including a lighting device that outputs non-white light. It is a figure which shows the example of the response function of the camera of FIG. It is a graph of the power spectrum of each light emitting diode in 1 aspect of the illuminating device of FIG. It is a graph which shows the comparison with the white light from the illuminating device of FIG. 2, and non-white light. FIG. 6 is a time-series schematic diagram for generating dynamic color channel lighting control factors. It is a figure which shows the filling amount of the electronic well of an image sensor about the color component of the image which took in the scene illuminated with the non-white light. It is a figure which shows the filling amount of the electronic well of an image sensor about the color component of the image which took in the scene illuminated with the non-white light. It is a figure which shows the filling amount of the electronic well of an image sensor about the color component of the image which took in the scene illuminated with the non-white light. FIG. 3 is a diagram illustrating a method of generating non-white light by relatively different on / off times over a period of output of illumination sources of various color components from the illumination device of FIG. 2. is there.

  In the drawings, the first digit of a reference number indicates the figure in which the element containing the reference number first appears.

  As used herein, electronic stereoscopic imaging includes the use of two imaging channels (ie, channels for left and right images).

  As used herein, a stereoscopic light path includes two channels (eg, channels for left and right images) within an endoscope to carry light from tissue. The light carried in each channel represents a different view of the tissue. The light can form one or more images. Without loss of generality or applicability, the aspects described more fully below can also be used in the context of a field sequential stereoscopic acquisition system and / or a field sequential display system.

  As used herein, an illumination path includes a path within an endoscope that provides illumination to tissue.

  As used herein, white light is composed of three (or more) visible color components, eg, a red visible color component, a green visible color component, and a blue visible color component. Visible white light. White light can also refer to a plurality of continuous spectra within the visible spectrum, as can be seen by a human, for example, from a heated tungsten filament.

  As used herein, non-white light means three (or more) visible color components, eg, a red visible color component in a different combination than the combination used to form white light. , Visible light composed of a green visible color component and a blue visible color component. Non-white light can also refer to a broad spectrum of wavelengths of the visible electromagnetic spectrum, including multiple continuous spectra of the visible electromagnetic spectrum, for example, visible spectrum of multiple colors that are not visible to human observers as white light. it can. Non-white light is a combination of two narrow-spectrum-only light sources, such as a combination of two different narrow-spectrum blue light sources, or a narrow-spectrum blue light source and a narrow-spectrum green light source used to distinguish a particular tissue Does not include the combination.

  As used herein, a color component has a spectrum of wavelengths within the visible electromagnetic spectrum.

  As used herein, the visible electromagnetic spectrum ranges from a wavelength of about 400 nanometers (nm) to 700 nm.

  As used herein, a color image is a combination of all color components of a color model, as opposed to a color image that contains only a single color image or a subset of the color components of a color model. Including. For example, for a color model that includes red, green, and blue components, the color image includes a combination of red, green, and blue components. A red image, a green image, a blue image, a blue and a green image, and the like are not included in the definition of a color image because such an image does not include a combination of all the color components of the color model.

  In one aspect, the surgical site 203 is illuminated using light from an endoscope 201 that forms part of the remotely operated surgical system 200. The illumination is non-white light, for example, the light appears purple when directly viewed by a human. Using non-white light illumination minimizes image noise of the surgical site 203 presented on the stereoscopic display 251, sometimes referred to as the display 251, at the surgeon console 250. Surgeon console 250 is sometimes referred to as console 250.

  The color of the light used to illuminate the surgical site 203 is non-white light, for example, light having a purple hue, but the image displayed on the stereoscopic display 251 of the surgeon console 250 includes this hue. Not. For the viewer, the light that illuminates the surgical site 203 appears white when viewed through the surgeon console 250.

  As described more fully below, the cameras 220L, 220R and the image processing pipeline 240 of the teleoperated surgical system 200 are used to remove the purple shade of the surgical image displayed on the stereoscopic display 251. Correct the captured image. Only when the surgeon exits the surgeon console 250, pulls the endoscope 201 from the patient, and looks directly at the light emitted from the endoscope 201, the surgeon will see non-white light. .

  Aspects of the invention make it easier to illuminate the surgical site 203 using non-white illumination, and with improved signal-to-noise compared to images captured using the white light illumination. It is easy to obtain a color image of the surgical site 203 by the cameras 220L and 220R (FIG. 2) of the remote operation surgical system 200. One example of teleoperated surgical system 200 is the da Vinci® minimally invasive teleoperated surgical system marketed by Intuitive Surgical, Inc. of Sunnyvale, California. Teleoperated surgical system 200 is merely exemplary and is not limited to applying non-white illumination to this particular teleoperated surgical system to improve the signal to noise ratio of the image. In view of this disclosure, non-white illumination can be used in any surgical system that utilizes color camera (s) to improve the signal-to-noise ratio of color images captured by these color cameras.

  In this example, a surgeon at surgeon console 250 remotely operates endoscope 201 attached to a robot manipulator arm (not shown). There are other parts, cables, etc., such as those associated with the da Vinci® surgical system, but these are not shown in FIG. 2 to avoid departing from the present disclosure. Further information regarding remotely operated minimally invasive surgical systems can be found, for example, in US patent application Ser. No. 11 / 762,165 (disclosed “Minimally Invasive Surgical System,” filed Jun. 13, 2007), and US Pat. No. 6,331,181 (disclosing “Surgical Robotic Tools, Data Architecture, and Use”, filed on Dec. 18, 2001), both of which are hereby incorporated by reference. Incorporated in the description.

  An illumination system, such as illumination device 210, is coupled to endoscope 201. In one aspect, the lighting device 210 includes a light source 211 and a lighting control device 215. The illumination controller 215 is coupled to the light source 211 and to an optional variable non-white light device 218.

  The lighting control device 215 includes a non-white light module 217 connected between the system processing module 262 and the light source 211. In one aspect, the non-white light module controls the output illumination from the illuminator 210 such that the illuminator 210 outputs non-white light that is used for general illumination of the surgical site 203.

  In one aspect, the light source 211 includes an illumination light source 212 of a plurality of color components. In the embodiment shown in FIG. 2, the plurality of color component illumination sources includes a P color component illumination source, where P is a non-zero positive integer. In one aspect, the number P is selected such that the combination of color component illumination sources provides prior art broad spectrum white light. Also, in order to form non-white light, at least one light power output of the plurality of color component illumination sources 212 is compared to the state used to generate prior art broad spectrum white light. Change to increase or decrease.

  In one aspect, the plurality of color component illumination sources 212 includes a plurality of light emitting diodes (LEDs). The use of LEDs is merely illustrative and is not limited to this. For example, the plurality of color component illumination sources 212 may include a plurality of laser light sources instead of LEDs.

  In this aspect, the illumination device 210 is used in connection with at least one illumination path of the stereoscopic endoscope 201 to illuminate the surgical site 203. Non-white light from the illumination device 210 is guided to the connector 216. The connector 216 supplies non-white light to the illumination path of the stereoscopic endoscope 201 and then guides the light to the surgical site 203. Each of the illumination paths in the connector 216 and the stereoscopic endoscope 201 can be implemented with, for example, an optical fiber bundle, a single rigid or flexible rod, or an optical fiber. In one aspect, the endoscope 201 also includes two light channels that pass light from the surgical site 203, such as reflected non-white light, ie, a stereoscopic light path. However, the use of the stereoscopic endoscope is merely an example, and the present invention is not limited to this. In view of the present disclosure, an endoscope including a single light channel that allows light from the surgical site 203 to pass through may be used.

  Non-white light from the surgical site 203 (FIG. 2) is passed to the cameras 220L and 220R through the stereoscopic light channel of the endoscope 201. As described more fully below, in one aspect, the left camera 220L includes a color filter array and a left image sensor 221L. The left image sensor 221L captures light received from the left channel of the stereoscopic endoscope 201 as a left image 222L. Similarly, in this aspect, the right camera 220R includes a color filter array and a right image sensor 221R. The right image sensor 221R takes in the light received from the right channel of the stereoscopic endoscope 201 as the right image 222R. Thus, the cameras 220L and 220R are color cameras using a color filter array. However, this is merely an example, and the present invention is not limited to this.

  Here, the camera is configured to separate light incident on the camera into N color components, which are captured by the camera as N sets of pixels, each set of pixel sets being a different camera color channel. It is in. Thus, each camera 220L, 220R includes a plurality of color channels. In one aspect, the plurality of color channels are N color component channels, where N is a non-zero positive integer.

  Camera 220L is coupled to stereoscopic display 251 of surgeon console 250 by left camera control unit 230L and image processing pipeline 240. The camera 220R is coupled to the stereoscopic display 251 of the surgeon console 250 by the right camera control unit 230R and the image processing pipeline 240. The camera control units 230L and 230R receive signals from the system process 262. System process 262 represents various control devices within system 200.

  Display mode selection switch 252 provides a signal to user interface 261 and then passes the selected display mode to system process 262. 262 various controllers in the system process set up the non-white light module 217 in the lighting controller 215, set up the left and right camera control units 230L and 230R to obtain the desired image, and the surgeon Other elements in the image processing pipeline 240 necessary to process the requested image are set to present the requested image on the display 250. The imaging processing pipeline 240 is equivalent to a known image processing pipeline except for the details provided herein.

  Here, the capturing, processing, and display of the image captured by the camera 220L are the same as the capturing, processing, and display of the image captured by the camera 220R. Therefore, in the following description, only images related to the camera 220L will be considered below. The description is directly applicable to images associated with camera 220R, and therefore the description will not be repeated for camera 220R. The description is directly applicable to systems that utilize only a single camera and a single image processing pipeline with an endoscope having a single light channel.

  As indicated above, typically the illumination components of three (or more) visible color components are combined to form white light, ie, the white light is a first visible color component, A combination of a second visible color component and a third visible color component is included. Each of the three visible color components is a different visible color component, for example, a red component, a green component, and a blue component. More visible illumination components such as cyan components along with red, green and blue components can be used to produce white light.

  Further, as described above, in one aspect, the light source 211 includes a plurality of color component illumination sources 212. In one aspect, in order to generate non-white light illumination, the non-white light module 217 includes a plurality of color component illumination sources 212 compared to that required to form white light. At least one optical power output is varied. The illumination output of the (non-) illuminator 210 is non-white light. Non-white light has a hue compared to white light.

  In FIG. 2, the cameras 220 </ b> L and 220 </ b> R and the light source 212 are shown to exist outside the endoscope 201. However, in one aspect, the cameras 220L, 220R and the light source 212 are included in the distal tip of the endoscope 201 adjacent to the tissue 203.

  There are various ways to configure the lighting device 210 such that the lighting device outputs non-white light illumination. The first method is based on white surface calibration, and the second method is based on surgical site image calibration. In addition, generation of non-white light that changes the lighting time of at least one of a plurality of color component illumination sources will be described. Each of these processes is considered in turn.

Non-White Light White Surface Calibration In this embodiment, the non-white light mode is the same amount of reflected light when all of the multiple color channels of the camera 220L are viewed on a white (fully reflective) surface. The illumination controller 215 operates to change the illumination intensity from the illumination source 212 of the plurality of color components in the light source 211. When imaging a white surface, there are two beneficial effects that the camera 220L responds to light equally across the color channels. First, all color channels can fully utilize their well capacity, ie one channel is limited to a percentage of its well capacity by another channel filling the well first. Not. This increases the signal-to-noise ratio of all pixels. Second, the camera does not need to apply digital white balance gain to some color channels that eliminates noise amplification. Both these effects enhance the final surgical image displayed on the stereoscopic display 251.

  Here, the noise characteristic of the camera 220L is interpreted as a whole depending on how high the electron well of the image sensor 221L is filled. As the camera 220L collects the light, the captured light is converted into electrons and stored in the wells of these electrons. There is one electron well per pixel. As the image sensor 221L, such as the camera 220L, collects more electrons in its electron well, the camera's signal-to-noise ratio increases, i.e. the signal to noise improves.

  As indicated above, when the camera 220L collects too much light and overfills the electron well, the signal in that electron well is lost and no longer valid. Thus, the exposure time of the camera 220L is set to collect light to fill all of the electron wells as high as possible without overfilling any electron wells. Stopping light collection when the color channel receiving most light is about to overflow its electron well, the result is that the electron wells of the other color channels are not sufficient. The electron wells of other color channels may be only 50% of the total. The signal-to-noise ratio of the color channel with these incomplete electron wells is significantly smaller than the signal-to-noise ratio of the color channel where the electron well just overflows.

  As explained earlier, for normal white light illumination, a camera only has a degraded signal-to-noise performance when not all the color channels of the camera have complete or nearly complete wells. Instead, the signal from the incomplete well is amplified as part of the white balance stage in the image processing stage of the prior art camera.

  In contrast to normal white light illumination, one aspect of non-white light illumination mode is that when all of the multiple color channels of the camera 220L are directed to a white (fully reflective) surface, the same amount of light. The illumination controller 215 operates to change the illumination intensity of the illumination source 212 of the plurality of color components in the light source 211. Since each of the multiple color channels of camera 220L receives approximately the same amount of light, all of the camera's color channels can be fully utilized in the non-white illumination mode. This is because the color channel does not initially reach its well capacity. Instead, the wells of multiple color channels reach their well capacity in approximately the same time. When photographing the white surface, the color channel well will not be 50% of its well capacity.

  Also, white light illumination is used because the non-white illumination from lighting device 210 is designed so that the channel response of all three cameras is equal to the light when directed to a white surface. The white balance stage in the image processing pipeline 240 does not need to amplify any of the multiple color channel signals. The signals of multiple color channels are already equal. The overall effect is that, by fully utilizing the well, there is very little noise during capture, so the resulting image is significantly less noisy and there is no need to amplify any color channel with respect to the other color channels.

  Decide how to increase or decrease the intensity of the different color component illumination sources in the multiple color component illumination sources 212 so that when the camera 220L is pointed at the white surface, the color channels of the camera 220L respond equally. Therefore, the characteristics of the camera 220L are examined, and the control of the light source 211 is examined. As previously described, the considerations for camera 220R are the same as for camera 220L, and therefore the description thereof will not be repeated for camera 220R. The following description also applies to an endoscope system that uses only a single signal channel and a single camera.

  In the following description, a color space that uses red, green, and blue components will be considered. Typically, a color camera includes a color filter array, such as a Bayer color filter array. Regardless of the configuration of the color camera, in the following description, there is a first set of pixels associated with the camera's red channel captured by the image capture sensor 221L. There is a second set of pixels associated with the camera's green channel captured by the image capture sensor 221L, and a third set of pixels associated with the camera's blue channel captured by the image capture sensor 221L. . The use of three color channels, such as representing the multiple N color channels of camera 220L, is merely illustrative and not limiting. The use of the red, green, and blue channels as the three color channels is merely an example, and the present invention is not limited to this. In view of this disclosure, those skilled in the art will recognize that the number of channels associated with the camera 220L and the specific color associated with the number of channels based on the color space of interest and the color filter array of interest. Both can be specified.

  The optical system of the camera 220L, the color filter array used in the camera 220L, and the quantum efficiency of the camera 220L (which are collectively referred to as the camera sensitivity function) indicate how the camera 220L is sensitive to incident light of different wavelengths. In other words, how to respond to the light from the surgical site 203 is determined. FIG. 3 shows an example of the response function of the camera 220L. A response function exists for each of the plurality of N color channels of the camera 220L.

  Thus, in this example, there are three response functions, one response function 301 for the blue channel, one response function 302 for the green channel, and one response function 303 for the red channel. In FIG. 3, each of the response functions 301 to 303 is represented by a curve.

  A larger value of a particular response function means that the color channel of camera 220L is more responsive to light of that particular wavelength compared to other wavelengths of light that have lower values for that particular response function. Show. For example, the red response function 303 indicates that the red channel of the camera 220L passes more light in the 600-650 nm wavelength range than in the 450-500 nm wavelength range.

  A lower value of a particular response function indicates that the color channel of camera 220L is less responsive to light of that particular wavelength compared to other wavelengths of light that have higher values for that particular response function. Show. For example, the blue response function 301 indicates that the blue channel of the camera 220L transmits less light in the wavelength range of 600-650 nm than in the wavelength range of 440-470 nm. A zero value of the response function indicates that the color channel of the camera 220L cannot recognize the wavelength of the light.

  In one aspect, the response functions of the multiple color channels of camera 220L are converted to matrix notation as three column vectors that make up matrix R. The matrix R is an M × N matrix. Specifically, in one aspect, a set of M equally spaced wavelengths, eg, 400 nm to 700 nm, spaced every 1 nm, is selected, and then the value of the response function is the value of these selected wavelengths. Read by each. In this example, 301 values are generated for each response function. Thus, in this example, M is equal to 301 and N is equal to 3 for the red, green and blue response functions. As long as the range from 400 nm to 700 nm encompasses all the important non-zero parts of the response function, the interval spacing is small enough (1 nm), and then the vector form corresponds to a complete curve. Sampling ranges and intervals usually vary based on the application.

  As described above, in one aspect, the light source 211 includes a plurality of P color component illumination sources 212. As an example, consider an implementation in which P is 4 such that the light source 211 includes four different color component illumination sources, eg, four individual LEDs that can be tuned to emit different intensities of light. An example of a lighting device comprising four individual LEDs is shown in US 2012/0004508 (disclosed “Surgical Illuminator With Dual Spectrum Fluorescence”, filed Aug. 13, 2010). Which is incorporated herein by reference.

  An example of the illumination spectrum of each of the four different LEDs is shown in FIG. A spectrum 401 is illumination of a blue component. A spectrum 402 is illumination of a cyan component. The spectrum 403 is green component illumination, and the spectrum 404 is red component illumination. In this aspect, each of the plurality of color component illumination sources 212 is a different color component illumination source.

  The spectra 401 to 404 can also be expressed in matrix notation. Each spectrum is a column of matrix E. The matrix E is an M × P matrix. Specifically, in one aspect, a set of M equally spaced wavelengths, eg, 400 nm to 700 nm, spaced apart by 1 nm is selected, and then the value of the LED illumination spectrum is selected from these M values. Read at each wavelength.

ここで、ｅ^ｏｕｔは、Ｍ×１列ベクトルとして放射されたスペクトルであり、
tは、Ｍ×１列ベクトルとして表される内視鏡の照明チャンネルの分光透過率であり、
ｄｉａｇ（ｘ）は、他の全ての要素がゼロである行列の対角にベクトルｘを配置することを示し、
ｗは、Ｐ×１の単一の列重みベクトルであり、重みベクトルｗの各要素は、複数のＰ個のＬＥＤの対応するＬＥＤの強度を決定するために、非白色光モジュール２１７によって使用される。 In addition, since the illumination control device 215 controls the output intensity of each LED in the light source 211, and the illumination output of the LED is optically linearly combined with the mixed output, the distal end of the endoscope 201. The spectrum e ^out of the light emitted from the part can be expressed as follows.

Where e ^out is the spectrum emitted as an M × 1 column vector,
t is the spectral transmittance of the illumination channel of the endoscope expressed as an M × 1 column vector,
diag (x) indicates that the vector x is placed at the diagonal of the matrix where all other elements are zero;
w is a P × 1 single column weight vector, and each element of the weight vector w is used by the non-white light module 217 to determine the intensity of the corresponding LED of the plurality of P LEDs. The

ここで、目標は、重みベクトルｗを解くことであり、[１]は、１のＮ×１列ベクトルを表し、Ｒ^Ｔは、応答関数の行列Ｒの転置行列であるＮ×Ｍ行列を表す。 In order to achieve non-white light, the weight vector w is the weight vector w when the response from each of the multiple N camera color channels is such that the LED cannot emit negative light. It is determined to be equally subject to the constraint that the element must be positive. This can be written in the following matrix form:

Here, the goal is to solve the weight vector w, [1] represents an N × 1 column vector of 1, and ^RT represents an N × M matrix that is a transposed matrix of the response function matrix R. .

  In other words, the previous equation indicates that the non-white light illumination module 217 applies a different one of the P color channel illumination control factors CCIW to each of the P color component illumination sources in the light source 211. The illumination control factor CCIW (element of the weight vector w) of the P color channel is determined so that the illumination device 210 outputs non-white light, and the non-white light emitted by the endoscope 201 is completely Each incident well of the camera 220L is filled to 100% perfect level when incident on a simple reflective surface, ie a white surface.

Thus, the problem is to find a weight vector w that contains only positive components. If the number P of controllable LEDs is equal to the number N of camera color channels, the simple solution is to take an inverse matrix when the matrix R ^T * diag (t) * E is a square matrix. . In this case, it is as follows.

As long as the weight vector w is all positive or zero (and R ^T * diag (t) * E has an inverse matrix), there is a solution. If the weight vector w has a negative component, the LED cannot be controlled to equalize the camera color channel. However, in this situation, the negative component of the weight vector w can be clipped to zero to obtain the closest solution.

However, when the number P of controllable LEDs is not equal to the number N of camera color channels, a simple solution is not possible because there is no inverse matrix of the matrix R ^T * diag (t) * E. For example, a simple solution cannot be used with a camera 220L having four controllable LEDs and only three color channels.

ここで、ｗ^ａは、擬似逆行列解であり、
Ｖ^ｎは、Ｎ×（Ｐ−Ｎ）行列の（Ｐ−Ｎ）ゼロ空間の列ベクトルを規定する、すなわち、カメラの応答を変更することなく、解ｗを変化させることができる方向性が存在し、
αは、ｗに対する１つの特定解を（Ｐ−Ｎ）×１ベクトルとして規定する。
１つの重みベクトルｗに対する解法を制限するには、別の制約を解法に与えなければならない、すなわち、我々は、単一のα値を決定することにより、ゼロ空間の値を指定する必要がある。 In this particular case (ie whenever the number P of controllable color component illumination sources is greater than the number N of color channels), there are many solutions for the components of the weight vector w: The different solutions for the weight vector w will make the filling of the camera color channel well uniform for non-white light incident on a perfectly reflective surface. Specifically, the set of solutions for w can be expressed as:

Here, w ^a is a pseudo inverse matrix solution,
V ⁿ defines a column vector of (PN) zero space of an N × (PN) matrix, that is, there is a direction that can change the solution w without changing the response of the camera. And
α defines one specific solution for w as a (P−N) × 1 vector.
To limit the solution to one weight vector w, another constraint must be given to the solution, i.e. we need to specify a zero-space value by determining a single alpha value .

  Possible constraints include maximizing radiant power, minimizing radiant power, maximizing drive current, minimizing drive current, or changing the color of light seen outside the surgeon console to appear whiter. Although it is mentioned, it is not limited to these. Due to LED drive electronics constraints, a constraint was implemented that maximized the minimum LED brightness. Basically, since the illumination controller 215 cannot reliably control light at low brightness, what is desired is to make the minimum value of the weight vector w as high as possible.

There is a minimax optimization to solve such a constraint, but since there is only one extra degree of freedom in the solution (P−N = 1), the illumination from the green and cyan LEDs is always lowest camera 220L, noticed empirically to have opposite signs in the null space vector V ^n. The value of the weight vector w was determined by sweeping out the zero space component α until the green and cyan channels were equal.

ここで、ｓｖｄ（Ｘ）は、Ｘの特異値分解であり、
ｄｉａｇ（ｘ）は、ｘの対角行列であり、
２及び３のインデックスは、それぞれ緑色及びシアン色のＬＥＤに対応する。
ここでは、Ｕ，Ｓ，Ｖは、特異値分解（svd）の出力である。特異値分解（svd）は、行列を３つの別々の行列に分割するための一般的な方法である：Ｘ＝ＵＳＶ^Ｔ、ここでＵ及びＶは、正規直交行列であり、Ｓは、対角行列である。ＸのＵ，Ｓ，Ｖへのこの分解によって、擬似逆及びゼロ空間の成分を見つけることができる。 Table 1 is an example of MatLab computer code compiled with a MatLab® compiler, which is then executed on the processor and runs in zero space until the green and cyan illumination channels are equal. The weight vector w is determined by sweeping out the component α. (MatLab is a US registered trademark of Mathworks, Inc., USA, Zip Code 01760, Massachusetts, Natick, Apple Hill Drive, 3.)

Where svd (X) is the singular value decomposition of X,
diag (x) is a diagonal matrix of x,
Indexes 2 and 3 correspond to green and cyan LEDs, respectively.
Here, U, S, and V are outputs of singular value decomposition (svd). Singular value decomposition (svd) is a general method for dividing a matrix into three separate matrices: X = USV ^T , where U and V are orthonormal matrices, and S is diagonal It is a matrix. By this decomposition of X into U, S, V, pseudo-inverse and zero-space components can be found.

  FIG. 5 shows the difference between white light illumination 502 and non-white illumination 501 using the above procedure. As can be seen, the blue and red LEDs output much more light than the green and cyan LEDs in the non-white light mode compared to the white light mode. In one aspect, the response of the camera 220L to white light when reflected from a white surface is red (R): 61%, green (G): 100%, blue (B) 67%, but non-white The camera response using light is red (R): 100%, green (G): 100%, and blue (B): 100% as designed.

  This technique of generating non-white light demonstrates that noise reduction in the displayed surgical scene can be achieved by adjusting the color of light from the endoscope. Specifically, the signal-to-noise ratio of each pixel is increased in the displayed surgical scene when all color channels of the camera (s) capture the scene response equally when directed to the white surface. Can be made. As the signal to noise ratio increases, the perceived noise in the final surgical image decreases.

Non-white light scene-based calibration In the prior art, the non-white light module 217 was configured to drive the light source 211 such that the illumination device 210 illuminates the surgical site 203 with non-white light, so Adjustment of the white light color balance of the color image captured by the camera 220L is unnecessary when non-white light is reflected from the white surface. This improved the signal to noise ratio of the image displayed on the display 251 compared to the image captured using normal white light illumination and displayed on the display 251.

  However, non-white light can be generated in other ways to improve the signal to noise ratio of the image presented on the display 251. For example, the controller 215 is configured to change the output illuminance level, eg, the light power output, of at least one of the multiple color component illumination sources 212 such that the illumination device 210 outputs non-white light. . The change in the output illuminance level of at least one of the plurality of color component illumination sources 212 is based on the color component characteristics of the color scene captured by the camera 220L. The non-white light generated in this case is improved with the improved signal to noise of the scene displayed on the display 251 compared to the scene captured using normal white light illumination and displayed on the display 251. Bring the ratio.

  In one aspect, the color channel lighting control factor CCIW is generated in various ways. The initially captured scene information is used for the generated color channel lighting control factor CCIW used by the lighting controller 215.

  In one aspect, a dynamic approach determines the color channel lighting control factor CCIW used to generate non-white light. For example, a frame 610 (FIG. 6A) is captured and used to generate a first set of color channel lighting control factors CCIW 620. The first set of color channel lighting control factors CCIW 620 is used for the next Z consecutive frames captured, where Z is a non-zero positive integer. Z consecutive frames are captured, processed and displayed, but one of the frames 621 in this sequence is used to generate a second set of color channel lighting control factors CCIW 621. The second set of color channel lighting control factors CCIW 621 is used for the next Z consecutive frames captured and processing continues.

  In one aspect, a time weighted moving average is used to generate the color channel lighting control factor CCIW621. In this aspect, a frame channel lighting control factor CCIW_frame is generated for each frame and weighted to decrease over the next fixed number of frames as described above and added to the other weighted frame channel lighting control factor CCIW_frame. The frame channel lighting control factor CCIW_frame of the recent frame becomes dominant, but is averaged so that the reduced fraction (eg linearly) of the frame channel lighting control factor CCIW_frame of the previous frame is added. For example, the currently applied weighted time moving average channel lighting control factor CCIW 621, which is updated at the frame rate, is provided.

  In one aspect, the number Z is empirically determined as a time average of the number of frames in order to maintain the stability of the system 200. In another aspect, the number Z is not constant. Rather, a new set of color channel lighting control factors CCIW is generated when the average luminance of the color channel is monitored and the average luminance of any color channel changes to exceed a predetermined percentage, eg, 5%. The This approach allows the fullness of the well of the color channel of the camera during the surgical procedure so that the electron well stays close to the optimal fill if the characteristics of the surgical site in the camera field of view change. ) Is adaptively compensated and the color channel illumination control factor CCIW is adjusted.

  In another aspect, a histogram of the luminance of the pixels of each color channel of the camera 220L is created for the captured scene. A histogram of the luminance of each pixel of the plurality of N color channels of the camera 220L is created. As known to those skilled in the art, each pixel has a pixel value that is a single number representing the luminance of the pixel. The pixel value is an index of the filling amount of the pixel well of the pixel. Thus, N luminance histograms are created for the camera 220L, one for each color channel.

  In each of the N luminance histograms, the luminance value is plotted on the x-axis. The height of the bar for each luminance value represents the number of pixels in the color channel having that luminance value. The luminance histogram can be based on the entire captured image or the region of interest of the captured image. For example, the region of interest can be defined as a region within the fovea of a person using the remotely operated surgical system 200. The display of the filling amount of the well of the pixel of the image sensor 221L is at a position where the color channel is derived from the luminance histogram of each color channel. The display can be an average value or a value corresponding to the 90th percentile of all values.

Matrix B is defined as the connection between the illumination control and the color channel response of the camera sensor. This matrix converts lighting control, P-element vector to camera sensor color channel response, N-element vector. The matrix B can be measured by rotating on the illumination channel one by one at the reference level. Thus:

ここで、ＷＦｕｌｌは、Ｎ×１列ベクトルであり、各成分が、カメラ２２０Ｌのカラーチャンネルの画素ウェルの所望の充填量を表し、
ＣＣＩＷは、Ｐ×１の列ベクトルであり、各要素が、光源２１１の色成分の照明源のうちの１つのカラーチャンネル照明制御因子を表す。 The color channel illumination control factor CCIW is defined as follows.

Here, WFFull is an N × 1 column vector, and each component represents a desired filling amount of the pixel well of the color channel of the camera 220L,
CCIW is a P × 1 column vector, and each element represents one color channel illumination control factor among the illumination sources of the color components of the light source 211.

  The determination of the color channel lighting control factor CCIW is easy when P is equal to N and there is an inverse matrix of B. If P is greater than N, use the pseudo inverse of B.

  Matrix inverses are well known to those skilled in the art. A pseudo-inverse suitable for use here is called a Moore-Penrose pseudo-inverse. A common use of the Moore-Penrose pseudo-inverse is to compute an “optimal” least squares solution for a system of linear equations that lacks an inherent solution. Another use of the Moore-Penrose pseudo-inverse is to find a minimum (Euclidean) norm solution for a system of linear equations. In one aspect, an optimal least squares solution is used.

  The color channel lighting control factor CCIW thus generated can be used in the teleoperated surgical system 200 in either a static approach or a dynamic approach, as described above. Regardless of the technique used to generate the color channel illumination control factor CCIW, using non-white light illumination is compared to a display image formed from an image captured using white light illumination, The image quality displayed on the display 251 is improved by reducing the noise contribution to the display image.

  As an example, assume that camera 220L has red, green, and blue channels, for example, N is 3, and the desired fill amount WFFull is 75% for each color channel. With white light illumination, red channel R603B is 90% filled (FIG. 6B), green channel G602B is 50%, and blue channel B601B is 37.5%. For non-white illumination, the red channel R603C is about 75% fill (FIG. 6C), the green channel G602C is 75%, and the blue channel B601C is about 75% fill. The noise floor for both white and non-white illumination was 15%. Thus, the signal to noise ratio of the blue and green components is improved for non-white light.

  However, because the illumination from each of the multiple color component illumination sources 212 in the light source 211 is absorbed by the surgical site 203 and reflected in different directions, the electron well may be exactly 75% filled. There is no. Thus, in this aspect, when the response of the camera channel to non-white light is approximately equal, the response of the camera image sensor portion in the different color channels takes into account the difference in absorption and reflection of non-white light by the surgical site. Means equal. Nevertheless, the signal-to-noise ratio was improved at least for blue and green pixels.

  In the color image of the surgical site 203, the green and blue pixels provide most of the detail because the green and blue illumination components are less scattered and penetrate less than the red illumination components. As a result, improving the signal to noise ratio of the blue and green pixels improves the image quality provided to the display 251.

ここで、Ｉ０＝［ｒ０，ｇ０，ｂ０］^Ｔは、Ｎ×１の画素輝度行列であり、ＲＧＢ成分が同じ値を有し、
Ｌ０＝［Ｌ０１，．．．，Ｌ０Ｐ］^Ｔは、Ｐ×１の光パワー出力行列である。 When pixels with wells filled as shown in FIG. 6C are processed directly for display on the display 251, the displayed image is an electron in the color channel associated with non-white light illumination. Proper color balance adjustment will not be performed depending on the filling volume of the well. Thus, when each color channel pixel is acquired from the image sensor, the color channel pixel value is corrected to compensate for the color channel illumination control factor CCIW. Assuming that the illumination control L0 produces a white image when observing a scene of an intermediate color object (eg, white balance target):

Here, I0 = [r0, g0, b0] ^T is an N × 1 pixel luminance matrix, and the RGB components have the same value,
L0 = [L01,. . . , L0P] ^T is a P × 1 optical power output matrix.

Later, additional gain A = [ccw1,. . . , CcwP] ^T = CCW ^T is applied to the lighting control on top of L0, resulting in L, where L = [L1,. . . , LP] ^T = [ccw1 * L01,. . . , CcwP * L0P] ^T. The color response from the camera using illumination L is I = [r, g, b] ^T.
K = B * A = [kr, kg, kb]

  The adjusted pixel color I ′ = [r / kr, g / kg, b / kb] is presented on the display to achieve an accurate color. This results in typical pixel values as shown in FIG. 6D. The red channel noise has increased slightly, but the green and blue channel noise is gone. Thus, when the green and blue channel signals are amplified in the white balance adjustment process of the image processing pipeline 240, the signal to noise ratio is better than with white light illumination.

High Dynamic Range Image Using Non-White Light A standard method for forming a high dynamic range image using a video camera is to take successive images at different exposure levels with the video camera. The differently exposed images are merged into a single high resolution image. This technique requires that the video camera has a function of switching the exposure setting for each frame. This results in an effectively reduced frame rate compared to a video camera that generates a display image from each captured frame. Also, if there is motion in the scene, motion artifacts may be observed when images captured over time are merged into a single high resolution image.

  Another approach used in photography to form a high dynamic range image is to capture the image using a graded neutral density filter. The dimming filter is adjusted in stages so that the bright areas of the scene are attenuated more by the filter than the dim areas of the scene. This is suitable for scenes with known areas of different brightness, such as sunrise or sunset scenes, but the same part of the surgical site scene is not always the brightest area of the scene, so in a remote operated surgical system The use of a stepwise neutral density filter is not practical.

  For applications that do not require a color image, non-white light illuminating the camera 220L and the surgical site 203 can be used to generate a single color high dynamic range image. For example, unlike diagnostics, color images may not be important during navigation of surgical instrument (s), for example during guidance of lung navigation.

  In applications where monochrome images are acceptable, it is possible to form high dynamic range images using non-white light illumination and cameras that do not use different exposure settings or neutral density filters. The non-white light module 217 drives the multiple color component illumination sources 212 such that the multiple color component illumination sources 212 have an intensity other than the normal intensity used to form the white light illumination. Configured as follows.

  Thus, the camera 220L captures N images, and each of the N color channels of the camera is effectively exposed differently by the different weights used for the color component illumination sources 212. These N images are used to generate a high dynamic range image in a manner equivalent to that used for cameras with neutral density filters. Thus, a high dynamic range image can be obtained without the need for any special filter or variable exposure setting camera. The use of non-white light illumination allows the generation of high dynamic range images using conventional cameras that include color filter arrays used in teleoperated surgical systems.

  For the purposes of the example, the number P of color component illumination sources in the plurality of color component illumination sources 212 is three, and the three color component illumination sources are a red light source, a green light source, and a blue light source. Assume that there is. Also, for normal white light illumination, the non-white light module 217 weights each of the three light sources equally, eg, red, green and blue weights (color channel lighting control factor CCIW) are 1: 1: 1. Assuming that In one aspect, the weight (color channel lighting control factor CCIW) is changed so that the lighting device 210 provides non-white light of, for example, a red, green, blue light source, and the weight is 0.5: 1.0. : 2.0.

Generally, the weight of the color component is determined by taking the ratio of the pixel dynamic range (pixel dynamic range) of the color channel of the camera 220L to the dynamic range of reflected light from the surgical scene (scene dynamic range). The ratio of pixel dynamic range to scene dynamic range is 1: DR, where DR is a positive integer other than zero. In this example, the maximum weight of the first illumination color component is DR times the weight of the Nth illumination color component. The weights due to the (N-1) illumination color components between the second are uniformly spaced between the minimum and maximum weights by a power of 2. For example, when the pixel dynamic range is 1: 256 and the scene dynamic range is 1: 1024, the ratio between the two is 1: 4 (2 ² ). In the above example, the minimum weight was 1, so the maximum weight was 4. Since the power of 2 between 1 and 4 is 2 ¹ = 2, the third weight is 2.

As another example, consider an illumination source with a ratio of 1:16 and three color components. The minimum weight is 1 and the maximum weight is 16. The other weight is 4. If the ratio is 1:16 and there are four color component illumination sources, the weight will be 1: 2 ^(4/3) : 2 ^(8/3) : 16.

  When the scene is captured from non-white light reflected by the camera 220L, each of the N color channels has a grayscale scene with a reflected light intensity that varies depending on the difference in light power output of the P color component illumination source at the light source 212. take in. The N grayscale scene is processed to produce a high dynamic range grayscale image. One technique for doing this is in Nayar and Mitunga, “High Dynamic Range Imaging: Spatially Varying Pixel Exposures,” IEEE Conference on Computer Vision and Pattern Recognition, Vol.1, pp.472-479 (2000). Which is described and incorporated herein by reference.

  Rather than using images obtained with different camera exposure settings or neutral density filters, using a camera with fixed exposure using non-white light illumination that contains a combination of color illumination components of different intensity Acquire different exposure images. Since N different exposure images are captured simultaneously, there are no motion artifacts.

  In the previous example, different intensity color components of non-white light were used to capture differently exposed images into a single frame. A similar effect can be obtained by changing the length of time that the illumination source for each color component is output within the exposure time of the frame. In this aspect, the weight applied by the non-white light module 217 is the same as that used for white light, but switching element (s) are added to the non-white light module 217.

  Consider the same example where the ratio of the optical power output of the red, green and blue components is 0.5: 1: 2 and the exposure time is time t. In this example, the blue light source is output over time (2 / 3.5) * t. The green light source is output over time (1 / 3.5) * t, and the red light source is output over time (0.5 / 3.5) * t. Thus, each of the color component light sources is pulse width modulated to have a specific rate of exposure time.

  FIG. 7 is a diagram illustrating an example in which the output from the illumination sources of the red, blue, and green components of the illumination device 210 is changed in order to control the output light from the illumination device 210. As long as the blue light source is turned on for 58% of the exposure time, the green light source is turned on for 28% of the exposure time, and the red light source is turned on for 14% of the exposure time, illumination of each color component The particular order in which the sources are turned on / off is not important. Of course, instead of switching the light source on / off, the light source can be kept on, and the output of the light source is from the output of the illuminator 210 during the exposure time as shown in the off state of FIG. It can be directed away or reaching its output.

  The particular number of light component illumination sources used in the above examples and the light source illumination source weights are merely exemplary, and the particular number of light component illumination sources and weights used in the examples are It is not limited.

  In another embodiment, the output of the color component illumination source varies as shown in FIG. 7, but a different frame is captured for each color component illumination source. When a fixed frame shutter is used, the lighting switch is synchronized to the image acquisition. When a rolling shutter is used, the illumination must be switched to a frequency that does not cause flicker in the captured image by switching the illumination. As used herein, a rolling shutter does not capture the entire frame at once, but rather means that information is read from each row of the frame, for example, from top to bottom.

  Thus, in this example, the camera exposure time is fixed, but the illumination is changed to obtain a different exposure image. In this case, the high dynamic range image is formed by a method corresponding to a known technique for obtaining a high dynamic range image for images taken at different exposures.

  Changing the output capability of the lighting device 210 can be achieved in many ways. The output of the light source having a plurality of components can be directly controlled as described above. Element 218 may be a rotating wheel having a plurality of sections arranged in the path of light output from lighting device 210. Each of the plurality of sections has one color of the plurality of color components, and each of the plurality of sections has a different light attenuation level. In another aspect, element 218 is an acousto-optic light modulator coupled to the controller.

  Alternatively, element 218 can be a liquid crystal device. In one aspect, the liquid crystal device is configured in an on / off pulse width modulated shutter mode, which provides an on / off time variation ratio that includes one or more on / off cycles per capture of the camera image frame. Have. In another aspect, the liquid crystal device is configured as an adjustable attenuator, such as, for example, an incident polarizer, a compensated liquid crystal variable retarder, and an exit polarizer, where the entrance and exit polarizers are linear polarizers. Intersect. In yet another aspect, the liquid crystal device is configured as a wavelength tunable filter. The use and operation of liquid crystal wavelength tunable filters are well known to those skilled in the art.

  Thus, in some aspects, a fixed filter is used to change the illuminance level of at least one of the multiple color component illumination sources so that the illumination device outputs non-white light.

  Also, in some aspects, illuminance level fluctuations that generate non-white light are controlled to adjust for unequal power loss of aging induced by the lifetime of illumination sources 212 of multiple color components. . Thus, when the optical power output, such as a light emitting diode or laser diode, decreases over time due to power loss induced by aging, the color channel lighting control factor CCIW can be used for each of the plurality of color component illumination sources 212. Adjustments can be made statically or dynamically to compensate for unequal power loss induced by aging.

  The various modules described herein can be implemented by software executed by a processor, hardware, firmware, or any combination of the three. When the module is implemented as software executed on a processor, the software is stored in memory as computer-readable instructions, which are executed by the processor. All or a portion of the memory may be in a different physical location than the processor as long as the processor is coupled to the memory. Memory refers to volatile memory, non-volatile memory, or any combination of the two.

  Also, as described herein, the functions of the various modules can be performed by one unit or divided between different components or different modules, each function being then Can be implemented by any combination of software running on the processor, and firmware. When dividing between different components or modules, the components or modules may be centralized or distributed across the system 200 for distributed processing. The execution of the various modules results in a method for performing the processes described above for the various modules and the controller 260.

  Thus, the processor is coupled to a memory that contains instructions to be executed by the processor. This can be accomplished within the computer system or through coupling with other computers via modems and analog lines or digital interfaces and digital carrier lines.

  Here, the computer program product is configured to store computer readable code required for some or all of the processes described herein, or stores part or all of the computer readable code for these processes. Computer readable media. Some examples of computer program products represent CD-ROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tape, computer hard drives, servers on a network, and computer readable program code. , A signal transmitted via a network. A non-transitory tangible computer program product is configured to store part or all of the computer readable instructions of a process, or a tangible computer readable medium having stored part or all of the computer readable instructions of a process including. Non-transitory tangible computer program products are CD-ROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tape, computer hard drives, and other physical storage media.

  In view of this disclosure, instructions used for some or all of the processes described herein may be implemented in a wide variety of computer system configurations using operating systems and computer programming languages that are of interest to the user. it can.

  The foregoing detailed description and accompanying drawings, which illustrate aspects and embodiments of the present invention, are not to be construed as limiting, and the claims define the invention to be protected. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from the spirit and scope of the specification and the claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail to avoid obscuring the present invention.

  Also, the terminology in this detailed description does not limit the invention. For example, `` beeneath '', `` below '', `` lower '', `` above '', `` upper '' ”,“ Proximal ”,“ distal ”, and other space related terms describe the relationship of one element or mechanism to another element or mechanism shown in the figure. Used for. These space-related terms are intended to encompass different positions (ie, placement) and orientations (ie, rotational positions) of the device in use or operation in addition to the positions and orientations shown in the drawings. Yes.

  For example, when the device in the drawing is flipped over, an element described as “below” or “beneath” of another element or mechanism is “Above” or “over”. Thus, the exemplary term “below” can encompass both “above” and “below” positions and orientations. The device may be oriented in other ways (90 degree rotation or other orientation), and the space related description used herein will be interpreted accordingly.

  Similarly, the description of movement along and around various axes includes various special positions and orientations of the device. The singular forms “a, an” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. The terms “comprises, comprising”, “including”, etc. identify the presence of the described feature, step, operation, element, and / or component, but one or more The presence or addition of other features, steps, operations, elements, components, and / or groups is not excluded.

  Components described as “coupled” may be coupled electrically or mechanically directly, or indirectly through one or more intermediate components. In view of this disclosure, instructions used in any or any combination of operations described with respect to an enhanced display system are implemented in a wide variety of computer system configurations using operating systems and computer programming languages that are of interest to the user. can do.

References to all examples and descriptions are non-limiting and should not be used to limit the claims to the specific implementations or embodiments described herein and their equivalents. Headlines are just for formatting and the text under one heading can be cross-referenced, i.e. applied to text under one or more headings, thus limiting the subject in any way Should not be used to do. Finally, in light of the present disclosure, certain features described in connection with one aspect or embodiment are not specifically illustrated in the drawings or described in the text, although not disclosed herein. It can be applied to other aspects or embodiments described.

Claims (21)

A device, the device
A camera configured to separate light incident on the camera into a set of pixels, each set of the set of pixels being in a different color channel of the camera;
An illumination device, wherein the illumination device is configured to output non-white light, whereby each color channel of the camera has a substantially equal response to the non-white light reflected from a perfect reflective surface. A lighting device,
machine.   The apparatus of claim 1, wherein the lighting device comprises a plurality of color component illumination sources, wherein the plurality of color components are configured such that the lighting device outputs the non-white light.   And a control device coupled to the illumination sources of the plurality of color components, wherein the control device is configured such that the combination of the outputs of the illumination sources of the plurality of color components is non-white. The apparatus of claim 2, configured to weight each output of a component illumination source. A device, the device
A camera,
A lighting device;
A control device coupled to the lighting device,
The camera is configured to separate light incident on the camera into color components, the color components being captured as a set of pixels by the camera, each set of pixel sets being a different camera color. In the channel,
The controller adjusts the characteristics of the light output by the illuminator to increase the signal-to-noise ratio of one camera color channel pixel for a color image captured by the camera. Composed,
machine.   The apparatus according to claim 4, wherein the illumination device includes an illumination source having a plurality of color components.   The controller is coupled to the illumination source of the plurality of color components, and the controller determines a signal-to-noise ratio of the pixels of the one camera color channel for the color image captured by the camera. The apparatus of claim 5, wherein the apparatus is configured to adjust a characteristic of at least one of the plurality of color component illumination sources to increase.   6. The apparatus of claim 5, wherein the plurality of color component illumination sources are a plurality of light emitting diodes.   6. The apparatus of claim 5, wherein the plurality of color component illumination sources are a plurality of laser diodes.   The controller is configured to control the output of the illumination source of the plurality of color components of the illumination device such that the illumination device outputs non-white light, whereby each camera color channel is fully The device of claim 4, having a substantially equal response to the non-white light reflected from a reflective surface.   5. The controller of claim 4, wherein the controller is configured to change an illuminance level of at least one of a plurality of color component illumination sources of the illumination device such that the illumination device outputs non-white light. Equipment.   The apparatus of claim 4, wherein the controller is further configured to form a high dynamic range image from a monochromatic image captured by the camera.   The apparatus of claim 11, wherein the controller is configured to change an illuminance level of at least one of a plurality of color component illumination sources of the illumination device.   And further comprising a rotating wheel having a plurality of sections coupled to the controller, each of the plurality of sections having a color of the plurality of color components, and each of the plurality of sections. 13. The device of claim 12, wherein the devices have different light attenuation levels.   The apparatus of claim 12, further comprising a liquid crystal device coupled to the controller.   15. The liquid crystal device according to claim 14, wherein the on / off time variation ratio is set to an on / off pulse width modulated shutter mode, wherein the on / off time variation ratio includes one or more on / off cycles per camera image frame capture. The equipment described.   The apparatus of claim 14, wherein the liquid crystal device is configured as an adjustable attenuator.   The apparatus of claim 14, wherein the liquid crystal device is configured as a wavelength tunable filter.   The apparatus of claim 12, further comprising an acousto-optic light modulator coupled to the controller. A method comprising:
Illuminating the scene with non-white light, wherein the non-white light is configured such that each camera color channel of the camera has a substantially equal response to light reflected from a perfectly reflective surface. Illuminating,
Capturing an image of the scene using the camera;
Outputting a display image based on the captured image without adjusting the white color balance of the captured image.
Method. A method comprising:
Capturing a color image including a set of pixels, each set of the pixel set being in a different color channel of the camera;
Constructing a high dynamic range image from the set of pixels;
Method. A method comprising:
Illuminating a site with non-white light, wherein the non-white light is configured such that each camera color channel of the camera has a substantially equal response to the non-white light reflected from the site And steps to
Capturing an image of the part using the camera;
Outputting an image for display based on the captured image.

Images