JP7552002B2 - Light level adaptive filter and method - Google Patents

Description

RELATED APPLICATIONS This patent application claims the benefit of priority and the filing date of U.S. Provisional Patent Application No. 62/288,956, entitled "LIGHT LEVEL ADAPTIVE FILTER AND METHOD," filed on January 29, 2016, which is incorporated herein by reference in its entirety.

The present invention relates generally to image pipelines for computer-assisted surgery systems, and more specifically to adaptive filters used in image pipelines.

The surgical system 100 is a computer-assisted surgical system that includes an endoscopic imaging system 192, a surgeon console 194 (master), and a patient support system 110 (slave), all of which are interconnected by wired (electrical or optical) or wireless connections 196. One or more electronic data processors may be variously located within these major elements to provide system functionality. Examples are disclosed in U.S. Patent Application Publication No. 2008/0065105, which is incorporated herein by reference.

The patient side support system 110 includes an entry guide manipulator 130. At least one surgical device assembly is coupled to the entry guide manipulator 130. Each surgical device assembly includes an instrument, which in turn includes either a surgical instrument or an image capture unit. For example, in FIG. 1, one surgical device assembly includes an instrument 135-1 having a shaft 137-1 and an image capture unit that extends through the entry guide 115 during a surgical procedure. The instrument 135-1 is sometimes referred to as an endoscope, or alternatively an imaging system device or camera instrument. Typically, the entry guide 115 includes multiple channels.

The imaging system 192 performs image processing functions, for example, on captured endoscopic imaging data of the surgical site and/or pre-operative or real-time image data from other imaging systems external to the patient. The imaging system 192 outputs processed image data (e.g., images of the surgical site as well as associated control and patient information) to the surgeon at the surgeon console 194. In some aspects, the processed image data is output to an optional external monitor for viewing by other participants in the operating room or at one or more locations remote from the operating room (e.g., a surgeon in a different location may monitor the video, a live feed video may be used for training, etc.).

The Surgeon Console 194 includes multiple degrees of freedom (DOF) mechanical input devices (masters) that allow the surgeon to steer the instruments, entry guide(s), and imaging system devices (collectively referred to as slaves). In some aspects, these input devices can provide tactile feedback to the surgeon from the instruments and components of the surgical device assembly. The Surgeon Console 194 also includes stereoscopic video output displays positioned such that the images on the displays are generally focused at a distance corresponding to the surgeon's hands working behind/below the display screen. These aspects are discussed more fully in U.S. Pat. No. 6,671,581, which is incorporated herein by reference.

Control during insertion of the instrument may be achieved, for example, by the surgeon moving the instrument shown in the image with one or both of the masters. The surgeon uses the master to move the instrument to the left or right of the image, pulling the instrument toward the surgeon. The movement of the master commands the imaging system and associated surgical device assembly to steer toward a fixed center point on the output display and advance inside the patient.

In one aspect, the camera control device is designed to give the impression that the master is fixed in the image so that the image moves in the same direction as the master handle is moved. This design places the master in the correct position for instrument control when the surgeon ends camera control, and as a result, this design avoids the need to clutch (disengage), move, and declutch (engage) the master back to its original position before starting or resuming instrument control.

The base 101 of the patient side support system 110 supports arm assemblies including a passive, uncontrolled setup arm assembly 120 and an actively controlled manipulator arm assembly 130. The actively controlled manipulator arm assembly 130 is sometimes referred to as an entry guide manipulator 130. The entry guide manipulator assembly platform 132 (sometimes referred to as platform 132) is coupled to the distal end of the fourth manipulator link 119. The entry guide manipulator assembly 133 is rotatably attached to the platform 132. Arrows 190 indicate distal and proximal directions.

The entry guide manipulator assembly 133 includes an instrument manipulator positioning system. The entry guide manipulator assembly 133 rotates the multiple instrument manipulators 140-1, 140-2 as a group around the axis 125.

Each of the multiple instrument manipulators 140-1, 140-2 is coupled to the entry guide manipulator assembly 133 by a different insert assembly 136. In one aspect, each insert assembly 136 is a telescoping assembly that moves a corresponding instrument manipulator away from and towards the entry guide manipulator assembly 133. In FIG. 1, each of the insert assemblies is in a fully retracted position.

Each of the multiple instrument manipulator assemblies 140-1, 140-2 includes multiple motors that drive multiple outputs at the output interface of the instrument manipulator. For an example of an instrument manipulator and a surgical instrument that can be coupled to the instrument manipulator, see U.S. Patent Application No. 61/866,115, filed Aug. 15, 2013, which is also incorporated by reference.

Each of the plurality of surgical device assemblies 180 includes a plurality of different instrument manipulator assemblies and an instrument including one of a surgical instrument and an image capture unit. Each of the instruments 135-1, 135-2 includes a body housing a transmission unit. The transmission unit includes an input interface including a plurality of inputs. Each of the instruments 135-1, 135-2 also includes a shaft 137-1, 137-2, sometimes referred to as a main tube, extending distally from the body. An end effector is coupled to the distal end of the shaft of the surgical instrument assembly, and an image capture unit, e.g., a camera, is included at the distal end of the different surgical instrument assembly. For examples of instrument manipulator assemblies and surgical instruments, see U.S. Patent Application No. 61/866,115, filed Aug. 15, 2013, which is also incorporated by reference.

Each of the instruments 135-1, 135-2 is coupled to an instrument mounting interface of a corresponding instrument manipulator assembly 140-1, 140-2, such that the inputs at the input interface of the transfer unit of the instruments 135-1, 135-2 are driven by the outputs at the instrument mounting interface of the instrument manipulator assembly 140-1, 140-2. See U.S. Patent Application No. 61/866,115 (filed Aug. 15, 2013).

As shown in FIG. 1, the shafts of the multiple surgical device assemblies 180 extend distally from the body of the instruments. The shafts extend through a common cannula 116 that is placed at an entry port into the patient (e.g., through a body wall or at a natural orifice). In one aspect, an entry guide 115 is positioned within the cannula 116, and each instrument shaft extends through a channel in the entry guide 115 to provide additional support to the instrument shaft.

The system includes an image sensor, an imaging pipeline, and a display device. The image sensor is configured to capture a first frame of pixel data. The imaging pipeline is coupled to the image sensor to receive the first frame of pixel data.

The imaging pipeline includes an adaptive noise filter. The adaptive noise filter is configured to filter pixels using an estimated pixel noise parameter for a first pixel and using a difference between a signal level of the first pixel and a signal level of at least a second pixel. The adaptive noise filter is also configured to output a second frame of pixel data. The second frame of pixel data includes pixels filtered by the adaptive noise filter.

The imaging pipeline is configured to output a frame of pixel data based on the second frame of adaptive noise filtered pixel data, where the frame of pixel data based on the second frame of adaptive noise filtered pixel data is a frame of pixel data that is a result of further processing of the adaptive noise filtered second frame of pixel data by the imaging pipeline. A display device is coupled to the imaging pipeline to receive the frame of pixel data based on the second frame of adaptive noise filtered pixel data. The display device is configured to display the frame of pixel data based on the second frame of adaptive noise filtered pixel data.

In one aspect, the second pixel is at the same location in a third frame captured by the image sensor as the location of the first pixel in the first frame. Because the third frame was captured at a time prior to the capture of the first frame, the signal level difference used by the adaptive noise filter is the time signal level change between pixels at the same location in the two different frames. In another aspect, the difference between the signal level of the first pixel and the signal level of at least the second pixel is the difference between the signal level of the first pixel and the signal level of one of a plurality of pixels (sometimes referred to as a plurality of adjacent pixels) adjacent to the first pixel. One of the plurality of adjacent pixels is the second pixel.

The adaptive noise filter includes an adaptive temporal noise filter. The adaptive temporal noise filter is configured to compare the temporal change in the signal levels of the first and second pixels to an estimated pixel noise level of the first pixel. In this aspect, the estimated pixel noise level is an estimated pixel noise parameter. The adaptive temporal noise filter is also configured to output a pixel noise frame.

In one aspect, the imaging pipeline includes a plurality of stages, and the adaptive temporal noise filter is included in a first stage of the plurality of stages. Also, each of the plurality of stages subsequent to the first stage of the plurality of stages is configured to process the input pixel noise frame and to output a stage-dependent pixel noise frame.

In another aspect, the adaptive noise filter includes an adaptive spatial noise filter. The adaptive spatial noise filter is configured to determine a difference between a signal level of the first pixel and a signal level of each of the multiple neighboring pixels. The adaptive spatial noise filter also uses a noise-dependent signal level parameter, where the noise-dependent signal level parameter is a function of the noise of the first pixel. In this aspect, the noise-dependent signal level parameter is an estimated pixel noise parameter of the pixel. In another aspect, the adaptive spatial noise filter includes a distance filter and a signal level range filter. The signal level range filter is configured to filter the pixel based on a difference between a signal level of the pixel and each of the multiple neighboring pixels and based on the noise-dependent signal level parameter. In one aspect, the noise-dependent signal level parameter is a signal level range parameter.

In a further aspect, the imaging pipeline stage includes an adaptive temporal noise filter including a signal level vs. noise lookup table. The adaptive temporal noise filter is configured to receive a frame of pixels, filter each pixel of a plurality of pixels of the frame by comparing a change in the signal level of the pixel over time to a value in the signal level vs. noise lookup table corresponding to the signal level of the pixel, and output an adaptive temporal noise filtered frame of pixels based on the comparison.

In another aspect, the imaging pipeline stage includes a current pixel frame, a signal level vs. noise lookup table, a modified lookup table, a pixel output frame, a pixel noise frame, and an adaptive temporal noise filter coupled to the current pixel frame, the signal level vs. noise lookup table, the modified lookup table, the pixel output frame, and the pixel noise frame. The signal level vs. noise lookup table is configured to receive signal levels of pixels of the current pixel frame and output a noise standard deviation corresponding to the signal levels of the pixels. The modified lookup table is configured to receive a time change in the signal level of the pixels represented by a numerical value of the noise standard deviation, and output a percentage of the allowable time change in the signal level of the pixels that passes the filter. The adaptive temporal noise filter is configured to output the adaptive temporal noise filtered pixels to the pixel output frame. The adaptive temporal noise filter is also configured to output pixel noise, i.e., output pixel noise corresponding to the adaptive temporal noise filtered pixels, to the pixel noise frame.

In yet another aspect, the adaptive spatial noise filter is configured to receive an input pixel frame and an input pixel noise frame, and is configured to filter a pixel of the input pixel frame based on a spatial change between a signal level of the pixel and a signal level of a plurality of adjacent pixels, and based on a noise-dependent signal level parameter to generate an adaptive spatial noise processed pixel. The noise-dependent signal level parameter is determined using pixel noise in the input pixel noise frame corresponding to the pixel. The spatial filtering of the pixel is related to a plurality of adjacent pixels in the input pixel frame. The adaptive spatial noise filter is also configured to output the adaptive spatial noise filtered pixel in the adaptive spatial noise filtered pixel frame.

In yet another aspect, the imaging pipeline stage includes an input pixel frame and an input pixel noise frame. Each pixel noise element in the input pixel noise frame corresponds to a different pixel in the input pixel frame. The imaging pipeline stage further includes a pixel output frame and an adaptive spatial noise filter coupled to the input pixel frame, the input pixel noise frame, and the pixel output frame.

ここで、

は、位置ｘにおける画素の適応空間ノイズフィルタ処理された信号レベルであり、

は、正規化定数であり、

は、空間的なガウス分布であり、

は、位置ｙの画素と位置ｘの画素との間の距離のノルムであり、

は、距離パラメータであり、

は、位置ｘにおける画素のノイズの関数である信号レベル範囲のガウス分布であり、

は、位置ｙにおける画素の信号レベルであり、

は、位置ｘにおける画素の信号レベルであり、

は、位置ｙにおける画素（第２の画素）の信号レベルと位置ｘにおける画素（第１の画素）の信号レベルとの差の絶対値であり、

は、位置ｘにおける画素のノイズの関数である信号レベルレンジパラメータであり、

は、位置ｘに中心画素を有する画素のブロックであり、

は、位置ｘに中心画素を有する画素ブロック内の画素に亘る総和を表す。 In one aspect, the adaptive spatial noise filter includes an adaptive spatial noise bilateral filter: For a pixel at location x, the adaptive spatial noise bilateral filter is defined as follows:

Where:

is the adaptive spatial noise filtered signal level of the pixel at location x,

is a normalization constant,

is a spatial Gaussian distribution,

is the norm of the distance between the pixel at position y and the pixel at position x,

is the distance parameter,

is a Gaussian distribution of the signal level range that is a function of the noise of the pixel at location x,

is the signal level of the pixel at position y,

is the signal level of the pixel at position x,

is the absolute value of the difference between the signal level of the pixel (second pixel) at position y and the signal level of the pixel (first pixel) at position x,

is a signal level range parameter that is a function of the noise of the pixel at location x,

is a block of pixels with its center pixel at position x,

represents the sum over the pixels in the pixel block having its center pixel at position x.

In one aspect, the method includes receiving a first pixel of a first frame and adaptively noise filtering the first pixel using an estimated pixel noise parameter for the first pixel and using a difference between a signal level of the first pixel and a signal level of at least a second pixel to obtain a filtered pixel. The filtered pixel is output in an output frame. The method also includes outputting pixel noise corresponding to the filtered pixel in a noise output frame.

In another aspect, the second pixel is at a location in a second frame captured by the image sensor that is the same as the location of the first pixel in the first frame. The second frame is captured at a time prior to capturing the first frame. In this aspect, adaptive noise filtering the first pixel includes adaptively noise filtering the first pixel based on a comparison of a change in signal level of the first pixel and the second pixel over time, also referred to as a change in signal level of the first pixel over time, to an estimated pixel noise level of the first pixel. The estimated pixel noise level is an estimated pixel noise parameter in this aspect.

The step of adaptively noise filtering the first pixel includes inputting the signal level of the first pixel into a signal level vs. noise lookup table, and using the signal level vs. noise lookup table to output a noise standard deviation corresponding to the signal level of the first pixel. The noise standard deviation is an estimated pixel noise level of the first pixel in this aspect. The step of adaptively noise filtering the first pixel also includes dividing the time change in the signal level of the first pixel by the noise standard deviation to obtain the time change in the signal level of the first pixel as the standard deviation of the noise, the division being the comparison. The step of adaptively noise filtering the first pixel further includes inputting the time change in the signal level of the first pixel with the standard deviation of the noise into a modified lookup table, and using the modified lookup table to output the percentage of the allowed time change in the signal level of the first pixel that passes through the filter. Finally, the step of adaptively filtering the first pixel includes the steps of: generating an allowable change in the signal level of the first pixel by multiplying a percentage of the allowable change in the signal level of the first pixel by the change in the signal level of the first pixel over time; and generating a filtered pixel by adding the allowable change in the signal level of the first pixel to a corresponding pixel, the corresponding pixel being a pixel of a second frame, the second frame immediately preceding the first frame in time.

In yet another aspect, the at least one second pixel of the method is included in a plurality of pixels adjacent to the first pixel, and adaptive noise filtering the first pixel includes adaptive spatial noise filtering the first pixel based on a difference between a signal level of the first pixel and each of the signal levels of the plurality of adjacent pixels and based on a noise-dependent signal level parameter. The noise-dependent signal level parameter is a function of the noise of the first pixel, and the noise-dependent signal level parameter in this aspect is an estimated pixel noise parameter of the first pixel.

The step of adaptively spatial noise filtering the pixel includes generating the filtered pixel by combining a distance filter and a signal level range filter, the signal level range filter being a function of a noise-dependent signal level parameter. In particular, the signal level filter includes a Gaussian distribution of signal level ranges having the signal level range parameter. The signal level range parameter is a noise-dependent signal level parameter.

FIG. 1 is a diagram of a prior art computer-assisted surgery system. FIG. 1 is a diagram of an imaging system including an imaging pipeline having an adaptive noise filter. A method for implementing an adaptive temporal noise filter. FIG. 2 is a diagram of an adaptive temporal noise filter. FIG. 4 is a graphical representation of data contained in a signal level versus noise lookup table in the embodiment of FIGS. 3A and 3B. FIG. 4 is a graphical representation of the data contained in the modified lookup table in the embodiment of FIGS. 3A and 3B. FIG. 2 is a diagram of an imaging pipeline including multiple adaptive noise filters. A method for implementing an adaptive spatial noise filter. FIG. 2 is a diagram of an adaptive spatial noise filter.

In the drawings, the first digit of a three-digit reference number is the number of the figure in which the element with that reference number first appears.

An imaging pipeline, such as the imaging pipeline in endoscopic imaging system 192, includes many noise filters. These noise filters include parameters that are based on the noise expected in the scene. While these noise filters work well for captured frames having a given signal level, e.g., the signal level for which the noise-based parameters are selected, these noise filters have problems processing frames captured at different signal levels and image frames that include varying signal levels.

These problems are solved by an imaging pipeline that includes one or more adaptive noise filters. Each of the adaptive noise filters selects a noise-dependent parameter that is related to a signal level on a local basis, as measured by an image sensor in an image capture unit, e.g., a camera. The noise-dependent parameters used in each of the adaptive noise filters may vary from image frame to image frame and may vary from pixel to pixel within an image frame. Specifically, the adaptive noise filters adjust the noise filtering based on the signal level of a region of interest in the frame. In some examples, the region of interest is a pixel, while in other examples, the region of interest is a group of pixels, where the group of pixels includes more than one pixel and less than the number of pixels in the frame. Each of the adaptive noise filters adaptively filters a pixel by comparing a value of a first pixel to a value of at least a second pixel, e.g., a temporal or spatial change, and an estimated pixel noise parameter for that pixel, where the estimated pixel noise parameter for the pixel is based on a number less than the total number of pixels in the frame of pixel data. Because the estimated pixel noise may vary from pixel to pixel, the adaptive filter compensates for such changes in the estimated pixel noise parameter.

ここで、
ｓｔｄ（Ｎｏｉｓｅ）は、画素のノイズの標準偏差であり、
Ｏは、センサの電気ノイズによるオフセットであり、
Ｇは、ゲイン値であり、
ＳｉｇｎａｌＬｅｖｅｌは、画像取込みセンサによって読み取られる画素信号レベルであり、
ＭａｘＳｉｇｎａｌＬｅｖｅｌは、画像取込みセンサの最大画素信号レベルである。
ここで、画素信号レベル（ＳｉｇｎａｌＬｅｖｅｌ）は、画像取込みセンサの画素に入射する光の尺度である。上の式は、低画素信号レベルでは、ノイズが高画素信号レベルよりもはるかに小さいことを示している。しかしながら、低信号レベルでは、ノイズは高信号レベルよりも信号のはるかに大きな部分を占める。 In image capture sensors used in endoscopes and other miniature image capture sensors, a pixel counts the photons that are incident on it during an exposure time. Thus, the signal level of a pixel, sometimes called the pixel value, includes the count of incident photons over the exposure time. However, this count, or signal level, may also include a contribution related to pixel noise. Pixel noise is dominated by electronic (shot) noise, and there is also a fundamental contribution to pixel noise due to electrical variations in the image capture sensor. Pixel noise is estimated as a square root function that deals with the variance in the number of electrons that strike the image capture sensor, for example, an estimated pixel noise parameter is the estimated pixel noise level. The estimated pixel noise level, sometimes called the estimated pixel noise in terms of a standard deviation number, can be estimated as follows:

Where:
std(Noise) is the standard deviation of the noise of the pixels,
O is the offset due to electrical noise of the sensor,
G is the gain value,
SignalLevel is the pixel signal level read by the image capture sensor,
MaxSignalLevel is the maximum pixel signal level of the image capture sensor.
Here, pixel signal level (SignalLevel) is a measure of the light incident on a pixel of an image capture sensor. The above equation shows that at low pixel signal levels, the noise is much smaller than at high pixel signal levels. However, at low signal levels, the noise is a much larger portion of the signal than at high signal levels.

For example, consider a first pixel with an offset O of 2, a gain G of 40, a pixel signal level of 6, and a second pixel with a pixel signal level of 4000. In this example, 12-bit pixels are used, so the Max Signal Level is 4095.
std(1st pixel noise)=2+40*(6/4095)**.5=3.5
std(2nd pixel noise)=2+40*(4000/4095)**.5=41.5

Thus, the standard deviation of the estimated pixel noise of the second pixel is more than 10 times the standard deviation of the estimated pixel noise of the first pixel. However, the standard deviation of the estimated pixel noise of the first pixel is about 58% of the pixel signal level of the first pixel, while the standard deviation of the estimated pixel noise of the second pixel is about 1% of the pixel signal level of the second pixel.

In one aspect, the image capture unit 201 captures a frame of pixel data, which is processed by the imaging pipeline 230 of the imaging system 292. Typically, the imaging pipeline 230 includes multiple stages. The imaging pipeline 230 of the imaging system 292 is similar to the imaging pipeline of the conventional imaging system 192, except that at least one noise filter in a stage of the imaging pipeline of the conventional imaging system 192 is replaced with an adaptive noise filter 210.

A stage input frame of pixel data 214 (sometimes referred to as an input frame 214) is a function of pixel data input to a particular stage of the imaging processing pipeline 230. The adaptive noise filter 210 in the imaging pipeline 230 of the imaging system 292 operates by determining whether a change in signal level of a pixel in the input frame 214 is noise or a real change. A real change is a change due to a real change in signal level, not a change due to noise. If the adaptive noise filter 210 determines that a change in signal level of a pixel is due to noise, the adaptive noise filter 210 filters the pixel and passes the filtered pixel to the frame of filtered pixel data 212. Conversely, if the adaptive noise filter 210 determines that a change in signal level of the pixel is due to a change in signal level, i.e., a real change, the adaptive noise filter 210 passes the unfiltered pixel to the frame of filtered pixel data 212. The adaptive noise filter 210 determines that a change in a pixel's signal level is due to noise by comparing the change in signal level associated with the pixel to an estimated pixel noise parameter for that pixel, and then filtering the pixel based on the comparison.

Thus, the adaptive noise filter 210 examines each pixel of at least the region of interest in the frame 202 captured by the image capture sensor of the image capture unit 201 and processes the signal level changes at each pixel, e.g., changes over time or spatial changes between adjacent pixels, based on whether the signal level changes are deemed to be caused by noise alone. This adaptive noise filtering causes the display unit 220 to display a better image than those obtained using prior art noise filters.

In prior art noise filters, when the parameters used for the noise filter are intended for low signal level noise, noise in pixels exhibiting high signal levels simply passes through the filter. As a result, high signal level noise in a scene displayed on a display unit can cause bright areas of the displayed scene to vary from frame to frame, which in turn can cause noise-induced flicker in the displayed scene.

Conversely, if the parameters used in the prior art noise filters are targeted at high signal level noise, then pixels with low signal levels will be filtered out because they are all determined to be noise. As a result, information from the low signal level pixels will be lost.

The adaptive noise filter described herein overcomes these problems by adjusting the noise filtering based on the signal level of each pixel. Thus, filtering for low signal levels differs from filtering for high signal levels, where the estimated pixel noise of a pixel changes as the signal level changes, and the filtering is adjusted to preserve the information conveyed by each pixel while smoothing the noise contribution so that the effect of the pixel noise in the displayed image is not visible. For example, the adaptive noise filter 210 uses the signal level of a pixel to access the noise of that pixel in signal level vs. noise data 211, which has appropriate noise parameters for each signal level across the possible range of signal levels for the pixel. The adaptive noise filter 210 ensures that each pixel is appropriately filtered based on the noise of that pixel's signal level by comparing the change in the pixel's signal level to the noise level from the signal level vs. noise data 211 for that pixel. This significantly reduces or eliminates problems encountered with prior art noise filters that attempt to use the same parameters across a complete scene in a frame of pixel data.

In one aspect, the adaptive noise filter 210 stores a noise value, e.g., pixel noise, for each pixel in an output frame of pixel noise data 213 (sometimes referred to as a noise frame 213). In one aspect, the values of the pixel noise data 213 are expressed as noise standard deviations. The noise frame 213 passes through the pipeline 230 and is processed by each stage of the pipeline 230. As a result, each stage of the pipeline 230 has a representation of the noise input to that stage, which may be used for any adaptive noise filtering operations performed at that stage.

As described more fully below, in one aspect, the imaging pipeline 230 includes an adaptive temporal noise filter in an early stage of the imaging pipeline 230. When the adaptive noise filter 210 is an adaptive temporal noise filter, the stage input frames of pixel data 214 are input frames of raw pixel data from an image capture sensor in the image capture unit 201.

In another aspect, the imaging pipeline 230 includes an adaptive spatial noise filter at a later stage in the imaging pipeline 230. When the adaptive noise filter 210 is an adaptive spatial noise filter, the stage input frame of pixel data 214 is, for example, an input frame of demosaiced and color transformed pixel data.

In yet another aspect, the imaging pipeline 230 includes both an adaptive temporal noise filter in an early stage of the imaging pipeline and an adaptive spatial noise filter in a later stage of the imaging pipeline. See FIG. 5.

When the adaptive noise filter 210 is an adaptive temporal noise filter, the value of a current pixel (first pixel) at a position in a current frame of raw pixel data is compared to the value of a previously filtered pixel (second pixel) at the same position in a frame of data that was filtered immediately before the current frame in time. Here, raw pixel data refers to data from the image capture unit 201 that has not yet been demosaiced. The difference between the two values of the two pixels, i.e., the change in the current pixel over time, is used to determine whether the difference in the values of the two pixels is due to noise or an actual change.

In one aspect, the standard deviation of the noise (estimated pixel noise parameter) relative to the value (signal level) of the current pixel is obtained from the signal level vs. noise data 211 by the adaptive noise filter 210. The time change of the current pixel is then divided by the standard deviation of the noise to obtain the time signal change relative to the value of the standard deviation of the noise. The time signal change relative to the value of the standard deviation of the noise is used to determine what percentage of the time change of the current pixel can pass through the filter 210. Thus, the filter 210 compares the time signal change of the current pixel with the estimated pixel noise parameter, e.g., the estimated pixel noise level, of the current pixel and filters based on the comparison. In one aspect, the adaptive temporal noise filtered pixel is defined as the percentage of the time change of the current pixel plus the value of the second pixel. The value of the adaptive temporal noise filtered pixel is placed in the stage output frame 212 of filtered pixel data (sometimes referred to as the output frame 212). If the percentage of the allowable time change of the current pixel that can pass through the filter 210 is 100%, the current pixel passes through the adaptive temporal noise filter unchanged. If the allowable time change percentage of the current pixel that can pass through the filter 210 is less than 100%, the current pixel is filtered based on its signal level. The output frame 212 continues through the imaging pipeline 230 and is displayed on the display unit 220. Thus, a frame based on the output frame 212 is sent by the imaging pipeline 230 to the display unit 220. That is, the frame sent to the display unit 220 is the output frame 212 after the output frame 212 has been processed by subsequent stages in the imaging pipeline 230.

When the adaptive noise filter 210 is an adaptive spatial noise filter, the value of the current pixel at a location in the current frame of demosaiced and color transformed pixel data is compared to the values of neighboring pixels in the current frame. The percentage of spatial variance of the current pixel that passes the filter as an adaptive spatial noise filtered pixel is weighted based on the spatial relationship of the current pixel to the neighboring pixels, based on the relationship of the signal level of the current pixel to the signal levels of the neighboring pixels, and based on the noise component of the current pixel. In one aspect, as described more fully below, the adaptive spatial noise filter is configured to filter the pixel based on the spatial variance between the signal level of the pixel and the signal levels of multiple neighboring pixels, and based on a noise-dependent signal level parameter. The noise-dependent signal level parameter is a function of the noise of the current pixel. Here, in this aspect, the noise-dependent signal level parameter is an estimated pixel noise parameter for the pixel. The stage output frame 212 continues through the imaging pipeline 230 and is displayed on the display unit 220.

Figure 3A is a process flow diagram of one implementation of an adaptive temporal noise filter that may be used in the imaging pipeline 230. Figure 3B is a block diagram of one implementation of the adaptive temporal noise filter 300 of the method 350 of Figure 3A.

In this example, pixels in a current pixel frame 315 (see FIG. 3B) are described as being processed sequentially. This is for illustrative purposes only and is not intended to be limiting. In light of the present disclosure, multiple pixels may be processed in parallel, pixels in a frame may be processed as a group, or some other matter. The process flow diagram of FIG. 3A is intended to show that each pixel in the current pixel frame 315 (see FIG. 3B) is processed to determine whether a temporal change in the current pixel is due to noise. In some aspects, it is not necessary to process all pixels in the frame. Thus, at least pixels in a region of interest of the frame, e.g., multiple pixels in the frame, are typically processed by the adaptive temporal noise filter 300, where the region of interest is known and designated to be a particular portion of the pixel frame.

First, a new frame search process 301 retrieves a new pixel frame 302 from the image capture sensor in the image capture unit 210. The retrieved new pixel frame 302 is saved as a current pixel frame 315 (see FIG. 3B). The new frame search process 301 transfers to a pixel acquisition process 303.

The pixel acquisition process 303 retrieves a first pixel from a current pixel frame 315 (see FIG. 3B) and a second pixel corresponding to the first pixel from a stored previously filtered pixel frame 304. Sometimes, the first pixel from the current pixel frame 315 is called the current pixel. The stored previously filtered pixel frame 304 is a pixel frame that was filtered at a time immediately before the time associated with the current pixel frame 315. If the current pixel frame 315 is a frame for time t, then the stored previously filtered pixel frame 304 is a frame for time t-1. Here, the second corresponding pixel from the stored previously filtered pixel frame 304 is a pixel that is at the same position in the frame 304 as the position of the first pixel in the frame 315. Here, when a pixel is said to be retrieved, it is that the value of the pixel is retrieved. The value of the pixel represents a signal level. The pixel acquisition process 303 passes to a frame-to-frame pixel difference process 305.

The frame to frame pixel difference process 305 first determines the time change between a first pixel and a second pixel, e.g., the time change of the value of the first pixel at time t from the value of the second pixel at time t-1. The frame to frame pixel difference process 305 provides the time change of the value of the first pixel to the pixel generation process 309.

Then, the frame pixel difference process 305 takes the absolute value of the time change of the first pixel. The frame pixel difference process 305 provides the absolute value of the time change of the first pixel to the signal level modification process 307.

The signal level modification process 307 not only receives the absolute value of the time change of the first pixel, but also receives a value from the signal level to noise look-up table 306 (upper and lower case letters are not used to distinguish elements herein). For example, the signal level to noise look-up table 306 (see FIG. 3A), the signal level to noise look-up table 306 (see FIG. 3B), the signal level to noise look-up table 306 (see FIG. 3C), the signal level to noise look-up table 306 (see FIG. 3D), the signal level to noise look-up table 306 (see FIG. 3E), the signal level to noise look-up table 306 (see FIG. 3F), the signal level to noise look-up table 306 (see FIG. 3G), the signal level to noise look-up table 306 (see FIG. 3H ...
The signal level to noise look-up table 306 (see FIG. 3B) and the signal level to noise look-up table 306 all have the same elements.

4A is a graphical representation of the data in the signal level vs. noise lookup table 306 in one embodiment. In this embodiment, the noise specification (1) above with an offset O of 2, a gain G of 40, and a Max Signal Level of 4095 was used to generate the curve 401. The curve 401 is merely illustrative and is not intended to be limited to this particular example. In light of this disclosure, one skilled in the art can generate a signal level vs. noise lookup table 306 that represents the characteristics and associated gains of the image sensor used to provide frames to the imaging pipeline.

In FIG. 4A, the signal level is plotted along the horizontal x-axis and the standard deviation of the noise is plotted along the vertical y-axis. Thus, when the signal level vs. noise lookup table 306 receives a current pixel value from the pixel acquisition process 303, which is a value along the x-axis, the signal level vs. noise lookup table 306 outputs a curve value 401 along the y-axis for the current pixel value. For example, assume that the signal level of the current pixel (current pixel value) is 1750. When the signal level vs. noise lookup table 306 receives an input value of 1750, the signal level vs. noise lookup table 306 outputs a standard deviation value of 28.1 for the current pixel. This lookup is represented by the dotted line in FIG. 4A.

In one aspect, the signal level vs. noise lookup table 306 includes an entry for each possible signal level of the current pixel. However, a fewer number of entries may be used and interpolation between entries may be used. The signal level vs. noise lookup table 306 provides the standard deviation of the noise for the signal level of the current pixel to the signal level modification process 307 as well as to the pixel noise generation process 311.

The signal level modification process 307 determines the standard deviation value included in the time change of the first pixel, i.e., divides the absolute value of the time change of the value of the first pixel (the current pixel) by the noise standard deviation input value received from the signal level vs. noise lookup table 306. The change in the value of the first pixel is thus compared to the estimated pixel noise level of the first pixel, which in this example is the estimated pixel noise parameter. The standard deviation value included in the absolute value of the time change is input to the modification lookup table 308.

A graphical representation of the data contained in the modification lookup table 308 is shown in FIG. 4B. The range of possible input values of the temporal signal change of the current pixel in standard deviations of the noise is shown along the horizontal x-axis, in this example the possible input values range from zero to four standard deviations. Curve 402 represents the output of the percentage change that can pass the adaptive temporal noise filter for each of the possible input values, and the value of the curve for a given input is read off the vertical y-axis as the percentage of the temporal change of the current pixel that can pass the filter.

As shown by curve 402, for a time change of a current pixel of 1 noise standard deviation or less, approximately 19 percent of the time change of the current pixel, i.e., a first fixed constant percentage, is allowed to pass through the filter. For a time change of a current pixel of 3 noise standard deviations or more, 100 percent of the time change of the current pixel, i.e., a second fixed constant percentage, is allowed to pass through the filter, i.e., the current pixel passes through the filter unfiltered. For a time change of a current pixel of 2 noise standard deviations or more but less than 3 noise standard deviations, the percentage of time change allowed to pass through the filter varies linearly from approximately 19 percent of the time change of the current pixel for 1 noise standard deviation to 100% for 3 noise standard deviations.

For example, for an input value of 2 standard deviations of the noise in the time evolution of the current pixel, the modification lookup table 308 outputs a value of approximately 59%, which is the percentage of the time evolution of the current pixel that is allowed to pass through the filter. An example of this lookup is represented by the dotted line in FIG. 4B.

In one aspect, the modified lookup table 308 is not used and is replaced with a piecewise function as shown in FIG. 4B. For curves more complex than those shown in FIG. 4B, the curve is implemented with a modified lookup table 308 having approximately 512 entries.

Thus, in response to the standard deviation value included in the absolute value of the time change of the current pixel input to the modification lookup table 308, the modification lookup table 308 outputs a corresponding percentage change that allows the current pixel to pass the adaptive temporal noise filter. The percentage change that allows the current pixel to pass the adaptive temporal noise filter is provided to the pixel generation process 309 and the pixel noise generation process 311.

The pixel generation process 309 receives as input the percentage of time change of the current pixel allowed to pass the filter from the modified lookup table 308, the time change of the current pixel from the frame-to-frame pixel difference process 305, and the value of the second pixel from the stored previously filtered pixel frame 304. The pixel generation process 309 multiplies the percentage of time change of the current pixel allowed to pass the filter from the modified lookup table 308 by the time change of the current pixel from the frame-to-frame pixel difference process 305, and then adds the result to the value of the second pixel from the stored previously filtered pixel frame 304 to obtain an adaptive temporal noise filtered pixel that is passed to the pixel output frame 310. Thus, an adaptively filtered output pixel is output based on the comparison described above.

The pixel noise generation process 311 receives as input the percent change allowed through the adaptive temporal noise filter, the noise standard deviation for the signal level of the current pixel (stdCurr), and the noise standard deviation for the signal level of the corresponding filtered pixel in the previous frame in time (stdPrev). The noise standard deviation for the signal level of the corresponding pixel in the previous frame in time (stdPrev) is received from a stored previous pixel noise frame 314, which is a frame of pixel noise standard deviations generated by the previous adaptive noise filter.

The pixel noise generation process 311 generates the pixel noise standard deviation (stdOut) for the current filtered pixel. In this aspect, the pixel noise standard deviation (stdOut) for the current filtered pixel is defined as follows:

The pixel noise generation process 311 writes the pixel noise standard deviation (stdOut) for the current filtered pixel to the pixel noise output frame 312 (sometimes called the noise output frame 312). The current pixel noise represents the noise component of the adaptive temporal noise filtered pixel. The pixel noise generation process 311 passes processing to the final pixel confirmation process 313.

The Last Pixel Check Process 313 determines whether all pixels in the current pixel frame have been filtered. If all pixels have been filtered, the Last Pixel Check Process 313 passes to the New Frame Search Process 301, otherwise it passes to the Pixel Acquisition Process 303.

Once all pixels of interest in a frame have been filtered, the pixel output frame 310 (called the filtered pixel frame) is moved to the stored previously filtered pixel frame 304, and the pixel noise output frame 312 is moved to the stored previous pixel noise frame 314 (sometimes called the stored previous noise frame 314).

The method 350 of FIG. 3A adaptively temporally noise filters each pixel in at least one region in a frame and adjusts the filter based on the estimated pixel noise for that pixel. The method further generates a pixel noise output frame that is processed by each stage of the imaging pipeline until another filter stage is reached, such that the noise contribution to each pixel is available to each filter stage of the imaging pipeline. Each filter stage includes an option to generate a pixel noise output frame from that stage.

3A. The adaptive temporal noise filter 300 includes a first adder 320, an absolute value module 321, a divider 322, a multiplier 323, a second adder 324, second through seventh multipliers 325, 326, 328, 329, 330, 331, a third and fourth adder 327, 332, and a square root module 333. The adaptive temporal noise filter 300 is connected to a current pixel frame 315, a stored previously filtered pixel frame 304, a signal level vs. noise lookup table 306, a modification lookup table 308, a pixel output frame 310, a pixel noise output frame 312, and a stored previous pixel noise frame 314. In one aspect, each of the current pixel frame 315, the stored previously filtered pixel frame 304, the signal level vs. noise lookup table 306, the modification lookup table 308, the pixel output frame 310, the pixel noise output frame 312, and the stored previous pixel noise frame 314 are stored in memory. In this example, the pixel output frame 310 is coupled to the stored previously filtered pixel frame 304 so that the pixel output frame 310 can be moved to the stored previously filtered pixel frame 304. Similarly, the pixel noise output frame 312 is coupled to the stored previous pixel noise output frame 314 so that the pixel noise output frame 312 can be moved to the stored previous pixel noise output frame 314.

The current pixel frame 315, the stored previously filtered pixel frame 304, the signal level vs. noise lookup table 306, the modified lookup table 308, the pixel output frame 310, the pixel noise output frame 312, and the stored previous pixel noise frame 314 are shown separately from the adaptive temporal noise filter 300 for illustrative purposes only. For example, the adaptive temporal noise filter 300 may be shown in FIG. 3B to include all or some combination of the current pixel frame 315, the stored previously filtered pixel frame 304, the signal level vs. noise lookup table 306, the modified lookup table 308, the pixel output frame 310, the pixel noise output frame 312, and the stored previous pixel noise frame 314.

To implement the pixel acquisition process 303, a signal on line (Control) drives a first pixel from a current pixel frame 315 (sometimes referred to as frame 315) on a first input (In1) of an adder 320 of the adaptive temporal noise filter 300, and drives a second pixel from a stored previously filtered pixel frame 304 (sometimes referred to as frame 304) to an inverted second input (In2) of the adder 320. As mentioned above, the first pixel is sometimes referred to as the current pixel. Also, when a pixel is said to be driven or provided to an input, it means that the value of the pixel is on the input.

The current pixel is also provided to an input (In) of the signal level vs. noise lookup table 306 of the adaptive temporal noise filter 300. A second pixel from the stored previously filtered pixel frame 304 is also provided to a second input (In2) of the second adder 324.

The adder 320 subtracts the value of the second pixel from the stored previously filtered pixel frame 304 from the value of the first pixel from the current pixel frame 315. The difference between these two pixels, i.e., the time change of the first pixel, is provided to a first input (In1) of the multiplier 323 of the adaptive temporal noise filter 300 and to an input (IN) of an absolute value module 321. The absolute value module 321 takes the absolute value of the time difference between the two pixels. The absolute value of the time difference between the two pixels is provided to a (fraction) numerator input (NUM) of a divider 322. In this embodiment, the adder 320 and the absolute value module 321 are used to implement the frame-to-frame pixel difference process 305.

In response to the value of the current pixel on the input (In) of the signal level vs. noise lookup table 306, the signal level vs. noise lookup table 306 outputs on the input (In) the noise standard deviation corresponding to the input signal level as represented by the value of the current pixel. The output noise standard deviation is provided to a second input (input (DENOM)) of the divider 322 and to both inputs of the multiplier 325.

The signal level modification process 307 is implemented in the adaptive temporal noise filter 300 by the divider 322. As mentioned above, the absolute value of the difference between two pixels, i.e. the absolute value of the time change of the current pixel, is received at a first input (Num) of the divider 322. As just explained, the signal level vs. noise lookup table 306 (see FIG. 3B) receives the value of the current pixel on an input (In) and receives the noise standard deviation corresponding to the input value in response to the input output, see FIG. 4A. The noise standard deviation is provided to a second input (Denom) of the divider 322.

The divider 322 divides the value of the first input (Num) by the value of the second input (Denom) and the result is at the output (Out). The result on the output (Out) of the divider 322 is fed to the input (In) of the modification lookup table 308. Thus, the divider 322 divides the absolute value of the time change of the current pixel by the noise standard deviation of the current pixel to obtain the time signal change in the noise standard deviation which is input to the modification lookup table 308.

The modified lookup table 308 outputs the percentage of the time change that can pass through the filter 300 in response to the input time signal change in the standard deviation of the noise. See FIG. 4B. The percentage of the time change that can pass through the filter 300 is provided to the second input (In2) of the multiplier 323, the inverted second input (In2) of the adder 327, and both inputs of the multiplier 326.

Thus, the multiplier 323 receives the proportion of the time change that can pass the filter 300 on a second input (In2) and the time change of the first pixel on a first input (IN1). The multiplier 323 multiplies these two inputs to obtain the time change that can pass the filter 300. The time change that can pass the filter 300 is output at the output (Out) of the multiplier 323 and is supplied to a first input (In1) of a second adder 324.

The adder 324 receives the temporal change that can pass through the filter 300 on a first input (In1) and the value of the second pixel from the stored previously filtered pixel frame 304 on a second input (In2). (As used herein, the second pixel from the stored previously filtered pixel frame 304 does not literally refer to the second pixel in that frame, but rather refers to a pixel from the stored previously filtered pixel frame 304 that has a position that is the same as, i.e., corresponds to, the position of the current pixel in the current pixel frame 315.) The adder 324 adds the temporal signal change that can pass through the filter 300 to the value of the second pixel from the stored previously filtered pixel frame 304 and outputs the sum at an output (Out). The sum is the adaptive temporal noise filtered pixel value. The output (Out) of the adder 324 is connected to the pixel output frame 310. Thus, the adaptive temporal noise filtered pixel is written to the pixel output frame 310. The multiplier 323 and the adder 324 perform the pixel generation process 309.

The pixel output frame 310 is processed by the remaining stages of the imaging pipeline (see FIG. 5) and then sent to a display unit. Since the frame of pixel data sent to the display unit is the result of the imaging pipeline processing the pixel output frame 310, the frame of pixel data sent to the display unit is said to be based on the pixel output frame 310. Thus, the frame of pixel data based on the pixel output frame 310 is displayed by the display unit and, as described above, provides a better image than would be obtained with non-adaptive temporal noise filtering.

The noise standard deviation (stdCurr) of the current pixel is present at both input terminals (In1) and (In2) of multiplier 325. Multiplier 325 multiplies these two values to generate the squared noise standard deviation (stdCurr2). The squared noise standard deviation (stdCurr2) is driven on an output terminal (Out) that is connected to the first input terminal (In1) of multiplier 330.

The change in time that allows the filter to pass is present at both input terminals (In1) and (In2) of multiplier 326. Multiplier 326 multiplies these two values to generate the square of the change in time that allows the filter to pass (change2). The square of the change in time that allows the filter to pass (change2) is driven on an output terminal (Out) that is connected to a second input terminal (In2) of multiplier 330.

The first input terminal (In1) of the adder 327 receives a "1" as an input, and the inverted second input (In2) receives an input of the time change allowed to pass the filter, expressed as a number between zero and one. The adder subtracts the percentage of time change allowed to pass the filter from one to generate a value (1-change). The value (1-change) is driven on an output terminal (Out) that is connected to the first input terminal (In1) and the second input terminal (In2) of the multiplier 329.

The signal on the line for the stored previous pixel noise frame (Control) drives the previously filtered pixel noise standard deviation (stdPrev) corresponding to the current pixel on both input terminals (In1) and (In2) of the multiplier 328. The previously filtered pixel noise standard deviation (stdPrev) corresponding to the current pixel is also the previously filtered pixel noise standard deviation (stdPrev) for the same position of the current pixel in the current pixel frame 315 as it is in the stored previous pixel noise frame 314. The multiplier 328 multiplies these two values to generate the previously filtered pixel noise standard deviation squared (stdPrev2). The previously filtered pixel noise standard deviation squared (stdPrev2) is driven on an output terminal (Out) that is connected to the second input terminal (In2) of the multiplier 331.

The value (1-change) is present at both the first input terminal (In1) and the second input terminal (In2) of multiplier 329. Multiplier 329 multiplies the two values to generate the value (1-change)2. The value (1-change)2 is driven on the output terminal (Out) which is connected to the first input terminal (In1) of multiplier 331.

The square of the time change (change2) that allows the filter to pass is on the second input terminal (In2) of the multiplier 330, and the square of the noise standard deviation of the current pixel (stdCurr2) is on the second input terminal (In2) of the multiplier 330. The multiplier 330 multiplies these two values to generate the value (change2*stdCurr2). The value (change2*stdCurr2) is driven on an output terminal (Out) that is connected to a first input terminal (In1) of the adder 332.

The square of the noise standard deviation (stdPrev2) of the previously filtered pixel corresponding to the current pixel is present on a second input terminal (In2) of the multiplier 331 and the value (1-change)2 is present on a first input terminal (In1) of the multiplier 331. The multiplier 331 multiplies the two values to generate the value ((1-change)2*stdPrev2). The value ((1-change)2*stdPrev2) is driven on an output terminal (Out) which is connected to a second input terminal (In2) of the adder 332.

Thus, the adder 332 receives the value (1-change)2*stdPrev2 on its second input terminal (In2) and the value change2*stdCurr2 on its first input terminal (In1). The adder 332 adds these two values to generate the value ((1-change)2*stdPrev2+change2*stdCurr2). The value ((1-change)2*stdPrev2+change2*stdCurr2) is driven on an output terminal (Out) that is connected to the input terminal (In) of the square root module 333.

The square root module 333 takes the square root of the value ((1-change)2*stdPrev2+change2*stdCurr2) and stores the result (stdOut) in a location in the pixel noise output frame 312 that corresponds to the location of the current pixel in the current pixel frame 315, i.e., in a location in the pixel noise output frame 312 that corresponds to the current pixel. The second through seventh multipliers 325, 326, 328, 329, 330, 331, the third and fourth adders 327, 332, and the square root module 333 implement the pixel noise generation process 311.

Figure 5 is a block diagram of representative stages 501-504 of an imaging pipeline 530 that includes multiple adaptive noise filters. The imaging pipeline 530 is an example of the imaging pipeline 230. In this example imaging pipeline, an adaptive temporal noise filter 300 is included in a first stage 501 and an adaptive spatial noise filter 570 is included in a fourth stage 504. The imaging pipeline 530 outputs a frame to a display unit based on the output of the adaptive noise filters of the pipeline.

In the example of FIG. 5, the image capture unit 201 includes an image capture sensor with a Bayer red-green-blue color filter. The Bayer pixel data-in frame 515 is the current pixel frame of the adaptive temporal noise filter 300. Here, the Bayer pixel data-in frame 515 includes a set of red pixel data, a set of green pixel data, and a set of blue data. However, in stage 501, the temporal filtering does not depend on the color attributes of a particular set of pixels, but only on the signal level of each pixel.

3A and 3B, each pixel in the Bayer pixel data input frame 515 accesses the signal level vs. noise lookup table 506, which outputs the noise standard deviation (estimated pixel noise parameter) for each pixel to the Bayer pixel noise in frame 516. The signal level vs. noise lookup table 506 is similar to the signal level vs. noise lookup table 306.

The adaptive temporal noise filter 300 receives as input the Bayer pixel data input frame 515 and the Bayer pixel noise in frame 516, and uses the modified lookup table 508 as described above for the modified lookup table 308. The modified lookup table 508 is similar to the modified lookup table 308. The adaptive temporal noise filter 300 generates an adaptive temporal noise filtered pixel in the Bayer pixel data output (data-out) frame 510 for each pixel in the Bayer pixel data input frame 515, and generates pixel noise in the Bayer pixel noise output (noise out) frame 512 for each pixel in the Bayer pixel data input frame 515, as described above with respect to Figures 3A and 3B. Therefore, the description will not be repeated here.

The Bayer pixel data output frame 510 and the Bayer pixel noise output frame 512 are input frames to the second stage 502 (demosaic stage) of the imaging pipeline 530. The demosaic process 550 processes the adaptive temporal noise filtered pixels of the Bayer pixel data output frame 510 to generate a full resolution red pixel frame (imRed), a full resolution green pixel frame (imGreen), and a full resolution blue pixel frame (imBlue). The demosaic process 550 also processes the pixel noise of the Bayer pixel noise data output frame 512 to generate a full resolution red pixel noise frame (imNoiseRed), a full resolution green pixel noise frame (imNoiseGreen), and a full resolution blue pixel noise frame (imNoiseBlue). The process of demosaicing a set of Bayer red, green, and blue pixels to obtain full resolution frames of red, green, and blue pixels is well known and will not be discussed in further detail.

The full resolution red pixel frame (imRed), full resolution green pixel frame (imGreen), full resolution blue pixel frame (imBlue), full resolution red pixel noise frame (imNoiseRed), full resolution green pixel noise frame (imNoiseGreen), and full resolution blue pixel noise frame (imNoiseBlue) are input frames to the third stage (color space conversion stage) of the imaging pipeline 530. The color conversion process 560 converts the red, green, and blue pixels, i.e., color pixels in a first color space, into a luminance pixel Y and two chrominance pixels U, V, i.e., color pixels in a second color space. The conversion from RGB color space to YUV color space is well known and therefore will not be considered in further detail.

The color conversion process 560 converts the full resolution red pixel frame (imRed), full resolution green pixel frame (imGreen), and full resolution blue pixel frame (imBlue) into a full resolution luminance pixel frame (imY), a first full resolution chrominance pixel frame (imU), and a second full resolution chrominance pixel frame (imV). The color conversion process 560 also converts the full resolution red pixel noise frame (imNoiseRed), full resolution green pixel noise frame (imNoiseGreen), and full resolution blue pixel noise frame (imNoiseBlue) into a full resolution luminance pixel noise frame (imNoiseY), a first full resolution chrominance pixel noise frame (imNoiseU), and a second full resolution chrominance pixel noise frame (imNoiseV).

The full resolution luminance pixel frame (imY), the first full resolution chrominance pixel frame (imU), the second full resolution chrominance pixel frame (imV), the full resolution luminance pixel noise frame (imNoiseY), the first full resolution chrominance pixel noise frame (imNoiseU), and the second full resolution chrominance pixel noise frame (imNoiseV) are inputs to the fourth stage 504 (adaptive spatial noise filter stage) of the imaging pipeline 530.

One of the reasons for converting from RGB color space to YUV color space is that most of the luminance information important to human visual perception is preserved in the luminance component Y of the YUV color space. Thus, in stage 504, the adaptive spatial noise filter 570 only adaptively filters the spatial noise of the luminance component pixel. This improves the final displayed image without the time and expense of adaptively filtering the spatial noise of the two chrominance components of the pixel. However, if no color transform is used in the imaging pipeline 530, or if the color transform is converted to another color space that does not include a luminance component, the adaptive spatial noise filter 570 adaptively spatially filters the pixels of each color component.

The full resolution luma pixel frame (imY) and the full resolution luma pixel noise frame (imNoiseY) are inputs to the adaptive spatial noise filter 570. The value of a pixel in the full resolution luma pixel noise frame (imNoiseY) represents the noise component of the corresponding pixel in the full resolution luma pixel frame (imY), i.e., the location of the pixel noise in frame (imNoiseY) is the same as the location in frame (imY). Because the pixel noise value in frame (imNoiseY) is used in specifying a filter for the corresponding pixel in frame (imY), the filter is said to be an adaptive spatial noise filter. The filter is adapted to the spatial noise associated with each pixel, rather than assuming that the same constant parameters for all pixels can be used in the spatial noise filter.

The adaptive spatial noise filter 570 generates an output frame of adaptive spatial noise filtered pixels (imYout) and an output frame of pixel noise (imNoiseYout). The chrominance pixel frame (imU) and the chrominance pixel frame (imV) are output unfiltered as chrominance pixel frame (imUout) and chrominance pixel frame (imVout), respectively. The chrominance pixel noise frame (imNoiseU) and the chrominance pixel noise frame (imNoiseV) are output unfiltered as chrominance pixel noise frame (imNoiseUout) and chrominance pixel noise frame (imNoiseVout). In one aspect, the adaptive spatial noise filtered frames of pixels (imYout), chrominance pixel frame (imUout), and chrominance pixel frame (imVout) are output from the pipeline stage 504. These frames can be further processed in the pipeline and sent to a display device, or in one aspect can be sent to a display device.

ここで、

は、フレーム内の位置ｘにおける画素のフィルタ処理された信号レベルである。
適応空間ノイズフィルタ５７０は、現在の画素をどの様にフィルタ処理するかを決定するために、現在の画素の周りの正方形の画素のブロック、例えば、複数の隣接画素を使用する。距離フィルタ（Distance Filter）は、現在の画素からのブロック内の隣接画素のユークリッド距離から決定された重みに基づいて、現在の画素をフィルタ処理する。信号レンジフィルタ（時には、レンジフィルタ（Range Filter）又は強度フィルタとも呼ばれる）は、フィルタ処理される画素に対応するフレーム（imNoiseY）内の画素ノイズの値に基づく適応ノイズ信号レンジフィルタである。こうして、レンジフィルタで使用される重みは、ブロック内の隣接画素と現在の画素との間の信号レベルの差だけでなく、現在の画素のノイズ成分にも基づく。入力画素フレーム内の現在の画素の位置と同じ位置にある画素ノイズ入力フレームの画素ノイズは、現在の画素に対応すると言われ、時には、現在の画素のノイズ成分と呼ばれることもある。 In one aspect, the adaptive spatial noise filter 570 is implemented as an adaptive spatial noise bilateral filter, for example as follows:

Where:

is the filtered signal level of the pixel at position x in the frame.
The adaptive spatial noise filter 570 uses a square block of pixels, e.g., a number of neighboring pixels, around the current pixel to determine how to filter the current pixel. The Distance Filter filters the current pixel based on weights determined from the Euclidean distance of the neighboring pixels in the block from the current pixel. The signal range filter (sometimes called a range filter or intensity filter) is an adaptive noise signal range filter based on the value of the pixel noise in the frame (imNoiseY) that corresponds to the pixel being filtered. Thus, the weights used in the range filter are based not only on the difference in signal level between the neighboring pixels in the block and the current pixel, but also on the noise component of the current pixel. Pixel noise of the input frame that is at the same location as the current pixel's location in the input pixel frame is said to correspond to the current pixel, and is sometimes also referred to as the noise component of the current pixel.

Non-noise adaptive bilateral filters are known. See, for example, C. Tomasi and R. Manduchi, "Bilateral Filtering for Gray and Color Images," Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay India, pp. 839-846 (1998), which is incorporated herein by reference as a demonstration of knowledge in the art.

Like prior art bilateral filters, adaptive spatial noise filter 570 is a nonlinear edge-preserving filter. However, as noted above, unlike conventional bilateral filters, for a pixel, adaptive spatial noise filter adjusts the weights used in the filter based not only on the Euclidean distance of neighboring pixels from the pixel, but also on the difference in signal level between the neighboring pixels and the current pixel and the noise component of the current pixel. Specifically, as described more fully below, adaptive spatial noise filter 570 takes a weighted sum of pixels in the neighborhood of the current pixel. The weights depend not only on the noise component of the current pixel itself, but also on the spatial distance and signal level of the neighboring pixels. Thus, the signal value at each pixel in the frame is replaced by a weighted average based on the noise of the signal values from the neighboring pixels.

Unlike previous implementations of bilateral filters that rely on a constant signal range parameter (σr) for all pixels to archive the effects of edge preservation while averaging and reducing noise, the adaptive spatial noise filter 570 adjusts the signal range parameter (σr) for each pixel based on the noise content of that pixel. Thus, each pixel in the frame is bilaterally filtered based on the noise associated with the pixel. That is, the spatial bilateral filter adapts to the noise at each pixel.

ここで、

は、位置ｘにおける画素のフィルタ処理された信号レベルであり、

は、正規化定数であり、

は、空間的なガウス分布であり、

は、位置ｙの画素と位置ｘの画素との間の距離のノルム（norm）であり、

は、通常１～１０の範囲の距離パラメータであり、

は、位置ｘにおける現在の画素のノイズの関数である信号レベル範囲のガウス分布（signal level range Gaussian distribution）であり、

は、位置ｙにおける画素の信号レベルであり、

は、位置ｘにおける画素の信号レベルであり、

は、位置ｙにおける画素の信号レベルと位置ｘにおける画素の信号レベルとの差の絶対値であり、

は、位置ｘにおける画素のノイズの関数である信号レベルレンジパラメータであり、

は、位置ｘに中心画素を有する画素のブロックであり、

は、位置ｘに中心画素を有する画素ブロック内の画素に亘る総和を表す。
位置ｘ及び位置ｙは、画素のブロック内の位置を指定し、各位置は２つの座標によって表されることに留意されたい。 In one aspect, the following equation is the definition of the adaptive spatial noise filter 570 implemented in the imaging pipeline 530:

Where:

is the filtered signal level of the pixel at location x,

is a normalization constant,

is a spatial Gaussian distribution,

is the norm of the distance between the pixel at position y and the pixel at position x,

is a distance parameter, typically ranging from 1 to 10,

is the signal level range Gaussian distribution that is a function of the noise of the current pixel at position x,

is the signal level of the pixel at position y,

is the signal level of the pixel at position x,

is the absolute value of the difference between the signal level of the pixel at position y and the signal level of the pixel at position x,

is a signal level range parameter that is a function of the noise of the pixel at location x,

is a block of pixels with its center pixel at position x,

represents the sum over the pixels in the pixel block having its center pixel at position x.
Note that position x and position y specify a location within a block of pixels, and each location is represented by two coordinates.

FIG. 6A is a process flow diagram of one implementation of a method 650 for adaptive spatial noise pixel filtering. FIG. 6B is a block diagram of one implementation of the method 650 of FIG. 6A in an adaptive spatial noise filter 670. In one aspect, the adaptive spatial noise filter 670 can be used in the imaging pipeline 230 and the imaging pipeline 530. The adaptive spatial noise filter 670 is an example of the adaptive spatial noise filter 570.

First, the new frame search process 601 retrieves a new pixel frame 602, e.g., frame (imY). The new frame search process 601 passes to the block acquisition process 603.

The Get Block process 603 retrieves an input block of nxn pixels centered around the current pixel from the current pixel frame 616 (see FIG. 6B). Each pixel in the frame is adaptively spatial noise filtered, or at least a number of pixels in the frame are adaptively spatial noise filtered, so that the current pixel is an adaptive spatial noise filtered pixel. In one aspect, the input block of nxn pixels is a 5x5 pixel block with the current pixel in the center of the block. The Get Block process 603 passes to a Distance Filter process 604.

The Distance Filter process 604 generates a value of the spatial Gaussian function defined above for each pixel of the input block of pixels, i.e., for each pixel in the input pixel block, resulting in a distance filter value for each position in the n x n distance filter value block. The Distance Filter process 604 passes to a Range Difference process 605.

The Range Difference process 605 determines the difference in signal level between the current pixel and each of the other pixels in the input pixel block. Specifically, the Range Difference process 605 evaluates I(y)-I(x), where x is the position of the central pixel in the block, I(x) is the value of the central pixel, and I(y) is the value of the pixel at position y in the input pixel block, where y is a range of positions in the input pixel block, to generate a range difference for each position in the n by n range difference block. The Range Difference process 605 passes to a Range Filter process 606.

The range filter process 606 implements a Gaussian distribution of signal level ranges that is a function of the noise of the current pixel as described above. To implement a Gaussian distribution of signal level ranges, the range filter process 606 requires a noise-dependent value of the signal level range parameter (σ(Noise)r) of the current pixel. Here, the inclusion of noise in parentheses of the signal level range parameter (σ(Noise)r) means that the signal level range parameter (σr) is a function of the noise component of the current pixel. The result of this function is that the range filter of the bilateral filter is adaptive and changes based on the noise component of the pixel being filtered bilaterally. As a result, the spatial noise filter is called an adaptive spatial noise filter because it adapts to noise. Thus, the adaptive spatial noise filter compares the signal level of the current pixel to the signal levels of multiple neighboring pixels and the estimated pixel noise parameter by dividing the square of the absolute value of the difference between the signal level of the pixel at position y and the signal level of the pixel at position x by the square of the signal level range parameter (σ(Noise)r). Here, the signal level range parameter (σ(Noise)r) is the estimated pixel noise parameter in this embodiment. This comparison is used to output the adaptive spatial noise filtered pixel value.

In one aspect (not shown), the signal level range parameter (σ(Noise)r) is a linear function of the noise component of the current pixel. In this aspect, the position of the current pixel is used to retrieve pixel noise from a corresponding location, e.g., the same location, in the frame of pixel noise data input to the stage of the pipeline. The retrieved pixel noise corresponds to the center pixel. The pixel noise, which is the noise component of the center pixel, is input to the linear function to obtain the signal level range parameter (σ(Noise)r). In the case of the imaging pipeline 530, the input frame 612 of pixel noise data is frame (imNoiseY). In one aspect, the slope of the linear function is between 2 and 3. In one aspect, the slope and offset are empirically determined by using different values for the slope and offset, filtering the noise image with an adaptive spatial filter for each value, and then selecting the value that produces the best filtered image as determined by a group of observers.

In another aspect, as shown in FIG. 6A, the current pixel location is used to look up the pixel noise from the pixel noise data frame 612, which is input to a noise vs. Sigma R lookup table 613. This table is empirically generated using a procedure similar to that described for linear functions, but with more complex curves. Thus, the noise vs. Sigma R lookup table 613 includes a signal level range parameter (σ(Noise)r) for each of a plurality of pixel noise levels, where in one aspect the plurality of pixel noise levels include a range of pixel noise commonly encountered in surgical images. The noise vs. Sigma R lookup table 613 outputs a signal level range parameter σ(Noise)r corresponding to the input pixel noise to the range filter process 606, which in turn outputs a signal level range parameter σ(Noise)r that depends on the noise content of the current pixel.

Thus, Range Filter process 606 generates an nxn block of range filtered values, where the range filtered value at a given location of the nxn block of range filtered values is a Gaussian distributed value of the signal level range using the input range difference and input signal level (σ(Noise)r) from the same location, e.g., corresponding location, of the given location and the range difference of the nxn block. Thus, Range Filter process 606 passes to Combine Filters process 607.

The filter combination process 607 multiplies the value of a position in the n-by-n block of distance filter values by the value of the same position in the n-by-n block of range filter values for each position in the two blocks to generate a composite filter value for each position in the n-by-n composite filter value block. The filter combination process 607 passes to a normalization process 608.

Normalization process 608 first determines the value of the normalization constant C defined above. Specifically, normalization process 608 sums the values of the n×n block of composite filter values to generate the normalization constant C. Normalization process 608 then divides the value at each position of the n×n composite filter value block by the normalization constant C to generate an n×n normalized composite filter value block. Normalization process 608 passes to pixel generation process 609.

The pixel generation process 609 generates adaptive spatial noise filtered pixels and writes the adaptive spatial noise filtered pixels to the output frame of filtered pixel data 614 (sometimes referred to as the output frame of pixel data or the filtered pixel output frame). Specifically, the pixel generation process 609 generates an applied filter value for a position in the n×n applied filter value block by multiplying the value of the same position in the n×n normalized composite filter value block with the value of the same position in the n×n input pixel block. The multiplication is performed for each position in the block. Finally, to generate an adaptive spatial noise filtered pixel for the pixel in the center of the block, the pixel generation process 609 sums the values of the applied filter values of the n×n block to obtain an adaptive spatial noise filtered pixel, which is written to the appropriate location in the output frame of filtered pixel data 614, i.e., the location in the output frame of filtered pixel data 614 that is the same as the location of the center pixel in the current pixel frame 616. The filtered pixel data output frame 614 is sometimes referred to as the filtered pixel data output frame 614.

The pixel noise generation process 610 is optional and is typically implemented if there is another filter stage later in the pipeline that uses an adaptive noise filter. As stated above, the normalization constant C is defined as follows:

In determining the normalization constant C, an n×n block of composite filter values is generated, and the composite filter value (C x y ) at a position within the block is expressed as follows:

Also, as explained above, the value at each position of the n×n composite filter value block is divided by a normalization constant C to generate a normalized composite filter value (CFN) for the n×n block, as shown below.

A pixel noise generation process 610 receives the normalized composite filter value (CFN) for this n×n block from the normalization process 608. It also includes, for each pixel in the n×n pixel block centered on the current pixel, a corresponding pixel noise standard deviation (stdPixi,j) within the frame of pixel noise data 612. The pixel noise output standard deviation (stdOut) for the current pixel is defined as:

The pixel noise output standard deviation (stdOut) is passed to the pixel noise output frame 615 (sometimes called the pixel noise data output frame 615). The pixel noise generation process 610 passes the process to the final pixel confirmation process 611.

The Check Last Pixel process 611 determines whether all pixels in the current pixel frame have been filtered. If all pixels have been filtered, the Check Last Pixel process 611 transfers to the Find New Frame process 601, otherwise it transfers to the Get Block process 603.

Below is a pseudocode implementation of one embodiment of the adaptive spatial noise filter 670.
sigmaD = 1 "sigma value of the distance filter"
I = [5, 5] "specifies the size of the block"
distFilt = fspecial('gausian',[5, 5], 10) "Evaluate the distance filter for each location in the block"
cp = I(3, 3) "specifies the center pixel"
diff = I-cp "signal level difference between each pixel and the center pixel"
np = location of cp in noise frame "Specifies the pixel noise corresponding to the central pixel"
sigmaR = lookup(np) "Get SigmaR as a function of the noise pixel corresponding to the center pixel"
rangeFilt = exp(-diff.^2/2*sigmaR^2))) "Evaluate the range filter at each position in the block"
combFilt = rangeFilt. * distFilt "Multiply each element of the range filter by the corresponding element of the distance filter."
sumFilt = sum(combFilt(:)) "sum up the values of the combined filters"
combFiltNorm = comFilt./sumFilt "Divide each value of the combined filter by a normalization factor"
appliedFilter = I. * comFiltNorm "Multiply each pixel in the block by the normalized combined filter"
newValue = sum(appliedFilter(:)) "The filtered center pixel is the sum of the values of the applied filters"
In one aspect, the adaptive spatial noise filter is written in a computer programming language and compiled into an executable module that can run on a processor in the imaging pipeline. Alternatively, the adaptive spatial filter can be implemented in hardware, firmware, or any combination of hardware, firmware, and combination of processors and executable modules. See, e.g., FIG. 6B.

6A. The adaptive spatial noise filter 670 includes a distance filter module 620, a first summer 621, a range filter module 622, a first multiplier 623, a second summer 624, a divider 625, a second multiplier 626, a third summer 627, a multiplier bank 628, a fourth summer 629, and a square root module 630. The adaptive spatial noise filter 670 is connected to the current pixel frame 616, the current pixel noise frame 612, the noise vs. sigma R lookup table 613, the output frame of filtered pixel data 614, and the pixel noise output frame 615. In one aspect, each of the current pixel frame 616, the current pixel noise frame 612, the noise vs. Sigma R lookup table 613, the filtered pixel output frame 614, and the pixel noise output 614 are stored in memory.

The depiction of the current pixel frame 616, the current pixel noise frame 612, the noise vs. sigma R lookup table 613, the filtered pixel output frame 614, and the pixel noise output frame 615 as separate from the adaptive spatial noise filter 670 is for illustrative purposes only. For example, the adaptive spatial noise filter 670 may be depicted in FIG. 6B as including all or some combination of the current pixel frame 616, the current pixel noise frame 612, the noise vs. sigma R lookup table 613, the filtered pixel output frame 614, and the pixel noise output frame 615.

To implement the block acquisition process 603, a control signal is provided to the current pixel frame 616 (sometimes also referred to as frame 616), a block of pixels I(n,n) is provided to a distance filter module 620 and to a first input of an aggregator 621, and a center pixel (cp) of the block I(n,n) is provided to a second inverted input (In2) of the aggregator 621 of the adaptive spatial noise filter 670, where n is a positive non-zero odd number.

As mentioned above, the center pixel is sometimes called the current pixel. Also, when a pixel is said to be driven on an input or fed to an input, it means that the value of the pixel is on the input.

The distance filter module 620 performs the spatial Gaussian function defined above for each pixel of the block I(n,n), i.e., for each location of the block I(n,n), to generate a distance filter value for each location of the block I(n,n). The distance filter module 620 performs the distance filter process 604.

The aggregator 621 subtracts the value of the center pixel from each value in the block I(n,n). That is, the aggregator 621 determines the signal level difference between the current pixel and each of the other pixels in the block I(n,n) to generate a signal level range difference for the nxn block. The aggregator 621 outputs the signal level range difference for the nxn block at an output (Out). The aggregator 621 implements the range difference process 605.

The Range Filter module 622 implements a Gaussian distribution of signal level ranges that is a function of the noise of the current pixel as described above. To implement the signal level, the Range Filter module 622 requires a noise-dependent value of the signal level range parameter (σ(Noise)r) for the central pixel of the block I(n,n).

In this embodiment, the location of the current pixel (cp) is used to retrieve the pixel noise of the same location within the frame of pixel noise data 612, which is input to the noise-to-sigma R lookup table 613. The retrieved pixel noise is the pixel noise corresponding to the center pixel of the block I(n,n). In response to the input pixel noise, the noise-to-sigma R lookup table 613 provides a signal level range parameter (σ(Noise)r) on a second input (In2) to the range filter module 622. The range filter module 622 receives the signal level range difference of the n×n block from the aggregator 621 on a first input (In1).

Thus, the range filter module 622 generates a range filter value for each position in the block I(n,n). The range filter value at a given position of the n by n range filter value block is a Gaussian distribution of the signal level range using the input range difference and the input signal level (σ(Noise)r) for that given position of the n by n signal level range difference block. The range filter module 622 implements the range filter process 606.

The multiplier 623 receives the distance filter value of the n×n block on a first input (In1) and the range filter value of the n×n block on a second input terminal (In2). The multiplier 623 multiplies the distance filter value of the n×n distance filter value block by the corresponding range filter value of the n×n range filter value block and puts the result into the corresponding position of the n×n combined filter block. Here, the distance filter value at position (1,1) of the distance filter value block corresponds to the range filter value at position (1,1) of the range filter value block and the composite filter value at position (1,1) of the composite filter value block; the distance filter value at position (1,2) of the distance filter value block corresponds to the range filter value at position (1,2) of the range filter value block and the composite filter value at position (1,2) of the composite filter value block; ... the distance filter value at position (2,1) of the distance filter value block corresponds to the range filter value at position (2,1) of the range filter value block and the composite filter value at position (2,1) of the composite filter value block, and so on. Thus, generally speaking, entries in the same position in two n×n blocks are said to correspond. The composite filter value for the n×n blocks is provided to the output (Out) of multiplier 623, which implements combined filter process 607.

The aggregator 624 receives an n x n block of composite filter values on its input (In). The aggregator 624 sums the values of the n x n block of composite filter values to generate the normalization constant C defined above. The normalization constant C is provided at the output (Out) of the aggregator 624.

Divider 625 receives the n×n block of composite filter values at its (fractional) numerator input (Num) and a normalization constant C at its denominator terminal (Denom). Divider 625 divides each composite filter value of the n×n block of composite filter values by the normalization constant C to generate a normalized composite filter value of the n×n block. The normalized composite filter value of the n×n block is provided at the output (Out) of divider 625. Aggregator 624 and divider 625 implement the normalization process 608.

The multiplier 626 receives the pixel block I(n,n) on a first input (In1) and the normalized composite filter value of the n×n block on a second input (In2). The multiplier generates the applied filter value of the n×n block by multiplying each of the normalized composite filter values by a corresponding pixel in the pixel block I(n,n). Again, the pixel value at position (1,1) of the pixel block corresponds to the normalized composite filter value at position (1,1) of the normalized composite filter value block and the applied filter value at position (1,1) of the applied filter value block; the pixel value at position (1,2) of the pixel block corresponds to the normalized composite filter value at position (1,2) of the normalized composite filter value block and the applied filter value at position (1,2) of the applied filter value block, and so on. The applied filter values of the n×n block are provided to the output (OUT) of the multiplier 626.

The aggregator 627 receives an n×n block of applied filter values on the input (IN). The aggregator 624 sums the values of the n×n block of applied filter values to generate an adaptive spatial noise filtered pixel for the center pixel of the pixel block I(n,n). The adaptive spatial noise filtered pixel for the center pixel is provided at the output (OUT) of the aggregator 627 so that the adaptive spatial noise filtered pixel is written to the appropriate position of the filtered pixel output frame 614, i.e., the same position of the filtered pixel output frame 614 as the position of the center pixel (cp) of the current pixel frame 616. The multiplier 626 and the aggregator 627 implement the pixel generation process 609.

The multiplier bank 628 receives on a first input (In1) an n×n block of pixel noise standard deviations (stdPixi,j) from the pixel noise data frame 612 and on a second input terminal (In2) an n×n block of normalized composite filter values. The multiplier bank 623 multiplies each of the input values by itself and multiplies the squared value of one block by the corresponding squared value of the other block. An n×n block consisting of the products of the squares of the pixel noise standard deviations and the squares of the normalized composite filter values is supplied to the output (Out) of the multiplier bank 628 and drives the input terminal (In) of the aggregator 629.

The aggregator 629 receives on its input (In) an n x n block of products of the squares of the pixel noise standard deviations and the squares of the normalized composite filter values. The aggregator 624 sums the values of the n x n block of products of the squares of the pixel noise standard deviations and the squares of the normalized composite filter values. The result is provided to the output (Out) of the aggregator 629.

The square root module 630 receives the output (Out) of the aggregator 629 at its input (In). The square root module 630 takes the square root of the input value and provides the pixel noise output standard deviation (stdOut) to the pixel noise output frame 615. The multiplier bank 628, the aggregator 629, and the square root module 630 implement the pixel noise generation process 610.

The adaptive noise filter described above may actually be implemented by any number of modules, and each module may include any combination of components. Each module and each component may include hardware, software running on a processor, and firmware, or any combination of the three. Also, as described herein, the functions and operations of the adaptive noise filter may be performed by one module, or may be divided among different modules or among different components of a module. When divided among different modules or components, the modules or components may be centralized in one location or distributed across the computer-assisted surgery system for distributed processing purposes. Thus, reference to an adaptive noise filter should not be construed as requiring a single physical entity.

In the above examples, a single imaging pipeline is shown and described. However, in a system utilizing stereoscopic scenes, a second pipeline identical to the one described above is used, with one pipeline processing left captured frames from a left image capture unit and another pipeline processing right captured frames from a right image capture unit. It would be redundant to repeat the above description of the second pipeline, and so for the sake of clarity, it is not included here.

As used herein, "first," "second," "third," etc. are adjectives used to distinguish between different components or elements. Thus, "first," "second," and "third" are not intended to imply an order of the components or elements or a total number of components or elements.

The above description and accompanying drawings illustrating aspects and embodiments of the present invention should not be construed as limiting, but rather define the invention for which the claims are protected. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail to avoid obscuring the invention.

Moreover, the terms of this description are not intended to limit the invention. For example, spatially related terms such as "beneath," "below," "lower," "above," "upper," "proximal," and "distal" refer to the relationship of one element or feature to another element or feature as depicted in the figures. These spatially related terms are intended to encompass different positions (i.e., configurations) and orientations (i.e., rotated configurations) of the device during use or operation in addition to the positions and orientations depicted in the figures. For example, if the device in the figures is turned over, an element described as "below" or "beneath" another element or feature would then be "above" or "over" the other element or feature. Thus, the exemplary term "below" can encompass both above and below positions and orientations. The device may be otherwise oriented (rotated 90 degrees or in other orientations) and the spatially related descriptions used herein interpreted accordingly. Similarly, descriptions of movement along and around various axes include various specific device positions and orientations.

The singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. Terms such as "comprises," "comprises," "includes," and the like specify the presence of stated features, steps, operations, elements, and/or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. Components that are described as being coupled may be directly coupled electrically or mechanically, or may be indirectly coupled through one or more intermediate components.

All example and illustrative references are non-limiting and should not be used to limit the scope of the claims to the specific implementations and embodiments described herein and their equivalents. Because text under a heading may be cross-referenced or applied to text under one or more headings, the headings are merely formative and should not be used to limit the subject matter. Finally, in light of this disclosure, a particular feature described in connection with one aspect or embodiment may be applied to other disclosed aspects or embodiments of the invention even if not specifically shown in the drawings or described in the text.

The above embodiments are illustrative of the present disclosure, but are not intended to be limiting. It should also be understood that many modifications and variations are possible in accordance with the principles of the present disclosure. For example, in many aspects, the devices described herein are used as single port devices, i.e., all components necessary to complete a surgical procedure enter the body through a single entry port. However, in some aspects, multiple devices and ports may be used.

ここで、

は、位置ｘにおける前記画素の適応空間ノイズフィルタ処理された信号レベルであり、

は、正規化定数であり、

は、空間的なガウス分布であり、

は、前記位置ｙの画素と前記位置ｘの画素との間の距離のノルムであり、

は、距離パラメータであり、

は、前記画素のノイズの関数である信号レベル範囲のガウス分布であり、

は、前記位置ｙにおける前記画素の信号レベルであり、

は、前記位置ｘにおける前記画素の信号レベルであり、

は、前記位置ｙにおける前記画素の信号レベルと前記位置ｘにおける画素の信号レベルとの差の絶対値であり、

は、前記位置ｘにおける前記画素のノイズの関数である信号レベルレンジパラメータであり、

は、前記位置ｘに中心画素を有する画素のブロックであり、

は、前記位置ｘに中心画素を有する画素ブロック内の画素に亘る総和を表す、実施例１３に記載の撮像パイプラインステージ。
［実施例１６］
方法であって、当該方法は、
第１のフレームの第１の画素を受信するステップと、
前記第１の画素の推定画素ノイズパラメータを用いて、且つ前記第１の画素の信号レベルと少なくとも第２画素の信号レベルとの間の差を用いて、前記第１の画素を適応ノイズフィルタ処理して、フィルタ処理された画素を取得するステップと、
前記フィルタ処理された画素を出力フレームに出力するステップと、を含む、
方法。
［実施例１７］
前記フィルタ処理された画素に対応する画素ノイズをノイズ出力フレームに出力するステップをさらに含む、実施例１６に記載の方法。
［実施例１８］
前記第２の画素は、前記第１のフレーム内の前記第１の画素の位置と同じ、画像センサによって取り込まれた第２のフレーム内の位置にあり、前記第２のフレームは、前記第１のフレームを取り込む前の時間に取り込まれ、前記第１の画素を適応ノイズフィルタ処理するステップは、前記第１及び第２の画素の信号レベルの変化と前記第１の画素の推定画素ノイズレベルとの比較に基づいて前記第１の画素を適応時間ノイズフィルタ処理するステップを含み、前記推定画素ノイズレベルは前記推定画素ノイズパラメータであり、前記第１及び第２の画素の信号レベルの変化は、前記第１の画素の前記信号レベルの時間変化である、実施例１６に記載の方法。
［実施例１９］
前記第１の画素を適応時間ノイズフィルタ処理するステップは、
前記第１の画素の信号レベルを信号レベル対ノイズ・ルックアップテーブルに入力するステップと、
前記信号レベル対ノイズ・ルックアップテーブルを用いて、前記第１の画素の信号レベルに対応するノイズ標準偏差を出力するステップであって、前記ノイズ標準偏差は、前記第１の画素の前記推定画素ノイズレベルである、出力するステップと、を含む、実施例１８に記載の方法。
［実施例２０］
前記第１の画素を適応時間ノイズフィルタ処理するステップは、
前記第１の画素の前記信号レベルの前記時間変化を前記ノイズ標準偏差で除算して、前記第１の画素の前記信号レベルの時間変化をノイズの標準偏差として取得するステップであって、前記除算は前記比較である、取得するステップをさらに含む、実施例１９に記載の方法。
［実施例２１］
前記第１の画素を適応時間ノイズフィルタ処理するステップは、
前記第１の画素の前記信号レベルの時間変化をノイズの標準偏差で変更ルックアップテーブルに入力するステップと、
前記変更ルックアップテーブルを用いて、前記フィルタを通過する前記第１の画素の信号レベルの許容される時間変化の割合を出力するステップと、をさらに含む、実施例２０に記載の方法。
［実施例２２］
前記第１の画素を適応時間ノイズフィルタ処理するステップは、
前記第１の画素の前記信号レベルの許容される時間変化の割合に、前記第１の画素の前記信号レベルの前記時間変化を乗算することによって、前記第１の画素の前記信号レベルの許容される時間変化を生成するステップと、
前記第１の画素の前記信号レベルにおける前記許容される時間変化を対応する画素に加算することによって、前記フィルタ処理された画素を生成するステップであって、前記対応する画素は、第２のフレーム内の画素であり、前記第２のフレームは、時間的に前記第1のフレームの直前にある、生成するステップと、をさらに含む、実施例２１に記載の方法。
［実施例２３］
前記少なくとも１つの第２の画素は、前記第１の画素に隣接する複数の画素に含まれ、前記第１の画素を適応ノイズフィルタ処理するステップは、前記第１の画素の信号レベルと前記複数の隣接する画素の信号レベルのそれぞれとの間の差に基づいて、及びノイズ依存信号レベルパラメータに基づいて、前記第１の画素を適応空間ノイズフィルタ処理するステップを含み、前記ノイズ依存信号レベルパラメータは、前記第１の画素のノイズの関数であり、前記ノイズ依存信号レベルパラメータは、前記第１の画素の推定画素ノイズパラメータである、実施例１６に記載の方法。
［実施例２４］
前記第１の画素を適応空間ノイズフィルタ処理するステップは、
距離フィルタと信号レベルフィルタとを組み合わせることによって前記フィルタ処理された画素を生成するステップであって、前記信号レベルフィルタは、前記ノイズ依存信号レベルパラメータの関数である、生成するステップを含む、実施例２３に記載の方法。
［実施例２５］
前記信号レベルフィルタは、信号レベルレンジパラメータを有する信号レベル範囲のガウス分布を含み、前記信号レベルレンジパラメータは、前記ノイズ依存信号レベルパラメータである、実施例２４に記載の方法。




The contents of the claims as originally filed are described below as examples.
[Example 1]
1. A system comprising:
an image sensor configured to capture a first frame of pixel data;
an imaging pipeline coupled to the image sensor to receive the first frame of pixel data, the imaging pipeline including an adaptive noise filter, the adaptive noise filter comprising:
filtering the first pixel using an estimated pixel noise parameter for the first pixel and using a difference between a signal level of the first pixel and a signal level of at least a second pixel;
configured to output a second frame of pixel data including the filtered first pixels;
an imaging pipeline configured to output an output frame of pixel data based on the second frame of pixel data;
a display device coupled to the imaging pipeline to receive the output frames of pixel data, the display device configured to display the output frames of pixel data.
system.
[Example 2]
The system of example 1, wherein the adaptive noise filter includes an adaptive temporal noise filter, the second pixel is at a position in a third frame captured by the image sensor that is the same as the position of the first pixel in the first frame, the third frame being captured at a time prior to capturing the first frame, and the adaptive temporal noise filter is configured to filter the first pixel based on a comparison of changes in signal levels of the first and second pixels to an estimated pixel noise level of the first pixel, the estimated pixel noise level being an estimated pixel noise parameter.
[Example 3]
3. The system of claim 2, wherein the imaging pipeline includes a plurality of stages, and the adaptive temporal noise filter is included in a first stage of the plurality of stages.
[Example 4]
3. The system of claim 2, wherein the adaptive temporal noise filter is configured to output a pixel noise frame.
[Example 5]
The system of example 4, wherein the imaging pipeline includes a plurality of stages, each of the plurality of stages subsequent to a first stage of the plurality of stages configured to process an input pixel noise frame and configured to output a stage-dependent pixel noise frame.
[Example 6]
2. The system of claim 1, wherein the adaptive noise filter includes an adaptive spatial noise filter, the at least one second pixel being included in a plurality of pixels adjacent to the first pixel, the adaptive spatial noise filter being configured to filter the first pixel based on a difference between a signal level of the first pixel and each of the signal levels of the plurality of adjacent pixels and based on a noise-dependent signal level parameter, the noise-dependent signal level parameter being a function of noise of the first pixel, and the noise-dependent signal level parameter being the estimated pixel noise parameter of the first pixel.
[Example 7]
The system of Example 6, wherein the adaptive spatial noise filter includes a distance filter and a signal level range filter, the signal level range filter configured to filter the first pixel based on a difference between a signal level of the first pixel and each of the signal levels of the multiple adjacent pixels.
[Example 8]
7. The system of claim 6, wherein the noise-dependent signal level parameter is a signal level range parameter.
[Example 9]
An imaging pipeline stage, the imaging pipeline stage comprising:
an adaptive temporal noise filter including a signal level vs. noise lookup table;
The adaptive temporal noise filter comprises:
receiving a frame of pixels;
filtering each pixel of the plurality of pixels of the frame by comparing a change in signal level of the pixel over time with a value in the signal level vs. noise lookup table corresponding to the signal level of the pixel;
and configured to output a frame including a plurality of adaptive temporal noise filtered pixels based on the comparison.
Imaging pipeline stages.
[Example 10]
10. The imaging pipeline stage of example 9, wherein the signal level vs. noise lookup table is configured to receive a value of a pixel of a current pixel frame and output a noise standard deviation corresponding to the signal level of the pixel.
[Example 11]
The imaging pipeline stage of Example 9, wherein the adaptive temporal noise filter further comprises a modified lookup table configured to receive a temporal signal change of a pixel represented by a noise standard deviation numerical value and to output a percentage of an acceptable temporal change of the signal level of the pixel.
[Example 12]
10. The imaging pipeline stage of example 9, wherein the adaptive temporal noise filter is configured to output a pixel noise frame.
[Example 13]
An imaging pipeline stage, the imaging pipeline stage comprising:
It has an adaptive spatial noise filter,
The adaptive spatial noise filter comprises:
receiving an input pixel frame and an input pixel noise frame;
filtering a pixel of the input pixel frame based on a difference between a signal level of a pixel and each of a plurality of adjacent pixel signal levels and based on a noise-dependent signal level parameter to generate an adaptive spatial noise filtered pixel, the noise-dependent signal level parameter being determined using pixel noise in the input pixel noise frame corresponding to the pixel;
configured to output the adaptive spatial noise filtered pixels into an adaptive spatial noise filtered pixel frame.
Imaging pipeline stages.
[Example 14]
14. The imaging pipeline stage of example embodiment 13, wherein the adaptive spatial noise filter comprises an adaptive spatial noise bilateral filter.
[Example 15]
The pixel is the pixel at location x, and the adaptive spatial noise bilateral filter is defined as:

Where:

is the adaptive spatial noise filtered signal level of the pixel at location x,

is a normalization constant,

is a spatial Gaussian distribution,

is the norm of the distance between the pixel at the position y and the pixel at the position x,

is the distance parameter,

is a Gaussian distribution of the signal level range that is a function of the noise of the pixel,

is the signal level of the pixel at the location y,

is the signal level of the pixel at the location x,

is the absolute value of the difference between the signal level of the pixel at the position y and the signal level of the pixel at the position x,

is a signal level range parameter that is a function of the noise of the pixel at the location x,

is a block of pixels having a center pixel at said location x,

represents a sum over pixels in a pixel block having a center pixel at the position x.
[Example 16]
1. A method, comprising:
receiving a first pixel of a first frame;
adaptive noise filtering the first pixel using estimated pixel noise parameters of the first pixel and using a difference between a signal level of the first pixel and a signal level of at least a second pixel to obtain a filtered pixel;
and outputting the filtered pixels into an output frame.
method.
[Example 17]
17. The method of example 16, further comprising outputting pixel noise corresponding to the filtered pixels in a noise output frame.
[Example 18]
17. The method of claim 16, wherein the second pixel is at a position in a second frame captured by an image sensor that is the same as the position of the first pixel in the first frame, the second frame being captured at a time prior to capturing the first frame, and the step of adaptively noise filtering the first pixel includes a step of adaptively temporal noise filtering the first pixel based on a comparison of a change in signal level of the first and second pixels to an estimated pixel noise level of the first pixel, the estimated pixel noise level being the estimated pixel noise parameter, and the change in signal level of the first and second pixels being a temporal change in the signal level of the first pixel.
[Example 19]
The step of adaptively temporal noise filtering the first pixel comprises:
inputting the signal level of the first pixel into a signal level vs. noise lookup table;
and outputting a noise standard deviation corresponding to a signal level of the first pixel using the signal level vs. noise lookup table, the noise standard deviation being the estimated pixel noise level of the first pixel.
[Example 20]
The step of adaptively temporal noise filtering the first pixel comprises:
20. The method of claim 19, further comprising: dividing the time change in the signal level of the first pixel by the noise standard deviation to obtain the time change in the signal level of the first pixel as a noise standard deviation, wherein the division is the comparison.
[Example 21]
The step of adaptively temporal noise filtering the first pixel comprises:
inputting the time change of the signal level of the first pixel into a lookup table modified by the standard deviation of noise;
21. The method of example 20, further comprising: using the modified lookup table to output an allowable percentage of time change in signal level of the first pixel passing through the filter.
[Example 22]
The step of adaptively temporal noise filtering the first pixel comprises:
generating an allowed change in signal level of the first pixel over time by multiplying a percentage of an allowed change in signal level of the first pixel over time by the change in signal level of the first pixel;
22. The method of example 21, further comprising: generating the filtered pixel by adding the allowed change in signal level of the first pixel to a corresponding pixel, the corresponding pixel being a pixel in a second frame, the second frame immediately preceding the first frame in time.
[Example 23]
17. The method of claim 16, wherein the at least one second pixel is included in a plurality of pixels neighboring the first pixel, and the step of adaptively noise filtering the first pixel comprises a step of adaptively spatial noise filtering the first pixel based on a difference between a signal level of the first pixel and each of the signal levels of the plurality of neighboring pixels and based on a noise-dependent signal level parameter, the noise-dependent signal level parameter being a function of noise of the first pixel, and the noise-dependent signal level parameter being an estimated pixel noise parameter of the first pixel.
[Example 24]
The step of adaptively spatial noise filtering the first pixel comprises:
24. The method of claim 23, comprising generating the filtered pixel by combining a distance filter and a signal level filter, the signal level filter being a function of the noise-dependent signal level parameter.
[Example 25]
25. The method of claim 24, wherein the signal level filter comprises a Gaussian distribution of signal level ranges having a signal level range parameter, the signal level range parameter being the noise dependent signal level parameter.

Claims (14)

1. A system comprising:
an image sensor configured to capture a first frame of pixel data;
an adaptive noise filter coupled to the image sensor;
The adaptive noise filter comprises:
determining an estimated pixel noise parameter based on a signal level of a first pixel of the first frame of pixel data;
determining a difference between the signal level of the first pixel and a signal level of at least a second pixel, the second pixel being a pixel at the same position as the first pixel in the first frame in a previous frame captured by the image sensor prior to the capture of the first frame;
determining a quantity of the difference that is allowed to pass through the adaptive noise filter based on a comparison of the difference to the estimated pixel noise parameters;
obtaining a filtered pixel based on the amount of the difference that is allowed to pass through the adaptive noise filter; and
outputting a second frame of pixel data including the filtered pixels;
system. a second adaptive noise filter coupled to the adaptive noise filter, configured to use estimated pixel noise parameters of the filtered pixels and to filter the filtered pixels using a difference between a signal level of the filtered pixels and a signal level of at least a third pixel to obtain further filtered pixels;
the adaptive noise filter is an adaptive temporal noise filter and the second adaptive noise filter is an adaptive spatial noise filter;
The system of claim 1 , wherein the second adaptive noise filter outputs a third frame of pixel data comprising the further filtered pixels. The system of claim 2, wherein the adaptive noise filter is configured to output pixel noise corresponding to the filtered pixels in a noise output frame. The system of claim 3, wherein the second adaptive noise filter is configured to determine the estimated pixel noise parameters for the filtered pixels based on the pixel noise corresponding to the filtered pixels in the noise output frame. The system of any one of claims 2 to 4, wherein the estimated pixel noise parameter of the first pixel is different from the estimated pixel noise parameter of the filtered pixel. The system of any one of claims 2 to 5, wherein the estimated pixel noise parameter of the filtered pixel is a function of the noise of the filtered pixel. 7. The system of claim 1, wherein the adaptive noise filter comprises a signal level vs. noise lookup table configured to receive the signal level of the first pixel and output a noise standard deviation corresponding to the signal level of the first pixel, the noise standard deviation being the estimated pixel noise parameter. 8. The system of claim 7, wherein obtaining the filtered pixel includes using the noise standard deviation to determine a percentage of difference between the signal level of the first pixel and the signal level of at least the second pixel that is allowed to pass through the adaptive noise filter. 1. A method of adaptively noise filtering a first pixel of pixel data for a frame, the method comprising:
determining an estimated pixel noise parameter based on a signal level of the first pixel;
determining a difference between the signal level of the first pixel and a signal level of at least a second pixel, the second pixel being a pixel at the same position as the first pixel in the first frame in a previous frame captured by an image sensor prior to the capture of the first frame;
determining a quantity of the difference that is allowed to pass through an adaptive noise filter based on a comparison of the difference with the estimated pixel noise parameters;
obtaining a filtered pixel based on the amount of the difference that is allowed to pass through the adaptive noise filter;
and outputting the filtered pixels into an output frame of filtered pixel data.
method. adaptively noise filtering the filtered pixel using an estimated pixel noise parameter of the filtered pixel and using a difference between a signal level of the filtered pixel and a signal level of at least a third pixel to obtain a further filtered pixel;
10. The method of claim 9 , further comprising the step of: outputting the further filtered pixels in a second output frame of filtered pixel data. 11. The method of claim 10 , further comprising the step of outputting pixel noise corresponding to the filtered pixels in an output frame of pixel noise data. 12. The method of claim 11 , further comprising determining the estimated pixel noise parameter for the filtered pixel based on the pixel noise corresponding to the filtered pixel in the output frame of pixel noise data. 13. The method of claim 9, wherein determining the estimated pixel noise parameter comprises using a signal level vs. noise lookup table configured to receive a signal level of the first pixel and output a noise standard deviation corresponding to the signal level of the first pixel, the noise standard deviation being the estimated pixel noise parameter. 14. The method of claim 13, wherein obtaining the filtered pixel comprises using the noise standard deviation to determine the percentage of the difference between the signal level of the first pixel and the signal level of at least the second pixel that is allowed to pass to the filtered pixel .

Images