TWI403976B - Image enhancement using hardware-based deconvolution - Google Patents

Abstract

An image enhancement circuit (26, 60, 190, 260) includes an input interface (64, 262), which is operative to accept a stream of input pixel values belonging to pixels (32) of an input image. The input image includes a plurality of different input sub-images including respective subsets of the pixels, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream. A plurality of filter cells (92, 144, 206, 222, 238, 364) are connected in a two-dimensional array configuration and are arranged to separately filter the input pixel values of each of the input sub-images with respective two-dimensional deconvolution kernels so as produce respective output sub-images that include output pixel values. A multiplexer (88, 332) is coupled to multiplex together the output pixel values of the output sub-images so as to produce a filtered output image.

Description

Image enhancement using hardware-based deconvolution

The present invention is generally directed to digital imaging and, in particular, to methods and systems for enhancing image quality in digital cameras.

In many digital imaging applications, images obtained by imaging optics may be blurred, out of focus, or otherwise distorted. For example, imaging optics for certain low cost and small size cameras typically have a non-negligible optical point spread function (PSF). Several methods are known in the art for using digital image processing to compensate for non-ideal PSF and other optical distortion effects.

For example, U.S. Patent No. 6,154,574, the disclosure of which is incorporated herein by reference in its entirety in its entirety in the the the the the the the The average step response is obtained by dividing the defocused image into sub-images and calculating step responses relative to the edge directions in each sub-image. The average step response is used to calculate the PSF coefficients, which in turn are applied to determine the image restoration transfer function. The in-focus image is obtained by multiplying this function by the frequency domain of the out-of-focus image.

PCT International Publication No. WO 2004/063989 A2, the disclosure of which is hereby incorporated hereinby incorporated by reference in its entirety in its entirety in its entirety in its entirety in its entirety A deblurring function in the form of a convolution filter (DCF) is applied to the signal output by the array to produce an output image with less blur. This blur reduction can be designed and used with camera optics with poor intrinsic PSF while restoring the electronic image produced by the sensing array to provide an acceptable output image.

Low cost color cameras typically use a single solid state image sensor with a multi-color mosaic filter overlay. The mosaic filter is a reticle of miniature color filter elements in which the filter elements are secured in front of each detector element of the image sensor. For example, U.S. Patent No. 4,697,208, the disclosure of which is incorporated herein by reference in its entirety, is incorporated herein by reference in its entirety in its entirety in the the the the the the the Any type of image sensor with a color mosaic filter, regardless of the color selection and configuration in the mosaic, is referred to below as a "mosaic image sensor."

The filter elements in the mosaic filter are typically alternated between the main RGB colors, or between complementary colors cyan, magenta, and yellow. A common type of color mosaic filter is called "Bayer sensor" or "Bayer mosaic", which has the following general form (where the letters represent color: R for red, G for green, and B for blue): Different color filters have individual bandpasses that can be overlapped. Bayer inlays are described in U.S. Patent No. 3,971,065, the disclosure of which is incorporated herein by reference.

Processing an image produced by a mosaic image sensor typically involves reconstructing a solid color image by extracting three color signals (red, green, and blue) from the sensor output. An image signal processor (ISP) processes the image sensor output to calculate the luminosity (Y) and chrominance (C) values for each pixel of the output image. The ISP then outputs these values (or corresponding R, G, and B color values) in a standard image format.

Therefore, in accordance with an embodiment of the present invention, an image enhancement circuit is provided, comprising: an input interface operable to accept a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different inputs a sub-image comprising an individual subset of pixels, such that input pixel values of pixels in different input sub-images are interleaved in the stream; a plurality of filter elements are connected in a two-dimensional array configuration Configuring to separately filter the input pixel values of each of the input sub-images using an individual two-dimensional inverse-folding core to generate an individual output sub-image comprising the output pixel values; and a multiplexer coupled to The output pixel values of the output sub-images are multiplexed together to produce a filtered output image.

In some embodiments, the input image has input blur, and after the inverse deconvolution core filtering, the output image has an output blur that is less than the input blur.

In another embodiment, the array configuration has columns and rows of filter cells, the input interface including a memory configured to buffer input pixel values and provide successive rows of buffered input pixels to the filter unit, and Each row of the filter unit is configured to propagate the line of input pixel values to the next row of filter elements in the array configuration to deconvolute the input pixel values using the inverse-folded core.

In another embodiment, the inverse-folding core includes filter coefficients, and each of the filter units includes a multiplier configured to multiply one of the input pixel values by one of the filter coefficients to produce a filter One of the units outputs. The multiplier can be configured to perform two multiplications in each cycle of the pixel clock in which the input pixel is received. In another embodiment, the circuit includes an accumulator circuit configured to accumulate the output of the filter unit to produce an output pixel value. The accumulator is sometimes configured to accumulate output within two consecutive cycles of the pixel clock at which the input pixel is received.

In one embodiment, the array configuration has columns and rows of filter cells configured to provide successive rows of input pixel values to the filter cells to provide each input pixel value to the filter cell at the same time. All of the filter cells in the column, each filter cell comprising a multiplier configured to multiply the supplied input pixel value by the filter coefficient of one of the inversely folded cores to produce a multiplication result, and the circuit includes An accumulation circuit configured to calculate a sum of rows of multiplication results in each row of the filter unit to apply a sum of rows delayed by the sum of the rows and combined with the delay to produce an input pixel value.

In another embodiment, the mosaic image sensor provides a stream of input pixel values to generate pixel values in each of the input sub-images by the image sensor to respond to different individual color lights, and to filter The unit is configured to filter the input sub-images to produce an output sub-image corresponding to the individual colors. In another embodiment, the input sub-image includes green, red, and blue sub-images, and the filter unit is disposed in the first two-dimensional array, the first two-dimensional array configured to filter the green sub-images Entering a value; and in the second two-dimensional array, the second two-dimensional array is configured to alternately filter input values belonging to the red and blue sub-images.

In another embodiment, the inverse-folding core includes filter coefficients, and the filter unit includes an individual multiplier configured to multiply the input pixel value by the filter coefficient according to a predetermined designation, and the circuit includes selection and accumulation logic, It is configured to separately accumulate the output of the multiplier multiplied by the input pixel value of each of the input sub-images to produce an output pixel value.

In some embodiments, the input image is divided into a plurality of segments, and the filter unit is configured to use at least the first and second different inversely-folded cores to filter the at least first and first portions of the input image. The value of the input pixel in the two regions. In some embodiments, the first and second segments have individually different first and second dimensions that are individually determined in response to spatial variability in blurring in the first and second segments.

In another embodiment, the input image is divided into symmetry regions according to the symmetry characteristic of the blur in the input image, and the filter unit is configured to filter the input pixels positioned in one of the symmetry regions using the baseline deconvolution core The value, and the value of the input pixel positioned in the other symmetry zone is filtered using the individual deconvolution kernel determined by applying the symmetric operation to the baseline core.

In addition, in accordance with an embodiment of the present invention, an imaging method is provided, comprising: accepting a stream of input pixel values belonging to pixels of an input image, the input image comprising a plurality of different input sub-images including individual pixels of the pixel Set, so the input pixel values of the pixels in different input sub-images are interlaced in the stream; the filter unit is implemented using a plurality of hardware connected in the two-dimensional array configuration, and the individual two-dimensional deconvolution core is used to separate The input values of each of the input sub-images are filtered to generate an individual output sub-image containing the output pixel values; and the output pixel values of the output sub-images are multiplied to produce a filtered output image.

According to an embodiment of the present invention, an imaging apparatus is further provided, comprising: a mosaic image sensor configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responding to an incident Different individual colored lights on the mosaic image sensor, so the input pixel values of the pixels in the different input sub-images are interlaced in the stream; an image enhancement circuit that is coupled to use the connection in the two-dimensional array A plurality of filter units in the configuration receive and filter input pixel values in each of the input sub-images to generate a corresponding plurality of enhanced output sub-images; and an image signal processor (ISP) coupled The plurality of output sub-images are received and combined to generate a color output image.

Overview

Embodiments of the present invention provide methods and circuits for enhancing the quality of color images produced by a mosaic image sensor. These methods and circuitry are typically used in digital cameras to compensate for blurring caused by camera optics in the input image. The methods and circuits described herein can be used to reduce blurring in any suitable optical system. In particular, relatively simple, small, and low cost optical devices that reduce the overall cost and size of the camera can be used to produce high quality images.

In some embodiments, one of the input pixel values of the color input image is streamed to the image enhancement or image restoration circuit. Depending on the mosaic used, the input stream typically includes a staggered pattern of pixel values of different colors. The image enhancement circuit uses a separate inverse-folded core to filter the pixel values of each color to produce a monochrome output sub-image for each color.

The inversely-folded core coefficients are typically set to be close to the reciprocal of the point spread function (PSF) of the optical device that caused the blur. Therefore, the inversely-compressed input pixel values using the inverse-folded core can effectively compensate for distortion caused by the optical device. The enhanced monochrome sub-image is then combined to produce a color output pixel that has less blur than the input image.

Several alternative deconvolution configurations are described below. In general, a deconvolution includes a plurality of multipliers arranged in a two dimensional array. The multiplier multiplies the input pixel value by the appropriate inverse fold factor and accumulates the multiplication result to produce a total deconvolution result. In some embodiments, the multiplier operates at a clock rate higher than the pixel rate, thereby reducing the number of multipliers used.

Some of the inverse deconvolution configurations described herein use a pool of multipliers that are assigned to perform deconvolution operations of different color pixels in accordance with a predetermined multiplication scheme. In some embodiments, the deconvolution has a pipeline configuration in which partial multiplication results are propagated across the multiplier array to reduce the number of registers in the circuit.

In some embodiments, the image enhancement circuit uses different deconvolution cores in different regions of the input image. The following describes the number of deconvolution coefficients used to reduce storage by using symmetry.

instructions

1 is a block diagram schematically illustrating an electronic imaging camera 20 in accordance with an embodiment of the present invention. The configuration of camera 20 is shown here by way of example to illustrate the principles of the invention. However, such principles are not limited to the configuration shown in Figure 1, and can be applied to reduce blur in other types of imaging systems in which the input image is divided into multiple interlaced sub-images of different colors. Then, the sub-images are combined to produce an enhanced color output image.

For example, camera 20 may be suitable for a mobile communication terminal (eg, a mobile phone) or an in vivo medical imaging device. Alternatively, camera 20 may be part of a security system, an automotive system, or any other suitable system or application.

In camera 20, objective optics 22 focuses light from an imaged scene or object onto tessellation sensor 24. Sensor 24 can include any suitable type of image sensor, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. In this example and the following description, it is assumed that the sensor has a Bayer type mosaic filter, so each pixel 32 in the image signal output by the sensor responds to red, green or blue light.

Thus, an image produced by a mosaic sensor can be considered to include red, green, and blue sub-images that are composed of pixel values corresponding to the sensor elements. Pixel values belonging to different sub-images are usually interleaved in the output signal in the order of the colored elements in the mosaic filter. The sensor typically outputs a list of RGRGRG... (alternating red and green filters), then outputs a subsequent column, GBGBGB... (alternating green and blue filters), and so on in alternate lines. Alternatively, as necessary to correct, the methods and circuits described below can be used with other types of damascene sensor patterns.

The image produced by sensor 24 is received and processed by digital recovery circuit 26, which is typically provided as a stream of pixel values. Several exemplary embodiments of such a recovery circuit, also referred to as image enhancement circuitry, are described in detail below. The pixel values are digitized by analog to digital converters (not shown) and then processed by circuit 26. Circuitry 26 processes the red, green, and blue input sub-images produced by sensor 24 to reduce image blur, as explained below. Circuit 26 then outputs red, green, and blue sub-images with less blur.

In general, circuit 26 outputs a sub-image in the same format that it receives sub-images from sensor 24. For example, circuit 26 may interleave pixel values in the sub-image to produce a single output stream, where the pixel values have the same interlace as the input pixel values from sensor 24. Alternatively, circuit 26 can be configured to demultiplex and output each sub-image as a separate data block or data stream.

The ISP 28 receives the deblurred red, green, and blue output sub-images from the circuitry 26 and combines the sub-images to produce a color output image (or image sequence) using a standard image format. The output image can be displayed on screen 30, transmitted over a communication link, and/or stored in memory.

In general, the recovery circuit 26 is embodied in an integrated circuit die, which may include any suitable custom or semi-custom component. Although the restoration circuit 26 and the ISP 28 are shown as separate functional blocks in FIG. 1, the restoration circuit and the functions of the ISP can be implemented in a single integrated circuit component. As desired, image sensor 24 can be combined with circuitry 26 on the same semiconductor substrate and also with ISP 28 in a system-on-chip (SoC) or on-camera wafer design. Alternatively, some or all of the functionality of the restoration circuit 26 may be implemented in software on a programmable processor, such as a digital signal processor. The software can be downloaded to the processor in electronic form, or it can be additionally provided on a tangible medium such as an optical, magnetic or electronic memory.

With respect to certain additional aspects of the operation of the camera 20, and in particular, the operation of the circuit 26 is illustrated in the PCT application PCT/IL2006/001294 filed on Nov. 7, 2006, the application being assigned to The assignee of the present invention is hereby incorporated by reference.

Deconvolution product in mosaic space

The restoration circuit 26 reduces blur in the input image by applying separate and individual de-convolution filters for each input sub-image (i.e., pixel values associated with each different color). In other words, each input sub-image is converted into a corresponding output sub-image with less blur. For example, when the input image includes green, red, and blue input sub-images originating from a Bayer mosaic sensor, circuit 26 produces a corresponding green, red, and blue late output sub-image.

The inverse-folding operation determines the value of each output pixel by calculating the sum of the weights of the plurality of input pixel values around the position of the output pixel, the input pixel having the same color as the questionable output pixel. In general, the tens of pixel values that are located in a square surrounding the output pixel are weighted. Alternatively, any other suitable number or geometric configuration of pixel values can be used.

The weighting coefficients for the deconvolution procedure form a two-dimensional filter response that is close to the reciprocal of the point spread function (PSF) of the optical device 22 to compensate for the distortion caused by the optical device. Each filter coefficient set is also known as the deconvolution core. In general, separate deconvolution cores are defined for green, red, and blue pixels.

The method for calculating such a deconvolution core is described in, for example, the above-mentioned WO 2004/063989 A2 and the U.S. Patent Application Serial No. 11/278,255, filed on March 31, 2006, which is assigned to The assignee of the invention and the disclosure thereof are hereby incorporated by reference.

2 is a schematic illustration of green, red, and blue anti-folded cores in accordance with an embodiment of the present invention. In this example, the input image is derived from a Bayer mosaic sensor. Thus, the input pixels alternate between green, red, and blue pixels in accordance with the Bayer mosaic pattern described above.

Figure 2 shows three different sets of pixels participating in the green, red, and blue deconvolution cores. The figure shows the locations of the different input pixels that contribute to the inverse of the specific output pixel. For example, the green pixel is denoted as G, and the value of the particular green output pixel 40 (column 8, line 9) is determined based on the value of one hundred and thirteen green pixels 44 surrounding the pixel 40.

The remaining pixels can be red or blue. The value of the particular red or blue output pixel 48 (column 8, line 8) is determined based on the value of forty-nine pixels 52 of the same color surrounding pixel 44. When pixel 48 includes a red pixel, pixel 52, denoted R/B, also includes a red pixel. In this case, the blue pixel is represented as B/R. When pixel 48 includes a blue pixel, pixel 52 includes forty-nine surrounding blue pixels. In these cases, the pixels denoted as B/R represent red pixels.

Figure 3A shows the pixel locations participating in the red and blue deconvolution cores. As can be seen in this figure, the input pixels 52 that contribute to the inverse of the output R/B pixel 48 are located in every second column and every second row. The remaining pixels include green pixels or B/R pixels (ie, blue pixels when pixel 48 is red, and vice versa).

Figure 3B shows the pixel locations participating in the green deconvolution core. The green input pixels 44 that contribute to the inverse of the output pixel 40 are positioned in alternating positions in each column and row.

Image restoration circuit configuration

4 is a block diagram schematically illustrating an image restoration circuit 60 that can be used as the circuit 26 of FIG. 1 above in accordance with an embodiment of the present invention. Circuitry 60 includes an input memory 64 that accepts and buffers the stream of input pixel values produced by sensor 24. Therefore, the memory 64 is used as an input interface to the circuit 60. The input stream is provided to memory 64 at a pixel clock rate. The operation of circuit 60 (especially the operation of memory 64) is synchronized with the pixel clock.

From top to bottom, the input image is scanned column by column (i.e., written into memory 64). The pixels are read from the memory 64 in successive vertical lines of fifteen pixels so as to go horizontally in each pixel clock cycle. In other words, if the output of the memory 64 includes pixels [j...j+14,i] in a certain pixel clock cycle (ie, the i-th pixel from the jth to j+14th columns), the memory will be in the next clock cycle. The output pixel [j...j+14, i+1]; that is, a fifteen pixel row below the same column. After reading the last line of the jth...j+14 column from the memory 60, the reading continues at the beginning of the next column, and the pixels [j+1...j+15,1] are output. The memory 60 is typically sized to store fourteen pixel columns, although any other suitable memory size can be used.

The output of the memory 60 is provided to a green pixel de-folder 68 and a red/blue pixel de-folder 72 controlled by the timing/selection unit 76. The inverse depreciator 68 uses the one hundred and thirteen coefficient sets stored in the green coefficient memory 80 to deconvolute the green sub-image, i.e., the green input pixel value. The inverse depreciator 72 uses the two individual sets of forty-nine coefficients stored in the R/B coefficient memory 84 to decompose the red and blue sub-images. An exemplary deconvolution configuration that can be used to implement the deconvolutions 68 and 72 is further described below.

As shown in Figures 2 and 3B above, the green pixels are positioned in alternating locations in each of the input image and the green deconvolution core. As shown in Figures 2 and 3A above, the R/B pixel is positioned in a fixed position in each of the second line of the input image and the R/B de-convolution core. To control the deconvolutions 68 and 72, the timing/selection unit 76 triggers between the two states in response to the pixel clock. In one state, unit 76 causes inverse depreciator 68 to load and latch the even-numbered pixel values in the fifteen pixel rows of its input. In the second state, unit 76 causes inverse depreciator 68 to load and latch odd-numbered pixel values. At the same time, in one state, unit 76 causes inverse deconvolution 72 to load and latch seven even-numbered pixel values from the fifteen pixel rows of its input. In the second state, unit 76 causes inverse deconvolution 72 to not load any pixel values. Unit 76 may be implemented using a flip-flop device and/or any other suitable logic.

A multiplexer (MUX) 88 synchronized and controlled by unit 76 alternates between the outputs of inverse depreciators 68 and 72 in each pixel clock cycle. Thus, the output of MUX 88 includes a stream of output pixel values that follow the same Bayer mosaic pattern of the input stream. Since the output sub-image is generated by decomposing the individual input sub-images by using a plurality of sets of inverse-folding coefficients, the level of blur and other optical distortions in the output sub-image is reduced relative to the input sub-image.

The output pixel value stream is provided to ISP 28, which produces a combined color output image, as explained above.

5A and 5B are block diagrams that schematically illustrate details of a green pixel de-folder 68 in accordance with an embodiment of the present invention. The inverse depreciator 68 includes a two-dimensional array of sixty-four filter units 92. The internal structure of unit 92 is illustrated below in Figure 5B. Sixty-four filter units are arranged in an array of eight columns by eight rows.

The leftmost line of unit 92 is connected to the inverse depreciator input; that is, the fifteen pixel rows read from memory 64 are accepted, as illustrated in Figure 4 above. In each pixel clock cycle, each cell 92 accepts even or odd pixel values from its left neighboring cell (or from the input when the leftmost cell in the cell family), multiplying the pixel value by the appropriate deconvolution product. The coefficient produces a result in its output 埠 (represented as P). At the same time, each cell (except for cells in the rightmost row of the array) transmits the pixel values used in the previous clock cycle to its right neighboring cell in the array.

Each filter unit 92 performs a single multiplication in each pixel clock cycle. The deconvolution of each green pixel value is performed during two consecutive pixel clock cycles. If a given unit 92 processes an even number of pixel values during the first clock cycle, the unit will process an odd number of pixel values in the second clock cycle, and vice versa, since the pixel values traverse the array. Thus, the total deconvolution result (i.e., the value corresponding to the green output pixel) is equal to the sum of all the outputs P of all cells 92 in the two clock cycles.

The convolver 68 includes a 64-input adder 96 that summarizes the output of sixty-four outputs P of unit 92. In some embodiments, adder 96 has a pipelined internal structure, although any other suitable adder can be used. The output of the adder 96 in the two consecutive clock cycles is accumulated by the register 100 and the adder 104. In the first clock cycle, the output of the adder 96 (i.e., the partial deconvolution result) is latched by the register 100. In the second clock cycle, adder 104 uses the value of register 100 to summarize the output of adder 96 to produce a total deconvolution result.

Figure 5B is a block diagram schematically illustrating the internal structure of a filter unit 92 in accordance with an embodiment of the present invention. The even and odd input pixel values are alternately loaded into the even pixel latch 108 and the odd pixel latch 112, respectively. The pixel value is loaded from the left neighboring cell of the cell in the array or from the memory 64 when the cell is the rightmost cell in its column.

The corresponding inversely-folded coefficients are loaded from the memory 80 into the even coefficient register 116 and the odd coefficient register 120. Any suitable method can be used to read the appropriate coefficients. The two switches 124 and 128 controlled by unit 76 are selected to multiply the even or odd pixel values in each pixel clock cycle. Switches 124 and 128 typically include a CMOS multiplexer.

In some embodiments, the loading and multiplication of pixel values are performed side by side. In these particular embodiments, the even-numbered pixel values are multiplied during the clock cycle in which the odd-numbered pixel values are loaded, and vice versa. The multiplier 132 multiplies the appropriate value by the corresponding deconvolution coefficient and outputs the result in the output P of the filter unit. The multiplier operates at the pixel clock rate, which performs a single multiplication in each pixel clock cycle.

In some embodiments, the inverse-folding coefficients are represented using a signed 10-bit representation, the input pixels are represented by unsigned 10-bits, and the multipliers comprise signed 10-bits x An unsigned 10-bit multiplier. A signed 20-bit representation multiplier output can be used (in this and other embodiments). However, in place of specific implementations, any other suitable quantification may be used.

In some embodiments, different numbers of bits can be used to represent different coefficients. For example, a higher precision (i.e., a larger number of bits) may be used to represent the coefficients at the center of the deconvoluted core as compared to coefficients that are positioned further away from the center. In these specific embodiments, the multiplier can also be implemented with a non-uniform number of bits. Using non-uniform quantization can reduce the size and cost of the circuit.

Figure 6A is a block diagram schematically illustrating details of a red/blue pixel de-folder 72 in accordance with an embodiment of the present invention. The inverse depreciator 72 includes twenty eight filter cells 144 arranged in a two-dimensional array of four columns by seven rows. Each cell 144 accepts two R/B pixel values from its left neighboring cell or from memory 64, multiplies the two pixel values by the appropriate deconvolution factor, and outputs the result in its output 埠P. Each cell 144 (except for cells in the rightmost row of the array) also passes the previous input pixel value to its right neighbor.

The loading and propagation of pixel values across the array is controlled by unit 76, as illustrated in Figure 4 above. Since only the R/B pixel is positioned in each second line of the image, all of the inverse of the R/B pixel values is performed during the two consecutive pixel clock cycles. The inverse depreciator 72 includes a 28 input adder 148 that summarizes the output of twenty eight output P of unit 144. The register 152 and the adder 156, similar to the register 100 and adder 104 of FIG. 5A above, accumulate the output of the adder 148 in two consecutive clock cycles. In the first clock cycle, the output of adder 148 is latched by register 152. In the second clock cycle, the output of adder 148 is added by adder 156 to the value of register 152 to produce a total deconvolution result.

Figure 6B is a block diagram schematically illustrating the internal structure of a filter unit 144 in accordance with an embodiment of the present invention. The two even input pixel values are loaded into two even pixel latches 160 and 164. Unlike the green pixel filter cell configuration of Figure 5B above, the two latches are latched simultaneously, rather than alternately, in this configuration. On the other hand, pixel values are only loaded in every second pixel clock cycle. The pixel value is loaded from the left neighboring cell of the cell in the array or from the memory 64 when the cell is the leftmost cell in its column.

The corresponding deconvolution coefficients are loaded from the memory 84 into the two coefficient registers 168 and 172. Any suitable method can be used to read the appropriate coefficients. The two switches 176 and 180 controlled by unit 76 are selected to multiply one of the two even-numbered pixel values in each pixel clock cycle. Switches 176 and 180 are similar to switches 124 and 128 of Figure 5B above. The multiplier 184 multiplies the appropriate value by the corresponding deconvolution coefficient and outputs the result in the output P of the filter unit. Multiplier 184 operates at a pixel clock rate.

The filter unit configuration of Figures 5A, 5B, 6A, and 6B assumes that all of the cells 92 are identical to each other, and that all of the cells 144 are identical to each other. In an alternative embodiment, some of the filter elements may be simplified to reduce the cost and size of the circuit 60. For example, the filter unit in the bottom column of the array in the inverse depreciator 68 and the inverse depreciator 72 processes only a single pixel value. These filter units can be replaced by filter units that have only a single pixel latch, a single coefficient register, a multiplier, and no switch.

By way of further example, the filter unit 92 in the rightmost row of the array of Figure 5A also processes only a single pixel value. These filter units can also be implemented using a simpler configuration that includes only a single pixel latch, a single coefficient register, a multiplier, and no switches.

In addition, the multiplier in the filter unit of the bottom column and the rightmost row of the inverse depreciator 68 and the bottom column of the inverse depreciator 72 is utilized only during 50% of the time. Therefore, the filter unit pairs in these columns/rows can share a single multiplier. Using this technique, the number of multipliers in the inverse depreciator 68 can be reduced from sixty-four to fifty-seven, and the number of multipliers in the inverse-folder 72 can be reduced from twenty-eight to twenty-five. .

The configuration of circuit 60 described above is an exemplary configuration. Any other suitable configuration can be used. For example, the size of memory 64 and the number of filter elements 92 and 144 in inverse depreciators 68 and 72 are selected to match the 15x15 pixel anti-aliased core used. When using different core sizes, the input memory and inverse depreciator configuration can be modified accordingly. The inverse deconvolution configuration can also be modified to suit other mosaic patterns.

FIG. 7 is a block diagram schematically illustrating an image restoration circuit 190 in accordance with another embodiment of the present invention. The overall configuration of circuit 190 is similar to circuit 60 of FIG. 4 above, and like circuit 60, circuit 190 can be used to implement circuit 26 of FIG. 1 above. Certain components of circuit 190 (e.g., memories 64, 80, and 84 and MUX 88) are similar to corresponding components of circuit 60.

However, in circuit 190, the inverse-folder multiplier operates at twice the clock rate of the clock rate of the multipliers of the de-folders 68 and 72. Therefore, the number of multipliers is reduced, and the cost and size of the circuit are correspondingly reduced.

Circuitry 190 includes a green pixel de-folder 194 and a red/blue pixel inverse-folder 198, which are described in more detail below in Figures 8 and 9, respectively. The four phase timing unit 202 generates a four phase timing signal in each of two consecutive pixel clock cycles. The timing signal is used to control and synchronize the higher clock rate operations of the two deconvolutions. The four phase timing signals can be provided to the deconvolution 194 and 198 using four discrete lines. Alternatively, unit 202 may encode four phases on two lines, or perform any other suitable encoding that enables the deconvolution to distinguish between four different phases of the timing signal.

FIG. 8 is a block diagram schematically illustrating details of a green pixel de-folder 194 in accordance with an embodiment of the present invention. The inverse depreciator 194 includes thirty-two filter cells 206 arranged in a two-dimensional array of four columns by eight rows. Each filter unit 206 processes four input pixel values and one of four corresponding inversely-folded coefficients.

Each filter unit 206 includes a single multiplier that operates at twice the pixel clock rate. During each phase of the four phase timing signal provided by unit 202, the multiplier multiplies the single input pixel value by a single coefficient and produces a result in the output P of the unit. Thus, each unit 206 performs multiplication during every two consecutive pixel clock cycles. Each filter unit 206 functions similar to the two units 92 as compared to the filter unit 92 of Figure 5A above. The total number of multipliers in the inverse depreciator 194 is thirty-two, that is, half the number of multipliers in the inverse depreciator 68.

The thirty-two outputs P of unit 206 are summarized by a 32-input adder 210. The adder 214 and the register 218, which are configured in the feedback configuration, accumulate the partial sums produced by the adder 210 during the four clock phases. The register 218 is initially cleared in the first clock phase and then the next output from the adder 210 is added to the register 218 in each successive clock cycle. At the end of the fourth clock phase, the output of adder 214 is equal to the desired deconvolution result, i.e., the value of the decomposed green pixel.

The following table illustrates the cumulative operation of adder 210, adder 214, and register 218:

FIG. 9 is a block diagram schematically illustrating details of a red/blue pixel de-folder 198 in accordance with an embodiment of the present invention. The inverse depreciator 198 includes fourteen filter units 222 arranged in a two-dimensional array of two columns by seven rows. Each unit 222 includes a single multiplier that operates at twice the pixel clock rate. Each multiplier multiplies four input pixel values by a corresponding deconvolution factor during the four clock phases provided by unit 202. The total number of multipliers in the inverse depreciator 198 is fourteen compared to the twenty-eight multipliers of the inverse deconvolution 72 of Figure 6A above.

The 14 input adder 226 summarizes the fourteen outputs P of the unit 222 in each of the four clock phases. Similar to adder 214 and register 218 of Figure 8, above, adder 230 and register 234 accumulate four partial results to produce a total deconvolution result.

Figure 10 is a block diagram schematically illustrating a reverse-fold filter unit 238 in accordance with an embodiment of the present invention. Filter unit 238 can be used as unit 206 in inverse deconvolution 194 of Figure 8 above, or unit 222 in inverse deconvolution 198 of Figure 9 above. In unit 238, four registers 242 latch the values of four adjacent vertical input pixels. The odd-numbered pixels are latched in phase 1 of the four-phase signal by unit 202 and the even-numbered pixels are latched in phase 3 of the four-phase signal.

The four corresponding inversely folded coefficients are stored in the coefficient register 246. Any suitable method can be used to load the appropriate coefficients from memory 80 or 84. During each of the four phases of the clock signal of unit 202, two 4-input multiplexers 250 sequentially select a single pixel value from register 242, and a corresponding coefficient value from register 246. In each phase, the selected pixel value and coefficient value are provided to multiplier 254. The multiplier multiplies the pixel value by the coefficient and outputs the result in the unit output P.

Shared multiplier configuration

Figure 11 is a block diagram schematically illustrating an image restoration circuit 260 in accordance with another embodiment of the present invention. Unlike the image restoration circuit described above, the multiplier in circuit 260 is shared between the green and red/blue pixel deconvolution programs. This sharing of the multipliers further reduces the number of multipliers and thus further reduces the cost of the image restoration circuit.

As explained above, the number of multiplications performed in the deconvolution of a single green pixel is 7*7+8*8=113, and the number of multiplications performed in the deconvolution of the red/blue pixels is 7*7=49. . The total number of multiplications in any two consecutive pixel clock cycles is therefore 113 + 49 = 162.

Circuitry 260 includes forty-one multipliers that operate at a clock rate that is twice the pixel clock rate. In every two consecutive pixel clock cycles, the circuit performs 162 multiplications according to a predetermined assignment scheme by assigning a multiplier to the coefficient/pixel pair. Each of the forty-one multipliers performs four multiplications in every two consecutive pixel clock cycles. At this time, the 41st is the smallest feasible multiplier because the ceiling is (162/4) = 41.

The input pixel stream is stored in input memory 262, similar to memory 64 of Figure 4 above, and is read in a 15 pixel vertical line. A 4-phase timing unit (not shown) similar to unit 202 of Figure 7 above produces a four-phase timing signal at twice the pixel clock rate. In other words, the signal alternates between four phases in every two pixel clock cycles.

Pixel array 264 maintains a matrix of 15x15 pixels. The pixel values in the pixel array are offset to the right side every two phases of the clock signal (ie, whenever a new pixel row is read from memory 262).

Multiplier repository 268 includes forty-one multipliers. Each multiplier performs a single multiplication in each phase of the 4-phase timing signal. A pixel multiplexer repository 272 comprising forty-one 4-input multiplexers provides each multiplier with appropriate pixel values from pixel array 264 during each phase of the 4-phase timing signal. A coefficient multiplexer repository 276, also including forty-one 4-input multiplexers, provides a corresponding inverse-fold factor for each multiplier. The inverse fold coefficient values are stored in the coefficient memory 280. Thus, one of each multiplier in the repository 268 is input to a multiplier in the repository 272 and another multiplier input is coupled to a multiplier in the repository 276.

Each of the forty-one multipliers is assigned in each of the four phases of the timing signal to multiply a particular coefficient value (which may be green or red/blue) by a particular coefficient value. Any suitable multiplier assignment to pixel values and coefficients can be used. The following two tables illustrate an exemplary multiplier assignment scheme that simplifies and minimizes the routing path in the circuit and thus minimizes circuit power consumption. The tables specify which pixel values (i.e., columns and rows in pixel array 264) are assigned to each multiplier in each of the four clock phases. The tables also indicate whether each pixel is green or red/blue.

The multiplex scheme defined by the above two tables assumes that the first pixel in each column of array 264 is a green pixel. When the first pixel in the column is an R/B pixel, the multiplier assignment is slightly modified by selecting the right two rows of R/B pixels.

To produce a deconvolution result, one hundred and sixty-two multiplication results performed by forty-one multipliers during the four clock phases should be properly accumulated. All multiplier outputs corresponding to the green pixels should be accumulated to produce a green output pixel value. Likewise, all outputs corresponding to the red/blue pixels should be accumulated to produce a red/blue output pixel value.

Green pixel selection block 284 selects a multiplier output corresponding to the green pixel in each of the four clock phases. (In the above two multiplex tables, the multiplier output corresponding to the green pixel is identified as "G".) The output of block 284 is summarized by 41 input adder 288, which is at each of the four clock phases The sum of the partial values of the green pixels is generated. The four partial results produced by adder 288 are accumulated by an accumulation circuit that includes an adder 292, a register 300, and a multiplexer 296. At the end of the four clock phases, the total de-complex green pixel values are stored in the scratchpad 304.

A similar procedure is applied to the multiplication result corresponding to the R/B pixel. R/B pixel selection block 308 selects the multiplier output corresponding to the R/B pixel (the multiplier output identified as "R/B" in the multiplex table above). The output of block 308 is summarized by a 33 input adder 312 which produces a sum of the four portions of the R/B pixel values in each of the four clock phases. The four partial results are accumulated by an accumulation circuit including an adder 316, a register 320, and a multiplexer 324. At the end of the four clock phases, the total decomposed R/B pixel values are stored in register 328. The multiplexer 332 alternately selects the outputs of the registers 304 and 328 in each pixel clock cycle to produce a Bayer mosaic output pixel stream.

It should be noted that although adder 288 includes 41 input adders in FIG. 11 and adder 312 includes 33 input adders, adders with fewer inputs may be used in the event of impairing higher complexity multiplexes and lines. In the exemplary multiplier assignment table provided above, there are at most thirty-nine multiplier outputs (in phase 3) corresponding to green pixels in each of the four phase timing signals, and corresponding to R/B Up to twenty-seven outputs of the pixel (in phase 2). Thus, assuming an appropriate multiplex, the adder 288 can include a 39 input adder and the adder 312 can include a 27 input adder.

In summary, the circuit configuration of Figure 11 uses a library of forty-one multipliers (each multiplier operates at twice the pixel clock rate), implementing the deconvolution of green and red/blue pixels.

Pipelined deconvolution configuration

In the image restoration circuit configuration described above, the inverse of each output pixel is calculated by multiplying the appropriate input pixel value by the corresponding deconvolution coefficient and accumulating the product. The incoming input pixel values are stored in a scratchpad (e.g., memory 64). For example, when the size of the deconvolution core is 15x15 pixels, one hundred and sixty registers are used to store the input pixel values at any given time.

In some embodiments, the number of registers required to store input pixel values is reduced by using a pipelined deconvolution configuration. Reducing the number of registers can reduce the die area and thus the cost of the image restoration circuit.

In the tube linear configuration described below, the inverse-folder calculates the contribution of the pixel values introduced in each row to the fifteen partial deconvolution results. Therefore, only partial deconvolution results are stored, rather than individual input pixel values. At the same time, the inverse depreciator accumulates the previous partial results and outputs the total deconvolution result.

Obviously, the number of partial results is considerably less than the number of individual pixel values, thus greatly reducing the number of registers in the circuit.

Figure 12 is a block diagram schematically illustrating a pipelined de-folder 340 in accordance with an embodiment of the present invention. The inverse depreciator 340 includes sixteen multipliers arranged in a two-dimensional four by four array. This simplified configuration shows the principle used to demonstrate and clarify the pipelined deconvolution.

Four consecutive rows of pixels enter the deconvolution at the pixel clock rate from the right. The pixel value row is represented as (P0 _i , P1 _i , P2 _i , P3 _i ) ^T , where i=0, 1, 2, 3... represents the pixel clock cycle. The inverse depreciator filters the introduced pixel values using the core of the sixteen coefficients stored in coefficient register 344. The coefficients are expressed as Cmn, m, n = 0...3. Each coefficient register is coupled to a multiplier 348 that operates at a pixel clock rate.

Each introduced pixel value is simultaneously supplied to all of the multipliers in its column. For example, the input pixel value P1 _{i is} simultaneously supplied to all four multipliers of the second column, that is, multipliers coupled to the coefficients C10, C11, C12, and C13. In each clock cycle, each multiplier multiplies the value of the input pixel in its input by the coefficient coupled to the multiplier.

The multiplication product in each multiplier row is accumulated by adder 352. The sum of the multiplication products of the leftmost row of the array is represented as S0. Similarly, the cumulative multiplication of the other lines is expressed as S1, S2, and S3.

Use the horizontal delay line to sum the line sums so that the partial contribution of each pixel is correctly accumulated. (The array simultaneously processes pixel values belonging to different pixels. For example, when the row sum S0 refers to the output of pixel i, the next row sum S1 refers to the output of pixel i+1. Therefore, the row sum S1 is delayed by one clock cycle with respect to S0.) Delay The lines include delay lines 356, each of which delays its input by a single pixel clock cycle. Delay and summarize the row sum to produce a deconvolution result expressed as OUT. The delayed partial results in the output of delay line 356 are represented as A, B, and C.

In each pixel clock cycle, the inverse depreciator 340 calculates a partial deconvolution result belonging to four different pixels. Each of the partial deconvolution results S0...S3 belongs to a different pixel.

The operation of the pipelined deconvolution is demonstrated by the following table showing the sum of the row sums S0...S3, points A, B, C and OUT during the six consecutive pixel clock cycles. The summary operations (denoted as Σ) in these tables are in the range of i=0...3.

As can be seen from these tables, the outputs A, B, C, and OUT are not initially defined and their values are represented as X. In the case of each clock cycle, the output begins to fill until all outputs produce a reasonable value in the fourth clock cycle. From this stage, the output of the deconvolution is equal to Where k represents the index of the introduced pixel row. As can be seen from the above equation, OUT _k is indeed equal to the inverse of the pixel P introduced by the inverse fold coefficient C.

FIG. 13 is a block diagram schematically illustrating a green pixel pipeline type de-folder 360 in accordance with an embodiment of the present invention. The inverse depreciator 360 includes sixteen filter units 364, each of which operates and processes four deconvolution coefficients at twice the pixel clock rate. Thus, a de-folder 360 can be used in place of the deconvener 194 in the circuit 190 of Figure 7 above. The internal structure of the filter unit 364 is explained below in FIG.

The inverse depreciator 360 has a pipelined configuration similar to the configuration of Figure 12 above. Each filter unit 364 produces a portion of the result (denoted as PR) and provides the result to the next lower filter unit in its row. Each filter unit that accepts partial results from neighboring cells adds the accepted partial results to its own multiplication result and then forwards the combined result to the next unit of the next row.

The partial result of each row of the cells is accumulated by the serial adder 368. Each of the serial adders 368 accumulates partial results of the cells in its row during the four clock phases of every two consecutive pixel clock cycles. Each serial adder 368 passes its output to the next string adder on its right side, where the result is added to the partial result of the next line. The rightmost serial adder produces the total deconvolution result.

Each serial adder 368 includes a multiplexer 372, a register 376, and an adder 380. The multiplexer 372 transmits the output of the left adjacent row to the second input of the adder 380 during the first clock phase. During the other three phases, multiplexer 372 sends the output of register 376 to this input. In the multiplexer 372 of the leftmost serial adder, a zero value is sent to the adder during the first phase. The pipeline delay between adjacent tandem adders is four phases, a two pixel clock cycle. This delay corresponds to the gap between two consecutive green pixels.

Figure 14 is a block diagram schematically illustrating a red/blue pixel pipelined deconvolution 384 in accordance with an embodiment of the present invention. The inverse depreciator 384 includes fourteen filter units 364 and operates in a manner similar to the inverse deconvolution 360 of Figure 13 above.

Figure 15 is a block diagram schematically illustrating the internal structure of a filter unit 364 used in the anti-folders 360 and 384 of Figures 13 and 14 above in accordance with an embodiment of the present invention.

Unit 364 includes a coefficient register 388, a multiplexer 392, and a multiplier 396 that functions similarly to the coefficient register 246, multiplexer 250, and multiplier 254 of the filter unit 238 illustrated in FIG. However, in the present embodiment, the output of multiplier 396 is stored in scratchpad 400 and added to the partial results accepted from neighboring cells using adder 404. The output of adder 404 is provided as a partial result (PR) output of the unit.

Although the above embodiments of Figures 13 through 15 refer to green and R/B de-folders operating at twice the pixel clock rate, these embodiments are shown by way of example to demonstrate a pipelined de-folder configuration. In an alternative embodiment, a similar pipelined deconvolution device can be used to implement other types of monochromatic (ie, green or R/B) de-folders with other internal clock rates, such as for Figure 4 above. The inverse of the circuit 60. The pipeline configuration can also be used for a shared multiplier deconvolution, such as the shared multiplier deconvolution in circuit 260 of Figure 11 above.

Image segmentation and anti-folding filter symmetry

In some practical situations, the blurring effect caused by optical device 22 is not uniform across the input image. For example, the PSF of optical device 22 may vary due to the area of the image. Thus, in some embodiments, different sets of deconvolution coefficients are used in different regions of the input image.

On the other hand, the use of a large number of different deconvolution cores greatly increases the memory used to store the deconvolution coefficients. For example, consider a 1,920,000 pixel input image with 1,600 rows by 1,200 columns. In a typical application, this image is divided into 768 segments of 50 x 50 pixels. Each segment is mapped to a particular deconvolution kernel of 15 x 15 coefficients, of which two hundred and eleven are non-zero coefficients. In this exemplary configuration, the use of different cores for each segment requires storage of 211 x 768 = 162,048 decoupling coefficients.

However, in many cases, blur or other distortion of the input image is circularly symmetric around the center of the image. For example, in many optical systems, the PSF of a lens that is part of the optical device 22 is symmetrical with respect to the axis of the lens. In such cases, because the lens system is symmetric around its axis, all PSFs that have the same distance from the center of the input image will be a rotated version of a single PSF function.

However, when dealing with discrete PSFs that are defined, for example, as 15x15 square grids, if and only if the angle of rotation is a multiple of 45°, then the rotated version of a given PSF will map to other grid locations of 15x15 squares. .

When the optical system has such circular symmetry, a relatively small baseline deconvolution core set can be used. Additional cores applied in regions having similar distances to the baseline core but different angles of rotation are generated from the baseline core, for example by symmetrically reversing the baseline core in 45[deg.] increments.

Figure 16 is a graph showing the symmetry of an anti-folding filter in accordance with an embodiment of the present invention. In the exemplary embodiment of Figure 16, the input image is divided into eight sector symmetrical regions around its center. The symmetry zones are numbered 1 to 8. The deconvolution core used in Zone 1 is considered a baseline, unrotated core. In each of these cores, as shown in the figure, the upper left coefficient is represented as 0,0; the upper right coefficient is represented as 1,14; the lower left coefficient is represented as 14,0, and the lower right coefficient is represented as 14 , 14. Use this core symmetric replicator for the other seven zones.

To calculate a core in zone 2, the following steps are taken: First, select the baseline core corresponding to zone 2 (ie, the core of zone 1). The selection of the appropriate baseline core is performed by reversing the position of the core of the zone 2 relative to the 45[deg.] axis separating zone 1 from zone 2. The selected baseline core is then inverted relative to this 45[deg.] axis to produce the desired core.

Cores for other symmetric regions are generated in a similar manner. Each core includes an inverted, symmetrical view of its adjacent core relative to the axis separating the two. With this technique, it is only necessary to store the coefficient values of the baseline core in the memory.

Figure 17 is a graph showing symmetric assignment of an inverse-folding filter to a segment in accordance with an embodiment of the present invention. The figure shows a symmetric equivalent region in a 1,920,000 pixel image, assuming 50 x 50 pixel segments. Each segment in Fig. 17 has a number within the range 1...8 of the symmetric region in Fig. 16 above.

Figure 18 is a graph showing the assignment of an inversely-folded coefficient to a segment of an image in accordance with an embodiment of the present invention. The figure shows the image of Figure 17 above, which is divided into seven hundred and sixty-eight 50 x 50 pixel segments 408. The segment belonging to the symmetry zone 1 is identified by an incremental core index between 1...126. In the other seven symmetrical regions, each segment 408 is identified by its index of the corresponding baseline core. Therefore, one hundred and twenty-six different cores can be used instead of seven hundred and sixty-eight cores to deconvolute the entire image.

It should be noted that because the input image is rectangular rather than square, the eight-shaped 45° symmetry only applies to the center and square of the image. As can be seen from Figure 18, the four leftmost segment rows have a 90° symmetry with the four rightmost segment rows (corresponding to segment indices 78...125). Determining the deconvolution cores in such regions is performed by inverting the appropriate baseline cores in zone 1 relative to the vertical and/or horizontal axes.

In an alternative embodiment, the input image can be divided into different numbers of segments and/or symmetric regions. For example, the same 1,920,000 pixel image can be divided into 100 x 100 pixel segments, resulting in thirty-two different symmetrically equivalent segments in each symmetric region.

Alternatively, different regions of the input image can be divided into segments having different sizes. The segment size can be selected to match the spatial variability of the PSF in different regions of the image. In other words, the area of the input image in which the PSF changes rapidly can be divided into smaller sections, and other areas in which the PSF changes more slowly can be divided into larger sections.

Although the specific embodiments described herein primarily refer to mosaics having green, red, and blue filters, the methods and systems described herein can also be used with other types of mosaic filters. For example, such inlays may include yellow, magenta, cyan, and green filters typically used in charge coupled device (CCD) image sensors.

Although the specific embodiments described herein address image quality enhancement in digital cameras, the principles of the present invention can also be applied to other image processing applications, such as pattern recognition applications.

Therefore, it should be understood that the specific embodiments described above are cited by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention includes the combinations and sub-combinations of the various features described above, and variations and modifications which will be apparent to those skilled in the art of <RTIgt;

20. . . camera

twenty two. . . Optical equipment

twenty four. . . Sensor

26. . . Circuit

28. . . ISP

40. . . Pixel

44. . . Pixel

48. . . Pixel

52. . . Pixel

60. . . Circuit

64. . . Memory

68. . . Green anti-folding device

72. . . Red/blue reverse folding device

76. . . Timing/selection unit

80. . . Green coefficient memory

84. . . R/B coefficient memory

88. . . MUX

92. . . unit

96. . . 64 input adder

100. . . Register

104. . . Adder

108. . . Even pixel latch

112. . . Odd pixel latch

116. . . Even coefficient register

120. . . Odd coefficient register

124. . . switch

128. . . switch

132. . . Multiplier

144. . . Filter unit

148. . . 28 input adder

152. . . Register

156. . . Adder

160. . . Latches

164. . . Latches

168. . . Register

172. . . Register

176. . . switch

180. . . switch

184. . . Multiplier

190. . . Circuit

194. . . Anti-folder

198. . . Anti-folder

202. . . Timing unit

206. . . Filter unit

210. . . Adder

214. . . Adder

218. . . Register

222. . . Filter unit

226. . . 14 input adder

230. . . Adder

234. . . Register

238. . . Filter unit

242. . . Register

246. . . Register

250. . . Multiplexer

254. . . Multiplier

260. . . Circuit

264. . . Pixel array

268. . . Repository

272. . . Repository

276. . . Repository

280. . . Coefficient memory

284. . . Select block

288. . . 41 input adder

292. . . Adder

296. . . Multiplexer

300. . . Register

304. . . Register

308. . . Select block

312. . . 33 input adder

316. . . Adder

320. . . Register

324. . . Multiplexer

328. . . Register

332. . . Multiplexer

340. . . Anti-folder

344. . . Register

348. . . Multiplier

352. . . Adder

356. . . Delay line

360. . . Anti-folder

364. . . Filter unit

368. . . Tandem adder

372. . . Multiplexer

376. . . Register

380. . . Adder

384. . . Anti-folder

388. . . Coefficient register

392. . . Multiplexer

396. . . Multiplier

400. . . Register

404. . . Adder

408. . . Section

The invention will be more fully understood from the foregoing detailed description of the preferred embodiments of the invention and the appended drawings, in which: FIG. 1 is a schematic illustration of an electronic imaging camera in accordance with an embodiment of the present invention. 2, 3A and 3B are schematic illustrations of a deconvolution core according to an embodiment of the present invention; and FIG. 4 is a block diagram schematically illustrating an image restoration circuit according to another embodiment of the present invention; 5A and 5B are block diagrams schematically illustrating a green pixel de-folder used in the circuit of FIG. 4 in accordance with an embodiment of the present invention; FIGS. 6A and 6B are schematic illustrations in accordance with the present invention; A block diagram of a red/blue pixel de-folder used in the circuit of FIG. 4 of one embodiment; FIG. 7 is a block diagram schematically illustrating an image restoration circuit in accordance with another embodiment of the present invention; 8 is a block diagram schematically illustrating a green pixel de-folder used in the circuit of FIG. 7 in accordance with an embodiment of the present invention; FIG. 9 is a schematic illustration of an embodiment in accordance with the present invention. In the circuit of Figure 7 Block diagram of a red/blue pixel inverse-folder used; Figure 10 is a block diagram schematically illustrating a reverse-fold filter unit used in the circuits of Figures 8 and 9 in accordance with an embodiment of the present invention. Figure 11 is a block diagram schematically illustrating an image restoration circuit in accordance with another embodiment of the present invention; and Figure 12 is a block diagram schematically illustrating a pipeline-type de-folder in accordance with an embodiment of the present invention; Figure 13 is a block diagram schematically illustrating a green pixel pipeline type de-folder in accordance with an embodiment of the present invention; Figure 14 is a schematic illustration of a red/blue pixel in accordance with an embodiment of the present invention. Block diagram of a pipelined de-folding device; Figure 15 is a block diagram schematically illustrating a reverse-folding filter unit used in the pipelined de-folding apparatus of Figures 13 and 14 in accordance with an embodiment of the present invention. Figure 16 is a graph showing the symmetry of the inverse-folding filter according to an embodiment of the present invention; Figure 17 is a diagram showing the symmetric assignment of the anti-folding filter to an image segment in accordance with an embodiment of the present invention. The graph; and Figure 18 shows the basis of the present One embodiment of a section of a graph image of the product of the coefficient assigned to the fold.

60. . . Circuit

64. . . Memory

68. . . Anti-folder

72. . . Anti-folder

76. . . Timing/selection unit

80. . . Memory

84. . . Memory

88. . . MUX

Claims (30)

An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream; a plurality of filter units are coupled in a two-dimensional array configuration and configured Separatingly filtering each of the input pixel values of the input sub-images using an individual two-dimensional doconvolution kernel to generate an individual output sub-image comprising output pixel values; wherein the plurality of filter elements Each includes a circuit component; wherein each of the circuit components provides an output, and wherein the image enhancement circuit includes an adder that sums the outputs; and a multiplexer coupled to The output pixel values of the output sub-images are multiplexed together to produce a filtered output image. The circuit of claim 1, wherein the input image has an input blur, and wherein after the inverse deconvolution core filtering is employed, the output image has an output blur that is less than the input blur. An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are Interleaving in a stream; a plurality of filter elements coupled in a two-dimensional array configuration and configured to separately filter each of the input sub-images using an individual two-dimensional de-convolution core Numerical values to produce an individual output sub-image comprising output pixel values; wherein each of the plurality of filter elements comprises a circuit component; and a multiplexer coupled to output the output sub-images Pixel values are multiplexed together to produce a filtered output image; wherein the array configuration has columns and rows of the filter elements, wherein the input interface includes a memory configured to buffer the input pixel values And providing successive rows of the buffered input pixel values to the filter cells, and wherein each row of the filter cells is configured to propagate the input pixel values to the array configuration The lower row of the filter unit is used to deconvolve the input pixel values using the deconvolution cores. The circuit of claim 3, wherein the deconvolution kernels comprise filter coefficients, and wherein each of the filter units comprises a multiplier configured to multiply one of the input pixel values by the One of the filter coefficients to produce the output of the filter unit. A circuit as claimed in claim 4, wherein the multiplier is configured to perform two multiplications in each cycle of receiving one of the pixel clocks of the input pixels. The circuit of claim 4, comprising an accumulator circuit configured to The outputs of the filter cells are accumulated to produce the output pixel values. The circuit of claim 6, wherein the accumulator is configured to accumulate the outputs in two consecutive cycles of accepting one of the pixel clocks of the input pixels. An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream; a plurality of filter units are coupled in a two-dimensional array configuration and configured Separatingly filtering each of the input pixel values of the input sub-images using an individual two-dimensional inverse-folding core to generate an individual output sub-image comprising output pixel values; wherein each of the plurality of filter elements comprises a circuit component; and a multiplexer coupled to multiplex the output pixel values of the output sub-images to produce a filtered output image; wherein the array configuration has the filter elements a row and a row, wherein the input interface is configured to provide successive rows of the input pixel values to the filter cells such that each input pixel value is the same All such units to provide one such filter unit filters the corresponding column, wherein each filter unit comprises a multiplier configured to those provided by the output The pixel value is multiplied by the filter coefficient of one of the deconvolution kernels to produce a multiplication result; and includes an accumulation circuit configured to calculate the row in each of the rows of the filter cells The sum of the rows of the multiplication results, such that the delay is applied to the sum of the rows and the sum of the rows of the delays is combined to produce the output pixel values. An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream; a plurality of filter units are coupled in a two-dimensional array configuration and configured Separatingly filtering each of the input pixel values of the input sub-images using an individual two-dimensional inverse-folding core to generate an individual output sub-image comprising output pixel values; wherein each of the plurality of filter elements comprises a circuit component; and a multiplexer coupled to multiplex the output pixel values of the output sub-images to produce a filtered output image; wherein the input pixel is provided by a mosaic image sensor The stream of values is such that the image sensor produces the pixel values in each of the input sub-images in response to a different individual color of light Wherein the filter and the other filter unit is configured to generate such input sub-images corresponding to the color of these individual sub-images of such output; wherein such input sub-Movies The image includes green, red, and blue sub-images, and the filter elements are disposed in a first two-dimensional array configured to filter the input values belonging to the green sub-image; In a second two-dimensional array, the second two-dimensional array is configured to alternately filter the input values belonging to the red and blue sub-images. The circuit of claim 9, wherein the deconvolution kernels comprise filter coefficients, wherein the filter units comprise individual multipliers configured to multiply the input pixel values by the filter according to a predetermined designation The coefficients, and including selection and accumulation logic, are configured to separately accumulate the outputs of the multipliers multiplied by the input pixel values of each of the input sub-images to produce the output pixel values. An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream; a plurality of filter units are coupled in a two-dimensional array configuration and configured Separatingly filtering each of the input pixel values of the input sub-images using an individual two-dimensional inverse-folding core to generate an individual output sub-image comprising output pixel values; wherein each of the plurality of filter elements comprises a circuit component; and a multiplexer coupled to output the output sub-images The pixel values are multiplexed together to produce a filtered output image; wherein the input image is divided into a plurality of segments, and wherein the filter cells are configured to use at least a first and a second different deconvolution product The core filter positions the values of the input pixels in at least one of the first and second regions of the input image; wherein the first and second segments have individually different first and second sizes It is determined individually that one of the first and second sections should be ambiguous and one of the spatial variability is blurred. An image enhancement circuit includes an input interface operative to receive a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual pixels a subset, such that the input pixel values of the pixels in the different input sub-images are interleaved in the stream; a plurality of filter units are coupled in a two-dimensional array configuration and configured Separatingly filtering each of the input pixel values of the input sub-images using an individual two-dimensional inverse-folding core to generate an individual output sub-image comprising output pixel values; wherein each of the plurality of filter elements comprises a circuit component; and a multiplexer coupled to multiplex the output pixel values of the output sub-images together to produce a filtered output image; wherein one of the ones of the input image is blurred A symmetrical characteristic divides the input image into symmetrical regions, and wherein the filter units are configured such that The baseline deconvolution core filter is used to locate the values of the input pixels in one of the symmetric regions, and filter positioning is performed using individual deconvolution kernels determined by applying symmetric operations to the baseline cores. The values of the input pixels in the other symmetrical regions. A method for imaging, comprising: receiving a stream of input pixel values of pixels belonging to an input image, the input image comprising a plurality of different input sub-images including individual subsets of the pixels, such The input pixel values of the pixels in different input sub-images are interleaved in the stream; using a plurality of hardware-implemented filter units connected to a two-dimensional array configuration, using individual two-dimensional The inverse-folding core separately filters the input values of each of the input sub-images to generate an individual output sub-image comprising the output pixel values; wherein the two-dimensional array configuration has the plurality of filter elements and Rowing; wherein filtering the input pixel values comprises propagating the rows of the input pixel values from each of the plurality of filter cells to a row below the plurality of filter cells in the two-dimensional array configuration, In order to use the deconvolution cores to deconvolve the input pixel values; and to multiply the output pixel values of the output sub-images to produce a filtered output Images. The method of claim 13, wherein the input image has an input blur, and wherein filtering the input pixel values comprises causing the output image to have an output blur that is less than the input blur. The method of claim 13, wherein the accepting the stream of input pixel values comprises buffering the input pixel values and providing successive rows of the buffered input pixel values to the filter units. The method of claim 13, wherein the deconvolution kernel comprises a filter coefficient, wherein each of the filter units comprises a multiplier, and wherein filtering the input pixel values comprises using each of the multipliers One of the input pixel values is multiplied by one of the filter coefficients to produce an output of the filter unit. The method of claim 16, wherein multiplying the input pixel values by the filter coefficients is included in each cycle of receiving one of the pixel clocks of the input pixels, and performing two multiplications by each multiplier . The method of claim 16, wherein filtering the input pixel values comprises accumulating the outputs of the filter cells to generate the output pixel values. The method of claim 18, wherein the accumulating the outputs of the filter cells is included in two consecutive cycles of accepting one of the pixel clocks of the input pixels, accumulating the outputs. The method of claim 13, wherein the array configuration has columns and rows of the filter cells, wherein the stream that accepts input pixel values includes a continuous row providing the input pixel values to the filter cells such that Providing each input pixel value to all of the filter cells in a corresponding column of one of the filter cells, and wherein filtering the input pixel values includes providing the input pixel in each filter cell The value is multiplied by the filter coefficient of one of the inverse-folded cores to produce a multiplication result, which is calculated The sum of the rows of the multiplication results in each of the rows of the filter cells is applied to delay the sum of the rows and combine the sum of the rows of the delays to produce the output pixel values. The method of claim 13, wherein the accepting the stream of input pixel values comprises accepting the stream from a mosaic image sensor to generate the input pixels in each of the input sub-images by the image sensor A value in response to a different, individually colored light, and wherein filtering the input pixel values comprises filtering the input sub-images to produce the output sub-images corresponding to the individual colors. The method of claim 21, wherein the input sub-images comprise green, red, and blue sub-images, and wherein filtering the input pixel values comprises filtering the green color using one of the first two-dimensional arrays of the filter units The input values of the sub-images are used to alternately filter the input values belonging to the red and blue sub-images using a second two-dimensional array of the filter elements. The method of claim 13, wherein filtering the input pixel values comprises multiplying the input pixel values by the filter coefficients of the deconvolution cores using multipliers in the filter units according to a predetermined multiplier assignment And separately accumulating the outputs of the multipliers multiplied by the input pixel values of each of the input sub-images to produce the output pixel values. The method of claim 13, wherein filtering the input pixel values comprises dividing the input image into a plurality of segments, and using at least one of the first and second different inverse-folding cores to filter and respectively locate the input image individual The values of the input pixels in at least one of the first and second regions. The method of claim 24, wherein dividing the input image comprises selecting different first and second sizes for the first and second segments, respectively, to wait for one of the first and second segments to be blurred. A spatial variability. The method of claim 13, wherein filtering the input pixel values comprises dividing the input image into a symmetric region according to a symmetry characteristic of one of the input images, and filtering the positioning in the symmetric region using a baseline deconvolution kernel The values of the input pixels in one of the plurality of input pixels, and the individual deconvolution kernels determined by applying the symmetric operations to the baseline cores to filter the input pixels positioned in the other symmetric regions Wait for the value. An imaging apparatus comprising: a mosaic image sensor configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image being responsive to being incident on the mosaic image sensor a different, individually colored light such that the input pixel values of the pixels in the different input sub-images are interlaced in the stream; an image enhancement circuit coupled to use the connection in a second And a plurality of filter units in the array configuration to receive and filter the input pixel values in each of the input sub-images to generate a corresponding plurality of enhanced output sub-images; wherein the two-dimensional array configuration Having the columns and rows of the plurality of filter cells; wherein each of the plurality of filter cells is configured to propagate The lines of the input pixel values are to the next row of the filter elements in the two-dimensional array configuration to recombine the input pixel values using the deconvolution cores; and an image signal processing An ISP is coupled to receive and combine the plurality of output sub-images to produce a color output image. The imaging device of claim 27, wherein each of the input sub-images has an input blur, and wherein after the deconvolution core filtering is employed, each of the output images has a smaller than the input blur The output is blurred. The imaging device of claim 27, wherein the deconvolution kernels comprise filter coefficients, wherein each of the plurality of filter cells comprises a multiplier configured to multiply one of the input pixel values by the multiplier One of the filter coefficients is equal to produce one of the output of the filter unit. The imaging device of claim 29, wherein the multiplier is configured to perform two multiplications in each cycle of receiving one of the pixel clocks of the input pixels.