TWI419079B - Image enhancement in the mosaic domain - Google Patents

Description

Image enhancement in the mosaic domain

The present invention relates generally to digital imaging and, in particular, to methods and devices for enhancing image quality in digital cameras.

Limited by size, cost, aperture size, and other factors imposed by the camera manufacturer, the objective optics used in digital cameras are typically designed to minimize the optical point spread function (PSF) and maximize the modulation transfer function ( MTF). The PSF of the resulting optical system may still differ from the ideal due to focus variations and aberrations. A number of methods are known in the art for compensating for such PSF bias by digital image processing. For example, U.S. Patent No. 6,154,574 describes a method for digitally focusing an out-of-focus image in an image processing system, the disclosure of which is incorporated herein by reference. An average step response is obtained by dividing a defocused image into sub-images and calculating step responses relative to the edge directions in each sub-image. The average step response is used to calculate the PSF coefficients, which are then applied to determine an image reduction transfer function. A focused image is obtained by multiplying this function by an out-of-focus image in the frequency domain.

PCT International Publication No. WO 2004/063989 A2, which describes an electronic imaging camera comprising an image sensing array and an image processor applying a deblurring function (generally in the form of a deconvolution filter (DCF)) The signals output to the array are used to produce an output image with reduced blur, the disclosure of which is incorporated herein by reference. This blur reduction makes it possible to design and use camera optics with a poor inherent PSF while restoring the electronic image produced by the sensing array to provide an acceptable output image.

Low cost color video cameras typically use a single solid state image sensor with a multi-color mosaic filter overlay. A mosaic filter is a mask of a miniature color filter component, wherein a filter component is positioned in front of each detector component of the image sensor. For example, U.S. Patent No. 4,697,208 describes a color image pickup device having a solid-state image sensing element and a complementary color mosaic filter. Any type of image sensor with a color mosaic filter (regardless of color configuration and selection in the mosaic) is hereinafter referred to as a "mosaic image sensor."

Filter elements in a mosaic filter typically alternate between major RGB colors or between complementary colors cyan, magenta, and yellow. A common color mosaic filter type is called a "Bell Sensor" or "Bell Mosaic", which has the following general form (where the letters indicate color: R for red, G for green, and B for blue): The different color filters have individual passbands that may overlap. The Bell Mosaics are described in U.S. Patent No. 3,971,065, the disclosure of which is incorporated herein by reference.

Processing an image produced by a mosaic image sensor generally involves reconstructing all of the color image by extracting three color signals (red, green, and blue) from the sensor output. An image signal processor (ISP) processes the image sensor output to calculate luminance (Y) and chrominance (C) values for each pixel of the output image. The ISP then outputs the values (or corresponding R, G, and B color values) in a standard video format.

Embodiments of the present invention provide methods and devices for processing and enhancing electronic images, particularly images produced by a mosaic image sensor. The sensor outputs a stream of pixel values belonging to a plurality of input sub-images, each sub-image being caused by a different, individual color of light incident on the mosaic image sensor. An image restoration circuit filters the pixel values in each of the input sub-images of the input sub-images to produce a corresponding output sub-image having enhanced quality (eg, reduced blur). An image signal processor (ISP) then combines the output sub-images to produce a color video output image.

This configuration provides superior results when comparing conventional methods in which the mosaic sub-images are combined to reconstruct a color output image prior to deblurring. In addition, in some embodiments, the output sub-images generated by the image restoration circuit are in the same format as the input sub-images generated by the mosaic image sensor, so that the image restoration circuit can be The existing sensor/ISP combination is integrated between the detector and the ISP with little or no change to the sensor or ISP design.

In accordance with an embodiment of the present invention, an imaging apparatus is provided, comprising: a mosaic image sensor configured to generate an input pixel value stream belonging to a plurality of input sub-images, each sub-image response being incident on A different, individual color of light on the mosaic image sensor.

An image restoration circuit coupled to receive and digitally filter the input pixel values in each of the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images; and an image signal processor ( ISP) is coupled to receive and combine the plurality of output sub-images to produce a color video output image.

In a disclosed embodiment, the input sub-images and the output sub-images have the same format such that the ISP can be coupled to receive and combine the output sub-images or the input sub-images. Generally, the input pixel values in the plurality of input sub-images are interlaced by a predetermined interlaced pattern in a single input pixel stream output by the mosaic image sensor, and the output sub-images include Output pixel values are interleaved by the image restoration circuit in an output pixel stream in accordance with the predetermined interlaced pattern.

In some embodiments, each sub-image has an input blur, and the output sub-images have an output blur after filtering by the image restoration circuit, which is less than the input blur. In a specific embodiment, the apparatus includes an objective optical element having a point spread function (PSF) that causes the input blur, and the image restoration circuit includes a deconvolution filter (DCF) having a basis The filtering core determined by the PSF. The PSF can be changed on a plane of the image sensor, and the image restoration circuit can be configured to respond to a PSF change on a plane of the image sensor, applying a different filter core to the image sensing The input pixel values of different regions of the device. Additionally or alternatively, the image restoration circuit can be configured to apply different filter cores to the input pixel values based on one of the characteristics of the input sub-images.

In some embodiments, the mosaic image sensor includes a filter array configured with a Bell mosaic pattern. In one embodiment, the input pixel values include a first column of alternating green and blue pixels and a second column of alternating green and red pixels, and the image restoration circuit includes a green balance unit coupled to compensate One of the green pixels between the first and second columns changes in sensitivity. The sensitivity change may be non-uniform over a region of the image sensor, and the green balance unit may be configured to apply a non-uniform compensation in response to uneven sensitivity changes.

In a disclosed embodiment, the apparatus includes an objective optical component configured to focus light onto the mosaic image sensor under a predetermined blur, and the image restoration circuit includes a speckle a removing unit that identifies, within each input sub-image of the input sub-images, a defective pixel having an input pixel value that is more than a maximum difference from an input pixel value of an adjacent pixel, the maximum difference being determined according to the blur And correcting the input pixel values of the defective pixels.

In some embodiments, the image restoration circuit includes a noise filter configured to digitally filter each of the input sub-images to reduce noise within the sub-images. In a specific embodiment, the noise filter is configured to determine a direction and magnitude of a local gradient in the input sub-images, and to determine an equal direction and the magnitude selection filter core to reduce Noise.

Additionally or alternatively, the image restoration circuit includes a deconvolution filter (DCF) configured to filter the input sub-images after the noise filter reduces noise. In an embodiment, the image restoration circuit includes an edge detector configured to identify an edge region within the input image and control one of the DCF inputs such that the DCF is received at the edge The input pixel values in the region that are not filtered by noise and from the noise filter receive noise reduced input pixel values outside of the edge regions. The edge detector can be configured to detect edge pixels, and the image restoration circuit can include a widening unit configured to widen the pixel by adding pixels to edge regions around the edge pixels Equal edge area. In general, the widening unit is configured to receive, from the edge detector, one of an edge pixel in a first sub-image of the sub-images, and at least one of the sub-images The pixels adjacent to the edge pixel are added to the edge region in the two sub-images.

In a specific embodiment, the image restoration circuit includes a digital filter configured to use the core of a given size to convolve the input sub-images and the frame extension unit. The method is configured to add a pixel value boundary around the input sub-images, the boundary having a width selected by filtering the core size.

In some embodiments, the input pixel values in the plurality of input sub-images are interlaced by a predetermined interlaced pattern in a single input pixel stream output by the mosaic image sensor, and the image is interlaced. The reduction circuit includes a digital filter configured to echo the interlaced pattern, using a shadowed individual core to fold the input sub-images of the input sub-images for sub-images of the sub-images Filter separately from other sub-images.

According to an embodiment of the present invention, an imaging method is further provided, comprising: receiving, from a mosaic image sensor, an input pixel value stream belonging to a plurality of input sub-images, each sub-image response being incident on the mosaic image sensing a different, individual color light on the device; digitally filtering the input pixel values in each of the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images; and an image signal processor ( The output sub-images are combined within the ISP) to produce a color video output image.

Overview

1 is a block diagram schematically illustrating an electronic imaging camera 20, in accordance with an embodiment of the present invention. This particular, simplified camera design is shown by way of example in order to illustrate and detail the principles of the invention. However, the principles are not limited to this design, but can be applied to reduce image blur in other types of imaging systems, where a sensor produces a plurality of sub-images of different colors and then combines them to produce a enhanced color output. image.

In camera 20, objective optical element 22 focuses light from a scene onto a mosaic image sensor 24. In the camera, any suitable image sensor type can be used, such as a CCD or CMOS image sensor. In this example and in the following description, the sensor is assumed to have a Bell-type mosaic filter such that each pixel 32 within the image signal output by the sensor responds to red, green or blue light. Thus, the mosaic sensor output can be viewed as containing red, green, and blue sub-images, consisting of pixel values of corresponding sensor elements. The pixel values belonging to the different sub-images are generally interleaved in the output signal according to the order of the color elements in the mosaic filter, that is, the sensor outputs a column of RGRGRG using alternating lines... (alternating red And the green filter), then output a successor column GBGBGB... (alternating green and blue) and so on. Alternatively, the methods and circuits described below (with the necessary modifications) can be used in conjunction with other types of mosaic sensor patterns.

The pixel value stream output by the image sensor 24 is received and processed by a digital reduction circuit 26. This circuit is described in detail with reference to the drawings. The pixel values are digitized by an analog/digital converter (not shown) prior to processing by circuit 26, which may integrate sensor 24 or circuit 26 or may be a separate component . In either case, circuit 26 processes the red, green, and blue input sub-images produced by sensor 24 to reduce image blur as described below. Circuit 26 then outputs red, green, and blue sub-images with reduced blur.

In general, circuit 26 outputs the sub-images in the same format as they were used to receive sub-images from sensor 24. For example, circuitry 26 may interleave the pixel values in the output sub-images to produce a single output stream, wherein the pixel values have the same interlace as the input pixel values from sensor 24. Alternatively, circuit 26 can be configured to demultiplex and output each sub-image as a separate data block or data stream.

An ISP 28 receives the deblurred red, green, and blue output sub-images from the restoration circuit 26 and combines the sub-images to produce a color video output image (or image sequence) using a standard video format. The output image can be displayed on a video screen 30 and transmitted and/or stored in a memory via a communication link. In a particular embodiment where circuit 26 outputs the sub-images in the same format as receiving the sub-images from sensor 24, ISP 28 is used interchangeably for processing the output of circuit 26 or directly processing sensor 24 Output. Moreover, the feature of the reduction circuit 26 is advantageous because it allows the restore circuit to be used with an existing sensor and ISP without modifying the sensor or the ISP. It also allows the restore function of circuit 26 to be turned on and off simply by enabling or disabling a bypass link (not shown) between the sensor and the ISP.

The color video output image produced by ISP 28 contains brightness and color information for each pixel in the image. This information can be encoded based on brightness and chrominance (such as Y/C or other color coordinates) or on individual color values (such as RGB). In contrast, the sub-images processed and output by the restore circuit 26 are monochrome images that contain luminance information only with respect to the particular color they represent. Each sub-image contains only a subset of the pixels that will appear in the color video output image, i.e., the pixels produced by the elements of the image sensor that cover the corresponding color filter. In other words, in the example of the Bell Matrix shown in Figure 1, the R and B sub-images will each contain a quarter of the pixels in the output image, and the G sub-image will contain the remaining half.

In general, the reduction circuit 26 and the ISP 28 are embedded in one or more integrated circuit wafers, which may include custom or semi-custom components. Although the restore circuit 26 and the ISP 28 are shown as separate functional blocks in FIG. 1, the functions of the restore circuit and the ISP can be implemented using a single integrated circuit component. As desired, image sensor 24 can combine circuit 26 on the same semiconductor substrate in a system-on-a-chip (SoC) or on-camera wafer design and can also combine ISP 28. Alternatively, some or all of the functions of restore circuit 26 and ISP 28 may be implemented in software on a programmable processor (e.g., a digital signal processor). The software can be downloaded to the processor in electronic form or it can alternatively be provided on a tangible medium, such as an optical, magnetic or electronic memory medium.

2 is a block diagram schematically showing the functional components of the reduction circuit 26 in accordance with an embodiment of the present invention. In general, the functional components are embedded together in a single custom or semi-custom integrated circuit device. Alternatively, the functions shown in Figure 2 can be divided among a number of components that can be implemented in hardware or software. In the exemplary embodiment illustrated in FIG. 2, circuit 26 performs image restoration by deconvolution filtering to reduce blurring of the sub-images before ISP 28 incorporates the sub-images into a single color output image. . Other image restoration functions performed by circuit 26 on the sub-images include speckle removal and noise filtering. Alternatively or in addition, circuitry 26 may be configured to implement only one or both of the restore functions or to implement additional digital filtering functions within the space of the mosaic sub-images.

A green balance unit 40 balances the green (Gr) and green (Gb) pixel values for possible amplitude variations. Gr and Gb refer to the green pixels appearing in the RGRGRG... column and the GBGBGB... column, respectively. The details of the operation of unit 40 are described below in the section entitled "Green Balance."

The green balance image data is maintained in the input fuzzy bell (IBB) buffer 42. The bounding frame extension unit 44 adds virtual pixel columns and rows to the buffer to ensure that the pixels are properly processed at the actual image boundaries. The organization and content of buffer 42 and the operation of unit 44 are described below with reference to FIG.

A speckle removal unit 46 identifies and modifies the defective pixel values to prevent noise from propagating from the pixels into the processed image. The speckle removal operation is explained below with reference to FIG.

An edge detection unit 48 determines the position of the edge pixels within each sub-image based on the adaptive threshold 50. A widening unit 52 then applies a morphing operation to produce an output edge mask (OEM) containing the edge regions. For example, the mask may have a value of "1" at pixels within the edge regions and "0" elsewhere. Circuit 26 prevents the application of noise suppression to the edge regions to avoid loss of edge information. These edge recognition functions are explained below with reference to FIG. 5.

A false dynamic noise filter 54 is applied to reduce noise within each sub-image, thereby producing pixel values for IBB modification (IBBM). The operation of the filter 54 is explained below with reference to FIG. A selector 56 then selects the appropriate value for each pixel based on the corresponding OEM value provided by unit 52 for delivery to a deconvolution filter (DCF) 60. The selector selects the IBBM values of the non-edge pixels and the unmodified IBB initial (IBBO) pixel values taken directly from the IBB buffer 42 in the edge regions. The IBBO values are delayed by a delay line 58 to maintain proper synchronization of the IBBO and IBBM pixel streams.

A deconvolution filter (DCF) 60 individually performs a deblurring operation on each of the sub-images of the sub-images. Deconvolution filter (DCF) 60 typically uses a core that is substantially opposite to the point spread function (PSF) of optical element 22 in order to "eliminate" the aberration effects of the optical element. Such a method of calculating the deconvolution of the core is described, for example, in the above-mentioned WO 2004/063989 A2, and in the U.S. Patent Application Serial No. 11/278,255, filed on March 31, 2006, assigned to The assignee and the disclosure thereof are hereby incorporated by reference. Alternatively, the deconvolution filter (DCF) 60 may use other types of filtering cores for deblurring or for other image processing functions as is known in the art. The cores are typically masked such that each sub-image of the sub-images is filtered independently of the other sub-images, although the sub-images in the input pixel stream are interlaced. An exemplary mask pattern is shown in Figures 8A and 8B.

In many cameras, the PSF of the optical component is not uniform across the field of view of the camera. Thus, for example, in camera 20, different image sensor 24 regions may be affected by different PSF profiles. Thus, for optimal deblurring, a DCF selector 62 applies different filtering cores to pixels within different image portions. One exemplary configuration of such a filtering core is described below with reference to FIG. After being processed by the deconvolution filter (DCF) 60, the deblurred sub-image systems are then output to the ISP 28.

Green balance

This section will explain the operation of the green balance unit 40 in further detail. In theory, the sensor sensitivities of the sensor elements corresponding to the Gr and Gb type green pixels should be the same, but in practice these sensitivities may differ due to design and manufacturing tolerances. Failure to correct for sensitivity differences may result in artifacts in the fixed image in the output image. Thus, circuit 26 multiplies the Gr or Gb pixel values (or both) by a correction factor.

In order to determine the correction factor, it is assumed that the average (and therefore the sum) of the pixel values in successive columns of the green sub-images should be equal. These sums are calculated by circuit 26 and represented as sumGreenRed and sumGreenBlue, respectively. The quotient of the sum gives the correction factor f: To avoid artifacts in low light conditions (where values are close to zero), circuit 26 calculates f only when sumGreenRed and sumGreenBlue exceed a certain minimum threshold. Furthermore, the ε system is limited to a specific desired green imbalance range, for example . The values of the Gb pixels are corrected by multiplying them by f : G _{b _ output} = G _b * f (3) The correction values replace the initial Gb pixel values in subsequent calculations. Alternatively, the correction factor can be calculated and applied to the Gr pixel values or used to correct both Gb and Gr, respectively.

The sensitivity changes of the Gr and Gb pixels may be uneven over the area of the sensor 24. In this case, circuit 26 can apply a variable value f to the image area in response to a change in non-uniform sensitivity.

Boxing and buffering

After the green balance, the pixel values are inserted into an input fuzzy bell (IBB) buffer 42. Assuming that the circuit 26 is "on-the-fly" operationally processed, the pixel values are output by the sensor 24, and the deconvolution filter (DCF) 60 uses a core containing L column values (eg, L = 15), the buffer. 42 can be designed to maintain the input pixel values of the L completion column. As each new column is read into the buffer, the oldest column is discarded from the buffer. Additional virtual pixel columns and rows are added to the top and bottom of the actual image frame to avoid artifacts caused by filtering near the image boundary.

3 is a detailed schematic diagram of one portion 70 of the contents of the buffer 42 near the upper boundary of one of the images captured by the sensor 24 in accordance with an embodiment of the present invention, illustrating a method for framing an image. The column 72 in the buffer contains the actual pixel values in alternating columns GBGB... and RGRG.... The subscripts 1, 2, 3, 4, ..., indicate the number of columns, starting from column 1 at the upper edge of the input image. A bounding box boundary 76 of seven additional columns is added above the first column within the input image. These extra columns reflect the pixel values in the first two columns of the actual input frame. A comparable boundary that reflects the two boundary columns or rows is added below and to the left and right of the input image.

The bounding extension unit 44 adds a boundary 76 to ensure that the deconvolution filter (DCF) 60 does not introduce false data into the output pixel values of the ISP 28. The extent of the bounding box boundary depends on the size of the DCF core. Thus, for L = 15, seven framing columns and rows are required for this purpose, as shown in Figure 3. Alternatively, different sized cores and framing boundaries can be used. Edge detection unit 48 may find a false edge within the column of bounding boundary 76, and deconvolution filter (DCF) 60 may use the pixels within the bounding box below the previous frame to decompose the content of the bounding boundary. However, the actual pixel data in column 72 will not be affected by these operations. (If necessary, according to The cores used by edge detection unit 48 and widening unit 52 use additional framing columns and/or rows to avoid false edges within the OEM from being propagated from the framing boundaries into the frame of actual pixel data. )

The pixel values in the columns and rows of the bounding box boundary are used only within circuit 26. The pixel values are discarded after the deconvolution filter (DCF) 60 has completed its operation and are not passed to the ISP 28.

Speckle removal

In some applications of circuit 26, optical component 22 is relatively low quality and produces a blurred image on image sensor 24, as explained in the above-referenced WO 2004/063989 A2 and U.S. Patent Application Serial No. 11/278,255. . In other words, the PSF of the optical element extends over a number of adjacent pixels and the optical element acts as a low pass spatial filter. Thus, each detector element within image sensor 24 senses light collected from an image scene area that overlaps the sensing area of an adjacent detector element. Therefore, the edges and other local variations in the input image are smooth, such that it is physically impossible to make a given pixel value too far from the adjacent pixel values of the same color. The PSF of the optical component can be used to determine a threshold such that a difference between the value of one pixel greater than the threshold and the value of the adjacent pixels of the same color must be due to noise in an image sensor or Caused by a defect.

4 is a schematic illustration of one of the masks 90 used by the speckle removal unit 46 to identify and correct defective pixels in accordance with an embodiment of the present invention. For pixel p(i,j), the speckle removal unit considers adjacent pixels 92 of the same color, ie, the pixels above or below the two columns and the left or right rows of pixels, as shown. (Thus, the Gr and Gb pixels are typically considered separately, but for the purpose of speckle removal, the two green pixel types can be combined in combination. As described above, for pixels at the border of the image frame, the speckle shift The dividing unit may only consider the actual pixel values falling within the mask 90 and not the mirrored values within the boundary 76.) For each input pixel value p(i,j), the speckle removal unit determines an output as follows. Pixel values:

In other words, if the value of p(i,j) is greater than the maximum of its neighbors of the same color (as provided by mask 90) is greater than threshold ₊ , then the value is the largest by the neighbors The value is replaced by threshold ₊ . Similarly, if the value of p(i,j) is smaller than the minimum of its neighbors of the same color by more than threshold _- , the value is replaced by the minimum of the neighbors minus threshold _- . The thresholds may be based on the optical characteristics of the camera and/or the type of image (indicating whether the pixel values are highly localized), or may be tentatively set according to other criteria. Due to the operation of unit 46, the pixel systems that are much brighter or darker than other pixels adjacent thereto are all smoothed to avoid deviations from their neighbors above the allowed deviation.

Edge detection and widening

Edge detection unit 48 can use any suitable core to detect edge locations in the input image. The inventors have found that, in order to simplify the calculation, it is advantageous in the edge calculation to include only pixels of the same row or column as the current pixel in the edge detection core. Alternatively, the core may also include diagonal elements. Some example core and edge detection procedures are described below. In compromised edge sharpness and residual noise that may remain in the edge area of the OEM, the optimal core choice depends on the image content and user preferences. Alternatively, unit 48 may employ other edge detection methods known in the art.

As described above, the edge detection methods applied by unit 48 use an adaptive threshold 50, which is referred to as Edge_threshold_value in the description that follows. This threshold is typically calculated based on the light conditions of the image being processed. These conditions can be measured within circuit 26 based on the average pixel values. Alternatively, the threshold can be calculated based on the color gain parameters automatically calculated and output by the ISP 28. For example, the threshold can be calculated as a color gain weighted sum or other brightness parameter. The weighting coefficients can be selected to provide the desired tradeoff of edge sharpness to residual noise.

Mode 1 - Simple Gradient Edge Detector

The edge detector detects an edge with high resolution near the pixel p(i,j). The pixels in the following formula are indexed according to the scheme shown in Figure 4, and the gradient values are calculated as follows: d _{x -1} =| p ( i , j )- p ( i -2, j ) | d _{x +1} =| p ( i , j )- p ( i +2, j )| d _{y -1} =| p ( i , j )- p ( i , j -2)| d _{y +1} =| p ( i , j )- p ( i , j +2)| (5) when the pixel p(i, j) is on an edge, the edge value e(i, j) output to the widening unit 52 is 1, otherwise 0 such that e(i,j)=1 in the case where any of the gradient values is greater than the threshold (ie, in the case where any of the following inequalities is evaluated as TRUE).

Mode 2 - Complex Gradient Edge Detector

This edge detector is a compromise between high resolution and better step detection with minimal error detection. In this case, the gradient values are given by: d _{x -1} = p ( i -2, j )- p ( i , j ) g _{x -1} = p ( i -4, j )- p ( i -2, j ) d _{x +1} = p ( i +2, j )- p ( i , j ) g _{x +1} = p ( i +4, j )- p ( i +2, j ) d _{y -1} = p ( i , j -2) - p ( i , j ) g _{y -1} = p ( i , j -4) - p ( i , j -2) d _{y +1} = p ( i , j +2) - p ( i , j ) g _{y +1} = p ( i , j +4) - p ( i , j +2) (7) In the case where any of the following inequalities is evaluated as TRUE, the edge value e(i, j) is set Is 1:

Mode 3 - Step Edge Detector

This edge detection method detects the step edge. It uses an edge step parameter edge_step, which is typically set to a value of two. The method is as follows: Calculate delta _minus1_ x =| p ( i -3 * edge _ step , j )- p ( i - edge _ step , j )|. Calculate delta _ null _ x =| p ( i - edge _ step , j )- p ( i + edge _ step , j )|. Calculate delta _ plus 1_ x =| p ( i +3 * edge _ step , j )- p ( i + edge _ step , j )|. Calculate delta _minus1_ y =| p ( i , j -3 * edge _ step )- p ( i , j - edge _ step )|. Calculate delta _ null _ y =| p ( i , j - edge _ step )- p ( i , j + edge _ step )|. Calculate delta _ plus 1_ y =| p ( i , j +3 * edge _ step )- p ( i , j + edge _ step )|The edge value e(i, in the case where either of the following conditions is satisfied j) is set to 1: delta _ null _ y -max( delta _ min us 1_ y , delta _ plus 1_ y )> Edge _ Threshold _ Value or delta _ null _ x -max( delta _min us 1_ x , delta _ plus 1_ x )> Edge _ Threshold _ Value .

It should be noted that in this case, the value of e(i,j) does not depend on the actual strength of the pixel p(i,j).

Edge widening

Figure 5 is a schematic illustration of one of the masks 100 used in the widening unit 52 in accordance with an embodiment of the present invention. The edge detection unit 48 outputs an edge map E(x, y) containing the edge values e(i, j) determined for each pixel. The widening unit is expanded to the edge map using the mask 100 application form, which is referred to as W(x, y). The output of this widening step is the OEM: OEM(x, y) = E(x, y) ⊕ W(x, y) (9) This OEM is applied to the selector 56 as a switching input, as explained above.

The degree of widening applied to the edges depends on the mask parameter W1. In the example shown in FIG. 5, W1=3. In other words, if a given pixel 102 at the center of the mask 100 is recognized by the edge detection unit 48 as an edge position (e(i, j) = 1), the pixel and its surroundings are within the mask region. Pixel 104 receives a value of "1" within the OEM. It has been found to be advantageous to extend this edge widening on pixels of different colors (i.e., belonging to different sub-images) (unlike the color-specific other digital filtering procedures implemented in circuit 26). The value of W1 can be selected according to the application needs and user preferences according to the desired width of the edge regions to be protected from low-pass noise filtering.

Noise filter

The noise filter 54 uses a core that uses the values of neighboring pixels of the same color (e.g., pixel 92 in Figure 4) to average the current pixel values. The filter determines the output pixel values as follows: Here v(i,j) is the value of the (i,j) filter core coefficient, and p(i,j) is the input pixel value. In the following description, the central element of the filter core v(i,j) is denoted as v _c ; the peripheral elements of the filters v(i±2,j) and v(i,j±2) are denoted as v _p ; The corner elements of the filter v(i±2, j±2) are denoted as v _f .

FIG. 6 is a flow chart schematically illustrating a pseudo dynamic noise filtering method applied to a filter 54 according to an embodiment of the present invention. The purpose of this method is to avoid image resolution and contrast degradation caused by the filter operating in an image region with varying gradients in pixel values. For this purpose, at a difference calculation step 110, the filter 54 calculates the local directional difference at each pixel. The differences can be calculated as follows: Δ v =| p ( i , j -2) - p ( i , j +2)| Δ h =| p ( i -2, j )- p ( i +2, j ) △ d ₁ =| p ( i -2, j -2)- p ( i +2, j +2)| Δ d ₂ =| p ( i -2, j +2)- p ( i +2, j -2) (11) (The vertical difference term Δv should not be confused with the core coefficient v(i,j).)

At a core selection step 112, filter 54 then selects the appropriate core based on the direction difference magnitude. For this purpose, the calculated values Δv, Δh are compared to a vertical/horizontal threshold value th _vh , and the equal values _{Δd 1} , Δd ₂ are compared to a pair of angular thresholds th _d :

_‧If Δv>th _vh , then v(i,j±2) does not participate in noise removal, and the corresponding v _p value is 0.

_‧If Δh>th _vh , then v(i±2,j) does not participate in noise removal, and the corresponding v _p value is 0.

‧If Δd ₁ >th _d , then v(i+2, j+2) and v(i-2, j-2) do not participate in noise removal, and the corresponding v _f value is 0.

‧If Δd ₂ >th _d , then v(i-2,j+2) and v(i+2,j-2) do not participate in noise removal, and the corresponding v _f value is 0.

Filter 54 selects a core that satisfies the above conditions for use at each pixel. An exemplary set of nine filter cores that can be used for this purpose is in Appendix A below. It will be understood by those skilled in the art that other noise removal filter cores (dynamic or static) can be used as such.

In a filtering step 114, the filter 54 applies selected cores to the respective pixel values adjacent to each pixel to generate output pixel values for each pixel. ( i , j ).

Deconvolution filter

7 is a schematic diagram of a set 120 of a plurality of DCF cores 122 applied to pixel values input to a deconvolution filter (DCF) 60 by a selector 56 in accordance with an embodiment of the present invention. In this example, it is assumed that an input image of 1600 x 1200 pixels is divided into 192 segments of 100 x 100 pixels each. A deconvolution filter (DCF) 60 applies a different core 122 (identified as K0, K1, ..., K32) to each segment based on the local PSF at the corresponding location in the image. Because of the circuit symmetry of optical component 22, it is also assumed that the PSF variations are symmetrical on the image plane, so that only three thirty-three of these configurations need to be stored and used. Different cores. The central area 124 all uses twenty-one cores of the same set, while the peripheral area 126 uses twelve sets of another set. Alternatively, the image area can be divided into larger or smaller segments for core assignment purposes, or a single DCF core can be applied to the entire image.

Additionally or alternatively, the DCF selector 62 can instantly change the set of filtered cores based on the characteristics of the image captured by the camera 20 (and thus the input sub-image). For example, the DCF selector can select different filter cores depending on image illumination level, image content type, or other factors.

8A and 8B are diagrams showing individual layouts of core masks 130 and 140 to which a deconvolution filter (DCF) 60 is applied in accordance with an embodiment of the present invention. Each mask indicates a pixel of a different color, that is, a pixel belonging to a particular sub-image within a 15 x 15 pixel mask area. A mask 130 is applied to the green pixels, and a mask 140 is applied to the red and blue pixels. Thus, when a central pixel 132 covered by the mask 130 is green, the shaded pixel 134 indicates other green pixels adjacent 15 x 15 pixels around the central pixel. When the central pixel 132 covered by the mask 140 is red or blue, the shaded pixel 134 indicates an adjacent other red or blue pixel.

For each pixel of the input image, a deconvolution filter (DCF) 60 selects the appropriate 15x15 core for the region 120 in which the pixel is located, and then masks the core using mask 130 or 140 based on the current pixel color. In each case only the appropriately shaded shadow pixels 134 participate in the DCF operation. The coefficient of the unshaded pixel is set to zero. Therefore, only pixel values belonging to the same sub-image (i.e., pixels of the same color) are used to fold each pixel value. As described above, the pixel values to be decomposed are selected by the selector 56, which selects the delay. The IBBO values in the edge region and the noise in the non-edge regions output by line 58 reduce the IBBM value.

As previously described, the core coefficients applied by the deconvolution filter (DCF) 60 can be selected in response to the defective PSF of the optical component 22 to deblur the output image. The cores may vary depending on the current pixel location in the image plane and other factors, such as image distance. Optical components and DCF cores can be selected to provide specific image enhancement functions, such as an increased effective field depth of camera 20. Alternatively, a configuration of a deconvolution filter (DCF) 60 can be used to implement various other image enhancement functions within the mosaic color space in a manner that alternately applies different filtering operations to different color sub-images.

Therefore, it is to be understood that the above specific embodiments are cited by way of example, and the invention is not limited to what has been specifically shown and described. Rather, the scope of the invention includes both combinations and sub-combinations of the various features described above, as well as variations and modifications which are apparent to those skilled in the art and are not disclosed in the prior art.

Appendix A - Noise Removal Filter Core

The table below shows exemplary noise filter coefficient values for different edge conditions. The sum of all the coefficients in each filter is 1, so that the average pixel value does not change after filtering. In the following examples, all coefficient values for the possible cores in filter 54 are explicitly written. Alternatively, only some of the coefficients of the coefficients may be explicitly given, and the remainder is inferred from the condition that the sum is 1. For simplicity, the center coefficient v _c can be assumed to be equal to 1 minus the sum of all other coefficients for a particular core.

20. . . Electronic imaging camera

twenty two. . . Objective optical component

twenty four. . . Mosaic image sensor

26. . . Image restoration circuit / digital reduction circuit

28. . . Image Signal Processor (ISP)

30. . . Video screen

32. . . Pixel

40. . . Green balance unit

42. . . Input Fuzzy Bell (IBB) Buffer

44. . . Fixed frame extension unit

46. . . Speckle removal unit

48. . . Edge detection unit

50. . . Adaptive threshold

52. . . Widening unit

54. . . False dynamic noise filter

56. . . Selector

58. . . Delay line

60. . . Deconvolution filter (DCF)

62. . . DCF selector

70. . . Part of the contents of the buffer 42

72. . . Column

76. . . Fixed frame boundary

90. . . Mask

92. . . Pixel

100. . . Mask

102. . . Pixel

104. . . Pixel

120. . . DCF core collection / mask

122. . . DCF core

124. . . Central region

126. . . Surrounding area

130. . . Core mask

132. . . Center pixel

134. . . Shadow pixel

140. . . Core mask

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be more fully understood from the following detailed description of the embodiments of the invention, wherein FIG. 1 is a block diagram of an electronic imaging camera in accordance with an embodiment of the invention; FIG. A block diagram of a detail of an image restoration circuit is schematically illustrated in accordance with an embodiment of the present invention; and FIG. 3 is a detailed schematic diagram of one of the contents of an input buffer in accordance with an embodiment of the present invention. An image framing technique used in an image restoration circuit; FIG. 4 is a schematic diagram of a mask for speckle removal in an image restoration circuit according to an embodiment of the present invention; FIG. 5 is in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A schematic diagram of a mask for edge enhancement in an image restoration circuit; FIG. 6 is a flow chart schematically illustrating a method for noise filtering according to an embodiment of the present invention; 7 is a schematic diagram of a set of deconvolution filters according to an embodiment of the present invention, and their deconvolution filters are applied to different parts of a restored image. And Figures 8A and 8B are schematic diagrams of a de-folded core mask applied to a sub-image output by a mosaic image sensor in accordance with an embodiment of the present invention.

twenty four. . . Mosaic image sensor

26. . . Image restoration circuit / digital reduction circuit

40. . . Green balance unit

42. . . Input Fuzzy Bell (IBB) Buffer

44. . . Fixed frame extension unit

46. . . Speckle removal unit

48. . . Edge detection unit

50. . . Adaptive threshold

52. . . Widening unit

54. . . False dynamic noise filter

56. . . Selector

58. . . Delay line

60. . . Deconvolution filter (DCF)

62. . . DCF selector

Claims (30)

An imaging apparatus includes: a mosaic image sensor configured to generate a stream of input pixel values belonging to a plurality of input sub-images, wherein the plurality of input sub-images The input pixel values are interleaved in the input pixel value stream by using a predetermined interlaced pattern, and each input sub-image responds to a different, individual color light incident on the mosaic image sensor, wherein the input sub- The image has an input blur caused by a point spread function (PSF) of a set of objective optical elements; an image restoration circuit coupled to receive and digitally filter the input sub-images of the input sub-images Entering a pixel value to generate a corresponding plurality of enhanced output sub-images, the image restoration circuit comprising a deconvolution filter (DCF) having a filter core determined according to the PSF, the DCF eliminating the PSF a plurality of effects such that the output sub-images have an output blur that is less than the input blur, wherein the DCF is configured to respond to the interlaced pattern and use a Copying each input sub-image of the input sub-images to individual cores of the mask to separate the sub-images of the input sub-images from other input sub-images; and an image signal processor (ISP), It is coupled to receive and process the plurality of output sub-images to produce a color output image. The device of claim 1, wherein the input sub-images and the output sub-images have the same format so that the ISP can be coupled to receive and process the output sub-images or the input sub-images. The device of claim 2, wherein the output sub-images comprise output pixel values that are interleaved by the image restoration circuit in an output pixel stream in accordance with the predetermined interlaced pattern. The device of claim 1, wherein the PSF is changeable on a plane of the image sensor, and wherein the image restoration circuit is configured to respond to a PSF change on a plane of the image sensor, the application Different filter cores to the input pixel values from different regions of the image sensor. The device of claim 1, wherein the image restoration circuit is configured to apply different filter cores to the input pixel values according to one of the characteristics of the input sub-images. The apparatus of any one of claims 1 to 5, wherein the mosaic image sensor comprises a filter array configured with a Bell mosaic pattern. The device of claim 6, wherein the input pixel values comprise a first column of alternating green and blue pixels and a second column alternating green and red pixels, and wherein the image restoration circuit includes a green balance unit coupled to compensate One of the green pixels between the first and second columns varies in sensitivity. The device of claim 7, wherein the sensitivity change is non-uniform in an area of the image sensor, and wherein the green balance unit is configured to apply an unevenness in response to uneven sensitivity changes. make up. The apparatus of any one of claims 1 to 5, further comprising the objective optical elements configured to focus light onto the mosaic image sensor under a predetermined blur, wherein The image restoration circuit includes a speckle removal unit that recognizes the input sub-images Determining the defective pixels in the sub-images and correcting the input pixel values of the defective pixels, wherein the input pixel values of the defective pixels are different from the input pixel values of the adjacent pixels by more than a maximum difference, and the maximum difference is based on the predetermined blur To decide. The device of any one of claims 1 to 5, wherein the image restoration circuit includes a noise filter configured to digitally filter each of the input sub-images of the input sub-images to reduce the inputs Noise in the sub-image. The device of claim 10, wherein the noise filter is configured to determine a direction and magnitude of the local gradient within the input sub-images, and to select an equal direction and the magnitude to select the filter core Used to reduce noise. An imaging device includes: a mosaic image sensor configured to generate an input pixel value stream belonging to a plurality of input sub-images, each input sub-image responding to an incident incident on the mosaic image sensor Different, individual color light, wherein the input sub-images have one input blur caused by a point spread function (PSF) of a set of objective optical elements; an image restoration circuit coupled to receive and digitally filter at the Inputting the input pixel values in each input sub-image of the sub-image to generate a corresponding plurality of enhanced output sub-images, the image restoration circuit comprising a deconvolution filter (DCF) having a basis according to the PSF Determining a filter core, the DCF canceling multiple effects of the PSF such that the output sub-images have an output blur that is less than the input blur, The DCF is configured to use the filter core of a given size to convolve the input sub-images; the fixed frame extension unit is configured to add a pixel value boundary around the input sub-images, The boundary has a width selected for the filter core size; and an image signal processor (ISP) coupled to receive and process the plurality of output sub-images to produce a color output image. An imaging device includes: a mosaic image sensor configured to generate an input pixel value stream belonging to a plurality of input sub-images, each input sub-image responding to an incident incident on the mosaic image sensor Different, individual color light; an image restoration circuit coupled to receive and digitally filter the input pixel values in each of the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images, The image restoration circuit includes a noise filter configured to digitally filter each input sub-image to reduce noise in the input sub-images; wherein the image restoration circuit includes a deconvolution filter (DCF), configured to filter the input sub-images after the noise filter reduces noise, and wherein the image restoration circuit includes an edge detector configured to identify the Inputting an edge region within the sub-image and controlling one of the inputs of the DCF so that the DCF receives input pixel values that are not noise filtered in the edge regions and from the noise filter Receiving input noise is reduced by such an edge region outside of the pixel value; and an image signal processor (ISP), which system is coupled to receive and process the complex Several output sub-images are produced to produce a color output image. The device of claim 13, wherein the edge detector is configured to detect edge pixels, and wherein the image restoration circuit includes a widening unit configured to be added around the edge pixels A plurality of pixels to the edge regions to widen the edge regions. The device of claim 14, wherein the widening unit is configured to receive, from the edge detector, one of an edge pixel in a first sub-image of the input sub-images, and to input A plurality of pixels adjacent to the edge pixel in at least one second sub-image of the sub-image are added to the edge region. An imaging method includes: receiving, from a mosaic image sensor, an input pixel value stream belonging to a plurality of input sub-images, wherein the input pixel values in the plurality of input sub-images are in the input pixel value stream Interspersed with a predetermined interlaced pattern, and each input sub-image responds to a different, individual color of light incident on the mosaic image sensor, wherein the input sub-images have a point spread function by a set of objective optical elements (PSF) causing one of the input blurs; the digitally filtering the input pixel values in the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images, wherein the digital filter includes the application one having one Decoupling filter (DCF) of the filter core determined according to the PSF to the input sub-images to eliminate multiple effects of the PSF such that the output sub-images have an output blur smaller than the input blur, wherein Applying a DCF containing will be subject to The individual cores of the mask are applied to the input sub-images of the input sub-images to echo the interlaced pattern to separate the sub-images of the input sub-images from the other input sub-images; and process the output sub-images To produce a color output image. The method of claim 16, wherein the input sub-images and the output sub-images have the same format. The method of claim 17, wherein the digit filtering the input pixel values comprises outputting pixel values of the output sub-images in an output pixel stream in accordance with the predetermined interlaced pattern. The method of claim 16, wherein the PSF is changeable on a plane of the image sensor, and wherein applying the DCF comprises responding to a PSF change on a plane of the image sensor, applying a different filter core The input pixel values from different regions of the image sensor. The method of claim 16, wherein applying the DCF comprises applying different filter cores to the input pixel values based on one of the characteristics of the input sub-images. The method of any one of claims 16 to 20, wherein the mosaic image sensor comprises a filter array configured with a Bell mosaic pattern. The method of claim 21, wherein the input pixel values comprise a first column of alternating green and blue pixels and a second column alternating green and red pixels, and wherein the method further comprises processing the input pixel values to filter the digits A sensitivity change of one of the green pixels between the first and second columns is compensated prior to inputting the pixel value. The method of claim 22, wherein the sensitivity change is in the image sensing One of the regions is not uniform, and processing the input pixel values includes applying a non-uniform compensation in response to uneven sensitivity changes. The method of any one of claims 16 to 20, wherein the light system is focused on the mosaic image sensor by the objective optical elements having a predetermined blur, and wherein the method further comprises identifying at the inputs Defective pixels in each of the input sub-images of the sub-image and correcting the input pixel values of the defective pixels, the input pixel values of the defective pixels and the input pixel values of the adjacent pixels are more than a maximum difference, the maximum difference is based on This pre-defined blur is determined. The method of any one of claims 16 to 20, wherein the digitally filtering the input pixel values further comprises processing the input sub-images of the input sub-images to reduce noise within the input sub-images. The method of claim 25, wherein processing the input sub-images of the input sub-images comprises determining a direction and a magnitude of a local gradient in the input sub-images, and responding to the equal direction and the magnitudes to select filtering The core is used to reduce noise. An imaging method includes: receiving, from a mosaic image sensor, an input pixel value stream belonging to a plurality of input sub-images, each input sub-image responding to a different, individual color light incident on the mosaic image sensor The input sub-images have one input blur caused by a point spread function (PSF) of a set of objective optical elements; the digitally filters the input pixel values in the input sub-images of the input sub-images so that Generating a corresponding plurality of enhanced output sub-images, Wherein the digital filtering comprises applying a deconvolution filter (DCF) having a filter core determined according to the PSF to the input sub-images to eliminate multiple effects of the PSF such that the output sub-images have a smaller The input blurred fuzzy output, wherein applying a DCF comprises applying a DCF of one of the filter cores having a given size, and adding a pixel value boundary around the input sub-images before the digitally filtering the input sub-images The boundary has a width selected for the size of the filter core; and the output sub-images are processed to produce a color output image. An imaging method includes: receiving, from a mosaic image sensor, an input pixel value stream belonging to a plurality of input sub-images, each input sub-image responding to a different, individual color light incident on the mosaic image sensor Digitally filtering the input pixel values in each of the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images, wherein digit filtering the input pixel values includes processing the input sub-images to reduce Noise in the input sub-images and applying a deconvolution filter (DCF) to filter the input sub-images after processing the input sub-images of the input sub-images to reduce noise, wherein the application is performed The DCF includes identifying edge regions within the input sub-images and controlling one of the DCF inputs such that in the edge regions, the DCF receives input pixel values that are unprocessed to reduce noise, and Outside the edge region, the DCF receives input pixels that are processed to reduce noise reduction after noise; The output sub-images are processed in an image signal processor (ISP) to produce a color output image. The method of claim 28, wherein identifying the edge regions comprises detecting edge pixels and widening the edge regions by adding a plurality of pixels around the edge pixels to the edge regions. The method of claim 29, wherein widening the edge regions comprises detecting an edge pixel in a first sub-image of the input sub-images and in at least a second sub-image of the input sub-images A plurality of pixels adjacent to the edge pixel are added to the edge region.