TW200821989A - Image enhancement in the mosaic domain - Google Patents

Abstract

Imaging apparatus includes a mosaic image sensor (24), which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor. An image restoration circuit (26) is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images. An image signal processor (ISP) (28) is coupled to receive and combine the plurality of the output sub-images in order to generate a color video output image.

Description

200821989 IX. INSTRUCTIONS: FIELD OF THE INVENTION The present invention relates generally to digital imaging and, more specifically, to methods and devices for enhancing image quality in digital cameras. [Prior Art] Due to size, cost, aperture size, and other factors imposed by the camera manufacturer, the objective optical components used in digital cameras are generally designed to minimize optical point spread function (PSF) and maximize tuning. Change the f transfer function (MTF). The PSF of the resulting optical system may still differ from the ideal due to the poly dance = and aberrations. A number of methods are known in the art to compensate for such pSF bias by digital image processing. For example, U.S. Patent No. 6' 154' 574 describes a method for digitally dancing in a video processing system - a method of defocusing images, the disclosure of which is incorporated herein by reference. An average step response is obtained by dividing a defocused image into sub-images and calculating the step response in each sub-image relative to the edge direction. The average step is used to calculate the PSF coefficient, which is then applied to the decision-image reduction transfer function. A focused image is obtained by multiplying this function by an out-of-focus image in the frequency domain (~~ domain). Ppt International Bulletin w〇2〇〇4/〇63989 A2, which illustrates an electronic imaging camera that includes an image sensing array and an image processor that should be defuzzified (usually using a deconvolution filter) The form of the device (dcf) to the output of the array to produce an output with reduced blurring, such as the disclosure of which is incorporated herein by reference. This blur reduction enables the design and use of camera optics having a poor inherent PSF to restore the electronic image produced by the sensing array to provide an acceptable output image. Low cost color video cameras typically use a single-solid image sensor with a multi-color mosaic filter overlay. The mosaic filter is a mask of a micro color filter component, wherein a filter component is positioned in front of each detector component of the image sensor. For example, U.S. Patent No. 4,697,208 describes a color image pickup device having a solid-state image sensing element and a complementary color mosaic filter. Any type of image sensor with a color mosaic waver (regardless of the color configuration and selection in the mosaic) is hereinafter referred to as a "mosaic image sensor". The chopper element in a mosaic chopper is generally Between main RGB colors or between complementary colors cyan, magenta, and yellow. A common color mosaic filter type is called a "Bell Sensor" or "Belle Mosaic M, which has the following general form ( The letters indicate color: R for red, G for green, and B for blue

R G R G R G G B G B G B R G R G R G G B G B G B R G R G R G G G G G G G These different color filters have individual pass bands that may overlap. The Belmars is described in U.S. Patent No. 3,971, G, the disclosure of which is incorporated herein by reference. Processing - The image produced by the mosaic image sensor generally involves processing three color signals (red, all color images - image signal processor (10)) from the sensing 1 § output for processing The pixels are used to calculate the luminance (γ) and chrominance (c) values. The ISP then outputs the values (or corresponding R, G, and B color values) in a standard video format. SUMMARY OF THE INVENTION The present invention provides a method and apparatus for processing and enhancing an electronic image (especially an image produced by a video sensor). The sensor outputs a stream of pixel values belonging to a plurality of input sub-images, each sub-image being caused by a different, individual color of light incident on the mosaic image sensor. - Image restoration circuit (4) The pixel values in the input sub-images of the input sub-images to produce corresponding output sub-images having enhanced quality (e.g., reduced blur). An image signal processor (ISP) then combines the output sub-images to produce a color video output image. This configuration provides superior results when comparing the conventional methods in which the mosaic sub-images were combined to reconstruct the color output image prior to deblurring. In addition, in some embodiments, the output sub-images generated by the image restoration circuit are identical to the input sub-images generated by the mosaic image sensor, and the image is restored. The circuit can integrate a sensor/ISP combination between the sensing $ and the ISP with little or no change to the sensor or 18? design. 119049.doc 200821989 In accordance with an embodiment of the present invention, there is thus provided an imaging apparatus comprising: ', a mosaic image sensor configured to generate an input pixel value stream belonging to a plurality of rounded sub-images Each sub-image responds to a different, individual color of light incident on the mosaic image sensor. An image restoration circuit coupled to receive and digitally filter the input pixel values in each of the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images; and an image signal processor ( ISP) is coupled to receive and combine the plurality of output sub-images to produce a color video output image. In the specific implementation of the disclosure, the input sub-image and the output sub-images have the same pattern so that the ISP can be coupled to receive and combine the output sub-images or the input sub-images. Generally, the input pixel values of the plurality of input sub-image towels are in a single input pixel stream output by the mosaic image sensor, and a predetermined interlaced pattern is used, and the output is The image includes output pixel values that are interleaved by the image restoration circuit in accordance with the predetermined interlaced pattern in an output pixel stream. In some embodiments, each sub-image has an input blur, and the output sub-images have an output blur after the image reduction circuit material, which is less than the input blur. In a specific embodiment, the apparatus includes an objective optical element having a point spread function (SF) that causes the input blur, and an I# like restore circuit includes a _dissolved wave generator (dcf), which has: 3PSF decided to wave the core '. The chat can be intertwined on the plane of the image and the J device, and the image restoration circuit can be configured to return 119049.doc 200821989 _ PSF change on the plane of the image sensor, applying different filter cores The input pixel values from different regions of the image sensor. Additionally or alternatively, the image restoration circuit can be configured to apply different filter cores to the input pixel values based on one of the characteristics of the input sub-images. In some embodiments, the mosaic image sensor includes a filter array configured with a Bell mosaic pattern. In one embodiment, the input pixel values include a first column of alternating green and blue pixels and a second column of alternating green and red pixels, and the image restoration circuit includes a green balance unit coupled to compensate One of the green pixels between the first and second columns changes in sensitivity. The sensitivity change may be uneven over an area of the image sensor, and the green balance unit may be configured to apply a non-uniform compensation in response to non-uniform sensitivity changes. In a disclosed embodiment, the apparatus includes an objective optical component configured to focus light onto the mosaic image sensor under a predetermined blur, and the image restoration circuit includes a speckle a (Spike) removal unit that identifies, within each input sub-image of the input sub-images, a defective pixel having a larger input pixel value than an input pixel value of the adjacent pixel, and a maximum difference is based on the blur To determine and correct the input pixel values of the defective pixels. In some embodiments, the image restoration circuit includes a noise filter configured to digitally filter each of the input sub-images to reduce noise within the sub-views. In a specific embodiment, the noise filter is configured to determine a direction and amount of a local gradient in the input sub-images, and to return the equal value and the magnitude of the value 119049.doc -10- 200821989 The filter core is selected to reduce noise. Additionally or alternatively, the image restoration circuit includes a deconvolution filter state (DCF)' configured to pass 4 input sub-images after the noise filter reduces noise. In a specific embodiment, the image restoration circuit includes an edge detector configured to identify an edge region within the input image, and (4) one of the (10) inputs, such that the (10) reception is at the The input pixel values in the edge region $ without noise filtering and receiving noise reduced input pixel values outside the edge regions from the noise filter. The edge detector can be configured to detect edge pixels, and the image restoration circuit can include a widening unit configured to widen the pixel by adding pixels to edge regions around the edge pixels Equal edge area. In general, the widening unit is configured to receive, from the edge detector, one of the edge pixels in the first sub-image of the sub-images, and at least one of the sub-images The pixels adjacent to the edge pixel are added to the edge region in the two sub-images. In a disclosed embodiment, the image restoration circuit includes a digital filter, the filter is configured to use the core of a given size to fold the input sub-images, and a certain frame extension unit. The system is configured to add a pixel value boundary around the input sub-images, the boundary having a width selected for filtering the core size. In some embodiments, the input pixel values in the plurality of input sub-images are interlaced by a predetermined interlaced pattern in a single input pixel stream output by the mosaic image sensor, and the image is interlaced. 119049.doc -11 - 200821989 The road includes a digital c-wave a, which is arranged to echo the interlaced pattern, using the individual cores of the shadow to fold the sub-images of the input sub-images (10) Separating the respective sub-materials of the material from the other sub-images. According to a specific embodiment of the present invention, an imaging method is further provided, comprising: ϋ receiving, from a mosaic image sensing device, a plurality of input sub-images - input pixel value streams, each sub-image responding person shooting Mosaic (4) a different, individual color of light on the sensor; in each of the input sub-images, the input sub-image (four) digits (four), the input pixel values are more corresponding to the plurality of enhanced sub-images; And combining the output sub-images in an image signal processing H (ISP) to generate a color video output image. [Embodiment]

A specific embodiment of G Kangben & Month, Fig. 1 is a block diagram schematically illustrating an electronic imaging camera. In this context, this particular, simplified camera set is shown by way of example in order to clarify and embody the principles of the present invention, and the principles are not limited to this design, but can be applied to other types of imaging systems. The small image is blurred, wherein the sensor generates a plurality of sub-images of different colors and then combines them to produce a enhanced color output image. In camera 2G, objective optical element 22 concentrates the light from the scene onto a mosaic image sensor 24. In this camera, any suitable image sensor type, such as a CCD or CMOS image sensor, can be used as 119049.doc -12 200821989. In this example and in the following description, the sensor assumes a Bell-type mosaic filter such that each pixel 32 within the image signal output by the sensor responds to red, green or blue light. Thus, the mosaic sensor output can be viewed as containing red, green, and blue sub-images, consisting of the pixel values of the corresponding sensor. The pixel values belonging to the different sub-images are generally interleaved in the output signal according to the order of the color elements in the mosaic filter, that is, the sensor outputs a column of RGRGRG·. And green filter), then output - subsequent columns GBGBGB · (. alternating green and blue) and so on. Alternatively, the methods and circuits described below (with the necessary modifications) can be used with other types of mosaic sensor patterns. The pixel value stream output by the image sensor 24 is received and processed by a digital restore circuit %. This circuit is described in detail with reference to the drawings. The pixel values are digitized by an analog/digital converter (not shown) prior to processing by circuit 26, which may integrate sensor or circuit 26 or may be a separate component . In either case, circuit % processes the red, green, and blue input sub-images produced by sense 24 to reduce image blur, as described below. Circuit 26 then outputs red, green, and blue sub-images with reduced blur. Typically, circuit 26 outputs the sub-images in the same manner as it did when receiving sub-images from sensor 24. For example, circuitry 26 may interleave the pixel values in the sub-images to produce a single output stream having the same number of inputs as the input pixel values from sensor 24 119049.doc. 13- 200821989 Wrong. Alternatively, circuit 26 can be configured to demultiplex and output each sub-image as a separate data block or data stream.

The UP 28 receives the deblurred red, green, and color wheel sub-images from the restoration circuit 26 and combines the sub-images to produce a color video output image (or image sequence) using a standard video format. The output image can be displayed on a video screen 30 and transmitted and/or stored in a body via a communication link. In a specific embodiment in which circuit 26 outputs the sub-images in the same format as receiving the sub-images from sensor 24, ISP 28 is used interchangeably for processing the output of circuit % or directly processing sensor 24 Output. In addition, the feature of this reduction circuit is advantageous because it allows: the restoration circuit cooperates with an existing sensor and the stage without modifying the sense to or 4 ISP, which also allows only by the sensor and the The ISP's $ or & bypass link (not shown) is used to turn the circuit on and off. The color video image generated by ISP Gu 28 contains image 2 redundancy and color information for the image. This information can be encoded based on brightness and chrominance (such as Y/C U color coordinates) or on individual color values (such as deletion). The sub-images processed and outputted compared to == 2 are monochromatic. Only the inclusion of ::: = fixed color contains brightness information. Each sub-image 隹... The color video outputs the pixels of the pixels in the image, and the G sub-images will contain the remaining ::: each containing a quarter of the image 119049.doc -14- 200821989 In general, The reduction circuit 26 and the ISP 28 are embedded in one or more integrated circuit wafers, which may include custom or semi-custom components. Although the reduction circuits 26 and 181 > 28 are shown as separate functional blocks in Figure 1, the functions of the reduction circuit and the ISP can be implemented using a single integrated circuit component. Optionally, image sensor 24 can combine circuit 26 on the same semiconductor substrate in a system-on-chip (S〇C) or on-camera wafer design and can also combine up 28. Alternatively, some or all of the functions of restore circuit 26 and isp 28 may be implemented in software on a programmable processor (e.g., a digital signal processor). The software can be downloaded to the processor in electronic form or alternatively provided on a tangible medium, such as an optical, magnetic or electronic memory medium. 2 is a block diagram schematically showing the components of the reduction circuit 26 in accordance with an embodiment of the present invention. In general, the functional components are embedded together in a single custom or semi-custom integrated circuit device. Alternatively, the functions described in Figure 2 may be divided among a number of components that may be implemented in hardware or software. In the exemplary embodiment shown in FIG. 2, circuit 26 performs image restoration by deconvolution filtering to reduce the sub-images before iPS 28 merges the sub-images into a single-color image. blurry. Other image restoration functions performed by circuit 26 on the sub-images include speckle removal and noise filtering. Alternatively or in addition, circuitry 26 may be configured to perform only one or both of the restore functions or to implement additional digital over-the-counter functions within the space of the mosaic sub-images. A green balance unit 40 balances the green (Gr) and green (Gb) pixel values for possible amplitude variations. (^ and Gb are respectively referred to in RGRGRG•• column and 119049.doc -15- 200821989

GBGBGB··· Columns Ψ Phase Μ M A Green pixels appear. The details of the operation of unit 40 are described below in the section entitled, Green Balance,.彔 # # 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 像 输入 输入 输入 ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib ib . The operation of the buffers, groups, ports, and valleys and units 44 is described below with reference to the drawings. A speckle removal unit 46 identifies and modifies the defective pixel values to prevent the pixels from propagating into the processed image. The speckle removal operation will be described below with reference to FIG. An edge detection unit 48 determines the position of the edge pixels within each sub-image based on the adaptive threshold value 〇. _ widening the single ^ 52 and then applying the _ morph operation to produce an output edge mask (1) 包含 containing the edge regions. For example, the mask may have a value of "1" at the pixels within the edge regions and have elsewhere. Circuit 26 prevents the application of noise suppression to the edge regions to avoid loss of edge information. These edge recognition functions are explained below with reference to FIG. A false dynamic noise filter 54 is applied to reduce the noise within each sub-image, thereby producing a modified (ΙΒΒΜ) pixel value. The operation of the filter 54 will be described below with reference to FIG. A selector 56 then selects the appropriate value for each pixel based on the corresponding OEM value provided by unit 52 for transmission to a deconvolution filter 60. The selector selects the IBBM values of the non-edge pixels and the unmodified IBB initial (IBBO) pixel values taken directly from the IBB buffer 42 in the edge regions. The IBBO values are delayed by a delay line 58 to maintain proper synchronization of the IBBO and IBBM pixel streams. 119049.doc -16- 200821989 Deconvolution Filter (DCF) 60 individually performs a deblurring operation on each sub-image of the sub-images. The filter 6 〇 generally uses a core that is substantially opposite to the point spread function (PSF) of the optical element U in order to "eliminate" the aberration effect of the optical element. Such a computational deconvolution core method is described (for example) The above-mentioned WO 2004/063989 A2 and the U.S. Patent Application Serial No. 11/278,255, filed on Mar. Alternatively, filter 60 may use other types of filtering cores for deblurring or other image processing functions as are known in the art. The cores are typically masked such that each of the sub-images The sub-images are filtered independently of the other sub-images, although the sub-images in the input pixel stream are interlaced. The exemplary mask patterns are shown in Figures 8A and 8B. In many cameras, the PSF of the optical components It is uneven over the entire field of view of the camera. Thus, for example, in camera 20, different image sensor 24 regions may be affected by different pSF profiles. Thus, for optimal deblurring 'a DCF selection 62 Applying different filter cores to pixels within different image portions. An exemplary configuration of such a filter core is described below with reference to Figure 7. After being processed by filter 6〇, the deblurred sub-image systems are then output to ISP 28 Green Balance This section will explain the operation of the green balance unit 4 in further detail. Theoretically, the light sensitivity of the sensor elements corresponding to the Gr and Gb type green pixels should be the same, but in fact the sensitivity may be Due to design and manufacturing tolerances, the inability to correct for sensitivity differences may cause artifacts in the output image to be fixed. Therefore, circuit 26 will have these Gr or Gb pixel values (or two). Multiply by a correction factor. To determine the correction factor, it is assumed that the average (and therefore the sum) of the pixel values in consecutive columns of green sub-images should be equal. The sums are calculated by circuit 26, respectively. And expressed as sumGreenRed and sumGreenBlue. The quotient of the sum gives the correction factor f:

sumGreenKcd Λ -= 1 + ε sumGreenBlue

sumGreenKed - sumGreenBlue ε-- (2) sumGreenBlue In order to avoid artifacts in low light conditions (where values are close to zero), circuit 26 only calculates / when sumGreenRed and sumGreenBlue exceed a certain minimum threshold. Furthermore, the ε system is limited to a particular desired green imbalance range, such as -. The values of these Gb pixels are corrected by multiplying them by /:

Gb_output = Gb^f (3) These correction values replace these initial Gb pixel values in subsequent calculations. Alternatively, the correction factor can be calculated and applied to the GH pixel values instead or to both Gb and Gr, respectively. The sensitivity changes of the Gr and Gb pixels may be uneven over the area of the sensor 24. In this case, circuit 26 can apply a variable value f to the image area in response to a change in non-uniform sensitivity. Boxing and Buffering After the green balance, the pixel values are inserted into an input fuzzy bell 119049.doc -18 - 200821989 (IBB) buffer 42. Assuming circuit 26 "instant, operational processing, then the pixel values are output by sensor 24, and filter 6 〇 uses a core containing L column values (eg, L = 15), buffer 42 can be designed The input pixel value used to maintain the completion column. As each new column is read into the buffer, the oldest column is discarded from the buffer. Additional virtual pixel columns and rows are added to the top and bottom of the actual image frame to avoid artifacts caused by filtering near the image boundary. 3 illustrates a method for framing an image in a portion of a portion 70 of the buffer 42 near the upper boundary of the captured image, in accordance with an embodiment of the present invention. Column 72 in the buffer contains the actual pixel values in alternating columns GBGB... and RGRG.... The subscripts 1, 2, 3, 4, ... indicate the number of columns starting from column i at the upper edge of the input image. The bounding box boundaries of the seven additional columns are added above the first column in the input image. These extra columns reflect the pixel values in the first two columns of the actual input frame. & The corresponding boundary of the two boundary columns or rows is added below and below the input image. The frame extension unit 44 adds a boundary 76 to ensure that the DCF 60 does not introduce false data into the output pixel values communicated to the ISP 28. The extent of the framing boundary depends on the size of the DCF core for this purpose. Thus, for the five seven frame columns and rows, as shown in Figure 3. Alternatively, different sizes of core and framing boundaries can be used. The edge detection unit 48 can find the false edge in the column of the frame boundary %, and the filter 6 can use the pixels in the frame boundary below the previous frame to fold the content of the frame boundary, but in column 72 The actual pixel material inside will not be affected by these operations. (If necessary, additional framed columns and/or rows may be used according to the core used by edge detection unit 119049.doc -19-200821989 48 and widening unit 52, in order to avoid false edges within the OEM from such The frame boundaries are propagated into the frame of the actual pixel data.) The pixel values in the columns and rows of the bounding box boundary are used only within circuit 26. The pixel values are discarded after the filter 60 has completed its operation and are not passed to the ISP 28. Speckle Removal In some applications of circuit 26, optical component 22 is relatively low quality and produces a blurred image on image sensor 24, such as w〇2004/063989 A2 and U.S. Patent Application Serial No. 11 /278,255 explained. In other words, the PSF of the optical element extends over a number of adjacent pixels, and the optical 70 is used as a low pass spatial filter, waver. Thus, each of the detector elements of the image sensor paste senses light collected from an image scene region that overlaps the sensing regions of adjacent detector elements. Therefore, the edges and other local variations in the input image are smooth, so that it is physically impossible to make the given pixel value too far from the adjacent pixel values of the same color. The PSF of the optical component can be used for the mosquito-threshold value such that the difference between the value of the -pixel value greater than the threshold value and the value of the adjacent pixels of the same color must be due to an image sensor ten noise or Caused by a defect. Figure 4 is a schematic illustration of a mask used by the speckle removal unit 46 to identify and correct defective pixels in accordance with an embodiment of the present invention. For the pixel P(i,j), the speckle removal unit considers the adjacent pixel % of the same color, that is, the pixels above or below the current pixel and the left or right rows of pixels, as shown in the figure. (Thus 'typically separate considerations ~ and Gb pixels, but for the purpose of speckle 119049.doc • 20-200821989 removal purpose' can be combined to combine the two green pixel types. As described above, for at the border of the image frame The pixels, the speckle removal unit may only consider the actual pixel values falling within the mask 90 and do not consider the mirror values within the boundary 76.) For each input pixel value p(ij), the speckle removal The unit determines the round pixel value as follows: P(hj) = ^ ^ \p{U JX [max(p(i + Δ/5 j + Δ/*))δ/ ^_2 〇2; [maxjp(/5 [min(^(/ + Δ/? j + Λ-threshold^ | a threshold-\

Equation (4) In other words, if the value of p(i,j) is greater than the maximum of its neighbors of the same color (as provided by mask 90), the value is mostly greater than -+, and the value of "Bei" is used by The maximum of these neighbors is replaced by thresh〇ld+. Similarly, if the value of p(i,j) is smaller than the minimum of the neighbors of the same color by more than thRshoicL, then the value is replaced by the minimum of the neighbors minus the threshold. The thresholds may be based on the optical characteristics of the camera and/or the type of image (indicating whether a pixel value is highly localized) or may be tentatively set according to other criteria. Due to the operation of unit 46, the pixels that are much brighter or darker than the other pixels adjacent thereto are all smooth, and the deviation from their neighbors is higher than the allowed deviation. Edge Detection and Widening Edge Detection Unit 48 can use any suitable core to detect edge locations in the input image. The inventors have found that, in order to simplify the calculation, it is advantageous that the edge-calculated towel core includes only pixels that are adjacent to or in the column of the current pixel. Alternatively, the core may also include diagonal elements. The following describes some example heart and edge _ procedures. In compromise margins and 119049.doc -21 - 200821989 residual gamma that may remain in the edge area of the OEM, the optimal core choice depends on the image content and user preferences. Alternatively, unit 48 may employ other edge detection methods known in the art.

As described above, the edge detection methods applied by unit 48 use an adaptive threshold 50, which is referred to as Edge_thres^id_value in the description that follows. This threshold is typically calculated based on the light conditions of the image being processed. These conditions can be measured within circuit 26 based on the average pixel values. The drag value can be calculated based on the color gain parameter automatically calculated and output by the ISP 28. For example, the threshold can be calculated as a color increase or other exemption parameter. These weighting coefficients can be selected to provide the desired tradeoff of edge sharpness to residual noise. & Mode 1 - Simple Gradient Edge Detector This edge (four) is a high resolution near the pixel P ( i, j ). The pixels in the following formula are indexed according to the scheme shown in Fig. 4 and the specific gradient values are calculated as follows: , , (5) ^xi = \p(^ J) ~ Pi} - 2, 71)| dx+x=\p(}J)-p{i^2j)\ dy^=\p{Uj)-p{Uj-2)\ dy+l=\p(hj)-p(hj ^2)\ 119049.doc -22- 1 A 4 paste mountain king knife edge sees the edge value of early morning 52 v (i, j) is 1, otherwise 0, so that any of the gradient values is large, critical In the case of a value (that is, in the case where any of the following inequalities is evaluated as =) e(i, j) = l. (6) 200821989 dx^x > Edge _ threshold _ value dx+l > Edge _ threshold _ value dy_x > Edge _ threshold _ value dy+l > Edge _ threshold _ value Mode 2 - Complex gradient edge detection This edge detector is a compromise between high resolution and detection with minimal error detection. In this case, the gradient values are given by: dX~\ = SXA z = ice-4, force-2, y) h = p{i + 2J)-p{iJ) Sx+l = d^i = p(hj-2)-p(Uj) g less-1 = = P(UH, -P(Uj - 2) dy+l = p(hj + 2)-p(i,j) Sy +l = =p(Uj + ^)-p(Uj + 2) (7) In the case where any of the following inequalities is evaluated as TRUE, the edge value e(L j) is set to 1:

Dddd ίν /IV /V 1 1 Less 1 Less +1 > Edge — threshold — value) > Edge _ threshold _ value) > Edge a threshold value and > Edge _ threshold ___ value) Kl > S x-\ (^x+l > S x+\ (七一1 > gy-l (^y+\ > Sy+\ or or dx-\'gx-l <〇dx+ l,gx+l /ox(8) dy^rSy+i <〇3-step edge detector This edge detection method detects step edge mode 3-step edge detector This edge detection The method detects the step edge, which uses an edge step parameter edge-step, which is generally set to a value of 2. The method proceeds as follows: • Calculate/to a min w*sl — λ: = |p(,· A 3 * edge pair of e;?, force - - edge - print, y)| • Calculate delta-null_x = \p(i-edge"tep,j)-p(i + edge_step,jj •calculate Do /to _ ;?/(10) 1 λ: = |p(z· + 3 * Mge - you p, force - + edge one he;?, • count the nose plus - min wsl one;; = |/7 (z',j· - 3 * Mge a pair of ep) - ;?(,·,)- —对叩】 • ef delta_null _y = \p(i9 j - edge_step)- p{i, j + edge_step)^ • ^ Delta_plus\_y = |/?(/?y + 3* edge _step)- p(i, j H-edge_stepi 119049.doc -23- 200821989 In the case where either of the following conditions is satisfied, the edge value e( i, D is set to 1: system such as a secret one 3; -max (·^minWtyl -% secret a-p / Wtyl - less) > £ coffee - hemp or a secret one X-dirty (_a-minw ( x, ^a - coffee > five coffee - · - heart. It should be noted that in this case, the value of e (i, j) does not depend on the actual strength of the pixel p (i, j). A schematic diagram of one of the masks 100 used by the widening unit 52 in accordance with an embodiment of the present invention. The edge detection unit 48 outputs an edge map E(x, y) containing the edges determined for each pixel. The value e(iJ). The widening unit is expanded to the edge map using a mask 100 application form, which is referred to as w(x, y). The output of this widening step is the OEM: 0EM(x, y) = E(x, y) ten W(x, y) (9) This OEM is applied to the selector 56 as a switching input, as explained above. The extent to which the edges are applied to these edges depends on the mask parameter boundary. In the example shown in Figure 5, W1 = 3. In other words, if a given pixel 102 at the center of the mask 1 is recognized by the edge detecting unit 48 as an edge position (e(i, j) = 1) ', the pixel is in the mask region. And the surrounding pixel ι〇4 receives the value "丨" within the OEM. It has been found to be advantageous to extend this edge broadening on pixels of different colors (i.e., belonging to different sub-images) (unlike the color-specific other digital filtering procedures implemented in circuit 26). The value of W1 can be selected according to the application needs and user preferences according to the desired width of the edge regions to be protected from low-pass noise filtering. Noise Filter The noise filter 54 uses a core that uses the values of neighboring pixels of the same color (e.g., pixel 92 in Figure 4) to average the current pixel value. The chopper determines the output pixel value as follows:

α=-2,0,2έ=—2,〇, 2 V where v(i,j) is the value of the (iJ) filter core coefficient, and p(iJ) is the input pixel f, value. In the following description, the central component of the filter core V(i,j) is expressed as

Vc; the peripheral elements of the waver v(i±2,j) and v(i,j±2) are denoted by 〜; and the corner elements of the filter v(i±2, j±2) are denoted by Vf. FIG. 6 is a flow chart schematically illustrating a method of filtering a false dynamic noise applied to a waver 54 in accordance with an embodiment of the present invention. The purpose of this method is to avoid image resolution and contrast degradation caused by the filter operating in the image region with varying gradients in pixel values. For this purpose, at a difference calculation step 110, the filter 54 calculates the local directional difference at each I pixel. These differences can be calculated as follows: Δν = \p(h 7-2)- p(i, j + 2)| △ do =|ρ〇· - 2, force - ρ〇· + 2, force | = \p{i - 25 y - 2) - p(i + 25; + 2)| ( 11) △d2 = |p(z· - 2, j· + 2) - ;?〇· + 2,) — 2)| (The vertical difference term Δν should not be confused with the core coefficient v(i,j).) At a core selection step 112, the filter 54 then selects the appropriate core based on the direction difference magnitude. For this purpose, the calculated values Δν, the heart is compared to a vertical/horizontal threshold value thvh, and the equivalent is compared to a 119049.doc -25-200821989 diagonal threshold thd: • if Δν> ί1ιν1ι, then v (i, j±2) does not participate in noise removal, and the corresponding value is 0 〇 • If Ah> tvh, v(i±2, j) does not participate in noise removal, and the corresponding Vp value is 0 〇 • If Δdpthd ' then v(i+2, j+2) and v(i-2, j_2) do not participate in noise removal, and the corresponding vf value is 〇. • If Ad2 >thd ' then v(i_2, j+2) and v(i+2, j-2) do not participate in noise removal, (and the corresponding Vf value is 0. The waver 54 selects the above conditions. A core is used at each pixel. An exemplary set of nine filter cores that can be used for this purpose is listed in Appendix A below. It will be understood by those skilled in the art that other noise removal can be used similarly. Filtering core (dynamic or static). In a filtering step 114, the filter 54 applies selected cores to the respective pixel values adjacent to each pixel to generate output pixel values for each pixel. (^ Deconvolution filtering Figure 7 is based on this One embodiment of the invention applies to a schematic diagram of a set 120 of a plurality of DCF cores 122 input by a selector brother to a pixel value of DCF 60. In this example, an input image of 16 〇〇 xl 2 〇〇 pixels is assumed It is divided into 192 segments of 100XI00 pixels. The DCF 60 applies a different core 122 (identified as K0, Κ1, . . . , Κ3 2) to each segment according to the local PSF at the corresponding position in the image. Because the optical component 22 Circuit symmetry, also assuming that the PSF changes are in the image Symmetrical in the plane, this configuration requires storage and use of only thirty-three different cores. The central area 丨24 all 119049.doc -26 - 200821989 eleven cores, while the peripheral area 126 uses another. Or, can be For the purpose of core assignment, the image area may be applied, or a single dc:f core may be applied to the whole or replace the ground. The DCF selector 62 may be based on the characteristics of the image captured by the camera 20 (and thus the input sub-image). Instantly change the filter core of the set. For example, the DCF selector can select different filter cores depending on image illumination level, image content type or other factors.

The twelve core fields of the second set using the same set are divided into larger or smaller images. Fig. 8 is a schematic diagram showing the individual layouts of the cores ~ masks 〇3〇 and 丨4G applied in accordance with the present invention, dcf 6〇. Each mask indicates - a pixel of an individual color, that is, a pixel belonging to a particular sub-image within a 15x15 pixel mask area. The mask 13G is suitable for the green pixels, and the mask 140 is suitable for red and blue pixels such as § Hai. Thus, when a central pixel 132 covered by the mask is green, the shaded pixel 134 indicates other green pixels adjacent 15 pixels of 15 pixels around the central pixel. When the central pixel 132 covered by the mask 14 is red or blue, the shaded pixel 丨 34 indicates other red or blue pixels that are adjacent. For each pixel of the input image, DCF 60 selects the appropriate 15 XI 5 core for region 120 in which the pixel is located, and then masks the core using mask 130 or 140 based on the current pixel color. In each case only the appropriately shaded shadow pixels 134 participate in the DCF operation. The coefficient of the unshaded pixel is set to zero. Therefore, only pixel values belonging to the same sub-image (i.e., pixels of the same color) are used to fold each pixel value. As described above, the pixel values to be decomposed are selected by the selector 56. The selector 56 selects the 119049.doc -27-200821989 output from the delay line 58 and the IBB0 values in the edge region. The noise in the edge region reduces the IBBM value. As previously described, the core coefficients applied by the DCF 60 can be selected in response to the defective psF of the optical component 22 to effect the image. These cores may vary depending on the current pixel location in the image plane and other factors, such as image distance. Optical components and DCF cores can be selected to provide specific image enhancement functions, such as an increased effective field depth of camera 20. Alternatively, the DCF 60 configuration can be used to implement various other image enhancement functions in the mosaic color space in a manner that alternately applies different filtering operations to different color sub-images. Therefore, it is to be understood that the above specific embodiments are cited by way of example, and the invention is not limited to what has been specifically shown and described. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described above, as well as variations and modifications which have been made by those skilled in the art and which are not described in the prior art. Appendix A_ Noise Removal Filter Core The following table shows sample noise filter coefficient values for different edge conditions. The sum of the coefficients in each filter is 丨 so that the average pixel value does not change after filtering. In the example below, all coefficient values for the possible cores in filter 54 are explicitly written. Alternatively, only some of the coefficients of the coefficients can be explicitly given, and the rest are inferred from the conditions of the sum. For simplicity, it can be assumed that the center coefficient vc is equal to 丨 minus the sum of all other coefficients for a particular core. 119049.doc -28 - 200821989 Center filter value Vc non-zero peripheral vp filter value non-zero corner vf filter value condition 1 0.375 0.125 0.03125 ΔvSthvh and Ah^thVh and Adi^tha and Ad2<tha 2 0.375 0.1875 0.0625 [ (Av>thVh and AhSthVh) or (AvSthvh and Δh>thVh)] and AdiShd and Ad〗 幺thd 3 0.625 0.09375 (Av>thVh and Ah>thVh) and AdiSthd and Ad2<thd 4 0.375 0.125 0.0625 (AvSthVh and AhSthvh And [(Adi>thd and Δd2<thd) or AdiSthd and AdaXlid)] 5 0.5 0.185 0.0625 [(Δv>thVh and AhSthVh) or (ΔvSthvh and Δh>thvh)] and [(Μ^ΐΐΐίΐ and Δ(ΐ22ΐι<〇 or (Adi<thd J.Ad2>thd)] 6 0.75 * 0.125 (Av>thVh and Ah>thVh) and [(Adi>thd and Δd2Sthd) or (Mi^thd and ^2^13⁄4 )] 7 0.5 0.125 ΔvSthVh and Δΐ^ΐΐΐνΐ! and Adi>thd and Δd2>thd 8 0.5 0.25 [(Av>thvh and Ah^thvh) or (ΔvSthvh and Δh>thVh)] and Adi>thd and Ad2>thd 9 1.0 - - Others [Simplified description of the drawings] The above detailed description of the specific embodiments of the present invention with reference to the accompanying drawings BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram schematically illustrating an electronic imaging camera in accordance with an embodiment of the present invention; FIG. 2 is a schematic representation of an image in accordance with an embodiment of the present invention. FIG. 3 is a detailed diagram of a portion of an input buffer in accordance with an embodiment of the present invention, illustrating an image framing technique used in an image restoration circuit; 4 is a schematic diagram of a mask for speckle removal in an image restoration circuit according to an embodiment of the present invention; 119049.doc -29- 200821989 FIG. 5 is a diagram of a specific embodiment of the present invention FIG. 6 is a flow chart schematically illustrating a method for noise filtering according to an embodiment of the present invention; FIG. 7 is a diagram of a method according to the present invention; A schematic diagram of a set of deconvolution filters of a specific embodiment, the deconvolution filters applied to different portions of a restored image; and ~ Figures 8A and 8B A schematic diagram of a deconvolution core mask applied to a sub-image output by a mosaic image sensor according to an embodiment of the present invention. [Main component symbol description] 119049.doc 20 Electronic imaging camera 22 Objective optical component 24 Mosaic Image sensor 26 Image restoration circuit / digital reproduction circuit 28 Image signal processor (ISP) 30 Video screen 32 pixels 40 Green balance unit 42 Input fuzzy Bell (IBB) buffer 44 Frame extension unit 46 Speckle removal unit 48 Edge Detection Unit 50 Adaptive Threshold -30 - 200821989 52 Widening Unit 54 False Dynamic Noise Filter 56 Selector 58 Delay Line 60 Deconvolution Filter (DCF) 62 DCF Selector 70 Content of Buffer 42 Part 72 Column 76 Boxing Boundary 90 Mask 92 Pixels 100 Mask 102 Pixels 104 Pixels 120 DCF Core Set / Mask 122 DCF Core 124 Center Area 126 Peripheral Area 130 Core Mask 132 Center Pixel 134 Shadow Pixel 140 Core Mask 119049 .doc -31 -

Claims (1)

200821989 X. Patent application scope: 1 · An imaging device comprising: - a mosaic image sensor configured to generate a stream of _ input pixel values belonging to a plurality of input sub-images, each sub-image responding to a human shot a different, individual color light on the mosaic image sensor; an image reduction coupled to receive and digitize (4) the input pixel values in each of the input sub-images of the input sub-images to generate Corresponding plurality of enhanced output sub-images; and an image signal processor (ISP) coupled to receive and combine the plurality of output sub-views "images" to generate a color video output image. 2. 2 request item i The device, wherein the input sub-images and the output sub-images have the same format so that the Isp can be coupled to receive and combine the output sub-images or the input sub-images. The 3 pixel values in the plurality of input sub-images are interlaced by a predetermined interlaced pattern in a single input pixel stream output by the mosaic image sensor. And wherein the output sub-shirt images comprise output pixel values, which are interleaved by the image restoration circuit in an output pixel stream according to the predetermined interlaced pattern. The device of the θθ item 1 wherein each sub-image has one Input blur, and wherein the output sub-images have an output blur after being filtered by the image restoration circuit, which is smaller than the input blur. 5_ The device of claim 4, comprising an objective optical element, the objective optical Yuan 4 Niu Xing right "has" - the point spread function (PSF) causing the input blur, and the pot in the pot; the ceremony, ', μ 篆 篆 reduction circuit contains a deconvolution filter (DCF), which has 119049. Doc 200821989 6 · -: Filter core determined according to the PSF. The device of claim 5, wherein the PSF is changeable on a plane of the image sensor, and the image restoration circuit is configured to respond to the plane of the image sensor - The pSF changes, applying different waves to the input values from different regions of the image sensor. A device of claim 5 wherein the image restoration circuit is configured to apply a different filter core to the input pixel values for the feature of the input sub-images. = n 7 - the device of the item 'where the mosaic image sensing '匕 contains a filter array configured with a Bell mosaic pattern. 9. The device of claim 8, wherein the input pixel values comprise alternating green and blue pixels of the first column and alternating green and red pixels of the second column, and the image restoration circuit of A comprises a green balance unit. The system is consumed to compensate for the change in sensitivity of the green pixels between the 4th and second columns. 1〇: The device of claim 9, wherein the sensitivity change is in the image sensing unevenness and wherein the green balance unit is configured to respond to the sensitivity change of the uneven hook to apply - unevenly make up. U. The device of any one of claims 1 to 7 is incorporated in the J member device, which comprises an objective optical element, and the immersion optical element can be 70 pieces, "the state is used to be under - predetermined blur The light is focused on the mosaic image sensor, wherein the image restoration circuit includes a speckle removal unit, such as a defective pixel in each input sub-image of the input sub-image, and corrects the ^^. The input pixel value of Yang 1豕京, the round-in pixel value of the special (10) pixel and the input pixel value difference of the adjacent pixel are more than 119049.doc 200821989. The maximum difference is determined according to the blur. The apparatus of any one of claims 1 to 7, wherein the image restoration circuit comprises a noise filter configured to digitally filter each of the input sub-images to reduce noise within the sub-images. 13. The device of claim 12, wherein the noise filter is configured to determine a direction and magnitude of a local gradient within the input sub-images, and to select an equal direction and the magnitude to select a filter core Used to reduce noise. 14. The device of claim 12, wherein the image restoration circuit comprises a de-folding wave transformer (DCF) configured to filter the input sub-images after the noise filter reduces noise. The device of claim 14, wherein the image restoration circuit includes an edge detector configured to identify an edge region within the input image and to control an input of the DCF to receive at the edge An input pixel value that is not filtered by noise in the region and receives noise-reduced input pixel values outside the edge regions from the noise filter. 16. For the device of claim 15, wherein the edge is detected The detector is configured to detect edge pixels, and wherein the image restoration circuit includes a widening unit configured to widen the edge regions by adding pixels to edge regions around the edge pixels 17. The apparatus of claim 16, wherein the widening unit is configured to identify from the edge phase H purely one of an edge pixel of the image of the image, and at least at least one of the sub-images a second child The apparatus of the present invention is the same as the apparatus of any one of claims 1 to 7, wherein the image restoration circuit comprises a digital filter Is configured to use the core of a given size to fold the input sub-images; and a frame extension unit configured to add a pixel value boundary around the input sub-images, the boundary having a The apparatus of the present invention, wherein the input pixel values in the plurality of input sub-images are in a mosaic image sensor The output of the single input pixel stream is interleaved using a predetermined interlaced pattern, and wherein the image restoration circuit includes a digital filter configured to be used to "reproduce the wrong core pattern - the individual core of the material is to be folded Each of the sub-images of the sub-images is input to separate the sub-images from the other sub-images. 20· an imaging method, comprising: receiving, from a mosaic image sensor, a plurality of input sub-images, a pixel value μ′ each sub-image responder shot on the mosaic image sensor The color of the individual colors; and the digits of the input pixel values in the input sub-images of the input sub-images to generate a corresponding plurality of enhanced output sub-images; ^Image Signal Processor (ISP) combines the #轮出 sub-image to generate a color video wheeled image. 2 1 · As requested in item 20, the emperor is a silly and h η 〃 ( (4) input sub-image and the output sub-images have the same format, so that ## έ ίί + $ # + 〇 ISP to receive And combine the temple powder out ~ like or such a loser image. The method of claim 21, wherein the input pixel values of the plurality of input persons are interlaced by a predetermined interlaced pattern in the output pixel stream output by the mosaic image sensor, and The mid-digits include an output pixel value that interleaves the output sub-images in an output pixel stream in accordance with the predetermined interlace pattern. 23. The method of claim 20, wherein each sub-image has an -input blur, and wherein the digit filter input pixel value comprises processing the input sub-image f: the input pixel values in each input sub-image So that the output sub-images have an output blur that is less than the input blur. The method of claim 23, wherein the light is focused by the objective optical element on the mosaic image sensor, the objective optical element having a point spread function (pSF) that causes the input blur, and Processing the input pixel values includes applying a deconvolution filter (DCF) to the input pixel values in each of the sub-images of the sub-images, the DCF having a filter core determined according to the PSF. The method of claim 24, wherein the PSF is changeable on a plane of the image sensor, and applying the DCF comprises responding to a PSF change on a plane of the image sensor, applying a different filter core The input pixel values from different regions of the image sensor. The method of claim 24, wherein applying the DCF comprises applying different wave cores to the input pixel values based on one of the characteristics of the input sub-images. The method of any one of claims 20 to 26, wherein the mosaic image sensor comprises a filter array configured with a Bell mosaic pattern. The method of claim 27, wherein the input pixel values comprise a first column of alternating green and blue pixels and a second column alternating green and red pixels, and comprising processing the input pixel values to A sensitivity change of one of the green pixels between the second columns is compensated by digitally filtering the input pixel values. 29. The method of claim 27, wherein the sensitivity change is non-uniform in an area of the image sensing region, and wherein processing the input pixel values comprises applying an uneven compensation to respond to uneven sensitivity Variety. The method of any one of claims 20 to 26, wherein the light system is focused on the mosaic image sensor by an objective optical element having a predetermined blur and includes identifying the input sub-image Each of the defective pixels in the input sub-images and the input pixel values of the defective pixels are corrected, and the input pixel values of the defective pixels are different from the input pixel values of the adjacent pixels by more than a maximum difference, and the maximum difference is based on the blur To decide. The method of any one of claims 20 to 26, wherein the digitally filtering the input pixel values comprises processing the input sub-images of the input sub-images to reduce noise within the sub-images. 32. The method of claim 31, wherein processing each of the input sub-images of the input sub-images comprises determining a direction and magnitude of a local gradient within the input sub-images, and echoing the equal direction and the magnitudes The filter core is chosen to be used to reduce noise. 33. The method of claim 31, wherein digit filtering the input pixel values comprises applying a deconvolution filter (DCF) to filter the input sub-images of the input sub-images to reduce noise Enter a sub-image. The method of claim 33, wherein applying the DCF includes identifying an edge region within the wheeled image and controlling one of the DCF inputs such that the DCF receives no within the edge region The input pixel values are processed to reduce noise, and outside of the edge regions, the DCF receives input pixels that are processed to reduce noise reduction after noise. 35. The method of claim 34, wherein identifying the edge regions comprises detecting edge pixels and widening the edge regions by adding pixels to the edge regions surrounding the edge pixels. 36. The method of claim 35, wherein widening the edge regions comprises detecting an edge pixel in a first sub-image of the sub-images and in at least a second sub-image of the sub-images The pixels adjacent to the edge pixel are added to the edge region. 37. The method of claim 2, wherein the input pixel value of the number of (4) (four), etc., comprises using a given size of the core to convolve the input sub-images and includes the number of digits. A pixel value boundary is added around the input sub-images before entering the pixel value, and the boundary has a width selected in response to the size of the wave core. 38. The method of requesting (4) to the financial-item, wherein the input pixel values in the plurality of input sub-images are in a single-input pixel stream outputted by the mosaic image sensing stolen Using a shirt-interlaced pattern and intersecting: and wherein the digits are filtered, the input pixel values are included in the interlaced pattern--the individual cores of the mask are used to fold the input sub-images of the input sub-images to The images of the sub-images are separated and filtered. Thick /, /, he at 119049.doc