JP2005004767A - Method and system for filtering noise adaptively to luminance change - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and system for filtering noise adaptively to a luminance change. <P>SOLUTION: The luminance noise filtering system and method is used to determine virtual filtered luminance for a pixel position including a predetermined pixel data region received from an image sensor. For the purpose of eliminating the luminance noise, the pixel data region directly received from the image sensor is used to eliminate a frame memory. Further, in this adaptive noise filtering, the virtual filtered luminance is selected as final luminance for a dark image, and reference luminance is selected without virtual noise filtering for a light image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image pickup apparatus, and more particularly, to a method and system for filtering luminance noise for pixels using a predetermined area of pixel data received directly from an image sensor so that line memory capacity is minimized.

  FIG. 1 is a view for explaining an image pickup apparatus 102 such as a conventional camera system including an image sensor 104. The image sensor 104 generates pixel data for an image of the object 106 projected through the objective lens 108. For example, the image sensor 104 may be a CIS (CMOS Image Sensor) mainly used in mobile communication terminals such as mobile phones and PDAs.

  The signal processor 110 receives the pixel data from the image sensor 104 and displays the image of the object 106 on the display 112, for further processing by the image recognition system 114, or the image is displayed on the remote display 118. The data is processed so that the image can be sent through the transmission system 116. Referring to FIGS. 1 and 2, the image sensor 104 generates pixel data 120 in the form of a Bayer Filter Array placed on the image sensor 104.

  The image sensor 104 generates an intensity signal of a corresponding color at each pixel position in the form of a Bayer filter array. “R” corresponds to a pixel position in the image sensor 104 that generates a red color component sensitivity signal. “G” corresponds to a pixel position in the image sensor 104 that generates a sensitivity signal of a green color component. “B” corresponds to a pixel position in the image sensor 104 that generates a sensitivity signal of a blue color component.

  An interpolation algorithm is used in the signal processor 110 to determine the sensitivity signal of the interpolated RGB color component for each pixel location. The interpolation algorithm uses pixel data in the form of the Bayer filter array 120.

  The interpolation algorithm is well known to those skilled in the art as disclosed in Patent Document 1, Patent Document 2, or Patent Document 3. As shown in FIG. 3, in the interpolation algorithm, in order to determine the color components R ′, G ′, and B ′ interpolated at a specific pixel position 124, the area of the pixel data 126 around the pixel position 124 is determined. used.

  Instantaneous noise is detected by the image pickup device 102 and affects the quality of the generated object image. Even if the image sensor 104 is irradiated with a certain amount of light, a change appearing in the output signal from the image sensor 104 is instantaneous noise. The instantaneous noise may be caused by “shot” noise and 1 / f noise in an optical element included in the image sensor 104. In addition, the instantaneous noise may be caused by “thermal” noise in transistors and other circuit elements used in the image sensor 104, or may be caused by quantization error of an analog-digital converter used in the image sensor 104. sell.

  The instantaneous noise increases with the brightness of the image. However, the effect of instantaneous noise appearing in the image is more fatal at low illuminance, because the SNR (Signal to Noise Ratio) is reduced by the decrease in illuminance. Actually, instantaneous noise acts as a limit of the dynamic range of the image sensor 104 under dark conditions.

  FIG. 4 is a view for explaining a conventional technique for performing the instantaneous noise removal. The pixel data 120 is generated by the image sensor 104 in the form of a buyer filter array. The signal processor 110 interpolates the pixel data 120 to generate interpolated RGB color components 122A, 122B, and 122C, respectively, stored in a conventional frame memory 122.

  In the prior art, after the interpolated RGB color components 122A, 122B, and 122C for the n * n pixel array are generated and stored in the frame memory device 122, the noise removing unit 132 removes the bad influence of instantaneous noise. For this purpose, the interpolated RGB color components are used. The noise remover 132 using a 3 * 3 pixel array of the interpolated RGB color components 122A, 122B, 122C is shown in FIG. However, 5 * 5, 7 * 7, or other n * n pixel arrays of the interpolated RGB color components 122A, 122B, 122C may be used to remove instantaneous noise.

The conventional noise removal technique as described above requires a frame memory 122 sufficient to store the interpolated RGB color components of the n * n array used in the noise removal unit 132. However, when the camera system 102 is built in a mobile communication terminal such as a mobile phone or a PDA, the relatively large memory 122 is disadvantageous as described above. Accordingly, when the camera system 102 is built in a mobile communication terminal, it is required to remove the frame memory 122 in order to reduce the size of the terminal, reduce power consumption, and particularly reduce cost.
US Pat. No. 5,382,976 US Pat. No. 5,506,619 US Pat. No. 6,091,862

  The technical problem to be solved by the present invention is to provide a method and system for filtering luminance noise for a pixel using a relatively small area of pixel data received directly from an image sensor so that the frame memory is eliminated. There is to offer.

  According to one aspect of the present invention for achieving the above technical problem, in a method or system for luminance noise filtering, a predetermined region of pixel data received from an image sensor is used, and a certain pixel in the predetermined region is used. Determine virtually filtered luminance. The virtually filtered luminance is calculated by averaging each pixel data multiplied by a weighting factor for each pixel in the region.

  In another aspect of the invention for achieving the above technical problem, the color component for each pixel is determined from the pixel data of the region.

  In still another aspect of the present invention for achieving the above technical problem, a reference luminance for each pixel is determined from the color components. The final brightness of each pixel is selected between the brightness that is virtually filtered by the brightness representing the image brightness and the reference brightness. The present invention provides adaptive noise filtering that selects the virtually filtered luminance as the final luminance for a dark image.

  In this manner, noise filtering is performed on each pixel using the virtual luminance determined using the pixel data of the predetermined area received from the image sensor. Accordingly, the frame memory for storing the interpolated pixel data is removed. The removal of the frame memory is particularly advantageous for reducing the size, power consumption, and cost of the camera system built in the mobile communication terminal.

  The noise removal filter according to the present invention can effectively remove noise corresponding to a change in luminance without adding a line memory. Therefore, a display using such a noise removal filter is suitable for realizing low cost and downsizing of a DSC or a mobile phone, and can improve the image quality of the display screen so that there is no noise.

  For a full understanding of the invention, its operational advantages, and the objectives achieved by the practice of the invention, reference should be made to the drawings illustrating the preferred embodiments of the invention and the contents described in the drawings. .

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The same reference numerals provided in each drawing denote the same members.

  Referring to FIGS. 5 and 6, the system 200 according to an aspect of the present invention performs noise filtering using a predetermined area of pixel data received directly from the image sensor 104. In addition, noise filtering is used while determining the luminance using the predetermined pixel data region as described above for a certain pixel position in the region. FIG. 7 is a flowchart for explaining the operation of the system of FIG.

  The system 200 is typically implemented within the signal processor 110 to process pixel data received from the image sensor 104. 6 to 8, the RGB matrix 202 includes a line memory device 204. The memory device 204 receives and stores a predetermined pixel data area 206 generated in the form of a buyer color filter array from the image sensor 104 (step 302 in FIG. 7).

  The line memory device 204 may be realized in any form of data storage device. For example, when the image sensor 104 sequentially outputs pixel data row by row, the line memory device 204 stores pixel data for 4 * H pixel positions. Here, H is the number of pixel columns in the image sensor 104 (for example, 684 columns).

  The RGB matrix 202 also includes an interpolation processor 208 that determines interpolated color components R ', B', G 'at a pixel location 210 in the region 206 (step 304 of FIG. 7). Referring to FIG. 8, the interpolation processor 208 interpolates the pixel data region 206 to generate the interpolated color components R ', B', G '. Such interpolation algorithms for generating the interpolated color components R ', B', G 'are well known to those skilled in the art.

  In addition, the RGB matrix 202 determines a reference luminance Y1H from the interpolated color components R ′, B ′, and G ′ as shown in Equation 1 (step 304 in FIG. 7).

  Y1H = (19R '+ 38G' + 7B ') / 64 (1)

  Such reference luminance Y1H is calculated by a common standard for calculating luminance, as is well known to those skilled in the art.

  The RGB matrix 202 includes a virtual luminance processor 212 that determines the virtual luminance arrays Y0V, Y1V, and Y2V (step 304 in FIG. 7). Referring to FIG. 9A, A, B, and C constituting the first virtual luminance array Y0V = [A, B, C] are boxes indicated by A, B, and C in the pixel data area 206, respectively. It is the average of the sensitivity values of the four pixels.

  Similarly, referring to FIG. 9B, D, E, and F constituting the second virtual luminance array Y1V = [D, E, F] are respectively represented by D, E, and F in the pixel data area 206. Is the average of the sensitivity values of 4 pixels in the box. Referring to FIG. 9C, G, H, and I constituting the third virtual luminance array Y2V = [G, H, I] are displayed as G, H, and I in the pixel data area 206, respectively. The average of the sensitivity values of 4 pixels in the box. Such a virtual luminance array Y0V, Y1V, Y2V is used for the pixel location 210 using the pixel data region 206 that determines the interpolated color components R ′, B ′, G ′ for the pixel location 210. Determined.

  In FIG. 6, the RGB matrix 202 transmits the reference luminance Y1H and the virtual luminance arrays Y0V, Y1V, Y2V to the noise filter 214. In addition, the RGB matrix 202 transmits the interpolated color components R ′, B ′, and G ′ to a color difference signal processor 215 in a Y / C (luminance / color difference) processor 216. Also, the RGB matrix 202 transmits the reference luminance Y1H and the interpolated color components R ′, B ′, and G ′ to the data processor 218.

  FIG. 10 is a block diagram showing the noise filter 214 for determining the virtually filtered luminance GOUT from the virtual luminance arrays Y0V, Y1V, Y2V (step 306 in FIG. 7). The noise filter 214 includes multipliers 222, 224 and 226 and an adder 228 in the multiplication and addition unit 230.

  The first multiplier 222 multiplies the first luminance array Y0V and the first weighted coefficient array GAD0 [39 63 39] as shown in Equation 2.

  39 * A + 63 * B + 39 * C (2)

  Similarly, the second multiplier 224 multiplies the second luminance array Y1H and the second weighting coefficient array GAD1 [63 104 63] as shown in Equation 3.

  63 * D + 104 * E + 63 * F (3)

  Also, the third multiplier 226 multiplies the third luminance array Y2V and the third weighting coefficient array GAD2 [39 63 39] as shown in Equation 4.

  39 * G + 63 * H + 39 * I (4)

  The adder 228 adds all the result values calculated by the multipliers 222, 224 and 226. The shifter 232 in the brightness control unit 234 divides the calculation result of the adder 228 by all the added values of the weighting coefficients GAD0, GAD1, and GAD2, that is, 512. The fourth multiplier 236 in the brightness control unit 234 multiplies the calculation result of the shifter 232 and the luminance compensation coefficient α to generate the virtual filtered luminance GOUT.

  In the present invention, the luminance compensation coefficient α and the weighting coefficient arrays GAD0, GAD1, and GAD2 are determined so as to obtain an optimal image quality by using a Gaussian distribution equation and stored in the data register 220 of FIG. Thus, the virtually filtered luminance GOUT is determined by averaging pixel data multiplied by each weighting factor for each pixel location in the region 206.

  In addition, referring to FIG. 6, the data processor 218 determines a threshold value THV (step 306 in FIG. 7). Referring to FIG. 11, the data processor 218 determines a threshold value THV that varies depending on the adaptive luminance AY, as shown in the graph 223 of FIG. In the embodiment of the present invention, the adaptive luminance AY represents the overall brightness of the previous image.

  For example, in FIG. 12, the threshold value THV may be determined according to another graph 225 that varies depending on the automatic luminance gain used in the camera system 102. As is well known to those skilled in the art, the automatic luminance gain represents the brightness of an image detected by the image sensor 102 in the camera system 102. A high auto luminance gain represents a dark image for the high threshold THV by the graph 225 of FIG. At this time, the automatic luminance gain represents the adaptive luminance AY.

  In another aspect of the present invention, the reference luminance Y1H is used to represent the adaptive luminance AY in FIG. At this time, the threshold value THV is determined according to the brightness level for each pixel position. In other words, the adaptive luminance AY is an average of the reference luminances Y1H for a predetermined pixel data area.

  In FIG. 10, the comparison unit 238 includes a subtractor 24 and an absolute value generator 242. The comparison unit 238 determines an absolute value DELTA of the difference between the virtually filtered luminance GOUT and the reference luminance Y1H (step 308 in FIG. 7). The comparator 224 in the comparator 238 compares the absolute value DELTA with the threshold value THV (step 310 in FIG. 7).

  In FIG. 10, a MUX (multiplexer) 246 receives the reference luminance Y1H and the virtual filtered luminance GOUT. When the absolute value DELTA is less than or equal to the threshold value THV, the comparator 224 controls the MUX 246 so as to select and output the virtually filtered luminance GOUT as the final luminance for the corresponding pixel position 210 ( Step 312 in FIG. In other words, when the absolute value DELTA is larger than the threshold value THV, the comparator 224 controls the MUX 246 to select and output the reference luminance Y1H as the final luminance for the corresponding pixel position (FIG. 7). 314 stage).

  If the pixel position 210 is the last pixel position of the image (step 316 of FIG. 7), the operation of the flowchart of FIG. 7 is terminated. Otherwise, the operations of steps 302, 304, 306, 308, 310, 312, 314, and 316 are repeated for other pixel data regions from the image sensor 104. Such repeated steps are repeated to determine the luminance and interpolated color components for each pixel location in each pixel data region of the image.

  In general, a larger pixel data array is generated from the image sensor 104 to provide a data array of small processed images, as is well known to those skilled in the art. Pixel data located in the outer peripheral direction of the image sensor 104 is used for image signal processing of adjacent pixels located in the center direction of the processed image. However, as is well known to those skilled in the art, extra pixel data regions around a pixel location are not required, so that pixel data located in such an outer peripheral direction is derived from the processed image. Removed. The operations of steps 304, 306, 308, 310, 312, 314, and 316 of FIG. 7 are repeated for each pixel location for the processed image.

  Thus, noise filtering is performed on a pixel location using the pixel data region 206 from the image sensor 104 while determining the final luminance Y1H / GOUT for the pixel location 210. The pixel data area 206 is also used to determine the interpolated color components R ′, G ′, B ′ of the pixel location 210. Since noise filtering is performed using the pixel data region 206 from the image sensor 104, the frame memory storing the interpolated pixel data can be eliminated by the present invention.

  Therefore, the luminance noise filter uses the image data from the image sensor to determine the final luminance Y1H / GOUT at the pixel location 210 and the interpolated color components R ′, G ′, and B ′. The capacity of the memory device included in the image pickup device can be minimized. When the camera system is built in the mobile communication terminal, the reduced memory capacity as described above is advantageous for reducing the size of the terminal, reducing the power consumption, and particularly reducing the cost.

  In addition, the present invention provides adaptive noise filtering by varying the threshold THV according to image brightness. For a brighter image, the reference brightness Y1H is selected as the final brightness instead of the virtually filtered brightness GOUT. Noise filtering distorts the image so that the effect of instantaneous noise appears smaller for brighter images. Accordingly, the reference luminance Y1H without distortion is selected as the final luminance for a brighter image by the virtual noise filtering. Conversely, the negative effects of instantaneous noise are even greater for darker images. Therefore, for a darker image, the virtual filtered luminance GOUT after noise filtering is selected as the final luminance.

  In FIG. 6, a luminance signal processor 217 in the Y / C processor 216 performs contour compensation processing on the final luminance Y1H / GOUT in order to generate a luminance signal Y to be output to a Y / C (luminance / color difference) formatter 219. The color difference signal processor 215 in the Y / C processor 216 receives the interpolated color components R ′, G ′, B ′ and automatic white balance data (AWBD: Auto White Balance Data) from the data processor 218. The color difference signals R ″ -Y and B ″ -Y to be output to the Y / C formatter 219 are generated. Here, R ″ represents the sensitivity of the interpolated color component R ′ adjusted by automatic white balance data, and B ″ represents the sensitivity of the interpolated color component B ′ adjusted by automatic white balance data. .

  The Y / C formatter 219 has luminance and color difference data Y, Cb, Cr and color data R ″ ′, G ″ ′ according to a standard data format required by the display 112, the image recognition system 116, or the transmission system 116. And B ″ ′. The components output by the Y / C processor 216 and the Y / C formatter 219 are well known to those skilled in the art. Further, as is well known, automatic luminance gain data AED is determined from Y1H, R ′, and B ′, and is output to the image pickup device.

  As mentioned above, the optimal embodiment was disclosed by drawing and the specification. Certain terminology has been used herein for the purpose of describing the invention only and is intended to limit the scope of the invention as defined in the meaning and claims. It was not used. Accordingly, those skilled in the art will appreciate that various modifications and equivalent other embodiments are possible. For example, the present invention has been described for a camera system 102 that can be part of a mobile communication terminal. However, the present invention can be used for any form of imaging device that performs image signal processing. In addition, the components according to an embodiment of the present invention may be realized by various combinations of hardware, software, or individual devices or integrated circuit devices. In addition, a certain number described here, that is, the number of pixels or the like is merely an example and is not limited to a specific number. Therefore, the technical scope of the present invention should be determined based on the description in the claims.

  The noise filtering method and system according to the present invention can be used in a camera system of a mobile communication terminal such as a mobile phone or a PDA.

1 is a diagram for explaining an image pickup apparatus such as a conventional camera system. 5 is a diagram for explaining a conventional technique for interpolating pixel data generated in a buyer color filter array form with color components; 6 is a diagram for explaining a conventional technique for determining a color component of a pixel in a predetermined area of pixel data generated in the form of a buyer color filter array. 6 is a diagram for explaining a conventional technique for performing noise removal after color components of an n * n array are determined. 6 is a diagram illustrating an embodiment of the present invention in which noise filtering is performed using a predetermined area of pixel data in order to determine luminance and color components at a pixel. FIG. 6 is a block diagram of a system according to an embodiment of the present invention that performs noise filtering as in FIG. 5 such that the frame memory is removed. It is a flowchart for operation | movement description of the system of FIG. 8 is a diagram for explaining an embodiment of the present invention for determining color components of pixels in a predetermined region of pixel data generated in the form of a buyer color filter array in FIG. 7. FIG. 9 is a diagram representing a region of pixel data of FIG. 8 for determining a number of virtual luminance arrays. FIG. 9 is a diagram representing a region of pixel data of FIG. 8 for determining a number of virtual luminance arrays. FIG. 9 is a diagram representing a region of pixel data of FIG. 8 for determining a number of virtual luminance arrays. It is a block diagram showing the noise filter of the system of FIG. FIG. 7 is a diagram illustrating an example of a graph used in a data processor of the system of FIG. 6 for determining a threshold value by luminance. 7 is a diagram illustrating another example of a graph used in the data processor of the system of FIG. 6 for determining a threshold value according to an automatic luminance gain of a camera system.

Explanation of symbols

200 System 202 RGB Matrix 204 Line Memory Device 208 Interpolation Processor 212 Virtual Luminance Processor 214 Noise Filter 215 Color Difference Signal Processor 216 Y / C (Luminance / Color Difference) Processor 217 Luminance Signal Processor 218 Data Processor 219 Y / C (Luminance / Color Difference) Formatter 220 Data register Y1H Reference luminance Y0V, Y1V, Y2V Virtual luminance array GAD0 First weighting coefficient array GAD1 Second weighting coefficient array GAD2 Third weighting coefficient array Y1H / GOUT Final luminance THV Threshold AED Automatic luminance gain data AWBD Automatic white balance data Y , Cb, Cr Luminance and color difference R ′, B ′, G ′ Color components

Claims (27)

Inputting a predetermined area of pixel data from the easy sensor;
Determining a virtually filtered luminance from a pixel data region for a pixel location within the predetermined region;
A luminance noise filtering method comprising: The luminance noise filtering method according to claim 1, further comprising determining an interpolated color component for the pixel position from the pixel data region. The luminance noise filtering method according to claim 2, wherein the luminance noise filtering method further comprises determining a reference luminance for the pixel position from the interpolated color component. The luminance noise according to claim 3, further comprising: selecting a final luminance of the pixel position that varies according to adaptive luminance between the virtual filtered luminance and the reference luminance. Filtering method. The luminance noise filtering method includes:
Determining a threshold from the adaptive brightness;
If the absolute value of the difference between the virtual filtered luminance and the reference luminance is less than or equal to the threshold, selecting the virtual filtered luminance;
Selecting the reference brightness if the absolute value of the difference between the virtual filtered brightness and the reference brightness is greater than the threshold;
The luminance noise filtering method according to claim 4, further comprising: The luminance noise filtering method according to claim 5, wherein the adaptive luminance is determined from the overall brightness of a previous image. The luminance noise filtering method according to claim 5, wherein the adaptive luminance is determined from an average reference luminance for a predetermined pixel data area. The luminance noise filtering method according to claim 5, wherein the threshold value is larger when the adaptive luminance is small. The luminance noise filtering method according to claim 4, wherein the adaptive luminance is an automatic luminance gain for the image sensor. The luminance noise filtering method according to claim 4, wherein the adaptive luminance is the reference luminance. The luminance noise filtering method according to claim 1, wherein the virtual filtered luminance is determined by averaging pixel data multiplied by a weighting coefficient for each pixel position of the region. The luminance noise filtering method according to claim 1, wherein the image sensor is built in an image pickup apparatus for mobile communication having a minimized line memory capacity. A memory for storing a predetermined area of pixel data from the image sensor;
A noise filter that determines a virtual filtered luminance from a pixel data region for a pixel location within the predetermined region;
A luminance noise filtering system comprising: The luminance noise filtering system of claim 13, further comprising a matrix that determines an interpolated color component for the pixel location from the pixel data region. The luminance noise filtering system according to claim 14, wherein the matrix determines a reference luminance for the pixel position from the interpolated color component. The luminance noise filtering system according to claim 15, wherein the noise filter selects a final luminance of the pixel position that varies depending on adaptive luminance between the virtual filtered luminance and the reference luminance. The luminance noise filtering system further comprises a data processor for determining a threshold value from the adaptive luminance,
The noise filter selects the virtual filtered luminance if the absolute value of the difference between the virtual filtered luminance and the reference luminance is less than or equal to the threshold, and the virtual filtered luminance and the virtual filtered luminance The luminance noise filtering system according to claim 16, wherein the reference luminance is selected if an absolute value of a difference between reference luminances is larger than the threshold value. The luminance noise filtering system according to claim 17, wherein the adaptive luminance is determined from the overall brightness of a previous image. The luminance noise filtering system according to claim 17, wherein the adaptive luminance is determined from an average reference luminance for a predetermined pixel data region. The luminance noise filtering system according to claim 17, wherein the threshold value is further increased when the adaptive luminance is small. The luminance noise filtering system according to claim 16, wherein the adaptive luminance is an automatic luminance gain for the image sensor. The luminance noise filtering system according to claim 16, wherein the adaptive luminance is the reference luminance. The luminance noise filtering system according to claim 13, wherein the virtual filtered luminance is determined by averaging pixel data multiplied by a weighting coefficient for each pixel position of the region. The luminance noise filtering system according to claim 13, wherein the image sensor is incorporated in an image pickup device for mobile communication having a minimized line memory capacity. Means for inputting a predetermined area of pixel data from the image sensor;
Means for determining a virtual filtered luminance from a pixel data region for a pixel location within the region;
A luminance noise filtering system comprising: The luminance noise filtering system of claim 25, further comprising means for determining an interpolated color component for the pixel location from the pixel data region. The luminance noise filtering system includes:
Means for determining a reference luminance for the pixel location from the interpolated color component;
Means for selecting between the virtual filtered luminance and the reference luminance a final luminance at the pixel location that varies depending on adaptive luminance;
The luminance noise filtering system according to claim 26, further comprising:

Images