TWI458335B - Image signal processor and method for image enhancement - Google Patents

Description

Image signal processor for image enhancement and image processing method

The invention relates to image processing technology, and more relates to image processing technology for enhancing images.

The Bayer filter, widely used in digital cameras and camcorders, is a color filter array (CFA) that arranges RGB color filters on an image sensor. Figure 1 shows the configuration of a Bayer filter. Since the human eye is most sensitive to green light, the green sensing element configured in the Bayer filter has twice the number of red or blue sensing elements. As shown in Figure 1, the Bayer filter is configured with 50% green sensing elements, 25% red green sensing elements, and 25% blue sensing elements.

Figure 2 is a diagram for explaining image signal processing (ISP) in the prior art. After the image sensor processes the color sensing program (step S202), a demosaicking process is performed (step S204), and the color data obtained by the image sensor is obtained by some interpolation method ( The original data format) was reorganized into a full color image. In this full color image, each pixel has an RGB signal, which in turn includes a red component (R), a green component (G), and a blue component (B). In this prior art, in order to perform image enhancement operations such as reducing noise and improving sharpness (step S208), the RGB signal of the pixel must be further converted by a step called color space conversion (CSC). It is a brightness and color difference signal (ie YCbCr signal). Because the conventional technology regards the brightness signal as being more important than the color signal in reducing the image noise and improving the image sharpness, when performing image enhancement, the luminance signal is usually processed only and the color difference signal is ignored. After the image enhancement operation (step S208), the conventional technique again converts the processed luminance and color difference signals into the original RGB signals (step S210).

For the same reason (the human eye is more sensitive to green light), the green sensor is usually designed to have better and higher signal noise than the other two color sensors (red and blue sensors). Signal to noise ratio (SNR). Therefore, the brightness signal obtained by the prior art in the color gamut conversion of the RGB signal is greatly affected by the green signal (for example, according to the ITU 601 standard, the luminance signal Y and the RGB signal have The following relationship: Y = 16 + 0.299 × R + 0.587 × G + 0.114 × B , where the Y value is obviously affected by the G value is large). For image enhancement, the green signal may also introduce more noise into the brightness signal that it later converts into. In addition, the conventional technique only improves the brightness of the luminance signal, and can also cause the image to have a more serious false color after the demosaicing process.

Therefore, there is a need for a new image signal processor and image processing method capable of solving the aforementioned problems.

The invention provides an image signal processor for enhancing images. The image signal processor receives a first, a second, and a third color component of a color signal of each pixel in an image, and includes: an image intensifier for the first color component of the color signal Performing image enhancement to generate a reinforced first color component; a gain generator coupled to the image intensifier for generating an image enhancement gain according to the enhanced first color component of the color signal; and a gain The multiplier is coupled to the gain generator for image enhancement of the second and third color components according to the image enhancement gain to generate a strengthened second color component and a strengthened third color component.

The invention further provides an image signal processing method for enhancing an image. The image signal processing method includes the steps of: receiving a first, a second, and a third color component of a color signal of each pixel in an image; performing image enhancement on the first color component of the color signal to Generating a strengthened first color component; generating an image enhancement gain according to the enhanced first color component of the color signal; and performing image enhancement on the second and third color components according to the image enhancement gain to generate a An enhanced second color component and a strengthened third color component.

The following is a description of the preferred embodiment of the invention. The examples are intended to illustrate the principles of the invention, but are not intended to limit the invention. The scope of the invention is defined by the appended claims. To simplify the description, the image signal processor and the image signal processing method in the following embodiments are all examples of processing RGB signals. However, it will be appreciated that the apparatus or method of the present invention is equally applicable to processing other color signals, such as CYMK signals. More specifically, in the following, when the color signal is an RGB signal, the first, second, and third color component colors of the color signal are respectively green, red, and blue components of the RGB signal; and when the color signal is In the case of the CYMK signal, the first, second and third color components of the color signal may be the cyan, yellow, magenta and black components of the CMYK signal.

Image signal processor - embodiment 1

FIG. 3A is a schematic diagram of image signal processing for performing an image enhancement operation according to an embodiment of the invention. The image signal processor 300 receives the RGB signal image of each pixel in an image (for example, receiving the RGB signal after the demosaicing process), and directly performs image enhancement on the RGB signal without performing a color gamut conversion process. operation. Wherein, as described above, the RGB signal includes a red component (R), a green component (G), and a blue component (B). The image signal processor 300 of the present invention includes at least an image intensifier 310, a gain generator 320, and a gain multiplier 330. When the image intensifier 310 receives one of the color components of the RGB signal, the gain multiplier 330 receives the image. Two other color components.

The image intensifier 310 is used to enhance the image processing of the single color component in the RGB signal, and the image enhancement includes the noise reduction operation and the sharpness lifting operation. Since the green component typically has a higher sampling rate and a higher signal to noise ratio (SNR), in a preferred embodiment, the image intensifier 310 can perform image enhancement on the green component of the RGB signal alone. It is to be noted that the embodiment of FIG. 3 will be described in detail later, and this embodiment is for convenience of description and is not intended to limit the present invention. In other embodiments, the image intensifier 310 can also process the red component R or the blue component B alone. In the embodiment of Fig. 3, after image enhancement of the green component G, the image intensifier 310 will produce a enhanced green component G_out.

The gain generator 320 of the present invention is coupled to the image intensifier 310. In this embodiment, an image enhancement gain Igain can be generated according to the green component G and the enhanced green component G_out. For example, the gain generator 320 divides the enhanced green component G_out by the green component G to generate the image enhancement gain Igain. This means that the image enhancement gain Igain is the ratio of the enhanced green component G_out to the green component G ( ).

The gain multiplier 330 of the present invention is coupled to the gain generator 320, and image enhancement of the red and blue components R and B according to the image enhancement gain Igain generated by the gain generator 320, thereby generating a enhanced Red component R_out and a strengthened blue component B_out. For example, the gain multiplier 320 can multiply the red component R and the blue component B by the image enhancement gain Igain to obtain the enhanced red component R_out and the enhanced blue component B_out ( R_out = I _gain × R ; B_out = I _gain × B ).

As shown in FIG. 3A, the image intensifier 310 further includes an edge detecting unit 312, a noise reducing unit 314, a sharpness increasing unit 316, and a mixing unit 318.

The noise reduction unit 314 of the present invention is used to reduce noise in an image. More specifically, in one embodiment, the noise reduction unit 314 can reduce its noise by convolving both the original image and a mask representing the low pass filter. In this way, the noise reduction unit 314 recalculates the green component of the pixel with pixels around its pixel. For example, suppose the green component array is G and the filter mask used to reduce noise is F _ nr as follows:

Then, for the original green component G ₁₂ , the green component G _ nr after reducing the noise can be recalculated by the following formula:

G _ nr = a ×[(G ₆ +G ₈ +G ₁₆ +G ₁₈ )+2×(G ₇ +G ₁₁ +G ₁₃ +G ₁₇ )+4×G ₁₂ ]/16+(1- a ) ×G ₁₂ ,

The image signal processor can further adjust the green component G _ nr after noise reduction by using the coefficient a .

The sharpness enhancement unit 316 of the present invention is used to enhance the sharpness of the image. Similar to the noise reduction unit 314, in one embodiment, the sharpness enhancement unit 316 can be boosted by convolution of both the original image and a mask representing another filter. Its sharpness. In this way, the sharpness enhancement unit 316 recalculates the green component of the pixel with pixels around it. For example, suppose the green component array G , and the filter mask used to increase the sharpness is F _ se , as shown below:

Then, for the original green component G ₁₂ , the green component G_nr after the sharpness is improved can be obtained by recalculating the following formula:

G_se = b × [8 × G ₁₂ - (G ₆ + G ₇ + G ₈ + G ₁₁ + G ₁₃ + G ₁₆ + G ₁₇ + G ₁₈ )] + G ₁₂ ,

The image signal processor can further adjust the green component G_se after the sharpness is improved by using the coefficient b .

The edge detection unit 312 of the present invention is used to detect the edge of a pattern in an image, and more specifically, to identify a point in the image where the brightness or color is discontinuous. In image signal processing, removing noise is usually accompanied by the sacrifice of partial image detail or the attenuation of image sharpness. In order to balance the noise reduction and sharpness improvement, the edge detection unit 312 further determines the relationship between a pixel and a pattern edge. As the pixel is closer to the edge of the pattern, the sharpness needs to be increased and the noise is less needed. Conversely, as the pixel is farther away from the edge of the pattern, the sharpness is not required but the noise removal needs to be enhanced. Then, the mixing unit 318 (coupled to the noise reduction unit 314, the sharpness enhancement unit 316, and the edge detection unit 312) uses the relationship between the pixel and the edge obtained by the edge detection unit 312 to reduce the green component G_nr of the noise. Both the sharpness- adjusted green component G_se is moderately mixed and produces the final image-enhanced green component G_out. For example, suppose the green component array is G and the filter mask used to detect the edge of the pattern is F_ed as follows:

Then, for the original green component G ₁₂ , the relationship between the pixel and the edge of the pattern can be obtained by recalculating the following formula:

Ed =|8×G ₁₂ -(G ₆₊ G ₇₊ G ₈₊ G ₁₁₊ G ₁₃₊ G ₁₆₊ G ₁₇₊ G ₁₈ )|.

When there is an upper limit Ed _max in the neighbor relationship, Ed can be further converted into a ratio according to the following relationship:

When Ed> Ed _max, α = 1 ;

otherwise,

Finally, the image-enhanced green component Gout can be obtained according to the following formula:

G_out = α × G_se + (1 - α) × G_nr,

In this way, a moderate balance can be obtained between noise reduction and sharpness improvement.

Image signal processor - embodiment 2

FIG. 3B is a schematic diagram of an image signal processor for performing image enhancement according to another embodiment of the present invention. Similarly, the image signal processor 300' receives the RGB signals of each pixel in an image (for example, receiving the RGB signal after the demosaicing process), and directly outputs the RGB signal without performing the color gamut conversion process. Perform image enhancement work. Wherein, as described above, the RGB signal includes a red component (R), a green component (G), and a blue component (B). The image signal processor 300' also includes an image intensifier 310', a gain generator 320', and a gain multiplier 330', wherein the image intensifier 310' receives one of the color components of the RGB signal, and the gain multiplier The 330' receives the other two color components.

Similarly, the image intensifier 310' is used for image enhancement of a single color component (such as a green component) in the RGB signal, and the image enhancement includes a noise reduction operation and a sharpness enhancement operation. After image enhancement of the green component G, the image intensifier 310' produces a strengthened green component G_out'. Gain generator 320' is coupled to image enhancer 310' to produce an image enhancement gain. The gain multiplier 330' is coupled to the gain generator 320', and the image enhancement of the red and blue components R and B is performed according to the image enhancement gain Igain generated by the gain generator 320, thereby generating a enhanced red component R_out. 'And a strengthened blue component B_out'.

The image intensifier 310' also includes an edge detecting unit 312', a noise reducing unit 314', a sharpness lifting unit 316', and a mixing unit 318', wherein each element has a similar use as the foregoing embodiment. That is to reduce noise and improve sharpness.

However, the difference from Embodiment 1 is that the image signal processor 300' in Embodiment 2 further includes three low-pass filters 3401, 3402, and 3403 for outputting red, green, and blue component signals, respectively. The high frequency noise therein is filtered out before the gain generator 320' or the gain multiplier 330'. The low pass filters 3401, 3402, and 3403 can respectively generate a low pass filtered red component R_LPF, a low pass filtered green component G_LPF, and a low pass filtered blue component B_LPF. For example, the low-pass filtered green component G_LPF can be expressed as:

G_out = ( G ₆ + G ₇ + G ₈ + G ₁₁ + G ₁₂ + G ₁₃ + G ₁₆ + G ₁₇ + G ₁₈ ) / 9

Then, the gain generator 320' generates an image enhancement gain Igain', which may be Finally, the gain multiplier 320' further multiplies the red component R and the blue component B by the image enhancement gain to produce a strengthened red component R_out' and a strengthened blue component B_out' ( R_out '= I _{gain '} × R ; B_out _{'= I gain' × B)} .

By using the low-pass filters 3401, 3402, and 3403, the image signal quality output by the image signal processor can be greatly improved. Although the present embodiment is only exemplified by three filters, those skilled in the art can understand that the number, type and arrangement of the filters used in the present invention are not limited thereto.

Image signal processing method

In addition to the foregoing image signal processor 300, the present invention further provides an image signal processing method for the purpose of image enhancement. FIG. 4 is a flow chart of a method for processing video signals according to an embodiment of the invention. The image signal processing method includes: in step S402, receiving a first color, a second color, and a third color component of a color signal of each pixel in an image, wherein the first, the second, and the third color component The color components are respectively selected from a red component, a green component, and a blue component of the RGB signal; in step S404, the first color component of the color signal is image-enhanced to generate a reinforcement a first color component; in step S406, an image enhancement gain is generated according to the enhanced first color component of the color signal; and in step S408, the second and the third color component are enhanced according to the image enhancement gain Image enhancement is performed to produce a strengthened second color component and a strengthened third color component. In one embodiment, the image enhancement gain can be obtained by dividing the enhanced first color component by the unenhanced first color component; and by multiplying the second and third color components by the image enhancement A strengthened second color component and a strengthened third color component are obtained. Since the steps S402-408 of the method can be implemented by the components in the image signal processor of the foregoing FIG. 3 and achieve the same function, other embodiments of the method of the present invention will not be further described herein.

It can be understood from the foregoing that whether the image signal processor or the image signal processing method of the present invention can reduce noise and improve sharpness without color space conversion (CSC), it is helpful. To reduce the cost of image processing on the hardware.

The present invention has been described above with reference to the preferred embodiments thereof, and is not intended to limit the scope of the present invention, and the invention may be modified and modified without departing from the spirit and scope of the invention. The scope of the invention is defined by the scope of the appended claims.

300. . . Video signal processor

310. . . Image intensifier

312. . . Edge detection unit

314. . . Noise reduction unit

316. . . Sharpness lifting unit

318. . . Mixing unit

320. . . Gain generator

330. . . Gain multiplier

310’. . . Image intensifier

312’. . . Edge detection unit

314’. . . Noise reduction unit

316’. . . Sharpness lifting unit

318’. . . Mixing unit

320’. . . Gain generator

330’. . . Gain multiplier

3401. . . Low pass filter

3402. . . Low pass filter

3403. . . Low pass filter

R. . . Red ingredient

G. . . Green ingredients

B. . . Blue ingredients

G_nr. . . Noise reduction green ingredients

G_se. . . Sharpness-enhancing green ingredients

G_out. . . Enhanced green ingredients

R_out. . . Fortified red ingredients

B_out. . . Fortified blue ingredients

G_LPF. . . Low pass filtered green ingredients

R_LPF. . . Low pass filtered red component

B_LPF. . . Low pass filtered blue component

Figure 1 shows the configuration of a Bayer filter.

Figure 2 is a diagram for explaining the image signal processing technique in the prior art.

FIG. 3A is a schematic diagram of image signal processing for performing an image enhancement operation according to an embodiment of the invention.

FIG. 3B is a schematic diagram of an image signal processor for performing image enhancement according to another embodiment of the present invention.

FIG. 4 is a flow chart of a method for processing video signals according to an embodiment of the invention.

300. . . Video signal processor

310. . . Image intensifier

312. . . Edge detection unit

314. . . Noise reduction unit

316. . . Sharpness lifting unit

318. . . Mixing unit

320. . . Gain generator

330. . . Gain multiplier

R. . . Red ingredient

G. . . Green ingredients

B. . . Blue ingredients

G_nr. . . Noise reduction green ingredients

G_se. . . Sharpness-enhancing green ingredients

G_out. . . Enhanced green ingredients

R_out. . . Fortified red ingredients

B_out. . . Fortified blue ingredients

Claims (14)

An image signal processor for enhancing an image, which receives a first, a second, and a third color component of a color signal of each pixel in an image, including: an image intensifier for the color The first color component of the signal is image-enhanced to generate a reinforced first color component, wherein the image intensifier further comprises: a noise reduction unit for receiving the first color component, the first color component Performing noise reduction operations and generating a noise-reduced color component; a sharpness enhancement unit for receiving the first color component, sharpening the first color component, and generating a sharpness enhancement a color detecting component; an edge detecting unit configured to detect an edge of the unrelated pattern in the image and determine a relationship between each pixel and the edge; a mixing unit coupled to the noise reducing unit and the sharpness improving unit And the edge detecting unit, configured to mix the color component of the noise reduction and the color of the sharpness improvement according to the relationship determined by the edge detecting unit A first component to generate the enhanced color components; a gain generator, coupled to the image intensifier, for generating a gain image intensifier according to the first color components of the color signal of the enhanced; and a gain multiplier coupled to the gain generator for image enhancement of the second and third color components according to the image enhancement gain to generate a enhanced second color component and a enhanced third color Ingredients. The image signal processor for enhancing image according to claim 1, wherein the image enhancement gain is a ratio of the enhanced first color component to the first color component. An image signal processor for enhancing an image as described in claim 2, wherein the gain multiplier multiplies the second and the third color components by the image enhancement gain to generate the enhanced second and Three color components. The image signal processor for enhancing image according to claim 1, wherein the noise reduction operation is achieved by convolving the image with a low pass filter. The image signal processor for enhancing image according to claim 1, wherein the color signal is an RBG signal, and the first, second, and third color components are respectively included in the RGB signal. One of the red ingredients, one green ingredient, one and one blue ingredient selected. An image signal processor for enhancing an image according to claim 5, wherein when the RGB signal is from a Bayer filter, the first color component is the RGB signal. Green ingredients. The image signal processor for enhancing images according to claim 1, wherein the color signal is a CYMK signal, and the first, the second, and the third color components are respectively used by the CMYK signal. One of the cyan components, one yellow component, one magenta component, one and one black component is selected. An image signal processing method for enhancing an image, comprising the steps of: receiving a first color, a second color, and a third color component of a color signal of each pixel in an image; receiving the first color component, The first color component performs a noise reduction operation, and generates a noise-reduced color component; receives the first color component, performs a sharpness enhancement operation on the first color component, and generates a sharply enhanced color component; Measure the edge of the unrelated pattern in the image, and determine the relationship between each pixel and the edge; mix the color component of the noise reduction and the color component of the sharpness enhancement according to the relationship judged by the edge detection unit to generate a reinforcement a first color component; generating an image enhancement gain based on the enhanced first color component of the color signal; The second and the third color components are image enhanced according to the image enhancement gain to generate a strengthened second color component and a strengthened third color component. The image signal processing method for enhancing image according to claim 8 , wherein the image enhancement gain is a ratio of the enhanced first color component to the first color component. The image signal processing method for enhancing image according to claim 9, wherein the enhanced second and third color components are obtained by multiplying the second and third color components by the image enhancement gain. produce. The image signal processing method for enhancing image according to claim 8 of the patent application, wherein the noise reduction operation is achieved by convolving the image with a low pass filter. The image signal processing method for enhancing image according to claim 8 , wherein the color signal is an RBG signal, and the first, second, and third color components are respectively included in the RGB signal. One of the red ingredients, one green ingredient, one and one blue ingredient selected. The image signal processing method for enhancing image according to claim 12, wherein when the RGB signal is from a Bayer filter, the first color component is the green color of the RGB signal. Ingredients. The image signal processing method for enhancing image according to claim 8 , wherein the color signal is a CYMK signal, and the first, the second, and the third color components are respectively used by the CMYK signal. One of the cyan ingredients, one yellow ingredient, one magenta ingredient, and one black ingredient are selected.