JP5557635B2 - Outline enhancement device - Google Patents

Description

  The present invention relates to a contour enhancement device for an image signal output from an image sensor.

  A contour emphasizing device has been proposed that enhances the contour of an image signal output from an image sensor while suppressing noise. Such an apparatus is disclosed in US Pat.

  In the contour emphasis device of Patent Document 1, the contour emphasis coefficient that determines the degree of contour emphasis is changed according to the luminance difference between the target pixel and the adjacent pixel, and the luminance difference approaches the reference value for the amount of contour emphasis applied to the target pixel. It increases as the brightness difference decreases, and decreases as the brightness difference moves away from the reference value. Thereby, when the luminance difference between adjacent pixels is too large or too small, it is possible to prevent edge enhancement on an image, and thus it is possible to suppress the occurrence of halation and noise enhancement in a low luminance region.

JP 9-233369 A JP 2010-4266 A

  However, in the conventional contour enhancement device as described above, when a certain pixel has an incorrect luminance value due to noise, not only in a low luminance region, an erroneous luminance difference is calculated by the noise. As a result, the pixel may be unexpectedly edge-enhanced and noise may be emphasized, or the noise may adversely affect surrounding pixel enhancement determination, and the noise component of the pixel may increase. is there. In particular, an image signal with a poor S / N ratio may generate an image in which noise is excessively emphasized and image quality is noticeably deteriorated.

  The present invention has been made in view of the above. An object of the present invention is to provide a contour emphasizing device capable of enhancing a contour while removing noise included in an image signal without enhancing the noise.

  In order to solve the above-described problem, the contour emphasizing device of the present invention is an edge emphasizing device that performs contour emphasis processing on an image signal output from each pixel of an image sensor. Noise removal means for removing noise components included in the luminance of the pixel, and contour information amount calculation means for calculating the amount of contour information of the target pixel from the brightness of the target pixel and the peripheral pixels of the target pixel in a predetermined pixel area A contour detection unit that weights the luminance of the target pixel by a predetermined filter process, a contour enhancement coefficient determination unit that determines a contour enhancement coefficient for the weighted luminance based on the contour information amount, and a contour information amount Is smaller than the predetermined threshold value, the luminance from which the noise component has been removed by the noise removing means is output, and if the contour information amount is greater than or equal to the predetermined threshold value, A luminance signal correction unit that outputs the luminance of the image signal output from the target pixel of the element, and a contour enhancement process for the luminance output from the luminance signal correction unit using the weighted luminance and the contour enhancement coefficient Contour emphasizing means. Therefore, in the present invention, when the amount of noise information is determined based on the amount of contour information calculated in an area composed of a plurality of pixels, rather than the luminance difference between adjacent pixels, the amount of noise information is larger than the amount of contour information. Since the edge enhancement can be performed using the luminance signal from which noise has been removed, particularly when an image signal having a poor S / N ratio is handled, a pixel adjacent to the target pixel exhibits erroneous luminance due to noise. However, contour enhancement can be performed without enhancing noise.

  Preferably, the contour emphasis coefficient determining means determines the contour emphasis coefficient using the contour information amount and a predetermined function, and the contour emphasizing means uses the result of multiplying the weighted luminance by the contour emphasis coefficient, Edge enhancement processing is performed on the luminance output from the luminance signal correction unit. As a result, even when the contour information amount and the contour enhancement coefficient are expressed as a complex nonlinear function, it is possible to determine a contour enhancement coefficient appropriate for the contour enhancement processing.

ここで、Ｍ及びＮは、所定の画素領域における注目画素及び周辺画素がＭ×Ｎ画素であるとしたときの値であり、Ｙ（ｉ，ｊ）は、注目画素と周辺画素の位置を座標（ｉ，ｊ）（ｉ＝１〜Ｍ，ｊ＝１〜Ｎ）で表したときの各画素の輝度であり、α（ｉ，ｊ）は、輝度Ｙ（ｉ，ｊ）を有する画素に割り当てられた値であり、輪郭情報量計算部は、以下の式（２）により輪郭情報量Ｄを計算し、

輝度信号補正部は、以下の式（３）により求められる輝度Ｙ_ｃを輪郭強調手段に出力する。

ここで、Ｄ_Ｔは、所定の閾値であり、Ｙ_ｉｎは、注目画素の輝度である。 More preferably, the noise removing means calculates a luminance Y from which noise is removed by the following equation (1):

Here, M and N are values when the target pixel and the peripheral pixels in the predetermined pixel region are M × N pixels, and Y (i, j) is a coordinate position of the target pixel and the peripheral pixels. (I, j) (i = 1 to M, j = 1 to N) is the luminance of each pixel, and α (i, j) is assigned to a pixel having luminance Y (i, j). The contour information amount calculation unit calculates the contour information amount D by the following equation (2),

Luminance signal correction unit outputs the luminance Y _c obtained by the following equation (3) to the edge enhancement unit.

Here, _DT is a predetermined threshold value, and Y _in is the luminance of the pixel of interest.

ここで、Ｄ_１〜Ｄ_４は、輪郭情報量の値であり、ｋ_１〜ｋ_４は、輪郭強調係数の値である。 The predetermined function is given by the following equation (4).

Here, D _{1 to} D ₄ are values of the contour information amount, and k _{1 to} k ₄ are values of the contour enhancement coefficient.

ここで、β（ｉ，ｊ）は、輝度Ｙ（ｉ，ｊ）を有する画素に割り当てられた値である。 Contour detecting unit, by the following equation (5), calculates the weighted is performed luminance Y _e.

Here, β (i, j) is a value assigned to a pixel having luminance Y (i, j).

The contour emphasizing means calculates the luminance Y _out of the target pixel after contour emphasis by the following equation (6).

  According to the contour emphasizing apparatus of the present invention, it is possible to perform contour emphasis by automatically adjusting the degree of contour emphasis while removing noise components based on the amount of contour information.

FIG. 1 is a block diagram illustrating a schematic configuration of an image display system including an outline emphasis device according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a schematic configuration of the contour emphasizing unit according to the embodiment of the present invention. FIG. 3A is a schematic diagram showing the luminance of the pixels used in the contour emphasizing process according to the embodiment of the present invention, and FIGS. It is the schematic which shows the coefficient currently allocated to the pixel. 4A and 4B are graphs used when determining the edge enhancement coefficient in the edge enhancement processing according to the embodiment of the present invention. FIG. 5 is a flowchart of the contour enhancement process according to the embodiment of the present invention.

  Hereinafter, an outline emphasis device according to an embodiment of the present invention will be described with reference to the drawings. In addition, the same number is attached | subjected, when showing the same member over several figures.

  FIG. 1 is a block diagram illustrating a schematic configuration of an image display system including an outline emphasis device according to an embodiment of the present invention. The imaging device 1 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor having a Bayer-type pixel arrangement, and charges an optical image formed by each pixel on the light receiving surface in accordance with the amount of light. And converted into an image signal corresponding to each color of R, G, and B. The converted image signal is amplified by an analog amplifier in an AFE (Analog Front End) 2, then AD converted, and sent to the color conversion unit 3 and the luminance conversion unit 4, respectively.

  The color conversion unit 3 and the luminance conversion unit 4 generate a color signal and a luminance signal based on the image signal output from the AFE 2, respectively. The generated luminance signal is sent to a contour emphasizing unit 5 as a contour emphasizing device. A detailed description of the processing in the contour emphasizing unit 5 will be described later. The color signal output from the color conversion unit 3 and the luminance signal subjected to the enhancement processing by the contour enhancement unit 5 are respectively sent to the signal mixing unit 6, and the signal mixing unit 6 outputs the luminance signal and the color difference signal, that is, YUV. A signal is generated. The luminance signal and the color difference signal are sent from the signal mixing unit 6 to the RGB signal conversion unit 7 and converted into RGB signals. The converted RGB signal is corrected by the gamma correction unit 8 so that the brightness of the image signal becomes natural. Then, the image output unit 9 performs NTSC (National Television System Committee) or PAL (Phase Alternating). The image signal is converted into an image signal conforming to a predetermined standard such as Line) and output to the monitor 10. As a result, the captured image is displayed on the monitor 10.

  FIG. 2 is a block diagram showing a schematic configuration of the contour emphasizing unit 5 in the present embodiment. The luminance signal generated by the luminance conversion unit 4 is sent to the line memory 51 of the contour enhancement unit 5. In this embodiment, it is assumed that contour enhancement processing is performed using an image signal of 3 × 3 pixels. Therefore, the line memory 51 has a capacity capable of storing image signals for three lines of the image sensor 1. The image signal stored in the line memory 51 is sent to a low-pass filter (LPF) 52, a contour information amount calculation unit 53, a contour detection unit 54, and a luminance signal correction unit 56, respectively.

  First, the process of the LPF 52 will be described. 3A to 3C show the luminances of the target pixel and the peripheral pixels in the contour emphasizing process of the contour emphasizing unit 5, and the coefficients assigned to the target pixel and the peripheral pixels. FIG. 3A shows the position of each pixel by coordinates (i, j) (i = 1 to M, j = 1 to N) when the target pixel and the surrounding pixels constitute M × N pixels. The luminance of each pixel is shown. FIG. 3B shows a coefficient α (i, j) assigned to each pixel used in the LPF 52 as will be described later. FIG. 3C shows a coefficient β (i, j) assigned to each pixel, which is used in the contour detection unit 54 as described later. In this embodiment, the edge enhancement process is performed using an image signal of 3 × 3 pixels. Therefore, M = N = 3 in FIGS. In the present embodiment, a pixel having luminance Y (2, 2) is set as a target pixel.

  The LPF 52 outputs a value obtained by multiplying each luminance and each corresponding coefficient using the luminance Y (1,1) to Y (3,3) and the coefficient α (1,1) to α (3,3). To do. As the coefficient values, for example, a moving average filter may be employed to set 1/9 to all the coefficients α (1, 1) to α (3, 3) to make them uniform, or Gaussian. Employing a filter, α (1,1), α (1,3), α (3,1) and α (3,3) are 0.0113, α (1,2) and α 0.0838 can be set for (2,1), α (2,3), and α (3,2), and 0.6196 can be set for α (2,2). The LPF 52 outputs the luminance Y obtained by removing the noise by performing the calculation according to the following expression (7) to the contour information amount calculation unit 53 and the luminance signal correction unit 56.

  Note that, as described above, in the present embodiment, since the target pixel and the peripheral pixels constitute 3 × 3 pixels, the calculation is performed with M = N = 3. In the present embodiment, the following equations are also calculated with M = N = 3.

  The contour information amount calculation unit 53 calculates the contour information amount of the image in the pixel region occupied by the target pixel and the peripheral pixels that are the processing target of the LPF 52. The contour information amount calculation unit 53 removes noise Y (1,1) to Y (3,3) of the target pixel and peripheral pixels sent from the line memory 51 and the luminance Y from which noise sent from the LPF 52 is removed. The contour information amount D is calculated by performing the calculation according to the following equation (8) using

  When D is a small value from the contour information amount D calculated by the contour information amount calculation unit 53, it is considered that noise has occurred in the pixel area to be processed by the LPF 52, and a large value is taken. Can be regarded as including the edge portion of the image in the pixel region. The contour information amount D calculated by the contour information amount calculation unit 53 is sent to the luminance signal correction unit 56 and the contour enhancement coefficient calculation unit 58, respectively.

The contour enhancement coefficient calculator 58 determines the contour enhancement coefficient k with reference to a contour enhancement coefficient look-up table (LUT) 57 based on the contour information amount D calculated by the contour information amount calculator 53. To do. FIG. 4A shows an example of a graph of functions stored in the edge enhancement coefficient LUT 57 in the present embodiment. For convenience of explanation, FIG. 4B shows a graph in which the graph of FIG. The graph of FIG. 4B can be expressed using the following formula (9) with D = D ₁ , D ₂ , D ₃ , and D ₄ as boundaries.

Here, D ₁ , D ₂ , D ₃ , and D ₄ are threshold values of the contour information amount, and k ₁ , k ₂ , k ₃ , and k ₄ are values of the contour enhancement coefficient. The contour enhancement coefficient calculation unit 58 sends the determined contour enhancement coefficient k to the multiplication unit 60.

The contour detection unit 54 performs calculation for detecting the contour of the image in 3 × 3 pixels formed by the target pixel and the peripheral pixels. Contour detection unit 54 performs calculation according to formula (10) below using each of the luminance of the target pixel and peripheral pixels output from the line memory 51, obtains the luminance Y _e weighted is performed for the pixel of interest.

Here, β (i, j) is a contour detection coefficient assigned to each pixel in the region formed by the target pixel and the peripheral pixels. As the value of the coefficient, for example, 1 can be set for β (2, 2) of the target pixel, and −0.125 can be set for the coefficients of other peripheral pixels. 2), 1 for β (1,1), β (1,3), β (3,1) and β (3,3), −1/12, β (1,2) , Β (2,1), β (2,3), and β (3,2) can be weighted by setting −1/6. The contour detection unit 54 performs calculation using a Laplacian filter, a LOG (Laplacian of Gaussian) filter, or the like. The calculation result by the contour detection unit 54 is sent to the multiplication unit 60. Multiplication unit 60 performs multiplication of the coefficient k which is determined by the luminance Y _e and edge enhancement coefficient calculator 58 weight calculated by the contour detection unit 54 is performed, and sends the calculation result to the adder 59.

The luminance signal correction unit 56 determines which one of the luminance Y (2, 2) of the target pixel output from the line memory 51 and the luminance Y of the target pixel output from the LPF 52 using Expression (7) is used as the correction luminance. decide. The luminance signal correction unit 56 compares the contour information amount D output from the contour information amount calculation unit 53 with the threshold value D _T of the contour information amount stored in the noise information amount threshold memory 55, and the following equation ( 11) is used to determine which of the luminance Y _{in the} target pixel output from the line memory 51 and the luminance Y from which noise is output from the LPF 52 is selected as the corrected luminance Y _c . When the contour information amount D takes a value smaller than the threshold value _DT, it is considered that the luminance signal of the target pixel includes more noise information than the contour information. The threshold value _DT is a value that affects the influence of noise removal. That is, the smaller the threshold _DT , the smaller the influence of noise removal, and the edge portion of the image can be held. Further, the larger the threshold value _DT , the greater the influence of noise removal. However, if the value is too large, the edge portion of the image may not be retained and the image may become unclear. Therefore, for example, an appropriate threshold _DT is determined based on the statistics of the information amount of noise components obtained for an image acquired in the image display system.

The value of _Yc determined by the luminance signal correction unit 56 is sent to the addition unit 59. The addition unit 59, by using the calculation result output from the Y _c a multiplication unit 60 which is output from the luminance signal correction section 56, the following equation (12), luminance Y which has been subjected to edge enhancement processing to the pixel of interest _out is calculated and output to the signal mixing unit 6.

FIG. 5 shows a flowchart of the contour emphasis process in the present embodiment. In step S <b> 101, the luminance of the target pixel and the luminance of surrounding pixels converted by the luminance conversion unit 4 are input to the line memory 51. In step S103, the luminance Y from which noise has been removed in the LPF 52 is calculated, the contour information amount D is calculated in the contour information amount calculation unit 53, and the luminance Y _e weighted in the contour detection unit 54 is calculated. In step S105, the contour enhancement coefficient calculator 58 receives the contour information amount D from the contour information amount calculator 53, and determines the contour enhancement coefficient k with reference to the contour enhancement coefficient LUT57.

Subsequently, in step S107, the luminance signal correction unit 56 compares the contour information amount D output from the contour information amount calculation unit 53 with the threshold value D _{T of the} contour information amount read from the noise information amount threshold memory 55, and It is determined which of the brightness Y _in output from the line memory 51 and Y output from the LPF 52 is used for edge enhancement. If the luminance signal correction unit 56 determines that the contour information amount D is smaller than the threshold value D _T , the luminance signal correction unit 56 proceeds to step S109 and adopts Y as the luminance Y _c for performing contour enhancement using Expression (11). If the contour information amount D is equal to or greater than the threshold value _DT , the luminance signal correction unit 56 proceeds to step S111 and employs Y _in as the luminance Y _c for performing contour enhancement using Expression (11). .

After determining the _{Y c} for performing edge enhancement in step S109 or S111, then the process proceeds to step S113, a series of operations performed by the adding unit 59 and the multiplication unit 60 constituting the contour emphasis means, from equation (12) Y _{Find out} .

When _{Y out} is obtained in step S113, _{Y out} is the subsequent processing block as a luminance signal contour is emphasized in step S115, in the present embodiment is output to the signal mixing unit 6. By performing the above-described series of processing of steps S101 to S115 on each pixel of the image sensor 1, a favorable contour emphasis process can be performed on the image signals output from all the pixels of the image sensor 1. Therefore, in the present invention, when the amount of noise information is determined based on the amount of contour information calculated in an area composed of a plurality of pixels, rather than the luminance difference between adjacent pixels, the amount of noise information is larger than the amount of contour information. Since the edge enhancement can be performed using the luminance signal from which noise has been removed, particularly when an image signal having a poor S / N ratio is handled, a pixel adjacent to the target pixel exhibits erroneous luminance due to noise. However, contour enhancement can be performed without enhancing noise.

  The above is the description of the embodiment of the present invention, but the present invention is not limited to the above-described configuration, and various modifications can be made within the scope of the technical idea of the present invention. For example, in the above description, a CMOS or CCD having a Bayer type pixel arrangement is used as the image pickup element 1, but various image pickup elements having an arrangement other than the Bayer type pixel arrangement may be used in accordance with the pixel arrangement. The effect of the present invention can be obtained by changing the capacity stored in the line memory and performing the above calculation on the target pixel and the peripheral pixels.

  Further, in the above description, the 3 × 3 pixels of the same type as the pixel of interest with the pixel of interest as the center are processed, but FIGS. 3A to 3C and equations (7), (8), (10 As shown in (), a configuration may be adopted in which M × N pixels are processed. In this case, the line memory needs to have a capacity enough to store the image signal from the M × N pixels. It should be noted that the coefficients α (i, j) and β (i, j) for each pixel are both set to 1 / M × N, as described above, or a larger value is assigned to a pixel closer to the target pixel. You can also.

3 Outline Enhancement Unit 4 Brightness Conversion Unit 51 Line Memory 52 Low Pass Filter 53 Outline Information Amount Calculation Unit 54 Outline Detection Unit 55 Noise Information Amount Threshold Memory 56 Brightness Signal Correction Unit 57 Outline Enhancement Coefficient Lookup Table 58 Outline Enhancement Coefficient Calculation Unit 59 Addition Part 60 multiplication part

Claims (6)

In an edge emphasis device that performs edge emphasis processing on an image signal output from each pixel of an image sensor,
Noise removing means for removing a noise component included in the luminance of the pixel of interest in a predetermined pixel region of the image sensor;
Contour information amount calculating means for calculating the amount of contour information of the target pixel from the luminance of the target pixel and the peripheral pixels of the target pixel in the predetermined pixel region;
A contour detection unit that weights the luminance of the target pixel by a predetermined filter process;
Contour enhancement coefficient determination means for determining an edge enhancement coefficient for the weighted luminance based on the contour information amount;
When the amount of contour information is smaller than a predetermined threshold, the luminance from which the noise component has been removed by the noise removing unit is output, and when the amount of contour information is equal to or greater than the predetermined threshold, the imaging element A luminance signal correction unit that outputs the luminance of the image signal output from the target pixel;
Using said weighting carried out the luminance and the edge enhancement coefficients, have a, and edge enhancement means for performing edge enhancement processing to the luminance output from the luminance signal correction section,
The contour information amount calculating means calculates the contour information amount D by the following equation (1):

Here, M and N are values when the target pixel and the peripheral pixels in the predetermined pixel region are M × N pixels, and Y (i, j) is the value of the target pixel and the peripheral pixels. The luminance of each pixel when the position is represented by coordinates (i, j) (i = 1 to M, j = 1 to N), and Y is the target pixel from which the noise component has been removed by the noise removing unit Is the brightness of
When the contour information amount is less than a first threshold, the contour enhancement coefficient determining means determines the contour enhancement coefficient to be a larger value as the contour information amount is larger, and the contour information amount is the first threshold value. When the second threshold value is greater than or equal to the second threshold value, the contour enhancement coefficient is determined to be smaller as the contour information amount is larger.
Outline enhancement device. The contour enhancement coefficient determination means determines the contour enhancement coefficient using the contour information amount and a predetermined function,
The contour enhancement means performs a contour enhancement process on the luminance output from the luminance signal correction unit using a result obtained by multiplying the weighted luminance by the contour enhancement coefficient.
The contour emphasizing device according to claim 1. The noise removing means calculates the luminance Y from which the noise component has been removed by the following formula ( 2 ):

Here, α (i, j) is a value assigned to a pixel having luminance Y (i, j).
The luminance signal correction unit outputs luminance Y _c obtained by the following equation (3) to the contour emphasizing unit.

Here, D _T is the predetermined threshold, and Y _in is the luminance of the target pixel.
The contour emphasizing device according to claim 1 or 2. The predetermined function is given by the following equation (4):

Here, D _{1 to} D ₄ are values of the contour information amount, and k _{1 to} k ₄ are values of the contour enhancement coefficient.
The contour emphasizing device according to claim 3 quoting claim 2 . The contour detecting unit, by the following equation (5), calculates the luminance Y _e where the weighting is performed,

Here, β (i, j) is a value assigned to a pixel having luminance Y (i, j).
The contour emphasizing device according to claim 4. The contour emphasizing unit calculates a luminance Y _out of the target pixel after contour emphasis according to the following equation (6).

The contour emphasizing device according to claim 5 .

Images