JP5991090B2 - Image signal processing apparatus and method - Google Patents

Description

  The present invention relates to an image signal processing apparatus and method capable of reducing luminance unevenness caused by discontinuity of image brightness due to flash emission or the like by another imaging device during moving image capturing.

  In recent years, in image pickup apparatuses such as video cameras, many CMOS (Complementary Metal Oxide Semiconductor) sensors are employed as image sensors. Reading of the imaging signal in the CMOS sensor is performed by a rolling shutter system in which the readout timing is sequentially shifted in the line direction by sequentially performing reset, exposure, and readout in units of lines, not at the same timing in all pixels.

  For this reason, when many video cameras and still cameras photograph the same subject at the same time, strong light with a short light emission period due to the flash of another person enters the CMOS sensor during the exposure period of only some lines. There is. In this case, the exposure amount is different in a part of the screen to be imaged, and a bright white band-like area is generated in the screen, resulting in uneven brightness.

  When the image has uneven brightness as described above, for example, as described in Patent Document 1, the image is corrected by replacing the screen including the uneven brightness (high brightness region) with a normal screen in the past. To be done.

JP 2010-135922 A

  For example, in a scene of arrival at a goal in sports, a large number of flashes may emit light. When determining the arrival order, the luminance unevenness due to flash emission is corrected by a method as described in Patent Document 1, and when the subject moves quickly, the temporal position of the image shifts. Therefore, accurate determination cannot be made. Therefore, there is a need for an image signal processing apparatus and method that can correct an image with uneven brightness without shifting the temporal position of the image.

  In order to meet such a demand, an object of the present invention is to provide an image signal processing apparatus and method that can correct an image having luminance unevenness without shifting the temporal position of the image.

In order to solve the above-described problems of the related art, the present invention provides a line for determining a change in luminance based on a sum value of pixels in each line of an image signal or a value obtained by dividing the sum value by a predetermined value. The calculation unit (11) that calculates the luminance value, the first storage unit (12) that stores the line luminance value calculated by the calculation unit for the number of lines in the vertical effective image period, and the calculation unit The line luminance value in the target line of the target frame is compared with the line luminance value in the line at the same position as the target line in the previous frame from the target frame read from the first storage unit. A determination unit (15) for determining the degree of change, and when the degree of change in luminance determined by the determination unit exceeds a predetermined threshold, Only when the correction coefficient generation unit (152) that generates a correction coefficient for correcting the image signal of the target line of the target frame and the degree of change in luminance determined by the determination unit is equal to or less than a predetermined threshold, The write control unit (151) for controlling the line luminance value calculated by the calculation unit to be stored in the first storage unit, and the correction coefficient generation unit so as to match the timing of generating the correction coefficient A second storage unit (13) for delaying the image signal of the target line of the target frame; and multiplying the image signal of the target line of the target frame delayed by the second storage unit by the correction coefficient. An image signal processing apparatus comprising an output multiplication unit (16) is provided.

  In the above image signal processing device, the determination unit divides a line luminance value in the target line of the target frame by a line luminance value in a line at the same position as the target line in a frame past the target frame. It is preferable that the correction coefficient generation unit converts the division value into the correction coefficient based on a characteristic obtained by linearly approximating the division value.

  In the image signal processing apparatus, the correction coefficient generation unit divides a region where the division value exceeds the predetermined threshold into a plurality of regions, and a straight line for converting the division value into the correction coefficient in each region. It is preferable to set the characteristics.

  In addition, in order to solve the above-described problems of the conventional technology, the present invention is for determining a change in luminance using a sum value of pixels in each line of an image signal or a value obtained by dividing the sum value by a predetermined value. And the calculated line luminance value is stored in the first storage unit (12) by the number of lines in the vertical effective image period, and the calculated line luminance value in the attention line of the attention frame is calculated. The degree of change in luminance is determined by comparing the line luminance value in a line at the same position as the target line in the frame prior to the target frame read from the first storage unit, and the determined luminance When the degree of change exceeds a predetermined threshold value, a correction coefficient for correcting the image signal of the target line of the target frame is generated according to the degree of change in luminance, The attention is made so that the calculated line luminance value is stored in the first storage unit only when the degree of the luminance change is equal to or less than a predetermined threshold value, and is matched with the timing of generating the correction coefficient. The image signal of the target line of the frame is delayed by being stored in the second storage unit (13) and read out, and the image signal of the target line of the target frame delayed by the second storage unit is corrected. Provided is an image signal processing method characterized by multiplying a coefficient and outputting the result.

  According to the image signal processing apparatus and method of the present invention, it is possible to correct an image having luminance unevenness without shifting the temporal position of the image.

It is a block diagram which shows the image signal processing apparatus of 1st Embodiment. It is a figure for demonstrating the operation | movement of a CMOS sensor, and the phenomenon when a flash light-emits. It is a figure which shows the example of the flame | frame which the brightness nonuniformity has generate | occur | produced by light emission of flash. It is a timing diagram for demonstrating operation | movement of the image signal processing apparatus of 1st Embodiment. It is a characteristic view of the correction coefficient K produced | generated by the determination part 15 in FIG.

  Hereinafter, an image signal processing apparatus and method according to each embodiment will be described with reference to the accompanying drawings.

<First Embodiment>
Before describing the image signal processing apparatus according to the first embodiment shown in FIG. 1, the operation of the rolling shutter type CMOS sensor and the phenomenon when the flash emits light will be described with reference to FIGS. As shown in FIG. 2, one frame period is 1125 lines (1125H), a vertical effective image period is 1080 lines (1080H), and a vertical blanking period is 45 lines (45H). The exposure time of the CMOS sensor is set to 500H. In FIG. 1, for convenience of illustration, the exposure time of 500H is shown in a state shorter than actual.

  In the first line in the vertical effective image period, a photodiode (not shown) is reset at a position indicated by a broken line 500H before the timing of reading out the signal of one line. Then, exposure (charge accumulation) is performed for 500H period, and the charge accumulated in the photodiode is read out at the read timing of the signal of the first line indicated by the solid line. In the second line, the photodiode is reset with a delay of one line from the first line, and the charge of the second line is read out with a delay of one line.

  Similarly, in the m-th line, the photodiode is reset with a delay of (m−1) lines from the first line, and the charge of the m-th line is read with a delay of (m−1) lines.

  It is assumed that the flash is emitted between the (m) line and the (m + 2) line of the n frame. The exposure period of the signal output from the image sensor to the mth line of the n frame is a 500H period from the (m-500) th line to the (m-1) th line, and is exposed with normal ambient light. n-1) There is no significant difference in luminance between the m-th line signal of the frame and the m-th line signal of the n frame. The signal output from the image sensor in the (m + 1) th line is affected by the flash for the 1H period because the exposure period is from the (m−499) th line to the mth line, and the (n−1) th frame The luminance of the image signal of the (m + 1) th line in the nth frame is higher than that of the image signal of the (m + 1) line.

  In the (m + 2) line, the luminance is higher because of being affected by the flash only for the 2H period, and in the (m + 3) line, the luminance is further increased because of being affected by the flash for the 3H period.

  Thereafter, up to the (m + 500) line is affected by the flash only for the 3H period, the (m + 501) line is affected by the flash for the 2H period, and the (m + 502) line is affected by the flash for the 1H period. Each has a higher luminance than the image signal of (n-1) frames. From the (m + 503) th line onward, the exposure is not affected by the flash and is exposed to normal ambient light, so there is no great difference between the image signal of (n-1) frames and the luminance.

  Therefore, as shown in FIG. 3, the (n−1) frame is a normal image, and the n frame is hatched from the (m + 1) th line to the (m + 502) line, so that the brightness is higher than usual. The image has high luminance unevenness. In the (n + 1) frame, a normal image is obtained again.

  The configuration and operation of the image signal processing apparatus according to the first embodiment that corrects an image signal having uneven luminance as described above will be described. In FIG. 1, m lines of n frames in an image signal are represented by subscripts (n, m). It is determined whether or not the m lines of n frames are affected by the flash, and if the luminance is uneven due to the influence of the flash, it is assumed that it is the target line of the target frame for correcting the luminance unevenness.

  The pixel data of the image signal Sin output from the CMOS sensor and converted into a digital signal is input to the summation unit 11 and the storage unit 13. The summation unit 11 sums the pixel data of the image signal Sin for one line and outputs a sum value SUM. If the image signal Sin input to the summation unit 11 is the image signal Sin (n, m + 1), the summation value SUM output from the summation unit 11 is the summation value SUM (n, m). The sum value SUM that is the summation result by the summation unit 11 is updated in one line cycle.

  The sum value SUM output from the summation unit 11 is input to the storage unit 12, the terminal Ta of the switch 14, and the determination unit 15. The determination unit 15 includes a write control unit 151 and a correction coefficient generation unit 152. The storage unit 12 stores 1080 total values SUM of each line, which is the number of vertical lines in the vertical effective image period. The storage unit 12 updates the total value SUM of the lines only when a write permission pulse WENB described later is input to each line.

  The determination unit 15 includes the sum value SUM (n−1, m) of m lines one frame before the target frame read from the storage unit 12 and the sum value SUM (n, m) output from the summation unit 11. ) Is entered. When the sum value SUM (n−1, m) read from the storage unit 12 is input to the determination unit 15, the switch 14 is connected to the terminal Tb. The determination unit 15 compares the sum value SUM (n-1, m), which is the sum value of the same line before and after one frame, with the sum value SUM (n, m), and the attention line of the attention frame is affected by the flash. It is determined whether or not it has received.

  The switch 14 is connected to the terminal Ta only at the first frame which is the first frame. In the first frame, the sum value SUM one frame before is not output from the storage unit 12. Therefore, the switch 14 is connected to the terminal Ta at the first frame, and the sum value SUM obtained by the summation unit 11 is input to the determination unit 15. In the first frame, the determination unit 15 compares the sum value SUM of the same value, so the determination unit 15 always determines that it is not affected by the flash.

  The correction coefficient generation unit 152 in the determination unit 15 corrects the image signal Sin based on the determination result of whether or not the target line of the target frame is affected by the flash. Is supplied to the multiplication unit 16.

  The storage unit 13 stores pixel data corresponding to the number of horizontal effective pixels in one line, for example, 1920 pixels, and outputs it with a delay of one line. Since the image signal Sin input to the summation unit 11 is the image signal Sin (n, m + 1), the image signal Sin output from the storage unit 13 is the image signal Sin (n, m).

  The total value SUM for one line is obtained in the summation unit 11 after all the pixel data for one line has been input. The determination unit 15 calculates the correction coefficient K (n, m) using the sum value SUM output from the summation unit 11, and the multiplication unit 16 corrects the image signal Sin because the pixel data of each line is input. It will be one line after the start of being done. Therefore, the storage unit 13 delays the image signal Sin by one line.

  That is, the storage unit 13 sets the image signal Sin (n, m) of the target line of the target frame so as to match the timing of generating the correction coefficient K (n, m) for correcting the image signal of the target line of the target frame. Delay.

  The multiplication unit 16 multiplies the image signal Sin (n, m) output from the storage unit 13 by the correction coefficient K (n, m), and outputs a corrected image signal Sout (n, m).

  A specific operation of the image signal processing apparatus according to the first embodiment will be described with reference to FIGS. As shown in FIGS. 4A and 4B, 2200 pixels (2200 clocks (ck)) in one line period, 1920 pixels (1920 ck) in horizontal effective pixels (effective line period), and 280 in horizontal blanking period Assume that the pixel is 280 ck.

  4, (c) is an image signal Sin input to the summation unit 11, (d) is a summation value SUM output from the summation unit 11, (e) is a multiplication coefficient K generated by the determination unit 15, ( f) shows the write permission pulse WENB output from the determination unit 15. 4, (g) shows the image signal Sin output from the storage unit 13, and (h) shows the corrected image signal Sout output from the multiplication unit 16.

  As described with reference to FIG. 2, it is assumed that the flash is emitted during the period from the (m + 1) th line to the (m + 3) th line of the n frame. In FIG. 4, the subscript n indicating n frames is omitted.

Based on the following equation (1), the determination unit 15 adds the sum value SUM (n−1, m) of the m line one frame before the frame of interest ((n−1) frames) and the sum of the frame of interest. A comparison result R (n, m) with SUM (n, m) is calculated. The comparison result R is a value indicating the degree of change in luminance.
R (n, m) = SUM (n, m) / SUM (n-1, m) (1)

  A conversion characteristic for converting the comparison result R into the correction coefficient K as shown in FIG. 5 is set in the correction coefficient generation section 152 in the determination section 15. The correction coefficient generation unit 152 may have a table showing the conversion characteristics shown in FIG. 5, or may have a calculation formula showing the conversion characteristics shown in FIG. In the m-th line, R (n, m) is approximately 1.0 because it is not affected by the flash. When the comparison result R (n, m) is R (n, m) ≦ 1.0 and 1.0 <R (n, m) ≦ TH1, the correction coefficient generation unit 152 multiplies the correction coefficient K (n, m) by 1.0. 16 is output.

  In FIG. 5, TH1 is a threshold value for distinguishing between a state not affected by the flash and a state affected by the flash. If the comparison result R is equal to or less than the threshold value TH1, it is determined that there is no influence from the flash, and there is no need to correct the image signal Sin (n, m) of the target line of the target frame. If the comparison result R exceeds the threshold value TH1, it is determined that the flash is affected, and the image signal Sin (n, m) of the target line of the target frame is corrected.

  In the example shown in FIG. 5, the threshold value TH1 is set to 1.25. When not affected by the flash, R (n, m) is set to approximately 1.0, but the total value SUM varies slightly depending on the movement of the subject and noise. Therefore, in order not to unnecessarily correct the image signal Sin, the correction coefficient K (n, m) is set to 1.0 up to a value slightly larger than 1.0.

  As shown in (f) of FIG. 4, the write control unit 151 in the determination unit 15 sends the sum of m lines to the (n−1) frame value SUM (n−1, m) to the storage unit 12. ), A write permission pulse WENB is output for rewriting to the value SUM (n, m) of n frames.

  Next, in the (m + 1) -th line, the pixel data of the image signal Sin (n, m + 1) of the (m + 1) -th line is sequentially written into the storage unit 13 and at the same time, the image signal of the m-th line from the storage unit 13. Sin (n, m) is read and output to the multiplication unit 16. The multiplier 16 multiplies the image signal Sin (n, m) by the correction coefficient K (n, m) and outputs the result as a corrected image signal Sout (n, m). Since K (n, m) = 1.0 in the m-th line, the multiplication unit 16 outputs the input image signal Sin (n, m) as it is as the corrected image signal Sout (n, m).

  In parallel with this processing, the summation unit 11 sums the image signal Sin of the (m + 1) th line for one line (1920 pixels) to generate a summation value SUM (n, m + 1) and outputs it to the determination unit 15. To do. Similarly to the equation (1), the determination unit 15 adds the sum SUM (n-1, m + 1) of the (m + 1) th line before one frame and the sum SUM (n) of the (m + 1) th line of the frame of interest. , m + 1) and a comparison result R (n, m + 1) are obtained. In the (m + 1) th line, since the flash emission period is included in the exposure period, R (n, m + 1)> 1.0.

  The correction coefficient generation unit 152 determines a correction coefficient K (n, m + 1) corresponding to the magnitude of the comparison result R (n, m + 1) based on the characteristics shown in FIG. Assuming that the comparison result R (n, m + 1) = 2.0, K (n, m + 1) = 0.5 is output to the multiplication unit 16 from FIG. Since the (m + 1) th line of the n frame is affected by the flash, the sum SUM of the (m + 1) th line cannot be used for the determination of the next frame. Therefore, the write control unit 151 does not output the write permission pulse WENB to the storage unit 12 as shown in (f) of FIG.

  Next, in the (m + 2) -th line, the pixel data of the image signal Sin (n, m + 2) of the (m + 2) -th line is sequentially written in the storage unit 13 and at the same time, the image signal Sin ( n, m + 1) is read out and output to the multiplier 16. The multiplier 16 multiplies the image signal Sin (n, m + 1) by the correction coefficient K (n, m + 1) and outputs a corrected image signal Sout (n, m + 1). Since K (n, m + 1) = 0.5 in the (m + 1) th line, the multiplication unit 16 halves the luminance value of the image signal Sin (n, m + 1), and the luminance value is reduced by flash. The image signal Sin (n, m + 1) increased almost twice is corrected and output.

  Here, when the comparison result R = 2.0, the luminance value of the n-frame image signal Sin is twice the luminance value of the (n−1) -frame image signal Sin. By multiplying the image signal Sin of the frame, the luminance value is corrected appropriately. That is, if K = 1 / R, the luminance value is corrected appropriately. However, such reciprocal calculation increases the scale of hardware and the calculation processing time. Therefore, in FIG. 5, the comparison result R is converted into the correction coefficient K by linear approximation.

In the characteristics shown in FIG. 5, the correction coefficient K is set as follows for the areas Ar1 to Ar4.
Area Ar1: R <1.25 K = 1.0 Correction unnecessary area Ar2: 1.25≤R <2.0 1.0≥K> 0.5 Undercorrection area Ar3: 2.0≤R <4.0 0.5≥K> 0.25 Almost appropriate correction area Ar4: 4.0≤RK = 0.25 Incorrect correction

  It is the area Ar3 that is likely to have uneven brightness under actual use conditions, and the uneven brightness is corrected almost appropriately in the area Ar3. Due to the linear approximation, the areas Ar2 and Ar4 are insufficiently corrected, but there is no practical problem.

  In the (m + 3) -th to (m + 502) -th lines, since the flash emission period is included in the exposure period, the comparison result R is R >> 1.0, and the correction coefficient K is K <1.0. Therefore, the image signal Sin is corrected and output. After the (m + 503) th line, the comparison result R is R <1.25, so the image signal Sin is output as it is.

  As described above, the n-frame image signal Sin having luminance unevenness due to flash emission shown in FIG. 3 is output as the corrected image signal Sout in which the luminance unevenness is corrected by the image signal processing device of FIG. Therefore, the quality of the image is improved.

  In the image signal processing apparatus and method of the present embodiment, the n-frame image signal Sin having luminance unevenness due to flash emission is not corrected by the past frame or future frame screen for the n frame, and the luminance unevenness. The luminance value of the portion where the occurrence is generated is corrected. Therefore, according to the image signal processing apparatus and method of the present embodiment, it is possible to correct an image having uneven brightness without shifting the temporal position of the image.

  By the way, in this embodiment, the summation unit 11 obtains a summation value SUM of pixels for one line, and the determination unit 15 compares the summation value SUM of the same line in the frame one frame before the attention frame with the determination frame 15. It is determined whether or not it is affected by the flash, but it is compared with the average value obtained by dividing the total value SUM by the number of pixels or the value obtained by dividing the total value SUM by a predetermined constant, for example, a power of 2. May be. What is necessary is just to use the value which can detect the difference in illumination light conditions of the attention frame and the frame one frame before the attention frame.

  That is, a calculation unit may be provided that calculates a sum value of pixels in each line of the image signal Sin or a value obtained by dividing the sum value by a predetermined value as a line luminance value for determining a change in luminance. The summation unit 11 is an example of a calculation unit.

Second Embodiment
The storage unit 12 in FIG. 1 is configured by a plurality of frames (P frames), and in the determination unit 15, the average value of the sum value SUM in the same line of the P frame or the maximum value and the minimum value of the sum value SUM in the P frame The average value of the remaining frames excluding the value is calculated. These average values are set as an average total value SUMave of a plurality of frames. The determination unit 15 compares the sum value SUM (n, m) of the frame of interest with the average sum value SUMave to determine whether or not it is affected by flash.

  By setting the value of TH1 in FIG. 5 to about 1.5, it is possible to prevent excessive correction from being applied to noise.

<Third Embodiment>
The capacity of the storage unit 13 in FIG. 1 is set for a plurality of lines (Q lines), and the storage unit 13 outputs the image signal Sin with a delay of Q lines. The correction coefficient K generated by the determination unit 15 is stored for Q lines in a Q-stage shift register, and a low-pass filter is applied to the Q correction coefficients K. In this way, it is possible to prevent excessive correction from being applied to noise in line units.

<Fourth embodiment>
It is possible to combine the configuration of the second embodiment and the configuration of the third embodiment.

  The present invention is not limited to the embodiments described above, and various modifications can be made without departing from the scope of the present invention. Although FIG. 1 shows an example in which the image signal processing device is configured by hardware, it can also be configured by software (computer program). It is also possible to configure the image signal processing apparatus by appropriately combining hardware and software.

11 Summation part (calculation part)
12, 13 Storage unit 14 Switch 15 Determination unit 16 Multiply unit 151 Write control unit 152 Correction coefficient generation unit

Claims (4)

A calculation unit that calculates a sum value of pixels in each line of the image signal or a value obtained by dividing the sum value by a predetermined value as a line luminance value for determining a change in luminance;
A first storage unit that stores the line luminance values calculated by the calculation unit for the number of lines in the vertical effective image period;
The line luminance value in the target line of the target frame calculated by the calculation unit, and the line luminance value in the line at the same position as the target line in the frame past the target frame read out from the first storage unit; And a determination unit that determines the degree of change in luminance,
When the degree of change in luminance determined by the determination unit exceeds a predetermined threshold value, a correction coefficient for correcting the image signal of the target line of the target frame is generated according to the degree of change in luminance. A correction coefficient generator,
A write control unit that controls to store the line luminance value calculated by the calculation unit in the first storage unit only when the degree of change in luminance determined by the determination unit is equal to or less than a predetermined threshold;
A second storage unit that delays the image signal of the target line of the target frame so that the correction coefficient generation unit matches the timing of generating the correction coefficient;
A multiplying unit that multiplies the image signal of the target line of the target frame delayed by the second storage unit by the correction coefficient, and outputs the result.
An image signal processing apparatus comprising: The determination unit indicates a degree of change in luminance by dividing a line luminance value of the target line of the target frame by a line luminance value of a line at the same position as the target line of the frame prior to the target frame. Value and
The image signal processing apparatus according to claim 1, wherein the correction coefficient generation unit converts the division value into the correction coefficient based on a characteristic obtained by linearly approximating the division value.   The correction coefficient generation unit divides an area where the division value exceeds the predetermined threshold into a plurality of areas, and sets a characteristic of a straight line that converts the division value into the correction coefficient in each area. The image signal processing apparatus according to claim 2. A sum value of pixels in each line of the image signal or a value obtained by dividing this sum by a predetermined value is calculated as a line brightness value for determining a change in brightness,
The calculated line luminance value is stored in the first storage unit by the number of lines in the vertical effective image period,
The calculated line luminance value of the target line of the target frame is compared with the line luminance value of the line at the same position as the target line of the past frame read from the first storage unit. , Determine the degree of brightness change,
If the determined brightness change exceeds a predetermined threshold, a correction coefficient for correcting the image signal of the target line of the target frame is generated according to the brightness change,
Control is made to store the calculated line luminance value in the first storage unit only when the determined degree of change in luminance is equal to or less than a predetermined threshold,
In order to match the timing to generate the correction coefficient, the image signal of the target line of the target frame is delayed by being stored in a second storage unit and read out,
An image signal processing method comprising: multiplying the image signal of the target line of the target frame delayed by the second storage unit by the correction coefficient, and outputting the result.

Images