JP5088361B2 - Image processing apparatus and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus that processes an image signal, and an imaging apparatus having this function, and in particular, an image processing apparatus and an imaging apparatus suitable for processing an image signal captured by an XY address scanning type solid-state imaging device. About.

  When shooting a subject with a video camera under the illumination of a blinking light source such as a fluorescent lamp that is turned on by a commercial AC power supply, the captured image is taken due to the difference between the luminance change (light quantity change) frequency of the light source and the vertical synchronization frequency of the camera. A change in light and dark over time, so-called fluorescent lamp flicker, occurs on the top. In particular, when an XY address scanning type imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) type image sensor is used, the exposure timing for each horizontal line is different, and therefore flicker on the captured image is periodic in the vertical direction. It is observed as a striped pattern due to variations in brightness level or hue.

  As a method for removing such flicker components from the captured image signal, a correction method based on the relationship between the shutter speed and the flicker level (referred to as a shutter correction method) and a flicker waveform are detected. A method of applying the inverse waveform to the image signal as a correction gain (referred to as a gain correction method) is known. Of these, the flicker reduction method using the gain correction method detects the flicker frequency spectrum by analyzing the change in the signal level of the image signal, and corrects the signal level of the image signal based on the amplitude value of the spectrum. There was a method (for example, refer patent document 1).

FIG. 21 is a diagram for schematically explaining a conventional flicker detection procedure.
As shown in FIG. 21, the flicker detection procedure disclosed in Patent Document 1 samples one period of the flicker waveform while processing the input image signal into an appropriate shape (step S11), and the sampling data is subjected to discrete Fourier transform. By performing transformation (DFT: Discrete Fourier Transform), the frequency spectrum of the flicker component having one period of the flicker waveform as a fundamental wave is calculated (step S12), and the flicker waveform is estimated using only the low-order term of the spectrum. It is mainly composed of three steps (step S13).

  Here, in step S11 for sampling the flicker waveform, specifically, the influence of the pattern is reduced by sequentially integrating the image signal, for example, for each line in the horizontal direction. Further, when performing DFT in step S12, an average value of integral values over a plurality of fields is obtained, and the integral value is normalized by the average value so that the luminance change and color change due to flicker in the screen area are matched. ing. By such processing, the flicker waveform can be accurately detected regardless of the subject and the image signal level.

  By the way, in recent years, the number of pixels of an image sensor mounted on a video camera or the like is rapidly increasing, and the number of horizontal lines has reached several hundred to several thousand. For this reason, if all the integral values of each line are used when sampling the flicker waveform for one period, the circuit scale of a memory or a DFT arithmetic circuit for temporarily holding those integral values becomes large. .

  On the other hand, the flicker waveform period of the fluorescent lamp is less than one vertical synchronization period in, for example, the NTSC system (National Television Standards Committee), and the flicker appears as several stripes on one screen. For this reason, considering from the sampling theorem, the number of sampling points by the L line corresponding to the flicker waveform for one period is redundant. For example, it is sufficient to detect about several tens of points (for example, 64 points) of the flicker waveform for one period. Accuracy is obtained. Therefore, in the actual flicker detection, the circuit scale is reduced by sampling the flicker waveform for one period by decimation in the vertical direction.

FIG. 22 is a diagram for schematically explaining the flicker detection procedure when the sampled data is thinned out.
In FIG. 22, first, similarly to step S11 of FIG. 21, one cycle of the flicker waveform, that is, L lines is sampled from the input image signal (step S21). Next, sampling data for L points are thinned out and output to L1 points satisfying L >> L1 (step S22). As the thinning process, for example, a method of simply outputting one by one at a predetermined interval or outputting one data by an operation such as LPF (Low Pass Filter) based on a certain number of data can be employed. Thereafter, as in the case of FIG. 21, DFT processing is performed on the sampling data at point L1 (step S23), and the flicker waveform is estimated from the result of the frequency analysis (step S24). By such processing, it is possible to reduce the capacity of a memory that holds sampling data for executing DFT.

JP 2004-222228 A (paragraph numbers [0072] to [0111], FIG. 4)

  By the way, in the above-described flicker detection method, the flicker waveform can be accurately detected by setting the sampling unit of the flicker waveform to exactly one cycle based on the sampling theorem. However, when sampling is performed by thinning lines at a constant rate, if the value of L / L1 does not become an integer, it is impossible to accurately sample the same period as one cycle of the flicker waveform. For this reason, the frequency spectrum sequence obtained from the sampling value in such a case is different from the original flicker waveform obtained by Fourier series expansion with a sine wave of one cycle, and a detection error occurs.

  In order to prevent such a detection error, it is necessary to design a circuit so as to be an optimum thinning unit according to the number of lines of the image sensor to be used. However, in recent years, there has been a demand for a circuit that can be used in common for various products for the purpose of reducing development costs, or that can easily cope with future specification changes. There are similar requirements. That is, there has been a demand for a flicker detection circuit capable of preventing the detection error as described above without greatly changing the circuit configuration even for products having different numbers of pixels of the image sensor.

  The present invention has been made in view of these points, and provides a highly versatile image processing apparatus that can remove flicker components of an image captured by an XY address scanning type solid-state imaging device with high accuracy. With the goal.

  Another object of the present invention is to provide an imaging apparatus capable of removing flicker components of an image captured by an XY address scanning type solid-state imaging device with high accuracy.

In the present invention, in order to solve the above-described problem, an integration unit that integrates an input image signal in units of one horizontal synchronization period or more, an integration value by the integration unit, or a difference value between integration values in adjacent fields or frames is normalized. Normalization means to generate and data corresponding to a predetermined number of sampling positions defined in advance for a predetermined period of flicker are generated by interpolation based on the integral value or difference value after normalization by the normalization means Then, the interpolation processing means for outputting data corresponding to the sampling position as an image signal, the frequency analysis means for extracting the spectrum by analyzing the image signal generated by the interpolation processing means, and the frequency analysis means Flicker estimation means for estimating flicker components from spectrum, and flicker components estimated by flicker estimation means A correction flicker component generation unit that generates a correction flicker component having the same phase and the same number of lines as the input image signal, and an image correction unit that removes the flicker component from the input image signal by the correction flicker component. , An image processing apparatus is provided.

In such an image processing apparatus, the input image signal is integrated by a unit of one horizontal synchronization period or more by the integration means, and the integration value by the integration means or the difference value of the integration values in adjacent fields or frames is normalized by the normalization means. It becomes. Interpolation means generates data corresponding to a predetermined number of sampling positions defined in advance for a period of a predetermined period of flicker by interpolating based on the integral value or difference value after normalization by the normalization means, An image signal which is data corresponding to the sampling position is generated. Thereafter, the spectrum of the image signal from the interpolation processing means is extracted by the frequency analysis means, and the flicker component is estimated from the extracted spectrum by the flicker estimation means. Then, a correction flicker component is generated by the correction flicker component generation means, and the flicker component is removed from the input image signal by the image correction means.

According to the image processing apparatus of the present invention, it is possible to stably improve the detection performance of the flicker component. Therefore, a highly versatile image processing apparatus with improved flicker removal performance can be realized.

1 is a block diagram illustrating a main configuration of an imaging apparatus according to a first embodiment of the present invention. It is a figure for demonstrating flicker. It is a block diagram which shows the internal structure of the flicker reduction part which concerns on 1st Embodiment. It is a figure for demonstrating the example of the thinning-out process of sampling data. It is a figure which shows typically the mode of the thinning-out process when a sampling period corresponds with 1 period of flicker waveforms. It is a figure which shows typically the mode of the thinning-out process when a sampling period is less than 1 period of a flicker waveform. It is a figure which shows typically the mode of the thinning-out process when a sampling period exceeds one period of flicker waveforms. It is a figure which shows typically the mode of the thinning-out process when the number of data points used is reduced so that a sampling period may correspond to 1 period of flicker waveforms. It is a figure for demonstrating the DFT process in case a sampling period does not correspond with one period of flicker waveforms. It is a block diagram which shows the 1st structural example of an integration process part. It is a figure for demonstrating operation | movement of the integration process part of FIG. It is a figure which shows the internal structural example of a V direction thinning-out calculating part. It is a block diagram which shows the 2nd structural example of an integration process part. It is a figure for demonstrating operation | movement of the integration process part of FIG. It is a block diagram which shows the internal structure of the flicker reduction part which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the internal structure of the flicker reduction part which concerns on the 3rd Embodiment of this invention. It is a figure for demonstrating the interpolation calculation by an estimated component interpolation process part. It is a block diagram which shows the internal structure of the flicker reduction part which concerns on the 4th Embodiment of this invention. It is a block diagram which shows the internal structure of the flicker reduction part which concerns on the 5th Embodiment of this invention. It is a figure for demonstrating the interpolation calculation by the estimated component interpolation process part of FIG. It is a figure for demonstrating schematically the conventional flicker detection procedure. It is a figure for demonstrating roughly the flicker detection procedure in the case of thinning out the sampled data.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[First Embodiment]
<System configuration>
FIG. 1 is a block diagram showing a main configuration of the imaging apparatus according to the first embodiment of the present invention.

  1 includes an optical block 11, a driver 11a, a CMOS image sensor (hereinafter abbreviated as a CMOS sensor) 12, a timing generator (TG) 12a, an analog front end (AFE) circuit 13, and a camera processing circuit 14. , A system controller 15, an input unit 16, a graphic I / F (interface) 17, and a display 17a.

  The optical block 11 includes a lens for condensing light from the subject on the CMOS sensor 12, a drive mechanism for moving the lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. The driver 11 a controls driving of each mechanism in the optical block 11 based on a control signal from the system controller 15.

  In the CMOS sensor 12, a plurality of pixels including a photodiode (photogate), a transfer gate (shutter transistor), a switching transistor (address transistor), an amplification transistor, a reset transistor (reset gate), etc. are two-dimensionally formed on a CMOS substrate. And a vertical scanning circuit, a horizontal scanning circuit, an image signal output circuit, and the like are formed. The CMOS sensor 12 is driven based on a timing signal output from the TG 12a, and converts incident light from the subject into an electrical signal. The TG 12 a outputs a timing signal under the control of the system controller 15.

  The AFE circuit 13 is configured, for example, as one IC (Integrated Circuit), and improves the S / N (Signal / Noise) ratio with respect to the image signal output from the CMOS sensor 12 by CDS (Correlated Double Sampling) processing. The sample is held so as to be maintained, the gain is controlled by AGC (Auto Gain Control) processing, A / D conversion is performed, and a digital image signal is output. Note that the circuit for performing the CDS process may be formed on the same substrate as the CMOS sensor 12.

  The camera processing circuit 14 is configured as, for example, one IC, and performs various camera signal processes such as AF (Auto Focus), AE (Auto Exposure), and white balance adjustment on the image signal from the AFE circuit 13, or a part of the process. Execute. In the present embodiment, in particular, a flicker reducing unit 20 is provided that reduces the flicker signal component generated on the screen during imaging under a fluorescent lamp from the image signal.

  The system controller 15 is a microcontroller composed of, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and by executing a program stored in the ROM or the like, Each part of this imaging apparatus is controlled in an integrated manner.

  The input unit 16 includes various operation keys such as a shutter release button, a lever, a dial, and the like, and outputs a control signal according to an input operation by the user to the system controller 15.

  The graphic I / F 17 generates an image signal to be displayed on the display 17a from the image signal supplied from the camera processing circuit 14 via the system controller 15, and supplies this signal to the display 17a to display an image. Let The display 17a is composed of, for example, an LCD (Liquid Crystal Display), and displays a camera-through image being captured, a reproduced image based on data recorded on a recording medium (not shown), and the like.

  In this image pickup apparatus, the signals received and photoelectrically converted by the CMOS sensor 12 are sequentially supplied to the AFE circuit 13, subjected to CDS processing and AGC processing, and then converted into digital signals. The camera processing circuit 14 performs image quality correction processing on the digital image signal supplied from the AFE circuit 13, and finally converts the digital image signal into a luminance signal (Y) and a color difference signal (RY, BY) and outputs the result.

  The image data output from the camera processing circuit 14 is supplied to the graphic I / F 17 via the system controller 15 and converted into a display image signal, whereby a camera-through image is displayed on the display 17a. Further, when image recording is instructed to the system controller 15 by a user input operation from the input unit 16, the image data from the camera processing circuit 14 is supplied to an encoder (not shown), and a predetermined compression encoding process is performed. Is recorded on a recording medium (not shown). At the time of recording a still image, image data for one frame is supplied from the camera processing circuit 14 to the encoder, and at the time of recording a moving image, the image data processed by the camera processing circuit 14 is continuously transmitted to the encoder. Supplied.

<Basic procedure for flicker reduction processing>
FIG. 2 is a diagram for explaining flicker.
Flicker occurs when shooting is performed under a blinking light source such as a fluorescent lamp. When an image is captured by an XY address scanning type imaging device such as a CMOS sensor, the vertical period is as shown in FIG. It is observed as a change in luminance level and hue. FIG. 2A shows a state in which flicker appears as a bright and dark stripe pattern when the subject is uniform. In FIG. 2B, such light and dark repetitions are represented as waveforms (flicker waveforms).

  For example, a fluorescent lamp with a commercial alternating current power supply with a frequency of 50 Hz has a blinking frequency of 100 Hz. Therefore, in an NTSC video signal with a field frequency of 60 Hz, if the number of readout lines per field including the vertical blanking period is M, The number L of lines corresponding to the flicker waveform for one cycle is (M × 60/100) lines. In addition, such periodic fluctuation in one field occurs 100/60 = 1.66 period. That is, such a periodic variation is repeated every three fields. In the following description, it is assumed that flicker occurs under such conditions.

FIG. 3 is a block diagram illustrating an internal configuration of the flicker reducing unit according to the first embodiment.
The flicker reduction unit 20 detects an image signal, normalizes the detection value, outputs a normalized integral value calculation unit 110, performs a DFT process on the normalized detection value, and a DFT spectrum. A flicker generation unit 130 that estimates a flicker component from the analysis result, and a calculation unit 140 that executes a calculation for removing the estimated flicker component from the image signal are provided. In addition, the normalized integral value calculation unit 110 includes an integration processing unit 111, an integration value holding unit 112, an average value calculation unit 113, a difference calculation unit 114, and a normalization processing unit 115.

  The integration processing unit 111 is a block that samples a flicker waveform for one period by integrating an input image signal. The integration processing unit 111 includes a line integrator 210 and a thinning processing unit 220. The line integrator 210 integrates the input image signal for each line. As will be described later, the thinning processing unit 220 thins out and outputs the integration result for L lines corresponding to one cycle of the flicker waveform to a predetermined number of sampling points L1 (L >> L1).

  The integral value holding unit 112 temporarily holds integral values for two fields. The average value calculation unit 113 averages the integral values in three consecutive fields. The difference calculation unit 114 calculates the difference between the integral values in two consecutive fields. The normalization processing unit 115 normalizes the calculated difference value.

  The DFT processing unit 120 performs DFT on the normalized difference value and performs frequency analysis to estimate the flicker component amplitude and initial phase. The flicker generation unit 130 calculates a correction coefficient indicating the ratio of flicker components included in the image signal from the estimated value obtained by frequency analysis. The calculation unit 140 performs a calculation for removing the flicker component from the image signal based on the calculated correction coefficient.

  Note that at least a part of the processing by each of the blocks may be executed by software processing in the system controller 15. Further, in the imaging apparatus according to the present embodiment, the processing by the block shown in FIG. 3 is executed for each of the luminance signal and color difference signal constituting the image signal. Alternatively, it may be executed for at least the luminance signal, and may be executed for the color difference signal and each color signal as necessary. The luminance signal may be executed at the stage of the color signal before being synthesized with the luminance signal, and the process at this stage of the color signal is executed at any stage of the color signal by the primary color or the color signal by the complementary color. May be. When these color signals are executed, processing by the block shown in FIG. 3 is executed for each color signal.

Hereinafter, flicker detection and reduction processing will be described with reference to FIG.
In general, the flicker component is proportional to the signal intensity of the subject. Therefore, if an input image signal (RGB primary color signal or luminance signal before flicker reduction) in an arbitrary sampling period n and an arbitrary pixel (x, y) for a general subject is In ′ (x, y), In ′ (X, y) is expressed by the following equation (1) as a sum of a signal component not including a flicker component and a flicker component proportional to the signal component.
In ′ (x, y) = [1 + Γn (y)] × In (x, y) (1)
Here, In (x, y) is a signal component, Γn (y) × In (x, y) is a flicker component, and Γn (y) is a flicker coefficient. One horizontal period is sufficiently shorter than the light emission period (1/100 second) of the fluorescent lamp, and the flicker coefficient can be regarded as constant in the same line of the same field, so the flicker coefficient is represented by Γn (y).

  In order to generalize Γn (y), it is described in a form expanded to a Fourier series as shown in the following equation (2). Thus, the flicker coefficient can be expressed in a format that covers all the light emission characteristics and afterglow characteristics that differ depending on the type of fluorescent lamp.

  In equation (2), λ0 is the wavelength of the flicker waveform, and when the number of read lines per field is M, it corresponds to L (= M × FPS / 100) lines. ω0 is a normalized angular frequency normalized by λ0.

  γm is the amplitude of each flicker component (m = 1, 2, 3,...). Φm, n indicates the initial phase of each next flicker component, and is determined by the light emission period (1/100 second) of the fluorescent lamp and the exposure timing. However, since Φm, n has the same value every three fields, the difference of Φm, n from the previous field is expressed by the following equation (3).

In the flicker reduction unit 20 shown in FIG. 3, first, in order to reduce the influence of the picture for flicker detection, the integration processing unit 111 shows the input image signal In ′ (x, y) as shown in Expression (4). Are integrated in units of one line in the horizontal direction of the screen to calculate an integrated value Fn (y). However, the integral value Fn (y) output here is an output obtained by thinning out the integral value for the L line to a certain L1 line. Note that α _n (y) in the equation (4) is an integral value over one line of the signal component In (x, y) as represented by the equation (5).

  The integration value Fn (y) output from the integration processing unit 111 is temporarily stored in the integration value holding unit 112 for flicker detection in subsequent fields. The integral value holding unit 112 is configured to be able to hold integral values for at least two fields.

By the way, if the subject is uniform, the integral value α _n (y) of the signal component In (x, y) becomes a constant value, so the integral value Fn (y) of the input image signal In ′ (x, y). It is easy to extract the flicker component α _n (y) × Γn (y) from However, in a general subject, α _n (y) also includes an m × ω0 component, so that a luminance component and a color component as a flicker component and a luminance component and a color component as a signal component of the subject itself are separated. It is not possible to extract purely flicker components. Further, since the flicker component of the second term is very small with respect to the signal component of the first term of the equation (4), the flicker component is almost buried in the signal component.

Therefore, the flicker reducing unit 20 uses integral values in three consecutive fields in order to remove the influence of α _n (y) from the integral value Fn (y). That is, in this example, when the integral value Fn (y) is calculated, the integral value Fn_1 (y) of the same line one field before and the integral value Fn_2 (y) of the same line two fields before are calculated from the integral value holding unit 112. ) Is read, and the average value calculation unit 113 calculates the average value AVE [Fn (y)] of the three integrated values Fn (y), Fn_1 (y), and Fn_2 (y).

Here, if the subjects in the period of three consecutive fields can be regarded as almost the same, α _n (y) can be regarded as the same value. This assumption is not a problem in practice if the movement of the subject is sufficiently small over a period of three fields. Further, calculating the average value of the integral values in the three consecutive fields is based on the relationship of Expression (3), and adds the signals whose phases of the flicker components are sequentially shifted by (−2π / 3) × m. As a result, the flicker component is canceled out. Therefore, the average value AVE [Fn (y)] is expressed by the following equation (6).

  However, the above explanation is about the case where the average value of the integral values in three consecutive fields is calculated on the assumption that the approximation of the above equation (7) holds, but when the subject motion is large, the equation (7) The approximation of 7) does not hold. However, in such a case, by setting the number of consecutive fields related to the averaging process to a multiple of 3, the influence of motion can be reduced by the low-pass filter action in the time axis direction.

  The flicker reduction unit 20 in FIG. 3 has a configuration in the case where the approximation of Expression (7) is established. In this example, the difference calculation unit 114 further calculates the integration value Fn (y) in the field from the integration processing unit 111 and the integration value Fn_1 (y) one field before from the integration value holding unit 112. The difference is calculated, and a difference value Fn (y) −Fn_1 (y) expressed by the following equation (8) is calculated. Note that equation (8) also assumes that the approximation of equation (7) holds.

  Further, in the flicker reduction unit 20 of FIG. 3, the normalization processing unit 115 uses the difference value Fn (y) −Fn — 1 (y) from the difference calculation unit 114 as the average value AVE [Fn ( Normalize by dividing by y)].

  The normalized difference value gn (y) is expanded as shown in Expression (9) by the above-mentioned Formula (6), Expression (8) and the trigonometric product product formula. It is represented by (10). In addition, | Am | and θm in the equation (10) are represented by the equations (11) and (12), respectively.

  Note that the difference value Fn (y) −Fn — 1 (y) remains affected by the signal strength of the subject, and therefore the level of luminance change and color change due to flicker differs depending on the region. However, by normalizing as described above, the luminance change and color change due to flicker can be matched to the same level over the entire region.

  Here, | Am | and θm represented by the equations (11) and (12) are the amplitude and initial phase of each spectrum of the normalized difference value gn (y), respectively. If the subsequent difference value gn (y) is Fourier-transformed to detect the amplitude | Am | and the initial phase θm of each spectrum, the following equation (2) is obtained by the following equations (13) and (14). Further, the amplitude γm and the initial phase Φm, n of each subsequent flicker component can be obtained.

  Therefore, in the flicker reduction unit 20 of FIG. 3, the DFT processing unit 120 corresponds to one cycle (L1 line) of the flicker waveform of the normalized difference value gn (y) from the normalization processing unit 115. Discrete Fourier transform the data.

  If the DFT operation is DFT [gn (y)] and the DFT result of order m is Gn (m), the DFT operation is expressed by the following equation (15). However, W in Formula (15) is represented by Formula (16). In this way, by setting the data length of the DFT operation to one cycle of the flicker waveform (for the L1 line), it is possible to directly obtain a discrete spectrum group that is an integral multiple of the normalized angular frequency ω0. Processing can be simplified.

  Moreover, the relationship between Formula (11) and (12) and Formula (15) is represented by following Formula (17) and (18) by the definition of DFT, respectively.

  Therefore, from the equations (13), (14), (17), and (18), the amplitude γm and the initial phase Φm, n of each next flicker component can be obtained by the following equations (19) and (20).

  The DFT processing unit 120 first extracts a spectrum by the DFT calculation defined by Expression (15), and then calculates the amplitude γm and the initial phase Φm of each next flicker component by the calculations of Expressions (19) and (20). n is estimated.

  In general, fast Fourier transform (FFT) is used as Fourier transform in digital signal processing. However, since FFT requires that the data length be a power of 2, in the present embodiment, frequency analysis is performed by DFT, and data processing is simplified accordingly. Under actual fluorescent lamp illumination, flicker components can be sufficiently approximated even if the order m is limited to several orders, so that it is not necessary to output all data in the DFT calculation. Therefore, there is no disadvantage in terms of calculation efficiency compared to FFT.

  The flicker generation unit 130 uses the estimated value of the amplitude γm and the initial phase Φm, n by the DFT processing unit 120 to execute the calculation process of the above equation (2), and the flicker coefficient Γn that correctly reflects the flicker component (Y) is calculated. Even in the calculation processing of equation (2), under actual fluorescent lamp illumination, the total order is not infinite, but limited to a predetermined order, for example, the second order, and the higher order processing is omitted. However, the flicker component can be approximated sufficiently in practice.

Here, said Formula (1) can be deform | transformed like following Formula (21). Based on this equation (21), the arithmetic unit 140 suppresses the flicker component by adding “1” to the flicker coefficient Γn (y) from the flicker generating unit 130 and then dividing the image signal by this added value. To do.
In (x, y) = In ′ (x, y) / [1 + Γn (y)] (21)
According to the above flicker detection / reduction processing, the flicker component is completely buried in the signal component with the integrated value Fn (y), even in a region such as a black background portion where the flicker component is minute or a portion with low illuminance, By calculating the difference value Fn (y) -Fn_1 (y) and normalizing it with the average value AVE [Fn (y)], the flicker component can be detected with high accuracy.

  Further, in calculating the flicker coefficient Γn, since the order can be limited to several orders, flicker detection can be performed with high accuracy by a relatively simple process. Note that estimating the flicker component from the spectrum up to an appropriate order approximates the normalized difference value gn (y) without completely reproducing it. Even if a discontinuous part occurs in the difference value gn (y) after conversion, the flicker component of that part can be estimated with high accuracy.

  In the above process, the difference value Fn (y) −Fn — 1 (y) is normalized by the average value AVE [Fn (y)], thereby effectively ensuring a finite calculation accuracy. However, for example, when the required calculation accuracy can be satisfied, the integral value Fn (y) may be directly normalized by the average value AVE [Fn (y)].

  Further, normalization may be performed with the integral value Fn (y) instead of the average value AVE [Fn (y)]. In this case, even if the flicker waveform does not have repeatability for each of a plurality of screens due to the relationship between the flicker waveform cycle and the screen rate, the flicker can be detected with high accuracy and its components can be reduced.

<Problem of thinning process>
Next, the thinning process of the sampling data (that is, the integration value in the integration processing unit 111) in the flicker reduction unit 20 will be described.

  In the flicker reduction unit 20 shown in FIG. 3, at least two fields of the integration value Fn (y) output from the integration processing unit 111 are accumulated in the integration value holding unit 112 for calculating the average value. The storage capacity of the integrated value holding unit 112 increases as the number of sampling points increases. Further, in the DFT processing unit 120, the calculation amount and necessary parameters (such as a twiddle factor) increase dramatically according to the number of data points used for the calculation, so that this DFT processing can be performed by either hardware or software. Obviously, as the number of sampling points increases, the circuit scale and the required computing power increase. Therefore, from the viewpoint of the system scale, it is desirable that the number of sampling points by the integration processing unit 111 is small.

  On the other hand, from the viewpoint of flicker detection accuracy, generally, as the number of sampling points is larger, more accurate signal processing is possible, and the above-described flicker detection method does not change this. However, in reality, since the frequency of the flicker waveform is sufficiently small with respect to the sampling interval for each line, considering the sampling theorem, the sampling data for the L line corresponding to the flicker waveform for one period is redundant as a point. is there. In particular, since the number M of lines of an image sensor mounted on a recent image pickup apparatus is on the order of hundreds to thousands, the number L of lines corresponding to one cycle of the flicker waveform is also in the same order. On the other hand, for example, when an image is captured by the NTSC system with a field frequency of 60 Hz under fluorescent lamp illumination with a commercial AC power supply with a frequency of 50 Hz, the flicker waveform is 1.666 stripes on one screen. In the flicker detection described above, sufficient detection accuracy can be obtained if, for example, about several tens of points (for example, 64 points) of a flicker waveform for one cycle can be sampled.

  Therefore, the integration processing unit 111 reduces the circuit scale of the subsequent stage by thinning and outputting the sampling data for L lines from the line integrator 210 as the sampling data of the L1 point where L >> L1. However, as L1 is naturally larger, the detection accuracy is improved. Therefore, it is desirable to select a value as large as possible within the range where the circuit installation area, the manufacturing cost, and the like are allowed.

  Next, problems when sampling data is thinned will be described. Here, from the viewpoint of the above flicker detection algorithm, the relationship between the number of sampling points and the detection accuracy is considered.

  As a general property of DFT processing, a discrete spectrum sequence that appears when DFT processing is performed on a data sequence obtained by sampling an arbitrary waveform at an X point is a sine whose sampling waveform is “one cycle at the X point. It is known that a wave (cosine wave) corresponds to a component of each order when Fourier series expansion is performed with a fundamental wave (m = 1) ”.

  Here, as a simplest example, when considering the case where the integration value for L lines integrated by the line integrator 210 of the integration processing unit 111 is passed to the subsequent stage as it is, the DFT processing unit 120 converts the sampling data of the L point to 1 DFT processing is performed as a period, and the flicker generation unit 130 estimates a flicker waveform using a spectrum of a low-order term (for example, only m = 1, 2). That is, in this example, since the L line corresponding to one cycle of the flicker waveform is sampled at the L point, each order spectrum output by the DFT is “the sine wave for one cycle of the flicker waveform as it is. This is equivalent to the component of each order when the Fourier series is expanded in a sequence. ”By extracting only the spectrum sequence of the appropriate order and writing it down in the manner of Fourier series expansion, the flicker waveform can be accurately obtained. It can be estimated well.

Consider a case where sampling data is thinned out in such an algorithm. FIG. 4 is a diagram for explaining an example of sampling data thinning-out processing.
FIG. 4 shows a thinning process when the number of data points is halved as an example. The method of FIG. 4 (A) reduces the number of data points by simply thinning out sampling data of L points by line integration, and the circuit scale can be minimized. In the method of FIG. 4B, new data is calculated by calculation from sampling data at a plurality of points using LPF. Since the frequency of the flicker waveform is sufficiently smaller than the sampling interval for each line, for example, even when the LPF is subjected to simple averaging, the flicker waveform to be detected is not dulled but rather the noise component is suppressed. Therefore, the effect of improving detection accuracy can be obtained.

  However, since the flicker detection algorithm described above is premised on accurate sampling in a period corresponding to a flicker waveform for one cycle, as described below, the detection accuracy decreases depending on conditions. May end up.

FIG. 5 is a diagram schematically showing a thinning process when the sampling period coincides with one flicker waveform cycle.
FIG. 5 shows an example of processing in a system in which the number M of lines in one field is 1000 and the number of sampling points L1 per field output from the integration processing unit 111 is 100. In this case, the number L of lines in one cycle of the flicker waveform is 1000 × 60/100 = 600. In order to thin out the data for the L lines to the L1 point permitted by the system, the number of lines is related to the type of thinning processing. The thinning data unit (hereinafter referred to as the thinning unit) D is L / L1 = 6. That is, if the decimation unit D is set to 6 when the number of data points is changed from L to L1 by some decimation processing, data of L1 point (= 100 points) allowed by the system (for example, data in the allocated memory area) All of the above can be used to accurately sample one cycle of the flicker waveform, and therefore the above flicker detection algorithm can perform highly accurate detection.

On the other hand, the case where the number of sampling points L1 allowed by the system remains 100 and the number of lines M per field is 900 will be described with reference to FIGS.
FIG. 6 is a diagram schematically showing a thinning process when the sampling period is less than one flicker waveform cycle. FIG. 7 is a diagram schematically showing a thinning process when the sampling period exceeds one flicker waveform cycle. Further, FIG. 8 is a diagram schematically showing a thinning process when the number of data points used is reduced so that the sampling period matches one cycle of the flicker waveform.

  When the number of lines M is 900, the number of lines L in one cycle of the flicker waveform is 900 × 60/100 = 540. In order to thin out the data for L lines to the L1 point allowed by the system, thinning is performed. Regardless of the type of processing, the thinning unit D must be L / L1 = 5.4.

  However, the thinning unit D needs to be an integer. Therefore, as shown in FIG. 6, for example, if 5 is selected as the value closest to the true D value, even if all the sampling data of the L1 point allowed by the system is output, the sampling period is 500 (= 5 × 100) lines, and one cycle of flicker waveform cannot be sampled. Also, as shown in FIG. 7, when 6 is selected as the thinning unit D, sampling of 600 (= 6 × 100) lines is performed in order to sample all the L1 points allowed by the system. Therefore, the subsequent processing is executed using sampling data in a period exceeding one flicker waveform.

  Further, when 6 is selected as the thinning unit D and the number of sampling points L1 after thinning is 90, as shown in FIG. 8, a period corresponding to one flicker waveform can be sampled accurately. However, 100 points of data are allowed to be sampled as a system, and for example, only 90 points of them are used even though a corresponding memory area is prepared. Of course, it is only necessary to construct a system that processes only 90 points of data in anticipation of this, but this limits the number of pixels of the image sensor that can be mounted, and the versatility is lost. In general, it is difficult to make D an integer by properly combining the thinning unit D and the number of sampling points L1 so that the period of one flicker waveform can be accurately sampled.

FIG. 9 is a diagram for describing DFT processing when the sampling period does not match one cycle of the flicker waveform.
As described above, the spectrum sequence obtained by the DFT processing unit 120 in the subsequent stage corresponds to “components of each order when Fourier series expansion is performed with a sine wave having a sampling period of one cycle”. If the period does not coincide with one cycle of the flicker waveform, the obtained spectrum sequence is no longer developed in “one cycle of flicker”.

  As shown in the upper part of FIG. 9, when one period of the flicker waveform can be sampled, a so-called DFT window indicated by a broken line in the figure coincides with one period of the original flicker waveform, and the DFT processing unit 120 The Fourier transform is performed under the assumption that the waveform in the broken line frame is repeated infinitely. That is, the signal in the window is developed on the frequency axis with a spectrum sequence of a sine wave having one cycle of the DFT window.

  On the other hand, as shown in the middle and lower stages of FIG. 9, when the sampling period deviates from one flicker cycle, the DFT window size is different. The Fourier transform is performed under the assumption that “the signal is repeated indefinitely”. For this reason, the spectrum sequence obtained by such processing is different from the one obtained by expanding the original flicker waveform by Fourier series expansion with a sine wave of one cycle. Thus, in the above flicker detection algorithm, when the sampling period does not coincide with one period of the flicker waveform, there is a problem that a detection error corresponding to the amount of deviation occurs and the detection accuracy is lowered. .

  As a method for avoiding this problem, it is conceivable to significantly increase the number of sampling points L1, but the system scale increases as the number increases. Further, the above problem can be avoided to some extent by properly selecting the combination of the number of sampling points L1 and the thinning unit D, but a plurality of sensor variations corresponding to the system (that is, variations of the number of lines M in one field) can be avoided. Not all solutions have an appropriate solution, and there may be variations in performance from sensor to sensor.

  In addition, the flicker detection algorithm described above has a problem with respect to setting of a twiddle factor necessary for DFT processing. DFT processing generally requires a sine wave (cosine wave) called a twiddle factor, and the phase of this twiddle factor usually needs to match the phase of each sampling data. In order to detect high-order terms, a twiddle factor corresponding to the order is also necessary.

  If a single system supports multiple sensor variations, even if a combination of L1 and D that is closest to one cycle is selected for each sensor, the DFT process requires an appropriate twiddle factor for each. These must be set in the DFT processing unit 120, for example, by setting them in the ROM table and reading them, or reading them by communication. For this reason, every detection order must be prepared separately each time the corresponding number of sensors increases, which is a practically significant problem from the viewpoint of system scale and development cost.

<Configuration Example 1 of Integration Processing Unit>
FIG. 10 is a block diagram illustrating a first configuration example of the integration processing unit.
As described above, the integration processing unit 111 illustrated in FIG. 10 includes the line integrator 210 and the thinning processing unit 220. The thinning processing unit 220 includes the V-direction thinning calculation unit 221 and the interpolation processing unit 222. Yes.

  The line integrator 210 integrates the input image signal for each line, and outputs an integral value over the entire screen, that is, an integral value for M lines per field to the thinning processing unit 220. The V direction decimation unit 221 decimates the number of data points of the input integral value for each line in a decimation unit so that M is L2 per field. As a thinning calculation method, LPF thinning as described later can be applied. In addition, instead of calculation, the input data may be simply thinned out every certain number and output. In this case, thinning may be performed by controlling the integration timing in the line integrator 210, thereby simplifying the circuit configuration. The interpolation processing unit 222 interpolates and generates sampling data at point L1 that accurately corresponds to a period corresponding to one cycle of the flicker waveform based on the sampling data at point L2 output after thinning.

FIG. 11 is a diagram for explaining the operation of the integration processing unit of FIG.
The upper part of FIG. 11 shows a case where the V-direction thinning-out calculation unit 221 outputs one data using an integration value for the D1 line by an operation such as LPF. In this example, D1 is set to 4. At this time, the number of data points L2 output as a result of thinning is set to be equal to or greater than the number of lines L corresponding to one cycle of the flicker waveform. That is, when L / L2 does not become an integer due to the number of lines of the image pickup device to be mounted, L2 is set to L or more so that L / L2 (= D1) becomes an integer, thereby a period of one flicker waveform or more. Can be sampled reliably.

  Note that the V-direction thinning calculation unit 221 does not need to process and output all the L2 points in parallel. When this block is processed in time series as much as possible (that is, the input sampling data is sequentially processed), the circuit scale may vary depending on the thinning unit D1, but the number of sampling points per line Therefore, the circuit scale does not increase. For this reason, even if the value of L2 becomes larger than L, the circuit scale does not increase significantly.

  Further, as shown in the lower part of FIG. 11, the interpolation processing unit 222 newly calculates the L1 point data sampled from the thinned L2 point data in a period that exactly matches one flicker waveform cycle by interpolation. Generated and output. This sampling point number L1 is set to a value that sufficiently satisfies the sampling theorem and obtains necessary detection accuracy within a range allowed for the system, such as a memory capacity in the integral value holding unit 112, and is mounted. Regardless of the number of lines M, the value is constant. For this reason, L / L1 (= D2) may not be an integer depending on the number M of lines of the image sensor, but even in that case, the data at the L1 point is reliably obtained by performing the interpolation calculation from the sampling data at the L2 point. Can be generated.

  Accordingly, the DFT window in the DFT processing unit 120 at the subsequent stage can be accurately matched with one period of the flicker waveform, so that the flicker waveform can be estimated with high accuracy. Further, since the cycle of the flicker waveform is sufficiently longer than the sampling interval, sufficient detection accuracy can be obtained by a process of linear interpolation as the interpolation calculation.

FIG. 12 is a diagram illustrating an internal configuration example of the V-direction thinning-out calculation unit.
As the decimation process in the V-direction decimation unit 221, as described above, “simple decimation” that intermittently outputs input data, decimation using LPF, or the like can be applied. FIG. 12 shows a configuration example when LPF is used. As shown in FIG. 12, when the decimation unit D1 is 2 to the nth power (n is an integer of 0 or more), the V-direction decimation unit 221 can have a simple configuration of an adder 223 and an n-bit shift unit 224. The adder 223 adds and outputs the integral value at the point D1, and the n-bit shift unit 224 shifts the input data to the lower side by n bits.

  Further, even when the thinning unit D1 is not 2 to the nth power, for example, the adder 223 always performs addition processing of data of 2n, and the input data to the adder 223 is the interval of the thinning unit D1. In this case, the circuit configuration can be simplified although some data is discarded. Further, it becomes possible to easily cope with image pickup devices having various numbers of lines. On the other hand, when the restrictions on the circuit area and the manufacturing cost are loose, the detection accuracy can be improved by calculating a complete average value using the n-bit shift unit 224 as a divider.

  As described above, the flicker waveform cycle is sufficiently longer than the sampling interval for each line, so that the flicker waveform to be detected is not blunt even in the configuration of the LPF that performs the averaging described above. Rather, the effect of suppressing noise components can be obtained. For this reason, with the above configuration, both simplification of the circuit configuration and improvement of detection accuracy can be achieved.

  According to the flicker reduction unit 20 to which the integration processing unit 111 of FIG. 10 is applied, the period for one cycle of the flicker waveform is always accurately sampled and detected without being restricted by the number of pixels of the mounted image sensor. -Since reduction processing can be executed, flicker detection accuracy can be stably improved, and variation in detection performance for each sensor variation can be reduced. Further, since the twiddle factor necessary for the DFT processing can always be set to the same value, for example, the flicker detection accuracy can be improved without increasing the circuit scale such as a memory for holding parameters. Furthermore, since the subsequent calculations are performed using all sampling data (for example, all sampling data stored in the integral value holding unit 112) as many points as the system allows, the circuit scale and manufacturing cost are further increased. Can be prevented.

  Such an effect can be obtained by simply changing the configuration of the integration processing unit with respect to the conventional flicker detection / reduction circuit in which the circuit scale is reduced by thinning sampling data. Changes in the circuit configuration and control procedure when optimizing for different image sensors are reduced. Therefore, it is possible to realize a small circuit with high versatility and improved flicker detection performance.

<Configuration Example 2 of Integration Processing Unit>
FIG. 13 is a block diagram illustrating a second configuration example of the integration processing unit.
The integration processing unit 111 illustrated in FIG. 13 is different from the configuration illustrated in FIG. 10 in the configuration of the thinning processing unit 220. The thinning processing unit 220 includes a V-direction gate 225 and an interpolation processing unit 226.

  The V-direction gate 225 receives the integral value over the entire screen from the line integrator 210, and outputs the integral value to the subsequent stage for the minimum period for sampling one period of the flicker waveform, that is, the period for the L line. During this period, output is stopped. The interpolation processing unit 226 generates sampling data of L1 point that accurately corresponds to a period of one flicker waveform by interpolation calculation using the input integral value of L lines. The interpolation processing unit 226 is configured to directly generate sampling data of the L1 point by calculation from the result of line integration, and thereby, when sampling data is converted in two stages as in the first embodiment. As compared with the above, the flicker waveform detection performance can be improved.

FIG. 14 is a diagram for explaining the operation of the integration processing unit in FIG. 13.
As the interpolation processing in the interpolation processing unit 226, since the cycle of the flicker waveform is sufficiently longer than the sampling interval, necessary and sufficient detection accuracy can be maintained by general processing using an appropriate interpolation function. Since the cycle of the flicker waveform is sufficiently long, in principle, the processing of the degree of linear interpolation is sufficient, but the above-mentioned LPF effect is enhanced by generating data of the L1 point using a larger number of sampling data. And detection accuracy can be increased.

  As such an example, so-called cubic interpolation can be applied. In this method, as shown in FIG. 14, a cubic interpolation function is applied to each point based on a certain number of integrated values (four lines in FIG. 14) sandwiching the sampling positions of the data at point L1. Generate data for This makes it possible to freely set the center of gravity of newly generated data at a position between the original data intervals (that is, the data interval for each line), so that the flicker detection accuracy is stable regardless of the number of pixels of the mounted image sensor Can be improved. That is, even when the thinning unit D2 (= L / L1) does not become an integer due to the number M of lines of the image pickup device to be mounted, a flicker waveform is prepared by preparing an interpolation function in advance in accordance with the number M of lines. It is possible to accurately sample in a period of one cycle, and to match the DFT window with that period. Therefore, as in the case of FIG. 10, a small-scale circuit with high versatility and improved flicker detection performance can be realized.

  In the first embodiment, the case where the input image signal In ′ (x, y) is integrated for each line in the integration processing unit 111 has been described. However, each integration period is not limited to one line. . However, the longer the integration period is, the more precisely the flicker component can be sampled by eliminating the influence of the pattern, so the integration may be performed over a time of one line or more. For example, the integration period of the line integrator 210 may be a plurality of lines. For example, when integration is performed in units of two lines, sampling data of (M / 2) points per field is supplied to the thinning processing unit 220. Further, in each integration period, the data of all pixels may not be used as an integration target, but may be used intermittently or only in a specific area.

  In the first embodiment, data sampled by the thinning processing unit 220 in a period corresponding to one cycle of the flicker waveform is output. However, this sampling period does not necessarily need to be one cycle, and a plurality of data are output. It may be a period of a period.

  Further, in the first embodiment, the DFT is used as a method for converting the flicker component to the frequency component by setting the number of data points L1 per cycle of the flicker waveform output from the thinning processing unit 220 to a power of 2. Instead, FFT can be used. When FFT is used, the amount of calculation can be reduced as compared with DFT, so that the circuit scale of the flicker reduction unit 20 can be suppressed. In addition, the DFT processing function can be easily realized by software processing.

  In the first embodiment, the case where the screen rate (frame frequency or field frequency) is constant has been described. However, for example, an imaging device with a variable screen rate is conceivable as an additional function of the imaging device, such as enabling imaging at a higher screen rate. As described above, when not only the number of lines on the image sensor but also the screen rate is changed, not only the integral multiple of the sampling interval by the integration processing unit 111 often does not coincide with one period of the flicker waveform. It is extremely rare that the number of lines corresponding to one cycle of the flicker waveform becomes an integer. Therefore, as described above, the sampling data accurately corresponding to the period of one cycle of the flicker waveform is obtained by interpolation, so that the flicker detection accuracy is high and the change in the number of pixels of the image sensor and the change in the screen rate is achieved. A circuit with wider versatility can be realized.

[Second Embodiment]
FIG. 15 is a block diagram showing an internal configuration of the flicker reducing unit according to the second embodiment of the present invention. In FIG. 15, blocks corresponding to those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted.

  In the second embodiment shown in FIG. 15, as in the first embodiment described above, the integrated value by the line integrator 210 is exactly L1 corresponding to the period of one flicker waveform (or a plurality of periods). Instead of interpolating as point data, one cycle (or a plurality of flicker waveforms) based on a flicker component (that is, a difference value gn (y)) calculated by a difference calculation based on an integral value for each frame or field or a normalization process. Sampling data accurately corresponding to the period of period) is generated by interpolation calculation and supplied to the DFT processing unit 120.

  In the flicker reducing unit 20a shown in FIG. 15, the integration processing unit 111a includes a line integrator 210 and a thinning processing unit 230. Similar to the first embodiment, the line integrator 210 integrates the input image signal for each line. The decimation processing unit 230 is a block having substantially the same function as the V-direction decimation unit 221 shown in FIG. 10, and outputs the number of data points of the input integral values for each line by decimation in a certain decimation unit. Note that the above-described LPF thinning or the like can be applied as a thinning calculation method in the thinning processing unit 230. Further, instead of the calculation, the input data may be simply thinned out every certain number and output.

  The thinning processing unit 230 outputs a predetermined number (for example, L2 points) of data for each period of one cycle or more of the flicker waveform. That is, since the sampling period based on these data does not necessarily correspond to the period of one flicker waveform depending on the number of pixels of the mounted image sensor, sampling is performed in a slightly longer period than this period, and the subsequent flicker component interpolation processing unit 150 The interpolation data corresponding to the period exactly corresponding to one flicker waveform is obtained.

  As in the first embodiment, the integration period by the line integrator 210 may be a period of one line or more, such as a plurality of lines. Further, in each integration period, the data of all pixels may not be used as an integration target, but may be used intermittently or only in a specific area. Further, the integrated value obtained by the line integrator 210 may be output to the subsequent stage without providing the thinning processing unit 230. In short, the integration processing unit 111a only needs to output integration values representing a predetermined area on the screen at regular intervals.

  In FIG. 15, the flicker component interpolation processing unit 150 uses the flicker component extracted by the normalized integral value calculation unit 110, that is, the normalized difference value gn (y) output from the normalization processing unit 115. Based on this, a predetermined number of data obtained by equally dividing the period of one cycle of the flicker waveform is generated by interpolation calculation and supplied to the DFT processing unit 120. The flicker component interpolation processing unit 150 selects two or more data centered on each point after interpolation from the output data from the normalization processing unit 115, and performs an interpolation operation based on these data. As the interpolation calculation method, linear interpolation, a method using an interpolation function such as cubic interpolation described with reference to FIG. 13, or the like can be applied.

  Since the flicker component interpolation processing unit 150 inputs the data of the L1 point accurately corresponding to the period of one cycle of the flicker waveform to the DFT processing unit 120, the DFT processing unit 120 performs the flicker for one cycle. The component can be estimated with high accuracy. Therefore, as in the first embodiment, it is possible to realize a circuit with high flicker component detection / correction accuracy and high versatility with respect to the number of pixels of the image sensor and the screen rate.

  As in the case of the first embodiment, a method of converting a flicker component into a frequency component by setting the number of data points per flicker waveform output from the flicker component interpolation processing unit 150 to a power of two. As a result, FFT can be used instead of DFT, the circuit scale of the flicker reduction unit 20a can be suppressed, and the DFT processing function can be easily realized by software processing.

[Third Embodiment]
FIG. 16 is a block diagram showing an internal configuration of the flicker reducing unit according to the third embodiment of the present invention. In FIG. 16, blocks corresponding to those in FIG. 15 are denoted by the same reference numerals, and description thereof is omitted.

  The flicker reduction unit 20b shown in FIG. 16 is different from the flicker reduction unit 20a in FIG. 15 in that an estimated component interpolation processing unit is provided between the flicker generation unit 130 and the calculation unit 140 instead of the flicker component interpolation processing unit 150. 160 is provided. Here, the flicker component (corresponding to the flicker coefficient Γn (y)) estimated by the operations of the DFT processing unit 120 and the flicker generation unit 130 is actually synchronized with, for example, data input to the DFT processing unit 120 The data is output from the flicker generation unit 130 as discrete data. The estimated component interpolation processing unit 160 interpolates the discrete data group of the flicker component output from the flicker generation unit 130 so as to match the phase of the data group to be corrected by the calculation unit 140.

FIG. 17 is a diagram for explaining the interpolation calculation by the estimated component interpolation processing unit.
The flicker reducing unit 20b in FIG. 16 has a function of interpolating sampling data so that the sampling interval matches one cycle of the flicker waveform (a function of the thinning processing unit 220 in FIG. 3 or the flicker component interpolation processing unit 150 in FIG. 15). Not provided. In such a case, as described above, depending on the number of pixels of the image sensor and the screen rate, an integer multiple of the sampling interval by the line integrator 210 or an integer multiple of the data output interval of the thinning processing unit 230 thereafter is Very little coincides with the period of one cycle of the flicker waveform. Therefore, since the phase of the discrete data of the waveform estimated by the DFT processing and the phase of the correction target data input to the calculation unit 140 do not match, a correction error occurs.

  For example, in the example of FIG. 17, since the DFT processing is performed based on the sampling data in a period that is less than one period of the flicker waveform, the DFT processing unit 120 causes the actual flicker waveform (included in the correction target image data ( A waveform having a short cycle (corresponding to the upper waveform in the figure) is estimated from the lower waveform in the figure. For this reason, if correction is performed using discrete data obtained as a result of estimation as it is, a correction error occurs, and flicker components cannot be accurately removed from the input image signal.

  Therefore, in the present embodiment, the estimated component interpolation processing unit 160 scales and outputs the discrete data output from the flicker generation unit 130 in synchronization with the correction timing in the calculation unit 140. That is, based on the discrete data from the flicker generation unit 130, data of arbitrary points synchronized with the correction timing in the calculation unit 140 is generated by interpolation calculation and output.

  In the example of FIG. 17, one piece of data is generated by performing an interpolation operation using four consecutive pieces of discrete data from the flicker generation unit 130. The flicker waveform from the flicker generation unit 130 has a period different from that of the actual flicker waveform to be corrected. Therefore, the period of the flicker waveform from the flicker generation unit 130 is converted in accordance with the period of the actual flicker waveform. Of the subsequent discrete data, four points in the vicinity centering on each point synchronized with the correction timing in the calculation unit 140 are selected and applied to the interpolation calculation. Actually, a specific number of discrete data of the estimated flicker component is converted into an arbitrary number of predetermined data, and each converted data is sequentially output in synchronization with the correction timing in the calculation unit 140. Will be.

  In the present embodiment, since the minimum unit of integration processing in the integration processing unit 111a is one line, in the estimated component interpolation processing unit 160, image data corresponding to one line or an integer multiple thereof is calculated by the calculation unit 140. The number of data after scaling is determined so as to synchronize with the timing input to. For example, if the number of data per flicker waveform generated by the estimated component interpolation processing unit 160 matches the number of data per flicker waveform output from the line integrator 210 or the thinning processing unit 230, Good. That is, each scaled data is generated so as to correspond to each period obtained by dividing one period or a plurality of periods of the actual flicker component in the image signal by one line or an integral multiple thereof, and in each of these periods The generated data is sequentially output in synchronization with the input timing of the corresponding image signal to the calculation unit 140.

Further, as an interpolation method in the estimated component interpolation processing unit 160, a method using linear interpolation, an interpolation function such as the cubic interpolation described above, or the like can be applied.
Through the above processing, the calculation unit 140 can reduce flicker components with high accuracy without causing an error, and can obtain a captured image with high image quality.

[Fourth Embodiment]
FIG. 18 is a block diagram showing an internal configuration of the flicker reducing unit according to the fourth embodiment of the present invention. In FIG. 18, blocks corresponding to those in FIG. 16 are denoted by the same reference numerals, and description thereof is omitted.

  In the flicker reduction unit 20c shown in FIG. 18, the calculation unit 140c includes a correction gain calculation unit 141, a gain interpolation processing unit 142, and an image correction unit 143. The correction gain calculation unit 141 calculates a correction gain that is a correction parameter for canceling the flicker component based on the discrete data of the flicker component from the flicker generation unit 130. This correction gain corresponds to, for example, 1 / [1 + Γn (y)] in Equation (21) described above. The gain interpolation processing unit 142 scales and outputs discrete data of the correction gain calculated by the correction gain calculation unit 141 so as to be synchronized with the correction timing of the image correction unit 143. The image correction unit 143 multiplies the input image signal by the correction gain scaled by the gain interpolation processing unit 142 according to the equation (21), and performs an operation for removing the flicker component.

  In the flicker reducing unit 20c, a gain interpolation processing unit 142 is provided instead of the estimated component interpolation processing unit 160 in FIG. 16, and is synchronized with the timing at the time of correction based on the correction gain obtained from the estimated flicker waveform. Scaling prevents the occurrence of correction errors. The number of data output after scaling, the interpolation calculation method used for scaling, and the like are the same as those of the estimated component interpolation processing unit 160 described above. With this configuration, as in the case of FIG. 16, flicker components can be reduced with high accuracy, and a captured image with high image quality can be obtained.

[Fifth Embodiment]
FIG. 19 is a block diagram showing an internal configuration of the flicker reducing unit according to the fifth embodiment of the present invention. In FIG. 19, blocks corresponding to those in FIGS. 15 and 16 are denoted by the same reference numerals, and description thereof is omitted.

  The flicker reducing unit 20d shown in FIG. 19 has a function of interpolating the sampled data in accordance with one period (or a plurality of periods) of the flicker waveform and the DFT processing as in the first and second embodiments described above. And a function of scaling the discrete data of the flicker waveform estimated by the method so as to synchronize with the correction timing in the calculation unit 140. In this example, the flicker component interpolation processing unit 150 described in FIG. 15 is provided as the former function, and the estimated component interpolation processing unit 160 described in FIG. 16 is provided as the latter function. As a result, it is possible to obtain both a detection error reduction effect when flicker is detected by the DFT processing and a correction error reduction effect when the flicker component estimated by the DFT processing is applied to the correction. The same effect can be obtained even when the integration processing unit 111 described in FIG. 3 is provided as the former function or when the calculation unit 140c described in FIG. 18 is provided as the latter function. it can.

FIG. 20 is a diagram for explaining the interpolation calculation by the estimated component interpolation processing unit of FIG.
In the flicker reduction unit 20d of FIG. 19, the data sampled at equal intervals in the period corresponding to one cycle (or a plurality of cycles) of the flicker waveform by the processing of the flicker component interpolation processing unit 150 is sent to the DFT processing unit 120. Entered. For this reason, the cycle of the flicker waveform (the upper waveform in FIG. 20) output from the flicker generation unit 130 matches the cycle of the actual flicker waveform (the lower waveform in FIG. 20) included in the image signal. However, since the interval of the discrete data on the estimated flicker waveform is different from the interval of the data sampled by the integration processing unit 111a, the estimated flicker waveform is used as it is when the calculation unit 140 corrects it. If it does, a correction error will occur.

  Therefore, the discrete data group corresponding to the estimated flicker waveform is converted into a data group synchronized with the correction timing in the calculation unit 140 by the interpolation calculation in the estimated component interpolation processing unit 160. In FIG. 20, as an example, four pieces of discrete data in the estimated flicker waveform are applied to the interpolation calculation to generate one piece of data. Thereby, the synchronization of the correction timing in the calculating part 140 is achieved, and generation | occurrence | production of a correction error can be prevented. Even when the sampling data to the DFT processing unit 120 is interpolated, the estimated component interpolation processing unit 160 may need to convert the estimated flicker waveform cycle.

  As described above, the function of interpolating the sampled data in accordance with one period (or a plurality of periods) of the flicker waveform, and the function of scaling the discrete data of the flicker waveform estimated by the DFT processing so as to synchronize with the correction timing. Since both the flicker detection error and the correction error during the DFT processing can be reduced, the flicker component can be more accurately removed, and the quality of the captured image can be further improved.

  In each of the above embodiments, the case where the output data of the line integrator 210 is thinned has been described. However, the present invention can be applied even when thinning is not performed. For example, when there is a margin in the calculation performance in the flicker reduction unit, the installation area of the memory circuit holding the integrated value, and the manufacturing cost thereof, the sampling unit of the integrated value can be set to one line unit at the shortest. . As described above, depending on the number of lines on the image sensor and the screen rate, the number of lines corresponding to one cycle of the flicker waveform may not be an integer. In such a case, the present invention is applied to the flicker. Detection errors and correction errors can be reduced.

  In each of the above-described embodiments, a case where a CMOS image sensor is used as an image sensor has been described. However, when another XY address scanning image sensor such as a MOS image sensor other than a CMOS image sensor is used. The present invention is also applicable. The present invention can also be applied to various imaging devices using XY address scanning type imaging devices, and devices such as mobile phones and PDAs (Personal Digital Assistants) having such an imaging function. .

  Furthermore, for an image processing apparatus that performs processing on an image signal by a small camera such as a videophone or game software connected to a PC (personal computer) or the like, or processing for correcting a captured image. Also, the present invention can be applied.

  Further, the above processing functions can be realized by a computer. In that case, a program describing the processing contents of the functions that the apparatus should have (functions corresponding to the above-described flicker reducing section) is provided. And the said processing function is implement | achieved on a computer by running the program with a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical disk, and a semiconductor memory.

  In order to distribute the program, for example, portable recording media such as an optical disk and a semiconductor memory on which the program is recorded are sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Further, each time the program is transferred from the server computer, the computer can sequentially execute processing according to the received program.

  DESCRIPTION OF SYMBOLS 11 ... Optical block, 11a ... Driver, 12 ... CMOS type image sensor (CMOS sensor), 12a ... Timing generator (TG), 13 ... Analog front end (AFE) circuit, 14 ... Camera processing circuit, DESCRIPTION OF SYMBOLS 15 ... System controller, 16 ... Input part, 17 ... Graphic I / F, 17a ... Display, 20 ... Flicker reduction part, 110 ... Normalization integral value calculation part, 111 ... Integration processing part, 112 …… Integral value holding unit, 113 …… Average value calculating unit, 114 …… Difference calculating unit, 115 …… Normalization processing unit, 120 …… DFT processing unit, 130 …… Flicker generation unit, 140 …… Calculation unit, 210 …… Line integrator, 220 …… Thinning processing unit

Claims (12)

Integration means for integrating the input image signal in units of one horizontal synchronization period or more;
Normalization means for normalizing the integration value by the integration means or the difference value of the integration values in adjacent fields or frames;
Data corresponding to a predetermined number of sampling positions defined in advance during a predetermined period of flicker is generated by interpolating based on the integrated value or difference value after normalization by the normalizing means, and Interpolation processing means for outputting corresponding data as an image signal;
Frequency analysis means for extracting a spectrum by analyzing the image signal generated by the interpolation processing means;
Flicker estimation means for estimating a flicker component from the spectrum extracted by the frequency analysis means;
Correction flicker component generation means for generating a flicker component for correction having the same phase as the input image signal and the same number of lines from the flicker component estimated by the flicker estimation means;
Image correction means for removing the flicker component from the input image signal by the correction flicker component;
An image processing apparatus comprising: The interpolation processing means performs an interpolation operation using an interpolation function based on a predetermined number of integrated values or difference values after normalization by the normalizing means for each sampling position, and data corresponding to the sampling position The image processing apparatus according to claim 1, wherein: The interpolation processing means selects the predetermined number of normalized integration values or difference values from the integration values or difference values after normalization by the normalization means in a period of one cycle or a plurality of cycles of the flicker. The image processing apparatus according to claim 2, wherein interpolation processing is performed to generate data corresponding to each sampling position. Integration means for integrating the input image signal in units of one horizontal synchronization period or more;
Normalizing means for normalizing the integrated value by the integrating means, or the difference value of the integrated values in adjacent fields or frames, and outputting the normalized integrated value or difference value as an image signal;
Frequency analysis means for extracting a spectrum by analyzing the image signal generated by the normalization means;
Flicker estimation means for estimating a flicker component from the spectrum extracted by the frequency analysis means;
A predetermined number of flicker components included in the input image signal are defined for one period or a plurality of periods, and each output timing is defined so as to be synchronized with an integration unit by the integration means or a period of an integral multiple thereof. Correction flicker component generation means for generating the data by interpolation based on the discrete value of the flicker component estimated by the flicker estimation means, and outputting as a correction flicker component;
Image correction means for removing the flicker component from the input image signal by the correction flicker component;
An image processing apparatus comprising: The correction flicker component generation means performs a linear interpolation based on a predetermined number of discrete values of the estimated flicker component, so that a period corresponding to one period or a plurality of periods of the flicker component included in the input image signal The image processing apparatus according to claim 4, wherein each data defined in the above is generated. The correction flicker component generation means performs an interpolation operation using an interpolation function based on a predetermined number of discrete values of the estimated flicker component, thereby performing one cycle of the flicker component included in the input image signal. The image processing apparatus according to claim 4, wherein each data defined in a period of a plurality of cycles is generated. Integration means for integrating the input image signal in units of one horizontal synchronization period or more;
Normalizing means for normalizing the integrated value by the integrating means, or the difference value of the integrated values in adjacent fields or frames, and outputting the normalized integrated value or difference value as an image signal;
Frequency analysis means for extracting a spectrum by analyzing the image signal generated by the normalization means;
  Flicker estimation means for estimating a flicker component from the spectrum extracted by the frequency analysis means;
A correction parameter for canceling the flicker component is calculated based on an estimated value of the flicker component by the flicker estimation means, and a fixed number of flicker components included in the input image signal for one period or a plurality of periods. Flicker for correction is generated by interpolating data defined so that each output timing is synchronized with an integration unit by the integration means or a period that is an integral multiple of the integration unit based on the discrete value of the correction parameter. Correction flicker component generation means for outputting as a component;
Image correction means for removing the flicker component from the input image signal by the correction flicker component;
An image processing apparatus comprising: The correction flicker component generation means generates each data defined in one period or a plurality of periods of the flicker component by performing linear interpolation based on a predetermined number of discrete values of the correction parameter. The image processing apparatus according to claim 7. The correction flicker component generation means performs an interpolation operation using an interpolation function based on a predetermined number of discrete values of the correction parameter, thereby defining a period corresponding to one cycle or a plurality of cycles of the flicker component. 8. The image processing apparatus according to claim 7, wherein each data is generated. In an imaging apparatus that captures an image using an XY address scanning type solid-state imaging device,
Integrating means for integrating the input image signal obtained by imaging in units of one horizontal synchronization period or more;
Normalization means for normalizing the integration value by the integration means or the difference value of the integration values in adjacent fields or frames;
Data corresponding to a predetermined number of sampling positions defined in advance during a predetermined period of flicker is generated by interpolating based on the integrated value or difference value after normalization by the normalizing means, and Interpolation processing means for outputting corresponding data as an image signal;
Frequency analysis means for extracting a spectrum by analyzing the image signal generated by the interpolation processing means;
Flicker estimation means for estimating a flicker component from the spectrum extracted by the frequency analysis means;
Correction flicker component generation means for generating a flicker component for correction having the same phase as the input image signal and the same number of lines from the flicker component estimated by the flicker estimation means;
Image correction means for removing the flicker component from the input image signal by the correction flicker component;
An imaging device comprising: Integrate the input image signal in units of one horizontal synchronization period or more,
Normalize the integral value or the difference value of the integral value in the adjacent field or frame,
Data corresponding to a predetermined number of sampling positions defined in advance for a predetermined period of flicker is generated by interpolation based on the normalized integral value or difference value, and the data corresponding to the sampling positions is imaged As a signal ,
Extracting the spectrum by analyzing the generated image signal,
Estimate the flicker component from the extracted spectrum,
From the estimated flicker component, a correction flicker component having the same phase as the input image signal and the same number of lines is generated,
Removing the flicker component from the input image signal by the correction flicker component;
An image processing method. In an image processing program for causing a computer to execute processing for detecting flicker generated on an image under fluorescent lamp illumination,
Integration means for integrating the input image signal in units of one horizontal synchronization period or more;
Normalization means for normalizing the integration value by the integration means or the difference value of the integration values in adjacent fields or frames
Data corresponding to a predetermined number of sampling positions defined in advance during a predetermined period of flicker is generated by interpolating based on the integrated value or difference value after normalization by the normalizing means, and Interpolation processing means for outputting corresponding data as an image signal,
Frequency analysis means for extracting a spectrum by analyzing the image signal generated by the interpolation processing means;
Flicker estimation means for estimating a flicker component from the spectrum extracted by the frequency analysis means;
  A correction flicker component generation unit that generates a correction flicker component having the same phase as the input image signal and the same number of lines from the flicker component estimated by the flicker estimation unit;
  Image correction means for removing a flicker component from the input image signal by the correction flicker component;
  An image processing program for causing the computer to function as:

Images