JP5383886B2 - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to a technique for correcting a flicker component in an image captured using an image sensor.

  In an image pickup apparatus using an image pickup device, when a subject under fluorescent light illumination is shot continuously for a plurality of frames (or fields) like continuous shooting of moving images or still images, periodic brightness and darkness may appear in the captured image. . This phenomenon is called fluorescent lamp flicker, and is due to the difference between the commercial power supply frequency and the vertical synchronization frequency of the imaging device.

  Therefore, conventionally, a technique for detecting and correcting a flicker component by detecting a periodic signal level fluctuation component (hereinafter referred to as a flicker component) and multiplying by its inverse gain has been proposed (see Patent Document 1). ).

Japanese Patent Laid-Open No. 11-122513

  The flicker component of the fluorescent lamp flicker for the imaging device varies depending on the power supply frequency and the light emission characteristics of the illumination, but the driving time such as the accumulation time and frame rate for the imaging device of the imaging device, and the change in the readout time due to the pixel addition method and thinning method It also changes depending on the state. This driving state changes depending on the control of the imaging system or an instruction from the user, and may change abruptly. In this case, the flicker component greatly fluctuates as the driving state changes.

  However, the correction method described in Patent Document 1 is a method of detecting a flicker component using an intensity distribution in a vertical direction generated by integrating image signals of current and past frames (or fields) in the horizontal direction. In such a method, it is not possible to follow a sudden change in the flicker component, or a time difference occurs between the image signal used for detection and the image signal to be corrected, so that appropriate correction processing is not performed. On the other hand, there was a problem that the captured image was adversely affected.

  The present invention has been made in view of such problems of the prior art. An object of the present invention is to provide an image processing apparatus and an image processing method capable of accurately correcting a flicker component even when the flicker component is rapidly changed due to a change in a driving method of an image sensor.

In order to solve such a problem, as technical features of the present invention, acquisition means for acquiring a first image and a second image, detection means for detecting a flicker component from the first image, The flicker component amplitude for the second image based on the flicker component for the first image detected by the detection means and the difference in accumulation time between the first image and the second image. Alternatively, calculation means for calculating at least one of the phases, and correction means for correcting the second image using the amplitude or phase of the flicker component for the second image calculated by the calculation means, It is characterized by having.

Further, as another technical feature, an acquisition step of acquiring the first image and the second image,
A detection step for detecting a flicker component from the first image, a flicker component for the first image detected in the detection step, and an accumulation time between the first image and the second image A calculation step for calculating at least one of an amplitude or a phase of a flicker component with respect to the second image based on the difference between the two and an amplitude of the flicker component with respect to the second image calculated in the calculation step Alternatively, a correction step of correcting the second image using a phase is provided.

  As described above, according to the present invention, even when the flicker component changes suddenly due to a change in the driving method of the image sensor, the flicker component can be corrected with high accuracy and the image quality can be improved.

It is a block diagram for demonstrating schematic structure of the flicker detection and correction apparatus of 1st Embodiment. 2 is a timing chart from flicker detection to correction according to the first embodiment. FIG. 6 is a schematic diagram illustrating a flicker state in an image signal due to a difference in accumulation time according to the first embodiment. It is the schematic diagram which showed the relationship between the accumulation time of 1st Embodiment, and the modulation | alteration of the luminance signal by a blinking light source. 3 is a timing chart illustrating a relationship between an accumulation time and a phase difference according to the first embodiment. 6 is a timing chart from flicker detection to correction according to the second embodiment. 10 is a timing chart from flicker detection to correction according to the third embodiment. It is the schematic diagram showing the mode of the flicker in the image signal by the difference in the readout time of 3rd Embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a block diagram of an image processing apparatus.

  An image sensor 101 having a photoelectric conversion element is driven by a drive pulse based on a control signal generated by the control means 102, converts a subject image into an electrical signal by photoelectric conversion, and outputs the electric signal. The converted electrical signal (analog signal) is converted into a digital signal by the AD converter 106 and output. At this time, when the image sensor 101 has an electronic shutter function, the exposure time may be ensured by the control signal from the control means 102 so that the required exposure time is obtained. Reference numeral 103 denotes flicker detection means for detecting a flicker component from a signal level fluctuation component generated by the blinking light source included in the image signal output from the AD conversion unit 106.

  A control unit 102 controls the system, outputs a control signal such as an accumulation time to the image sensor 101, and outputs a control state of the image sensor 101 as control information to a correction value predicting unit 104 described later. Reference numeral 104 is a correction based on the flicker component detected by the flicker detection unit 103 and the control information of the image sensor 101 output from the control unit 102 so that the fluctuation component of the signal level generated in the image signal does not appear. Correction value predicting means for generating a correction value to do. Reference numeral 105 denotes flicker correction means for performing arithmetic processing on the image signal based on the correction value generated by the correction value prediction means 104 and suppressing flicker generated on the image signal.

  Next, a timing chart of the first embodiment of the present invention will be described with reference to FIG. Reference numeral 201 denotes a vertical synchronization signal determined by the imaging method of the imaging apparatus. r1 to r7 represent the reading drive of the image sensor 101. In this embodiment, the image sensor 101 uses a CMOS image sensor that employs an XY address type scanning method, and the following description will be continued. In the case of the image sensor of the XY address type scanning method, since the exposure timing for each pixel is sequentially performed in the line direction, all pixels of the image sensor are sequentially read out within one vertical synchronization signal period as shown in the figure. Further, s1 to s7 represent driving for resetting the charge accumulated in the charge conversion unit of the image sensor 101. In order to unify the exposure time of each pixel, the reset timing is sequentially performed in the line direction as in the read driving. The time between s and r in the figure is the accumulation time of the image sensor 101, the image signals from A to C in the figure have a short accumulation time, and the image signals from D to G in the figure have a long accumulation time. Is output as Reference numeral 202 denotes the timing at which the processing operation of the flicker detection means 103 is performed, and the detection operation is performed during a part of the period during which the image signal is read. Reference numeral 203 denotes the timing at which the control means 102 outputs the control signal of the image sensor 101 to the image sensor 101. In the case of the figure, for example, when acquiring the image signal D to the image sensor 101 at the timing 204. Is given as a control signal b. Reference numeral 205 denotes a timing at which the control unit 102 outputs control information of the image sensor 101 to the correction value prediction unit 104. It is output with a delay of one vertical synchronization period with respect to the timing at which the control means 102 outputs a control signal to the image sensor 101. In the case of the figure, the control unit 102 outputs the control signal b to the image sensor 101 at the timing of 204, while the control information b is output to the correction value prediction unit 104 at the timing of 206. Thus, when generating a correction value for correcting the image signal D, it is possible to refer to the accumulation time when the image signal D is acquired. Reference numeral 207 shows the timing at which the correction value prediction means 104 predicts the correction value from the control information obtained from the control means 102 and the flicker component obtained from the flicker detection means 103. In the figure, the image sensor 101 is driven so that the accumulation time changes rapidly between the image signal C and the image signal D. The correction value prediction unit 104 receives control information of the image sensor 101 when an image signal that is a target of flicker detection is acquired from the control unit 102, and is always obtained as control information before one vertical synchronization period. Are comparing. And when this control information changes, it operate | moves so that a correction value may be estimated. Reference numerals 208 to 214 indicate timings when the correction value generated by the correction value predicting unit 104 is set for the flicker correcting unit 105. That is, when the control information does not change, a correction value is generated from the flicker component obtained from the flicker detection means 102. When the control information changes, an operation is performed so as to generate a correction value by predicting based on the flicker component obtained from the flicker detection unit 102 and the control information output from the control unit. The set correction value is updated with the next vertical synchronization signal, and correction processing is performed on the image signal read from the image sensor 101. In the case of this figure, 210 is the timing at which the correction value generated by prediction is set.

  Next, the flicker component output from the flicker detection means will be described together with an example of detection operation.

  FIG. 3 schematically illustrates an image signal captured by the image sensor 101, where 301 is an image signal when the accumulation time of the image sensor 101 is short, and 303 is a long accumulation time of the image sensor 101. It shows the situation of time. The signal level is modulated by the blinking light source, and periodic level fluctuations depending on the light emission period of the blinking light source occur in the image signal as indicated by 302 and 304. The flicker component contained in the image signal can be extracted by converting the periodic signal level fluctuation in the time domain into the frequency domain and detecting it.

(Extraction of flicker components)
There are several possible methods for converting the image signal obtained from the image sensor 101 into the frequency domain, but for data sampled over a certain period, such as an image signal, it is converted into the frequency domain using the discrete Fourier transform. This is the most common. Hereinafter, a method for calculating the flicker component by converting into the frequency domain using the discrete Fourier transform will be described.

  Flickers generated by the flashing light source appear in the form of light and dark in the vertical direction from the driving of the image sensor 101 (see FIG. 3). Therefore, it is only necessary to generate sampling data in the vertical direction in the image signal. For example, the signal is averaged for each horizontal line, and the time axis (time sampling) and the signal level are 2 as in 302 and 304. There is a method to reduce the dimension information. Alternatively, the present invention is not limited to this, and if it is considered that the user feels uncomfortable when flicker occurs in the center of the image signal, a detection frame is provided in the center of the image, and a signal is sent for each horizontal direction within the detection frame. An averaging method is also conceivable. In any case, data reduced to two-dimensional information of a time axis (time sampling) and a signal level is generated using some method. Further, when the video signal is NTSC and the frequency of the commercial power supply is 50 Hz, the flicker appearing in the image signal is 100 / 60≈1.7 cycles. For this reason, it suffices to sample all or a part of one period or more in the vertical direction of the image signal. Further, when the video signal is PAL and the commercial power supply frequency is 60 Hz, since 120/50 = 2.4 cycles, all of the vertical direction of the image signal or a part of one cycle or more may be similarly sampled.

  Next, an example of converting to the frequency domain using the obtained sampling data will be described. Assuming that the sampling parameter is N and the sampled data is Xn (n = 0, 1,... N−1), discrete Fourier transform is performed in the frequency domain with discrete Fourier coefficients expressed by the following equation.

When the time L (sec) is sampled for N times, and the sampling interval is a (sec), the Fourier coefficient after the discrete Fourier transform is obtained in the range of the discrete frequency fk expressed by the following equation.

fs is a sampling frequency in the time domain.

  Since the data sequence sampled from the image signal is a real-time signal, (Expression 1) can be separated into a real part Re and an imaginary part Im when developed, and is expressed by the following expression.

  The amplitude spectrum Ak at a certain frequency fk is obtained by the following equation from (Equation 4).

  Further, the phase component Ωk at a certain frequency fk when converted into the frequency domain is obtained by the following equation from (Equation 4).

From (Equation 5) and (Equation 6), (Equation 4) includes a frequency component and a corresponding phase component as a flicker component, and the flicker component output from the flicker detection means 103 to the correction value prediction means 104 is Represents.

  Next, the correction value prediction unit 104 predicts and corrects the correction parameter from the flicker component (amplitude / phase of each frequency) obtained from the flicker detection unit 103 and the control information of the image sensor 101 obtained from the control unit 102. We will describe how to generate values.

  When the driving state of the image sensor 101 does not change, a method for generating a correction value when detecting a flicker component from the image signal B and correcting the image signal C in FIG. As described above, flicker generated by the blinking light source appears in the image signal at a frequency twice the power supply frequency. For example, when the imaging apparatus is NTSC and the power supply frequency is 50 Hz, it repeatedly appears in the same phase every three fields. It has been known. Accordingly, the difference θk between the phase of the flicker component of the flicker generated in the image signal C and the phase of the flicker component of the flicker generated in the image signal D is expressed by the following equation. Note that m is k at the frequency fk = 100 (Hz) calculated from (Expression 2), and the phase difference θk expressed by the following expression satisfies θk = 2πfk x (1/60).

Although the amplitude value of the flicker component depends on the accumulation time of the image sensor 101, since the accumulation time is the same for the image signal B and the image signal C, it can be considered that there is no fluctuation in the amplitude value between frames. Therefore, from the above, it is possible to convert the flicker component into the time domain by predicting only the phase component of the flicker component detected from the image signal B and performing inverse discrete Fourier transform. Become. The inverse discrete Fourier transform is expressed by the following expression from the real part and the imaginary part ((Expression 4)) of the discrete Fourier transform.

  It is generally known that the flicker component is proportional to the signal intensity of the subject, and in order to cancel the flicker generated in the image signal, the flicker component in the zero-center time domain calculated by the above equation is set to 1. A correction value can be generated by calculating the reciprocal at the center. An image signal without flicker can be generated by multiplying the image signal by the generated correction value.

  On the other hand, referring to FIG. 2 again, there is a timing at which a correction value is generated from the flicker component detected in the image signal C and a correction operation is performed on the image signal D. If a correction value is generated with the flicker component detected from the image signal C having a short accumulation time and the correction value is applied to the image signal D having a long accumulation time, an appropriate correction operation cannot be performed. This is because, as is apparent from 302 and 304 in FIG. 3, the amplitude values and phases of the flicker components are different. Therefore, when generating a correction value from the detected flicker component, the image signal D is appropriately corrected by predicting the amplitude value and phase from the control information of the image sensor 101 output from the control means 102. be able to. Hereinafter, a method for predicting the correction value will be described.

(Prediction of correction value amplitude)
FIG. 4 shows the relationship between the accumulation time of the image sensor and the degree of modulation of the luminance signal by the blinking light source. The degree of modulation is obtained by normalizing the level fluctuation (amplitude) when flicker is generated by the blinking light source with the signal level when flicker is not generated. 401 indicates how much the signal level is modulated by flicker in each accumulation time. As the accumulation time becomes shorter, the time for integrating the light quantity of the flashing light source becomes shorter, so the fluctuation of the signal level for each line becomes larger, and the characteristics shown in the figure are obtained from the integration time and the cycle of the flashing light source. The correction value predicting means 104 stores the accumulation time of the image sensor 101 and the modulation degree of the luminance signal from the blinking light source in a table in advance. The table may be stored by discretizing the characteristics of 401 in FIG. 4 measured in advance, or may be learned and stored in the imaging apparatus. When the accumulation time of the image sensor 101 changes from t1 to t2, the correction value prediction unit 104 calculates the accumulation time t2 of the image sensor when the image signal to be corrected is accumulated from the control unit 102 as the control information of the image sensor. receive. The modulation degree α1 at the previous accumulation time t1 and the modulation degree α2 of the received accumulation time t2 are obtained by referring to the table, and the correction value generated in (Equation 8) is modulated by α2 / α1. By generating a new correction value, it is possible to predict the amplitude of the correction value that is the correction parameter.

(Prediction of correction value phase)
A method of predicting the phase of the correction value will be described with reference to FIG. Reference numerals 503 and 505 denote driving for resetting the electric charge accumulated in the charge conversion unit of the image sensor 101, and reference numerals 504 and 506 denote readout driving, and at times t1, 505, and 506 between 503 and 504, respectively. The sandwiched time t2 indicates the accumulation time. In order to calculate the phase difference between the flicker components of t1 and t2, the time difference A between the centroids 501 and 502 of the accumulation time during which the flashing light source is accumulated may be converted into phase information, and the period of the vertical synchronization signal is 1 / In the case of 60 s, the time difference A is expressed by the following equation.

That is, the second term of (Equation 9) represents the phase difference of the flicker component generated by the change in the accumulation time. Therefore, when calculating the correction value, in addition to the phase difference θk of (Expression 7), the component of the second term of (Expression 9) is added as the phase difference component Φk due to the change in the accumulation time. Is possible. The phase difference component Φk is expressed by the following equation when the frequency fk = 100 (Hz) calculated from (Expression 2) is Φk = 2πfk x ((t1−t2) / 2). In l which satisfies, it is expressed by the following formula.

As described above, the correction value can be generated from (Equation 8) and (Equation 10) using the following equation to predict the phase difference due to the difference in accumulation time and generate the correction value.

(Prediction of harmonic component of correction value)
So far, the embodiment has been described in which detection and correction are performed by discrete Fourier transform / inverse discrete Fourier transform in the entire frequency domain determined by the sampling frequency fs. For example, referring to FIG. 3, it can be seen that when the accumulation time is long, it approximates to a sine wave of the frequency at which the flashing light source emits light. Therefore, even when generating the correction value, without generating the inverse discrete Fourier transform over the entire frequency range, by generating the correction value from the frequency fk of the generated flicker and the Fourier series of the frequencies in the vicinity thereof, It is possible to generate a sufficient correction value. On the contrary, when the accumulation time is shortened, the flicker generated on the image signal is approximated to a shape obtained by tracing the light emission shape of the blinking light source. Therefore, the generated flicker is difficult to approximate with a simple sine wave and includes harmonic components. At this time, it is determined from the accumulation time, which is control information of the image sensor 101 obtained from the control means 102, whether the generated flicker includes harmonics. The correction value predicting means 104 stores in advance the accumulation time of the image sensor 101 and the degree of modulation of the luminance signal from the blinking light source in a plurality of tables including harmonics. Then, at the time of the inverse discrete Fourier transform when generating the correction value, the modulation factor is referred to from the table so that the harmonic component is included according to the accumulation time, and the inverse discrete Fourier transform is performed in consideration of the harmonic component. Perform conversion. With the above processing, even when flicker including a harmonic component occurs, it is possible to realize good flicker correction.

  As described above, even when the flicker component changes suddenly due to a change in the driving method of the image sensor, the flicker component can be corrected with high accuracy and the image quality can be improved.

  Although this embodiment uses a CMOS image sensor that employs an XY address type scanning method, the present invention can also be applied to the case where an image pickup device of a type that resets and reads out all pixels at the same time, such as a CCD, is used. In that case, the generated flicker appears as a periodic signal level fluctuation between captured image signals. When the accumulation time of the image sensor 101 changes, the accumulation time and the modulation degree of the luminance signal by the blinking light source are stored in a table, and a correction value is predicted and generated based on the control information output from the control means 102. As a result, good flicker correction can be realized.

[Second Embodiment]
Hereinafter, a timing chart of the second embodiment of the present invention will be described with reference to FIG. A description of the same reference numerals as those in the first embodiment is omitted. The present embodiment shows a flicker detection / correction method when an image signal for detecting flicker and an image signal to be corrected are delayed by two vertical synchronization periods.

  Consider a case where the driving state of the image sensor 101 transitions from an intermittent reading operation to a normal operation. Reference numeral 601 denotes the timing at which the flicker detection means 103 is operating, and it operates during a part of the period during which an image signal is obtained. In the intermittent reading operation, the image sensor 101 forms a frame for reading an image signal and a frame for not reading in synchronization with the vertical synchronization period, and is repeated alternately in a part of FIG. In the normal read operation, an image signal is read from the image sensor 101 every vertical synchronization period. In other words, the imaging apparatus is driven so that the frame rate changes. In the following description, it is assumed that the accumulation time does not change between the intermittent reading operation and the normal operation. Reference numeral 602 denotes a control signal output from the control means to the image sensor 101, and reference numeral 604 denotes control information output from the control means 102 to the correction value prediction means 104. Reference numeral 604 denotes a timing at which the correction value prediction unit 104 operates, and reference numerals 605 to 609 denote timings at which the correction values generated by the correction value prediction unit 104 are set.

  In this case, as the timing of flicker detection and flicker correction, a delay of two vertical synchronization periods always occurs in detection and correction. For example, flicker detection is performed from the image signal A and the image signal B is corrected, or flicker detection is performed from the image signal B and the image signal C is corrected. Therefore, when the driving state of the image sensor 101 is changed, the correction for the image signal D is not performed because the image signal to be detected cannot be generated and the correction value itself cannot be generated. Therefore, the control unit 102 and the correction value prediction unit 104 predict the correction value for the image signal D at the timing described below.

  The control means 102 outputs a control signal for the image sensor 101 to the image sensor 101 at the timing indicated by 602. Reference numeral 602c denotes a drive signal for operating only the shutter without performing the read drive, d is a drive signal for only the read drive and the shutter operation is stopped, and e is the normal read drive. If the driving state of the image sensor 101 is changed from intermittent reading to normal reading by the control unit 102, a drive signal e is output to change the driving state of the image sensor 101 at a timing indicated by 602. As indicated by reference numeral 603, the control unit 102 outputs the driving state of the image sensor 101 to the correction value prediction unit 104 as control information. The control information is composed of information (High / Low in this embodiment) that can discriminate switching of the frame rate. The correction value prediction unit 104 generates a correction value for the image signal C from the flicker component detected by the flicker detection unit 102 from the image signal B. The phase of flicker generated in the detection frame and the phase of flicker generated in the correction frame are the phase difference expressed by (Expression 7) at k when the frequency fk = 100 (Hz) calculated from (Expression 2). It has a fixed value at m where θk satisfies θk = 2πfk x (1/30). Further, if the accumulation time of the image signal obtained from the image sensor 101 is not changed, the amplitude spectrum of the flicker that is generated hardly changes. Therefore, from (Equation 8), a correction value for the image signal C is generated from the detection result of the image signal B, and the correction value is set at the timing indicated by 606.

  After generating a correction value for the image signal C in this way, information indicating that the drive state of the image sensor 101 has transitioned (transition from intermittent readout to normal readout) is received from the control information indicated by 603. And it acquires beforehand that the correction value with respect to image signal D by driving changes will be lost. Then, a correction value for the image signal D is generated based on the flicker component detected from the image signal B and the control information 603 of the image sensor 101 output from the control unit 102. The flicker component detected from the image signal B is always held in the storage means, and is read from the storage means when the control information output from the control means 102 changes. Then, after the correction value for the image signal C is generated, the correction value for the image signal D is predicted and generated at the timing indicated by 604. The generated correction value is set at the timing indicated by 607 so that the image signal D is corrected, and is corrected for the image signal D. Thus, when the frame rate is high, flicker correction is performed based on the flicker component detected from the image obtained several frames before, and when the frame rate is low, the flicker detected from the image of the frame before the number of frames. Flicker correction based on the component.

  Next, a method for predicting the phase of the correction value in the correction value predicting unit 104 in the present embodiment will be described.

(Prediction of correction value phase)
When the image signal for detecting flicker and the image signal to be corrected are delayed by two vertical synchronization periods, the phase difference between the flicker occurring in the detected frame and the flicker of the frame to be corrected is as follows. In case of, it has a fixed value. That is, at k at the frequency fk = 100 (Hz) calculated from (Expression 2), the phase difference θk expressed by (Expression 7) is at m satisfying θk = 2πfk x (1/30). . When a correction value for correcting flicker of the image signal D is generated from the detection result of the image signal B as in the present embodiment, the phase difference for one vertical synchronization period may be added to Φk. Accordingly, at k when the frequency fk = 100 (Hz) calculated from (Equation 2), the phase difference Φk expressed by (Equation 10) is fixed at l satisfying Φk = 2πfk x (1/60). Have a value. Therefore, good flicker correction can be realized by generating a correction value from (Equation 11) below.

[Third Embodiment]
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. Since the block diagram of the flicker detection / correction device is the same as that of the first embodiment, it is omitted.

  Next, a timing chart of the third embodiment of the present invention will be described with reference to FIG.

  The present embodiment is an example of control for changing a driving method of an imaging method in order to capture a still image with higher definition than a moving image, for example, during moving image capturing. For example, in order to read a moving image at high speed, there is a method of reading the image from the image pickup device by adding, averaging, or thinning the images. In addition, when reading a still image, in order to read out a high-definition image, the number of additions / averages or thinnings is reduced or non-addition compared to that for moving images, and the number of signals is increased and the reading time is longer than that for moving images. There is a method of reading with.

  Reference numeral 701 denotes a vertical synchronization signal determined by the imaging method of the imaging apparatus. In the case of the image sensor of the XY address type scanning method, since the exposure timing for each pixel is sequentially performed in the line direction, all pixels of the image sensor are sequentially read out within one vertical synchronization signal period as shown in the figure. r1 to r6 represent readout driving of the image sensor 101. Further, s1 to s6 represent driving for resetting the charges accumulated in the charge conversion unit of the image sensor 101. In order to unify the exposure time of each pixel, the reset timing is sequentially performed in the line direction as in the read driving. In this figure, the read time of r4 is longer than r1, r2, r3, r5, r6. As described above, this indicates that the readout time has changed due to a change in the driving method of the image sensor in order to switch between capturing images for moving images and still images, for example. The image signal read out at r4 corresponds to the D image signal in the figure, and the image signals read out at r1, r2, r3, r5, r6 correspond to A, B, C, E, F, respectively. .

  FIG. 8 schematically illustrates an image signal captured by the image sensor 101. Reference numeral 801 denotes a state of an image signal when the readout time from the image sensor 101 is short when capturing a moving image, and reference numeral 803 denotes a state of an image signal when the readout time from the image sensor 101 is long when capturing a still image. . Actually, since the number of pixels to be read is different, the image size is different, but in order to compare flicker periods at the same angle of view, description will be made with the same size. The signal level is modulated by the blinking light source, and periodic level fluctuations depending on the light emission period of the blinking light source occur in the image signal as indicated by 802 and 804. If the light emission cycle of the flashing light source is constant, the level fluctuation cycle that appears on the screen depends on the readout time, so it is possible to predict the level fluctuation cycle by using the readout time information as control information. Become.

  Returning to FIG. 7, the control method will be described.

  Reference numeral 203 denotes a timing at which the control means 102 outputs a control signal for the image sensor 101 to the image sensor 101. A drive system that affects the readout time, such as an addition / average / thinning system in the image sensor, is performed. Is for. In the case of the figure, for example, a control signal f for capturing an image signal C as a moving image is sent to the image sensor 101 at timing 704, and the image sensor resets and reads out at timings s3 and r3. At timing 705, a control signal g for capturing an image signal D as a still image is sent to the image sensor 101, and the image sensor resets and reads out at timings s4 and r4. At timing 706, a control signal h indicating that driving is continued is sent to the image sensor 101, and the image sensor continues resetting and reading at timings s4 and r4. At timing 707, a control signal f for capturing an image signal E as a moving image is sent to the image sensor 101, and the image sensor resets and reads out at timings s5 and r5.

  In the present embodiment, for the image signals B, C, and F obtained at the time of moving image capturing, correction values are generated from the flicker components obtained from the image signals A, B, and E obtained immediately before the timings 713, 714, 717 can be set.

  Since the image signal D obtained at the time of still image capturing has a flickering state different from that of the image signal C obtained immediately before, a correction value is predicted from the flicker component obtained from the image signal C. Further, since the image signal E obtained at the time of moving image capturing has a different flickering state from the image signal D obtained immediately before, a correction value is predicted from the flicker component obtained from the image signal D. Reference numeral 708 denotes control information sent from the control unit 102 to the correction value predicting unit 104, which represents a ratio of read times. At timing 709, the time ratio between read r3 and r4 is sent as control information i, and at time 710, the time ratio between read r4 and r5 is sent out as control information j. The correction value prediction means 104 predicts the correction value at timings 711 and 712 based on the flicker component and the control information, and sets the correction value at timings 715 and 716. In other fields, the correction value is calculated based on the flicker component and set.

  Next, a method for predicting the correction value from the flicker component obtained from the flicker detection means 103 and the control information obtained from the control means 102 in the correction value prediction means 104 will be described. The idea of obtaining the correction value from the flicker component using the inverse discrete Fourier transform expressed by Equation 8 is the same as in the first embodiment.

  When the readout time of one field 1 from the image sensor is t1, and the readout time of another field 2 is t2, the ratio of the period of level fluctuation in the screen of field 2 to field 1 is the ratio T of the readout time. = T1 / t2.

  Therefore, by using T as the control information, specifically, by dividing each term by T as represented by (Equation 12), it is possible to predict the periodic component of the correction value.

  As described above, even when the flicker component changes suddenly due to a change in the driving method of the image sensor, the flicker component can be corrected with high accuracy and the image quality can be improved.

  In the present embodiment, for the image signal E obtained at the time of moving image capturing, a correction value is predicted from the flicker component obtained from the image signal D obtained immediately before and used for correction. However, the correction value may be obtained based on the flicker component obtained from the image signal C obtained before the still image is captured. Thus, flicker correction may be appropriately performed when correction is performed based on the flicker component obtained from the previous moving image.

  As described above, in the flicker correction in which the image signal is corrected based on the flicker component detected from the image signal obtained a predetermined number of frames before the frame from which the image signal is acquired, when the drive state is changed, Change the predetermined number of frames. Thereby, more appropriate flicker correction can be performed.

  The frame from which the image signal D of the first embodiment (FIG. 2) is obtained, the frame from which the image signal D of the second embodiment (FIG. 6) is obtained, and the third embodiment (FIG. 7). A frame in which the image signal D is obtained is a transition period. This transition period indicates a period from when the driving state of the image sensor is maintained to when the driving state is changed and when the driving state is maintained.

[Other Embodiments]
It should be noted that the software configuration and the hardware configuration in the above embodiments can be appropriately replaced.

  Further, the present invention may be applied to a system including a plurality of information processing apparatuses (for example, a host computer, an interface device, a camera head, etc.). Further, the present invention may be applied to an apparatus (for example, a digital still camera, a digital video camera, etc.) composed of a single device.

  The object of the present invention can also be achieved by the system or apparatus computer reading and executing the program code stored in the storage medium storing the program code of the software that implements the functions of the above-described embodiments. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be executed by the signal processing. Cases are also included. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered.

  Further, the program code read from the storage medium may be written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After being written in the memory, the CPU of the function expansion card or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the function of the above-described embodiment is performed by the processing. Realized.

  When the present invention is applied to the above-described storage medium, the storage medium stores program codes corresponding to the processing described with reference to either FIG. 2 and FIG. 6 described above or FIG. become.

DESCRIPTION OF SYMBOLS 101 Image pick-up element 102 Control means 103 Flicker detection means 104 Correction value prediction means 105 Flicker correction means

Claims (10)

Acquisition means for acquiring a first image and a second image;
Detecting means for detecting a flicker component from the first image;
Based on the flicker component for the first image detected by the detection means and the difference in accumulation time between the first image and the second image, the flicker component for the second image Calculating means for calculating at least one of amplitude or phase;
Correction means for correcting the second image using an amplitude or phase of a flicker component with respect to the second image calculated by the calculation means;
An image processing apparatus comprising:   The calculation means is configured to store the second image based on a relationship between an accumulation time and a modulation degree of a signal from a blinking light source of an imaging element that captures the first image and the second image. The image processing apparatus according to claim 1, wherein an amplitude of a flicker component with respect to an image is calculated.   The calculation means includes a phase difference between a flicker component for the first image and a flicker component for the second image obtained from an accumulation time of the second image, the first image, and the second image. 3. The image processing apparatus according to claim 1, wherein a phase of a flicker component with respect to the second image is calculated in consideration of a phase difference based on a difference in accumulation time of the second image.   The image processing apparatus according to claim 1, wherein the first image and the second image are images captured by an XY address type image sensor.   The said 1st image and said 2nd image are the images imaged by the image sensor which respectively resets and reads all the pixels simultaneously, The one of Claim 1 thru | or 3 characterized by the above-mentioned. Image processing device.   The image processing apparatus according to claim 1, wherein the calculating unit corrects a flicker component of the second image by predicting a harmonic component of the first image. . An acquisition step of acquiring a first image and a second image;
Detecting a flicker component from the first image;
The flicker component for the second image based on the flicker component for the first image detected in the detection step and the difference in accumulation time between the first image and the second image. A calculating step for calculating at least one of the amplitude or phase of
A correction step of correcting the second image using an amplitude or phase of a flicker component with respect to the second image calculated in the calculation step;
An image processing method comprising:   A program that can be executed by an information processing apparatus, and that causes the information processing apparatus that has executed the program to function as the image processing apparatus according to any one of claims 1 to 6.   A program executable by an information processing apparatus, comprising program code for realizing the signal processing method according to claim 7.   A storage medium readable by an information processing apparatus, wherein the program according to claim 8 or 9 is stored.