JP7336186B2 - IMAGING DEVICE AND CONTROL METHOD THEREOF, PROGRAM, STORAGE MEDIUM - Google Patents

Description

The present invention relates to technology for shooting HDR moving images under a flickering light source.

There are a global shutter and a slit rolling shutter in the electronic shutter system of the imaging device. When photographing is performed under a flickering light source in each method, the former causes so-called surface flicker, in which the brightness of the entire image changes due to flickering. In the latter, a so-called line flicker occurs in which a difference in brightness occurs in an image obtained by picking up an image of an object. Techniques for detecting and correcting flicker generated by each shutter method are already known in order to reduce the influence of flicker. For example, Patent Document 1 discloses a detection method and a correction method for line flicker generated by a slit rolling shutter.

By the way, in recent years, HDR (high dynamic range) synthesis, in which images captured under a plurality of exposure conditions are combined, has been proposed in order to express gradation exceeding the dynamic range of an image sensor, and the combined image is recorded as a moving image. HDR moving image technology has also been proposed. As one method for generating this HDR movie, a combination of one or more exposure conditions with a predetermined exposure step with respect to the reference luminance (for example, proper exposure and 2 steps under, etc.) There is a method of accumulating, synthesizing the read images, and recording them as one frame of a moving image. Patent Document 2 proposes a technique related to a flicker detection method and a correction method for such HDR moving image driving.

JP 2009-17213 A JP 2014-179924 A

The flicker detection method described in Patent Document 1 extracts a stationary image signal in the time direction using continuously captured images, and extracts the blinking component of the flicker by dividing one frame of the captured image. is a method for detecting flicker. However, if such flicker detection is performed in a shooting mode in which shooting is performed continuously under different exposure conditions, such as HDR moving images, the stationary image signal in the time direction will change depending on the exposure conditions. The flicker detection method described in Patent Document 2 proposes a technique for detecting flicker by providing an extractor for extracting flicker components for each exposure condition (for example, proper exposure and two-step underexposure).

However, when a flicker detector is provided for each exposure condition, there is a problem that the hardware configuration and memory scale increase as the number of combined exposure conditions increases.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and its object is to detect flicker while suppressing an increase in hardware configuration and memory size when performing HDR video shooting under a flickering light source. is.

An image pickup apparatus according to the present invention comprises an image pickup device for picking up an image of a subject and outputting an image signal, dividing means for vertically dividing an image represented by the image signal into a plurality of areas, and each of the plurality of areas. a detection means for detecting flicker based on the signal level of the corresponding image signal ; and a control means for causing the imaging device to perform moving image shooting by capturing images of a plurality of frames having exposure steps, wherein the detection means is flicker is detected from the plurality of image signals by adjusting the phase of flicker appearing in each of the plurality of image signals obtained by capturing the images of the plurality of frames, and a frame in which the level of flicker exceeds a predetermined value is detected. It is characterized in that it is determined that flicker exists when it continues for a predetermined number of frames or longer .

Advantageous Effects of Invention According to the present invention, flicker can be detected while suppressing an increase in hardware configuration and memory size when HDR moving image shooting is performed under a flickering light source.

1 is a diagram showing the block configuration of a digital camera that is an embodiment of an imaging apparatus of the present invention; FIG. FIG. 2 is a block diagram showing the configuration of a flicker detector in one embodiment; FIG. 5 is a diagram showing combinations of exposure conditions in HDR movie mode; 4 is a flowchart showing a flicker detection operation in one embodiment. FIG. 4 is a diagram for explaining flicker frequency detection processing according to an embodiment; 4 is a flowchart showing flicker frequency detection processing in one embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an imaging apparatus of the present invention will be described in detail below with reference to the accompanying drawings.

<Structure of Imaging Device>
FIG. 1 is a diagram showing the block configuration of a digital camera 100, which is an embodiment of the imaging apparatus of the present invention.

A lens group 101, which includes a mechanism for controlling exposure using an aperture and a neutral density filter, and a mechanism for controlling focus and zoom operations, forms an image of incident light on an image sensor 103 to form an optical image (object image). do. The formed optical image is converted into an electrical signal by the imaging device 103 and output to the signal processing unit 104 . At this time, the exposure of the captured image is controlled by driving the diaphragm included in the lens group 101, changing the neutral density filter, controlling the accumulation time in the image sensor 103, controlling the amplification factor of the electric signal, and the like. The signal processing unit 104 performs luminance correction such as gain correction, color correction such as white balance correction, and other signal processing on the obtained image signal, and then synthesizes the processed image signal with the image synthesis unit 105. output to The image synthesizing unit 105 generates an HDR (high dynamic range) image with an expanded dynamic range using a plurality of images input in chronological order, and outputs the HDR (high dynamic range) image to the recording processing unit 106 .

At this time, the image synthesizing unit 105 performs luminance matching by performing non-linear gradation conversion processing according to the exposure difference of the input images input in chronological order, and calculates the synthesizing ratio from the luminance values of the corresponding pixels. are determined, and synthesis is performed for each pixel. When synthesizing a plurality of images input in chronological order, it is necessary to store the images once in a storage area. The recording area for storing the image signals may exist anywhere. For example, a storage area for holding the input images may be provided inside the image synthesizing unit 105 and used for synthesizing the images input in chronological order.

In addition, although HDR moving images will be described in this embodiment, it is of course possible to operate in a normal camera mode. In that case, the image composition unit 105 may output the input image as it is without image composition. In this case, for example, an electronic device such as an external client device may be configured to generate an HDR moving image based on an image output from the digital camera 100 .

The image signal in a recordable format output from the recording processing unit 106 through the above-described processing is recorded on the recording medium 107 . This series of image processing flow is executed by controlling each block by the camera control unit 108 . The camera control unit 108 is composed of a microcomputer, and controls the entire digital camera 100 by loading a control program stored in the ROM 115 into the RAM 113 and executing it. The RAM 113 is also used as a work area for the camera control section 108 .

A camera control unit 108 receives an operation from a user through an operation unit 109, and sets parameters relating to zoom, focus, and brightness of a captured image according to the set shooting mode and subject brightness. The set parameters are transmitted to the lens control unit 102 and the exposure control unit 110, and each block is controlled so that the control positions and control states correspond to the set values. The flicker detection unit 111 receives captured images accumulated by the image sensor 103 as an input, and detects a flicker cycle (a light amount change cycle) from a brightness change cycle in a plurality of temporally continuous image data (for example, preceding and subsequent image data). to detect The details of the flicker period detection will be described later.

<Flicker detection>
Next, FIG. 2 is a block diagram showing the configuration of the flicker detector 111. As shown in FIG. A flicker detection operation will be described with reference to FIG.

When an image signal is input to the flicker detection unit 111 via the camera control unit 108, the evaluation value generation unit 201 sets an evaluation value frame for calculating signal values of color components. In an image captured by a rolling shutter type imaging device, the timing of exposure start and exposure end differs for each horizontal line, so the difference in brightness caused by blinking of the light source differs for each horizontal line. Therefore, the luminance value in the vertical direction is affected by ambient light and flickering of the flickering light source, resulting in a difference in average luminance (signal level). In order to extract the flicker component from this luminance value, the following calculation is performed by the cyclic low-pass filter processing unit 202 to extract a stationary signal component (environmental light) in the time direction.

mem=ave×k+mout×(1−k) (Formula 1)
Here, mem is a value stored in the evaluation value memory 203 as the output of the above equation, and ave represents the output result of the evaluation value generation unit 201 . k is the filter coefficient of the cyclic low-pass filter processing unit 202, and mout is the calculation result of the above equation calculated when the signal value of the image one frame before is input. By performing the above calculation for each horizontal line of the input image, a stationary signal component in the time direction can be extracted.

In the flicker component extraction unit 204, the flicker component (input The level fluctuation component of the image signal) is calculated. From the calculated flicker component, the flicker detection unit 205 generates a flicker model, which is the fluctuation characteristic of the signal level in the vertical direction.

The flicker model is a model approximating luminance variation due to flicker as a periodic function having specific amplitude w, frequency f, and phase θ in the vertical direction, for example. A sine wave (or cosine wave) is generally used as the periodic function to be modeled, since the voltage fluctuation of the AC power supply has trigonometric functions, but other ideal periodic functions may also be used. . The frequency f is determined by the frame rate and the power supply frequency of the light source. The phase θ can be calculated for each row by assuming that the variation ratio of the detected variation component is 1, and the row in which the change amount of the variation ratio in the vertical direction is 1 is set as the phase θ=0. The amplitude w is calculated from the variation ratio at π/2 and 3π/2 of the calculated phase.

<HDR video>
Next, drive control for an HDR moving image will be described with reference to FIG.

As described above, HDR moving images are captured multiple times continuously under one or more exposure conditions with a predetermined exposure step with respect to the reference exposure, and the read image signals are synthesized to create one moving image. This is a shooting mode that records as frames. FIG. 3 shows a case where an exposure step is provided by sensitivity (hereinafter referred to as ISO) and a case where an exposure step is provided by storage time (hereinafter referred to as Tv). The a-1 frame and the a-2 frame, and the b-1 frame and the b-2 frame form a pair of synthesis when generating one frame forming the HDR moving image, and are synthesized by the image synthesizing unit 105 and output as one frame image. be done.

<Operation of Imaging Device>
FIG. 4 is a flow chart showing the flicker detection operation in this embodiment. With reference to FIG. 4, a flicker detection method in the case where an exposure step is provided by Tv (accumulation time) in an HDR moving image will be described.

When the HDR moving image mode is started and flicker detection is started, the evaluation value generation unit 201 sets an evaluation frame for obtaining the average value of each color signal for the captured image signal in step S401.

In step S402, the camera control unit 108 determines whether or not the current frame is the first frame from which flicker detection is started. It is held in the value memory 203 . If it is determined in step S402 that the current frame is the accumulation timing corresponding to the second frame (second frame) after the start of flicker detection, the process advances to step S403, and the camera control unit 108 determines that the input image signal is an HDR moving image. It is determined whether or not the exposure conditions are the reference exposure conditions. FIG. 4 shows the case where the reference exposure condition is proper exposure.

If it is determined in step S403 that the input image signal is a captured image (properly exposed frame) obtained with proper exposure, the process proceeds to step S405. The details of the processing in step S405 are the same as those described in the operation of the cyclic low-pass filter processing unit 202 in FIG. 2, so description thereof will be omitted.

If it is determined in step S403 that the captured image (image signal) is not a proper exposure frame but a predetermined exposure step with respect to proper exposure, the evaluation value generation unit 201 performs evaluation value gain adjustment processing in step S404. Here, a gain is applied to the evaluation value and the difference in exposure steps from the reference frame. Amplifies the signal by applying a gain of 4 times, which is the reciprocal of it. When the evaluation value gain adjustment processing is completed, the cyclic low-pass filter processing unit 202 performs low-pass filter processing in step S405.

After the processing of step S405 is completed, the camera control unit 108 holds the evaluation value as the output of the cyclic low-pass filter processing unit 202 in the evaluation value memory 203 in step S406. After that, proceeding to step S407, the flicker component extraction unit 204 extracts a flicker component from the difference between the evaluation value of the input image signal and the evaluation value held in step S406. The details of step S407 are also the same as those described in the operation of the flicker component extraction unit 204 in FIG. 2, so description thereof will be omitted.

When flicker component extraction is completed, the process advances to step S408 to determine the frequency of the flicker extracted by the flicker detection unit 205 (that is, the flicker light amount change period), and flicker detection ends. This flicker frequency detection processing will be described later with reference to FIGS. 5 and 6. FIG. This detection result is fed back to Tv (accumulation time) and image processing in order to remove flicker, and as a result, a frame in which flicker disappears appears. continue detection in the frame

The flicker frequency (light amount change period) detection process in step S408 of FIG. 4 will be described with reference to FIG. The vertical axis is the time axis, and VD is the vertical synchronization signal. FIG. 5 shows how the image data of the first frame is read from time t1 to time t1', and the image data of the second frame is read from time t2 to time t2'. When the subject is under a flickering light source, the image data has a luminance change for each line as shown in FIG. If the power frequency is f [Hz], the luminance change of the flickering light source is
L(t)=A sin ² (2πft)dt (A is a constant)
can be modeled as When this is accumulated for Δt from time (t−Δt) to t, the signal value is

and becomes a signal with a frequency of 2f [Hz]. here,

is. By detecting this frequency component from the image data, the presence or absence of flicker at the power supply frequency f [Hz] is detected.

For this purpose, in step S401, the evaluation value generating unit 201 sets an evaluation frame for each of the regions obtained by vertically dividing the image read out for each frame by N, and obtains the average value of the signal for each divided region. In FIG. 5, N=16, and the average value of divided regions n (1≦n≦N) obtained from the image read out in the m-th frame is denoted as I[m][n]. This I[m][n] is used as an evaluation value for flicker detection.

When detecting the presence or absence of flicker at power supply frequency f[Hz], the flicker level is calculated from I[m][n] and the corresponding sine wave (and cosine wave) value at 2f[Hz]. Then, it is determined that there is flicker when frames having a flicker level exceeding a predetermined value continue for a predetermined number of frames.

FIG. 6 is a flow chart showing the detailed operation of the flicker frequency detection process in step S408 of FIG.

In step S601, the flicker detection unit 205 sets a variable dt_start. dt_start is a value that indicates how far the phase of the 2f [Hz] sine wave advances between the last evaluation value of frame m and the first evaluation value of frame m+1. dt_start is set to a value based on dt_start_org, which is determined by the readout mode of the image sensor, plus θ that depends on the accumulation time. For dt_start_org, as shown in FIG. 5, for example, if it takes R [ms] to read an image at a frame rate of F [fps],
dt_start_org={1/FR×(N−1)/N}×4πf
should be set as If the storage time is the same between frames, no phase shift occurs between frames, so there is no problem if dt_start=dt_start_org. However, as shown in (Eq.2) and (Eq.4), the flicker component of the image signal has a phase shift of .theta. Therefore, θ that satisfies (Equation 4) is calculated according to the accumulation time, and the phase shift is adjusted by reflecting it in dt_start, making it possible to detect flicker even when the accumulation time differs from frame to frame, such as in HDR video. and

For example, when trying to detect a flicker with a power frequency of 50 [Hz] with an accumulation time of 1/120 [s],
4πfΔt=4π×50/120=5/3×π
Substituting into (Formula 4) yields cos θ=−1/2 and sin θ=(√3)/2, so
θ=(2/3)×π
becomes.

After setting dt_start in step S601, the flicker detection unit 205 initializes a variable cnt to 0 in step S602 prior to a loop that processes N evaluation values.

In step S603, the camera control unit 108 determines whether or not cnt is smaller than N. If smaller than N, the process proceeds to step S604 to continue the loop.

In step S604, the camera control unit 108 determines whether or not cnt is 0. If it is 0, the process proceeds to step S605 to perform processing for the first time of the loop, and if not 0, the process proceeds to step S606.

In step S605, the flicker detection unit 205 sets variables wt_sin and wt_cos. wt_sin indicates the phase of the 2f[Hz] sine wave corresponding to the evaluation value, and wt_cos indicates the phase of the 2f[Hz] cosine wave corresponding to the evaluation value.

In step S605, the flicker detection unit 205 adds dt_start to wt stored in the previous frame in order to generate the phase for the first evaluation value (I[m][1] in FIG. 5) in that frame. In addition, obtain wt_sin. As will be described later, since the phase corresponding to the last evaluation value of the previous frame is stored in wt in step S612, the phase corresponding to the first evaluation value can be obtained by adding dt_start to this. A cosine wave is expressed by setting wt_cos to a value obtained by shifting the phase of wt_sin by π/2.

Except for the first estimate of the frame, wt_sin and wt_cos are set in step S606. In step S606, the flicker detection unit 205 updates wt_sin and wt_cos by adding dt to wt_sin and wt_cos at that time. As shown in FIG. 5, dt corresponds to the time to read out one divided area, and is a value determined by the readout mode of the sensor.
dt=R/N×4πf
should be set as

In step S607, similarly to step S403, the camera control unit 108 determines whether or not the frame is a proper exposure frame. k is a coefficient by which the evaluation value of the improperly exposed frame is multiplied in order to adjust the amplitude deviation due to the difference in accumulation time between the properly exposed frame and the improperly exposed frame. Here, let Δt1 be the accumulation time of frames that are not properly exposed in an HDR moving image, and Δt2=aΔt1 be the accumulation time of frames that are properly exposed. Then, when the signal value other than the flicker component of the frame not properly exposed is E1, the signal value other than the flicker component of the properly exposed frame E2=aE1. Therefore, the signal value S1 of the image data of the frame not properly exposed is ( Combined with the flicker component of equation 2),

The signal value S2 of the image data of the proper exposure frame is

It can be expressed as. here,

is. By transforming the formula, it becomes as follows.

From this, the DC component aA/2Δt1+aE1=A/2Δt2+E2 is obtained from the evaluation value of the properly exposed frame and a times the evaluation value of the frame that is not properly exposed, and the component other than that is obtained using r2/(ar1). to correct. It can be seen that this makes it possible to even out the amplitudes that have shifted due to the difference in accumulation time, and to enable stable flicker detection.

Since the gain multiplied in step S404 corresponds to a, mem (see (Equation 1)) stored in the evaluation value memory through steps S403 to S406 corresponds to the DC component here. Based on this idea, r2/(ar1) is obtained according to the equation from the accumulation time of the properly exposed frame and the accumulation time of the frame that is not properly exposed, and k is set. On the long side, k is set to 1 in step S609. This processing means that the evaluation value is used as it is without being multiplied by the coefficient for amplitude adjustment in the calculation using the evaluation value of the properly exposed frame.

In step S610, the flicker detection unit 205 uses the phases wt_sin and wt_cos set in steps S605 and S606 to multiply and integrate the evaluation value by the sine wave and cosine wave of frequency 2f [Hz]. Itg[cnt] is the evaluation value obtained in the frame, and mem[cnt] is the stationary signal component stored in the evaluation value memory in step S406, which are obtained and stored for each of the N divided regions. ds is the result of multiplying the value obtained by subtracting the stationary signal component from the evaluation value by the coefficient k and the sin wave (sin(wt_sin)), and dc is the result of multiplying the result by the cosine wave (sin(wt_cos)). Get in. ds and dc are initialized to 0 when the camera is started, and the integration process is performed each time an evaluation value is obtained.

After updating ds and dc, the flicker detection unit 205 increments cnt in step S611 and returns to step S603 to process the next evaluation value.

When the processing for N evaluation values is completed, the process proceeds to step S612, and wt_sin at that time is stored in wt. The stored wt is used in the next frame to calculate wt_sin, wt_cos in step S605. Note that wt is initialized to 0 when the camera is started.

In step S613, the flicker detection unit 205 obtains the sum of squares of ds and dc, and sets this as the flicker level F. FIG. In step S614, the camera control unit 108 determines whether frames in which F exceeds Th2 have continued for Th1 frames. flicker frequency detection processing (step S408).

If the condition is satisfied in step S614, the flicker detection unit 205 stores the flicker detection state in step S616, and ends the flicker frequency detection processing (step S408) for that frame.

As described above, according to the present embodiment, flicker can be detected while suppressing an increase in the hardware configuration and memory in a shooting mode in which different exposure conditions are continuously captured, such as an HDR moving image. It is also possible to detect flicker using only images with the same exposure conditions and using one flicker detector, but this method reduces the number of frames used for flicker detection. The more stably the same flicker frequency is detected in a plurality of frames, the higher the reliability. can be detected.

Note that in this embodiment, the reference frame is the properly exposed frame, and the gain adjustment process for amplifying the image signal is performed in step S404. However, the exposure condition used as the reference frame may be any combination of exposure conditions for HDR synthesis, and processing such as attenuating the image signal may be performed in step S404.

(Other embodiments)
In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

101: lens group, 102: lens control unit, 103: image sensor, 104: signal processing unit, 105: image synthesizing unit, 108: camera control unit, 110: exposure control unit, 111: flicker detection unit, 201: evaluation value Calculation unit 202: cyclic low-pass filter processing unit 203: evaluation value memory 204: flicker component extraction unit 205: flicker detection unit

Claims (11)

an imaging device that captures an image of a subject and outputs an image signal;
dividing means for vertically dividing an image represented by the image signal into a plurality of regions;
detection means for detecting flicker based on the signal level of the image signal corresponding to each of the plurality of areas;
a control means for causing the image sensor to perform video shooting of capturing images of a plurality of frames having exposure steps,
The detection means detects flicker from the plurality of image signals by adjusting the phase of flicker appearing in each of the plurality of image signals obtained by capturing the images of the plurality of frames, and the level of the flicker is a predetermined value. is continued for a predetermined number of frames or more, it is determined that flicker exists. 2. The method according to claim 1, wherein said detection means adjusts phases of flicker in said plurality of image signals based on a difference between a signal readout time and a frame rate in each of said plurality of frame images. The imaging device described. The detection means detects the phases of flickers in the plurality of image signals by calculating the phases of flickers progressing during the difference between the signal readout time and the frame rate in each of the plurality of frame images. 3. The image pickup apparatus according to claim 2, wherein adjustment is performed. 3. The image pickup apparatus according to claim 2, wherein said detection means adjusts phases of flickers in said plurality of image signals based on a difference in accumulation time between images of said plurality of frames. The detecting means adjusts the phase of flicker in the plurality of image signals by calculating the phase of flicker advancing during the time difference between the accumulation times in each of the images of the plurality of frames. 5. The imaging device according to claim 4. The detection means detects flicker from the plurality of image signals by correcting a difference in signal level of the plurality of image signals based on a difference in accumulation time between images of the plurality of frames. 5. The imaging device according to claim 4. 7. The controller according to any one of claims 1 to 6, wherein the control unit changes the accumulation time or sensitivity for each imaging of the plurality of frame images, and performs the imaging of the plurality of frame images. 10. The image pickup device according to claim 1. 8. The apparatus according to claim 7, further comprising synthesizing means for synthesizing an HDR (high dynamic range) image by synthesizing a plurality of image signals with different brightness obtained by imaging the images of the plurality of frames. Imaging device. A method for controlling an imaging device having an imaging device that captures an image of a subject and outputs an image signal, comprising:
a dividing step in which the dividing means vertically divides the image represented by the image signal into a plurality of regions;
a detection step in which the detection means detects flicker based on the signal level of the image signal corresponding to each of the plurality of areas;
a control step in which the control means causes the imaging element to perform moving image shooting, in which images of a plurality of frames having exposure steps are captured;
In the detecting step, the detecting means detects flicker from the plurality of image signals by adjusting the phase of flicker appearing in each of the plurality of image signals obtained by imaging the plurality of frame images, and detecting the flicker. 1. A control method for an imaging apparatus, comprising determining that flicker exists when frames whose level exceeds a predetermined value continue for a predetermined number of frames or more. A program for causing a computer to execute each step of the control method according to claim 9. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 9.

Images