JP2012257043A - Image pickup apparatus and image pickup method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup apparatus which achieves both the improvement of detection precision of a flicker component and the acquisition of a moving image having suppressed distortion.SOLUTION: An image pickup apparatus comprises: a rolling shutter type image pickup device for imaging a subject to obtain image data; a flicker detection unit for detecting a flicker component of a subject image on the basis of the image data obtained by the imaging; an operation mode setup unit for setting an operation mode out of modes including at least a flicker detection mode for detecting the flicker component and a moving image photography mode for photographing a moving image; and a curtain speed control unit for controlling a curtain speed of an electronic shutter of the image pickup device. In the curtain speed control unit, when the flicker detection mode is set, the curtain speed of the electronic shutter is set at a first curtain speed, and when the moving image photography mode is set, the curtain speed of the electronic shutter is set at a second curtain speed which is higher than the first curtain speed.

Description

  The present invention relates to a technique for controlling the curtain speed of a rolling shutter type image sensor depending on whether flicker detection or moving image shooting is performed.

  In shooting using a rolling shutter type imaging device under a flicker light source, line flicker generally occurs. The rolling shutter type imaging device is characterized in that the exposure start timing differs for each line (pixel row) of the imaging device, which causes line flicker.

  Patent Document 1 discloses a technique for extracting a flicker component from a difference between two images taken at two different shutter speeds.

JP 2010-98416 A

  Here, in order to accurately detect the flicker component, it is necessary to slow down the rolling shutter curtain speed. However, if the curtain speed is slowed down, an image obtained by moving image shooting tends to be distorted.

  An object of the present invention is to provide a technique that achieves both improvement in flicker component detection accuracy and acquisition of a moving image with suppressed distortion.

  An imaging apparatus according to an aspect of the present invention detects a flicker component of a subject image based on a rolling shutter type imaging device that captures an image of a subject and obtains image data, and image data obtained by imaging with the imaging device. An operation mode setting unit for setting an operation mode from a mode including at least a flicker detection unit for detecting a flicker by the flicker detection unit and a moving image shooting mode for shooting a moving image. And a curtain speed controller for controlling the curtain speed of the electronic shutter of the image sensor. When the flicker detection mode is set by the operation mode setting unit, the curtain speed control unit sets the curtain speed of the electronic shutter to the first curtain speed and the movie shooting mode is set. In this case, the curtain speed of the electronic shutter is set to a second curtain speed higher than the first curtain speed.

  An imaging method according to another aspect of the present invention is an imaging method in which imaging is performed by an imaging device including a rolling shutter type imaging device. Determining whether or not the processing is to detect a flicker component of the subject image, determining whether or not the processing performed by the imaging device is processing of capturing a moving image, and determining the flicker component of the subject image If it is determined that the detection process is to be performed, the curtain speed of the electronic shutter of the image sensor is set to the first curtain speed, and if it is determined to perform the process of shooting the moving image, the electronic shutter Setting the shutter curtain speed to a second curtain speed higher than the first curtain speed.

  According to the present invention, when detecting flicker, the curtain speed of the electronic shutter is set to the first curtain speed to detect flicker accurately, and when shooting a movie, the curtain speed of the electronic shutter is set. By setting the second curtain speed higher than the first curtain speed, a moving image in which distortion is suppressed can be acquired.

It is a block diagram which shows the structure of the digital camera which is an imaging device in one embodiment. It is a figure which shows an example of the exposure start timing of each line of the image sensor of a rolling shutter system. It is a figure which shows the exposure level of each line when a curtain speed is made into high speed. It is a figure which shows the exposure level of each line when a curtain speed is made into a low speed. FIGS. 5A and 5B are diagrams showing exposure levels of respective lines when moving image shooting is performed under a flicker light source. FIG. 5A shows a case where the curtain speed is high, and FIG. 5B shows a case where the curtain speed is low. FIG. FIG. 6A is a diagram illustrating an example of an image when moving image shooting is performed using a global shutter, and FIG. 6B is an example of an image when moving image shooting is performed with the curtain shutter speed being high. FIG. 6C is a diagram showing an example of an image when moving image shooting is performed with the curtain shutter speed being low. It is a flowchart which shows the processing content performed with the digital camera which is an imaging device in one Embodiment. It is a flowchart which shows the content of the process following the process of the flowchart shown to FIG. 7A. It is a figure for demonstrating the image processing which cancels a flicker.

  FIG. 1 is a block diagram illustrating a configuration of a digital camera 100 that is an imaging apparatus according to an embodiment. The digital camera 100 includes a lens 1, a diaphragm 2, an image sensor 3, an analog amplification unit 4 (hereinafter referred to as A-AMP 4), an A / D conversion unit 5 (hereinafter referred to as ADC 5), and a CPU 6. , An image processing unit 7, a video encoder 8, a liquid crystal display unit 9 (hereinafter referred to as an LCD 9), an operation unit 10, a DRAM 11, a FLASH memory 12, and a bus 13.

  The lens 1 focuses the optical image of the subject on the image sensor 3. The lens 1 may be a single focus lens or a zoom lens.

  The diaphragm 2 defines the passage range of the light beam reaching the image sensor 3 based on a command from the CPU 6.

  The image pickup device 3 is an image pickup device in which a Bayer array color filter is arranged in front of a photodiode constituting each pixel. The image pickup device 3 receives the light collected by the lens 1 by a photodiode constituting a pixel and performs photoelectric conversion, and outputs the amount of light to the A-AMP 4 as a charge amount. The image pickup device 3 is, for example, a CMOS image sensor, and performs shooting with a so-called mechanical shutter when shooting a still image, but performs shooting with a rolling shutter when shooting a moving image.

  FIG. 2 is a diagram illustrating an example of the exposure start timing of each line of the imaging device 3 of the rolling shutter system. In the rolling shutter type image pickup device 3, as shown in FIG. 2, the exposure is started for each line from the line A at the upper end of the image pickup device to the line C at the lower end. Accordingly, there is a time difference between the exposure start time of the upper end line (line A) and the exposure start time of the lower end line (line C). In this specification, the time difference between the exposure start times is referred to as “curtain speed”.

  The A-AMP 4 performs waveform shaping on the electrical signal (analog image signal) output from the image sensor 3 while reducing reset noise and the like, and further increases the gain so that the target brightness is obtained.

  The ADC 5 converts the analog image signal output from the A-AMP 4 into a digital image signal (hereinafter referred to as image data).

  The bus 13 is a transfer path for transferring various data generated in the digital camera to each unit in the digital camera. The bus 13 is connected to the ADC 5, the CPU 6, the image processing unit 7, the video encoder 8, the DRAM 11, and the FLASH memory 12.

  The image data output from the ADC 5 is temporarily stored in the DRAM 11 via the bus 13. The DRAM 11 is a storage unit that temporarily stores various data such as image data obtained by the ADC 5 and image data processed by the image processing unit 7 or the like.

  The image processing unit 7 performs various image processing such as optical black subtraction processing, white balance correction processing, synchronization processing, gamma correction processing, and color reproduction processing on the image data read from the DRAM 11. The image processing unit 7 also performs processing for detecting a flicker component from an image obtained by shooting and processing for canceling the flicker component from the image.

  The CPU 6 controls each unit in the imaging apparatus 100 in an integrated manner.

  The video encoder 8 reads the image data stored in the DRAM 11 or the FLASH memory 12, converts it into a video signal or the like, and outputs it to the LCD 9. Note that the converted video signal may be output to an external display device such as a television.

  The operation unit 10 is an operation member such as a power button, a release button, and various input keys. When any one of the operation members of the operation unit 10 is operated by the user, the CPU 6 executes various sequences according to the user's operation. The power button is an operation member for instructing power on / off of the digital camera. When the power button is pressed, the CPU 6 turns on or off the power of the digital camera. The release button has a two-stage switch of a first release switch and a second release switch. When the release button is pressed halfway and the first release switch is turned on, the CPU 6 performs a shooting preparation sequence such as AE processing and AF processing. When the release button is fully pressed and the second release switch is turned on, the CPU 6 performs shooting by executing a shooting sequence.

  The FLASH memory 12 stores various parameters necessary for the operation of the digital camera, such as a white balance gain and a low-pass filter coefficient corresponding to the white balance mode, and a manufacturing number for specifying the digital still camera. The FLASH memory 12 also stores various programs executed by the CPU 6. The CPU 6 reads parameters necessary for various sequences from the FLASH memory 12 according to a program stored in the FLASH memory 12, and executes each process. Further, the FLASH memory 12 records still images and moving images obtained by photographing.

  The FLASH memory 12 is a memory built in the main body of the digital camera 100, but may be replaced by a memory card that can be attached to and detached from the main body of the digital camera 100.

  Next, the exposure level of each line when shooting is performed under a flicker light source will be described with reference to FIGS. FIG. 3 is a diagram showing the exposure level of each line when the curtain speed is high, and FIG. 4 is a diagram showing the exposure level of each line when the curtain speed is low. 3 and 4, the flicker light source flickers at a constant cycle, and the change in luminance level with time draws the absolute value of the sine curve shape.

  When moving image shooting is performed using a rolling shutter type imaging device under a flicker light source, as shown in FIGS. 3 and 4, bright and dark horizontal stripes (line flicker) appear on the image. However, when the curtain speed is high, the difference in the exposure start timing of each line is small, so the number of horizontal stripes appearing on the image is small (see FIG. 3). On the other hand, when the curtain speed is set to a low speed, the difference in exposure start timing of each line increases, and the number of horizontal stripes appearing on the image increases (see FIG. 4).

  5A and 5B are diagrams showing the exposure level of each line when shooting is performed under a flicker light source. FIG. 5A shows a case where the curtain speed is high, and FIG. 5B shows a case where the curtain speed is low. FIG. When the curtain speed is set to a low speed, the exposure level change cycle becomes shorter than when the curtain speed is set to a high speed, and flicker detection accuracy is improved. Also, less line data is required to detect the flicker cycle.

  FIG. 6A is a diagram illustrating an example of an image when moving image shooting is performed using a global shutter, and FIG. 6B is an example of an image when moving image shooting is performed with the curtain shutter speed being high. FIG. 6C is a diagram showing an example of an image when moving image shooting is performed with the curtain shutter speed being low. Here, it is assumed that the subject is photographed while moving in parallel in the right direction, such as photographing from inside a moving car.

  In moving image shooting using a global shutter, all lines of the image sensor are exposed simultaneously, so that the subject image is not distorted as shown in FIG.

  On the other hand, in moving image shooting using a rolling shutter, as described above, there is a time difference between the exposure start time of the upper end line and the exposure start time of the lower end line. Therefore, when shooting a moving subject, the image is distorted. . In particular, when the curtain speed is low, the image distortion is larger than when the curtain speed is high (see FIGS. 6B and 6C).

  As described above, in the imaging apparatus according to the embodiment, when flicker is detected, the curtain speed is set to a low speed and flicker is detected with high accuracy. obtain.

  FIG. 7A and FIG. 7B are flowcharts showing the content of processing performed by the digital camera 100 that is the imaging apparatus in the embodiment. The process starting from step S10 is mainly performed by the CPU 6.

  In step S10 of FIG. 7A, it is determined whether or not the power button has been pressed. When the user presses the power button, the digital camera 100 is activated. If it is determined that the power button has been pressed, the process proceeds to step S20.

  In step S20, the digital camera 100 is set to the flicker detection mode. The flicker detection mode is a mode for detecting flicker.

  In step S30, the curtain speed is set to a low speed n1, and photographing using a rolling shutter is performed.

  In step S40, flicker is detected based on the image obtained by shooting at the curtain speed n1. Flicker detection is performed by the image processing unit 7 using a known method.

  In step S50, the digital camera 100 is set to the moving image shooting mode. The moving image shooting mode is a mode for shooting a moving image. Here, in order to perform live view display, it can also be referred to as a live view display mode.

  In step S60, shooting is performed with the curtain speed set to high speed n2 (n2 <n1). This shooting is so-called live view shooting.

  In step S70, it is determined whether or not flicker is detected as a result of the flicker detection process in step S40. If it is determined that no flicker is detected, the process proceeds to step S90. If it is determined that flicker is detected, the process proceeds to step S80.

  In step S80, image processing for canceling flicker is performed from an image obtained by photographing. This process is performed by the image processing unit 7. Details of image processing for canceling flicker will be described below.

  FIG. 8 is a diagram for explaining an example of image processing for canceling flicker. Here, it is assumed that the frequency of the flicker light source is 50 Hz and moving image shooting is performed at 30 fps (Frame Per Second). In this case, since the flicker level cycle is 100 Hz and the moving image shooting interval is 1/30 (second), the flicker phase change on the image is a three-frame cycle (= 1/10 sec). Accordingly, among the four images 31, 32, 33, and 34 obtained by moving image shooting at 30 fps, the flicker on the image 31 and the flicker on the image 34 are in phase.

  In order to cancel the flicker component of the image, it is necessary to calculate the correction gain of each line on the image. When the images are added for the phase period, an image with substantially uniform flicker components is obtained. In the example shown in FIG. 8, by adding the images 31, 32, and 33 for three frames, an image 35 in which the flicker components are almost uniform is obtained. By dividing the composite image 35 by the processing target image 34, a correction gain for canceling the flicker component of the processing target image 34 can be obtained. FIG. 8 shows an image 36 obtained by dividing the composite image 35 by the image 34 to be processed, and a correction gain graph 37 for each line obtained from the image 36.

  Although omitted in FIG. 8, in the actual image processing, processing for correcting the positional deviation is also performed so that the position of the subject does not shift when the images 31, 32, and 33 are combined.

  Finally, an image 38 in which the flicker component is canceled is obtained by multiplying the image 34 to be processed by the obtained correction gain.

  As described above, since shooting is performed at a high curtain speed during moving image shooting, clear line flicker as shown in the images 31 to 34 in FIG. 8 does not appear, but in FIG. 8, for ease of explanation. The images are taken when shooting at a curtain speed equivalent to a low speed.

  Another image processing method for canceling flicker will be described.

  As described above, shooting is performed at a low curtain speed when the flicker component is detected, and shooting is performed at a high curtain speed when shooting a moving image. Therefore, when canceling the flicker component from the image obtained by moving image shooting, it is necessary to calculate the cycle of the flicker component at the time of moving image shooting from the cycle of the flicker component detected at the time of detecting the flicker component.

  Assuming that the curtain speed at the time of flicker component detection is Ss, the flicker cycle on the image is Ts, and the curtain speed at the time of moving image shooting is Sf, the flicker cycle Tf on the image at the time of moving image shooting is expressed by the following equation (1). expressed.

Tf = Ts × Ss / Sf (1)
Since the flicker amplitude is also detected when the flicker component is detected, each flicker is canceled based on the detected flicker amplitude, the flicker cycle Tf obtained by the above equation (1), and the flicker phase. Calculate the line correction gain. The flicker phase can be estimated, for example, by matching the flicker shape whose amplitude and period are obtained with the flicker on the image to be processed.

  Finally, an image with the flicker component canceled is obtained by multiplying the image 34 to be processed by the obtained correction gain.

  Note that the method of canceling the detected flicker component from the image is not limited to the two methods described above.

  In step S90, a so-called live view display in which an image obtained by photographing is displayed on the LCD 9 is performed.

  In step S100, it is determined whether or not the power button has been pressed. If it is determined that the power button has been pressed by the user, the process of the flowchart is terminated. If it is determined that the power button has not been pressed, the process proceeds to step S110.

  In step S110, it is determined whether or not the moving image recording button has been pressed. The moving image recording button is a button for switching on / off of the moving image recording process. If it is determined that the moving image recording button has not been pressed, the process returns to step S60 and the live view shooting is continued. On the other hand, if it is determined that the moving image recording button has been pressed, the process proceeds to step S120 in FIG. 7B.

  In step S120, the digital camera 100 is set to the flicker detection mode.

  In step S130, shooting is performed with the curtain speed set to a low speed n1.

  In step S140, flicker is detected based on the image obtained by shooting at the curtain speed n1. Flicker detection is performed by the image processing unit 7 using a known method.

  In step S150, the digital camera 100 is set to the moving image shooting mode. Here, in order to perform a moving image recording process, the moving image shooting mode can also be referred to as a moving image recording mode.

  In step S160, shooting is performed with the curtain speed set to high speed n2 (n2 <n1).

  In step S170, it is determined whether or not flicker is detected as a result of the flicker detection process in step S140. If it is determined that no flicker is detected, the process proceeds to step S190. If it is determined that flicker is detected, the process proceeds to step S180.

  In step S180, image processing for canceling flicker is performed from an image obtained by photographing. This process is performed by the image processing unit 7. The image processing method for canceling flicker is the same as the image processing performed in step S80 in FIG. 7A, and detailed description thereof is omitted here.

  In step S190, an image obtained by shooting is recorded in the FLASH memory 12.

  In step S200, it is determined whether or not the power button has been pressed. If it is determined that the power button has been pressed by the user, the process of the flowchart is terminated. If it is determined that the power button has not been pressed, the process proceeds to step S210.

  In step S210, it is determined whether or not the moving image recording button has been pressed again. If it is determined that the moving image recording button has not been pressed, the process returns to step S160, and moving image shooting and recording processing continues. On the other hand, if it is determined that the moving image recording button has been pressed, the process proceeds to step S20 in FIG. 7A.

  As described above, according to the imaging apparatus in the embodiment, when the flicker detection mode is set in the imaging apparatus including the rolling shutter type imaging device, the curtain speed of the electronic shutter is set to the first curtain speed. When the moving image shooting mode is set, the curtain speed of the electronic shutter is set to a second curtain speed that is faster than the first curtain speed. Since flicker is detected at a curtain speed slower than the curtain speed during moving image shooting, the flicker component can be detected with high accuracy. In addition, since moving image shooting is performed at a higher curtain speed than that at the time of flicker detection, a high-quality image with less subject distortion can be obtained.

  In addition, according to the imaging device in one embodiment, when the moving image shooting mode is set, the flicker component of the subject image is removed from the image data obtained by imaging, so that the image quality with less influence of flicker is obtained. Can be obtained.

  Further, based on the flicker component detected in the state in which the flicker detection mode is set and the ratio between the first curtain speed and the second curtain speed, it is obtained by shooting in the state in which the movie shooting mode is set. The flicker component of the subject image in the obtained image data can be obtained, and the obtained flicker component can be removed. As a result, the flicker component can be accurately obtained and removed.

  Also, according to the imaging apparatus in one embodiment, when setting the moving image shooting mode, the flicker detection mode is set before setting the moving image shooting mode, and the detection of the flicker component of the subject image is completed. Set the shooting mode. Then, based on the detected flicker component, the flicker component of the subject image is removed from the image data obtained by imaging when the moving image shooting mode is set. Thus, since the flicker component is removed from the image obtained by moving image shooting based on the flicker component obtained immediately before moving image shooting, it is possible to obtain a high-quality moving image that is less affected by flicker.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment.

  In the above-described embodiment, the imaging apparatus has been described as a digital camera. However, a video camera or a movie camera may be used as long as the camera has a rolling shutter type imaging device and has a moving image shooting function. A camera built in a portable information terminal (PDA: Personal Digital Assist) or a game machine may be used.

  In the above-described embodiment, the image processing is performed to remove the flicker component from the moving image shot when the moving image shooting mode is set. However, the influence of flicker is based on the flicker component detected when the flicker detection mode is set. It is also possible to control so as to obtain a moving image with suppressed. For example, it is possible to obtain an image in which the influence of flicker is suppressed by controlling the exposure amount at the time of moving image shooting based on the detected flicker component. It is also possible to obtain an image in which the influence of flicker is suppressed by controlling the shutter speed based on the detected flicker component.

  In the embodiment described above, the CPU 6 sets the operation mode from the modes including at least the flicker detection mode and the moving image shooting mode, and controls the curtain speed of the electronic shutter according to the set operation mode. explained. However, without regard to the setting of the operation mode, even when flicker components are detected, a low curtain speed is set, and when a moving image is shot, a high curtain speed is set. Needless to say, an effect can be obtained.

3 ... Image sensor 6 ... CPU (operation mode setting unit, curtain speed control unit)
7. Image processing unit (flicker detection unit, flicker removal unit)
9 ... LCD
10 ... Operation unit 11 ... DRAM
12 ... FLASH memory 100 ... Digital camera

Claims (6)

A rolling shutter type imaging device that captures an image of a subject and obtains image data;
A flicker detection unit that detects a flicker component of a subject image based on image data obtained by imaging by the imaging element;
An operation mode setting unit for setting an operation mode from modes including at least a flicker detection mode for detecting a flicker component by the flicker detection unit and a moving image shooting mode for shooting a moving image;
A curtain speed controller for controlling the curtain speed of the electronic shutter of the image sensor;
With
When the flicker detection mode is set by the operation mode setting unit, the curtain speed control unit sets the curtain speed of the electronic shutter to the first curtain speed and the movie shooting mode is set. The curtain speed of the electronic shutter is set to a second curtain speed higher than the first curtain speed.
An imaging apparatus characterized by that. A flicker removing unit that removes the flicker component of the subject image detected by the flicker detecting unit from the image data obtained by imaging by the image sensor;
The flicker removing unit removes a flicker component of the subject image from image data obtained by imaging by the imaging element when the moving image shooting mode is set by the operation mode setting unit;
The imaging apparatus according to claim 1. The flicker removal unit is based on a flicker component detected by the flicker detection unit in a state where the flicker detection mode is set and a ratio between the first curtain speed and the second curtain speed. Obtaining the flicker component of the subject image in the image data obtained by shooting in a state in which the movie shooting mode is set, and removing the obtained flicker component;
The imaging apparatus according to claim 2. The operation mode setting unit, when setting the moving image shooting mode, sets the flicker detection mode before setting the moving image shooting mode, and detects the flicker component of the subject image by the flicker detection unit Is completed, set the video shooting mode,
The flicker removing unit removes the flicker component of the subject image from the image data obtained by imaging by the imaging device when the moving image shooting mode is set based on the flicker component detected by the flicker detection unit.
The imaging apparatus according to claim 2. The moving image shooting mode is a moving image recording mode for recording a captured moving image, or a live view display mode for performing a live view display.
The imaging apparatus according to any one of claims 1 to 4, wherein the imaging apparatus is characterized in that In an imaging method for performing imaging by an imaging device including a rolling shutter type imaging device,
Determining whether the processing performed by the imaging device is processing for detecting a flicker component of a subject image based on image data obtained by imaging;
Determining whether the process performed by the imaging apparatus is a process of capturing a moving image;
If it is determined that the process of detecting the flicker component of the subject image is to be performed, it is determined that the process of shooting the moving image is performed by setting the curtain speed of the electronic shutter of the image sensor to the first curtain speed. If so, the step of setting the curtain speed of the electronic shutter to a second curtain speed higher than the first curtain speed;
An imaging method comprising: