JP6587391B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention particularly relates to an imaging apparatus suitable for use in imaging under a light source whose luminance changes periodically, an imaging apparatus control method, and a program.

  2. Description of the Related Art Conventionally, a digital camera or the like that employs a rolling shutter system in which exposure start timing differs for each line is known. When such a camera is used to photograph under a light source (hereinafter referred to as flicker light source) whose luminance changes periodically such as a fluorescent lamp, uneven luminance or color unevenness may occur within the screen (hereinafter referred to as flicker). . Therefore, a method for correcting the flicker has been proposed. For example, Patent Document 1 discloses a technique of synchronizing the fluctuation of the light amount of the light source and the drive timing of the shutter, exposing at the timing when the light amount of the light source becomes the largest, and shooting so that the luminance difference in the screen is reduced. Has been.

JP 2010-74484 A

  In flicker light sources such as fluorescent lamps, since the afterglow characteristics of fluorescent substances in fluorescent lamps are wavelength-dependent, the RGB ratio (R / G, B / G) generally differs depending on each phase of the luminance fluctuation period. Is. In the prior art disclosed in Patent Document 1 described above, since the RGB ratio is different for each phase of the flicker light source, there is a possibility that photographing with little color unevenness in the screen cannot be performed.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to enable photographing with little color unevenness using a flicker light source.

An imaging apparatus according to the present invention detects an image capturing unit that captures a subject image to generate image data, and detects a change in color of the subject image captured by the imaging unit under a light source whose luminance changes periodically. and one of the detecting means, color and variation in detected by the first detecting means, a first calculation means on the basis of the shutter speed, and calculates the color ratio evaluation value for each phase under the light source, the The brightness for each phase under the light source based on the second detection means for detecting the brightness fluctuation of the subject image under the light source, the brightness fluctuation detected by the second detection means, and the shutter speed. Based on the second calculation means for calculating the ratio evaluation value, the color ratio evaluation value calculated by the first calculation means, and the luminance ratio evaluation value calculated by the second calculation means , the imaging Means And having a control means for controlling the optical timing, the.

  According to the present invention, it is possible to perform shooting with little color unevenness in a flicker light source.

It is a flowchart which shows an example of the process sequence of the still image photography by the digital camera which concerns on embodiment. It is a block diagram which shows the structural example of the digital camera which concerns on embodiment. It is a figure for demonstrating the period of a flicker. It is the figure which showed typically the relationship between a light source luminance value and a flicker table. It is a figure which shows the result of having predicted the flicker table computed for every RGB, and V mapping for every RGB. It is a figure for demonstrating the procedure which produces | generates an exposure timing evaluation value table. It is a figure for demonstrating the relationship between the in-plane or period average color ratio, and its evaluation value. It is a flowchart which shows an example of the process sequence which calculates a color ratio evaluation value. It is a flowchart which shows an example of the detailed process sequence which correct | amends a flicker for every line. It is a figure for demonstrating the relationship between a similarity and a weighting coefficient.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
FIG. 2 is a block diagram illustrating a configuration example of the digital camera 100 according to the present embodiment as an example of the imaging apparatus.
As shown in FIG. 2, a lens unit 200 is attached to the digital camera 100, and the lens unit 200 includes a photographing lens 1 and a photographing lens aperture 2.

  The light of the subject image incident on the digital camera 100 via the lens unit 200 is imaged on the image sensor 8 via the mechanical shutter 7. The image sensor 8 is, for example, a CCD sensor or the like, and has a Bayer array configuration in which R, G, and B pixels are regularly arranged in a checkered pattern. The mirror 3 is moved when an optical image is formed on the image sensor 8 and is excluded from the optical axis. The mirror 3 is a semi-transmissive half mirror, and the light reflected by the mirror 3 is diffused from the focus plate 14 and enters the pentaprism 4. The pentaprism 4 reflects the diffused incident light beam to reach the photometric sensor 5 that measures the luminance distribution of the object to be photographed.

  The photometric sensor 5 is, for example, a CCD sensor or the like, and has a Bayer array configuration in which R, G, and B pixels are regularly arranged in a checkered pattern. For this reason, luminance information and color information of the subject can be output. The photometric sensor 5 is a global shutter system in which the timing of charge accumulation time is the same in all pixels, whereas the image sensor 8 is a rolling shutter system in which the timing of starting the accumulation time of each line moves sequentially. The photometric sensor 5 operates in two operation modes, a photometric mode for automatic exposure control of the digital camera 100 and a flicker detection mode for detecting flicker, and the control unit 9 operates in these operation modes. Switch.

  The control unit 9 controls the entire digital camera 100. The control unit 9 also plays a role of determining the aperture value of the photographing lens aperture 2, the shutter speed of the mechanical shutter 7, and the shutter start timing based on the output from the photometric sensor 5. Based on an instruction from the control unit 9, the shutter drive unit 10 causes the shutter front curtain and the shutter rear curtain of the mechanical shutter 7 to travel in the vertical direction and controls the exposure time.

  The operation unit 21 includes a shutter button and other buttons for instructing the digital camera 100 to start photographing. When the shutter button is pressed halfway, a still image shooting preparation switch (SW1) for starting preparation for still image shooting is turned on. When the shutter button is fully pressed, the still image shooting switch (SW2) for starting still image shooting is turned on.

  The image processing unit 11 performs white balance processing that applies a uniform gain for each color on each of the RGB pixels in the screen of the output signal from the image sensor 8. Further, the image processing unit 11 has a function of adjusting the white balance gain for each color and for each line. In addition, the image processing unit 11 further performs interpolation processing, color conversion processing, gradation conversion processing, compression processing, and the like on the output signal from the image sensor 8 and stores the processed image data in the memory 12. The recording unit 13 records the image data output from the image processing unit 11 on a portable medium (not shown).

  The flicker detection unit 15 detects flicker based on the output of the photometric sensor 5. Here, in the flicker detection mode, the control unit 9 drives the photometric sensor 5 in order to detect flicker.

  FIG. 3 is a diagram for explaining the flicker cycle. The flicker cycle in the case of a light source with an AC power supply frequency is an inverse of twice the power supply frequency. Therefore, as shown in FIGS. 3A and 3B, the flicker cycle is 10 msec when the power supply frequency is 50 Hz, and is 8.33 msec when 60 Hz. In the flicker detection mode, the flicker detection unit 15 periodically repeats accumulation / readout operations from the photometric sensor 5 at a period of 1.67 msec in order to detect flickers of these two power supply frequencies. FIG. 3A shows the operation in the flicker detection mode when the power supply frequency is 50 Hz, and FIG. 3B shows the operation in the flicker detection mode when it is 60 Hz.

  The flicker detection unit 15 detects the presence / absence of flicker, the period, and the phase from the temporal variation of the value obtained by integrating the output of the entire screen read from the photometric sensor 5 (hereinafter, the light source luminance value). Here, the output of the entire screen is integrated so that the moving subject does not affect the light source luminance value. In the present embodiment, integration is performed for each color of the RGB pixels read out from the photometric sensor 5, and thus the light source luminance value of each RGB signal is actually obtained. At this time, if the temporal fluctuation width of the light source luminance value is less than the predetermined value, it is determined that there is no flicker. The period of reading from the photometric sensor 5 may be (1.67 / N) msec (N: integer). If N ≧ 2, the flicker cycle and phase can be detected with higher accuracy than when N = 1. Furthermore, when generating a flicker table, which will be described later, the flicker table can be generated with higher accuracy if N is increased. Therefore, the accumulation / reading operation from the photometric sensor 5 is repeated at a high speed within a feasible range. To do.

  The flicker table generation unit 6 generates a flicker table at each shutter speed based on the light source luminance value calculated by the flicker detection unit 15. When the light source luminance value at the time t is f (t) and the shutter speed is Tv, the flicker table FT (Tv, t) at the time t and the shutter speed Tv is expressed by the following equation (1).

  FIG. 4 is a diagram schematically showing the relationship between the light source luminance value and the flicker table. As shown in FIG. 4, as the exposure time becomes longer due to the integration effect, the variation in luminance within one cycle becomes smaller. In the present embodiment, the flicker table FT (Tv, t) is obtained for each RGB signal.

  Based on the flicker table generated by the flicker table generation unit 6, the mapping prediction unit 17 predicts a result (V mapping) integrated for each line of the image sensor 8. As described above, since the image sensor 8 is a rolling shutter system in which the start timing of the accumulation time of each line sequentially moves, exposure of each line of the image sensor 8 is performed as shown in FIG. Further, the time-flicker table and the V coordinate (Y coordinate of the line) -V mapping of the image sensor 8 have a high correlation as shown in FIG. FIG. 5 shows the flicker table calculated for each RGB of the photometric sensor 5 and the result of predicting the V mapping for each RGB of the image sensor 8.

  When the traveling speed of the mechanical shutter 7 in the imaging surface is constant and the spectral characteristics of the photometric sensor 5 and the imaging element 8 are equal, the flicker table and the V mapping completely match. The traveling speed of the mechanical shutter 7 is generally not constant. However, if information on the traveling speed measured in advance is stored in a nonvolatile memory (not shown), the mapping predicting unit 17 converts the flicker table to the V mapping. Conversion is possible. Even if the photometric sensor 5 and the image sensor 8 have different spectral characteristics, the mapping predicting unit 17 can convert the flicker table to the V mapping. In this case, the RGB color ratios of the photometric sensor 5 and the image sensor 8 are measured in advance under various light sources. Then, it can be converted into a V map by multiplying the flicker table with a gain that corrects the RGB color ratio based on the color temperature estimated from the photometric sensor 5 or the image sensor 8. Hereinafter, in order to simplify the description, the description will be made on the assumption that the traveling speed of the mechanical shutter 7 in the imaging surface is constant, and that the photometric sensor 5 and the imaging element 8 have the same spectral characteristics.

  The color ratio evaluation unit 19 calculates a color ratio evaluation value based on the flicker table generated by the flicker table generation unit 6. Here, a processing procedure for calculating the color ratio evaluation value will be described with reference to FIG.

FIG. 8 is a flowchart illustrating an example of a processing procedure for calculating a color ratio evaluation value by the color ratio evaluation unit 19.
First, calculation of a color ratio evaluation value is started in S800. Next, in S801, the in-plane color ratio is calculated from the flicker table. Here, the in-plane color ratio refers to the ratio between the maximum value and the minimum value of the color ratio (R / G, B / G) in the screen caused by the influence of the flicker light source. For example, when the flicker table as shown in FIG. 6A is generated by the flicker table generator 6, the in-plane as shown in FIG. 6B is obtained from the flicker table and the traveling speed of the mechanical shutter 7. The color ratio can be calculated.

  In step S802, an in-plane color ratio evaluation value is calculated from the in-plane color ratio calculated in step S801 and the color ratio evaluation value table shown in FIG. Here, the in-plane color ratio evaluation values of R / G and B / G are CinR and CinB, respectively. Further, the color ratio evaluation value table is stored in advance in a non-illustrated nonvolatile memory, and the in-plane color ratio is ideally set to 1, so that the in-plane color ratio is 1 and takes the maximum value. Such a color ratio evaluation value table is used.

  In step S803, the period average color ratio is calculated from the flicker table. Here, the period average color ratio is a ratio (R / G, B / G) of a value obtained by integrating the RGB value of the flicker table over one period of the flicker light source and a value obtained by integrating the RGB value of the flicker table during the exposure period. ). As described above, the period average color ratio varies depending on the shutter speed. When the flicker table shown in FIG. 6A is generated, the period average color ratio is calculated as shown in FIG.

  Next, in S804, a periodic average color ratio evaluation value is calculated from the periodic average color ratio calculated in S803 and the color ratio evaluation value table shown in FIG. Here, R / G and B / G period average color ratio evaluation values are CaveR and CaveB, respectively. In the present embodiment, the color ratio evaluation value tables used in S802 and S804 are the same, but different color ratio evaluation value tables may be used.

Next, in S805, the color ratio evaluation value C is calculated from the in-plane color ratio evaluation values CinR and CinB calculated in S802 and the periodic average color ratio evaluation values CaveR and CaveB calculated in S804 by the following formula (2). The calculation is completed, and the process ends (S806). FIG. 6D shows an example of the calculation result of the color ratio evaluation value C.
C = cw1 * CinR + cw2 * CinB + cw3 * CaveR + cw4 * CaveB Formula (2)
Here, cw1, cw2, cw3, and cw4 represent weighting factors.

The luminance ratio evaluation unit 20 calculates an in-plane luminance ratio evaluation value and a periodic average luminance ratio evaluation value for the luminance signal Y generated from the RGB signal of the flicker table. The in-plane luminance ratio evaluation value and the periodic average luminance ratio evaluation value are obtained by the same calculation method as the in-plane color ratio evaluation value and the periodic average color ratio evaluation value, respectively. The luminance ratio evaluation value L is calculated by the following equation (3). FIG. 6E shows an example of the calculation result of the luminance ratio evaluation value L.
L = lw1 × Lin + lw2 × Lave Equation (3)
Here, Lin represents the in-plane luminance ratio evaluation value, and Lave represents the periodic average luminance ratio evaluation value. Further, lw1 and lw2 represent weighting factors.

  The exposure timing calculation unit 16 calculates an exposure timing evaluation value table ET based on the color ratio evaluation value C calculated by the color ratio evaluation unit 19 and the luminance ratio evaluation value L calculated by the luminance ratio evaluation unit 20. In the present embodiment, the exposure timing evaluation value table ET is the sum of the color ratio evaluation value C and the luminance ratio evaluation value L. FIG. 6F shows an example of the exposure timing evaluation value table. The control unit 9 controls the shutter drive unit 10 based on the exposure timing evaluation value table ET, and causes the image sensor 8 to be exposed at the exposure timing Te at which the exposure timing evaluation value table is maximum as shown in FIG. .

Hereinafter, the flicker correction method according to the present embodiment will be described with reference to FIG.
FIG. 1 is a flowchart illustrating an example of a processing procedure of still image shooting by the digital camera 100 according to the present embodiment.
First, in S101, the process waits until the still image shooting preparation switch (SW1) for starting preparation for still image shooting is turned on. When SW1 is turned on, in S102, the control unit 9 operates the photometric sensor 5 in the flicker detection mode described above.

  Next, in S <b> 103, the flicker detection unit 15 determines the presence / absence of flicker by the above-described flicker detection method based on the output from the photometric sensor 5. If the result of this determination is that flicker has not been detected, the controller 9 determines in S115 whether or not the still image shooting switch (SW2) has been turned on. If the result of this determination is OFF, in S116, the controller 9 determines whether or not a predetermined time has elapsed since the still image shooting preparation switch (SW1) was turned ON. As a result of this determination, if the predetermined time has not elapsed, the process returns to S115, and if the predetermined time has elapsed, the process returns to S101. On the other hand, if SW2 is turned on as a result of the determination in S115, in S117, the control unit 9 instructs the photometric sensor 5 to operate in the photometric mode, and normal imaging is performed according to the control of the control unit 9. The operation is performed, and the process ends in S111.

  On the other hand, if flicker is detected as a result of the determination in S103, the flicker table generating unit 6 generates a flicker table for each shutter speed in S104. In step S105, the color ratio evaluation unit 19, the luminance ratio evaluation unit 20, and the exposure timing calculation unit 16 calculate an exposure timing evaluation value table ET according to the above-described procedure.

  Here, an example of the relationship between the weighting coefficients cw1, cw2, cw3, and cw4 of Expression (2) and the weighting coefficients lw1 and lw2 of Expression (3) related to the calculation method of the exposure timing evaluation value table ET is shown in FIG. Will be described with reference to FIG. The control unit 9 obtains these weighting coefficients using the degree of similarity with the previous shooting, and obtains the degree of similarity using the photometric sensor 5. Specifically, the screen of the photometric sensor 5 is divided into 10 × 10 small areas, the integral value for each area is obtained for each of the previous shooting and the current shooting, and the difference absolute value sum is calculated as the similarity. And

  First, it is considered that it is desirable to satisfy cw1 and cw2> lw1 because the color shift in the plane has a larger influence on the image quality than the luminance shift in the plane. Furthermore, when the degree of similarity with the previous shooting is low, it is considered that the in-plane luminance and the color shift should be emphasized rather than the luminance and the color shift with the previous shooting, so cw1, cw2> cw3 , Cw4 is desirable. However, when the degree of similarity with the previous shooting is high, especially when shooting continuously at high speed, it is better to focus on the brightness and color shift from the previous shooting than the in-plane shift. If it is high, it is considered preferable to set cw1, cw2 <cw3, cw4. Based on the above thinking, the weighting coefficient is changed as shown in FIG. 10 according to the degree of similarity with the previous shooting.

  Next, in S106, the control unit 9 determines whether or not the still image shooting switch (SW2) is turned on. If the result of this determination is OFF, in S107, the control unit 9 determines whether or not a predetermined time has elapsed since the exposure timing evaluation value table ET was previously generated in S105. If it is determined that the predetermined time has not elapsed, the process returns to S106. If the predetermined time has elapsed, the calculated exposure timing evaluation value table ET is discarded and the process returns to S101.

  On the other hand, if SW2 is turned on as a result of the determination in S106, the control unit 9 instructs the photometric sensor 5 to operate in the photometric mode in S108. Further, the control unit 9 performs photometric calculation based on the output of the flicker detection unit 15 and determines the shutter speed (Tv value). In S109, the control unit 9 determines the timing at which the maximum value E of the exposure timing evaluation value table ET corresponding to the shutter speed Tv calculated in S108 is taken as the exposure timing Te, and starts photographing processing.

  Next, in S110, the control unit 9 determines whether or not the maximum value E of the exposure timing evaluation value table ET is equal to or greater than a threshold value. As a result of this determination, if the maximum value E of the exposure timing evaluation value table ET is equal to or greater than the threshold value, it can be considered that no flicker occurs in the screen when exposure is performed at the exposure timing Te, so the processing is terminated as it is (S111). .

  On the other hand, if the maximum value E of the exposure timing evaluation value table ET is smaller than the threshold value as a result of the determination in S110, in S112, the control unit 9 determines whether the in-plane color ratio evaluation values CinR and CinB are equal to or larger than the threshold value. Determine. If the in-plane color ratio evaluation values CinR and CinB are equal to or greater than the threshold value as a result of this determination, in S113, the image processing unit 11 performs flicker correction by applying a uniform gain to each RGB pixel in the screen. Here, the gain used in S113 is the reciprocal of the periodic average color ratio evaluation values CaveR and CaveB calculated by the color ratio evaluation unit 19 in S802 of FIG. 8, and the image processing unit 11 applies this gain as a white balance gain. To correct flicker.

  On the other hand, if at least one of the in-plane color ratio evaluation values CinR and CinB is smaller than the threshold value as a result of the determination in S112, sufficient flicker correction cannot be performed only by controlling the exposure timing. Accordingly, in S114, the flicker is corrected by changing the white balance gain for each line.

FIG. 9 is a flowchart illustrating an example of a detailed processing procedure for correcting flicker for each line in S114.
First, in S901, the mapping predicting unit 17 generates a V mapping based on the flicker table generated by the flicker table generating unit 6 and the timing when SW2 is turned on. In step S902, the image processing unit 11 initializes the first line.

In step S <b> 903, the image processing unit 11 performs the following correction on each pixel on the i-th line of the image sensor 8 (L × M pixel).
R (i, j) = R (i, j) × (Rm / Ri) × max (Rp / Rm, Bp / Bm)
G (i, j) = G (i, j) × (Gp / Gi) × max (Rp / Rm, Bp / Bm)
B (i, j) = B (i, j) × (Bm / Bi) × max (Rp / Rm, Bp / Bm)
i = 1 to M, j = 1 to L.

  Here, Ri, Gi, and Bi represent the values of the V mapping of the i-th line. When performing inter-frame correction when the degree of similarity with the previous shooting is high, Rp, Bp, and Gp are values when R, G, and B are the largest in the flicker table, and Rm and Bm are respectively R and B values when G of the flicker table is the largest. On the other hand, when performing in-plane correction when the degree of similarity with the previous shooting is low, Rp, Bp, and Gp are values of R, G, and B when the flicker table is the largest in the imaging plane, respectively. is there. Rm and Bm are values of R and B when G of the flicker table becomes the largest in the imaging surface.

  In step S904, the image processing unit 11 increments the value of i. In step S905, the image processing unit 11 determines whether the condition of i ≧ M is satisfied. As a result of this determination, if the condition is satisfied, the process returns to S903 to correct the next line. On the other hand, if the condition is not satisfied, the flicker correction by the image correction is ended in S906.

  As described above, according to the present embodiment, the exposure timing is calculated from the table based on the color ratio evaluation value and the luminance ratio evaluation value. Thereby, it is possible to perform photographing with less color unevenness in the screen under the flicker light source.

(Other embodiments)
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

5 Photometric sensor 8 Image sensor 9 Control unit 16 Exposure timing calculation unit 19 Color ratio evaluation unit

Claims (7)

Imaging means for capturing a subject image and generating image data;
First detection means for detecting a change in color of a subject image picked up by the image pickup means under a light source whose luminance changes periodically;
First calculation means for calculating a color ratio evaluation value for each phase under the light source based on the color variation detected by the first detection means and the shutter speed;
Second detection means for detecting a change in luminance of the subject image under the light source;
Second calculation means for calculating a luminance ratio evaluation value for each phase under the light source based on a variation in luminance detected by the second detection means and a shutter speed;
Control means for controlling the exposure timing of the imaging means based on the color ratio evaluation value calculated by the first calculation means and the luminance ratio evaluation value calculated by the second calculation means;
An imaging device comprising:   The first calculation means calculates the color ratio evaluation value based on a color change in a screen caused by a temporal change under the light source and a color change caused by a shutter speed. Item 2. The imaging device according to Item 1.   The control means controls a weight between a color change in the screen and a color change caused by the shutter speed based on a similarity to an image taken in the previous shooting. 2. The imaging device according to 2.   The second calculation means calculates the luminance ratio evaluation value based on a luminance variation in a screen caused by a temporal variation under the light source and a luminance variation caused by a shutter speed. Item 4. The imaging device according to any one of Items 1 to 3.   5. The imaging according to claim 1, further comprising a correcting unit that corrects the image data generated by the imaging unit by applying a gain based on a color variation of the subject image. apparatus. A method for controlling an image pickup apparatus having an image pickup means for picking up a subject image and generating image data,
A first detection step of detecting a change in color of a subject image captured by the imaging means under a light source whose luminance changes periodically;
A first calculation step of calculating a color ratio evaluation value for each phase under the light source based on the color variation detected in the first detection step and the shutter speed;
A second detection step of detecting a change in luminance of the subject image under the light source;
A second calculation step of calculating a luminance ratio evaluation value for each phase under the light source based on a variation in luminance detected in the second detection step and a shutter speed;
A control step of controlling the exposure timing of the imaging means based on the color ratio evaluation value calculated in the first calculation step and the luminance ratio evaluation value calculated in the second calculation step;
A method for controlling an imaging apparatus, comprising:   The computer-executable program which functions a computer as each means of the imaging device of any one of Claims 1-5.