JP6752688B2 - Imaging device, its control method, and control program - Google Patents

Description

The present invention relates to an imaging device, a control method thereof, and a control program, and more particularly to an imaging device that corrects the influence of flicker whose brightness changes with time.

In general, a so-called flicker phenomenon (hereinafter, simply referred to as flicker) in which brightness unevenness and color unevenness occur in an image when shooting under a light source such as a fluorescent lamp whose brightness changes periodically (temporally). ) May occur. In particular, when the image sensor is driven by the so-called rolling shutter method in which the exposure start timing is different for each line, flicker is likely to occur.

As a method for correcting such flicker, for example, a technique is known in which a subject is imaged at a timing when the amount of light of the light source becomes maximum according to the period and phase of the flicker (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-74484

However, in the method described in Patent Document 1, it is necessary to synchronize the shutter drive timing with the flicker phase. For this reason, the time lag in the release may be longer than in normal shooting. For example, when the frequency of the commercial power supply is 50 Hz, the continuous shooting speed is reduced by a maximum of 10 msec, and when the frequency is 60 Hz, the continuous shooting speed is reduced by a maximum of 8.33 msec.

Therefore, an object of the present invention is to provide an imaging device, a control method thereof, and a control program capable of suppressing an increase in release time lag and correcting the influence of flicker.

In order to achieve the above object, the imaging device according to the present invention is an imaging device that corrects the influence of the light and dark on the image obtained by the shooting when the image is taken in an environment where the light and dark change with time. Therefore, a determination means for detecting the presence or absence of light and darkness and a determination for determining the maximum gain amount used when correcting the image according to the detection result when the light and darkness is detected by the detection means. It is characterized by having means and correction means for correcting the image based on the maximum gain amount when the maximum gain amount is in a predetermined range.

According to the present invention, it is possible to suppress an increase in the release time lag and to correct the influence of flicker in which light and darkness changes with time.

It is a block diagram which shows the structure about the example of the image pickup apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the operation in the flicker detection mode in the camera shown in FIG. It is a figure which shows an example about the relationship between the light source luminance value used in the camera shown in FIG. 1 and a flicker table. It is a figure which shows the correlation between the flicker table obtained for each RGB in the photometric sensor shown in FIG. 1 and the V mapping predicted for each RGB in an image sensor. It is a figure for demonstrating an example about the phase of flicker and the timing of exposure. It is a figure for demonstrating the relationship between the flicker table and the gain table used in the camera shown in FIG. It is a flowchart for demonstrating the operation at the time of still image shooting by the camera shown in FIG. It is a flowchart for demonstrating the image correction shown in FIG.

An example of the image pickup apparatus according to the embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an example of an imaging device according to an embodiment of the present invention. In the illustrated example, one or more of the functional blocks may be realized by hardware such as an ASIC or a programmable logic array (PLA). Further, it may be realized by executing software by a programmable processor such as a CPU or MPU. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, the same hardware can be the main body even if different functional blocks are described as the main body of operation.

The illustrated imaging device is, for example, a digital camera (hereinafter referred to as a camera), and the camera includes a camera body 100 and a photographing lens unit (hereinafter simply referred to as a lens unit) 200. Then, the lens unit 200 is detachably attached to the camera body 100. In the illustrated example, the lens unit 200 is removable from the camera body 100, but the camera body 100 and the lens unit 200 may be integrated.

The lens unit 200 includes a photographing lens 1 including a zoom lens, a shift lens, a focus lens, and the like. Further, the lens unit 200 has a diaphragm 2 which is a light amount adjusting member.

The camera body 100 is provided with an image sensor 8, and a mechanical shutter 7 is arranged on the front side of the image sensor 8. Then, at the time of shooting, an optical image (subject image) is formed on the image sensor 8 via the lens unit 200. The image sensor 8 is, for example, a CCD sensor, in which storage-type photoelectric conversion elements (pixels) are arranged in a two-dimensional matrix, and an image signal corresponding to an optical image is output.

A mirror 3 is arranged in front of the mechanical shutter 7. When an optical image is formed on the image sensor 8, that is, when shooting, the mirror 3 is in a state of being retracted from the optical axis (mirror lockup state). Further, when an optical image is incident on the pentaprism 4, the mirror 3 is in a state of being positioned on the optical axis (mirror down state).

The mirror 3 is a semi-transmissive so-called half mirror, and the light reflected by the mirror 3 is diffused by the focus plate 14 and incident on the pentaprism 4. The pentaprism 4 is incident on the photometric sensor 5 for measuring the luminance distribution of the object to be imaged while reflecting the diffused light.

The photometric sensor 5 is, for example, a CCD sensor, and like the image sensor 8, storage-type photoelectric conversion elements (pixels) are arranged in a two-dimensional matrix. In the illustrated example, the photometric sensor 5 has an RGB (red, green, and blue) color filter, and the color filter sets a spectral sensitivity difference for a plurality of pixels. As a result, the photometric sensor 5 outputs information indicating the brightness of the subject and color information indicating the color.

The photometric sensor 5 is driven by a so-called global shutter method in which the timing of charge accumulation time in all pixels is the same. On the other hand, the image sensor 8 is driven by a rolling shutter method in which the start timing of the charge accumulation time is sequentially moved for each line.

The control unit 9 controls the control of the entire camera. Although not shown, the control unit 9 includes a CPU, a RAM area, and a ROM area. For example, the control unit 9 determines the aperture value of the aperture 2, the shutter speed of the mechanical shutter 7, and the travel start timing of the shutter based on the output of the photometric sensor 5.

The shutter drive unit 10 runs vertically the shutter front curtain and the shutter rear curtain of the mechanical shutter 7 under the control of the control unit 9 to control the exposure time. The image processing unit 11 performs A / D conversion, white balance processing, interpolation processing, color conversion processing, gradation conversion processing, and the like on the image signal output from the image sensor 8 to generate image data. Then, the image processing unit 11 saves the image data in the memory 12. Further, the image processing unit 11 adjusts the white balance gain for each line corresponding to each pixel of the image sensor 8 in the acquired image data. The recording unit 13 records the image data output from the image processing unit 11 on a portable medium (recording medium: not shown). The portable media is selectively attached to the camera body.

The flicker detection unit 15 detects the presence or absence of flicker based on the output of the photometric sensor 5. That is, the flicker detection unit 15 detects whether or not shooting is performed in a flicker environment. The mode in which the photometric sensor 5 is driven to detect flicker is called a flicker detection mode.

FIG. 2 is a diagram for explaining the operation in the flicker detection mode in the camera shown in FIG. FIG. 2A is a diagram showing flicker detection when the power supply frequency is 50 Hz, and FIG. 2B is a diagram showing flicker detection when the power supply frequency is 60 Hz.

The flicker cycle under a light source such as a fluorescent lamp is the reciprocal of twice the commercial power frequency (hereinafter simply referred to as the power frequency). Therefore, when the power supply frequency is 50 Hz, the flicker cycle is 10 msec, and when the power supply frequency is 60 Hz, the flicker cycle is 8.33 msec.

In the flicker detection mode, in order to detect flicker at these two power supply frequencies, the charge of the photometric sensor 5 is accumulated at a period of (1.67 / N) msec (N is a positive integer, N = 1 in FIG. 2). And reading is repeated for a predetermined period, and a plurality of charge accumulation results are output. The flicker detection unit 15 integrates the output of the photometric sensor 5 over the entire screen to obtain the light source luminance value.

The flicker detection unit 15 detects the presence / absence of flicker, the period, and its phase based on the temporal fluctuation of the light source luminance value. For example, based on the above-mentioned plurality of charge accumulation results, an evaluation value that evaluates five consecutive charge accumulation results and an evaluation value that evaluates six consecutive charge accumulation results are acquired. Then, the configuration may be such that the presence / absence of flicker, the period, and the phase are detected by comparing each evaluation value with a predetermined threshold value.
If the output of the photometric sensor 5 is integrated over the entire screen, the influence of the moving subject can be reduced. Further, when N ≧ 2, the flicker period and phase can be detected more accurately than in the case of N = 1. Further, also in the generation of the flicker table described later, the accuracy of the flicker table can be improved by increasing the integer N. Therefore, it is desirable that the charge accumulation and readout operations in the photometric sensor 5 be as fast as possible.

The flicker table generation unit 6 generates a flicker table (that is, a fluctuation amount of the brightness value) for each shutter speed based on the light source brightness value obtained by the flicker detection unit 15. Now, let the light source luminance value at time t be f (t) and the shutter speed be Tv. The flicker table g (Tv, t) at the time t and the shutter speed Tv is obtained by the following equation (1).

FIG. 3 is a diagram showing an example of the relationship between the light source luminance value used in the camera shown in FIG. 1 and the flicker table.

After the integration calculation, the solid line shows the flicker table when Tv = 1/1000 sec, and the broken line shows the flicker table when Tv = 1/250 sec. The alternate long and short dash line shows the flicker table when Tv = 1/125 sec. Are shown respectively. As shown in the figure, it can be seen from the integral calculation that as the shutter speed Tv increases, the amount of variation in the brightness value (g (Tv, t)) decreases.

With reference to FIG. 1 again, the mapping prediction unit 17 predicts the integration result (V mapping) for each line in the image pickup device 8 based on the flicker table generated by the flicker table generation unit 6. As described above, since the image sensor 8 is driven by the rolling shutter method in which the start timing of the charge accumulation time for each line is sequentially moved, between the time-flicker table and the V coordinate-V mapping of the image sensor 8. Is highly correlated.

FIG. 4 is a diagram showing the correlation between the flicker table obtained for each RGB in the photometric sensor 5 shown in FIG. 1 and the V mapping predicted for each RGB in the image sensor 8.

Now, it is assumed that the traveling speed of the mechanical shutter 7 on the imaging surface is constant and the spectral characteristics of the photometric sensor 5 and the image sensor 8 are the same. In this case, the flicker table and the V map are exactly the same. The traveling speed of the mechanical shutter 7 is generally not constant, but in this case, information on the traveling speed measured in advance is stored in a non-volatile memory (not shown), and the mapping prediction unit 17 is based on the traveling speed information. The flicker table may be converted into a V map.

Further, even if the photometric sensor 5 and the image sensor 8 have different spectral characteristics, it is possible to convert the flicker table into a V-mapping. For example, the RGB color ratios of the photometric sensor 5 and the image sensor 8 are measured in advance under various light sources. Then, by multiplying the flicker table by the gain for correcting the RGB color ratio based on the color temperature estimated by the photometric sensor 5 or the image sensor 8, it is possible to convert to a V-mapping.

In the following description, for convenience of explanation, it is assumed that the traveling speed of the mechanical shutter 7 on the imaging surface is constant and the spectral characteristics of the photometric sensor 5 and the image sensor 8 are the same.

The gain table generation unit 16 generates a gain table based on the flicker table generated by the flicker table generation unit 6. The gain table is a table for removing fluctuation components of the brightness of the light source from the imaging surface (correcting flicker). Here, the gain table is a table obtained by obtaining the maximum value of the gain to be multiplied for each line for each time t.

When correcting flicker, a method of removing brightness unevenness and color unevenness on the imaging surface (in-plane correction) and a method of removing brightness and color differences between a plurality of images in addition to the imaging surface (frame-to-frame correction). Is used.

FIG. 5 is a diagram for explaining an example of the flicker phase and the exposure timing. FIG. 5A is a diagram showing an example of the flicker phase and exposure timing in the case of in-plane correction and inter-frame correction, and FIG. 5B is a diagram showing the flicker phase in the case of in-plane correction and inter-frame correction. And another example of exposure timing. Further, FIG. 5C is a diagram showing still another example of the flicker phase and the exposure timing in the case of in-plane correction and inter-frame correction.

Now, the exposure start timing of the first line of the image sensor 8 is t1, the value of the flicker table at that time is L1, the exposure start timing of the M (M is an integer of 2 or more) line is tM, and the flicker table at that time. Let the value be LM. Further, the maximum value of the flicker table is Lmax, and the minimum value is Lmin.

In FIG. 5A, the maximum gain value required for flicker correction is LM / L1 for in-plane correction and Lmax / L1 for inter-frame correction. Similarly, in FIG. 5B, it is L1 / Lmin in the case of in-plane correction and Lmax / Lmin in the case of inter-frame correction. In FIG. 5C, Lmax / L1 is obtained in both the in-plane correction and the inter-frame correction.

As described in the example above. The maximum gain (maximum value) required for correcting flicker is obtained for each phase of flicker to obtain a gain table.

FIG. 6 is a diagram for explaining the relationship between the flicker table and the gain table used in the camera shown in FIG. FIG. 6A is a diagram showing a flicker table, and FIG. 6B is a diagram showing a gain table in the case of in-plane correction. Further, FIG. 6C is a diagram showing a gain table in the case of inter-frame correction.

As shown in FIG. 6A, it is assumed that the flicker tables R, G, and B with respect to R, G, and B change. In this case, when performing in-plane correction, the gain tables Gain R, Gain G, and Gain B with respect to R, G, and B are changed as shown in FIG. 6 (b). As shown in the figure, when the amount of change in the flicker tables R, G, and B is large, the values (that is, gain) of the gain tables Gain R, Gain G, and Gain B are large. On the other hand, when the amount of change in the flicker tables R, G, and B is small, the values of the gain tables Gain R, Gain G, and Gain B are small.

When performing inter-frame correction, the gain tables Gain R, Gain G, and Gain B for R, G, and B are changed as shown in FIG. 6 (c). Similar to the case of FIG. 6B, when the amount of change in the flicker tables R, G, and B is large, the values (that is, gain) of the gain tables Gain R, Gain G, and Gain B are large. On the other hand, when the amount of change in the flicker tables R, G, and B is small, the values of the gain tables Gain R, Gain G, and Gain B are small. However, in FIG. 6 (c), the change in gain is larger than in the case of FIG. 6 (b).

FIG. 7 is a flowchart for explaining an operation when a still image is taken by the camera shown in FIG. The process according to the illustrated flowchart is performed by the CPU provided in the control unit 9 expanding the program stored in the ROM area into the RAM area and executing the program.

First, the control unit 9 determines whether or not the still image shooting preparation switch (SW1) that starts the preparation for still image shooting is ON (step S101). When SW1 is OFF (NO in step S101), the control unit 9 stands by.

When SW1 is turned ON (YES in step S101), the control unit 9 operates the photometric sensor 5 in the above-mentioned flicker detection mode (step S102). The flicker detection unit 15 detects the presence or absence of flicker by the above-mentioned flicker detection method based on the output of the photometric sensor 5. Then, the control unit 9 determines the presence or absence of flicker according to the detection result by the flicker detection unit 15 (step S103).

If it is determined that flicker exists (YES in step S103), the control unit 9 generates a flicker table for each shutter speed Tv by the flicker table generation unit 6 (step S104). Then, the control unit 9 generates the gain table by the gain table generation unit 16 based on the flicker table as described above (step S105).

Subsequently, the control unit 9 determines whether or not the still image shooting switch (SW2) is ON (step S106). That is, the control unit 9 determines whether or not there is a shooting instruction. When SW2 is OFF (NO in step S106), the control unit 9 determines whether or not a predetermined time has elapsed since the previous generation of the gain table (step S107). When the predetermined time has elapsed (YES in step S107), the control unit 9 discards the gain table generated by the gain table generation unit 16 and returns to the process of step S101. On the other hand, if the predetermined time has not elapsed (NO in step S107), the control unit 9 returns to the process of step S106.

When SW2 is turned ON (YES in step S106), the control unit 9 sets the photometric sensor 5 in the photometric mode. Then, the control unit 9 performs a photometric calculation based on the output of the flicker detection unit 15 and the output of the photometric sensor 5, and determines the shutter speed (Tv) according to the photometric result (step S108).

Subsequently, the control unit 9 obtains the maximum values Gr, Gb, and Gg for each RGB according to the timing when SW2 is turned ON and the gain table generated in step S105 (step S109). Then, the control unit 9 determines the maximum value Gmax, which is the maximum gain amount of RGB, based on the maximum values Gr, Gb, and Gg (step S110).

Next, the control unit 9 determines whether or not the maximum value Gmax is a predetermined threshold value Th1 or less (first threshold value or less) (step S111). When Gmax> Th1 (NO in step S111), the control unit 9 determines whether or not the maximum value Gmax is a predetermined threshold value Th2 or less (second threshold value or less) (step S112). Here, Th2> Th1.

When Gmax> Th2 (NO in step S112), if the mechanical shutter 7 is driven by the shutter drive unit 10 at a normal timing, there is a concern that the image quality may deteriorate due to excessive gain. Therefore, the control unit 9 repeats the process of step S112 until Gmax ≦ Th2. Then, in response to the determination that Gmax ≦ Th2, the control unit 9 proceeds to the process of step S113 described later. The threshold values Th1 and Th2 may be changed according to the imaging sensitivity of the image sensor 8.

On the other hand, when Gmax ≦ Th2 (YES in step S112), the control unit 9 corrects the flicker by performing image correction described later (step S113). Then, the control unit 9 ends the still image shooting.

When Gmax ≦ Th1 (YES in step S111), the control unit 9 considers that the image is not affected by flicker, and executes a normal shooting operation without performing flicker correction processing (step S114). Then, the control unit 9 ends the still image shooting. If it is determined that the flicker does not exist (NO in step S103), the control unit 9 proceeds to the process of step S114 and executes a normal shooting operation.

FIG. 8 is a flowchart for explaining the image correction shown in FIG. 7.

When the image correction is started, the control unit 9 generates a V-mapping by the mapping prediction unit 17 according to the flicker table generated by the flicker table generation unit 6 and the timing when the SW2 is turned on (step S201). Then, the control unit 9 sets 1 for the line number (that is, the line number) i (step S202).

Subsequently, the control unit 9 performs the correction represented by the following equations (2) to (4) on the pixel of the i-line in the image sensor 8 by the image processing unit 11 (step S203). Here, it is assumed that the image sensor 8 has pixels of L rows × M columns.

R (i, j) = R (i, j) x Rm / Ri x max (Rp / Rm, Bp / Bm) (2)
G (i, j) = G (i, j) x Gm / Gi x max (Rp / Rm, Bp / Bm) (3)
B (i, j) = B (i, j) x Bm / Bi x max (Rp / Rm, Bp / Bm) (4)
It should be noted that i = 1 to M and j = 1 to L.

Here, Ri indicates the value of the V mapping of the i-line. At the time of inter-frame correction, Rp, Bp, and Gp indicate the values when R, G, and B are the largest in the flicker table. Further, Rm and Bm are the values of R and B when G in the flicker table is the largest. At the time of in-plane correction, Rp, Bp, and Gp indicate the values of R, G, and B when the flicker table is the largest on the imaging surface. Then, Rm and Bm are the values of R and B when G of the flicker table becomes the largest on the imaging surface.

Next, the control unit 9 increments the line number i (step S204). Then, the control unit 9 determines whether or not the line number i is M or less (step S205). When i ≦ M (YES in step S205), the control unit 9 returns to the process of step S203. On the other hand, when i> M (NO in step S205), the control unit 9 ends the image correction (that is, flicker correction).

As described above, in the embodiment of the present invention, the flicker correction is performed when the maximum gain amount exceeds the threshold value Th1 and is equal to or less than the threshold value Th2. Therefore, the flicker is suppressed while suppressing the increase in the release time lag. It can be corrected.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

For example, the function of the above-described embodiment may be used as a control method, and the image pickup apparatus may be made to execute this control method. Further, a program having the functions of the above-described embodiment may be used as a control program, and the control program may be executed by a computer included in the imaging device. The control program is recorded on a computer-readable recording medium, for example.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

5 Photometric sensor 6 Flicker table generator 8 Image sensor 9 Control unit 11 Image processing unit 15 Flicker detection unit 16 Gain table generator 17 Map prediction unit 100 Camera body 200 Lens unit

Claims (9)

An imaging device that corrects the effect of light and darkness on an image obtained by the shooting when shooting is performed in an environment in which light and darkness changes with time.
The detection means for detecting the presence or absence of light and darkness,
When the light and darkness is detected by the detection means, a determination means for determining the maximum gain amount used when correcting the image according to the detection result, and
A correction means that corrects the image based on the maximum gain amount when the maximum gain amount is within a predetermined range.
An imaging device characterized by having. Equipped with an image sensor that outputs an image according to the subject image
The determination means determines the maximum gain amount for each line in the image sensor.
The imaging device according to claim 1, wherein the correction means corrects the image according to the maximum gain amount for each line. The determining means obtains a fluctuation amount of the brightness value based on a brightness value indicating brightness and darkness, which is a detection result by the detection means, and a shutter speed, and obtains a gain table showing a gain amount corresponding to the fluctuation amount of the brightness value. The imaging apparatus according to claim 1 or 2, wherein the maximum gain amount is determined from the gain table at the timing when the image is generated and a shooting instruction is given. It has a photometric means that measures the subject and obtains the photometric result when the shooting instruction is given.
The determining means determines the shutter speed at the time of shooting according to the brightness value indicating brightness and darkness which is the detection result by the detecting means and the photometric result, and determines the maximum gain amount according to the shutter speed. The image pickup apparatus according to claim 3. The one according to any one of claims 1 to 4, wherein the predetermined range exceeds the first threshold value and is equal to or less than the second threshold value larger than the first threshold value. Imaging device. The image pickup apparatus according to claim 5, wherein the first threshold value and the second threshold value are set according to the sensitivity of an image pickup device that outputs an image according to a subject image. The correction means is characterized in that, when the maximum gain amount exceeds the second threshold value, the image obtained up to the timing when the maximum gain amount reaches the second threshold value is corrected. The imaging device according to claim 5 or 6. It is a control method of an image correction device that corrects the influence of the light and darkness on the image obtained by the shooting when the shooting is performed in an environment where the light and darkness changes with time.
The detection step for detecting the presence or absence of light and darkness and
When the light and darkness is detected in the detection step, a determination step of determining the maximum gain amount used when correcting the image according to the detection result, and a determination step.
A correction step that corrects the image based on the maximum gain amount when the maximum gain amount is within a predetermined range.
A control method characterized by having. When brightness makes a time varying photographed under environment, a control program used in the image correction apparatus for correcting the influence of the dark in an image obtained by the photographing, the computer,
The detection step for detecting the presence or absence of light and darkness and
When the light and darkness is detected in the detection step, a determination step of determining the maximum gain amount used when correcting the image according to the detection result, and a determination step.
A correction step that corrects the image based on the maximum gain amount when the maximum gain amount is within a predetermined range.
A control program characterized by executing.