JP2008067132A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To sufficiently apply a flicker suppression to a photographed image of a subject accompanied by movement. <P>SOLUTION: This imaging apparatus (9) equipped with an imaging device (12) of a rolling shutter system has: a frame rate setting means (13) for setting a frame rate of the imaging device to either of a high frame rate or a low frame rate; a flicker correction value operation means (15) for calculating a flicker correction value based on a plurality of high frame rate images to be periodically output from the imaging device when the frame rate of the imaging device is set to the high frame rate by the frame rate setting means; and a flicker correction means (25) for correcting a low frame rate image to be output from the imaging device when the frame rate of the imaging device is set to the low frame rate by the frame rate setting means using the flicker correction value. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, and more particularly, to an imaging apparatus that suppresses flicker when shooting under a light source whose brightness varies periodically.

  When a subject is photographed with an imaging device such as a digital camera or a video camera using an artificial light source as a main light source or an auxiliary light source, depending on the combination of the artificial light source and the imaging device, a horizontal band-like luminance change stripe ( Flicker).

  A typical combination of an artificial light source and an imaging device that generates flicker is a combination of a non-inverter type fluorescent lamp and an XY address type imaging device such as a CMOS sensor. An XY addressing imaging device is an imaging device that can freely read out information on a pixel located at the intersection point by specifying an X address (column address) and a Y address (row address). A CMOS sensor is a typical example.

  FIG. 7A illustrates a shooting state in which flicker may occur. In this figure, a non-inverter type fluorescent lamp 2 is attached to the ceiling of a room 1, and an image pickup apparatus 4 in which an XY address type image pickup device such as a CMOS sensor is incorporated by a photographer 3 in the room 1. Assume that the subject 5 is photographed using the fluorescent lamp 2 as a main light source using a (digital camera or digital video camera).

  Since the non-inverter type fluorescent lamp 2 repeats the fluorescent discharge every half cycle of the commercial power supply frequency, for example, if a commercial power supply of 50 Hz is used, the cycle of the half cycle (1/100 seconds = 10 milliseconds) That is, the brightness fluctuates at a cycle twice the commercial power frequency. In the figure, milliseconds are expressed as ms.

  FIG. 7B is a correspondence diagram between the light quantity fluctuation of the fluorescent lamp and the reading operation of an XY address type imaging device such as a CMOS sensor. In this figure, the brightness (light quantity) of the fluorescent lamp 2 fluctuates at a period (10 milliseconds) that is twice the commercial power supply frequency as described above. When the subject 5 is photographed using the imaging device 4 in which an XY address type imaging device such as the above is incorporated, a horizontal band-like luminance change stripe (flicker) is generated in the photographed image.

  The reason for this will be explained. Normal pixel information readout in an XY addressing imaging device such as a CMOS sensor is not a simultaneous readout of all pixels as in a CCD sensor, but a line-sequential readout for each line, also called a rolling shutter. It is. That is, as shown in FIG. 7B, the first line (L1), the second line (L2), the third line (L3),..., The nth line (Ln) of the imaging device little by little. This is because pixel information is read out while shifting in the time axis direction.

  In FIG. 7B, hatched L1 to Ln indicate each line constituting one image (hereinafter also referred to as a frame), and these lines are little by little (usually one pixel clock). It is shifted in the time axis direction. Therefore, since the brightness (luminance value) of L1 to Ln corresponds to the brightness of the fluorescent lamp 2 at the same time, the brightness for each line is not uniform, and as a result, the brightness of the horizontal band in the image A change stripe (flicker) is generated.

  As a flicker suppression technique, for example, a technique described in Patent Document 1 below is known. In this technique, an image signal photographed by an XY addressing type imaging device such as a CMOS sensor is integrated over a time of one horizontal period or more, and a difference value of integral values in adjacent fields is integrated in three consecutive fields. Is normalized by the average value, and the normalized integral value is subjected to discrete Fourier transform to extract a spectrum, the flicker coefficient is estimated from the spectrum, and the flicker component in the image signal is suppressed using the flicker coefficient. ing.

JP 2004-222228 A

  However, the technique described in Patent Document 1 has a problem that flicker suppression is insufficient for a captured image of a subject with movement for the following reason.

  That is, in the document, “the difference value of the integral values in adjacent fields is normalized by the average value of the integral values in three consecutive fields”, and this “three fields” is a paragraph “0235” of the document. 28 and FIG. 29, since one field period is 1/60 second, 3 × (1/60 seconds) = three fields acquired continuously over a time of 0.05 seconds. Is recognized.

  The above time (0.05 seconds) is “1/20 (second)” in terms of the shutter speed of a normal camera. In general, such a shutter speed falls within the range of a low-speed shutter. Therefore, particularly when a subject with motion is photographed, it is inevitable that the subject position in each of the three fields is slightly shifted.

  FIG. 8 is a diagram showing three fields in which a subject with motion is photographed at a 1/60 second period. In this figure, the same subject 6 (a car traveling from right to left in this figure) is shown in each of the first to third fields F1 to F3, but is shown in each field F1 to F3. The position of the inserted subject 6 is different. For this reason, these three fields F1 to F3 are applied to the technique of the above-mentioned Patent Document 1, and the difference value of the integral values in the “adjacent fields“ F1 and F2, F2 and F3 ”is consecutive three fields“ F1 ”. Even if “normalized by the average value of the integral values in .about.F3 ″”, the positions of the subject 6 in these three fields F1 to F3 are different, so that these three fields F1 to F3 are extremely different. In particular, the flicker coefficient in Patent Document 1 for a photographed image of a subject with movement must be inaccurate, and eventually, flicker suppression for a photographed image of a subject with movement is suppressed. The problem of becoming insufficient is inevitable.

  SUMMARY An advantage of some aspects of the invention is that it provides an image pickup apparatus that can sufficiently suppress flicker on a captured image of a subject with movement.

According to a first aspect of the present invention, there is provided an imaging apparatus including a rolling shutter type imaging device, frame rate setting means for setting a frame rate of the imaging device to one of a high frame rate and a low frame rate, and the frame When the frame rate of the imaging device is set to a high frame rate by the rate setting means, a flicker correction value is calculated based on a plurality of high frame rate images periodically output from the imaging device. When the frame rate of the imaging device is set to a low frame rate by the flicker correction value calculating means and the frame rate setting means, the low frame rate image output from the imaging device is converted to the flicker correction value. And flicker correction means for correcting by using Preparative an imaging apparatus according to claim.
According to a second aspect of the present invention, the flicker correction value calculating means obtains an intra-frame integrated value of a plurality of high frame rate images periodically output from the imaging device for each image, The imaging apparatus according to claim 1, wherein an average value of the integrated values is set as a flicker correction value.
The invention according to claim 3 further includes image dividing means for equally dividing the low frame rate image output from the imaging device into a plurality of blocks in the vertical direction, and the flicker correction means performs flicker correction for each block. The flicker correction value calculating means obtains an integrated value of a plurality of high frame rate images periodically output from the imaging device for each image, and the plurality of blocks The imaging apparatus according to claim 1, wherein an average value of intra-frame integrated values corresponding to each of the first and second frames is set as the flicker correction value.

  In the present invention, periodic light quantity fluctuations can be accurately detected from a plurality of high frame rate images, and horizontal band-like luminance change fringes (flicker) in a low frame rate image caused by such light quantity fluctuations are detected. Can be suppressed.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted that the specific details or examples in the following description and the illustrations of numerical values, character strings, and other symbols are only for reference in order to clarify the idea of the present invention, and the present invention may be used in whole or in part. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

  FIG. 1 is a configuration diagram of an imaging apparatus. In this figure, an imaging device 9 includes an imaging unit 10 that can output a two-dimensional image of a subject (not shown) existing at a fixed or variable shooting angle of view α at either one of high and low frame rates. . The imaging unit 10 includes an optical system 11 including a photographing lens, a diaphragm mechanism, a focusing mechanism (and a zoom mechanism if necessary), and an XY address type imaging device 12 (typically a CMOS sensor). The The operation of the imaging unit 10 (adjustment of the aperture size and zoom magnification, that is, adjustment and focusing of the shooting angle of view α, and exposure and reading operation and frame rate of the imaging device 12) are appropriately determined by a control signal from the control unit 13. Controlled.

  As described above, the frame rate of the imaging unit 10 (more precisely, the imaging device 12) is set to at least one of two high and low frame rates according to the control signal from the control unit 13. Here, the low frame rate is the frame rate of the imaging device 12 used for normal shooting, and the high frame rate is a frame rate N times the low frame rate. For example, if the low frame rate is 300 fps (fps: frame / second) and N = 4, the high frame rate is 300 × 4 = 1200 fps. In principle, such a high frame rate can be realized by increasing the operating frequency of the imaging device 12 N times. However, depending on the type of the imaging device 12, the operating frequency is increased N times due to operational limitations. May not be possible. In such a case, the XY addressing imaging device 12 such as a CMOS sensor can partially read out only a pixel region in an arbitrary range, so that the above high frame rate can be easily achieved without increasing the operating frequency. Can do. For example, if the pixel frequency of 1 / N of the XY addressing imaging device 12 such as a CMOS sensor is partially read out with the operating frequency left as it is, the time required for reading becomes 1 / N by simple calculation. Can be increased N times.

  Hereinafter, for convenience of explanation, the low frame rate is set to 300 fps, and the high frame rate is set to 1200 fps that is N = 4 times.

  As will be described in detail later, the image capturing unit 10 is configured to detect a light amount variation that operates at a high frame rate (1200 fps), an actual shooting period that operates at a low frame rate (300 fps), , And operates in three periods of a switching period of a predetermined length between them. That is, image shooting that requires flicker suppression is performed with three period transitions from the “light quantity fluctuation detection period” to the “real shooting period” through the “switching period”. Here, the “light quantity fluctuation detection period” is a period for detecting a flicker component associated with the light quantity fluctuation of the non-inverter type fluorescent lamp (refer to the fluorescent lamp 2 in FIG. 7). This is a period for actually capturing an arbitrary subject using a lamp as a main light source or an auxiliary light source, and acquiring and recording the captured image. The “switching period” is a period necessary for transition between these two periods.

  As described above, in image capturing that requires flicker suppression, the imaging unit 10 first selects an image with a high frame rate (1/1200 second period = 0.8 millisecond period) in the “light quantity fluctuation detection period”. Only the number M is output. The number M of images is a number exceeding at least one cycle of light quantity fluctuation of a fluorescent lamp. For example, one period of light quantity fluctuation of a non-inverter type fluorescent lamp operating with a 50 Hz commercial power supply is 1/100 seconds = 10 milliseconds. Therefore, it is M when the condition of “0.8 milliseconds × M + 1 ≧ 10 milliseconds” is satisfied. In other words, in this case, if M = 12, then 0.8 × 12 + 1 = 10.4 milliseconds, which satisfies the above-described condition. Therefore, the following description will be made assuming that M = 12.

  The predetermined number M of high frame rate images output from the imaging unit 10 in this manner are sequentially input to the flicker correction value calculation unit 15 through the analog-digital converter 14 (ADC) under the control of the control unit 13. The The flicker correction value calculation unit 15 includes an R integration unit 16, a G integration unit 17, a B integration unit 18, an R determination unit 19, a G determination unit 20, a B determination unit 21, an R correction value calculation unit 22, and a G correction value calculation. A unit 23 and a B correction value calculation unit 24 are provided.

  In addition, here, an integration unit, a determination unit, and a correction value calculation unit for each primary color of RGB are provided, but this is a “primary color system” in which the imaging unit 10 outputs an image signal for each primary color of RGB. Because there is. When the imaging unit 10 is a “complementary color system” that outputs an image signal for each complementary color of YCM, an integration unit, a determination unit, and a correction value calculation unit for each complementary color of YCM are provided. That is, the R integration unit 16, the G integration unit 17, the B integration unit 18, the R determination unit 19, the G determination unit 20, the B determination unit 21, the R correction value calculation unit 22, the G correction value calculation unit 23, and the B correction value calculation. The unit 24 includes a Y integrating unit 16, a C integrating unit 17, an M integrating unit 18, a Y determining unit 19, a C determining unit 20, an M determining unit 21, a Y correction value calculating unit 22, a C correction value calculating unit 23, and an M. What is necessary is just to read as the correction value calculation part 24. Hereinafter, in order to simplify the description, it is assumed that the imaging unit 10 is a “primary color system” that outputs an image signal for each primary color of RGB.

  A predetermined number M of high frame rate images converted into digital signals by the analog-to-digital converter 14 are, for each primary color image (R image, G image, B image), an R integration unit 16, a G integration unit 17, and a B integration. The information is distributed to the part 18 and input.

  The R integration unit 16, the G integration unit 17, and the B integration unit 18 calculate an intra-frame integrated value (integrated value of each pixel value in the image) of each of the input high frame rate images. That is, the R integration unit 16 calculates an intra-frame integration value of each R image of the first to Mth high frame rate images, and the G integration unit 17 calculates the first to Mth high frame rate images. The intra-frame integration value of each G image is calculated, and the B integration unit 18 calculates the intra-frame integration value of each B image of the first to Mth high frame rate images.

  Hereinafter, these integrated values in the frame are identified by the symbol “K _ # _ &”. Here, # is the number (1, 2, 3,... M) of the high frame rate image, and & is the distinction of R image, G image, and B image (R, G, B). For example, K_1_R indicates an intra-frame integrated value of the R image of the first high frame rate image.

  The R determination unit 19, the G determination unit 20, and the B determination unit 21 are the maximum of the R image, the G image, and the B image for K _ # _ & calculated by the R integration unit 16, the G integration unit 17, and the B integration unit 18, respectively. A value (MAX_K _ # _ &) and a minimum value (MIN_K _ # _ &) are obtained, and further, it is determined whether or not a difference between the maximum value and the minimum value exceeds a predetermined threshold value. If the difference between the maximum value and the minimum value exceeds a predetermined threshold, it is determined that flicker that requires correction has occurred, and if not, flicker It is determined that there is no occurrence (or only a slight amount of flicker that can be ignored).

  The R correction value calculation unit 22, the G correction value calculation unit 23, and the B correction value calculation unit 24 generate flickers that require correction in the R determination unit 19, the G determination unit 20, or the B determination unit 21. When it is determined that the flicker is corrected, the flicker correction value of the corresponding color image is calculated. For example, when it is determined that all of the R determination unit 19, the G determination unit 20, or the B determination unit 21 have flickers that require correction, the R correction value calculation unit 22, the G correction value, Each of the calculation unit 23 and the B correction value calculation unit 24 calculates an average value (AVE_K _ # _ &) of K _ # _ & for each primary color and for each of blocks B1 to B4 of the divided image 28 described later, and for each primary color and An average value (AVE_K _ # _ &) for each block B1 to B4 of a divided image 28 (see FIG. 3) described later is used as a flicker correction value for the R image, G image, and B image for each block B1 to B4 of the divided image 28 described later. The data is output to the correction unit 25.

  Alternatively, it is determined that any one of the R determination unit 19, the G determination unit 20, and the B determination unit 21 (for example, the R determination unit 19) has flickers that require correction. In this case, the R correction value calculation unit 22 calculates an average value (AVE_K _ # _ &) of K _ # _ & of the R image for each of blocks B1 to B4 of the divided image 28 described later, and calculates the average value (AVE_K _ # _ &). The flicker correction value for the R image of each block B1 to B4 of the divided image 28 described later is output to the correction unit 25.

  Alternatively, any two of the R determination unit 19, G determination unit 20, and B determination unit 21 (for example, the R determination unit 19 and the G determination unit 20) generate flickers that require correction. If it is determined that the average value of the R image and the G image K _ # _ & (AVE_K _ # _ &) is determined by the R correction value calculation unit 22 and the G correction value calculation unit 23, the blocks B1 to B1 of the divided image 28 described later are used. The calculation is performed for each B4, and the average value (AVE_K _ # _ &) is output to the correction unit 25 as a flicker correction value for R and G images for each of the blocks B1 to B4 of the divided image 28 described later.

  As described above, in this embodiment, the calculation of the in-frame integrated value (K _ # _ &), the flicker determination, and the calculation of the flicker correction value are performed for each primary color of RGB (or for each complementary color of YCM). This is due to the following reason.

  FIG. 2 is a light amount fluctuation diagram (represented by R and B) of the white fluorescent lamp. In general, white light is a mixed color that includes RGB, YCM, and other colors at a predetermined ratio. Although the color of these fluorescent lamps appears to be white to the human eye, each color is not necessarily seen when viewed in a minute flow. Is not uniform on the time axis, and there is some variation in the amount of light for each color as shown. This is because there is a variation for each color in the light emission characteristics of the fluorescent lamp. For this reason, flickering of a specific color can be suppressed when calculating the in-frame integrated value (K _ # _ &), flicker determination, and flicker correction value without distinguishing RGB (or CYM). This is because it disappears or the flicker of a specific color is excessively suppressed, and the result is conspicuous. Therefore, in practice, as shown in the figure, the calculation of the in-frame integrated value (K _ # _ &), the flicker determination, and the calculation of the flicker correction value are performed for each primary color of RGB (or for each complementary color of YCM). Is desirable.

  However, if the reduction in circuit scale is emphasized and there is no problem in compromising with a certain degree of flicker suppression effect, the integrated value (K _ # _ &) within the frame is not necessarily required for each primary color of RGB (or for each complementary color of YCM). It is not necessary to perform calculation of flicker, flicker determination, and calculation of flicker correction value. For each of the primary colors of RGB (or one of the complementary colors of YCM) or the luminance information of the image, the calculation of the integrated value (K _ # _ &) within the frame, the flicker determination, and the calculation of the flicker correction value are typically performed. May be performed.

  Next, when the imaging unit 10 completes the “light quantity variation detection period”, the imaging unit 10 transitions to a “real imaging period” after passing through a “switching period” of a predetermined length. In this “actual imaging period”, the imaging unit 10 outputs one or more images at a low frame rate.

  The low frame rate image output from the imaging unit 10 in this way passes through the analog-to-digital converter 14 and is then input to the image dividing unit 26 under the control of the control unit 13.

  The image dividing unit 26 equally divides one of the low frame rate images output from the imaging unit 10 in the “real shooting period” into N pieces in the vertical direction.

  FIG. 3 is a conceptual diagram of image division. In this figure, an image 27 is a single image output at a low frame rate (300 fps) from the imaging unit 10 in the “actual imaging period”, and has an image size corresponding to the number of effective imaging pixels of the imaging device 13. is doing. For example, the image has a size of 1024 (columns) × 768 (rows). On the other hand, the image 28 is an image divided by the image dividing unit 26, and this image 28 is obtained by equally dividing the image 27 into N pieces in the vertical direction (hereinafter referred to as a divided image 28). That is, the divided image 28 includes N blocks B1 to B4. The number of horizontal pixels of each block B1 to B4 is the same as the number of horizontal pixels (1024) of the image 27, and each of the blocks B1 to B4. The number of vertical pixels coincides with a value obtained by multiplying the number of vertical pixels (768) of the image 27 by 1 / N.

  The correction unit 25 corrects each of the R image, the G image, and the B image for each of the blocks B1 to B4 of the divided image 28 by applying the flicker correction value calculated by the flicker correction value calculation unit 15. That is, the R image of the block B1 of the divided image 28 is corrected by applying the average value (AVE_K _ # _ &) of the intra-frame integrated values (K_1_R to K_4_R) of the first to fourth high frame rate images, The R image of the block B2 of the divided image 28 is corrected by applying an average value (AVE_K _ # _ &) of intra-frame integrated values (K_5_R to K_8_R) of the fifth to eighth high frame rate images, .... For the R image of block B4 of the divided image 28, the average value (AVE_K _ # _ &) of the intra-frame integrated values (K_M-3_R to K_M_R) of the M-3rd to Mth high frame rate images is applied. To correct.

  Similarly, the G image of the block B1 of the divided image 28 is corrected by applying the average value (AVE_K _ # _ &) of the integrated values (K_1_R to K_4_G) of the first to fourth high frame rate images. The G image of the block B2 of the divided image 28 is corrected by applying the average value (AVE_K _ # _ &) of the intra-frame integrated values (K_5_R to K_8_G) of the fifth to eighth high frame rate images. ..., for the G image of the block B4 of the divided image 28, the average value (AVE_K _ # _ &) of the intra-frame integrated values (K_M-3_R to K_M_G) of the M-3rd to Mth high frame rate images. Apply and correct.

  Similarly, the B image of the block B1 of the divided image 28 is corrected by applying the average value (AVE_K _ # _ &) of the integrated values (K_1_R to K_4_B) of the first to fourth high frame rate images. The B image of the block B2 of the divided image 28 is corrected by applying the average value (AVE_K _ # _ &) of the in-frame integrated values (K_5_R to K_8_B) of the fifth to eighth high frame rate images. ..., for the B image of the block B4 of the divided image 28, the average value (AVE_K _ # _ &) of the in-frame integrated values (K_M-3_R to K_M_B) of the M-3rd to Mth high frame rate images. Apply and correct.

  Hereinafter, also for the second block B2 to the fourth block B4, the R image, the G image, and the B image are corrected in the same manner as described above.

  The image synthesis unit 29 synthesizes the corrected blocks B1 to B4 and returns them to the original image 27, and the image 27 is converted into an image processing unit (not shown) (for example, image processing of a digital camera or digital video camera). Part).

  FIG. 4 is a timing chart of the light amount variation detection period, the switching period, and the actual photographing period. In this figure, a waveform 30 showing a half-wave periodic change represents the light quantity of the non-inverter type fluorescent lamp, and one period length of the light quantity fluctuation is half of one cycle of the commercial power supply frequency (here 50 Hz). (10 milliseconds). This value (10 milliseconds) corresponds to one cycle length of a frequency (100 Hz) twice the commercial power supply frequency (50 Hz).

  On the time axis in the figure, scales 1 to M (1 to 12 since M = 12) are engraved. The interval between the scales corresponds to one cycle length (1/1200 seconds = 0.8 milliseconds) of the high frame rate in the light amount variation detection period. Therefore, in this figure, M = per cycle length of the light amount variation. This indicates that 12 high frame rate images can be acquired. M = 12 rhombus figures drawn on the lower side of the scale of 1 to 12 in the light quantity fluctuation detection period schematically represent the high frame rate images H1 to H12 from 1 to 12 acquired in the period. It is a thing.

  When the light amount fluctuation detection period is completed, an actual photographing period is started after a switching period of a predetermined length. The length of the switching period (SEL) is a length that does not cause excess or deficiency so that the frame rate of the imaging device 13 can be stably switched from low (300 fps) to high (1200 fps), for example, one frame at a low frame rate. This is a time obtained by adding a predetermined margin time β to the time (3.3 milliseconds). Here, the margin time β can be set to a time of about two frames with a high frame rate (0.8 milliseconds × 2). Therefore, in this case, SEL is 3.3 milliseconds + (0.8 milliseconds × 2) = 4.9 milliseconds.

  As described above, when the length of the switching period (SEL) is set to “4.9 milliseconds”, the actual photographing period is started after the scale 7 after completion of the light amount variation detection period.

  The band-like horizontal figure drawn below the actual photographing period represents the pixel information reading period for each line by the rolling shutter. Note that L1 (1) to Ln (1) represent the lines (L1 to Ln) of the first low frame rate image extracted from the imaging unit 13 during the actual shooting period, and L1 (2) to Ln (2). ) Represents each line (L1 to Ln) of the second low frame rate image extracted from the imaging unit 13 during the actual shooting period, and L1 (3) to Ln (3) are from the imaging unit 13 during the actual shooting period. Each line (L1 to Ln) of the extracted third low frame rate image is shown.

  FIG. 5 is a correspondence diagram of the high frame rate images H <b> 1 to H <b> 12, the low frame rate image, and the divided image 28. Each of the lines L1 (1) to Ln (1) of the first low frame rate image acquired during the actual shooting period and the high frame rate images H1 to H12 have a correspondence relationship as shown in the figure. Here, it is assumed that n = 768. That is, L1 (1) to L192 (1), which are the first 768 / N lines of each line of the low frame rate image, correspond to H7, H8, H9, H10, and H11 of the high frame rate image. Ln577 (1) to L768 (1), which are the last 768 / N lines of each line of the frame rate image, correspond to H10, H11, H12, H1, and H2 of the high frame rate image.

  Although not shown, L193 (1) to L384 (1) of each line corresponds to H8, H9, H10, H11, and H12 of the high frame rate image, and among the lines of the low frame rate image. Ln385 (1) to L576 (1) correspond to high frame rate images H9, H10, H11, H12, and H1.

  As shown in the figure, block 1 (B1) of the divided image 28 is composed of L1 (1) to L192 of the low frame rate image, and block 2 (B2) of the divided image 28 is L193 of the low frame rate image. (1) to L384. Similarly, block 3 (B3) of the divided image 28 is composed of L385 (1) to L576 of the low frame rate image, and block 4 (B4) of the divided image 28 is L577 (low frame rate image). 1) to L768.

Therefore, the high frame rate images H1 to H12 and the blocks B1 to B4 of the divided image 28 have the following correspondence relationship. However, this is the case where N = 4 and M = 12.
B1: H7, H8, H9, H10, H11
B2: H8, H9, H10, H11, H12
B3: H9, H10, H11, H12, H1
B4: H10, H11, H12, H1, H2

  The flicker correction value in the flicker correction value calculation unit 15 is calculated based on these correspondences. That is, the flicker correction value for B1 is calculated from the average value of H7, H8, H9, H10, and H11, and the flicker correction value for B2 is calculated from the average value of H8, H9, H10, H11, and H12. Similarly, the flicker correction value for B3 is calculated from the average value of H9, H10, H11, H12, and H1, and the flicker correction value for B4 is calculated from the average value of H10, H11, H12, H1, and H2. However, as described above, these flicker correction values are calculated individually for each of the R image, the G image, and the B image of each of the blocks B1 to B4 of the divided image 28.

  FIG. 6 is a diagram illustrating an operation flow of flicker suppression in the imaging device 9. This operation flow can be broadly divided into two processes: light amount detection (step S1) and flicker suppression (step S2). First, in the light amount detection in step S1, the frame rate of the imaging device 13 is set to a high frame rate (1200 fps) (step S1a). Next, M high frame rate images H1 to H12 are acquired over one period length (10 milliseconds) of the light amount fluctuation (step S1b), and the in-frame integrated value (K_ #) of the M high frame rate images H1 to H12 is acquired. _ &) Is calculated (step S1c). Then, the difference between the maximum value (MAX_K _ # _ &) and the minimum value (MIN_K _ # _ &) of the M intra-frame integrated values (K _ # _ &) is obtained (step S1d).

  Next, in the flicker suppression in step S2, the frame rate of the imaging device 13 is set to a low frame rate (300 fps) after a predetermined length (SEL) switching period (step S2a), and a low frame rate image is acquired (step S2a). Step S2b). Next, the low frame rate image is equally divided into N, and a divided image 28 including N (four) blocks B1 to B4 is generated (step S2c).

  Next, the “difference” obtained in step S1d, that is, the difference between the maximum value (MAX_K _ # _ &) and the minimum value (MIN_K _ # _ &) of the M intra-frame integration values (K _ # _ &) is predetermined. Is determined (step S2d), and if it exceeds, it is determined that correction for flicker suppression is required, and the following correction processing (step S2e and step S2f) and image are performed. While the synthesis process (step S2g) is executed to end the flow, if not exceeded, it is determined that correction for flicker suppression is not required, and the following correction processes (step S2e and step S2f) are performed. Pass, execute image composition processing (step S2g), and end the flow.

  That is, only when the correction for flicker suppression is necessary, the average value (AVE_K _ # _ &) of the integrated value in frame (K _ # _ &) is set to each of the “high frame rate images H1 to H12 and the divided images 28”. After calculating based on “correspondence with blocks B1 to B4” (step S2e) and applying the average value (AVE_K _ # _ &) to each block B1 to B4 of the divided image 28 to suppress flicker (step S2f) Then, the divided image 28 is synthesized in the form of the original image 27 (step S2g), and the flow ends.

  Here, the flicker correction values applied to the R image, G image, and B image of each of the blocks B1 to B4 of the divided image 28 are as follows. However, this is the case where N = 4 and M = 12.

Flicker correction value of R image of block B1:
K_7_R,
K_8_R,
K_9_R,
Flicker correction value of G image of average value block B1 of K_10_R and K_11_R:
K_7_G,
K_8_G,
K_9_G,
Flicker correction value of B image of average value block B1 of K_10_G and K_11_G:
K_7_B,
K_8_B,
K_9_B,
Flicker correction value of R image of average value block B2 of K_10_B and K_11_B:
K_8_R,
K_9_R,
K_10_R,
Flicker correction value of G image of average value block B2 of K_11_R and K_12_R:
K_8_G,
K_9_G,
K_10_G,
Flicker correction value of B image of average value block B2 of K_11_G and K_12_G:
K_8_B,
K_9_B,
K_10_B,
Flicker correction value of R image of average value block B3 of K_11_B and K_12_B:
K_9_R,
K_10_R,
K_11_R,
Flicker correction value of G image of average value block B3 of K_12_R and K_1_R:
K_9_G,
K_10_G,
K_11_G,
Flicker correction value of B image of average value block B3 of K_12_G and K_1_G:
K_9_B,
K_10_B,
K_11_B,
Flicker correction value of R image of average value block B4 of K_12_B and K_1_B:
K_10_R,
K_11_R,
K_12_R,
Flicker correction value of G image of average value block B4 of K_1_R and K_2_R:
K_10_G,
K_11_G,
K_12_G,
Flicker correction value of B image of average value block B4 of K_1_G and K_2_G:
K_10_B,
K_11_B,
K_12_B,
Average value of K_1_B and K_2_B

  As described above, in the present embodiment, using the partial reading of the XY address type imaging device 12, M frames are obtained at a high frame rate (1200 fps) that is M times faster than the light quantity fluctuation period of the non-inverter type fluorescent lamp. Images H1 to H12 are taken out from the imaging device 12, and flicker correction values associated with fluctuations in the amount of ambient light (in this case, fluorescent light) are calculated from the M high frame rate images H1 to H12, and then continued. Thus, the flicker is suppressed by using the above flicker correction value for each of the blocks B1 to B4 of the low frame rate image when the frame rate of the imaging device 12 is switched to the low frame rate (300 fps). Therefore, for example, even when shooting a subject with movement, flicker can be suppressed without hindrance. .

  One frame time at a high frame rate (1200 fps) is 1/1200 seconds = 0.8 milliseconds. This time corresponds to a high-speed shutter of a normal camera. In addition, when M frames (H1 to H12) are continuously acquired, the time required for the acquisition is 12 × (1/1200) = 1/100 seconds = 10 milliseconds, and eventually 50 Hz This corresponds to one cycle length (10 milliseconds) of the light amount fluctuation of the non-inverter type fluorescent lamp whose light amount fluctuates at a cycle twice that of the commercial power source. Therefore, from the M high frame rate images H1 to H12, it is possible to accurately detect the amount corresponding to one period of the light amount fluctuation of the environmental light (in this case, the light from the fluorescent lamp), and use the detection result to flicker. The correction value can be calculated accurately.

  This is clear from the description of FIG. In this figure, the time required to acquire M high frame rate images from H1 to H12 coincides with one cycle length (10 milliseconds) of the light quantity fluctuation of the fluorescent lamp. For this reason, it is possible to accurately detect the brightness of each portion (scales 1 to 12) of one period of the light amount fluctuation by each of the M high frame rate images H1 to H12.

  The one described in Patent Document 1 described at the beginning is merely a normalization by an average value of integral values in three fields having a one-field period of 1/60 seconds, and the time required for acquiring an image in three fields is as follows. In terms of shutter speed, it becomes “1/20 (seconds)”, and in particular, when shooting a subject with movement, it is inevitable that the position of the subject in each of the three fields is slightly shifted. The problem that the flicker suppression for the captured image of the subject with movement becomes insufficient cannot be avoided.

  On the other hand, in this embodiment, since one period of light quantity fluctuation can be accurately detected by the M high frame rate images H1 to H12, such a problem does not occur. In addition, after accurately detecting one cycle of the light amount variation in this way, N pieces of integrated values (K _ # _ &) of the M high frame rate images H1 to H12 are used to calculate N pieces. A flicker correction value (AVE_K _ # _ &) for each of the blocks B1 to B4 is calculated, and flicker suppression is performed for each of the blocks B1 to B4 of the divided image 28 of the low frame rate image by the flicker correction value. In spite of the rate change from the low frame rate to the low frame rate, the flicker correction for each of the blocks B1 to B4 can be performed without any trouble.

  Therefore, according to the present embodiment, when taking a still image or a moving image under a light source (for example, a non-inverter type fluorescent lamp) with a light amount variation, a “light amount variation detection period” with a high frame rate is executed. After that, only by executing the “real shooting period” of still image shooting or movie shooting after a “switching period” of a predetermined length, a still image with good image quality that eliminates the influence of flicker due to the light amount fluctuation of the above light source Or a movie can be taken.

  Incidentally, in the present embodiment, two periods of “light quantity fluctuation detection period” and “switching period” are required before the “actual shooting period”, so a delay corresponding to the length of these two periods (shutter) According to the example of the above embodiment, the time length of the “light quantity fluctuation detection period” is one cycle length (10 milliseconds) of the light quantity fluctuation and the “switching period”. The time length of is about half of one cycle length of light quantity fluctuation (SEL = 4.9 milliseconds). Therefore, even when these are combined, only a delay of about 14.9 milliseconds occurs. This time (14.9 milliseconds) is a time that is sufficiently permissible in a normal shooting scene (it is not experienced as a large delay), and thus may be practically used.

  In addition, since the flicker correction value in the present embodiment is performed based on the intra-frame integrated value in each of the M high frame rate images H1 to H12, no matter what the brightness deviation is within one frame. Not affected. That is, even if the light source is reflected in a part of the frame (image), the brightness of the light source can be accurately detected by the integrated value in the frame.

  In this embodiment, the high frame rate is set to 1200 fps and the low frame rate is set to 300 fps. However, this is an example for explanation, and the present invention is not limited to this. When the low frame rate is Afps, the high frame rate may be N times (N × Afps). Moreover, it is not limited to N = 4, but may be another integer, for example, N = 0 or more and 4 or less, or N = 4 or more. As N is increased, flicker suppression accuracy is improved, but the processing time is increased. Therefore, an appropriate value of N may be set in consideration of the required performance. The same applies to the value of M. Since the value of M defines the number of high frame rate images per light source fluctuation period output in the “light quantity fluctuation detection period”, the accuracy of flicker suppression improves as M increases, but the processing time increases. Therefore, like N, an appropriate value may be set in consideration of the required performance.

  In the above description, a non-inverter type fluorescent lamp is shown as a light source whose light amount periodically changes, but this is merely a typical example. Any light source whose light quantity varies periodically may be used. For example, a discharge lamp such as a mercury lamp or a sodium lamp may be used. Or even if it is an inverter type | mold fluorescent lamp, the operation | movement of an inverter becomes incomplete by failure etc., and the discharge voltage fluctuates periodically (thus, accompanying light quantity fluctuation) corresponds to the above-mentioned light source.

It is a block diagram of an imaging device. It is a light quantity fluctuation | variation figure of a white fluorescent lamp. It is a conceptual diagram of image division. It is a figure which shows the timing chart of a light quantity fluctuation | variation detection period, a switching period, and an actual imaging | photography period. FIG. 5 is a correspondence diagram between high frame rate images H1 to H12, a low frame rate image, and a divided image 28. FIG. 11 is a diagram illustrating an operation flow of flicker suppression in the imaging apparatus 9; It is a figure which shows the imaging | photography state which may produce a flicker, and a correspondence relation figure of the light quantity fluctuation | variation of a fluorescent lamp, and reading operation | movement of XY address type imaging devices, such as a CMOS sensor. It is a figure which shows three fields which image | photographed the to-be-photographed object with a movement with a 1/60 second period.

Explanation of symbols

9 Imaging device 12 Imaging device 13 Control unit (frame rate setting means)
15 Flicker correction value calculation unit (flicker correction value calculation means)
25 Correction unit (Flicker correction means)
26 Image segmentation unit (image segmentation means)

Claims (3)

In an imaging apparatus including a rolling shutter type imaging device,
Frame rate setting means for setting the frame rate of the imaging device to either one of a high frame rate and a low frame rate;
When the frame rate of the imaging device is set to a high frame rate by the frame rate setting means, a flicker correction value is calculated based on a plurality of high frame rate images periodically output from the imaging device. Flicker correction value calculating means for calculating;
Flicker correction means for correcting a low frame rate image output from the imaging device using the flicker correction value when the frame rate of the imaging device is set to a low frame rate by the frame rate setting means. An imaging apparatus comprising:   The flicker correction value calculating means obtains an intra-frame integrated value of a plurality of high frame rate images periodically output from the imaging device for each image, and flicker corrects an average value of the intra-frame integrated values of each image. The imaging device according to claim 1, wherein the imaging device is a value. Furthermore, image division means for equally dividing the low frame rate image output from the imaging device into a plurality of blocks in the vertical direction,
The flicker correction means performs flicker correction for each block, and
The flicker correction value calculating means obtains an intra-frame integrated value of a plurality of high frame rate images periodically output from the imaging device for each image, and includes an intra-frame corresponding to each of the plurality of blocks. The imaging apparatus according to claim 1, wherein an average value of integrated values is used as the flicker correction value.

Images