JP5630132B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus.

  Currently, an image pickup apparatus including a general XY address scanning type image pickup element employs a rolling shutter system that controls a charge accumulation time for each line by address designation.

  When moving pictures are taken under a fluorescent lamp using this rolling shutter method, horizontal stripes appear in the image due to the influence of periodic luminance changes due to the flicker frequency of the fluorescent lamp light source (for example, the commercial power supply frequency of 50 Hz or 60 Hz). In some cases, unevenness (hereinafter referred to as “flicker”) occurs.

  In view of this, there has been proposed an imaging apparatus that calculates the flicker frequency (50 Hz or 60 Hz) of a light source and suppresses flicker based on the flicker frequency (see, for example, Patent Document 1).

JP 2003-189129 A

  However, in the imaging apparatus of Patent Document 1, a plurality of flicker detection frames are set by dividing an image of one frame in the vertical scanning direction, and flicker components are obtained from luminance value data in each flicker detection frame that varies with time. After extraction, the flicker frequency of the light source is calculated. For this reason, the imaging device of Patent Document 1 has a problem that the arithmetic processing is generally complicated.

  In view of the above circumstances, an object of the present invention is to provide an imaging apparatus capable of calculating the blinking frequency of a light source without complicating arithmetic processing.

Imaging device according to one of the invention includes the acquisition unit, an average value calculation unit, and the frequency calculator, and a setting unit. Acquisition unit sequentially acquires the images signals captured area or al thinning read changeable imaging element rolling shutter speed. The average value calculation unit calculates the average value of the brightness of the image signal for each frame. The frequency calculation unit calculates the frequency of the light source based on the temporal change of the average value calculated for each frame. The setting unit sets a rolling shutter speed at the time of moving image shooting using the frequency of the light source calculated by the frequency calculation unit.

An imaging device according to another aspect of the invention includes an acquisition unit, a calculation unit, and a control unit. The acquisition unit sequentially acquires image signals that have been thinned and read out from the imaging region of the imaging device that can change the rolling shutter speed. The calculation unit calculates the brightness of the image signal for each frame. The control unit controls the rolling shutter speed during moving image shooting based on the temporal change in the brightness of the image signal calculated for each frame by the calculation unit.

  According to the imaging apparatus of the present invention, the blinking frequency of the light source can be calculated without complicating the arithmetic processing.

1 is a block diagram illustrating an internal configuration example of an electronic camera 1 that is an embodiment of an imaging apparatus of the present invention. As a comparative example, a diagram for explaining an example of flicker detection of a CMOS image sensor that is conventionally performed The flowchart which shows an example of the operation | movement of the flicker detection of the electronic camera 1. The figure explaining the principle of the flicker detection in this embodiment The figure explaining the principle of the flicker detection in this embodiment Partial enlarged view of the frame shown in FIG. The figure explaining the modification of 1st Embodiment

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating an internal configuration example of an electronic camera 1 which is an embodiment of an imaging apparatus of the present invention. The electronic camera 1 has not only a still image shooting function but also a moving image shooting function. As shown in FIG. 1, the electronic camera 1 includes a photographing lens 11, an image sensor 12, a timing generator (hereinafter referred to as “TG”) 13, an analog front end unit (hereinafter referred to as “AFE”) 14, and An image processing unit 15, a RAM (Random Access Memory) 16, a ROM (Read Only Memory) 17, a display monitor 18, a recording interface unit (hereinafter referred to as “recording I / F unit”) 19, and an operation unit. 20, a CPU (Central Processing Unit) 21, and a bus 22.

  Among these, the AFE 14, the image processing unit 15, the RAM 16, the ROM 17, the display monitor 18, the recording I / F unit 19, and the CPU 21 are connected to each other via a bus 22.

  The photographing lens 11 includes a plurality of lens groups including a focus lens and a zoom lens. For the sake of simplicity, FIG. 1 shows the photographing lens 11 as a single lens. The photographing lens 11 is controlled by a lens driving device (not shown).

  The image pickup device 12 is a CMOS (Complementary Metal-Oxide Semiconductor) type image pickup device in which a plurality of pixels are arranged two-dimensionally (M rows and N columns) as an example, and an arbitrary line can be read by XY addressing. In addition, on the image pickup surface of the image pickup device 12, three types of color filters of red (R), green (G), and blue (B) are arranged in a Bayer array as an example in order to detect the color of the subject image. Yes. In addition, the imaging element 12 includes a pixel addition circuit 12a that adds output values of arbitrary pixels by XY addressing. Details of the pixel addition circuit 12a will be described later in a modification.

  The TG 13 sends a drive signal to each of the image sensor 12 and the AFE 14 in accordance with an instruction from the CPU 21, thereby controlling the drive timing of both.

  The AFE 14 is an analog front end circuit that performs signal processing on an image signal (RGB signal) generated by the image sensor 12. The AFE 14 performs image signal gain adjustment, A / D conversion, and the like. The image signal output from the AFE 14 is temporarily recorded in the RAM 16.

  The image processing unit 15 reads the image signal recorded in the RAM 16 and performs various image processing (gradation conversion processing, contour enhancement processing, white balance processing, YC conversion processing, etc.).

  The ROM 17 is a non-volatile flash memory that stores a program for controlling the electronic camera 1 in advance. In the ROM 17, address information indicating a designated address to be thinned and read out under the control of the CPU 21 is recorded in advance.

  The display monitor 18 displays a still image, an operation menu of the electronic camera 1, and the like. As the display monitor 18, a liquid crystal monitor or the like can be appropriately selected and used.

  The recording I / F unit 19 is formed with a connector for connecting a detachable recording medium 30. Then, the recording I / F unit 19 accesses the recording medium 30 connected to the connector according to an instruction from the CPU 21 and performs recording processing of still images and moving images. The recording medium 30 is, for example, a non-volatile memory card. FIG. 1 shows the recording medium 30 after being connected to the connector.

  The operation unit 20 includes, for example, a command dial for command selection, a power button, a release button, and the like. Then, the operation unit 20 receives an instruction input for operating the electronic camera 1. In addition, the operation unit 20 receives, for example, a selection input of a moving image shooting mode via a command dial. Here, when an instruction input for a moving image shooting mode is received, a flicker detection program starts under the control of the CPU 21.

  The CPU 21 is a processor that performs overall control of the electronic camera 1. The CPU 21 controls each part of the electronic camera 1 by executing a program stored in the ROM 17 in advance. For example, the CPU 21 controls imaging such as reading of an image signal from the imaging element 12 using a rolling shutter system that controls the charge accumulation time for each line by XY addressing. In the present embodiment, for example, a flicker detection program used in the flow process shown in FIG. 3 to be described later is recorded on the recording medium 30, and the program recorded on the recording medium 30 is recorded on the ROM 17. Also good. As a result, a flicker detection program can be applied as firmware for adding a new function.

  The CPU 21 also functions as an acquisition unit 21a, an average value calculation unit 21b, and a frequency calculation unit 21c. In the present embodiment, in FIG. 1, the acquisition unit 21a, the average value calculation unit 21b, and the frequency calculation unit 21c are divided into blocks, but for example, an ASIC (Image Specific Integrated Circuit) of ASIC (Image Engine: Application Specific Integrated Circuit). In this way, each block may be integrated into one functional unit.

  The acquisition unit 21a instructs the image pickup device 12 to perform thinning readout in units of lines via the TG 13, and sequentially acquires the image signals of the partial areas read out from the imaging area at a predetermined frame rate. In this case, each frame generated based on the image signal of the partial area thinned and read out in units of lines is temporarily recorded in the RAM 16. The predetermined frame rate is a frame rate at which flicker can be detected, and is, for example, 600 fps as described later.

  The average value calculation unit 21b calculates the average value of the brightness of the image signal for each frame. Specifically, the average value calculation unit 21b adds the signal values for each of R, G, and B for each frame to the RGB signals after A / D conversion, and then obtains the average value. The image processing unit 15 performs YC conversion processing on the RGB signal to convert it into a luminance signal (Y) and a color difference signal (C), whereby the average value calculation unit 21b calculates the average value of the luminance values based on the luminance signal. You may calculate for every frame.

  The frequency calculation unit 21c calculates the blinking frequency based on the temporal change in the average value for each R, G, and B calculated for each frame. Details will be described with reference to a flowchart (FIG. 3) described later.

  Next, the flicker detection operation of the electronic camera 1 will be described. Here, for convenience of explanation, an example of conventional flicker detection will be described as a comparative example.

  FIG. 2 is a diagram for explaining an example of flicker detection of a CMOS image sensor conventionally performed as a comparative example. Fig.2 (a) is the figure which showed typically the mode of the brightness | luminance change of the fluorescent lamp. FIG. 2B is a timing chart of the vertical synchronization signal indicating the start of reading one frame image. FIG. 2C is a diagram schematically showing rolling shutter control. FIG. 2D is an exaggerated view of flicker generated due to the blinking frequency under a fluorescent lamp.

  Note that the fluorescent lamp causes a luminance change twice in each cycle of the blinking frequency (commercial power supply frequency). That is, this luminance change is repeated at a period corresponding to a frequency twice as high as the blinking frequency. Therefore, the blinking period is 1/100 second in the case of 50 Hz, and 1/120 second in the case of 60 Hz.

  Here, assuming the case of shooting a movie under a 50 Hz fluorescent lamp, the charge accumulation time for each line of the image sensor 12 is n times the blinking cycle (1/100 seconds) (n is an integer, where 0 is set) A case where the cycle does not correspond to (excluding) (that is, flicker is detected) will be described. In this case, the integral amount of the fluorescent lamp light quantity corresponding to the charge accumulation time of the line a shown in FIG. 2C and the integral of the fluorescent lamp light quantity corresponding to the charge accumulation time of the line b shown in FIG. It is different from the amount. Therefore, as shown in FIG. 2D, the brightness within one screen is not constant for each line, and flicker occurs. FIG. 2D schematically shows that detection is performed by finely dividing the flicker detection area.

  Conventionally, in flicker detection of a CMOS image sensor, the flicker frequency of a fluorescent lamp is calculated based on horizontal stripes appearing in one screen, but the arithmetic processing tends to be complicated. Therefore, in the present embodiment, a means for calculating the flicker frequency of the fluorescent lamp with high accuracy is provided. In this embodiment, since the charge accumulation time (rolling shutter speed) by the rolling shutter method is set to n times the blinking cycle, the blinking cycle (reciprocal of the blinking frequency) is calculated from the blinking frequency.

  FIG. 3 is a flowchart illustrating an example of the flicker detection operation of the electronic camera 1. This flowchart is started by the control of the CPU 21 when the moving image shooting mode is selected by a command dial or the like.

  Step S101: The CPU 21 drives the image sensor 11. Specifically, the CPU 21 sends a drive signal to the image sensor 11 and the AFE 14 via the TG 13 and issues an instruction to acquire a frame image signal at a frame rate of 600 fps. Here, the CPU 21 sequentially outputs image signals to the image sensor 12 by a rolling shutter system.

  For convenience of explanation, the frame rate of 600 fps is for flicker detection. At the time of actual moving image shooting, the CPU 21 sets a frame rate such as 24 fps, 30 fps, 50 fps or 60 fps according to the moving image format. To do. That is, when flicker is detected, the CPU 21 makes the frame rate faster than that during normal moving image shooting. Here, the CPU 21 may set the charge accumulation time for each line, or may set one charge accumulation time for a plurality of continuous lines.

  Step S102: The acquisition unit 21a of the CPU 21 instructs the image sensor 12 to perform thinning readout in units of lines. Thereby, the acquisition unit 21a sequentially acquires the image signals of the partial areas read out from the imaging area. The charge accumulation time in each thinning-out readout line is constant, but the acquisition timing is different in the time axis direction.

  4 and 5 are diagrams for explaining the principle of flicker detection in this embodiment. FIG. 4A is a diagram schematically showing how the fluorescent lamp changes in luminance. FIG. 4B is a timing chart of the vertical synchronization signal. FIG. 4C is a diagram schematically showing rolling shutter control in the present embodiment. FIG. 4D is a diagram schematically illustrating frame reading in the present embodiment. In FIG. 4D, the X direction of the image sensor 12 represents the horizontal scanning direction, and the Y direction of the image sensor 12 represents the vertical scanning direction.

  In FIG. 4D, the acquisition unit 21a instructs the image sensor 12 to read out image signals of, for example, lines L1, L2, L3, and L4 specified in advance in the image pickup region of the image sensor 12. Then, the image sensor 12 outputs a frame image signal based on the partial regions (lines L1 to L4). FIG. 5 schematically shows a state in which an image signal is output as an image every reading time of one frame. That is, in the present embodiment, one screen (one frame) can be read at a higher speed when thinning readout is performed than when thinning readout is not performed. In this way, the influence of the flickering of the fluorescent lamp is no longer a spatial change in horizontal stripes in one screen as shown in FIG. In other words, as shown in FIG. 5, the influence of the flickering of the fluorescent lamp is less uniform in one screen and becomes almost uniform in the screen, and is replaced with a change in brightness in the time axis direction of the entire frame screen. Here, the image signals of the frames based on the partial areas (lines L1 to L4) output from the image sensor 12 are A / D converted by the AFE 14 and sequentially stored in the RAM 16.

  Step S103: The average value calculation unit 21b of the CPU 21 calculates the average value of the brightness of the image signal for each frame.

  FIG. 6 is a partially enlarged view of the frame shown in FIG. For the sake of simplicity, FIG. 6 shows frames based on partial areas of four lines (L1 to L4) arranged in a Bayer array, and values of image signals corresponding to R, G, and B shown in FIG. Is recorded in the RAM 16.

  The average value calculation unit 21b sequentially reads out the image signals of the partial areas from the RAM 16, adds the signal values for each of R, G, and B, and then obtains an average value. Then, the CPU 21 temporarily records each average value for each frame in the RAM. As the sampling data for calculating the blinking frequency, the CPU 21 records, for example, an average value for each of 20 frames as a frame image. Note that 20 frames is an example, and as the number of frames increases, the frequency calculation unit 21c of the CPU 21 can more accurately calculate the blinking frequency.

  Step S104: The frequency calculation unit 21c of the CPU 21 calculates the blinking cycle based on the temporal change of the average value for each R, G, and B calculated for each frame. Specifically, the frequency calculation unit 21c first reads average values for R, G, and B for 20 frames from the RAM 16. Next, the frequency calculation unit 21c calculates a period in which a frame having a relatively low average value (meaning a frame with a dark screen) appears based on the frame rate. Then, the frequency calculation unit 21c calculates a blinking frequency (for example, 50 Hz or 60 Hz) based on a cycle in which a frame having a relatively low average value appears. Then, the frequency calculation unit 21c calculates a blinking cycle based on the blinking frequency. Note that the frequency calculation unit 21c may calculate the blinking frequency based on the period in which frames with relatively high average values appear. Alternatively, the frequency calculation unit 21c may calculate the blinking frequency based on a cycle in which a frame having a relatively low average value and a frame having a relatively high average value appear.

  Step S105: The CPU 21 determines whether or not the blinking cycle has been calculated. For example, the frequency calculating unit 21c cannot calculate the blinking cycle when the photographing is not performed under a fluorescent lamp. Therefore, when the frequency calculation unit 21c cannot calculate the blinking cycle (step S105: No), the process proceeds to step S108. The CPU 21 can arbitrarily set the rolling shutter speed during moving image shooting (step S108). That is, there is no influence of flicker. However, the setting of the rolling shutter speed at which the influence of flicker newly appears is excluded.

  On the other hand, when the frequency calculation unit 21c can calculate the blinking cycle (step S105: Yes), the process proceeds to step S106.

  Step S106: The CPU 21 sets the rolling shutter speed to n times the blinking cycle. As a result, flicker is not detected.

  Step S107: The CPU 21 starts moving image shooting. That is, the CPU 21 sends a drive signal to the image sensor 11 and the AFE 14 via the TG 13, and starts moving image shooting at a rolling shutter speed at which flicker is not detected if it is under a fluorescent lamp. Further, if it is not under a fluorescent lamp, moving image shooting can be started at an arbitrary rolling shutter speed. Then, the CPU 21 ends the process of the flowchart. Note that the CPU 21 may appropriately perform the sequence of the above flowchart even during moving image shooting.

  As described above, according to the electronic camera 1 of the present embodiment, it is not necessary to set a flicker detection frame. Thus, the electronic camera 1 can calculate the blinking frequency of the light source without complicating the arithmetic processing.

  That is, the CPU 21 can shorten the readout time of one frame by thinning the image sensor 12, and can read out an image signal in a shorter time than the blinking cycle (reciprocal of the blinking frequency) of the light source. As a result, the influence of the blinking of the light source results in a change in the brightness of the entire screen, so that the frequency calculating unit 21c can easily calculate the blinking frequency.

  Therefore, in the present embodiment, the blinking frequency can be calculated as a surface flicker generated between a plurality of frames, instead of the conventional method of detecting a plurality of flickers in one screen.

  Next, a modification of this embodiment will be described. In the present embodiment, the acquisition unit 21a instructs the image pickup device 12 to perform thinning readout in units of lines, thereby sequentially using image signals of partial areas that have been read out in units of lines as frames, at a predetermined frame rate (for example, 600 fps). It was set as the composition to acquire.

  In the modification of this embodiment, as shown in FIG. 1, a pixel addition circuit 12a including a first addition circuit and a second addition circuit described below is used. The first addition circuit adds the output values of the same color pixels in the same column in different rows of the imaging region. The second addition circuit adds the output values of the same color pixels in the same row in different columns of the imaging area. The acquisition unit 21a sequentially acquires the image signal after the output value is added at least one of the first addition circuit and the second addition circuit at a predetermined frame rate.

  FIG. 7 is a diagram for explaining a modification of the first embodiment. In the figure of the imaging region shown on the left side of FIG. 7 (a), R rows (R and G pairs) and B rows (B and G pairs) are alternately arranged. The lines are color-coded. Similarly, in the figure of the imaging region shown on the upper side of FIG. 7B, the R column (R, G group) and B column (B, G group) to be added are color-coded.

  In the diagram shown on the left side of FIG. 7A, the CPU 21 issues an instruction to the image sensor 12, so that the first adder circuit of the image sensor 12 is in the vertical direction (up and down direction in the figure) in different rows of the imaging area. The output values of adjacent pixels of the same color are added. The diagram shown on the right side of FIG. 7A schematically shows the result of addition.

  In the diagram shown in the upper side of FIG. 7B, the second addition circuit of the image sensor 12 adds the output values of pixels of the same color adjacent in the horizontal direction (left-right direction in the figure) in different columns of the imaging region. . The diagram shown at the bottom of FIG. 7B schematically shows the result of addition.

  The CPU 21 reads an image signal based on the output value after the addition in units of lines. Thereby, the acquisition unit 21a sequentially acquires the image signals of the partial areas read out from the imaging area. By doing so, the resolution of the image improves as the charge amount of each pixel increases, so that the frequency calculation unit 21c of the CPU 21 can calculate the blinking frequency more accurately. That is, in the present embodiment, when the image signal is read out after performing addition processing as shown in FIGS. 7A and 7B, the read time can be shortened.

  As described above, the CPU 21 can reduce the load of calculating the blinking frequency by causing the image sensor 12 to perform thinning readout and pixel addition processing.

  Note that the CPU 21 may combine the processing of the first addition circuit and the processing of the second addition circuit. Moreover, although the example which added the output value of the pixel of 2 columns or 2 rows was shown in FIG. 7, you may increase the number of the column or rows to add further. In addition, if the pixels have the same color, the image sensor 12 may add the output values of adjacent neighboring pixels.

  Further, the image sensor 12 may perform pixel addition by adopting a configuration in which a floating diffusion (FD), an amplifier, and the like are shared between the pixels to be added.

<Supplementary items of the embodiment>
(1) In this embodiment, in order to make the explanation easy to understand, the CPU 21 once records the image signal of the partial area in the RAM 16 as an image (frame). The average value calculation unit 21b may read the image signal for each line and sequentially calculate the average value for each of R, G, and B without being temporarily recorded as an image in the RAM 16, and may overwrite the RAM 16. In this way, the frequency calculation unit 21c can calculate the blinking frequency even faster.
(2) In the present embodiment, the average value calculation unit 21b may calculate only the average value of the G image signal, for example. Then, the frequency calculation unit 21c may calculate the blinking frequency using only the average value of the G image signals.
(3) In the present embodiment, three types of RGB color filters are employed. However, the present invention is not limited to this, and the average value calculation unit 21b may obtain an average value of brightness using a complementary color filter. The average value calculation unit 21b calculates the average brightness value for each of R, G, and B. However, the average value calculation unit 21b may determine the average brightness value by including all the RGB image signals.
(4) In this embodiment, the frequencies of 50 Hz and 60 Hz are adopted, but other frequencies may be used depending on the light source.
(5) In the present embodiment, the image pickup device 12 that picks up the main image to be recorded on the recording medium 30 is used as the flicker detection sensor. However, the image pickup device is provided separately from the image pickup device 12 that picks up the main image. (For example, a photometric sensor) may be employed.
(6) In the present embodiment, the AFE 14 is employed as an example of the analog output sensor. However, a type incorporating the TG 13 may be employed as the analog output sensor. In the present invention, a digital output sensor may be employed instead of the analog output sensor. In this case, the AFE 14 is built in the digital output sensor. In addition, the digital output sensor may include TG 13, image processing unit 15, RAM 16, ROM 17, CPU 21, etc. in any combination.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 12 ... Image sensor, 12a ... Pixel addition circuit, 21a ... Acquisition part, 21b ... Average value calculation part, 21c ... Frequency calculation part

Claims (8)

An acquisition section for sequentially acquiring the images signals captured area or al thinning read changeable imaging element rolling shutter speed,
An average value calculating unit for calculating an average value of the brightness of the image signal for each frame;
A frequency calculation unit that calculates a frequency of a light source based on a temporal change of the average value calculated for each frame;
An imaging apparatus comprising: a setting unit configured to set a rolling shutter speed during moving image shooting using the frequency of the light source calculated by the frequency calculation unit . The imaging apparatus of claim 1,
The image pickup device is a color image pickup device in which any one of a plurality of color filters is arranged in each pixel, and a first addition circuit that adds output values of the same color pixels in the same column in different rows of the image pickup region; Including at least one of a second addition circuit that adds output values of the same color pixels in the same row in different columns of the imaging region;
The acquisition unit, an imaging apparatus and obtains an image signal after the output value has been added at least one of said first adder circuit and the second adding circuit with line units. An acquisition unit for sequentially acquiring image signals read out from the imaging region of the imaging device capable of changing the rolling shutter speed;
A calculation unit that calculates the brightness of the image signal for each frame;
A control unit that controls a rolling shutter speed during moving image shooting based on a temporal change in brightness of the image signal calculated for each frame by the calculation unit;
An imaging apparatus comprising: The imaging apparatus according to claim 3,
The control unit calculates a cycle of a light source based on a temporal change in brightness of the image signal calculated for each frame by the calculation unit, and uses the calculated cycle of the light source to capture the moving image. Control the rolling shutter speed
An imaging apparatus characterized by that. An imaging apparatus according to claim 4, wherein
The control unit controls a rolling shutter speed at the time of moving image shooting to be an integral multiple of a cycle of the light source.
An imaging apparatus characterized by that. An imaging apparatus according to any one of claims 3 to 5,
The rolling shutter speed at which the acquisition unit sequentially acquires the image signals is different from the rolling shutter speed at the time of moving image shooting.
An imaging apparatus characterized by that. An imaging apparatus according to any one of claims 3 to 6,
The calculation unit calculates an average value of the brightness of the plurality of image signals for each frame,
The control unit controls a rolling shutter speed at the time of moving image shooting based on the average value calculated by the calculation unit.
An imaging apparatus characterized by that. An imaging apparatus according to any one of claims 3 to 7,
The image pickup device is a color image pickup device in which any one of a plurality of color filters is arranged in each pixel, and a first addition circuit that adds output values of the same color pixels in the same column in different rows of the image pickup region; Including at least one of a second addition circuit that adds output values of the same color pixels in the same row in different columns of the imaging region;
The acquisition unit acquires, in line units, an image signal after the output value is added to at least one of the first addition circuit and the second addition circuit.
An imaging apparatus characterized by that.

Images