JP2015019330A - Imaging device, method of controlling imaging device, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an excellent image signal by putting together a plurality of image signals obtained by successive imaging even if brightness modulation is imposed on an image signal.SOLUTION: Two image signals are picked up successively while the timing when electric charge of an imaging element 101 begins to be accumulated is shifted by a time based upon a representative light emission time (average light emission time) of an external electronic flash and a readout time of one line. Integral values (line integral value) of respective lines of those two image signals are derived, and the derived line integral values are compared with a threshold Th so as to determine a brightness modulation area. According to respective brightness modulation areas of the two image signals, an area where the two image signals are put together and an area where one of the two image signals is used are determined.

Description

  The present invention relates to an imaging apparatus, an imaging apparatus control method, and a computer program, and is particularly suitable for use in synthesizing and outputting a plurality of consecutively captured image signals.

  In an image pickup apparatus including a CMOS image sensor (an image pickup element adopting a line sequential scanning method), an image may be picked up in a situation where an external strobe (for example, a light emitting means provided in another image pickup apparatus) is sown. is there. In this case, since it becomes impossible to perform imaging under the exposure condition determined on the imaging device side, luminance modulation due to an external strobe occurs in the image signal, and significant image degradation occurs. In particular, in an imaging apparatus that synthesizes a plurality of consecutively captured image signals and outputs them as a single image signal, luminance modulation by an external strobe appears in the combined image signal, which may cause discomfort to the user. .

Therefore, Japanese Patent Laid-Open No. 2004-26883 discloses an imaging in which an image signal of a frame affected by an external strobe is replaced with an image signal obtained by synthesizing image signals of preceding and succeeding frames among a plurality of image signals continuously captured. An apparatus is disclosed.
Japanese Patent Application Laid-Open No. 2004-228561 discloses an imaging apparatus that captures an image at a frame rate that is an integral multiple of a predetermined format, and selects only an image signal that is not affected by an external strobe to synthesize the image signal.

JP 2007-306225 A JP 2011-015381 A

However, in the imaging apparatus described in Patent Document 1, the image signal of the frame affected by the external strobe is replaced with an image signal obtained by synthesizing the image signals of the preceding and succeeding frames. Therefore, there is a problem that the temporal continuity of the subject being imaged cannot be maintained despite continuous image capturing.
Further, in the imaging device described in Patent Document 2, an image signal that is partially affected is not a target of a plurality of image signals to be combined. For this reason, there is a problem that the entire image signal is not used even though there is a line that is normally imaged. In addition, in the imaging apparatus described in Patent Document 2, when only the image signals that are not affected by the external strobe are selected and combined, level compensation for the missing frame is performed in order to match the levels of the combined image signals. Must. Therefore, it is necessary to adjust the level substantially by increasing the gain, and there is a problem that the image has a low S / N ratio.
The present invention has been made in view of such problems, and can generate a good image signal by synthesizing a plurality of continuously captured image signals even when luminance modulation occurs in the image signal. The purpose is to do so.

  An image pickup apparatus according to the present invention is an image pickup apparatus that generates an image signal by combining a plurality of image signals continuously picked up by an image pickup element having a plurality of pixels arranged in a matrix. Setting means for setting a shift time, which is a time from the start of charge accumulation in the image pickup device for obtaining the next image signal after reading out the charge in the image pickup device for obtaining; Based on the shift time set by the setting means, the image sensor control means for controlling the operation of the image sensor, and the plurality of image signals captured by the image sensor whose operation is controlled by the image sensor control means. Based on the brightness value, a brightness modulation area, which is a brightness-modulated area, is determined by each of the plurality of image signals by the area determination means and the area determination means. The operation is controlled by the image sensor control means with a composite ratio calculation means for calculating a composite ratio of the plurality of image signals based on the luminance modulation area and a composite ratio calculated by the composite ratio calculation means. And combining means for combining the plurality of image signals picked up by the image pickup device.

  According to the present invention, even when luminance modulation occurs in an image signal, a good image signal can be generated by synthesizing a plurality of continuously captured image signals.

It is a figure which shows the 1st example of a structure of an imaging device. It is a figure explaining the outline | summary of operation | movement of the imaging device of a 1st example. It is a figure explaining the conditions which an image signal receives to the influence of an external strobe. It is a figure explaining operation | movement of the imaging device of a 1st example. It is a figure explaining the timing of the whole control of the imaging device of the 1st example. It is a figure explaining the determination method of a brightness modulation area. It is a figure explaining the method to produce | generate a synthetic | combination ratio control signal. It is a figure which shows the modification of a structure of the imaging device of a 1st example. It is a flowchart explaining the outline of operation | movement of the imaging device of a modification. It is a figure which shows the other modification of a structure of the imaging device of a 1st example. It is a figure which shows the 2nd example of a structure of an imaging device. It is a figure which shows the relationship between the luminance value and pixel value of a low exposure and high exposure image signal. It is a figure which shows the histogram of the pixel value of a low exposure and high exposure image signal. It is a figure explaining operation | movement of the image pick-up element of a 2nd example. It is a figure explaining the timing of the whole control of the imaging device of the 2nd example. It is a figure explaining the method to determine a synthetic | combination ratio. It is a figure which shows the 3rd example of a structure of an imaging device. 10 is a flowchart illustrating an outline of operation of an imaging apparatus according to a third example. It is a figure explaining the method to determine a luminance modulation area. It is a figure which shows the 4th example of a structure of an imaging device. It is a figure explaining the timing of the whole control of the imaging device of the 4th example. It is a figure explaining the mode of accumulation of electric charge in an image sensor. It is a figure which shows the image signal when not shifting the reset start timing.

Prior to describing the embodiments of the present invention, the technology that is the premise of the embodiments of the present invention will be described.
FIG. 22 is a diagram for explaining the state of charge accumulation in the image sensor. Specifically, FIG. 22 is a diagram for explaining the state of charge accumulation in the image sensor when a subject is continuously photographed as a moving image with an imaging device using a CMOS image sensor in a situation where an external strobe is emitted. It is.
A chart 2201 shows the timing at which the external strobe emits light. In the rising portion 2202 of the chart 2201, the external strobe emits light for the light emission time t.
A chart 2203 shows the vertical synchronization signal. An image signal is read from the CMOS image sensor at a timing at which a vertical synchronization signal determined by an imaging method of the imaging apparatus is asserted (see a portion protruding downward in the chart 2203).

  Charts 2204, 2205, and 2206 show readout periods of image signals from the CMOS image sensor provided in the imaging apparatus. Charts 2204, 2205, and 2206 show how the exposure timing for each pixel is sequentially performed in the line direction. The charge of each pixel that is the basis of the image signal read according to the chart 2205 is accumulated in a period between the charts 2204 and 2205. Then, the external strobe emits light within this period (see the light emission time t of the external strobe in FIG. 22). For this reason, the region 2207 is affected by the external strobe, so that the region 2207 is photographed with overexposure instead of the proper exposure condition determined by the imaging device. Similarly, the charge of each pixel that is the basis of an image signal read according to the chart 2206 is accumulated in a period between the charts 2205 and 2206. Then, the external strobe emits light within this period (see the light emission time t of the external strobe in FIG. 22). For this reason, the region 2208 is also affected by the external strobe, so that an image is overexposed instead of the appropriate exposure condition determined by the imaging apparatus.

FIG. 23 is a diagram illustrating an image signal when an image is captured under the above conditions.
An image signal 2301 indicates an image signal (image X) read according to the chart 2205 in FIG. An image signal 2302 schematically shows an image signal (image Y) read according to the chart 2206 in FIG.
As described with reference to FIG. 22, in FIG. 23, the line group corresponding to the region 2207 is affected by the luminance modulation. Similarly, the line group corresponding to the region 2208 is affected by the luminance modulation. When the image X and the image Y are combined, as a result, the logical sum of the line where the luminance modulation is generated in the image X and the line where the luminance modulation is generated in the image Y is calculated. . Therefore, an image subjected to luminance modulation by the external strobe over all lines is generated. The combined signal level 2304 indicates the signal level of the image signal 2303 obtained by combining the image X and the image Y. The combined signal level 2304 may be a higher level image signal than the original signal level 2305 obtained after the image pickup apparatus combines the image signals. An area 2306 is an area where a modulation area is generated by an external strobe in both the images X and Y. Accordingly, the line belonging to the region 2306 has a higher signal level than the surrounding lines, and this signal level may appear as a band on the synthesized image signal 2303.
The following embodiments of the present invention have been made in consideration of the above-described technique.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
First, a first embodiment of the present invention will be described.
FIG. 1 is a block diagram illustrating an example of the configuration of the imaging apparatus according to the present embodiment.
The image sensor 101 is configured by arranging pixels having photoelectric conversion elements in a matrix (two-dimensional matrix). The photoelectric conversion element provided in each pixel converts the subject image into an electrical signal (analog signal) by photoelectric conversion. In the present embodiment, the image sensor 101 is configured using a CMOS sensor. The image sensor 101 is driven with a drive pulse based on a control signal generated by the image sensor controller 102. The analog signal subjected to the photoelectric conversion is converted into a digital signal by an AD conversion unit provided in the image sensor 101 and is output from the image sensor 101. At this time, when the image sensor 101 has an electronic shutter function, the exposure time may be ensured by the control signal from the image sensor control unit 102 so that the required exposure time is obtained.

A luminance modulation area determination unit 103 determines a luminance modulation area that is an area subjected to luminance modulation by an external strobe from an image signal obtained from the image sensor 101, and synthesizes luminance modulation area information that is information indicating a result of the area determination. Output to the ratio calculation unit 104.
The composition ratio calculation unit 104 includes a memory 105 and a composition ratio calculation unit 106. The composite ratio calculation unit 106 calculates a composite ratio between the image signal output from the image sensor 101 and the image signal of the previous frame output from the frame memory 107, and outputs the composite ratio to the image composition unit 108 as a composite ratio control signal. To do. The calculation of the combination ratio is performed based on the luminance modulation area information output from the luminance modulation area determination unit 103 and the luminance modulation area information of one frame before output from the memory 105.

The frame memory 107 holds an image signal output from the image sensor 101 for one frame.
The image synthesis unit 108 synthesizes the image signal output from the frame memory 107 and the image signal output from the image sensor 101 to generate one image signal. The image composition unit 108 includes the following elements. The combination ratio adjustment unit 109 performs gain processing on the image signal output from the image sensor 101 according to the combination ratio control signal output from the combination ratio calculation unit 104. The combination ratio adjustment unit 110 performs gain processing on the image signal output from the frame memory 107 according to the combination ratio control signal output from the combination ratio calculation unit 104, as in the combination ratio adjustment unit 109. . The synthesizing unit 111 adds the two image signals after adjusting the synthesis ratio output from the synthesis ratio adjusting units 109 and 110, and outputs the resultant as a synthesized image 112.
The control unit 113 controls the entire imaging apparatus. The control unit 113 also has a role of determining a charge accumulation time of the image sensor 101 according to an exposure condition or the like in imaging and passing a control signal to the image sensor control unit 102.

Next, the influence that appears in the image signal due to the light emission of the external strobe which is an example of the external light emitting means will be described with reference to FIGS. FIG. 2 is a diagram for explaining an example of the outline of the operation in the image sensor 101. Specifically, FIG. 2 is a diagram for explaining an example of the operation of the image sensor 101 when the charge reset timing is delayed by the shift time (shift amount) Δt in the imaging with the image sensor 101. This shift amount Δt corresponds to the time from the start of charge accumulation in the image sensor 101 for obtaining a certain image signal to the start of charge accumulation in the image sensor 101 for obtaining the next image signal. .
A chart 201 shows the timing at which the external strobe emits light. The external strobe emits light at a rising portion 202 of the chart 201 for a light emission time t. In the example shown in FIG. 2, the vertical synchronization signal 203 determined by the imaging method of the imaging apparatus is asserted four times.

Charts 205, 206, 207, and 208 show the readout period of each image signal. Each image signal is read from the image sensor 101 at the timing when the vertical synchronization signal 203 is asserted. Time tl indicates the readout time for one line.
Charts 209 and 210 indicate that a reset signal is output. Specifically, the chart 209 indicates that a reset signal for determining the charge accumulation time of the image signal shown in the chart 206 is output. A chart 210 indicates that a reset signal that determines the charge accumulation time of an image signal read according to the chart 207 is output.

  When the external strobe emits light at the timing shown in FIG. 2, for the image signal read according to the chart 206, the region 211 of the image signal 204 with N lines is affected by the external strobe. For the image signal read in accordance with the chart 207, the area 212 of the image signal 204 is affected by the external strobe.

FIG. 3 is a diagram for explaining conditions under which an image signal is affected by an external strobe. A description of the portions denoted by the same reference numerals is omitted.
A period between the light emission start timing 301 of the external strobe and the light emission end timing 302 corresponds to the light emission time t of the external strobe described above. Further, the charge accumulation time in the image sensor 101 is determined by the timing shift amount Δt at the start of reset (start of reset signal output). A line subjected to luminance modulation by an external strobe generated between two consecutive image signals (between image signals read in accordance with charts 206 and 207) when there is no reset start timing shift amount Δt (Δt = 0). X corresponds to the area 2306 in FIG. The line X subjected to luminance modulation by the external strobe is calculated by the following (Equation 1).
X = t / tl (Formula 1)
Here, tl is the readout time for one line in the image sensor 101 as described above (see FIG. 2). Further, t is the light emission time of the external strobe as described above (see FIG. 2).

Next, reading of an image signal in consideration of the reset start timing shift amount Δt will be described.
Of the lines subjected to luminance modulation by the external strobe generated between two consecutive image signals, the number of lines in which the overlap of the influence of the external strobe is eliminated by the reset start timing shift amount Δt is set to ΔX. Then, a line X ′ subjected to luminance modulation by an external strobe generated between two consecutive image signals can be expressed as (Equation 2) below.
X ′ = X−ΔX (Expression 2)
From FIG. 3, ΔX indicates the number of lines that can be read at time 303 (that is, reset start timing shift amount Δt), and can be expressed as in (Equation 3) below.
ΔX = Δt / tl (Expression 3)

From the above, the line X ′ subjected to luminance modulation by the external strobe generated between the two image signals in consideration of the reset start timing deviation Δt is expressed by (Expression 1), (Expression 2), (Expression 3). Thus, the following (Formula 4) is obtained.
X ′ = (1 / tl) × (t−Δt) (Expression 4)

Here, (Equation 4) is subjected to luminance modulation by an external strobe generated between two consecutive image signals when the reset start timing deviation amount Δt is equal to or longer than the light emission time t of the external strobe. It means that the line becomes 0 (zero). That is, in such a case, it is shown that appropriate imaging is performed with either one of the image signals in all lines.
The control unit 113 performs a shift time derivation process based on (Expression 4). Specifically, the control unit 113 determines a reset start timing shift amount Δt such that the line X ′ shown in (Expression 4) becomes 0 (zero) based on the average flash time of the external strobe. And the control part 113 passes a control signal to the image pick-up element control part 102 so that it may image with this deviation | shift amount (DELTA) t. The average flash time of the external strobe is determined in advance based on the strobe flash time of a typical strobe light emitting means, and is stored in a nonvolatile storage medium or the like in the control unit 113 as information in advance in the imaging device. Retained.

FIG. 4 is a diagram for explaining the operation of the image sensor 101 in the present embodiment. A description of the portions denoted by the same reference numerals is omitted.
Charts 402 and 403 are reset signals that determine the charge accumulation time of image signals read according to charts 206 and 207, respectively. This reset signal is scanned with an offset of the shift start timing difference Δt. This reset start timing shift amount Δt is determined by giving the average flash time of the external strobe to (Equation 4).

When the external strobe emits light at a light emission time t between the light emission start timing 301 and the light emission end timing 302 shown in FIG. 4, for an image signal read according to the chart 206, the region 404 is subjected to luminance modulation by the external strobe. For the image signal read according to the chart 207, the area 405 is affected by the luminance modulation by the external strobe.
Image signals 406 and 407 are schematic diagrams of image signals read according to charts 206 and 207, respectively. When attention is paid to the areas 404 and 405 that have been subjected to luminance modulation by the external strobe in the two image signals 406 and 407 that are continuously photographed, appropriate imaging is achieved with either one of the image signals in all lines. Become so.

Next, an example of the timing of overall control of the imaging apparatus according to the present embodiment will be described with reference to FIG. A description of the portions denoted by the same reference numerals is omitted.
An image signal 501 indicates an image signal captured by the image sensor 101. Image signals read in accordance with charts 206, 207, and 208 are image A, image B, and image C, respectively.
An image signal 502 indicates an image signal output from the frame memory 107. The image signal output from the image sensor 101 is output from the frame memory 107 after being delayed by one frame.
The luminance modulation area information 503 indicates the luminance modulation area information output from the luminance modulation area determination unit 103. The result (luminance modulation area) determined from the image A is represented by (a). Similarly, the results (brightness modulation areas) determined from the images B and C are represented by (b) and (c), respectively.

With reference to FIG. 6, an example of a method for determining the luminance modulation area in the luminance modulation area determination unit 103 will be described.
An image signal 601 schematically shows an image signal (image A) captured by the image sensor 101. The luminance modulation area determination unit 103 calculates an integral value of each line of the input image signal. In the present embodiment, the image sensor 101 is a CMOS sensor. Therefore, the luminance modulation area generated on the image signal by the external strobe is uniformly generated in the line direction (on the same line) as the luminance modulation area information 603 in FIG. Therefore, the luminance modulation area determination unit 103 can determine whether or not the luminance modulation is performed by integrating the luminance value of the image signal in the line direction and evaluating the line integral value that is the integrated value.

  In FIG. 6, the line integration value 602 calculated by the luminance modulation area determination unit 103 is shown together with the image signal 601. The horizontal axis represents the line integral value, and the vertical axis represents the number of lines. The luminance modulation area determination unit 103 has a predetermined threshold Th, and determines that a line corresponding to the line integration value is a luminance modulation area when the calculated line integration value is larger than the threshold Th. On the other hand, when the line integral value is not larger than the threshold Th, it is determined that the line corresponding to the line integral value is not the luminance modulation region. It should be noted that the luminance modulation region can be determined with higher accuracy by determining the threshold Th with reference to the result of the line integral value of the past frame.

  In FIG. 6, the luminance modulation area information 603 output from the luminance modulation area determination unit 103 is shown together with the image signal 601. The horizontal axis represents the value of luminance modulation area information, and the vertical axis represents the number of lines. The luminance modulation area determination unit 103 outputs 1 as the luminance modulation area information for a line that has undergone luminance modulation, and outputs 0 (zero) as the luminance modulation area information for a line that does not.

Returning to FIG. 5 again, the overall control of the imaging apparatus of the present embodiment will be described.
The luminance modulation area information 504 indicates luminance modulation area information output from the memory 105 provided in the synthesis ratio calculation unit 104. The luminance modulation area information output from the luminance modulation area determination unit 103 is output from the memory 105 after being delayed by one frame.
A combination ratio control signal 505 indicates a combination ratio control signal generated by the combination ratio calculation unit 106 based on the luminance modulation area information output from the luminance modulation area determination unit 103 and the memory 105. Specifically, the combination ratio control signal γ1 indicates a combination ratio control signal calculated based on the luminance modulation area information (a) and (b), and the combination ratio control signal γ2 indicates the luminance modulation area information (b) and The synthesis ratio control signal calculated based on (c) is shown.

FIG. 7 is a diagram for explaining an example of a method for generating a synthesis ratio control signal. Specifically, FIG. 7 is a diagram illustrating an example of a method of generating the composite ratio control signal γ1 based on the luminance modulation region information (a) and (b) in the composite ratio calculation unit 106.
Image signals 701 and 702 schematically show images A and B, respectively. Graphs 703 and 704 show the luminance modulation area information (a) and (b) generated by the luminance modulation area determination unit 103, respectively. The luminance modulation area information (a) is output from the memory 105, and the luminance modulation area information (b) is output from the luminance modulation area determination unit 103.

Based on the two pieces of luminance modulation area information (a) and (b), the combination ratio calculation unit 106 generates a combination ratio control signal γ1 such that the luminance modulation area by the external strobe does not appear in the combined image 112.
A graph 705 schematically shows the synthesis ratio control signal γ1 generated by the synthesis ratio calculation unit 106. In the region 706, the image B has a luminance modulation region. Therefore, the composition ratio calculation unit 106 generates the composition ratio control signal γ1 so that the image B is not used in the region 706 (that is, the image A is used). In the region 707, the image A has a luminance modulation region. Accordingly, the composition ratio calculation unit 106 generates the composition ratio control signal γ1 so that the image A is not used in the region 707 (that is, the image B is used). In the present embodiment, the composite ratio control signal γ1 takes 0 (zero) or 1. When the value of the composite ratio control signal γ1 is 0 (zero), the image B is used and is 1. Indicates that the image A is used. An area 708 is an area where no luminance modulation area is generated in either the image A or the image B. Therefore, the synthesis ratio calculation unit 106 generates the synthesis ratio control signal γ1 so as to gradually change the synthesis ratio in the region 708. For the region 708, the value of the synthesis ratio control signal γ1 may be fixed at 0.5, or may be either 0 (zero) or 1.

Returning to FIG. 5 again, the description of the overall control of the imaging apparatus of the present embodiment will be continued.
A composite image 506 indicates a combination of the image A output from the frame memory 107 and the image B output from the image sensor 101 by the combining unit 111. In the synthesis ratio adjusting units 109 and 110, based on the synthesis ratio based on the synthesis ratio control signal γ1 output from the synthesis ratio calculation unit 106, gain processing is performed on the image signals (images A and B) at the synthesis ratio. In the present embodiment, the image B is used when the value of the synthesis ratio control signal γ1 is 0 (zero). For this reason, the composition ratio adjustment unit 109 performs the composition ratio obtained by performing the following (Equation 5) operation on the received composition ratio control signal γ1, and the input image signal (image B). Γ1 ← (1-γ1) (Formula 5)
The combining unit 111 adds pixel values corresponding to each other of the image signals (images A and B) after the combining ratio is adjusted, and generates and outputs a combined image 112.

  As described above, in the present embodiment, the charge accumulation time in the image sensor 101 is determined based on the typical light emission time (average light emission time) of the external strobe and the readout time of one line, and two image signals are obtained. Take images continuously. That is, the timing at which the image sensor 101 starts accumulating charges is shifted by a time based on the typical light emission time (average light emission time) of the external strobe and the readout time of one line, and the two image signals are continuously input. Take an image. For each of these two image signals, an integrated value (line integrated value) of each line is derived, and the derived line integrated value is compared with a threshold value Th to determine the luminance modulation region. A region where two image signals are synthesized and a region where a part (one) of the two image signals is used are determined according to the respective luminance modulation regions of the two image signals. Accordingly, it is possible to minimize image quality degradation by generating an image signal using a region that has not undergone luminance modulation without causing frame loss.

<Modification 1>
According to this embodiment, the charge accumulation time in the image sensor 101 is determined based on the typical light emission time (average light emission time) of the external strobe. Therefore, the exposure in the imaging device may be lower than the original exposure. In this case, the brightness value (amplification factor) set for a gain circuit (amplifier) (not shown) provided in the imaging apparatus is controlled to amplify (gain up) the signal value of the composite image, thereby obtaining an appropriate brightness. It is also possible.

<Modification 2>
Further, it is possible to determine whether or not to perform the control described in the present embodiment by determining the shooting situation. In this way, in a situation where the external strobe is not blown, the charge accumulation time in the image sensor 101 is determined based on the representative light emission time (average light emission time) of the external strobe so that no image is taken. Is possible.
FIG. 8 is a block diagram illustrating an example of the configuration of the imaging apparatus according to this modification. As a configuration of the imaging apparatus, it is assumed that a control line 801 is added to the imaging apparatus shown in FIG.
The control line 801 is used to transmit a control signal output from the control unit 113 to the synthesis ratio calculation unit 106, and specifically, to control a synthesis ratio control signal generated by the synthesis ratio calculation unit 106. Used for.

FIG. 9 is a flowchart for explaining an example of an outline of the operation of the imaging apparatus according to the present modification.
First, in step S901, the control unit 113 acquires an F value (F), an accumulation time (T), and a gain value (G) that are exposure control parameters of the imaging apparatus.
Next, in step S902, the control unit 113 performs a room determination calculation. The situation in which the external strobe emits light is mostly under an illumination light source such as a room. Therefore, the control unit 113 determines whether the image is taken indoors (or in an illumination environment equivalent to the indoor environment) using the exposure control parameter obtained in step S901. The determination is performed based on, for example, the following determination formula (Formula 6).
Evaluation value = (Fd / F) ² × (T / Td) × (G / Gd) (Expression 6)

Here, Fd, Td, and Gd are generally exposure control parameters (F value, accumulation time, and gain value) under indoor shooting conditions, and are stored in advance in a nonvolatile storage medium or the like in the imaging apparatus. deep.
Next, in step S903, the control unit 113 determines whether or not the evaluation value of (Expression 6) is 1 or more. As a result of the determination, if the evaluation value is 1 or more, it is determined that the image is taken indoors, and the process proceeds to step S904. On the other hand, if the evaluation value is less than 1, it is determined that the image is not taken indoors, and the process proceeds to step S905.

  In step S904, as described in the present embodiment, the imaging apparatus determines the charge accumulation time in the image sensor 101 based on the typical light emission time (average light emission time) of the external strobe and the readout time of one line. Then, the two image signals are continuously imaged. At this time, the control unit 113 passes a control signal to the image sensor control unit 102. Upon receiving this control signal, the image sensor control unit 102 passes the control signal to the image sensor 101 so that an image is taken in a desired charge accumulation time.

On the other hand, when the processing proceeds to step S905, the control unit 113 generates a control signal indicating that the composite ratio control signal is a constant value (0.5 or the like) regardless of the luminance modulation area information. 106. Further, the image sensor control unit 102 passes a control signal to the image sensor 101 so that the timing at which charge accumulation in the image sensor 101 starts does not deviate significantly (so that the time Δt in FIG. 2 becomes a predetermined value).
As described above, it is possible to adaptively switch the imaging method by determining whether or not it is a situation in which an external strobe is blown.

<Modification 3>
Further, as in the present embodiment, when the charge accumulation time in the image sensor 101 is determined based on the typical light emission time (average light emission time) of the external strobe, the exposure is lower than the original exposure that the image pickup apparatus wants to shoot. Become. An imaging apparatus that includes an F value as an exposure control parameter (that is, includes an aperture control mechanism) can also capture images with appropriate brightness by controlling the F value (aperture value).
FIG. 10 is a diagram illustrating an example of the configuration of the imaging apparatus according to the present modification. The imaging apparatus of this modification is obtained by adding an optical system 1001 and an optical system control unit 1002 that controls the optical system 1001 to the imaging apparatus shown in FIG.
The control unit 113 passes the control signal to the optical system control unit 1002. Upon receiving this control signal, the optical system control unit 1002 is based on the charge accumulation time of the image sensor 101 determined based on the typical light emission time (average light emission time) of the external strobe and the readout time of one line. The F value reflecting the exposure corresponding to the underexposure is determined. The optical system control unit 1002 controls the diaphragm of the optical system 1001 so that the determined F value is obtained. By doing in this way, it becomes possible to image | photograph with appropriate brightness.

<Modification 4>
In the present embodiment, the case where two image signals captured in succession are combined has been described as an example. However, the number of image signals to be combined may be three or more.
In this case, with respect to areas corresponding to each other of a plurality of image signals continuously captured, it is determined that some image signals are luminance modulation areas, and other partial image signals are not luminance modulation areas. If determined, the other part of the image signal can be used for the region. When there are a plurality of the other partial image signals, the plurality of image signals may be combined, or one of the other partial image signals may be selected.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In this embodiment, a configuration in which a high dynamic range image signal is synthesized and generated from a plurality of image signals captured at different exposures is added to the first embodiment. As a result, it is possible to generate a high dynamic range image that is not affected even when an external strobe emits light. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

FIG. 11 is a block diagram illustrating an example of the configuration of the imaging apparatus according to the present embodiment.
In FIG. 11, the control unit 113 determines an exposure amount when shooting with a high dynamic range. The image sensor control unit 102 controls the exposure amount of the image sensor 101 in accordance with a control signal passed from the control unit 113. Further, the control unit 113 determines a level adjustment gain for level adjustment performed on the image signal in accordance with the exposure amount determined by the image sensor control unit.
The image combining unit 1101 generates a single image signal by combining a plurality of image signals. The image composition unit 1101 includes level adjustment units 1102 and 1103, a motion detection unit 1104, a composition ratio determination unit 1105, composition ratio adjustment units 109 and 110, and a composition unit 111.
Level adjustment units 1102 and 1103 adjust the levels of image signals read from the image sensor 101 in accordance with control signals from the control unit 113, respectively.

The motion detection unit 1104 obtains difference information between the image signals of two image signals whose levels are matched, detects a motion vector, and combines the motion detection information including the detected motion vector with the synthesis ratio determination unit 1105. To pass.
A combination ratio determination unit 1105 determines a combination ratio for combining two image signals. This combination ratio includes the motion detection information output from the motion detection unit 1104, the image signal after level adjustment output from the level adjustment units 1102 and 1103, and the combination ratio control signal output from the combination ratio calculation unit 106. To be determined.

  Next, an example of a procedure for capturing two images with different exposures and combining them will be described. In the following, an example in which a relatively low exposure image (low exposure image) is first captured, and then a relatively high exposure image (high exposure image) is captured and combined. I will give you a description. However, the order of capturing the low exposure image and the high exposure image may be reversed.

First, the control unit 113 sets an exposure step with low exposure. Here, a case where an exposure level of −2 is set with respect to proper exposure will be described as an example.
The image sensor control unit 102 controls the exposure amount of the image sensor 101 so that the exposure amount corresponding to −2 steps set by the control unit 113 is obtained. That is, if only the shutter speed is controlled, the image sensor control unit 102 sets an exposure time that is ¼ times the appropriate exposure time. Further, if only the sensor gain unit included in the image sensor 101 is controlled, the image sensor control unit 102 sets a gain that is ¼ times that at the appropriate exposure to the sensor gain unit. Only one of these is set, or the exposure amount of -2 steps is set by a combination thereof.

The image sensor 101 performs low-exposure shooting in accordance with the exposure amount (exposure time and gain setting values) set by the image sensor control unit 102, and outputs a low-exposure image signal. Thus, the image signal obtained by the image sensor 101 is temporarily held in the frame memory 107.
Next, the control unit 113 sets an exposure step with high exposure. Here, a case where an exposure step of +2 is set with respect to proper exposure will be described as an example.
The image sensor control unit 102 controls the exposure amount of the image sensor 101 so that the exposure amount corresponds to +2 steps set by the control unit 113. That is, if only the shutter speed is controlled, the image sensor control unit 102 sets an exposure time four times as long as the proper exposure. Further, if only the sensor gain unit provided in the image sensor 101 is controlled, the image sensor control unit 102 sets a gain four times that at the time of proper exposure in the sensor gain unit. Only one of these is set, or a combination of these is set to +2 steps of exposure amount.
The image sensor 101 performs high-exposure shooting in accordance with the exposure amount (exposure time and gain setting values) set by the image sensor control unit 102, and outputs a high-exposure image signal.

Here, the control unit 113 performs level setting for the level adjustment units 1102 and 1103 in order to match the levels of the low-exposure image signal and the high-exposure image signal when combining two image signals. That is, the control unit 113 sets a level adjustment gain four times that at the time of proper exposure in the level adjustment unit 1103 that performs level adjustment of the low-exposure image signal. Similarly, the control unit 113 sets a level adjustment gain that is ¼ times that at the appropriate exposure to the level adjustment unit 1102 that performs level adjustment of the high-exposure image signal.
The low-exposure image signal is read from the frame memory 107, level-adjusted by the level adjustment unit 1103, and sent to the motion detection unit 1104, the composition ratio determination unit 1105, and the composition ratio adjustment unit 110. The high-exposure image signal is level-adjusted by a level adjustment unit 1102 and sent to a motion detection unit 1104, a composition ratio determination unit 1105, and a composition ratio adjustment unit 110.

The motion detection unit 1104 detects the motion detection information (for example, motion vector) in the image by comparing the low exposure image signal and the high exposure image signal whose levels are matched. For example, the motion detection unit 1104 divides the low-exposure image signal and the high-exposure image signal into fixed regions, obtains a difference between pixel values in each region corresponding to each other, and calculates an absolute value of the difference value. If is large, it is determined that the region is a motion region. In addition, since the detection of motion detection information is realizable with a well-known technique, the detailed description is abbreviate | omitted here.
The composition ratio determination unit 1105 determines the composition ratio based on the brightness information of the image signal after level adjustment, the motion detection information from the motion detection unit 1104, and the composition ratio control signal from the composition ratio calculation unit 106. To do.

More specifically, the composition ratio determination unit 1105 first compares the level-adjusted image signals (the low exposure image signal and the high exposure image signal). The composition ratio determining unit 1105 determines the composition ratio so that the low-exposure image signal is mainly synthesized in the bright part of the image, and the high-exposure image signal is mainly synthesized in the dark part of the image. Set the composition ratio to. In addition, the composition ratio determination unit 1105 outputs one of the low-exposure image signal and the high-exposure image signal for the region determined as the motion region, and the region that is not the motion region. Determines the composition ratio so as to output the other image signal. In the present embodiment, it is assumed that a low exposure image signal is output for an area determined to be a motion area. By doing so, it is possible to avoid deterioration in image quality such as accumulation blur due to the charge accumulation operation, and it is possible to avoid degradation in image quality such as motion region blur in the synthesized image signal.
In the present embodiment, the combination ratio is further determined based on the combination ratio control signal output from the combination ratio calculation unit 106, and the detailed contents thereof will be described later.

FIG. 12 is a diagram illustrating an example of a relationship between a luminance value and a pixel value of the low exposure image signal and the high exposure image signal. In FIG. 12, the horizontal axis indicates the luminance value of the captured image signal, and the vertical axis indicates the magnitude of the pixel value of the image data.
FIG. 12A shows the relationship before level matching is performed. In FIG. 12A, a white line L indicates the value of the low exposure image signal, and a black thick line H indicates the value of the high exposure image signal. N indicates the value of an image signal with proper exposure, which is a target for level matching. In the following, a case where the image signal (N) with proper exposure is set as a level alignment target will be described as an example.

  As shown in FIG. 12A, the luminance value of the low-exposure image signal (L) has a wide luminance range from luminance values Y0 to Y2, but the pixel value is a narrow range from pixel values V0 to V1. The bright image signal is photographed so as not to be overexposed. Conversely, the luminance value of the high-exposure image signal (H) has a narrow luminance range from luminance values Y0 to Y1, but the pixel value has a wide range from pixel values V0 to V2, and is a dark image signal. The photo is taken so that the image does not collapse.

FIG. 12B shows the relationship when level adjustment is performed to synthesize these image signals (the low exposure image signal and the high exposure image signal).
In FIG. 12B, the low-exposure image signal is level-adjusted according to the exposure step and becomes a low-exposure image signal (L2) having pixel values of pixel values V0 to V2. Similarly, the level of the high-exposure image signal is adjusted according to the exposure level difference, and the pixel value becomes a high-exposure image signal (H2) having data values from pixel values V0 to V1. As a result of the level adjustment, both the low exposure image signal (L2) and the high exposure image signal (H2) match the relationship between the luminance value and the pixel value of the image signal (N) in the case of proper exposure. It is done. As a result, in the image composition processing, the range of luminance values Y1 to Y2 is an image based on the low exposure image signal, and the low exposure image signal and the high exposure image signal are mixed for the range of luminance values Y0 to Y1. Image. Note that the motion detection is performed in a data range common to the low-exposure image signal (L2) after level adjustment and the high-exposure image signal (H2) after level adjustment, that is, a range in which the luminance value is a luminance value Y0 to Y1. Done in

FIG. 13 is a diagram illustrating an example of a histogram of pixel values of the low exposure image signal and the high exposure image signal. In FIG. 13, the horizontal axis indicates the pixel value, and the vertical axis indicates the number of pixels having the image value.
FIG. 13A shows a histogram before level matching is performed. In FIG. 13A, a thin line L indicates a histogram of the low-exposure image signal, and pixel values are distributed in a range of pixel values V0 to V1. A thick line H indicates a histogram of the high-exposure image signal, and pixel values are distributed in a range of pixel values V0 to V2. Of the luminance values of the high-exposure image signal, the luminance values equal to or higher than the luminance value Y1 in FIG. 12 are saturated. For this reason, the histogram (H) of the high-exposure image signal is a histogram of only the dark portion of the pixel value. A dotted line N indicates a histogram in the case of proper exposure.

FIG. 13B shows a histogram after level adjustment.
The histogram (L) of the low-exposure image signal becomes a histogram (L2) in which the pixel values are distributed from the pixel values V0 to V2 as a result of level adjustment according to the exposure step. Further, the histogram (H) of the high-exposure image signal is a histogram (H2) in which pixel values are distributed from pixel values V0 to V1 as a result of level adjustment according to the exposure step. In the range of pixel values V0 to V1, the histogram (L2) of the low-exposure image signal after level matching and the histogram (H2) of the high-exposure image signal are in a state of being coincident (overlapped).

FIG. 14 is a diagram for explaining the operation of the image sensor 101 in the present embodiment when high dynamic range imaging is realized with such a configuration. A description of the portions denoted by the same reference numerals is omitted.
A chart 1401 is a reset signal that determines the charge accumulation time of an image signal read according to the chart 206. As in the first embodiment, the image sensor 101 is scanned with an offset of the shift start time Δt determined by giving the average light emission time of the external strobe to (Equation 4). In addition, for imaging with a high dynamic range, the controller 113 controls the imaging element control unit so that the imaging element 101 can be driven with a reset timing for a low exposure image and a reset timing for a high exposure image every other frame. A control signal is transmitted to 102. The image sensor 101 is driven in accordance with a control signal transmitted from the image sensor control unit 102.

When the external strobe emits light at a light emission time t between the light emission start timing 301 and the light emission end timing 302 shown in FIG. 14, in the image signal read according to the chart 206, the line group corresponding to the luminance modulation area 1402 has a luminance by the external strobe. Undergo modulation. For the image signal read according to the chart 207, the line group corresponding to the luminance modulation area 1403 is affected by the luminance modulation by the external strobe.
Image signals 1404 and 1405 are schematic diagrams of image signals read according to charts 206 and 207, respectively. When attention is paid to the luminance modulation areas 1402 and 1403 which are areas subjected to luminance modulation by an external strobe in the two image signals 1404 and 1405 photographed continuously, it is appropriate for either one of the image signals in all lines. Imaging is achieved.

Next, an example of the timing of overall control of the imaging apparatus according to the present embodiment will be described with reference to FIG. A description of the portions denoted by the same reference numerals is omitted.
An image signal 1501 indicates an image signal captured by the image sensor 101. Image signals read by the charts 206, 207, 208, and 1517 are an image O, an image P, an image Q, and an image R, respectively.

An image signal 1502 indicates an image signal output from the frame memory 107. The image signal output from the image sensor 101 is output from the frame memory 107 after being delayed by one frame.
The luminance modulation area information 1503 and 1504 indicates the luminance modulation area information output from the luminance modulation area determination unit 103. The result (luminance modulation region) determined from the image O is represented by (o), and the result (luminance modulation region) determined from the image Q is represented by (q).
The luminance modulation area determination unit 103 is controlled to perform the luminance modulation area determination only during a period in which the low-exposure image signal is read from the image sensor 101 according to the control signal transmitted from the control unit 113. For this reason, as shown in FIG. 15, luminance modulation area information is output every other frame (intermittently). As in the first embodiment, the luminance modulation area is determined by integrating the luminance value of the input image signal in the line direction and comparing the integrated value of the line with the threshold value Th. (See FIG. 6).

The luminance modulation area determination units 1505 and 1506 indicate luminance modulation area information output from the memory 105 provided in the synthesis ratio calculation unit 104. The luminance modulation area information input from the luminance modulation area determination unit 103 is output from the memory 105 after being delayed by one frame.
The synthesis ratio control signals 1507 and 1508 indicate synthesis ratio control signals generated by the synthesis ratio calculation unit 106 based on the luminance modulation area information output from the memory 105. Specifically, the combination ratio control signal γ1 indicates a combination ratio control signal calculated based on the luminance modulation area information (o), and the combination ratio control signal γ2 is calculated based on the luminance modulation area information (q). A synthesis ratio control signal is shown.

Motion detection information 1509 and 1510 indicate motion detection information output from the motion detection unit 1104. As described above, the motion detection information 1509 and 1510 is generated based on the level-adjusted image signals (the low exposure image signal and the high exposure image signal) output from the level adjustment units 1102 and 1103. In FIG. 15, the motion detection information detected based on the image O and the image P is motion detection information α1, and the motion detection information detected based on the image Q and the image R is motion detection information α2.
In addition, the motion detection unit 1104 receives two image signals (the low exposure image signal and the high exposure image signal) to be combined in accordance with a control signal transmitted from the control unit 113 as input to the image combination unit 1101. Control is performed so that motion detection is performed only during a certain period. For this reason, as shown in FIG. 15, motion detection information is output every other frame (intermittently).

  The combination ratios 1511 and 1512 are generated by the combination ratio determination unit 1105 according to the level of the image signal after level adjustment (the low exposure image signal and the high exposure image signal) output from the level adjustment units 1102 and 1103. The synthesis ratio is shown. In FIG. 15, the composition ratio generated according to the levels of the image O and the image P is defined as a composition ratio β1, and the composition ratio generated according to the levels of the image Q and the image R is defined as a composition ratio β2.

The combination ratios 1513 and 1514 indicate the combination ratios output from the combination ratio determination unit 1105. In FIG. 15, the composition ratio when the image O and the image P are combined is a composition ratio δ1, and the composition ratio when the image Q and the image R are combined is a composition ratio δ2.
Composite images 1515 and 1516 indicate composite images obtained by combining the image signals (the low exposure image signal and the high exposure image signal) output from the level adjustment units 1102 and 1103. Based on the combination ratios δ1 and δ2 output from the combination ratio determination unit 1105, the combination ratio adjustment units 109 and 110 perform gain processing on the image signal at the combination ratio. In the present embodiment, as in the first embodiment, the images P and R are used when the value of the synthesis ratio δ1 is 0 (zero). For this reason, the composition ratio adjustment unit 109 performs the input image signal (with the composition ratio obtained by performing the operation of (Equation 5) on the received composition ratios δ1 and δ2 in the same manner as in the first embodiment. Gain processing is performed on the images P and R).

FIG. 16 is a diagram for explaining an example of a method for determining a composition ratio based on image signals (the low exposure image signal and the high exposure image signal) read from the image sensor 101. A description of the portions denoted by the same reference numerals is omitted.
A graph 1601 shows motion detection information α1 generated based on the image O and the image P. The composition ratio determining unit 1105 selects the image O for an area where there is a difference between the image signals (images O and P). Accordingly, the motion detection information α1 is generated so as to select the low-exposure image signal (that is, the image O) in the lines belonging to the luminance modulation regions 1402 and 1403.
A graph 1602 shows a synthesis ratio β1 generated according to the signal levels of the image O and the image P. In FIG. 16, for convenience of explanation, the synthesis ratio β1 is shown in units of lines. Actually, the signal levels of the images O and P are compared in units of one pixel, or the signal levels of the images O and P are compared with each other divided into certain divided areas.

  A graph 1603 shows a synthesis ratio control signal γ1 generated based on the image O. A composite ratio control signal γ1 is generated so that the luminance modulation area by the external strobe does not appear in the composite image 112. More specifically, the composition ratio control signal γ1 is generated so that the image O is not used (that is, the image P is used) for the luminance modulation region 1402. In the first embodiment, the two image signals are referred to when the synthesis ratio control signal is generated. On the other hand, in the present embodiment, only the luminance modulation area 1402 of the low-exposure image signal is referred to among the two image signals (the low-exposure image signal and the high-exposure image signal) to be combined. As described above, in this embodiment, the motion detection unit 1104 determines that a region where the absolute value of the difference value of the image signal to be combined is large is a motion region, and outputs the low-exposure image signal. For this reason, when luminance modulation by an external strobe occurs in the high-exposure image signal, the motion detection unit 1104 can detect a region (luminance modulation region) where the luminance modulation has occurred by the external strobe. Therefore, as described above, in the present embodiment, only the luminance modulation area 1402 of the low exposure image signal is referred to among the low exposure image signal and the high exposure image signal.

  A graph 1604 shows the composition ratio δ1 determined by the composition ratio determination unit 1105 based on the motion detection information α1, the composition ratio β1 corresponding to the level, and the composition ratio control signal γ1, and passed to the composition ratio adjustment units 109 and 110. Show. The composition ratio determining unit 1105 sets the composition ratio δ1 for the region where the composition ratio control signal γ1 is 0 (zero) to 0 (zero) accordingly. Next, the composition ratio determination unit 1105 sets the composition ratio δ1 for the area having the motion detection information α1 to 1 among the other areas (areas where γ1 = 1). Finally, the composition ratio determining unit 1105 determines the composition ratio δ1 of the region where the composition ratio is not determined according to the composition ratio β1 corresponding to the level.

  As described above, in the present embodiment, after selecting the low-exposure image signal for the motion detection region, the high-exposure image signal image O is preferentially selected for the luminance modulation region of the low-exposure image signal, and the rest For the region, the low-exposure image signal and the high-exposure image signal are combined according to the level. Therefore, even in an imaging apparatus that performs imaging with a high dynamic range, it is possible to suppress the influence of an external strobe on the combined image signal, which is the same as the effect described in the first embodiment. There is an effect. In addition, an image with a high dynamic range can be generated for a region where the low exposure image signal and the high exposure image signal are combined.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the first embodiment, the case where the reset start timing shift amount Δt is determined in advance based on the average flash time of the external strobe and fixed at a predetermined time has been described as an example. In contrast, in the present embodiment, a case will be described in which the shift start amount Δt of the reset start is dynamically determined. As described above, the present embodiment and the first embodiment mainly differ in the configuration and processing for determining the amount of shift Δt in the reset start timing. Therefore, in the description of this embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals as those in FIGS.

FIG. 17 is a block diagram illustrating an example of the configuration of the imaging apparatus according to the present embodiment.
The image capturing apparatus according to the present embodiment is different from the image capturing apparatus according to the first embodiment illustrated in FIG. 1 in that the composition ratio calculation unit 106 has a control line 1701 that passes information on the luminance modulation area that appears after the composition to the control unit 113. It has been added.
The imaging device captures an image in a state where the reset start timing shift amount Δt is 0 (zero, Δt = 0). Then, when the external strobe emits light, the imaging device determines a reset start timing shift amount Δt based on a luminance modulation region that appears after combining two image signals that are successively captured. For this purpose, the composition ratio calculation unit 106 determines the luminance modulation region appearing on the composite image 112 based on the luminance modulation region information obtained from the luminance modulation region determination unit 103, and displays the result of the determination using the control line 1701. To the control unit 113.

FIG. 18 is a flowchart for explaining an example of the outline of the operation of the image pickup apparatus of the present embodiment when driving the image pickup element 101.
First, in step S1801, the luminance modulation area determination unit 103 determines a luminance modulation area by light emission of an external strobe. As described in the first embodiment, an area subjected to luminance modulation by an external strobe is output as luminance modulation area information.
In step S1802, the composition ratio calculation unit 106 calculates the number of lines in which the luminance modulation area is generated after image composition.

FIG. 19 is a diagram for explaining an example of a method for determining a luminance modulation area appearing on the composite image 112.
In the imaging apparatus according to the present embodiment, when an external strobe emits light when imaging is performed at a reset start timing shift amount Δt = 0, for example, charge accumulation and readout in the image sensor 101 are performed as shown in FIG. Done.
Image signals read in accordance with charts 2205 and 2206 shown in FIG. 22 correspond to image signals 1901 and 1902 (images X and Y), respectively.

  In FIG. 19, a graph 1903 shows the luminance modulation area information generated by the luminance modulation area determination unit 103 based on the image X, and a graph 1904 shows the luminance generated by the luminance modulation area determination unit 103 based on the image Y. Indicates modulation area information. The composition ratio calculation unit 106 calculates the number of lines in which the luminance modulation area is generated after the image composition from the information on both of the luminance modulation areas via the memory 105. A graph 1905 shows the result of taking the logical product of both of these luminance modulation area information (graphs 1903 and 1904). A region 1906 corresponds to a line that remains as a luminance modulation region after image synthesis. The composite ratio calculation unit 106 passes the calculation result (number of lines) to the control unit 113.

  Returning to the description of FIG. 18, in step S1803, the control unit 113 determines the external strobe of the external strobe from the number of lines calculated by the combination ratio calculation unit 106 and the readout time tl of one line of the image sensor 101 according to (Equation 4). The light emission time t is calculated. The variable X ′ on the left side in (Expression 4) is a line that undergoes luminance modulation by an external strobe generated between two image signals, and corresponds to the number of lines calculated by the combination ratio calculation unit 106. Further, the reset start timing shift amount Δt is 0 (zero, Δt = 0). From the above, the light emission time t of the external strobe can be easily obtained.

  In step S1804, the control unit 113 calculates a reset start timing shift amount Δt based on the light emission time t of the external strobe calculated in step S1803. From (Equation 4), by setting the amount of deviation Δt of the reset start timing to a value equal to or longer than the light emission time t of the external strobe, the line subjected to luminance modulation by the external strobe generated between two consecutive image signals is set to 0. Can be. The control unit 113 passes a control signal to the image sensor control unit 102 so that the image sensor 101 captures an image with the shift start time difference Δt calculated in this way. Then, the image sensor control unit 102 controls the charge accumulation time in the image sensor 101 based on the reset start timing shift amount Δt.

  As described above, in the present embodiment, two image signals captured in succession are analyzed, and the number of lines corresponding to the luminance modulation area existing in common in the two image signals is derived. Then, the light emission time of the external strobe is calculated based on the derived number of lines, the readout time of one line of the image sensor 101, and the current shift amount of the reset start timing. Therefore, in addition to the effects described in the first embodiment, the following effects can be obtained. That is, before recording an image with the imaging device, the imaging device 101 can be driven so that the combined image signal is not affected by the external strobe, and the reset is performed in a situation where the external strobe is not emitting light. Images can be taken without shifting the start timing.

<Modification>
In the present embodiment, the light emission time t of the external strobe is calculated from the image signal captured when the reset start timing deviation amount Δt is 0 (zero, Δt = 0). However, even when the reset start timing deviation amount Δt is not 0 (zero) and the light emission time t of the external strobe is larger than the reset start timing deviation amount Δt, the same applies to (Equation 4). The light emission time t of the external strobe can be calculated. In this case, the current value of the reset start timing deviation amount Δt is given to (Equation 4). In addition, as shown in the first embodiment, the value of the shift start time Δt that is determined in advance based on the average light emission time of the external strobe can be adjusted.
In addition, the configuration of this embodiment can be applied to the second embodiment.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In this embodiment, a configuration is added to the first embodiment to compensate for exposure of an image signal obtained by synthesizing two image signals captured at a reset start timing shift amount Δt. Therefore, in the description of this embodiment, the same parts as those in the first to third embodiments are denoted by the same reference numerals as those in FIGS.

FIG. 20 is a block diagram illustrating an example of the configuration of the imaging apparatus according to the present embodiment.
Compared with the imaging apparatus of the first embodiment shown in FIG. 1, the imaging apparatus of this embodiment has two frame memories 2004 and 2005, two memories 2002 and 2003, and exposure correction. The difference is that a configuration for generating an image signal is added. Specifically, an exposure correction signal generation unit 2007 and an exposure correction unit 2008 are provided in the imaging apparatus as a configuration for executing a correction signal generation process for generating an image signal for exposure correction. The image pickup apparatus according to the present embodiment uses the image signal obtained by combining two image signals successively captured with the reset start timing shift amount Δt based on the image signal of another frame. To compensate.

20, the synthesis ratio calculation unit 2001 has two memories (a first memory 2002 and a second memory 2003), unlike the synthesis ratio calculation unit 104 shown in FIG. The composition ratio calculation unit 106 generates a composition ratio control signal based on the luminance modulation area information output from the first memory 2002 and the second memory 2003.
A first frame memory 2004 and a second frame memory 2005 are frame memories that hold an image signal output from the image sensor 101 for one frame. The second frame memory 2005 holds the image signal output from the first frame memory 2004 for one frame.

The image composition unit 2006 is obtained by adding an exposure correction signal generation unit 2007 and an exposure correction unit 2008 to the image composition unit 108 shown in FIG. The combination ratio adjustment units 109 and 110 correspond to the combination ratio control signals output from the combination ratio calculation unit 106 for the image signals output from the first frame memory 2004 and the second frame memory 2005, respectively. Gain processing is performed with the composition ratio.
The exposure correction signal generation unit 2007 generates an exposure correction image signal from the image signal output from the image sensor 101 according to the control signal sent from the control unit 113.
The exposure correction unit 2008 combines (adds) the combined image signal output from the combining unit 111 and the exposure correction image signal output from the exposure correction signal generation unit 2007, and outputs a combined image 2009. .

With reference to FIG. 21, an example of the timing of overall control of the imaging apparatus of the present embodiment will be described. A description of the portions denoted by the same reference numerals will be omitted.
An image signal 2101 indicates an image signal read from the image sensor 101.
Image signals 2102 and 2103 indicate image signals output from the first frame memory 2004 and the second frame memory 2005, respectively. The image signals are image O, image P, image Q, and image R in the order of output.
The first frame memory 2004 holds the image signal output from the image sensor 101 for one frame. The second frame memory 2005 holds the image signal output from the frame memory 2004 for one frame.

The luminance modulation area information 2104 indicates the luminance modulation area information output from the luminance modulation area determination unit 103.
The luminance modulation area information 2105 and 2106 indicate the luminance modulation area information output from the first memory 2002 and the second memory 2003, respectively. In FIG. 21, the luminance modulation area information (o), (p), (q), and (r) are written in order along the number of the image signal.
A composite ratio signal 2107 indicates a composite ratio control signal generated by the composite ratio calculator 106. The combination ratio control signal γ1 is generated based on the luminance modulation area information (p) output from the first memory 2002 and the luminance modulation area information (o) output from the second memory 2003. Similarly, the composition ratio control signal γ2 is generated based on the luminance modulation area information (q) output from the first memory 2002 and the luminance modulation area information (r) output from the second memory 2003. . The method for generating the synthesis ratio control signals γ1 and γ12 is the same as the method described in the first embodiment.

An image signal 2108 indicates the image signal synthesized by the synthesis unit 111. The image O and the image P are subjected to gain processing by the composition ratio adjustment units 109 and 110 in accordance with the composition ratio control signal γ1. This is the same as described in the first embodiment.
An exposure correction image signal 2109 indicates an exposure correction image signal generated by the exposure correction signal generation unit 2007. An exposure correction image signal based on the image Q is generated based on a control signal output from the control unit 113 at a timing when an image signal obtained by combining the image O and the image P is output. The control signal output from the control unit 113 is a signal for instructing a gain amount calculated so as to compensate for an underexposure corresponding to a reset start timing shift amount Δt in the image sensor 101. Based on this gain amount, the exposure correction signal generation unit 2007 generates an image obtained by performing gain processing on the image Q as an exposure correction image signal. The gain amount Ga is expressed by the following (Equation 7), where T is the charge accumulation time when the reset start timing shift amount Δt in the image sensor 101 is 0 (zero, Δt = 0).
Ga = Δt / (T−Δt) (Expression 7)

  A composite image 2110 indicates an image obtained by combining the image signal combined by the combining unit 111 and the exposure correction image signal generated by the exposure correction signal generation unit 2007 by the exposure correction unit 2008. An image correction signal Q ′ generated from the image Q as the next frame is added to the composite image of the image O and the image P. Similarly, the image correction signal R ′ generated from the image R which is the next frame is added to the composite image of the image P and the image Q.

As described above, in the present embodiment, an exposure correction image signal is generated by performing gain processing on the next image signal with a gain amount corresponding to the underexposure caused by the reset start timing shift amount Δt. Then, the image signal obtained by synthesizing the two image signals successively captured as described in the first embodiment is compensated with the exposure correction image signal. As a result, it is possible to obtain a composite image 2009 having a better S / N ratio than exposure compensation by increasing the gain. In addition, a sharper exposure compensation is possible than changing the aperture value of an optical system (not shown).
The configuration of the present embodiment can also be applied to the second embodiment or the third embodiment.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention is also realized by executing the following processing. That is, first, software (computer program) for realizing the functions of the above embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the computer program.

  DESCRIPTION OF SYMBOLS 101 Image sensor, 103 Brightness modulation area | region determination part, 104 Composition ratio calculation part, 108 Image composition part

Claims (23)

An imaging device that generates an image signal by combining a plurality of image signals continuously captured by an imaging element having a plurality of pixels arranged in a matrix,
Setting for setting a shift time, which is a time period from the start of charge accumulation in the image sensor for obtaining the next image signal after reading of the charge in the image sensor for obtaining the image signal Means,
Image sensor control means for controlling the operation of the image sensor based on the shift time set by the setting means;
Based on the luminance values of the plurality of image signals imaged by the image sensor whose operation is controlled by the image sensor control means, a luminance modulation area, which is an area in which the luminance is modulated, is defined as the plurality of image signals. An area determination means for determining each;
Based on the luminance modulation area determined by the area determination unit, a combination ratio calculation unit that calculates a combination ratio of the plurality of image signals;
A synthesizing unit that synthesizes the plurality of image signals picked up by the image pickup device whose operation is controlled by the image pickup element control unit at a combination ratio calculated by the combination ratio calculating unit;
An imaging device comprising:   The synthesis ratio calculation unit determines that the region determination unit determines that there is a region serving as the luminance modulation region in at least one image signal of the plurality of image signals, and the remaining images of the plurality of image signals When it is determined that a region corresponding to the region of the signal is not the luminance modulation region, the composition ratio is calculated so that the remaining image signals of the plurality of image signals are used for the region. The imaging apparatus according to claim 1.   The combining ratio calculating unit combines the plurality of image signals for the region when the region determining unit determines that the region corresponding to the plurality of image signals is not the luminance modulation region. The imaging apparatus according to claim 1, wherein the composite ratio is calculated so as to be used.   The imaging according to any one of claims 1 to 3, wherein the plurality of image signals are a high exposure image signal with relatively high exposure and a low exposure image signal with relatively low exposure. apparatus. Level adjusting means for matching the signal levels of the high exposure image signal and the low exposure image signal;
Motion detection means for detecting movement of a subject based on a difference in signal level between the high exposure image signal and the low exposure image signal, the signal level of which is adjusted by the level adjustment means;
The composition ratio calculated by the composition ratio calculation means, the motion detection area that is the area where the motion is detected, and the high-exposure image signal and the low-exposure image signal whose signal levels are matched by the level adjustment means The imaging apparatus according to claim 4, further comprising: a combination ratio determining unit that determines the combination ratio based on The composite ratio calculating means includes
The high exposure image signal is used for the luminance modulation area of the low exposure image signal, and the low exposure image signal is used for an area different from the luminance modulation area of the low exposure image signal. Calculate the composition ratio,
The synthesis ratio determining means includes
For the motion detection region, use one of the predetermined image signals of the high exposure image signal and the low exposure image signal, for the region different from the motion detection region, use the other image signal,
For the luminance modulation area of the low exposure image signal, regardless of the image signal that is supposed to be used based on the motion detection area, the high exposure image signal is used,
For other regions, the high exposure image signal and the low exposure image signal are combined and used in accordance with the signal levels of the high exposure image signal and the low exposure image signal whose signal levels are adjusted by the level adjusting means. The imaging apparatus according to claim 5, wherein the combination ratio is determined.   Shift time deriving means for deriving the shift time based on the number of lines of the image sensor present in the luminance modulation area common to both of the plurality of image signals and the readout time of one line of the image sensor; The imaging apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 1, wherein the shift time is a predetermined time based on an average light emission time of an external light emitting unit. The image signal picked up by the image pickup device whose operation is controlled by the image pickup device control means, and the image pickup signal corresponding to the shift time with respect to the image signal different from the plurality of image signals. Correction signal generation means for generating a signal obtained by applying a gain based on the charge accumulation time in the element as an exposure correction image signal;
The exposure correction means for combining the image signal for exposure correction generated by the correction signal generation means and the plurality of image signals combined by the combining means, further comprising: The imaging apparatus of Claim 1. An optical system including at least an aperture;
Optical system control means for controlling the operation of the optical system,
The optical system control means controls the aperture of the optical system so that the aperture value is based on the shift time and the charge accumulation time in the image sensor corresponding to the shift time. Item 9. The imaging device according to any one of Items 1 to 8. A determination means for determining whether to control the operation of the image pickup device based on the shift time according to an exposure control parameter of the image pickup apparatus;
When the determination unit determines that the operation of the image sensor based on the shift time is not to be controlled, the image sensor control unit controls the operation of the image sensor by setting the shift time to 0 (zero). The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. A method of controlling an imaging apparatus that generates an image signal by combining a plurality of image signals continuously captured by an imaging device having a plurality of pixels arranged in a matrix,
Setting for setting a shift time, which is a time period from the start of charge accumulation in the image sensor for obtaining the next image signal after reading of the charge in the image sensor for obtaining the image signal Process,
An image sensor control step for controlling the operation of the image sensor based on the shift time set by the setting step;
Based on the luminance values of the plurality of image signals imaged by the image sensor whose operation is controlled by the image sensor control step, a luminance modulation area, which is an area where the luminance is modulated, is determined as the plurality of image signals. An area determination step for determining each;
A synthesis ratio calculation step of calculating a synthesis ratio of the plurality of image signals based on the luminance modulation area determined by the area determination step;
A synthesis step of synthesizing the plurality of image signals picked up by the image pickup device, the operation of which is controlled by the image pickup device control step, at the combination ratio calculated by the combination ratio calculation step;
A method for controlling an imaging apparatus, comprising:   In the synthesis ratio calculation step, it is determined by the region determination step that at least one image signal of the plurality of image signals includes a region to be the luminance modulation region, and the remaining images of the plurality of image signals When it is determined that a region corresponding to the region of the signal is not the luminance modulation region, the composition ratio is calculated so that the remaining image signals of the plurality of image signals are used for the region. The method of controlling an imaging apparatus according to claim 12.   In the combination ratio calculation step, when it is determined by the region determination step that regions corresponding to each other of the plurality of image signals are not the luminance modulation region, the plurality of image signals are combined for the region. The method of controlling an imaging apparatus according to claim 12, wherein the composite ratio is calculated so as to be used.   15. The imaging according to any one of claims 12 to 14, wherein the plurality of image signals are a high exposure image signal with relatively high exposure and a low exposure image signal with relatively low exposure. Control method of the device. A level adjustment step of matching the signal levels of the high exposure image signal and the low exposure image signal;
A motion detection step of detecting a motion of a subject based on a difference in signal level between the high exposure image signal and the low exposure image signal, the signal level of which is adjusted by the level adjustment step;
The composition ratio calculated by the composition ratio calculation step, the motion detection region that is the region where the motion is detected, the high-exposure image signal and the low-exposure image signal whose signal levels are matched by the level adjustment step The control method for an imaging apparatus according to claim 15, further comprising: a composite ratio determining step for determining the composite ratio based on The synthesis ratio calculation step includes
The high exposure image signal is used for the luminance modulation area of the low exposure image signal, and the low exposure image signal is used for an area different from the luminance modulation area of the low exposure image signal. Calculate the composition ratio,
The synthesis ratio determining step includes
For the motion detection region, use one of the predetermined image signals of the high exposure image signal and the low exposure image signal, for the region different from the motion detection region, use the other image signal,
For the luminance modulation area of the low exposure image signal, regardless of the image signal that is supposed to be used based on the motion detection area, the high exposure image signal is used,
For other regions, the high exposure image signal and the low exposure image signal are combined and used in accordance with the signal levels of the high exposure image signal and the low exposure image signal whose signal levels are adjusted by the level adjustment step. The method according to claim 16, wherein the composition ratio is determined.   A shift time deriving step of deriving the shift time based on the number of lines of the image sensor present in the luminance modulation area common to both of the plurality of image signals and the readout time of one line of the image sensor. The method for controlling an imaging apparatus according to claim 12, further comprising:   The method of controlling an imaging apparatus according to any one of claims 12 to 17, wherein the shift time is a predetermined time based on an average light emission time of an external light emitting unit. The image signal picked up by the image pickup element whose operation is controlled by the image pickup element control step, and the image pickup corresponding to the shift time with respect to the image signal different from the plurality of image signals A correction signal generation step of generating a signal obtained by applying a gain based on the charge accumulation time in the element as an exposure correction image signal;
The exposure correction step of combining the exposure correction image signal generated by the correction signal generation step and the plurality of image signals combined by the combining step, further comprising: A control method for an imaging apparatus according to claim 1. An optical system control step for controlling the operation of the optical system including at least the stop, and
The optical system control step controls the aperture of the optical system so that the aperture value is based on the shift time and the charge accumulation time in the image sensor corresponding to the shift time. Item 20. The control method for an imaging apparatus according to any one of Items 12 to 19. A determination step of determining whether to control the operation of the image sensor based on the shift time according to an exposure control parameter of the imaging device;
If it is determined by the determination step that the operation of the image sensor based on the shift time is not to be controlled, the image sensor control step sets the shift time to 0 (zero) and controls the operation of the image sensor. The method of controlling an imaging apparatus according to any one of claims 12 to 21.   23. A computer program for causing a computer to execute each step of the control method for an imaging apparatus according to claim 12.

Images