JP7034792B2 - Imaging device and its control method and program - Google Patents

Description

The present invention relates to an image pickup device including a rolling shutter type image pickup device, and a control method and program thereof.

As an image pickup element of an image pickup device such as a digital camera or a digital camcorder, a rolling shutter type such as a CMOS sensor is widely used. In the rolling shutter type image pickup device, an image pickup is performed by sequentially reading out charges (image signals) from the pixel lines on the image pickup region of the image pickup device. Therefore, when a flash (for example, light having a short emission time and strong light such as strobe light) is irradiated from the outside of the image pickup apparatus, the flash may not be irradiated to the entire line of the image pickup region. In that case, a phenomenon occurs in which only the line on the image pickup region irradiated with the flash is imaged with high brightness, and the high brightness line generated in the captured image in this way is generally called a flash band. Flash bands are likely to occur in an environment where multiple flashes are continuously irradiated in a short period of time, such as when shooting a movie at a press conference or an event such as a wedding ceremony.

Images containing frames with flash bands are often perceived as unnatural when viewed, so captured images with flash bands are automatically deleted or corrected by image processing when the video is generated. Often. For example, Patent Documents 1 and 2 describe a method of correcting a flash band at the time of shooting by image processing in an image pickup apparatus.

Patent Document 1 proposes a method of reading an image signal from an image sensor at a high frame rate and synthesizing a plurality of read image images to generate a frame corrected with a flash band. In this method, the probability of flash band generation is reduced by reading the captured image from the image sensor at a high frame rate. Then, when a flash band is generated, adjacent captured images are combined to generate a frame equivalent to the state in which all the lines in the imaging region are irradiated with the flash.

Patent Document 2 reads an image signal at a high frame rate, determines whether or not a flash band is generated for each captured image read, and a captured image in which a flash band is generated does not generate a flash band. We are proposing a method to replace it with an captured image. In this method, before the replacement process for the captured image in which the flash band is not generated, the exposure is corrected by the gain or the like so as to be equivalent to the exposure when the flash is irradiated.

Japanese Unexamined Patent Publication No. 2013-165439 JP-A-2015-177303

However, in the technique described in Patent Document 1, in order to correct the flash band, it is necessary to match the boundaries of the flash bands on the imaging region of the adjacent captured images. That is, it is necessary to match the readout period of the previous captured image with the accumulation start period for the next captured image, and to match the readout timing and the accumulation start timing of each line. Due to this limitation, there is a problem that the shutter speed cannot be set freely.

Further, in the technique described in Patent Document 2, the flash band is corrected on the premise that the captured image in which the flash band is not generated exists. However, for example, in a shooting environment in which flashes such as strobe light are continuously irradiated, there is a possibility that captured images in which a flash band is generated are continuous. In this case, since the continuous captured images are replaced with the same captured images in which the flash band is not generated, the same captured images become still images in the continuous portion.

It is an object of the present invention to provide an image pickup apparatus capable of freely setting a shutter speed and generating a moving image in which a flash band is corrected without making the same captured image continuous.

In the image pickup apparatus according to the present invention, the storage period of the entire line of the image pickup device capable of performing the first image pickup and the second image pickup having different storage timings in parallel and the image pickup device in the second image pickup is described above. A control means for executing the first image pickup and the second image pickup by overlapping the accumulation start period and the read-out period of all the lines of the image pickup element in the first image pickup, and the first image pickup and the first image pickup. A storage means for storing the captured image obtained by the second image pickup, a detection means for detecting the occurrence of a flash band in the captured image stored in the storage means, and a flash band based on the detection result by the detection means. It is characterized by comprising a selection means for selecting a captured image that has not been generated, and a moving image generation means for generating a moving image using the captured image selected by the selection means.

According to the present invention, the shutter speed can be freely set, and a moving image with a corrected flash band can be generated without making the same captured image continuous.

It is a block diagram which shows the schematic structure of the image pickup apparatus which concerns on 1st Embodiment. It is a schematic diagram explaining the 1st example of the 1st image pickup and the 2nd image pickup. It is a flowchart of the process at the time of moving image shooting by the image pickup apparatus of FIG. It is a schematic diagram explaining the 2nd example of the 1st image pickup and the 2nd image pickup. It is a schematic diagram explaining the 3rd example of the 1st image pickup and the 2nd image pickup. It is a block diagram which shows the schematic structure of the image pickup apparatus which concerns on 2nd Embodiment of this invention. It is a schematic diagram explaining the 4th example of the 1st image pickup and the 2nd image pickup. It is a flowchart of the process at the time of moving image shooting by the image pickup apparatus of FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a schematic configuration of an image pickup apparatus 100 according to a first embodiment of the present invention. Specific examples of the image pickup apparatus 100 include, but are not limited to, a digital camera and a digital camcorder. The image pickup apparatus 100 includes a lens 1, a dimming section 2, an image pickup processing section 3, an image processing section 9, an MPU 10, a storage section 11, a display section 12, and an operation section 13. The image pickup processing unit 3 includes an image pickup unit 4, a first memory 5, a second memory 6, a flash detection unit 7, and an image selection unit 8.

The reflected light from the subject, which is the incident light, passes through the lens 1 and the dimming unit 2 and forms an image on the image pickup element (CMOS image pickup device) included in the image pickup unit 4 of the image pickup processing unit 3. For convenience, although one lens 1 is shown in FIG. 1, the lens 1 may be composed of a plurality of lenses. The dimming unit 2 is composed of a member such as a diaphragm blade and an ND filter that can adjust the amount of incident light. The details of the configuration of the image processing unit 3 will be described later, but in general, the image pickup processing unit 3 analyzes the image captured image read from the image pickup unit 4 and selects and selects the image captured image to be output to the image processing unit 9. The captured image is output to the image processing unit 9. The image processing unit 9 adjusts the color, brightness, and the like of the captured image acquired from the image pickup processing unit 3 (various signal processing such as white balance processing and gamma processing), and generates a moving image file in a predetermined format by cordic. ..

The moving image file generated by the image processing unit 9 is stored in the storage unit 11 and used for display by the display unit 12. The storage unit 11 is, for example, a semiconductor storage device such as a memory card, a hard disk drive, or the like, but is not limited thereto, and may be any one capable of storing a moving image file composed of a digital signal. The display unit 12 is, but is not limited to, a liquid crystal display, an organic EL display, or the like. The operation unit 13 provides an interface for instructing various settings of the image pickup apparatus 100 and execution of various operations. The operation unit 13 includes selection means (not shown) for allowing the user to select whether or not to execute moving image shooting for performing flash band correction processing described later with reference to FIG.

The MPU 10 is a microcomputer that controls the entire image pickup apparatus 100, and includes a CPU, a ROM, a RAM, and the like. Various programs for controlling the operation of the image pickup apparatus 100 are stored in the ROM, and the CPU expands a predetermined program stored in the ROM into the RAM to expand each part of the image pickup apparatus 100. Control the operation comprehensively.

Subsequently, the configuration of the image pickup processing unit 3 will be described. The image pickup unit 4 included in the image pickup processing unit 3 includes a rolling shutter type CMOS image pickup element, a driver for driving the CMOS image pickup element, a timing generation circuit, a CDS / AGC circuit, an A / D converter, and the like. The operation of each part is controlled by the MPU 10. The CMOS image sensor photoelectrically converts the optical image of the subject imaged on the image pickup surface. The CDS / AGC circuit reads out the electric charge (image signal) accumulated in each pixel of the CMOS image sensor by correlated double sampling (CDS) and amplifies it to a predetermined gain by automatic gain adjustment (AGC). The analog electric signal output from the CDS / AGC circuit is converted into an captured image composed of a digital signal by an A / D converter, and the captured image is stored in the first memory 5 and the second memory 6.

In the present embodiment, as the image pickup unit 4, a unit capable of performing a first image pickup and a second image pickup having different accumulation timings in parallel is used. Specifically, a pixel having a structure in which a first A / D conversion unit and a second A / D conversion unit are provided in each of a plurality of pixels constituting the CMOS image pickup device can be used. Since the configuration of such an imaging unit 4 is known, detailed description thereof will be omitted, but for example, it is disclosed in Japanese Patent Application Laid-Open No. 2013-207433. From the image pickup unit 4, the captured image output from the first A / D conversion unit and the captured image output from the second A / D conversion unit can be obtained at the same time.

In the following description, the first imaging refers to the imaging for obtaining the captured image through the first A / D conversion unit, and the second imaging refers to the imaging for obtaining the captured image through the second A / D conversion unit. Shall point to. As the image pickup unit 4, a unit having a configuration in which two A / D conversion units are alternately connected to a plurality of pixels constituting the CMOS image pickup device can also be used. Since the configuration of such an imaging unit 4 is known, detailed description thereof will be omitted, but for example, it is disclosed in Japanese Patent Application Laid-Open No. 2015-26938.

In the present embodiment, the imaging conditions are set so that the accumulation timing of the electric charge (image signal) in the CMOS image sensor differs between the first image pickup and the second image pickup at the time of moving image shooting. Then, the captured image obtained by the first imaging is stored in the first memory 5, and the captured image obtained by the second imaging is stored in the second memory 6. A specific example of "first imaging and second imaging with different accumulation timings" will be described later with reference to FIGS. 2, 4, 5, and 7.

The flash detection unit 7 detects the presence or absence of a flash band for each of the plurality of captured images stored in each of the first memory 5 and the second memory 6. For example, when the flash detection unit 7 scans the horizontal lines of the captured image one line at a time from top to bottom and detects that the luminance difference between the two lines adjacent to the top and bottom is larger than a predetermined value, the captured image is displayed. It is determined that a flash band has occurred. The flash detection unit 7 notifies the image selection unit 8 of the flash band detection result.

The image selection unit 8 selects and reads an image captured without a flash band from the first memory 5 and the second memory 6 based on the flash band detection result acquired from the flash detection unit 7. Then, the image selection unit 8 transmits the captured images read from the first memory 5 and the second memory 6 to the image processing unit 9 in synchronization with the control frame rate of the image pickup device 100.

When the flash emission amount is small, the luminance difference between the lines adjacent to the top and bottom becomes small, so that the captured image in which the flash band is generated is the captured image in which the flash band is not generated. There is a possibility of misunderstanding. In order to deal with this problem, the image selection unit 8 selects an captured image from the captured images stored in the first memory 5 and the second memory 6 which is most likely not to generate a flash band. By selecting it, control is performed to reduce the influence of the erroneous detection of the flash detection unit 7. Here, the captured image that is presumed to have no flash band is adjacent to the upper and lower sides of the captured image determined that the flash band is not generated, which is required for each captured image. It is a captured image in which the maximum value of the brightness difference between the two lines is relatively small. The flash detection unit 7 obtains the brightness of two lines adjacent to each other when inspecting whether or not a flash band is generated in the captured image. Therefore, if the flash band detection result notified from the flash detection unit 7 to the image selection unit 8 includes information on the maximum value of the luminance difference between the two vertically adjacent lines in each captured image, the image selection unit 8 will be used. The selection of the captured image can be performed simply.

In the moving image shooting by the image pickup device 100, the captured image selected from the captured images obtained by performing the first imaging and the second imaging at different charge (image signal) accumulation timings in the CMOS image sensor according to the user setting. Can be used as a frame to generate one moving image. FIG. 2 is a diagram illustrating a first example of a first image pickup and a second image pickup imaged at different accumulation timings. FIG. 2A is a diagram illustrating control of a CMOS image sensor in each of the first image pickup and the second image pickup. In the first example, the accumulation period of all the lines in the imaging region in the second imaging overlaps with the accumulation start period and the readout period of all the lines in the imaging region in the first imaging. In other words, during the period in which the accumulation is sequentially started for each line of the imaging region and the period in which the readout is performed in order to acquire one captured image in the first imaging, the entire imaging region is acquired in the second imaging. Make sure that the electric charge (image signal) is accumulated on the line. The image captured by the first imaging is stored in the first memory 5, and the image captured by the second imaging is stored in the second memory 6.

FIG. 2B is a diagram illustrating that when a flash band is generated in the captured image in the first imaging, the flash band is not generated in the captured image in the second imaging. It is assumed that the flash irradiation is performed during the readout period in the first image pickup, and a flash band is generated in the image captured in the first image. Although not shown, a flash band is also generated in the next captured image of the captured image of the storage 1. At this time, since the flash irradiation timing is within the accumulation period of all the lines of the accumulation 2b next to the accumulation 2a in the second imaging, the flash is irradiated to all the lines of the imaging region in the accumulation 2b. Therefore, the captured image of the storage 2a has the brightness not irradiated with the flash, the captured image of the storage 2b has the brightness irradiated with the flash, and no flash band is generated in any of the captured images. In this way, by making sure that the accumulation start period and the readout period of all the lines in the imaging region in the first imaging are covered by the accumulation period of all the lines in the imaging region in the second imaging, one time. It is possible to obtain an captured image in which no flash band is always generated for a flash of light.

In the present embodiment, the first imaging and the second imaging are performed in parallel, and one moving image file is generated based on the captured image obtained by the first imaging. Therefore, when the flash band is not generated in both the first image pickup and the captured image obtained by the second image pickup at a predetermined timing, the image selection unit 8 generates the flash band by the first image pickup. Select an uncaptured image. Then, when it is detected that a flash band is generated in the captured image obtained by the first imaging at a predetermined timing, the image selection unit 8 captures the image in which the flash band is not generated by the second imaging. Select an image. In the example of FIG. 4B, when it is detected that a flash band is generated in the captured image of the storage 1, the image selection unit 8 is used to generate a moving image instead of the captured image of the storage 1. Accumulation in the imaging of 2 Select the images captured by 2a and 2b. In this way, when a flash band is generated in the image captured by the first imaging, the captured image is replaced with the captured image in which the flash band is not generated by the second imaging.

Next, the flow of the process of performing the first imaging and the second imaging set at different accumulation timings in parallel and generating one moving image file based on the images captured by the first imaging will be described. FIG. 3 is a flowchart illustrating a process executed by the MPU 10 and the image pickup processing unit 3 when a moving image is taken by the image pickup apparatus 100. Each process (step) indicated by the S number in FIG. 3 is realized by the MPU 10 executing a predetermined program to comprehensively control the operation of each unit constituting the image pickup processing unit 3. It is assumed that the conditions for the first imaging are predetermined.

In S101, the MPU 10 stores the image captured by the first image capture in the first memory 5. In S102, in the first imaging, whether or not the charge (image signal) accumulation period (hereinafter simply referred to as “accumulation period”) for acquiring the captured image is twice or more the accumulation start period of all lines. Is determined. The accumulation period in the first imaging refers to the period from the start of accumulation to the start of reading for one line (in the case of FIG. 2, the period corresponding to the length of the upper side of the accumulation 1). Further, in the first imaging, it is assumed that the accumulation start period and the read period of all the lines have the same time length. Therefore, S102 may be performed by determining whether or not the accumulation period is twice or more the read period of all lines in the first imaging. When the MPU 10 determines that the accumulation period in the first imaging is twice or more the accumulation start period of all lines (YES in S102), the process proceeds to S103. On the other hand, when the MPU 10 determines that the accumulation period in the first imaging is less than twice the accumulation start period of all lines (NO in S102), the process proceeds to S108.

In S103, the MPU 10 determines whether or not the accumulation period in the first image pickup is equal to or less than the accumulation period of the control frame rate of the image pickup apparatus 100. Here, the case where the accumulation period of the first imaging is longer than the accumulation period of the control frame rate is specifically the case where long-second exposure is performed. For example, when the control frame rate is 60 fps, the storage period is 16.7 msec, and when the storage period is longer than this, it is determined that long-second exposure is being performed. When the MPU 10 determines that the accumulation period in the first imaging is equal to or less than the accumulation period of the control frame rate (YES in S103), the MPU 10 advances the process to S104. On the other hand, when the MPU 10 determines that the accumulation period in the first imaging is longer than the accumulation period of the control frame rate (NO in S103), the process proceeds to S106.

When the determination of S103 is YES, the accumulation start period and the readout period of all the lines in the first imaging are set to the same frame rate as the frame rate in the first imaging of the second imaging. The conditions that can be covered by the accumulation period of all lines are in place. Therefore, in S104, as described with reference to FIG. 2, the MPU 10 has a second imaging in which the accumulation period of all lines is set so as to cover the accumulation start period and the readout period of all the lines of the first imaging. To execute. As a result, even if a flash band is generated in the image captured by the first imaging due to the flash irradiation, the flash is applied to the entire line of the imaging region to obtain an captured image in which the flash band is not generated by the second imaging. Can be done. In S105, the MPU 10 stores the image captured by the second image in the second memory 6 (normal storage). The MPU 10 advances the process to S110 after S105.

In S104, it is sufficient that each of the accumulation start period and the readout period of all the lines in the first imaging is covered by the accumulation period of all the lines in the second imaging. Therefore, it is possible to set the frame rate of the second image pickup to be faster than the frame rate of the first image pickup. However, in that case, it is necessary to adjust the output from the CMOS image sensor so that the exposure of the image captured by the second imaging matches the exposure of the image captured by the first imaging.

In S106 after the determination of S103 is NO, the MPU 10 covers the accumulation start period and the readout period of all the lines in the first imaging (long exposure) with the accumulation period of all the lines in the second imaging. The accumulation period of the second imaging is set so that the second imaging can be performed. Here, the second image pickup set in S106 will be described with reference to FIG.

FIG. 4 is a diagram illustrating a second example of a first image pickup and a second image pickup imaged at different accumulation timings. FIG. 4A is a diagram illustrating control of the CMOS image sensor in each of the first image pickup and the second image pickup. In the first imaging, long-second exposure is performed, and it is assumed that the captured image indicated by the storage 11 is output from the imaging unit 4. In this case, in the second imaging, the accumulation 12a (first accumulation) is performed at a timing that covers the accumulation start period of all the lines of the first imaging, and the readout period of all the lines of the first imaging is covered. Accumulation 12c (second accumulation) is performed at the timing. In the second imaging, the accumulation 12b (third accumulation) is further performed between the accumulation 12a and the accumulation 12c. The reason for performing the accumulation 12b is to match the accumulation period (exposure) in the image captured by the second imaging with the accumulation period (exposure) in the image captured by the first imaging. In other words, the second imaging condition is that the accumulation period of each line when the accumulation 12a and the accumulation 12b are added and the accumulation period of each line when the accumulation 12b and the accumulation 12c are added are both at the accumulation 11. It is set to be the same as the accumulation period of each line.

In S107, the MPU 10 stores the captured image obtained in the second imaging in the second memory 6 (additional storage). FIG. 4B is a diagram illustrating a captured image stored in the second memory 6 in S107. In the second imaging, an image captured by the storage 12a and an image captured by the storage 12b are added and stored in the second memory 6, and the image captured by the storage 12b and the image captured by the storage 12c are added. The captured image is created and stored in the second memory 6. As a result, the accumulation period of the captured image by the second imaging can be matched with the accumulation period of the captured image by the first imaging. The MPU 10 advances the process to S110 after S107.

When the accumulation period in the first imaging is less than twice the accumulation start period of all lines, the accumulation start period and the readout period of all lines in the first imaging are set to the accumulation 2a and 2b shown in FIG. 2, respectively. Cannot be covered with separate accumulations such as. Therefore, in S108 after the determination of S102 becomes NO, the period from the start of accumulation of all lines in the first imaging to the end of reading is covered by the accumulation period of all lines in the second imaging. , Perform a second imaging.

Here, the second image pickup set in S108 will be described with reference to FIG. FIG. 5 is a diagram illustrating a third example of a first image pickup and a second image pickup imaged at different accumulation timings. FIG. 5 shows an example in which the accumulation period of all lines in the second imaging is the same as the period from the start of accumulation to the end of reading of all the lines in the first imaging. However, the present invention is not limited to this, and the accumulation period of all lines in the second imaging may cover the period from the start of accumulation to the end of reading of all the lines in the first imaging.

In S109, the MPU 10 stores the image captured by the second image in the second memory 6. At this time, since the image captured by the second imaging has a longer accumulation period than the image captured by the first imaging, the image is generally brighter than the image captured by the first imaging. Therefore, in the second imaging, the MPU 10 performs negative gain processing (output adjustment) when reading the charge (image signal) from the CMOS image sensor, and adjusts the brightness (exposure) of the image captured by the second imaging. Matches the brightness of the image captured by the first image pickup (gain adjustment storage). The MPU 10 advances the process to S110 after S109.

In S110, the flash detection unit 7 detects whether or not a flash band is generated in the captured images stored in the first memory 5 and the second memory 6, respectively, and sends the flash band detection result to the image selection unit 8. Notice. As described above, the moving image is generated based on the captured image obtained by the first imaging. Therefore, only when it is detected that a flash band is generated in the captured image stored in the first memory 5, in order to specify the captured image by the second imaging used for generating the moving image instead of them. The captured image stored in the second memory 6 may be inspected. In S111, the image selection unit 8 selects the captured image to be transmitted to the image processing unit 9 based on the information acquired in S110.

In S112, the image selection unit 8 transmits the captured image selected in S111 to the image processing unit 9 in accordance with the control frame rate of the image pickup device 100. At that time, when the image captured by the second imaging is transmitted to the image processing unit 9, the captured image in which the flash is irradiated to all the lines of the imaging region is included. Further, the captured image transmitted to the image processing unit 9 and the captured image that has been determined not to be used for moving image generation are sequentially erased from the first memory 5 and the second memory 6. As a result, the processing in the image processing unit 3 is completed, and the image processing unit 9 generates a moving image file from the acquired image.

By the above processing, the accumulation period in the first imaging for acquiring the main captured image constituting the moving image, that is, the shutter speed can be freely set. In addition, it is possible to generate a moving image including a captured image with a corrected flash band without delay without causing the same captured image to be continuous.

By the way, when the determination in S102 is NO (when the process proceeds to S108), the process can proceed as follows instead of the above-mentioned flow. That is, the same processing as in S108 described above is executed. After that, in the above S109, the minus gain process is performed and the captured image is stored in the second memory 6, but the captured image is stored in the second memory 6 without performing the minus gain process. Then, when the same processing as in S110 described above is executed and it is detected that a flash band is generated in the image captured by the first imaging, there is no flash band by the second imaging used in place of the flash band. Identify the captured image. Then, a negative gain process of the specified captured image is performed so as to match the exposure (brightness) of the specified captured image with the brightness (exposure) of the captured image obtained by the first imaging. In order to execute the negative gain processing of the captured image stored in the second memory 6, the imaging processing unit 3 needs to include a gain adjusting unit of the captured image stored in the second memory 6. Subsequent processing can be performed as described above after S111. Instead of providing the gain adjustment unit in the image processing unit 3, the negative gain processing execution information is transmitted to the image processing unit 9 together with the image captured by the second image pickup, and the image processing unit 9 executes the negative gain processing. It is also possible to do.

<Second Embodiment>
FIG. 6 is a block diagram showing a schematic configuration of the image pickup apparatus 200 according to the second embodiment of the present invention. The image pickup processing unit 3 of the image pickup apparatus 100 described in the first embodiment has a flash detection unit 7 and an image selection unit 8, whereas the image pickup apparatus 200 has an image pickup processing unit having a flash detection unit 27 and an image addition unit 28. It differs from the image pickup apparatus 100 in that it includes 23. Since the other configurations of the image pickup apparatus 200 are the same as those of the image pickup apparatus 100, the same reference numerals are given to the components of the image pickup apparatus 200 that are the same as the components of the image pickup apparatus 100, and duplicate description is omitted. ..

In the image pickup apparatus 200, the image captured by the first image pickup, which is the main image captured image used for generating the moving image, is stored in the first memory 5. The frame rate of the second imaging is set to be faster than the frame rate of the first imaging, and the captured image by the second imaging is stored in the second memory 6.

FIG. 7 is a diagram illustrating a fourth example of the first imaging and the second imaging. FIG. 7A is a diagram illustrating control of the CMOS image sensor in each of the first image pickup and the second image pickup. In the present embodiment, the accumulation start period of all lines in the first imaging is matched with the readout period of all lines in the second imaging, and the readout period of all lines in the first imaging is the second imaging. Match the accumulation start period of all lines in. Further, the accumulation period in the second imaging is set shorter than the accumulation period in the first imaging. That is, the frame rate in the second imaging is set to be faster than the frame rate in the first imaging. Further, the second imaging is set so that each of the accumulation start period and the readout period of all the lines in the first imaging is covered by a plurality of consecutive accumulation periods. FIG. 7A shows an example in which the accumulations 32a to 32e are performed so that the five captured images are continuously output.

FIG. 7B is a diagram illustrating an addition process of the captured image by the first imaging and the captured image by the second imaging. When a flash band is generated in the captured image obtained by the accumulation 31 in the first imaging, as shown in the upper figure, the flash band caused by the same flash is always accumulated in the second imaging 32a, 32b. It also occurs in the captured image obtained in. Therefore, by adding the two captured images with the flash band obtained by the accumulation 32a and 32b of the second imaging to the captured image with the flash band obtained by the accumulation 31 in the first imaging, the imaging region Generates a captured image in which all lines of are illuminated with a flash of light. That is, the flash was applied to all the lines in the imaging region by adding the required number of continuous short accumulation periods in the second imaging to the accumulation start period and the readout period of all the lines in the first imaging. The same state as in the case is generated, and the flash band of the image captured by the first imaging is corrected.

Note that the example of FIG. 7 shows an example in which a flash band is generated in one or two of the images captured by the second imaging when a flash band is generated in the image captured by the first imaging. However, the second imaging condition is not limited to this, and the frame rate may be set so that a flash band may occur in three or more captured images.

The flash detection unit 27 inspects the presence or absence of a flash band in the captured image stored in the second memory 6, and notifies the image addition unit 28 of the flash band detection result. Since the flash band detection method is based on the detection method by the flash detection unit 7 described in the first embodiment, the description thereof is omitted here. Since short-time (small exposure) imaging such as the accumulated 32a to 32e shown in FIG. 7 is not easily affected by external light other than the flash, the presence or absence of the flash band is determined from the image captured by the second imaging. Thereby, the detection accuracy of the flash band can be improved.

The captured image stored in the first memory 5 is transmitted to the image adding unit 28. Then, as described with reference to FIG. 7B, the image adding unit 28 takes an image by the second imaging in which it is determined that the flash band is generated based on the notification from the flash detecting unit 27. The image is added to the captured image in which the flash band generated by the first imaging is generated. In this way, it is possible to obtain a bright captured image in which the flash band is corrected and all the lines in the imaging region are exposed by flash irradiation. As described with reference to FIG. 7, when a flash band is generated in the image captured by the second imaging, a flash band caused by the same flash is always generated in the image captured by the first imaging. ing. Therefore, it is easy to identify the captured image having the flash band by the first imaging to be the target of the addition processing. The image addition unit 28 transmits the image captured by the first image capture as it is or after correcting the flash band to the image processing unit 9 in synchronization with the control frame rate of the image pickup device 200.

FIG. 8 is a flowchart illustrating a process executed by the image pickup processing unit 23 at the time of moving image shooting by the image pickup apparatus 200. Each process (step) indicated by the S number in FIG. 8 is realized by the MPU 10 executing a predetermined program to comprehensively control the operation of each unit constituting the image pickup processing unit 23. It is assumed that the conditions for the first imaging are predetermined.

In S201, the MPU 10 executes the first image pickup and stores the image captured by the first image pickup in the first memory 5. In S202, the MPU 10 executes a second image pickup and stores the image captured by the second image pickup in the second memory 6.

The second imaging condition is set according to the example described with reference to FIG. 7. That is, the second imaging is set so that the timing of reading out the same line of the second imaging coincides with the timing of the accumulation start of each line of the first imaging. Further, in the second imaging, a continuous predetermined number of captured images at a high-speed frame rate that can be regarded as covering the accumulation start period of all the lines of the first imaging with the accumulation period of all the lines. Take an image. As a result, even if a flash band is generated during the accumulation start period of the first imaging, the captured image in which the flash band is generated is corrected by using the captured image by the second imaging, and the entire line of the imaging region is corrected. It is possible to obtain a bright captured image exposed by flash irradiation.

Further, the second imaging is set so that the timing of reading out each line of the first imaging coincides with the timing of the start of accumulation of the same line of the second imaging. Further, in the second imaging, a continuous predetermined number of captured images at a high-speed frame rate can be regarded as covering the readout period of all the lines of the first imaging with the accumulation period of all the lines. do. As a result, even if a flash band is generated during the readout period of the first imaging, the captured image in which the flash band is generated is corrected by using the captured image by the second imaging, and the entire line of the imaging region is covered. It is possible to obtain a bright captured image exposed by flash irradiation.

In S203, the flash detection unit 7 performs flash band detection processing on the captured image by the second image pickup stored in the second memory 6, and notifies the image addition unit 28 of the flash band detection result. In S204, the image addition unit 28 determines whether or not the captured image in which the flash band is generated is detected based on the flash band detection result in S203. When the image addition unit 28 determines that the captured image in which the flash band is generated is detected (YES in S204), the process proceeds to S205, and when it is determined that the captured image in which the flash band is generated is not detected (YES). NO in S204), the process proceeds to S206.

In S205, the image addition unit 28 captures each image captured by the first image pickup and the second image pickup in which a flash band is generated due to a common flash, as in the example described with reference to FIG. 7 (b). In addition, an captured image in which all lines in the imaging region are irradiated with a flash is generated. In S206, the image addition unit 28 transmits the captured image used for generating the moving image to the image processing unit 9 in accordance with the control frame rate of the image pickup device 200. As a result, the processing in the image processing unit 23 is completed, and the image processing unit 9 generates a moving image file from the acquired image.

By the above processing, as in the first embodiment, the accumulation period in the first imaging for acquiring the main captured image constituting the moving image, that is, the shutter speed can be freely set. In addition, it is possible to generate a moving image including a captured image with a corrected flash band without delay.

Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various embodiments within the range not deviating from the gist of the present invention are also included in the present invention. included. For example, in the above embodiment, various processes executed inside the image pickup apparatus 100 are executed by a plurality of circuits (processing units), but all or a part of the plurality of circuits are included in the MPU 10. It may be configured. That is, the circuit configuration does not matter as long as the processing described with reference to FIGS. 3 and 8 can be performed.

Further, in the second example of the first imaging and the second imaging described with reference to FIG. 4, a configuration in which the accumulation 12b is performed between the accumulation 12a and the accumulation 12c has been described. On the other hand, the gain of the captured image obtained by the storage 12a and 12c may be adjusted so as to be the same as the exposure of the captured image obtained by the storage 11 without performing the storage 12b. Further, in the above embodiment, the image pickup apparatus 100 has a built-in storage unit 11 and a display unit 12, but the storage unit 11 and the display unit 12 are external devices communicably connected to the MPU 10. It doesn't matter. In the above embodiment, the image pickup apparatus 100 according to the present invention is a digital camera or a digital camcorder. However, the control method of the image pickup device described in the above embodiment can also be applied to various electronic devices (for example, smartphones, mobile phones, game machines, tablet PCs, etc.) having an image pickup function by the rolling shutter method.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

3,23 Image processing unit 4 Image processing unit 5 First memory 6 Second memory 7,27 Flash detection unit 8 Image selection unit 9 Image processing unit 28 Image addition unit 100, 200 Image pickup device

Claims (14)

An image sensor capable of performing a first image pickup and a second image pickup with different accumulation timings in parallel, and an image sensor.
The first image pickup and the said A control means for performing a second image pickup,
A storage means for storing the captured images obtained by the first imaging and the second imaging, and
A detection means for detecting the occurrence of a flash band in an captured image stored in the storage means, and a detection means.
A selection means for selecting a captured image in which no flash band is generated based on the detection result by the detection means, and a selection means.
An image pickup apparatus comprising: a moving image generating means for generating a moving image using an image taken by the selection means. The imaging device according to claim 1, wherein the accumulation period of all lines in the first imaging and the accumulation period of all lines in the second imaging are temporally the same length. When the length of the accumulation period of all the lines in the second imaging is different from the length of the accumulation period of all the lines in the first imaging, the exposure of the image captured by the second imaging is the first. The image pickup apparatus according to claim 1, further comprising a correction means for correcting the gain of the image captured image obtained by the second image pickup so as to match the exposure of the image captured by the image pickup. When the flash band is not generated in both the first image pickup and the captured image obtained by the second image pickup at a predetermined timing, the selection means generates the flash band by the first image pickup. The image pickup apparatus according to any one of claims 1 to 3, wherein an image captured image that has not been captured is selected. When the accumulation period in the first imaging is longer than the accumulation period of the control frame rate of the imaging device, the control means starts accumulating all lines in the first imaging in the second imaging. The first accumulation of the accumulation period covering the period, the second accumulation of the accumulation period covering the readout period of all the lines in the first imaging, and the period between the first accumulation and the second accumulation. Perform the third accumulation with the accumulation period as
The first aspect of the present invention is characterized in that the storage means stores an image taken by adding the first accumulation and the third accumulation and an image taken by adding the second accumulation and the third accumulation. The imaging device described. In the control means, when the accumulation period of each line in the first imaging is less than twice the accumulation start period of all lines, the accumulation period of all lines in the second imaging is the first. The imaging device according to claim 1, wherein the second imaging is performed so as to overlap with the period from the start of accumulation of all lines in the imaging to the end of reading. An adjusting means for adjusting the output from the image pickup element in the second imaging so as to match the exposure of the captured image obtained by the second imaging with the exposure of the captured image obtained by the first imaging is provided. The image pickup apparatus according to claim 6, wherein the image pickup device is characterized by the above. A correction means for correcting the gain of the captured image obtained by the second imaging so as to match the exposure of the captured image obtained by the second imaging with the exposure of the captured image obtained by the first imaging. The image pickup apparatus according to claim 6, further comprising. An image sensor capable of performing a first image pickup and a second image pickup with different accumulation timings in parallel, and an image sensor.
A control means for setting the second image pickup to a frame rate faster than that of the first image pickup and executing the first image pickup and the second image pickup.
A storage means for storing the captured images obtained by the first imaging and the second imaging, and
A detection means for detecting the occurrence of a flash band in an captured image obtained by the second imaging and stored in the storage means, and a detection means.
Based on the detection result by the detection means, an image captured by adding the image captured by the flash band obtained by the first imaging and the image generated by the flash band obtained by the second imaging is generated. Image addition means and
An image pickup apparatus comprising: an image pickup image obtained by the first image pickup in which a flash band is not generated, and a moving image generation means for generating a moving image using the image pickup image generated by the image addition means. .. The control means matches the accumulation start period of each line in the first imaging with the readout period of the same line in the second imaging, and the readout period of each line in the first imaging. The imaging device according to claim 9, wherein the image is matched with the accumulation start period of the same line in the second imaging. In the second imaging, the control means has a continuous predetermined number that can be regarded as covering the accumulation start period and the readout period of all the lines of the first imaging with the accumulation period of all the lines. The image pickup apparatus according to claim 9 or 10, wherein the image pickup image is acquired by accumulating the images. It is a control method of an image pickup device provided with an image pickup element capable of performing a first image pickup and a second image pickup having different accumulation timings in parallel.
The first image pickup and the said The step of performing the second image pickup and
A step of storing the captured image obtained by the first image pickup and the second image pickup in the storage means,
The step of detecting the occurrence of a flash band in the captured image stored in the storage means, and
A step of selecting a captured image in which no flash band is generated based on the result of detecting the occurrence of the flash band, and
A control method for an image pickup apparatus, comprising: a step of generating a moving image using a selected captured image. It is a control method of an image pickup device provided with an image pickup element capable of performing a first image pickup and a second image pickup having different accumulation timings in parallel.
A step of setting the second image pickup to a frame rate faster than that of the first image pickup and executing the first image pickup and the second image pickup.
A step of storing the captured image obtained by the first image pickup and the second image pickup in the storage means,
A step of detecting the occurrence of a flash band in the captured image obtained by the second imaging and stored in the storage means, and
Based on the result of detecting the generation of the flash band, the captured image obtained by adding the captured image obtained by the first imaging and the generated image of the flash band obtained by the second imaging is added. And the steps to generate
A control method for an image pickup apparatus, comprising: a captured image in which a flash band is not generated obtained by the first imaging and a step of generating a moving image using the captured image generated by the addition. A program comprising causing a computer to execute the control method of the image pickup apparatus according to claim 12 or 13.

Images