JP2024054604A - Imaging device, control method and program thereof - Google Patents

Abstract

[Problem] To obtain an HDR panoramic image having an appropriate contrast with reduced occurrence of whiteout and blackout in each region to be panoramic-combined. [Solution] An imaging device 100 including an image sensor 102 capable of outputting multiple images with different gains in one imaging is panned to capture multiple images, an area for determining the exposure step difference in the next imaging is detected from the image captured immediately before each of the multiple imaging, an exposure step difference in the next imaging is determined from the detected area, the gain in the next imaging is controlled so as to achieve the determined exposure step difference, and after the multiple imaging is completed, HDR composition is performed for each of the multiple imagings using the multiple images output in one imaging, and a panoramic image is generated by cutting out and connecting predetermined areas from the images obtained by HDR composition. [Selected Figure] FIG.

Description

The present invention relates to an imaging device and a control method and program for the same, and in particular to a technique for generating a panoramic image with an expanded dynamic range from multiple still images captured by panning the imaging device.

An imaging element (DGO (Dual Gain Output Sensor) sensor) is known that is capable of outputting images with different gains by providing two column circuits for pixel-by-pixel output signals and separate gains for the amplifiers in the column circuits. The DGO sensor can output two images with different gains (for example, high gain and low gain) with one image capture (exposure).

Therefore, the synthesis of two images captured by a DGO sensor is suitable for HDR synthesis, a method of obtaining an image with an expanded dynamic range. Furthermore, HDR synthesis of two images captured by a DGO sensor has the advantage that it does not require alignment processing and can reduce blurring of moving objects, compared to the synthesis of two images obtained by time-division exposure.

HDR synthesis is a useful technique for generating a panoramic image with a wider angle of view from multiple images (see, for example, Patent Documents 1 and 2). This is because generating a panoramic image requires capturing an extremely wide area, and when the capturing area is wide, the difference in brightness within the capturing area tends to become large.

In panoramic photography, the imaging device is generally moved in one direction while capturing images continuously with overlapping areas in the imaging range. A panoramic image can be generated by linking (splicing) the images obtained in this way while aligning the overlapping areas between the images (hereinafter referred to as "panoramic synthesis"). Two images with different gains obtained by capturing images using a DGO sensor are HDR-synthesized, and the obtained HDR images are then panorama-synthesized using conventional methods, thereby generating a panoramic image with a wide dynamic range (hereinafter referred to as "HDR panoramic image").

JP 2021-90175 A JP 2017-143390 A

At the start of panoramic shooting, the brightness difference of the entire imaging area is often unknown, so when performing panoramic shooting with HDR compositing, it is not easy to determine an appropriate exposure step difference in advance. For example, if the exposure step difference is determined in advance and panoramic shooting is started, a range that cannot be accommodated by the exposure step difference determined in advance may be captured during shooting, resulting in blown-out areas in the resulting HDR image. In contrast, if the exposure step difference is set larger than necessary and panoramic shooting with HDR compositing is performed, the contrast of the HDR image will decrease, resulting in reduced image quality. Thus, when performing panoramic shooting with HDR compositing, it is not easy to determine the optimal exposure step difference before starting shooting.

The present invention has been made in consideration of these circumstances, and aims to provide a technology that makes it possible to obtain an HDR panoramic image with appropriate contrast that reduces the occurrence of blown-out highlights and crushed shadows in each area that is panoramic synthesized.

The imaging device according to the present invention is an imaging device equipped with an imaging element capable of outputting multiple images with different gains in one imaging, and is characterized by comprising: an acquisition means for acquiring multiple images with different gains for each imaging by performing imaging multiple times with the imaging element while panning the imaging device; a detection means for detecting an area for determining an exposure step difference for the next imaging from an image obtained by imaging immediately before the multiple imaging times in the multiple imaging times; a determination means for determining an exposure step difference for the next imaging from the area detected by the detection means; a control means for controlling the different gains in the next imaging so as to achieve the exposure step difference determined by the determination means; a synthesis means for performing HDR synthesis for each of the multiple imaging times for the multiple images output in one imaging by the imaging element; and a generation means for generating a panoramic image by cutting out and connecting a predetermined area from the images obtained by the HDR synthesis.

According to the present invention, it is possible to obtain an HDR panoramic image with appropriate contrast that reduces the occurrence of whiteout and blackout in each area that is panoramic synthesized.

1 is a block diagram showing a schematic configuration of an imaging apparatus according to an embodiment. FIG. 2 is a block diagram showing a configuration of an imaging element. 4 is a circuit diagram showing a configuration of a column AMP unit. FIG. FIG. 2 is a block diagram showing a configuration of an HDR synthesis unit. 11A and 11B are diagrams illustrating the relationship between the input light amount of an H image and an L image and the output after A/D conversion. FIG. 2 is a schematic diagram of a method for generating an HDR panoramic image from an H image and an L image. 1 is a flowchart of a process from capturing an image to generating an HDR panoramic image.

The following describes in detail an embodiment of the present invention with reference to the attached drawings.

Fig. 1 is a block diagram showing a schematic configuration of an imaging device 100 according to an embodiment. The imaging device 100 has an optical lens 101, an imaging element 102, an image acquisition unit 103, a signal processing unit 104, an HDR synthesis unit 105, a panorama synthesis unit 106, a signal storage unit 107, an area detection unit 108, an exposure control unit 109, an imaging element control unit 110, and a system control unit 120.

The system control unit 120 is composed of a CPU that performs calculation processing, a ROM that stores programs executed by the CPU and various data required for executing the programs, and a RAM that has an area for the CPU to expand the programs and an area for temporarily storing various data. The system control unit 120 comprehensively controls the operation of each part of the imaging device 100 by having the CPU read out a specific program from the ROM and expand it into the RAM.

The optical lens 101 captures light from a subject and focuses the incident light on the imaging surface of the image sensor 102. The image sensor 102 is specifically a CCD image sensor or a CMOS image sensor, and is configured as the above-mentioned DGO sensor. The image sensor 102 converts the optical image focused on the imaging surface into an analog electrical signal, and further A/D converts the analog electrical signal to generate an image signal consisting of a digital signal such as LVDS, and outputs it to the image acquisition unit 103. The configuration of the image sensor 102 will be described in detail later.

The image acquisition unit 103 is composed of multiple blocks (not shown) that perform various processes on the image signal (digital signal) output from the image sensor 102, such as removing fixed pattern noise from the image sensor 102 and performing black level clamping. The image acquisition unit 103 also performs a process to separate the image signal into a signal (image data) to be used for storage and an evaluation signal for controlling the operation of the image sensor 102.

The signal processing unit 104 performs various processes such as pixel addition, noise reduction, gamma correction, knee, digital gain, and scratch correction, which are typical image processing functions of the imaging device 100. Note that the image acquisition unit 103 and the signal processing unit 104 each have a memory circuit (not shown) that stores setting values required for performing various processes.

The following describes the HDR synthesis unit 105, panorama synthesis unit 106, signal storage unit 107, area detection unit 108, exposure control unit 109, and image sensor control unit 110, focusing on their functions for generating HDR panorama images.

The HDR synthesis unit 105 synthesizes an HDR image using an arbitrary synthesis method from the image signal output from the signal processing unit 104. In this embodiment, a high-gain image is used as the normal image, and a low-gain image is synthesized in the bright, blown-out parts of the normal image.

The panorama synthesis unit 106 generates an HDR panorama image from multiple HDR images obtained by HDR synthesis using an arbitrary panorama synthesis method. In this embodiment, the panorama synthesis method used is a method in which strip areas are cut out from each HDR image, and the cut out strip areas are aligned and linked (joined together). Note that the aspect ratio of the strip areas does not matter, and therefore they may be substantially rectangular areas.

The signal storage unit 107 includes a storage medium (not shown) and stores image data of the HDR panoramic image output from the panorama synthesis unit 106 in the storage medium.

The area detection unit 108 detects an area for determining the exposure step difference between two images with different gains in the next capture from the image obtained by the immediately preceding capture based on the image signal output from the image acquisition unit 103 and the panning direction during panoramic shooting. Hereinafter, the exposure step difference between two images that can be obtained by one capture (exposure) is referred to as the "HDR step difference," and the area detected by the area detection unit 108 is referred to as the "HDR step difference determination area."

The exposure control unit 109 calculates the exposure amount and HDR step difference that will result in proper exposure for the next image capture from the HDR step difference determination area detected by the area detection unit 108, and notifies the image sensor control unit 110 of the determined exposure amount and HDR step difference.

During imaging operations to generate an HDR panoramic image, the image sensor control unit 110 controls the exposure of the image sensor 102 based on the exposure amount and HDR step difference notified by the exposure control unit 109.

The algorithms used for various processes inside the imaging device 100 (for example, algorithms for HDR compositing of two images with different gains, alignment and linking of rectangular regions, determination of HDR step difference determination region and HDR step difference, etc.) are not limited.

Figure 2 is a block diagram showing the schematic configuration of the image sensor 102. As described above, the image sensor 102 is configured as a DGO sensor. The image sensor 102 has a timing pulse control unit 201, a vertical scanning circuit 202, a pixel unit 203, a column AMP 204, a column ADC 205, a horizontal transfer circuit 206, a signal processing circuit 207, and an external output circuit 208.

The timing pulse control unit 201 supplies an operating clock (operation CLK) to each block of the image sensor 102, and also supplies a timing signal to each block to control the operation of the image sensor 102. The vertical scanning circuit 202 performs timing control to sequentially read out the pixel signal voltages of the two-dimensionally arranged pixel section 203 during one frame. In general, image signals are read out row by row, from the top row to the bottom row during one frame. The pixel section 203 is a photoelectric conversion element that performs photoelectric conversion according to the amount of incident light and outputs a voltage.

The column AMP 204 electrically amplifies the signal read from the pixel unit 203. This amplifies the signal level of the pixel relative to the noise generated by the subsequent column ADC 205, thereby equivalently improving the S/N ratio. In addition, the image sensor 102 is configured so that the gain of the column AMP 204 can be changed from the timing pulse control unit 201. Furthermore, in the image sensor 102, two input memories are provided in the column AMP 204, and it is possible to change the column AMP gain and output two types of gain. By providing two input memories, it is possible to multiply the signal read from the pixel unit 203 by different gains and output it, so although the amount of data increases, it is possible to obtain two images (still images) with different gains that are simultaneous.

The column ADC 205 performs A/D conversion on the read signal from the column AMP 204. The signals digitized by the column ADC 205 are sequentially read out by the horizontal transfer circuit 206. The output signal from the horizontal transfer circuit 206 is input to the signal processing circuit 207.

The signal processing circuit 207 is a circuit that performs signal processing digitally, and performs gain processing such as adding a fixed offset value, shift calculation processing, and multiplication processing. Note that it is also possible to intentionally provide a light-shielded pixel area in the pixel unit 203, and have the signal processing circuit 207 perform a black level clamp operation using the output from the light-shielded pixel area. The signal output from the signal processing circuit 207 is input to the external output circuit 208.

The external output circuit 208 has a serializer function and converts the multi-bit input parallel signal from the signal processing circuit 207 into a serial signal. The external output circuit 208 also converts the generated serial signal into, for example, an LVDS signal, and outputs it to an external device as image information.

Figure 3 is a circuit diagram showing the configuration of column AMP 204. One column in column AMP 204 is shown in Figure 3, and the operation of column AMP 204 during HDR synthesis will be explained here.

In the column AMP 204, the operational amplifier 305 (hereinafter referred to as "OP305") is connected to input capacitances C303 and C304 and feedback capacitances C306 and C308. The signal read from the pixel unit 203 can be connected to the input capacitances C303 and C304 by the switches SW301 and SW302, respectively. The switch SW307 controls the connection of the feedback capacitance C308.

The amplification factor of the column AMP204 is a value obtained by dividing the input capacitance by the feedback capacitance, since capacitance is used here. Since it has two input capacitances C303 and C304, first, switch SW301 is connected and switches SW302 and SW307 are disconnected, and the signal is multiplied by the gain of the input capacitance C303 and the feedback capacitance C306, and output to the column ADC205. Next, switch SW301 is disconnected, switches SW302 and SW307 are connected, and the signal is multiplied by the gain of the input capacitance C304 and the feedback capacitances C306 and C308, and output. This makes it possible to output two (two types of) images with different gains.

Figure 4 is a diagram showing the functional blocks of the HDR synthesis unit 105. The two images to which different gains have been applied are referred to as a high gain image (hereinafter referred to as an "H image") and a low gain image (hereinafter referred to as an "L image").

The HDR synthesis unit 105 uses an exposure correction block 401 to make the gains of the H image and L image the same in order to match the brightness of a pair of H and L images, and outputs them to an image synthesis block 402. The HDR synthesis unit 105 synthesizes an HDR image from the exposure-corrected H and L images using the image synthesis block 402, and outputs the HDR image to the panorama synthesis unit 106 (not shown in FIG. 4).

Figure 5 shows the relationship between the amount of input light (horizontal axis) and the output after A/D conversion (vertical axis) before and after HDR synthesis of the H image and the L image. Note that the unit of input light is generally candela (cd).

Figure 5 (a) shows the relationship between the amount of input light for the H image (thick line) and L image (thin line) and the output after A/D conversion. In this state, the brightness of the H image and the L image differs, so HDR synthesis cannot be performed as is. Therefore, a gain is applied in the exposure compensation block 401 to match the brightness of the H image and the L image.

Figure 5(b) shows an example in which a gain is multiplied to the L image so that the brightness of the L image matches that of the H image in order to match the brightness of the H image and the L image. The L image that has been exposure corrected to match the brightness of the H image is hereinafter referred to as the "L2 image." In this case, the gain multiplied to the L image in the exposure correction block 401 is a value greater than 1. Note that the gain is a value less than 1 when performing exposure correction to match the brightness of the H image to that of the L image.

Figure 5 (c) shows the relationship between the amount of input light and the output after A/D conversion in an HDR image that combines an H image and an L2 image. HDR synthesis is performed using the H image for areas where the amount of input light is small, and the L2 image for areas where the amount of input light is large. The HDR image obtained in this way has a linear output in relation to the input signal.

Figure 6 is a schematic diagram explaining a method of synthesizing an HDR image from a properly exposed image and an underexposed image, and then performing panoramic synthesis. This shows the flow of performing HDR synthesis and panoramic synthesis for each of frames A, B, C, D, and E obtained by taking five images (exposing the image sensor 102) while panning the imaging device 100 to the right. Note that imaging is performed so that 2/3 of the images overlap between adjacent frames. Also, it is assumed that the exposure correction described with reference to Figure 5 has already been performed on the underexposed image and properly exposed image.

In panoramic photography in which multiple captures are performed, the HDR step difference determination area for the next capture is detected from the properly exposed image obtained in the immediately preceding capture, excluding the area to be used for panoramic synthesis. For example, assume that a properly exposed image and an underexposed image of frame A were obtained in the current capture. The area to be used for panoramic synthesis in frame A is area b, and area c is detected as the HDR step difference determination area for the capture of the next frame B, and area c is used to determine the HDR step difference when capturing frame B. Similarly, the area to be used for panoramic synthesis in frame B is area c, and area d is detected as the HDR step difference determination area for the capture of the next frame C, and area d is used to determine the HDR step difference when capturing frame C. The rest is similar, so description will be omitted.

After HDR-compositing the properly exposed and underexposed images for each of frames A to E, a strip of area to be used for panorama composition is cut out from each HDR image, aligned, and linked to generate an HDR panoramic image. For example, area b' corresponding to area b in the properly exposed and underexposed images is cut out from the HDR image generated from frame A. Similarly, areas c', d', e', and f' are cut out from the HDR images generated from frames B to E, respectively, and areas b', c', d', e', and f' are linked to generate an HDR panoramic image.

Next, the process flow from capturing images to synthesizing an HDR panoramic image will be described. Here, while panning the imaging device 100, multiple captures are performed using the image sensor 102, which is capable of outputting two images with different gains applied in one capture (exposure), and HDR synthesis is performed for each frame, followed by panoramic synthesis.

Figure 7 is a flowchart showing the process flow from capturing images to synthesizing an HDR panoramic image. Each process (step) indicated by an S number in this flowchart is realized by the CPU in the system control unit 120 expanding a predetermined program stored in the ROM into the RAM and comprehensively controlling the operation of each part constituting the imaging device 100.

In S701, the exposure control unit 109 determines the HDR step difference at the start of panoramic shooting, that is, for the first of multiple image captures. For example, the ISO sensitivity of the H image is determined so that the H image is properly exposed, and the ISO sensitivity of the L image is determined so that the HDR step difference is the one determined for the ISO sensitivity of the H image. The ISO sensitivity may be determined by manual setting by the photographer, or automatically by the imaging device 100. In the case of automatic determination, it can be determined using luminance information of the entire imaging range obtained from the image sensor 102 before the first image capture.

In S702, the image sensor control unit 110 controls the image sensor 102 to capture images with the HDR step difference determined in S701, and obtain two images with different gains. Here, the H image is obtained as a properly exposed image, and the L image is an underexposed image with the HDR step difference determined for the properly exposed image.

In S703, the system control unit 120 determines whether or not to end the panoramic shooting. The determination criterion may be a user's operation to end the panoramic shooting, or the completion of a preset number of captures (exposures) (a predetermined number of frames have been obtained), etc. If the system control unit 120 determines not to end the panoramic shooting (NO in S703), it executes the process of S704, and if it determines to end the panoramic shooting (YES in S703), it executes the process of S706.

In S704, the area detection unit 108 detects an HDR step difference determination area from the properly exposed image obtained in the immediately preceding image capture. As the HDR step difference determination area, an area in the panning direction in the properly exposed image is usually detected. The panning direction can be estimated from the amount of blur of the image sensor 102, or can be estimated using information from a gyro sensor that detects vibrations such as shaking or blurring of the image capture device 100, and any method may be used. For example, a method of detecting a predetermined area according to the panning direction can be used to detect the HDR step difference determination area. Also, a method of estimating the panning speed from information from the gyro sensor, detecting a wider area in the panning direction as the panning speed increases, and detecting a narrower area closer to the center of the image capture range (image) as the panning speed decreases, may be used.

In S705, the exposure control unit 109 uses the HDR step difference determination area determined in S704 to determine the HDR step difference for the next image capture. The method for determining the HDR step difference may involve dividing the HDR step difference determination area into a predetermined number of blocks and determining the step difference from the maximum luminance value or luminance difference for each block, but is not particularly limited as long as it determines the HDR step difference.

After processing S705, the system control unit 120 executes processing S702. This causes the processing of S702 to S705 to be repeated, but when multiple images have been captured to perform panoramic composition, the determination in S703 becomes YES, and processing of S706 is executed.

In S706 after the panoramic shooting is completed, the HDR synthesis unit 105 performs HDR synthesis of the H image and the L image for each frame for all frames acquired in S702.

In S707, the panorama synthesis unit 106 cuts out rectangular regions to be used for panorama synthesis from each of all HDR images obtained in S706, and connects the cut out rectangular regions. When the HDR panorama image is thus generated, the system control unit 120 ends this process.

As described above, this embodiment makes it possible to obtain an HDR panoramic image with appropriate contrast that reduces the occurrence of whiteout and blackout in each area that is panoramic synthesized.

The disclosure of this embodiment includes the following configurations and methods.
(Configuration 1) An imaging device having an imaging element capable of outputting multiple images with different gains applied in one imaging, comprising: an acquisition means for acquiring multiple images with different gains applied for each imaging by panning the imaging device and performing multiple imaging using the imaging element; a detection means for detecting an area for determining an exposure step difference for the next imaging from an image obtained by imaging immediately before, in the multiple imaging operations; a determination means for determining an exposure step difference for the next imaging operation from the area detected by the detection means; a control means for controlling the different gains for the next imaging operation so that the exposure step difference is determined by the determination means; a synthesis means for performing HDR synthesis of the multiple images output in one imaging operation using the imaging element for each of the multiple imaging operations; and a generation means for generating a panoramic image by cutting out and connecting a predetermined area from the images obtained by HDR synthesis.
(Configuration 2) The imaging device described in Configuration 1, wherein the determination means determines the exposure step difference for a first image capture of the multiple image captures based on luminance information of the entire imaging range obtained from the imaging element before the first image capture.
(Configuration 3) The imaging device according to configuration 1 or 2, wherein the control means controls the different gains so that the multiple imaging operations result in imaging that obtains a set of a properly exposed image and an overexposed image.
(Configuration 4) The imaging apparatus according to configuration 1 or 2, wherein the control means controls the different gains so that the multiple imaging operations result in imaging that obtains a set of a properly exposed image and an underexposed image.
(Configuration 5) An imaging device described in any one of configurations 1 to 4, characterized in that the detection means uses a region other than a predetermined region cut out to generate the panoramic image as a region for determining a different gain for the next imaging.
(Structure 6) An imaging device as described in Structure 5, further comprising an estimation means for estimating the panning speed, wherein the detection means detects a wider area toward the panning direction as the area for determining the exposure step difference when the panning speed is faster, and detects a narrower area closer to the center of the imaging range when the panning speed is slower.
(Configuration 7) The imaging device according to any one of configurations 1 to 6, wherein the panoramic image has a wider dynamic range than the plurality of images.
(Configuration 8) A program for causing a computer to function as each of the means of the imaging device according to any one of configurations 1 to 7.
(Method 1) A method for controlling an imaging device having an imaging element capable of outputting a plurality of images with different gains in a single imaging operation, comprising the steps of:
acquiring a plurality of images with a different gain applied for each image capture by performing image capture a plurality of times using the image capture element while panning the image capture device;
performing HDR compositing for each of a plurality of images output by the image sensor in one imaging operation;
and generating a panoramic image by cutting out and connecting a predetermined area from the image obtained by the HDR synthesis,
The step of acquiring a plurality of images includes:
detecting an area for determining an exposure step difference in a next image taken from an image obtained by an immediately preceding image taken in the plurality of times of image taking;
determining an exposure step difference for the next image capture from the detected area;
and controlling the different gain in the next image capture so as to achieve the determined exposure step difference.

Although the present invention has been described above in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. For example, in the above embodiment, the image sensor 102 outputs a digital signal, but the image sensor 102 may not perform A/D conversion internally, in which case the image acquisition unit 103 is configured to include an analog front end that performs A/D conversion. In addition, the image sensor 102 is configured to obtain two images with different gains that are simultaneous by multiplying the pixel signal by two (different) gains, but the number of images that can be output simultaneously is not limited to two.

In the flowchart of FIG. 7, the H image is obtained as a properly exposed image and the L image is obtained as an underexposed image, but the H image may be obtained as an overexposed image and the L image as a properly exposed image. Also, the H image and L image obtained in each frame are HDR-composited, and then strip-shaped regions are cut out and linked to generate an HDR panoramic image. Without being limited to this, strip-shaped regions constituting an HDR panoramic image may be cut out from the H image and L image obtained in each frame, HDR-composited, and the obtained strip-shaped HDR images may be linked to generate an HDR panoramic image.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

REFERENCE SIGNS LIST 100 Imaging device 102 Imaging element 103 Image acquisition unit 104 Signal processing unit 105 HDR synthesis unit 106 Panorama synthesis unit 109 Exposure control unit 110 Imaging element control unit 120 System control unit 204 Column AMP

Claims (9)

An imaging device including an imaging element capable of outputting a plurality of images multiplied by different gains in one imaging operation,
an acquisition means for acquiring a plurality of images, each of which is multiplied with a different gain, by performing imaging a plurality of times using the imaging element while panning the imaging device;
a detection means for detecting an area for determining an exposure step difference in a next image taken from an image obtained by an immediately preceding image taken in the plurality of times of image taking;
a determination means for determining an exposure step difference for the next image capture from the area detected by the detection means;
a control means for controlling the different gains in the next image capture so as to achieve the exposure step difference determined by the determination means;
A synthesis means for performing HDR synthesis for each of a plurality of images output by the imaging element in one imaging operation;
and a generating means for generating a panoramic image by cutting out and connecting a predetermined area from the image obtained by the HDR synthesis. The imaging device according to claim 1, characterized in that the determining means determines the exposure step difference for the first of the multiple imaging operations based on luminance information of the entire imaging range obtained from the imaging element before the first image capture. The imaging device according to claim 1 or 2, characterized in that the control means controls the different gains so that the multiple imaging operations result in imaging that obtains a set of a properly exposed image and an overexposed image. The imaging device according to claim 1 or 2, characterized in that the control means controls the different gains so that the multiple imaging operations result in imaging that obtains a set of a properly exposed image and an underexposed image. The imaging device according to claim 1 or 2, characterized in that the detection means determines a region different from the predetermined region cut out to generate the panoramic image as a region for determining a different gain for the next imaging. An estimation means for estimating a speed of the panning,
The imaging device according to claim 5, characterized in that the detection means detects a wider area toward the panning direction as the panning speed increases, and detects a narrower area closer to the center of the imaging range as the panning speed decreases, as the exposure step difference is determined. The imaging device according to claim 1 or 2, characterized in that the panoramic image has a wider dynamic range than the multiple images. A method for controlling an image capturing apparatus having an image capturing element capable of outputting a plurality of images with different gains in a single image capturing operation, comprising the steps of:
acquiring a plurality of images with a different gain applied for each image capture by performing image capture a plurality of times using the image capture element while panning the image capture device;
performing HDR compositing for each of a plurality of images output by the image sensor in one imaging operation;
and generating a panoramic image by cutting out and connecting a predetermined area from the image obtained by the HDR synthesis.
The step of acquiring a plurality of images includes:
detecting an area for determining an exposure step difference in a next image taken from an image obtained by an immediately preceding image taken in the plurality of times of image taking;
determining an exposure step difference for the next image capture from the detected area;
and controlling the different gain in the next image capture so as to achieve the determined exposure step difference. A program that causes a computer to function as each of the means of the imaging device according to claim 1 or 2.

Images