JP7100535B2 - Image pickup device and image processing device, their control method, program, storage medium - Google Patents

Description

The present invention relates to a technique for obtaining a wider range of starry sky images, that is, starry sky panoramic images, by synthesizing continuously shot images while sequentially changing the shooting directions so that common areas can be captured.

A panoramic photography method is known as one of the photography methods for capturing a vast starry sky as a single image. As an example of this method, Patent Document 1 discloses the following photographing method. That is, a plurality of unit images constituting a part of the shooting range are imaged by sequentially changing the shooting direction, and an image area of a predetermined size is cut out from the captured unit image so as to generate overlapping areas with each other, and the cut out image area. To generate a panoramic image by sequentially superimposing.

Here, the problems when applying this method to panoramic photography of the starry sky will be described. When shooting a starry sky, the amount of light of the stars is very small, so long-time exposure such as 30 seconds or 1 minute is often performed. Since the celestial body moves diurnal in accordance with the rotation of the earth, the stars do not become point images but become light trails when exposed for a long time.

When taking a panoramic picture of a starry sky, it is necessary to take an image with a different shooting direction with an exposure time that does not cause the star to become a light trail, and to generate a panoramic image by stitching the images together. However, there are cases where you want to take a panoramic shot of the starry sky with long exposure. In this case, since the stars move between the images with the passage of time, it is easy to fail in the alignment of the images. FIG. 8 shows an example of misalignment when two images having different shooting directions are combined in panoramic shooting of a starry sky. FIG. 8A shows the first captured image, and FIG. 301 shows the background. FIG. 8B shows the second captured image, and FIG. 302 shows the background. The position of the star in the second image is the first one due to the passage of time, which is the sum of the time of the first long-second exposure shooting and the time when the user sets the image pickup device to change the shooting direction. It is moving with respect to the position of the star in the image. In FIG. 8 (c), since the star was used as the alignment reference when aligning the first and second captured images, the backgrounds 301 and 302 were combined by the amount of the movement of the star. It shows the state.

On the other hand, in Patent Document 2, the positional deviation on the imaging surface due to the diurnal motion of the celestial body is corrected by using an optical shake correction means, and imaging and composition are repeated to prevent misalignment. A technique for obtaining a photographed image having a wide angle of view is disclosed.

Japanese Unexamined Patent Publication No. 2005-328497 Japanese Unexamined Patent Publication No. 2016-005160

However, in the prior art disclosed in Patent Document 2, since the optical shake correction means is used, the maximum value of the change in the shooting direction is limited, and an astronomical image having a wider angle of view is taken. There is a problem that it cannot be done.

The present invention has been made in view of the above-mentioned problems, and an object thereof is an image pickup apparatus capable of taking a high-quality panoramic image even when a star as a subject moves between images having different shooting directions. Is to provide.

The image pickup device according to the present invention is an image pickup device capable of generating one composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common area, and captures a subject image. The image pickup means and the image pickup means are made to image the plurality of unit images, and the exposure time is longer than the exposure time of the unit image before and after the acquisition of each unit image of the plurality of unit images. A control means for capturing a short short-second exposed image, a first short-second exposed image captured after capturing the first unit image, and a second unit captured following the first unit image. Calculation to calculate the amount of movement of the subject between the first short-second exposed image and the second short-second exposed image using the second short-second exposed image captured before the image was captured. Based on the means and the movement amount of the subject calculated by the calculation means, the first unit image and the second unit image are aligned, and the first unit image and the second unit are aligned. It is characterized by comprising a compositing means for compositing images.

Further, the image processing apparatus according to the present invention is an image processing apparatus capable of generating one composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common region. Acquires the plurality of unit images and short-second exposed images captured with an exposure time shorter than the exposure time of the unit images before and after imaging of each of the plurality of unit images. The acquisition means to be performed, the first short-second exposed image captured after the first unit image is captured, and the second unit image captured following the first unit image captured before the capture. Using the second short-second exposed image, the calculation means for calculating the amount of movement of the subject between the first short-second exposed image and the second short-second exposed image, and the calculation means. A compositing means for aligning the first unit image and the second unit image based on the amount of movement of the subject and synthesizing the first unit image and the second unit image. It is characterized by having.

According to the present invention, it is possible to provide an image pickup device capable of taking a high-quality panoramic image even when a star as a subject moves between images having different shooting directions.

A conceptual diagram for panoramic composition of multiple captured images. The block diagram which shows the structure of 1st Embodiment of the image pickup apparatus of this invention. A flowchart illustrating normal shooting operation. The flowchart which shows the operation of the starry sky panoramic photography in 1st Embodiment. A data flow diagram illustrating the operation of the first embodiment. The flowchart which shows the operation of the starry sky panoramic photography in 2nd Embodiment. The conceptual diagram of the warning screen in 3rd Embodiment. A conceptual diagram explaining the problem of starry sky panoramic composition.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a diagram showing an outline of panoramic photography. In the present embodiment, panoramic shooting is performed by sequentially changing the shooting direction while the user 202 moves the image pickup device 201 as shown in FIG. 1A or by moving the image pickup device 201 on an automatic pan head or the like. .. As shown in FIG. 1 (b), a plurality of images are photographed so that each captured image partially includes a common area of the subject. Then, the feature points in the common area of each image are extracted, and the movement vector indicating how much the feature points have moved is detected. By calculating, for example, the coefficient of the affine transformation from the movement vector and superimposing the two images so that the feature points match, an image in which the portion other than the common region is expanded can be obtained. By repeating this a plurality of times, it is possible to generate a panoramic image having a wider angle of view than the angle of view obtained by one shooting.

FIG. 2 is a block diagram showing a configuration of a first embodiment of the image pickup apparatus of the present invention. In FIG. 2, the image pickup apparatus 100 includes a photographing lens 101 for forming an image of a subject image and an AF (autofocus) drive circuit 102 for adjusting the focus of the photographing lens 101. The AF drive circuit 102 is composed of, for example, a DC motor or a stepping motor, and adjusts the focus by changing the position of the focus lens of the photographing lens 101 under the control of the microcomputer 123.

The photographing lens 101 includes an aperture 103, and the aperture 103 is driven by an aperture drive circuit 104. The optical aperture value is calculated by the microcomputer 123, and the amount of the aperture drive circuit 104 driving the aperture 103 is determined based on the calculation.

A main mirror 105 is arranged behind the aperture 103. The main mirror 105 switches between a state in which the luminous flux passing through the photographing lens 101 is guided to the finder and a state in which the light beam is guided to the image pickup element 112. The main mirror 105 is arranged at a position where the light flux is reflected upward so that the light flux always directs to the finder, but when shooting is performed or when the live view is displayed, the light flux is directed to the image sensor 112. It jumps upward and escapes from the optical path so that Further, the central portion of the main mirror 105 is a half mirror that transmits a small amount of light, and a part of the light is transmitted and led to a focus detection sensor (not shown) for performing focus detection. The amount of defocus of the photographing lens 101 is obtained by calculating the output of the focus detection sensor. The microcomputer 123 evaluates the calculation result and instructs the AF drive circuit 102 to drive the focus lens.

The main mirror 5 is driven up and down via the mirror drive circuit 107 according to the instruction of the microcomputer 123. A sub-mirror 106 is arranged behind the main mirror to reflect the light flux transmitted through the main mirror 105 and guide it to the above-mentioned focus detection sensor. The luminous flux transmitted through the central portion of the main mirror 105 and reflected by the sub mirror 106 also enters the exposure amount calculation circuit 109 and reaches the photometric sensor arranged inside the circuit for performing photoelectric conversion.

A pentaprism constituting a finder is arranged above the main mirror 105. The viewfinder is also composed of a focus plate, an eyepiece lens (not shown), and the like.
The focal plane shutter 110 that opens and closes the optical path of the photographing lens 101 is driven by the shutter drive circuit 111. The opening time of the focal plane shutter 110 is controlled by the microcomputer 123.

An image sensor 112 is arranged behind the focal plane shutter 110. A CCD, CMOS sensor, or the like is used for the image pickup element 112, and the subject image formed by the photographing lens 101 is converted into an electric signal. The output of the image sensor 112 is input to the A / D converter 115. The A / D converter 115 converts the analog output signal of the image sensor 112 into a digital signal.

The video signal processing circuit 116 is realized by a logic device such as a gate array. The video signal processing circuit 116 includes a luminance adjustment circuit 116a, a gamma correction circuit 116b, a blur amount calculation circuit 116c, an alignment circuit 116d, a geometric conversion circuit 116e, and a scaling circuit 116f. The video signal processing circuit 116 further includes a trimming circuit 116e, a synthesis circuit 116j, a development circuit 116k, and a compression / decompression circuit 116l.

The brightness adjustment circuit 116a adjusts the brightness by a digital gain. The gamma correction circuit 116b adjusts the luminance according to the gamma characteristic. The blur amount calculation circuit 116c calculates the blur amount of a plurality of images. The alignment circuit 116d aligns a plurality of images according to the amount of blurring of the images. The geometric conversion circuit 116e corrects the distortion of the photographing lens 101. The scaling circuit 116f scales the size of the image. The trimming circuit 116e cuts out a part of the image. The synthesis circuit 116j synthesizes a plurality of images. The developing circuit 116k develops image data. The compression / decompression circuit 116l converts the image data into a general image format such as JPEG.

The video signal processing circuit 116 is connected to a display drive circuit 117, a display member 118 using a TFT, an organic EL, or the like, a memory controller 119, a memory 120, an external interface 121 that can be connected to a computer, and a buffer memory 122. ..

The video signal processing circuit 116 performs filter processing, color conversion processing, and gamma processing on the digitized image data, and also performs compression processing such as JPEG, and outputs the digitized image data to the memory controller 119. At that time, it is also possible to temporarily place the image in the process of processing in the buffer memory 122.

The video signal processing circuit 116 can also output the video signal from the image pickup element 112 and the image data reversely input from the memory controller 119 to the display member 118 via the display drive circuit 117. Switching between these functions is performed according to the instructions of the microcomputer 123.

The video signal processing circuit 116 can output information such as exposure information and white balance of the signal of the image pickup element 112 to the microcomputer 123 as needed. Based on the information, the microcomputer 123 gives instructions for white balance adjustment and gain adjustment.

In the case of continuous shooting operation, the shooting data is temporarily stored in the buffer memory 122 without being processed, the unprocessed image data is read out via the memory controller 119, and the image processing and compression processing are performed by the video signal processing circuit 116. And perform continuous shooting. The number of continuous image shots depends on the capacity of the buffer memory 122 and the size of the image in the case of panoramic shots. In the memory controller 119, the unprocessed digital image data input from the video signal processing circuit 116 is stored in the buffer memory 122, and the processed digital image data is stored in the memory 120. On the contrary, it is also possible to output the image data from the buffer memory 122 or the memory 120 to the video signal processing circuit unit 116. The memory 120 may be removable. The memory controller 119 can output an image stored in the memory 120 to the outside via an external interface 121 that can be connected to a computer or the like.

The operation member 124 conveys the state to the microcomputer 123, and the microcomputer 123 controls each part according to the change of the operation member. The switch SW1 (125) and the switch SW2 (126) are switches that are turned on and off by operating the release button, and are one of the input switches of the operation member 124, respectively.

When only the switch SW1 (125) is on, the release button is half-pressed, and in this state, the autofocus operation is performed and the photometric operation is performed. When both the switches SW1 (125) and SW2 (126) are on, the release button is fully pressed, and the release switch for recording an image is on. Shooting is performed in this state. Further, while the switches SW1 (125) and SW2 (126) are continuously turned on, the continuous shooting operation is performed.

The operation member 124 also has an ISO setting button, a menu button, a set button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, a move + (plus) button for moving a menu or a reproduced image, and a move. -(Minus) button, exposure compensation button, display image enlargement button, display image reduction button, playback switch, aperture button to narrow down the aperture 103 to the set aperture, erase button to erase the shot image, shooting A switch (not shown) such as an information display button related to playback or playback is connected, and the state of the switch is detected. Further, by assigning the above-mentioned plus button and minus button functions to the rotary dial switch, it becomes possible to select numerical values and functions more easily.

The liquid crystal drive circuit 127 causes the external liquid crystal display member 128 and the liquid crystal display member 129 in the finder to display an operation state display or a message using characters and images according to a display command of the microcomputer 123. Further, a backlight such as an LED (not shown) is arranged on the liquid crystal display member 129 in the finder, and the LED is also driven by the liquid crystal drive circuit 127.

The microcomputer 123 confirms the capacity of the memory via the memory controller 119 based on the predicted value data of the image size according to the ISO sensitivity, the image size, and the image quality set before shooting, and then the remaining number of shots that can be taken. Can be calculated. This information can also be displayed on the external liquid crystal display member 128 and the liquid crystal display member 129 in the finder, if necessary.

The non-volatile memory (EEPROM) 130 can store data even when the camera is not turned on. The power supply unit 131 supplies the necessary power to each IC and drive system. The internal clock 132 measures the elapsed time, saves the shooting time or the like in an image file recorded in the memory 120, and can superimpose the shooting time on the image itself as described later. The gyro 133 detects the angular velocity of rotation of the image pickup apparatus 100 on two or three axes. The azimuth meter 134 detects the direction in which the image pickup device is facing.

Next, the operation of the image pickup apparatus configured as described above will be described. FIG. 3 is a flowchart showing a shooting operation of the image pickup apparatus of the first embodiment.

First, before starting the shooting operation, the exposure amount is calculated in advance by the exposure amount calculation circuit 109, and the aperture amount, the accumulation time, and the ISO sensitivity are determined. When the switch SW2 (126) is pressed by the user, the shooting operation is performed.

In S401, the microcomputer 123 notifies the aperture drive circuit 104 of a predetermined aperture value, and adjusts the aperture 103 to a target aperture value. The image sensor 112, the AD converter 115, and the like are turned on to prepare for shooting. When the preparation is completed, the mirror drive circuit 107 is driven to flip up the main mirror 5 so that the subject image is incident on the image sensor 112. The shutter drive circuit opens a front curtain (not shown) of the focal plane shutter 110 so that the subject image is incident on the image pickup device 112. Subsequently, after a predetermined storage time, the rear curtain (not shown) of the shutter 110 is closed so that light enters the image sensor 112 only during the storage time. Exposure is performed by performing this series of operations.

In S402, the image signal is read into the video signal processing circuit 116 via the AD converter 115 and stored in the buffer memory 122. In S403, the read image signal is developed by the developing circuit 116k and converted into image data. At this time, image processing such as white balance processing or gamma processing for applying a gain to a dark portion by the gamma correction circuit 116b may be performed so that the image has an appropriate image quality.

In S404, the obtained image data is converted into a general-purpose data format such as JPEG by the compression / decompression circuit 116l. In S405, the converted image data is stored in a memory 120 such as an SD card or a compact flash (registered trademark). This completes the shooting operation.

In S403, the read image signal as it is may be losslessly compressed in S404 and stored in a storage medium in S405 without performing image processing or development processing. These switching can be switched by the user using the operation member 124.

Hereinafter, the starry sky panoramic shooting mode will be described. Starry sky panorama shooting has a mode of shooting while shifting the image pickup device in the horizontal direction and a mode of shooting while shifting the image pickup device in the vertical direction. Here, an example of shooting while shifting the image pickup device in the horizontal direction will be described.

When the user sets the starry sky panorama shooting mode using the operation member 124, power is supplied to the image sensor 112 and the AD converter 115, and the initial setting is performed. On the other hand, the main mirror 105 is flipped up, the shutter drive circuit 111 opens the shutter 110, and the subject image via the photographing lens 101 is incident on the image sensor 112.

The signal from the image pickup element 112 is converted into a digital signal by the AD converter 115, developed by the development circuit 116k of the video signal processing circuit 116, and converted into an appropriate image by the brightness adjustment circuit 116a and the gamma correction circuit 116b. After that, this image data is scaled to an image size suitable for the display member 118 by the scaling circuit 116f and displayed. By repeating this 24 to 60 times per second, so-called live view display is performed.

While checking the live view display, the user adjusts the shooting direction and angle of view and presses the switch SW1 (125). When the switch SW1 (125) is pressed, the exposure amount is calculated. In the case of not live view shooting, the light reflected by the sub mirror 106 is received by the exposure amount calculation circuit 109, and an appropriate exposure amount is calculated. During live view shooting, an appropriate exposure amount is determined by an exposure amount calculation circuit (not shown) included in the video signal processing circuit 116. Then, the microcomputer 123 drives the diaphragm 103 by using the diaphragm drive circuit 104, and controls the sensitivity and the accumulation time of the image pickup device 112. In starry sky panoramic photography, a program diagram is used so that the exposure time is such that the stars do not form a locus. On the other hand, the AF drive circuit 102 drives the photographing lens 101 to focus. When the preparation for shooting is completed, the user is notified by a buzzer (not shown). When the user presses the switch SW2 (126) in the direction in which he / she wants to start shooting the image pickup device, starry sky panoramic shooting is started.

Next, the starry sky panoramic photography will be described in detail using the flowchart of FIG. 4 and the data flow diagram of FIG.

When the starry sky panoramic photography is started, the microcomputer 123 first acquires the lens information in S501. This lens information includes data for correcting distortion and a decrease in the amount of light in the peripheral portion of the lens, which will be described later.

In S502, the first long-second exposure image is taken. Since the image sensor 112 and the AD converter 115 are set to drive for live view, they are switched to for still image shooting. The aperture 103 is adjusted to have the exposure amount determined earlier, and the focal plane shutter 110 is opened and closed to expose the image sensor 112. The image signal obtained by the image pickup element 112 is converted into a digital signal by the AD converter 115 and stored in the buffer memory 122. The image data is subjected to processing such as shading correction by a circuit (not shown) included in the image processing circuit 116. Image data that has undergone minimal processing in this way is called RAW image data 601. Next, this RAW image data 601 is developed by the developing circuit 116k to obtain YUV image data 602.

Next, in S503, the first high-sensitivity short-second exposure image is taken to obtain a short-second exposure image. Let A be the high-sensitivity short-second exposure shooting immediately before the long-second exposure shooting, and B be the high-sensitivity short-second exposure shooting immediately after the long-second exposure shooting. High-sensitivity short-second exposure photography is used only for calculating the amount of movement, and is not used as a panoramic composite image. What is performed in S503 is high-sensitivity short-second exposure shooting B. Since the stars become smaller and thinner in short-second exposure photography, the ISO sensitivity is increased before shooting. Similar to the long exposure photography, the RAW image data 604 is developed by the developing circuit 116k to obtain the YUV image data 605.

In S504, the gyro 133 is first initialized at the time of the first image in order to be able to acquire how much the image pickup device 100 has been swung (rotated) by the second image in the later S509.

In S505, the distortion aberration of the photographing lens 101 is corrected by the geometric conversion circuit 116e for the image data 602,605 after development of each of the long-time exposure photography and the short-time exposure photography, and the image after the distortion correction is performed. Data 603,606 is obtained. The image data 603 after distortion correction by long-second exposure is reduced according to the number of pixels of the liquid crystal monitor by the scaling circuit 116f and stored in the VRAM 611 in order to be displayed on the display member 118.

Next, in S506, the second high-sensitivity short-second exposure shooting A is performed to obtain image data 608. In S507, a second long-second exposure shot is taken to obtain image data 613. Further, in S508, the second high-sensitivity short-second exposure shooting B is performed to obtain image data 616. Similar to S502, each RAW image data 608, 613, 616 is developed by the developing circuit 116k to obtain YUV image data 609, 614, 617.

In S509, in order to acquire the swing amount of the image pickup apparatus 100 from the previous shooting, the gyro information which is the detection value of the gyro 133 is acquired. For the gyro information, the values in the two-axis directions of the Yaw direction and the Pitch direction of the image pickup device are acquired, but it is also preferable to acquire the values in the three-axis direction including the Roll direction which is the rotation with respect to the optical axis. .. The output itself from the gyro 133 is the angular velocity, but in panoramic shooting it is necessary how much it has been swung from the previous shooting, so the angular velocity from the previous shooting to the next shooting is integrated, and when the second and subsequent shots are taken from the previous time. The rotation angle 607 is calculated and stored.

In S510, this rotation angle 607 is converted into pixel units from the focal length and angle of view of the lens acquired in S501, information on the image sensor, and the like.

Generally, the lens without distortion or the angle of view (α) after distortion correction is calculated by the following equation 1 assuming that the effective focal length is f [mm] and the width of the image sensor is w [mm].

α [°] = 2 × arctan (w [mm] ÷ 2 ÷ f [mm]) (Equation 1)
Assuming that the size of the image sensor per pixel is p [μm] and the swing angle [°] is θ, the movement amount d [pix] in the image is as shown in Equation 2.

d [pix] = tan (α [°] ÷ 2) × f [mm] / p [μm] × 1000 (Equation 2)
In S511, the distortion aberration of each image of the second image is corrected in the same manner as the distortion aberration correction (S505) of the first image, and the image data 610, 615, 618 after the distortion correction is obtained. Similar to the first image, the long-time exposure photographed image data 615 after distortion correction is displayed on the display member 118, so that it is reduced according to the number of pixels of the liquid crystal monitor by the scaling circuit 116f and stored in the VRAM 619.

In S512, the image data 606 obtained in the first high-sensitivity short-second exposure shooting B and the image data 610 obtained in the second high-sensitivity short-second exposure shooting A are used by using the movement amount calculation circuit 116c. The amount of movement is calculated from. As described above, a known method can be used for the movement amount detection, but in the present embodiment, the affine coefficient 612 is calculated by finding some feature points in the image by the movement amount detection circuit 116c and sampling them.

Specifically, an edge is detected, feature points are extracted, and the amount of movement thereof is calculated. For example, here, the feature point 1 is from the coordinates (x1, y1) to the coordinates (u1, v1), the feature point 2 is from the coordinates (x2, y2) to (u2, v2), and the feature point 3 is the coordinates (x3, y3). It is assumed that the coordinates deviate from (u3, v3). Then, when simultaneous equations are created from Equation 1, Equations 3 and 4 are obtained.

By solving this equation, it is possible to calculate the affine coefficients a to f. If 4 or more feature points can be detected, normalize using the least squares method by excluding close points. If three points are not found, if the extracted three points are linear, or if two of the three points are close to each other, it is determined that the calculation of the movement amount has failed.

If the movement amount (affin coefficient) calculated from the image in this way and the movement amount based on the rotation angle 607 calculated from the detection value of the gyro sensor in S510 are significantly different, the image includes a repeating pattern or a moving body. It is possible that Therefore, it is conceivable to calculate the movement amount again under another condition, make this shot a failure, return to the next shooting (S506), or issue a warning that the starry sky panoramic shooting has failed.

In S513, based on the movement amount (affin coefficient) calculated from the image, the image obtained by long-time exposure photography with S502 and S507 is aligned using the alignment circuit 116d, and the alignment image data 621 is used. To get.

In S514, the first image data 620 and the image data 621 after the alignment of the second image are combined by using the synthesis circuit 116j to obtain the composite image data 622. When processing the Nth sheet (N> 2), the composition results so far, that is, the composite image data 620 up to the (N-1) th sheet and the image data 621 after the alignment of the Nth sheet are combined. do.

In S515, if the switch SW2 (126) is pressed, the process returns to the next shooting S506, and if the switch SW2 (126) is not pressed, the process of S516 is performed. In S516, the image data is compressed into a general-purpose format such as Jpeg using the compression / decompression circuit 116l, and stored in the memory 120 in S517.

At this time, it is also preferable to perform gamma correction by the gamma correction circuit 116b in order to make the dark part of the composite image easier to see, or to perform correction in order to unify the color tone of the entire image. Further, since the image obtained in this way has a large size, scaling may be performed by the scaling circuit 116f to a size specified in advance by the user. Further, it is also preferable to cut out the maximum inscribed rectangle or a predetermined region with the trimming circuit 116 g and then store the region.

In the above description, an example of capturing a plurality of images while shifting the image pickup device in the horizontal direction is shown, but the same method can be used when the image pickup device is shifted in the vertical direction.

As described above, even if the star as the subject moves between images in different shooting directions, it is possible to perform correct alignment and shoot a high-quality panoramic composite image without increasing the sensitivity.

(Second embodiment)
In this embodiment, an example will be described in which high-sensitivity short-second exposure photography for calculating the amount of movement is not necessary depending on the imaging conditions and environment of starry sky panoramic photography. FIG. 6 is a flowchart showing the operation of panoramic photography in the second embodiment.

The processing of S701 to S702 after the start of starry sky panoramic photography is the processing of S501 to S502 of the first embodiment, the processing of S704 to S706 is the processing of S503 to S505, and the processing of S708 to S709 is S506 to S507. Since the processes of S711 to S720 correspond to the processes of S508 to S517, the description of these processes will be omitted.

In S703, it is determined whether or not it is necessary to perform the high-sensitivity short-second exposure shooting B following the first long-second exposure shooting (S702). The determination of implementation is made, for example, as follows. First, the microcomputer 123 acquires the direction of the image pickup device detected by the azimuth meter 134, and calculates the amount of movement of the star during shooting. If it is determined that there is no movement of the star, the processing of S705 is performed without performing the high-sensitivity short-second exposure shooting B of S704. The accumulation time setting or the like may be used to determine the execution of the high-sensitivity short-second exposure shooting B.

Whether or not it is necessary to perform high-sensitivity short-second exposure photography A and B before and after the long-second exposure photography (S709) for the second and subsequent images is determined in S707 and S710 by the same processing as in S703. If it is determined that there is no movement of the star, the processes of S709 and S712 are performed without performing the high-sensitivity short-second exposure photography A of S708 and the high-sensitivity short-second exposure photography B of S711.

As explained above, by determining the shooting conditions and shooting environment for starry sky panoramic shooting, it is possible to reduce the power consumption related to shooting by omitting unnecessary high-sensitivity short-second exposure shooting for alignment. Become. In the present embodiment, the case where the implementation judgment is made before all the high-sensitivity short-second exposure shootings has been described, but when it is possible to judge that all the high-sensitivity short-second exposure shootings are unnecessary in the standby state of the image pickup apparatus. May be configured so that the determination process is performed only once.

(Third embodiment)
In the present embodiment, an example of displaying an appropriate warning to the user when it is considered that the alignment fails in the starry sky panoramic photography will be described with reference to FIGS. 4 and 7.

When the high-sensitivity short-second photography B of S504 and the high-sensitivity short-second photography A of S506 are performed, the microcomputer 123 acquires and stores the time from the clock 132. The microcomputer 123 calculates the interval between the two shooting times, determines that the star has moved too much when the interval is longer than a certain time, and displays the warning screen of FIG. 8A.

Further, the microcomputer 123 detects the swing amount of the image pickup device from the rotation angle 607 which is the gyro information acquired in S509, and if it exceeds a certain swing amount, it is determined that the alignment cannot be performed, and FIG. 8B is shown. Warning screen is displayed.

As described above, it is possible to improve the convenience of the user by displaying a warning in advance in a scene where alignment is expected to be impossible in starry sky panoramic photography.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Image pickup device, 101: Shooting lens, 112: Image pickup element, 116: Video signal processing circuit, 119: Memory controller, 123: Microcomputer, 133: Gyro, 134: Direction meter

Claims (18)

An image pickup device capable of generating one composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common area.
An imaging means for capturing a subject image and
The imaging means is made to image the plurality of unit images, and before and after the imaging of each unit image of the plurality of unit images, a short-second exposure having a shorter exposure time than the exposure time of the unit image is performed. A control means for capturing an image and
The first short exposure image taken after the first unit image was taken and the second short shot taken before the second short shot taken following the first unit image. A calculation means for calculating the amount of movement of the subject between the first short-exposure image and the second short-exposure image using the second-exposure image, and
Based on the movement amount of the subject calculated by the calculation means, the first unit image and the second unit image are aligned, and the first unit image and the second unit image are combined. Synthetic means and
An image pickup device characterized by being provided with. The image pickup apparatus according to claim 1, wherein the subject is a starry sky, and the unit image is an image taken by long-second exposure for taking a picture of the starry sky. The imaging device according to claim 2, wherein the short-second exposure image is an image captured by increasing the sensitivity of the imaging means in order to capture a starry sky with a short-second exposure. When the first determination means for determining whether or not it is necessary to capture the short exposure image is further provided, and it is determined by the first determination means that the imaging of the short exposure image is not necessary. The image pickup apparatus according to any one of claims 1 to 3, wherein the control means omits the acquisition of the short-time exposure image. The first determination means determines that it is not necessary to capture the short exposure image when there is no movement of the subject between the first short exposure image and the second short exposure image. The image pickup apparatus according to claim 4, wherein the image pickup apparatus is characterized by the above. The first unit image and the second unit image are determined by the second determination means for determining whether or not the alignment of the first unit image and the second unit image is possible, and the second determination means. The image pickup apparatus according to any one of claims 1 to 5, further comprising a warning means for issuing a warning when it is determined that alignment of a unit image is impossible. In the second determination means, when the amount of movement of the subject between the first short-second exposure image and the second short-second exposure image calculated by the calculation means exceeds a predetermined amount. The image pickup apparatus according to claim 6, wherein it is determined that the alignment of the first unit image and the second unit image is impossible. Further provided with a time measuring means for measuring the elapsed time between the acquisition of the first short-time exposure image and the acquisition of the second short-time exposure image, the second determination means has a predetermined elapsed time. The image pickup apparatus according to claim 6, wherein it is determined that the alignment of the first unit image and the second unit image is impossible when the time is exceeded. An image processing device capable of generating a single composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common area.
Acquires the plurality of unit images and short-second exposure images captured with an exposure time shorter than the exposure time of the unit images before and after imaging of each of the plurality of unit images. Acquisition method and
The first short exposure image taken after the first unit image was taken and the second short shot taken before the second short shot taken following the first unit image. A calculation means for calculating the amount of movement of the subject between the first short-exposure image and the second short-exposure image using the second-exposure image, and
Based on the movement amount of the subject calculated by the calculation means, the first unit image and the second unit image are aligned, and the first unit image and the second unit image are combined. Synthetic means and
An image processing device characterized by comprising. The image processing apparatus according to claim 9, wherein the subject is a starry sky, and the unit image is an image taken with a long exposure for photographing the starry sky. The image processing apparatus according to claim 10, wherein the short-second exposure image is an image taken with increased sensitivity of an imaging means in order to take a short-second exposure of a starry sky. The first unit image and the second unit image are determined by the second determination means for determining whether or not the alignment of the first unit image and the second unit image is possible, and the second determination means. The image processing apparatus according to any one of claims 9 to 11, further comprising a warning means for issuing a warning when it is determined that alignment of a unit image is impossible. In the second determination means, when the amount of movement of the subject between the first short-second exposed image and the second short-second exposed image calculated by the calculation means exceeds a predetermined amount. The image processing apparatus according to claim 12, wherein it is determined that the alignment of the first unit image and the second unit image is impossible. The acquisition means further acquires information on the elapsed time between the acquisition of the first short-second exposed image and the acquisition of the second short-second exposed image, and the second determination means obtains the elapsed time. The image processing apparatus according to claim 12, wherein when the image exceeds a predetermined time, it is determined that the alignment of the first unit image and the second unit image is impossible. It is a method of controlling an imaging device capable of generating one composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common area.
The imaging means for capturing the subject image captures the plurality of unit images, and the exposure time is longer than the exposure time of the unit image before and after the imaging of each unit image of the plurality of unit images. A control process for capturing a short short-second exposure image and
The first short exposure image taken after the first unit image was taken and the second short shot taken before the second short shot taken following the first unit image. A calculation step of calculating the amount of movement of the subject between the first short-exposure image and the second short-exposure image using the second-exposure image, and
Based on the movement amount of the subject calculated in the calculation step, the first unit image and the second unit image are aligned, and the first unit image and the second unit image are combined. And the synthesis process
A method for controlling an image pickup apparatus, which comprises. It is a method of controlling an image processing device capable of generating one composite image by synthesizing a plurality of unit images having different imaging directions and having a partially common region.
Acquires the plurality of unit images and short-second exposure images captured with an exposure time shorter than the exposure time of the unit images before and after imaging of each of the plurality of unit images. Acquisition process and
The first short exposure image taken after the first unit image was taken and the second short shot taken before the second short shot taken following the first unit image. A calculation step of calculating the amount of movement of the subject between the first short-exposure image and the second short-exposure image using the second-exposure image, and
Based on the movement amount of the subject calculated in this calculation, the first unit image and the second unit image are aligned, and the first unit image and the second unit image are combined. The synthesis process to synthesize and
A method for controlling an image processing apparatus, which comprises. A program for causing a computer to execute each step of the control method according to claim 15. A computer-readable storage medium that stores a program for causing a computer to execute each step of the control method according to claim 15.

Images