JP2020028079A - Imaging apparatus, image processing apparatus, control method of the same, program, and storage medium - Google Patents

Abstract

To provide an imaging apparatus capable of capturing a high-quality panoramic image even if a subject star moves between images with different shooting directions.SOLUTION: The imaging apparatus capable of generating one synthesized image by synthesizing a plurality of unit images having different imaging directions and having a partially common area, includes: an imaging unit for capturing a subject image; a control unit for causing the imaging unit to capture a plurality of unit images and a short-second exposure image whose exposure time is shorter than exposure time of the unit image before and after capturing each unit image of the plurality of unit images; a calculating unit for calculating a moving amount of the subject between a first short-second exposure image and a second short-second exposure image by using the first short-second exposure image captured after capturing a first unit image and the second short-time exposure image captured before capturing a second unit image captured after the first unit image; and a synthesizing unit for aligning the first unit image and the second unit image on the basis of the movement amount of the subject calculated by the calculation unit to synthesize the first unit image and the second unit image.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technique for obtaining a wider-range starry sky image, that is, a starry sky panoramic image, by synthesizing continuously shot images while sequentially changing the shooting direction so that a common region is captured.

  2. Description of the Related Art A panoramic imaging method is known as one of imaging methods for storing a vast starry sky as one image. As an example of this technique, Patent Document 1 discloses the following imaging method. That is, a plurality of unit images constituting a part of the photographing range are photographed by sequentially changing the photographing direction, and image regions of a predetermined size are cut out from the photographed unit images so that overlapping regions are generated with each other. Are sequentially superimposed to generate a panoramic image.

  Here, a problem in a case where this method is applied to panoramic shooting of a starry sky will be described. When photographing a starry sky, a long exposure such as 30 seconds or 1 minute is often performed because the amount of light of the star is very small. As the celestial body makes a diurnal motion in accordance with the rotation of the earth, long exposure will result in a star trail instead of a point image.

  When taking a panoramic image of a starry sky, it is necessary to generate panoramic images by photographing images having different photographing directions with an exposure time that does not cause a star to form a light trail, and connecting the images. However, there are cases where it is desired to perform panoramic shooting of the starry sky with long exposure. In this case, since the stars move between the images with the passage of time, alignment between the images tends to fail. FIG. 8 shows an example of a failure in alignment when two images having different shooting directions are combined in panoramic shooting of a starry sky. FIG. 8A shows a first captured image, and 301 indicates a background. FIG. 8B illustrates a second captured image, and 302 indicates a background. With the lapse of time including the time of the first long-time exposure shooting and the time for the user to set the imaging device to change the shooting direction, the position of the star in the second image is changed to the position of the first image. Moving relative to the position of the star in the image. FIG. 8C shows that when the first and second shot images are aligned, the background 301 and 302 are shifted by the amount of the movement of the stars because the stars are used as a reference for alignment. The state is shown.

  On the other hand, Patent Literature 2 discloses an optical shake correction unit that corrects a position shift on an imaging surface due to a diurnal motion of a celestial body, and repeats imaging and synthesis to prevent alignment failure while preventing misalignment. A technique for obtaining a captured image with a wide angle of view has been disclosed.

JP 2005-328497 A JP-A-2006-005160

  However, in the related art disclosed in Patent Document 2, since an optical shake correction unit is used, the maximum value of the change in the shooting direction is limited, and an astronomical image with a wider angle of view is shot. There is a problem that can not be.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an imaging apparatus capable of capturing a high-quality panoramic image even when a star serving as a subject moves between images having different shooting directions. It is to provide.

  An imaging apparatus according to the present invention is an imaging apparatus that can generate a single composite image by synthesizing a plurality of unit images having different imaging directions and partially common areas, and captures a subject image. And an imaging unit that causes the imaging unit to capture the plurality of unit images, and before and after capturing each unit image of the plurality of unit images, the exposure time is shorter than the exposure time of the unit image. Control means for capturing a short short-time exposure image, a first short-time exposure image captured after capturing the first unit image, and a second unit captured subsequent to the first unit image Calculation for calculating the amount of movement of a subject between the first short-time exposure image and the second short-time exposure image using a second short-time exposure image captured before capturing the image Means, and the subject calculated by the calculation means A synthesizing unit that aligns the first unit image and the second unit image based on a movement amount, and synthesizes the first unit image and the second unit image. And

  The image processing apparatus according to the present invention is an image processing apparatus capable of generating one composite image by combining a plurality of unit images having different imaging directions and having a partially common area, Obtaining the plurality of unit images and a short-second exposure image captured with an exposure time shorter than the exposure time of the unit image before and after capturing each unit image of the plurality of unit images. Acquiring means, a first short-time exposure image captured after capturing the first unit image, and an image captured before capturing a second unit image captured subsequent to the first unit image. Calculating means for calculating the amount of movement of the subject between the first short-time exposure image and the second short-time exposure image using the second short-time exposure image The first unit image based on the determined moving amount of the subject Said alignment was carried out of the second unit image, characterized in that it comprises a synthesizing means for synthesizing said second unit image and the first unit image.

  According to the present invention, it is possible to provide an imaging apparatus capable of capturing a high-quality panoramic image even when a star serving as a subject moves between images having different shooting directions.

FIG. 3 is a conceptual diagram when a plurality of shot images are synthesized in a panorama. FIG. 1 is a block diagram illustrating a configuration of a first embodiment of an imaging device according to the present invention. 5 is a flowchart illustrating a normal photographing operation. 5 is a flowchart illustrating an operation of starry sky panoramic imaging according to the first embodiment. 4 is a data flow diagram illustrating the operation of the first embodiment. 9 is a flowchart showing the operation of starry sky panoramic imaging according to the second embodiment. FIG. 13 is a conceptual diagram of a warning screen according to the third embodiment. The conceptual diagram explaining the subject of a starry sky panorama synthesis.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a diagram showing an outline of panoramic shooting. In the present embodiment, panoramic imaging is performed by sequentially changing the imaging direction while the user 202 moves the imaging device 201 or moving the imaging device 201 with an automatic camera platform or the like as shown in FIG. . As shown in FIG. 1B, a plurality of images are photographed so that each photographed image partially includes the common region of the subject. Then, feature points in the common area of each image are extracted, and a movement vector indicating how much the feature points have moved is detected. For example, an affine transformation coefficient is calculated from the movement vector, and the two images are overlapped so that the feature points coincide with each other, thereby obtaining an image in which a portion other than the common area is extended. By repeating this a plurality of times, it becomes possible to generate a panoramic image having a wider angle of view than the angle of view obtained by one shooting.

  FIG. 2 is a block diagram illustrating a configuration of the first embodiment of the imaging apparatus of the present invention. In FIG. 2, the imaging apparatus 100 includes a photographing lens 101 for forming a subject image, and an AF (autofocus) driving circuit 102 for adjusting the focus of the photographing lens 101. The AF drive circuit 102 is constituted by, for example, a DC motor or a stepping motor, and performs focus adjustment by changing the position of the focus lens of the photographing lens 101 under the control of the microcomputer 123.

  The taking lens 101 includes an aperture 103, and the aperture 103 is driven by an aperture driving circuit 104. The optical diaphragm value is calculated by the microcomputer 123, and the amount by which the diaphragm driving circuit 104 drives the diaphragm 103 is determined based on the calculated diaphragm value.

  Behind the stop 103, a main mirror 105 is arranged. The main mirror 105 switches between a state in which the light beam passing through the photographing lens 101 is guided to the finder and a state in which the light beam is guided to the image sensor 112. The main mirror 105 is arranged at a position where the light flux is reflected upward so that the light flux always goes to the finder. However, when the photographing is performed or the live view display is performed, the light flux is transmitted to the image sensor 112. Bounce upward so as to move away from the optical path. In addition, the main mirror 105 is a half mirror that transmits a small amount of light at the center thereof, transmits a part of the light, and guides it to a focus detection sensor (not shown) for performing focus detection. The defocus amount of the photographing lens 101 is obtained by calculating the output of the focus detection sensor. The microcomputer 123 evaluates the calculation result and instructs the AF drive circuit 102 to drive the focus lens.

  The main mirror 5 is driven up and down through a mirror driving circuit 107 according to an instruction from the microcomputer 123. A sub-mirror 106 is arranged behind the main mirror, reflects the light beam transmitted through the main mirror 105, and guides the light beam to the focus detection sensor. The light beam transmitted through the central portion of the main mirror 105 and reflected by the sub-mirror 106 also enters the exposure amount calculation circuit 109, and reaches a photometric sensor disposed therein for performing photoelectric conversion.

Above the main mirror 105, a pentaprism constituting a finder is arranged. The finder is configured by a focus plate, an eyepiece lens (not shown), and the like.
A focal plane shutter 110 that opens and closes the optical path of the taking lens 101 is driven by a shutter drive circuit 111. The opening time of the focal plane shutter 110 is controlled by the microcomputer 123.

  An image sensor 112 is arranged behind the focal plane shutter 110. A CCD, a CMOS sensor, or the like is used for the image sensor 112, and converts an object image formed by the imaging lens 101 into an electric signal. The output of the image sensor 112 is input to the A / D converter 115. The A / D converter 115 converts an analog output signal of the image sensor 112 into a digital signal.

  The video signal processing circuit 116 is realized by a logic device such as a gate array. The video signal processing circuit 116 has a luminance adjustment circuit 116a, a gamma correction circuit 116b, a blur amount calculation circuit 116c, a positioning circuit 116d, a geometric conversion circuit 116e, and a scaling circuit 116f. The video signal processing circuit 116 further includes a trimming circuit 116e, a synthesizing circuit 116j, a developing circuit 116k, and a compression / decompression circuit 116l.

  The brightness adjustment circuit 116a adjusts brightness by a digital gain. The gamma correction circuit 116b adjusts luminance according to gamma characteristics. The blur amount calculation circuit 116c calculates the blur amounts of a plurality of images. The positioning circuit 116d performs positioning of a plurality of images in accordance with the amount of image blur. The geometric transformation circuit 116e corrects the distortion of the taking lens 101. The scaling circuit f changes the size of the image. The trimming circuit 116e cuts out a part of the image. The combining circuit 116j combines a plurality of images. The development circuit 116k develops image data. The compression / expansion circuit 116l converts the image data into a general image format such as Jpeg.

  To the video signal processing circuit 116, a display drive circuit 117, a display member 118 using a TFT or an organic EL, a memory controller 119, a memory 120, an external interface 121 connectable to a computer or the like, and a buffer memory 122 are connected. .

  The video signal processing circuit 116 performs filtering, color conversion, and gamma processing on the digitized image data, performs compression processing such as JPEG, and outputs the result to the memory controller 119. At this time, it is also possible to temporarily store the image being processed in the buffer memory 122.

  The video signal processing circuit 116 can also output a video signal from the image sensor 112 and image data reversely input from the memory controller 119 to the display member 118 via the display drive circuit 117. Switching of these functions is performed according to an instruction from the microcomputer 123.

  The video signal processing circuit 116 can output information such as exposure information of a signal of the image sensor 112 and white balance to the microcomputer 123 as needed. Based on the information, the microcomputer 123 issues white balance adjustment and gain adjustment instructions.

  In the case of the continuous shooting operation, the shooting data is temporarily stored in the buffer memory 122 without being processed, the unprocessed image data is read out via the memory controller 119, and the video signal processing circuit 116 performs image processing and compression processing. Perform continuous shooting. The number of continuous image shots depends on the capacity of the buffer memory 122 and the size of the image in the case of panoramic shooting. In the memory controller 119, the unprocessed digital image data input from the video signal processing circuit 116 is stored in the buffer memory 122, and the processed digital image data is stored in the memory 120. Conversely, it is also possible to output image data from the buffer memory 122 or the memory 120 to the video signal processing circuit unit 116. The memory 120 may be removable. Note that the memory controller 119 can output an image stored in the memory 120 to the outside via an external interface 121 connectable to a computer or the like.

  The operation member 124 informs the microcomputer 123 of the state, and the microcomputer 123 controls each part according to the change of the operation member. The switch SW1 (125) and the switch SW2 (126) are switches that are turned on and off by operating a release button, and are each one of the input switches of the operation member 124.

  A state where only the switch SW1 (125) is ON is a state where the release button is half-pressed. In this state, the autofocus operation and the photometric operation are performed. When both the switches SW1 (125) and SW2 (126) are on, the release button is fully pressed, and the release switch for recording an image is on. Imaging is performed in this state. While the switches SW1 (125) and SW2 (126) are kept ON, a continuous shooting operation is performed.

  The operation member 124 includes an ISO setting button, a menu button, a set button, a flash setting button, a single-shot / continuous-shot / self-timer switching button, a move + (plus) button for moving a menu and a reproduced image, and a move. -(Minus) button, exposure correction button, display image enlargement button, display image reduction button, playback switch, aperture button for narrowing aperture 103 to the set aperture, erasure button for erasing a captured image, shooting A switch (not shown), such as an information display button for playback and playback, is connected, and the state of the switch is detected. Also, by assigning the functions of the plus button and the minus button to the rotary dial switch, it is possible to more easily select a numerical value and a function.

  The liquid crystal drive circuit 127 causes the external liquid crystal display member 128 and the finder liquid crystal display member 129 to display a display of an operation state using characters and images, a message, and the like according to a display command of the microcomputer 123. A backlight such as an LED (not shown) is disposed on the liquid crystal display member 129 in the finder, and the LED is also driven by the liquid crystal driving circuit 127.

  The microcomputer 123 checks the memory capacity via the memory controller 119 based on the ISO sensitivity, the image size, and the predicted value data of the image size according to the image quality set before shooting, and then checks the remaining number of shootable images. Can be calculated. This information can also be displayed on the external liquid crystal display member 128 and the finder internal liquid crystal display member 129 as necessary.

  The non-volatile memory (EEPROM) 130 can store data even when the power of the camera is not turned on. The power supply unit 131 supplies necessary power to each IC and drive system. The internal clock 132 can measure the elapsed time, store the shooting time in an image file recorded in the memory 120, and superimpose the shooting time on the image itself as described later. The gyro 133 detects the angular velocity of rotation of the imaging device 100 on two axes or three axes. The compass 134 detects the direction in which the imaging device is facing.

  Next, the operation of the imaging device configured as described above will be described. FIG. 3 is a flowchart illustrating a shooting operation of the imaging device according to the first embodiment.

  First, before the photographing operation is started, the exposure amount is calculated in advance by the exposure amount calculation circuit 109, and the aperture amount, the accumulation time, and the ISO sensitivity are determined. When the switch SW2 (126) is pressed by the user, a shooting operation is performed.

  In step S401, the microcomputer 123 notifies the aperture driving circuit 104 of a predetermined aperture value, and adjusts the aperture 103 to a target aperture value. The power is turned on to the image sensor 112, the AD converter 115, and the like, and the shooting is prepared. When the preparation is completed, the mirror driving circuit 107 is driven to flip up the main mirror 5 so that the subject image is incident on the image sensor 112. The shutter drive circuit opens the front curtain (not shown) of the focal plane shutter 110 so that the subject image enters the image sensor 112. Then, after a predetermined accumulation time, the rear curtain (not shown) of the shutter 110 is closed so that light enters the image sensor 112 only during the accumulation time. Exposure is performed by performing this series of operations.

  In S402, the image signal is read out to the video signal processing circuit 116 via the AD converter 115 and stored in the buffer memory 122. In S403, the read image signal is developed by the developing circuit 116k, and is converted into image data. At this time, image processing such as white balance processing or gamma processing for applying a gain to a dark portion by the gamma correction circuit 116b may be performed so that the image has an appropriate image quality.

  In S404, the obtained image data is converted into a general-purpose data format such as JPEG by the compression / decompression circuit 116l. In S405, the converted image data is stored in the memory 120 such as an SD card or CompactFlash (registered trademark). Thus, the photographing operation ends.

  In step S403, the read image signal may be losslessly compressed in step S404 without performing image processing or development processing, and may be stored in a storage medium in step S405. These switching can be performed by the user using the operation member 124.

  Hereinafter, the starry sky panorama shooting mode will be described. The starry sky panoramic shooting includes a mode in which the image capturing device is shifted in the horizontal direction and a mode in which the image capturing device is shifted in the vertical direction. Here, an example in which the image capturing is performed while shifting in the horizontal direction will be described.

  When the user sets the starry sky panorama shooting mode using the operation member 124, power is supplied to the image sensor 112 and the AD converter 115, and initialization is performed. On the other hand, the main mirror 105 is flipped up, the shutter drive circuit 111 opens the shutter 110, and causes the subject image via the photographing lens 101 to enter the image sensor 112.

  A signal from the image sensor 112 is converted into a digital signal by an AD converter 115, developed by a developing circuit 116k of a video signal processing circuit 116, and converted into an appropriate image by a luminance adjustment circuit 116a and a gamma correction circuit 116b. Thereafter, the image data is scaled by the scaling circuit 116f to an image size suitable for the display member 118 and displayed. By repeating this operation 24 to 60 times per second, a so-called live view display is performed.

  While checking the live view display, the user adjusts the shooting direction and the angle of view and presses the switch SW1 (125). When the switch SW1 (125) is pressed, the exposure amount is calculated. If it is not the live view shooting, the light reflected by the sub-mirror 106 is received by the exposure amount calculation circuit 109, and an appropriate exposure amount is calculated. During live view shooting, an appropriate exposure amount is determined by an exposure amount calculation circuit (not shown) included in the video signal processing circuit 116. Then, the microcomputer 123 drives the diaphragm 103 using the diaphragm driving circuit 104, and controls the sensitivity and the accumulation time of the image sensor 112. In the starry sky panoramic photography, a program diagram is used such that the exposure time is such that the stars do not form a trajectory. On the other hand, the AF drive circuit 102 drives the photographing lens 101 to focus. When the preparation for shooting is completed, the user is notified by a buzzer (not shown) or the like. When the user presses the switch SW2 (126) in a direction in which the user wants to start taking an image, the starry sky panoramic photographing is started.

  Next, the starry sky panoramic shooting will be described in detail with reference to the flowchart of FIG. 4 and the data flow diagram of FIG.

  When the starry sky panorama shooting is started, the microcomputer 123 first obtains lens information in S501. The lens information includes data for correcting distortion, which will be described later, and a decrease in the amount of light around the lens.

  In S502, the first long-time exposure shooting is performed. Since the image sensor 112 and the AD converter 115 are set to drive for live view, they are switched to those for still image shooting. The aperture 103 is adjusted so as to have the previously determined exposure amount, the focal plane shutter 110 is opened and closed, and the image sensor 112 is exposed. The image signal obtained by the image sensor 112 is converted into a digital signal by the AD converter 115 and stored in the buffer memory 122. This image data is subjected to processing such as shading correction by a circuit (not shown) included in the image processing circuit 116. The image data on which only the minimum processing has been performed is called RAW image data 601. Next, the RAW image data 601 is developed by the developing circuit 116k to be YUV image data 602.

  Next, in step S <b> 503, a first high-sensitivity short-second exposure shooting is performed to obtain a short-second exposure image. The high-sensitivity short-second exposure shooting immediately before the long-second exposure shooting is A, and the high-sensitivity short-second exposure shooting immediately after the long-second exposure shooting is B. High-sensitivity short-second exposure photography is used only for calculating the amount of movement, and is not used as a panoramic composite image. What is performed in S503 is high-sensitivity short-second exposure photography B. In short-second exposure shooting, stars are small and thin, so shooting is performed with an increased ISO sensitivity. As in the case of the long-second exposure shooting, the RAW image data 604 is developed by the developing circuit 116k to be YUV image data 605.

  In S504, first, the gyro 133 is initialized at the time of the first image so that the extent to which the imaging apparatus 100 has been swung (rotated) before the second image is captured can be acquired in S509.

  In step S505, the geometric conversion circuit 116e corrects the distortion of the photographing lens 101 using the existing technology for the developed image data 602 and 605 of the long-second exposure shooting and the short-second exposure shooting, respectively. Data 603 and 606 are obtained. The image data 603 after the distortion correction by the long-second exposure is reduced according to the number of pixels of the liquid crystal monitor by the scaling circuit 116f and stored in the VRAM 611 for display on the display member 118.

  Next, in S506, the second high-sensitivity short-second exposure shooting A is performed to obtain image data 608. In S507, the second long-time exposure shooting is performed to obtain image data 613. Further, in S508, the second high-sensitivity short-second exposure shooting B is performed to obtain image data 616. Similarly to S502, each of the RAW image data 608, 613, 616 is developed by the developing circuit 116k to obtain YUV image data 609, 614, 617.

  In step S509, gyro information, which is a detection value of the gyro 133, is obtained to obtain the swing amount of the imaging device 100 from the previous shooting. The gyro information acquires values in two directions of the imaging device in the Yaw direction and the Pitch direction, but it is also preferable to acquire values in three axes directions obtained by adding a Roll direction that is rotation with respect to the optical axis. . The output itself from the gyro 133 is the angular velocity, but in panoramic photography, it is necessary to know how much the camera has been swung from the previous photography, so the angular velocity between the previous photography and the next photography is integrated, and the The rotation angle 607 is calculated and stored.

  In step S510, the rotation angle 607 is converted into a pixel unit based on information such as the focal length and the angle of view of the lens acquired in step S501 and the image sensor.

  In general, the lens having no distortion or the angle of view (α) after distortion correction is calculated by the following equation 1 when the effective focal length is f [mm] and the width of the image sensor is w [mm].

α [°] = 2 x arctan (w [mm] ÷ 2 ÷ f [mm]) (Equation 1)
Assuming that the size per pixel of the image sensor is p [μm] and the swing angle [°] is θ, the amount of movement d [pix] in the image is as shown in Expression 2.

d [pix] = tan (α [°] ÷ 2) × f [mm] / p [μm] × 1000 (Equation 2)
In S511, similarly to the first image (S505), the second image is subjected to distortion correction to obtain distortion-corrected image data 610, 615, and 618. Similarly to the first image, the long-time exposure photographed image data 615 after the distortion correction is reduced according to the number of pixels of the liquid crystal monitor by the scaling circuit 116f and stored in the VRAM 619 for display on the display member 118.

  In S512, using the movement amount calculation circuit 116c, the image data 606 obtained in the first high-sensitivity short-second exposure shooting B and the image data 610 obtained in the second high-sensitivity short-second exposure shooting A are used. Is calculated from the distance. As described above, a known method can be used for the movement amount detection. In the present embodiment, the affine coefficient 612 is calculated by finding some characteristic points in the image by the movement amount detection circuit 116c and sampling it.

  Specifically, edges are detected, feature points are extracted, and the amount of movement is calculated. For example, here, feature point 1 is from coordinates (x1, y1) to coordinates (u1, v1), feature point 2 is from coordinates (x2, y2) to (u2, v2), and feature point 3 is coordinates (x3, y3). To (u3, v3). Then, when simultaneous equations are formed from Equation 1, Equations 3 and 4 are obtained.

By solving this equation, the affine coefficients a to f can be calculated. If four or more feature points are detected, close points are excluded and normalization is performed using the least squares method. When three points are not found, when the extracted three points are linear, and when two of the three points are close to each other, it is determined that the calculation of the movement amount has failed.

  If the movement amount (affine coefficient) calculated from the image in this way greatly differs from the movement amount based on the rotation angle 607 calculated from the detection value of the gyro sensor in S510, the image includes a repetitive pattern or a moving object. It is thought that it is. Therefore, a method of calculating the movement amount again under another condition, making this shot a failure and returning to the next photographing (S506), or issuing a warning that the starry sky panorama photographing has failed can be considered.

  In step S513, based on the movement amount (affine coefficient) calculated from the image, the images obtained by performing the long-time exposure shooting in steps S502 and S507 are aligned using the alignment circuit 116d, and the alignment image data 621 is obtained. Get.

  In step S514, the first image data 620 and the second-aligned image data 621 are combined using the combining circuit 116j to obtain combined image data 622. When the N-th (N> 2) processing is performed, the synthesis result so far, that is, the synthesized image data 620 up to the (N-1) -th image and the N-th aligned image data 621 are synthesized. I do.

  In step S515, if the switch SW2 (126) is pressed, the process returns to the next shooting step S506. If the switch SW2 (126) is not pressed, the process of step S516 is performed. In S516, the image data is compressed into a general-purpose format such as Jpeg using the compression / decompression circuit 116l, and stored in the memory 120 in S517.

  At this time, it is also preferable to perform the gamma correction by the gamma correction circuit 116b in order to make the dark portion of the composite image easy to see, or to perform the correction in order to unify the color tone of the entire image. Since the image obtained in this way has a large size, the image may be scaled by the scaling circuit 116f to the size specified by the user in advance. Further, it is preferable that the maximum inscribed rectangle or a predetermined area is cut out by the trimming circuit 116g and then stored.

  Note that, in the above description, an example in which a plurality of images are shot while the imaging device is shifted in the horizontal direction has been described.

  As described above, even if a star serving as a subject moves between images having different shooting directions, it is possible to perform correct positioning and shoot a high-quality panoramic composite image without increasing sensitivity.

(Second embodiment)
In the present embodiment, an example in which high-sensitivity short-second exposure shooting for calculating a moving amount is not necessary depending on shooting conditions and environment of starry sky panorama shooting will be described. FIG. 6 is a flowchart illustrating an operation of panoramic image capturing according to the second embodiment.

  The processing of S701 to S702 from the start of the starry sky panorama shooting is the processing of S501 to S502 of the first embodiment, the processing of S704 to S706 is the processing of S503 to S505, and the processing of S708 to S709 is S506 to S507. Since the processing of S711 to S720 corresponds to the processing of S508 to S517, the description of these processing will be omitted.

  In S703, it is determined whether or not it is necessary to perform high-sensitivity short-time exposure shooting B following the first long-time exposure shooting (S702). The determination of execution is performed, for example, as follows. First, the microcomputer 123 acquires the direction of the imaging device detected by the compass 134, and calculates the amount of movement of the star during imaging. If it is determined that there is no movement of the star, the processing of S705 is performed without performing the high-sensitivity short-second exposure shooting B of S704. The setting of the accumulation time or the like may be used to determine the execution of the high-sensitivity short-second exposure shooting B.

  In S707 and S710, it is determined in the same processing as S703 whether or not it is necessary to perform high-sensitivity short-time exposure shooting A and B before and after the long-time exposure shooting (S709) for the second and subsequent images. If it is determined that there is no movement of the star, the processes of S709 and S712 are performed without performing the high-sensitivity short-time exposure shooting A of S708 or the high-sensitivity short-time exposure shooting B of S711.

  As described above, by determining the shooting conditions and shooting environment of the starry sky panorama shooting, it is possible to reduce unnecessary power for shooting by omitting unnecessary high-sensitivity short-second exposure shooting for alignment. Become. Note that, in the present embodiment, the case where the execution determination is performed before all the high-sensitivity short-second exposure shootings has been described, but when it is possible to determine that all the high-sensitivity short-second exposure shootings are unnecessary in the standby state of the imaging device. May be configured to be a one-time determination process.

(Third embodiment)
In the present embodiment, an example in which an appropriate warning is displayed to the user when positioning is considered to be unsuccessful in starry sky panoramic imaging will be described with reference to FIGS. 4 and 7.

  When the high-sensitivity short-time shooting B in S504 and the high-sensitivity short-time shooting A in S506 are performed, the microcomputer 123 acquires the time from the clock 132 and stores it. The microcomputer 123 calculates two photographing time intervals, and determines that the star has moved too much if the time interval is longer than a predetermined time, and displays a warning screen of FIG. 8A.

  Further, the microcomputer 123 detects the swing amount of the imaging device from the rotation angle 607, which is the gyro information acquired in S509, and determines that the position cannot be adjusted if the swing amount exceeds a predetermined swing amount, and FIG. Display a warning screen.

  As described above, in a scene where positioning in starry sky panorama shooting is expected to be impossible, displaying a warning in advance can improve user convenience.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

(Other embodiments)
In addition, the present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read the program. It can also be realized by executing processing. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

100: imaging device, 101: photographing lens, 112: imaging element, 116: video signal processing circuit, 119: memory controller, 123: microcomputer, 133: gyro, 134: compass

Claims (18)

An imaging apparatus capable of generating one combined image by combining a plurality of unit images having different imaging directions and having a partially common area,
Imaging means for imaging a subject image;
The imaging unit causes the plurality of unit images to be imaged, and before and after the imaging of each unit image of the plurality of unit images, a short-second exposure having an exposure time shorter than the exposure time of the unit image. Control means for capturing an image,
A first short-time exposure image captured after capturing the first unit image, and a second short-time exposure image captured before capturing the second unit image captured following the first unit image. Calculating means for calculating a moving amount of the subject between the first short-time exposure image and the second short-time exposure image using a second-time exposure image;
The first unit image and the second unit image are aligned based on the movement amount of the subject calculated by the calculation unit, and the first unit image and the second unit image are combined. Means for synthesizing,
An imaging device comprising:   The imaging apparatus according to claim 1, wherein the subject is a starry sky, and the unit image is an image captured with a long exposure for capturing a starry sky.   The imaging apparatus according to claim 2, wherein the short-time exposure image is an image captured by increasing the sensitivity of the imaging unit in order to capture the starry sky with the short-time exposure.   A first determining unit that determines whether or not it is necessary to perform the imaging of the short-time exposure image; and when the first determining unit determines that the imaging of the short-time exposure image is not necessary. The imaging apparatus according to claim 1, wherein the control unit omits imaging of the short-time exposure image.   The determining unit determines that it is not necessary to capture the short-time exposure image when there is no movement of the subject between the first short-time exposure image and the second short-time exposure image. The imaging device according to claim 4, wherein   A second determination unit that determines whether the first unit image and the second unit image can be aligned, and the second determination unit determines whether the first unit image is aligned with the second unit image. The imaging apparatus according to claim 1, further comprising: a warning unit configured to issue a warning when it is determined that the alignment of the unit images is not possible.   The second determination unit is configured to determine that a movement amount of the subject between the first short-second exposure image and the second short-second exposure image calculated by the calculation unit exceeds a predetermined amount. 7. The imaging apparatus according to claim 6, wherein it is determined that the alignment between the first unit image and the second unit image is impossible.   The image forming apparatus further includes a timer that measures an elapsed time between the imaging of the first short-second exposure image and the imaging of the second short-second exposure image, wherein the second determination unit determines that the elapsed time is a predetermined time. 7. The imaging apparatus according to claim 6, wherein when the time has elapsed, it is determined that alignment between the first unit image and the second unit image is impossible. An image processing apparatus capable of generating one combined image by combining a plurality of unit images having different imaging directions and having a partially common area,
Obtaining the plurality of unit images and a short-second exposure image captured with an exposure time shorter than the exposure time of the unit image before and after capturing each unit image of the plurality of unit images. Acquisition means for
A first short-time exposure image captured after capturing the first unit image, and a second short-time exposure image captured before capturing the second unit image captured following the first unit image. Calculating means for calculating a moving amount of the subject between the first short-time exposure image and the second short-time exposure image using a second-time exposure image;
The first unit image and the second unit image are aligned based on the movement amount of the subject calculated by the calculation unit, and the first unit image and the second unit image are combined. Means for synthesizing,
An image processing apparatus comprising:   The image processing apparatus according to claim 9, wherein the subject is a starry sky, and the unit image is an image captured with a long exposure for capturing a starry sky.   The image processing apparatus according to claim 10, wherein the short-time exposure image is an image captured by increasing the sensitivity of the imaging unit in order to capture a starry sky with a short-time exposure.   A second determination unit that determines whether the first unit image and the second unit image can be aligned, and the second determination unit determines whether the first unit image is aligned with the second unit image. The image processing apparatus according to claim 9, further comprising: a warning unit that issues a warning when it is determined that alignment of the unit images is impossible.   The second determination unit is configured to determine that a movement amount of the subject between the first short-second exposure image and the second short-second exposure image calculated by the calculation unit exceeds a predetermined amount. 13. The image processing apparatus according to claim 12, wherein it is determined that the alignment between the first unit image and the second unit image is impossible.   The acquisition unit further acquires information on an elapsed time between the imaging of the first short-time exposure image and the imaging of the second short-time exposure image, and the second determination unit includes the elapsed time. 13. The image processing apparatus according to claim 12, wherein when the time exceeds a predetermined time, it is determined that the alignment between the first unit image and the second unit image is impossible. A method for controlling an imaging apparatus capable of generating one combined image by combining a plurality of unit images having different imaging directions and having a partially common area,
The imaging unit that captures the subject image causes the plurality of unit images to be captured, and before and after capturing each unit image of the plurality of unit images, the exposure time is shorter than the exposure time of the unit image. A control step of capturing a short short exposure image,
A first short-time exposure image captured after capturing the first unit image, and a second short-time exposure image captured before capturing the second unit image captured following the first unit image. Using a second exposure image, a calculation step of calculating the amount of movement of the subject between the first short exposure image and the second short exposure image;
The first unit image and the second unit image are aligned based on the movement amount of the subject calculated in the calculation step, and the first unit image and the second unit image are combined. A synthesis process,
A method for controlling an imaging device, comprising: A method for controlling an image processing apparatus capable of generating one combined image by combining a plurality of unit images having different imaging directions and having a partially common area,
Obtaining the plurality of unit images and a short-second exposure image captured with an exposure time shorter than the exposure time of the unit image before and after capturing each unit image of the plurality of unit images. An acquisition process to
A first short-time exposure image captured after capturing the first unit image, and a second short-time exposure image captured before capturing the second unit image captured following the first unit image. Using a second exposure image, a calculation step of calculating the amount of movement of the subject between the first short exposure image and the second short exposure image;
The first unit image and the second unit image are aligned based on the movement amount of the subject calculated in the calculation, and the first unit image and the second unit image are aligned. A synthesis step of synthesizing,
A method for controlling an image processing apparatus, comprising:   A program for causing a computer to execute each step of the control method according to claim 15.   A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 15.