JP2020043419A - Imaging apparatus, control method of the same, and program - Google Patents

Abstract

To provide an imaging apparatus capable of positioning both of a background and stars even if a star to be a subject is moved between photographs which have different field angle in a starry sky panoramic photography, a control method of the same, and a program.SOLUTION: After and before an acquisition of a long second photography image in each difficult field angle, a short second photography A image and a short second photography B image are acquired. Sequentially, for the long second photography image of N-th is composed by a comparison darkness combination compared with a comparison darkness combination image until N-1-th after the positioning with a blurring amount of the short second photography B image of N-1-th and the short second photography A image at N-th to only generate a panoramic image of a simple background. Sequentially, only for N-1-th star extracted from the long second photography image of N-1-th and the comparison darkness combination image until N-th, it is composed by a comparison brightness combination with a comparison darkness combination image until N-1-th after the positioning with the blurring amount of images only in the star to generate the panoramic image of the simple star. A starry panoramic image is generated from the two panoramic images to be generated.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an imaging device, a control method thereof, and a program. In particular, the present invention relates to an imaging apparatus that performs a starry sky panoramic image capturing, a control method thereof, and a program.

  One of the shooting methods for storing a vast celestial object as one image is starry sky panorama shooting.

  Further, as a technique for generating a panoramic image, the following technique is known (for example, see Patent Document 1). First, a plurality of unit images constituting a part of the imaging range are imaged by sequentially changing the imaging direction. Thereafter, image regions of a predetermined size, which constitute a part of each of the unit images captured on the (N-1) -th and N-th sheets, are cut out so that overlapping regions are generated. This is a method in which this extraction is performed on all the unit images, and the extracted image regions are sequentially superimposed and synthesized.

  On the other hand, there is also known a technique for obtaining a captured image having a wider angle of view while preventing misalignment by repeating photographing and synthesizing while performing optical shake correction for a positional shift on a photographing surface due to a diurnal motion of a celestial body. (For example, see Patent Document 2).

  When photographing an astronomical object, a long exposure time of 30 seconds or 1 minute is often performed because the amount of light of the star is very small. On the other hand, since the celestial body performs a diurnal motion in accordance with the rotation of the earth, stars are not point images but light trails when exposed for a long time without performing optical shake correction as in Patent Document 2. .

  Therefore, when taking a panoramic photograph of an astronomical object, it is necessary to photograph an image (astronomical image) having a different angle of view as a unit image with an exposure time that does not cause a light trail.

JP 2005-328497 A JP-A-2006-005160

  However, since stars move with the passage of time between astronomical images taken with long exposure, when the image regions cut out from each unit image are sequentially superimposed and synthesized, it is not possible to align the background and the star respectively. Have difficulty.

  FIG. 3 shows an example of a failure in alignment when two images 301 and 302 having different angles of view are combined during starry sky panoramic photography.

  In FIG. 3, first, the image 301 is obtained by performing long-second exposure shooting with the camera directed in a predetermined direction. Thereafter, after panning the camera to a predetermined angle, the image 302 is obtained by performing long-time exposure photography again.

  After that, a blur amount calculation and a position adjustment are performed on a superimposed region of the images 301 and 302, and the images are combined to generate a panoramic image 303. However, since the star to be the subject is a moving object, there is a problem that the star becomes a double image in the overlapping area of the generated panoramic image 303. In addition, it takes time to complete all of the multiple long-second exposure shootings performed during the starry sky panorama shooting, and the relative relationship between the background and the star changes between the image at the start of shooting and the image at the end of shooting. Also occurs.

  Further, in the prior arts disclosed in Patent Documents 1 and 2, alignment with respect to either the background or the star can be performed. However, when a star serving as a subject moves between shootings, the background and the star move. Cannot perform both alignments.

  Therefore, an object of the present invention is to provide an imaging apparatus and a control method thereof that can perform both positioning of a background and a star even when a star serving as a subject moves between shootings having different angles of view in a starry sky panoramic shooting, and Provide a program.

  An image pickup apparatus according to claim 1 of the present invention is an image pickup apparatus that generates a starry sky panoramic image using n long-time shot images obtained by long-time exposure shooting at a different angle of view. Acquiring means for acquiring first and second short-time photographed images by performing short-time exposure photographing before and after the long-time exposure photographing, respectively, and an (N-1) -th image (N is an integer from 2 to n) A) a first blur amount calculating means for calculating a blur amount between view angles from the second short-time shot image and the N-th first short-time shot image; (1) first alignment means for aligning the N-th long-second captured image with respect to the first long-second captured image, and the N-th long-second captured image sequentially aligned The image is compared with a comparative dark synthetic image which is a result of synthesizing the long-second captured images up to the (N-1) th image. A first synthesizing means for generating a panoramic image of only the background, and N-1 images representing the difference between the (N-1) -th long-time photographed image and the comparative dark-synthesized image, which is the synthesis result up to the N-th image Difference image extracting means for extracting an eye-only star image, and second blur amount calculating means for calculating a star-to-star image blur amount from the (N-1) th star-only image and the N-th star-only image Second positioning means for positioning the N-th star-only image with respect to the (N-1) -th star-only image using the star-only inter-image blurring amount; A comparatively bright composite of the N-th star-only image that has been combined and the comparative bright composite image that is the composite result of the (N-1) -th star-only image is performed to generate a star-only panoramic image. 2, combining the background only panoramic image and the star only panoramic image. Synthesized and characterized in that it comprises a third synthesizing means for generating a starry sky panoramic image.

  ADVANTAGE OF THE INVENTION According to this invention, even if the star used as a subject moves between imaging | photography with a different angle of view in starry sky panorama imaging | photography, both a background and a star can be aligned.

FIG. 2 is a block diagram illustrating a hardware configuration of the imaging device according to the first embodiment of the present invention. It is a figure for explaining the outline of panorama photography. It is a conceptual diagram explaining the subject of a starry sky panorama synthesis | combination. 3 is a flowchart illustrating a procedure of a normal photographing process in the imaging device in FIG. 1. 6 is a flowchart illustrating a procedure of a starry sky panorama shooting process according to the first embodiment of the present invention. FIG. 6 is a flowchart illustrating a procedure of a parallel process of photographing and generating only a background panoramic image in step S503 in FIG. 5. FIG. 7 is a data flow diagram showing a flow of parallel processing of shooting and background only panoramic image generation in the processing of FIG. 6. FIG. 6 is a flowchart illustrating a procedure of a star-only panorama image generation process in step S <b> 504 in FIG. 5. FIG. 9 is a conceptual diagram showing a flow of generating a panoramic image of only the starry sky in the processing of FIG. 8. 6 is a flowchart illustrating a procedure of a starry sky panoramic image generation process in step S505 in FIG. 9 is a flowchart illustrating a procedure of a starry sky panoramic image generation process according to the second embodiment of the present invention. It is a conceptual diagram showing the flow of generation of the starry sky panorama image concerning Example 2 of the present invention. It is a figure showing an example of a warning display concerning Example 2 of the present invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  First, an overview of panoramic shooting will be described with reference to FIG.

  First, as shown in FIG. 2A, the user 202 sequentially captures images while panning the image capturing apparatus 100 directly or with the image capturing apparatus 100 mounted on an automatic camera platform or the like, and acquires a plurality of images. As shown in FIG. 2B, shooting is performed such that each of the plurality of acquired images includes a superimposed region of the subject.

  When the first image and the second image are acquired, the feature points of the overlapping area are extracted from both images, and a movement vector (a blur amount) indicating how much the feature points have moved between the images is detected. . The movement vector is converted into, for example, affine transformation coefficients, and the composite image is obtained by superimposing the first image and the second image so that the feature points match.

  Thereafter, a panoramic image (whole view image) is generated by sequentially superimposing the composite image and the image captured immediately before. When the panorama image to be generated is a starry sky panorama image, shooting performed sequentially while panning is long-time exposure shooting, so shooting using a tripod is desirable.

  FIG. 1 is a block diagram illustrating a hardware configuration of the imaging apparatus 100 according to the first embodiment of the present invention.

  In FIG. 1, the imaging apparatus 100 includes a photographing lens 101, an AF (autofocus) driving circuit 102, a microcomputer 123, an aperture 103, an aperture driving circuit 104, a main mirror 105, a sub mirror 106, and a mirror driving circuit 107. Further, the imaging apparatus 100 includes a pentaprism 108, an exposure amount calculation circuit 109, a focal plane shutter (hereinafter, simply referred to as “shutter”) 110, a shutter driving circuit 111, an imaging sensor 112, an A / D converter 115, and a video signal. The processing circuit 116 is provided. The imaging device 100 further includes a memory controller 119, a memory 120, an external interface 121, a buffer memory 122, an operation member 124, a release switch 125, and a liquid crystal drive circuit 127. Further, the imaging apparatus 100 includes an external liquid crystal display member (hereinafter, referred to as “liquid crystal monitor”) 128, a liquid crystal display member 129 in a finder, a nonvolatile memory 130, a power supply unit 131, a clock 132, a gyro 133, and a compass 134.

  The AF drive circuit 102 is configured by, for example, a DC motor or a stepping motor, and focuses by changing the focus lens position of the photographing lens 101 under the control of the microcomputer 123.

  The aperture driving circuit 104 drives the aperture 103. The amount to be driven is calculated by the microcomputer 123 and changes the optical aperture value.

  The main mirror 105 is a mirror for switching between guiding the light beam incident from the photographing lens 101 to the pentaprism 108 constituting the viewfinder or the image sensor 112. The main mirror 105 always guides the light beam to the pentaprism 108, but in the case of photographing or live view display, the main mirror 105 jumps upward to guide the light beam to the image sensor 112 and escapes from the light beam. The main mirror 105 is a half mirror so that a central part thereof can transmit a part of light, and transmits a part of the light flux so as to be incident on a sensor (not shown) for performing focus detection. The finder is constituted by a focus plate, an eyepiece lens, etc., not shown in FIG.

  The sub-mirror 106 reflects a light beam transmitted from the main mirror 105 and guides the light beam to a sensor (not shown) for performing focus detection and a sensor (not shown) arranged in the exposure amount calculation circuit 109.

  The mirror driving circuit 107 drives the main mirror 105 under the control of the microcomputer 123.

  The exposure amount calculation circuit 109 receives the light beam transmitted through the central portion of the main mirror 105 and reflected by the sub-mirror 106 by a sensor for performing photoelectric conversion disposed therein. The defocus amount used for the focus calculation is obtained by calculating the output of this sensor. The microcomputer 123 evaluates the calculation result and instructs the AF driving circuit 102 to drive the photographing lens 101.

  The shutter drive circuit 111 drives the shutter 110 under the control of the microcomputer 123. That is, the opening time of the shutter 110 is controlled by the microcomputer 123.

  As the image sensor 112, a CCD, a CMOS sensor, or the like is used. The image sensor 112 converts a subject image formed by the imaging lens 101 into an analog image signal and outputs the analog image signal.

  The A / D converter 115 AD-converts the image signal output from the image sensor 112 and outputs the result.

  The video signal processing circuit 116 is realized by a logic device such as a gate array, performs various processes on the image of the image signal output from the A / D converter 115, and outputs the processed image data. Specifically, the video signal processing circuit 116 includes a luminance adjustment circuit 116a, a gamma correction circuit 116b, a blur amount calculation circuit 116c, a positioning circuit 116d, and a geometric conversion circuit 116e. The video signal processing circuit 116 includes a scaling circuit 116f, a trimming circuit 116g, a synthesizing circuit 116j, a developing circuit 116k, a compression / decompression circuit 116l, and a difference image extraction circuit 116m.

  The brightness adjustment circuit 116a adjusts the brightness of the image using a digital gain, and the gamma correction circuit 116b adjusts the brightness of the image using a gamma characteristic. The blur amount calculating circuit 116c calculates the blur amount between the two images, and the positioning circuit 116d aligns the two images according to the calculated blur amount. The geometric transformation circuit 116e corrects the distortion of the photographing lens 101, the scaling circuit 116f changes the size of the image, the trimming circuit 116g cuts out a part of the image, and the combining circuit 116j converts the plurality of images. Combine into one image. The development circuit 116k performs development for converting the image signal into image data, the compression / decompression circuit 116l performs conversion for compressing the image data into a general image format such as Jpeg, and the difference image extraction circuit 116m performs conversion from a plurality of images. Extract the difference image. Note that the synthesis circuit 116j performs not only simple addition synthesis and average addition synthesis but also comparative bright synthesis in which a pixel having the brightest or darkest value in each region of the image to be synthesized is selected to generate one image. And processing such as comparison dark synthesis.

  The memory 120 is a removable memory such as an SD card or a compact flash (registered trademark).

  The external interface 121 is an interface that allows the imaging device 100 to be connected to an external computer or the like.

  The memory controller 119 outputs the processed image data output from the video signal processing circuit 116 so as to be held in the memory 120. In addition, when image data being processed is transmitted from the video signal processing circuit 116, the memory controller 119 outputs the image data to be temporarily stored in the buffer memory 122. The output destination is switched by the memory controller 119 in accordance with an instruction from the microcomputer 123.

  Conversely, the memory controller 119 outputs image data from the buffer memory 122 or the memory 120 to the video signal processing circuit 116 according to an instruction of the microcomputer 123.

  The microcomputer 123 obtains information such as exposure information and white balance included in the image signal from the image sensor 112 as needed by the video signal processing circuit 116, and adjusts white balance and gain based on the information. Instructions to do so. Further, in the case of the continuous shooting operation, the video signal processing circuit 116 controls the memory controller 119 to temporarily store the image data in the process of being obtained for each shooting in the buffer memory 122, Compression processing is performed sequentially. The number of continuously shootable images depends on the capacity of the buffer memory 122. Further, the number of continuously shootable images during panoramic shooting is further influenced by the image size of image data obtained by one shot.

  The memory controller 119 can further output an image stored in the memory 120 via an external interface 121 that connects the imaging device 100 to an external computer or the like.

  The operation member 124 is connected to the release switch 125 and transmits a signal state of the release switch 125 to the microcomputer 123. The microcomputer 123 controls each part according to the state of the signal of the release switch 125 received via the operation member 124. Specifically, the release switch 125 turns on the state of the SW1 signal when the user is half-pressed by pressing the button. When the state of the SW1 signal is turned ON, the microcomputer 123 performs an autofocus operation and performs a photometric operation. The release switch 125 turns on the state of the SW2 signal when the switch is fully pressed by the user. When the state of the SW2 signal is turned on, the microcomputer 123 starts recording (photographing) an image. Further, while both the state of the SW1 signal and the state of the SW2 signal are ON, the continuous shooting operation is performed.

  The operation member 124 is also connected to various buttons and switches (not shown in FIG. 1), and notifies the microcomputer 123 of the states of these buttons and switches. Here, various buttons and switches (not shown) are not particularly limited. For example, an ISO setting button, a menu button, a set button, a flash setting button, a single-shot / continuous-shot / self-timer switching button, a move + (plus) button for moving a menu or a reproduced image, a move-(minus) button, exposure The correction button. The display image enlargement button, the display image reduction button, the reproduction switch, the aperture button for setting the aperture 103 to the set aperture amount, the erasure button for erasing a captured image, and the information display button for photography and reproduction are also not available. It may be included in the various buttons and switches shown. The functions of the plus button and the minus button are provided with a rotary dial switch so that numerical values and functions can be more easily selected.

  The liquid crystal driving circuit 127 drives a liquid crystal monitor 128 and a finder liquid crystal display member 129 which display an operation state, a message, and the like using characters and images according to a display content command of the microcomputer 123.

  A backlight such as an LED (not shown) is disposed on the liquid crystal display member 129 in the finder, and the LED is also driven by the liquid crystal driving circuit 127.

  The microcomputer 123 checks the capacity of the memory 120 through the memory controller 119 based on the predicted value data of the image size corresponding to the ISO sensitivity, the image size, and the image quality set before the photographing, and then sets the remaining photographable amount. Numbers can be computed. The calculation result is displayed on the liquid crystal monitor 128 or the liquid crystal display member 129 in the viewfinder as needed.

  The nonvolatile memory 130 is composed of an EEPROM, and can store data even when the power supply unit 131 of the imaging device 100 is off.

  The power supply unit 131 supplies necessary power to each IC and the drive system when in the ON state.

  The clock 132 can store the photographing time and the like in the image data recorded in the memory 120, and can superimpose the photographing time on the image itself as described later.

  The gyro 133 detects the angular velocity of rotation of the imaging device 100 on two axes or three axes.

  The compass 134 detects the direction of the optical axis of the imaging lens 101 of the imaging device 100.

  Hereinafter, a normal photographing process in the imaging device 100 will be described with reference to the flowchart of FIG. Before starting this processing, the exposure amount is calculated in advance by the exposure amount calculation circuit 109, and the aperture amount, the accumulation time, and the ISO sensitivity are determined in advance.

  This processing is started in the microcomputer 123 when the release switch 125 is fully pressed by the user and the state of the SW2 signal is turned on.

  First, the microcomputer 123 notifies a predetermined aperture amount to the aperture drive circuit 104 and sets the aperture 103 to a target aperture amount. Next, power is supplied from the power supply unit 131 to the image sensor 112, the A / D converter 115, and the like, and preparation for photographing is performed. When the preparation is completed, the mirror driving circuit 107 is driven to retract the main mirror 105 from the light beam, and the shutter driving circuit 111 is driven to open the front curtain (not shown) of the shutter 110, and the subject image is passed through the photographing lens 101. An image is formed on the image sensor 112.

  Subsequently, after a predetermined accumulation time, the shutter drive circuit 111 is driven to close the rear curtain (not shown) of the shutter 110 so that light enters the image sensor 112 only during the accumulation time. Exposure is performed by performing this series of operations (step S401).

  Subsequently, the image signal of the subject image formed on the image sensor 112 is converted into a digital signal by the A / D converter 115, and the image signal is read out by the video signal processing circuit 116 and stored in the buffer memory 122 (step S402). ).

  Next, a developing process of converting the image signal read by the video signal processing circuit 116 into image data is performed by the developing circuit 116k (step S403). At this time, the white balance may be adjusted so that the image is appropriate, or image processing such as applying a gain to a dark part by the gamma correction circuit 116b may be performed.

  Thereafter, the image data obtained by the development processing in step S403 is converted by the compression / decompression circuit 116l into a general-purpose image format such as JPEG (step S404), and is written in the memory 120 (step S405). This processing ends.

  Note that the process directly proceeds from step S402 to step S404 without performing the image processing and the developing process in step S403, losslessly compresses the read image signal as it is, and stores the losslessly compressed image signal in the memory 120 in step S405. May be switched. This switching can be performed by the user using the operation member 124.

  Hereinafter, the starry sky panorama photographing process executed by the microcomputer 123 will be described with reference to the flowchart of FIG. The starry sky panorama shooting mode includes a mode in which the image capturing apparatus 100 is shot while changing the direction in the horizontal direction and a mode in which the image is shot while changing the direction in the vertical direction. Here, the former mode will be described.

  When the user operates the menu button to select the starry sky panorama shooting mode (YES in step S500), preparation for shooting is performed (step S501).

  Here, the shooting preparation specifically indicates the following processing. First, power is supplied from the power supply unit 131 to the image sensor 112, the A / D converter 115, and the like, and initialization is performed. Next, the mirror drive circuit 107 is driven to retract the main mirror 105 from the light beam, and the shutter drive circuit 111 is driven to open the shutter 110 so that a subject image is formed on the image sensor 112 through the photographing lens 101. .

  Thereafter, the display of the live view on the liquid crystal monitor 128 is started. That is, the image signal from the image sensor 112 is converted into a digital signal by the A / D converter 115, and is developed into image data by the developing circuit 116k of the video signal processing circuit 116, and the luminance adjustment circuit 116a and the gamma correction are performed. The circuit 116b adjusts the brightness and brightness of the image. Further, the image data is scaled by the scaling circuit 116f to an image size suitable for the liquid crystal monitor 128 and displayed. This is repeated 24 to 60 times per second.

  Next, when the user adjusts the angle of view while checking the live view on the liquid crystal monitor 128 and then half-presses the release switch 125, the release switch 125 turns on the state of the SW1 signal. When the state of the SW1 signal is turned on, the microcomputer 123 performs a photometric operation, and the exposure calculation circuit 109 calculates the exposure. In this embodiment, when calculating the exposure amount, the live view is stopped, and the light reflected by the sub-mirror 106 is guided to the sensor in the exposure amount calculation circuit 109, so that the optimum exposure amount Is calculated. It should be noted that the live view may be continued even when calculating the exposure amount. In this case, an optimal exposure amount is determined by an exposure amount calculation circuit (not shown) included in the video signal processing circuit 116.

  Thereafter, exposure control is performed based on the calculated exposure amount. Specifically, the aperture amount is determined based on the calculated exposure amount, the aperture amount is notified to the aperture drive circuit 104, and the aperture 103 is set as the aperture amount. Also, the sensitivity and the accumulation time of the image sensor 112 are controlled based on the calculated exposure amount. At this time, in the starry sky panoramic imaging, the accumulation time is set using the program diagram so that the exposure time is such that the star does not follow the trajectory during long-second exposure imaging.

  After the exposure control, the photographing lens 101 is driven by the AF driving circuit 102, and when the focus adjustment is completed, the user is notified by a buzzer or the like (not shown) that preparation for the starry sky panorama photography is completed, and photography preparation is completed.

  In step S501, the preparation for photographing is completed. When the user receives the above notification and turns the release switch 125 fully toward the direction in which the user wants to start photographing the imaging apparatus 100, the release switch 125 turns on the state of the SW2 signal. When the state of the SW1 signal is turned ON (YES in step S502), the microcomputer 123 shifts to parallel processing of capturing and generating a panoramic image only for the background (step S503). In this process, a panoramic image of only the background is generated in parallel with the photographing for acquiring all the images necessary for generating the starry sky panoramic image. The method of generating the panoramic image with only the background will be described in detail below with reference to the data flow diagram of FIG. 7 and the conceptual diagram of FIG.

  In FIG. 6, first, lens information of the photographing lens 101 is obtained (step S601). The lens information includes data for correcting distortion and a decrease in the amount of light around the lens, which will be described later.

  Next, long-second exposure shooting at the first angle of view (hereinafter, referred to as "first long-time shooting") is performed (step S602). Since the imaging sensor 112 and the A / D converter 115 are set to drive for live view, they are switched to those for still image shooting. Thereafter, the processing of FIG. 4 described above is performed. In step S403, the image signal read out to the video signal processing circuit 116 is subjected to correction such as shading of the image sensor 112 by a circuit (not shown) included in the video signal processing circuit 116. Image data that has undergone minimal processing in this manner is hereinafter referred to as a RAW image. Subsequently, the RAW image is developed by the developing circuit 116k to be a YUV image. Specifically, when N in FIG. 7 is 2, a RAW image 601 is obtained by the long-time shooting of the first image, which is developed by the developing circuit 116k to become the first YUV image 602.

  In step S603, the first high-sensitivity short-second exposure shooting is performed. Hereinafter, the high-sensitivity short-time exposure photography immediately before the N-th long-time exposure is referred to as the N-th short-time exposure A, and the high-sensitivity short-time exposure immediately after the N-th long-time exposure is taken as the N-th exposure. Is referred to as short-time shooting B. The images obtained by the short-time shooting A and B are used only for calculating the amount of blur, and are not used as a panorama composite image. The high-sensitivity short-time exposure shooting performed in step S603 is a process immediately after the long-time shooting in step S602, and thus corresponds to the short-time shooting B. When the Nth short-time shooting A and B are performed at the same ISO sensitivity as the Nth long-time shooting, the stars of the images obtained in the Nth short-time shooting A and B become the Nth long-time shooting It is smaller and thinner than. For this reason, in the Nth short-time shooting A and B, the ISO sensitivity is switched to a higher ISO sensitivity than in the Nth long-time shooting, and the above-described processing in FIG. 4 is performed. Specifically, when N in FIG. 7 is 2, a RAW image 604 is obtained by the first short-time shooting B, and this is developed by the developing circuit 116k to become the first YUV image 605.

  In step S604, the gyro information at the time of shooting the first image is first initialized (reset) to obtain how much the imaging apparatus 100 has been swung (rotated) before the second long-time shooting.

  In step S605, the geometric conversion circuit 116e captures the YUV images 602 and 605 obtained in the first long-time shooting B in step S602 and the first short-time shooting B in step S603 using existing technology. The distortion of the lens 101 is corrected. With this correction, the images 603 and 606 after geometric transformation are obtained. In order to display the post-geometric transformation image 603 obtained by long-time shooting on the liquid crystal monitor 128, the image is reduced by the scaling circuit f according to the number of pixels of the liquid crystal monitor 128 and stored in the VRAM area 611 of the buffer memory 122. Thereafter, the counter N is set to 2, and the process proceeds to step S606.

  In step S606, after the RAW image 608 is obtained by the Nth short-time shooting A, the image is developed by the developing circuit 116k to be a YUV image 609.

  Subsequently, in step S607, after the RAW image 613 is obtained by the N-th long-time shooting, the image is developed by the developing circuit 116k to be a YUV image 614.

  Further, in step S608, after a RAW image 616 is obtained by the Nth short-time shooting B, the image is developed by the developing circuit 116k to be a YUV image 617.

  In step S609, the rotation angle of the imaging device 100 from the time of photographing the (N-1) th image to the time of photographing the Nth image is acquired as gyro information. As the gyro information, rotation angles about two axes in the Yaw direction and the Pitch direction with respect to the imaging device 100 are acquired, but it is also preferable to acquire rotation angles about three axes in the Roll direction that is rotation about the optical axis. . Since the output from the gyro 133 is an angular velocity itself, a rotation angle obtained by integrating the angular velocity is calculated as gyro information. Since the calculated rotation angle is necessary when calculating the gyro information when N> 2, it is stored as the gyro information 607.

  In step S610, the gyro information calculated in step S609 is converted into pixel units based on the focal length of the photographing lens 101, the angle of view, information on the image sensor 112, and the like acquired in step S601, and the amount of movement of the imaging device 100 is calculated. calculate.

Here, the angle of view (α) of the photographing lens 101 after the distortion correction is calculated by the following equation 1, assuming that the effective focal length is f [mm] and the width of the image sensor 112 is w [mm].
α [°] = 2 × arctan (w [mm] ÷ 2 ÷ f [mm]) (Equation 1)

Assuming that the size per pixel of the image sensor 112 is p [μm] and the swing angle [°] is θ, the moving amount d [pix] in the image is as shown in Expression 7.
d [pix] = tan (α [°] ÷ 2) × f [mm] / p [μm] × 1000 (Equation 2)

  In step S611, the distortion aberration of the photographing lens 101 is determined for the N-th YUV images 609, 614, and 617 obtained in the short-time shooting A, the long-time shooting, and the short-time shooting B in the same manner as in step S605. After the correction, the images 610, 615, and 618 after the geometric transformation are obtained. At this time, similarly to the post-geometric conversion image 603 obtained by the first long-time shooting, the post-geometric conversion image 615 obtained by the N-th long-time shooting is displayed on the liquid crystal monitor 128. For this reason, the image 615 after the geometric transformation is reduced by the scaling circuit 116f according to the number of pixels of the liquid crystal monitor 128, and then stored in the VRAM 619 of the buffer memory 122.

  In step S612, the image 606 obtained from the (N-1) th (previous) short-time shooting B and the N-th (current) short-time shooting A using the blur amount calculation circuit 116c are obtained. The amount of blur between the images 610 after the geometric transformation, that is, the amount of blur between the angles of view, is calculated. Although a known method can be used to calculate the blur amount, in this embodiment, the affine coefficient 612 as the blur amount is extracted by sampling the feature points in the superimposed regions in both images by the blur amount calculation circuit 116c and sampling the extracted feature points. It has been calculated. For example, here, feature point 1 is from coordinates (x1, y1) to coordinates (u1, v1), feature point 2 is from coordinates (x2, y2) to (u2, v2), and feature point 3 is coordinates (x3, y3). To (u3, v3). Then, when simultaneous equations are formed from Equation 1, Equations 3 and 4 are obtained.

  By solving this equation, the affine coefficients a to f can be calculated. If four or more feature points are detected, close points are excluded and normalization is performed using the least squares method. If three or more feature points cannot be detected, if the three detected feature points are linear, and if two of the three detected points are close, the calculation of the blur amount has failed. Judge.

  If the blur amount (affine coefficient) calculated in this way greatly differs from the movement amount calculated in step S610, the post-geometric transformation images 606 and 601 may include a repeated pattern or moving object. Conceivable. In this case, the blur amount is calculated again under another condition, or this photographing is determined to be failed, and the process returns to step S606 to warn that the Nth photographing is performed again or that the starry sky panoramic photographing has failed.

  In step S613, based on the shake amount (affine coefficient) calculated in step S612, the positioning circuit 116d performs N-image conversion on the post-geometric transformation image 603 obtained in the (N-1) -th long-time shooting. The position of the geometrically transformed image 615 obtained by long-second imaging of the eye is aligned. Thereby, the alignment image 621 is obtained.

  In step S614, the combining circuit 116j performs comparative dark combining on the alignment image 621 acquired in step S613 and the comparative dark combined image 620, which is the result of combining up to the (N-1) th sheet, and outputs the comparative dark combined image 622. get. When N = 2, the geometrically transformed image 603 obtained by the first long-time shooting is used as a comparative dark combined image 620 which is a result of combining up to the (N-1) th image. Hereinafter, the comparative dark synthesis processing will be described with reference to the conceptual diagram of the image processing in FIG.

  As shown in FIG. 9, while the star, which is the subject, is moving, the first to fourth geometrically transformed images (hereinafter simply referred to as “long 905, 908, 911, and 914 are acquired.

  On the other hand, an image after geometric transformation (hereinafter, referred to as a “short-time photographed B image” (second short-time photographed image)) 906 obtained by the first short-time photographing B immediately after the first image is photographed. Is obtained. Further, an image after geometric transformation (hereinafter, referred to as a “short-second photographed A image” (first short-second photographed image)) 907 obtained by the short-second photographing A immediately before the second long-second photographing is obtained, and Immediately after the second long-time shooting, a short-time shot B image 909 is obtained. Similarly, short-time shooting A images 910 and 913 are acquired immediately before the third and fourth long-time shooting, and short-time shooting is performed immediately after the third and fourth long-time shooting. B images 912 and 915 are acquired.

  First, a blur amount 916 of a star in an overlapping area of the first short-time photographed B image 906 and the second short-time photographed A image 907 is calculated. Similarly, the blur amount 917 is calculated using the second short-time B image 909 and the third short-time A image 910, and the third short-time B image 912 and the fourth The blur amount 918 is calculated using the second shot A image 913.

  The calculated blur amounts 916, 917, and 918 are used when comparative dark synthetic images 919, 920, and 921 are obtained by performing comparative dark synthetic processing on the long-second captured images 908, 911, and 914, respectively. More specifically, the comparative dark synthetic image 919 is generated by performing comparative dark synthetic on the first long second photographed image 901 and the second long second photographed image 908 in a state of being aligned based on the blur amount 916. Is done. As shown by the comparative dark synthetic image 919, in the superimposed area of the images to be subjected to the comparative dark synthetic (here, the long-second captured images 905 and 908), the background remains because the moving is not performed, but the star is moving. Do not remain.

  In the case of N> 2, the comparative dark synthetic image and the N-th long-time photographed image, which are the combined results up to the (N-1) th image, are compared with the (N-1) -th short-time photographed B image and the N-th image. The comparative dark synthesis is performed in a state where the position is adjusted based on the shake amount calculated using the short-time shooting A image. As a result, a comparative dark synthetic image, which is the synthesis result up to the Nth sheet, is generated. For example, the comparative dark synthetic image 920 which is the synthetic result up to the third image is aligned with the comparative dark synthetic image 919 which is the synthetic result up to the second image and the third long-time captured image 911 based on the blur amount 917. Are generated by performing comparative dark synthesis in a state in which. By repeating the similar comparative dark synthesis, a panoramic image can be generated only for the background where the background area is gradually connected like the comparative dark synthetic image 921.

  Returning to FIG. 6, in step S615, when the user fully presses the release switch 125 during a period until the predetermined time elapses after the short-time shooting B in step S608 is completed, the release switch 125 turns on the SW2 signal. I do. In this case, it is determined that the shooting has not been completed yet (NO in step S615), and after the count N is incremented by one, the processing from step S606 is repeated. On the other hand, if the user does not fully press the release switch 125 during the above period, it is determined that the shooting is to be ended (YES in step S615), and this processing ends. In the following processing, a case will be described in which the value of the counter N is n when it is determined in step S615 that the shooting is to be ended.

  As described above, according to the processing in FIG. 6, by incrementing the counter N from 2 to n, long-second exposure shooting at each angle of view and short-second exposure shooting before and after the angle of view are performed. The position alignment and the comparison dark combination based on the blur amount calculated from the short-time captured image are repeated. As a result, a panoramic image can be generated only from the background where only the background portion is gradually connected like the comparative dark synthetic image 921.

  Also, as shown in steps S602 and S603, at the time of acquiring the first angle of view, that is, the first long-shot image, short-second exposure shooting is performed only after long-second exposure shooting.

  Returning to FIG. 5, when the parallel processing of the photographing and the background only panorama image generation in step S503 described in detail in FIG. 6 is completed, the processing shifts to the star only panorama generation processing (step S504).

  Hereinafter, the star-only panorama image generation processing in step S504 will be described in detail with reference to the flowchart in FIG. 8 and the conceptual diagram of the image processing in FIG.

  In FIG. 8, first, a counter N is initialized to 1 (step S801).

  Subsequently, a difference image is extracted by the difference image extraction circuit 116m from the Nth long-time captured image and the comparison dark combination image that is the combination result up to the (N + 1) th image to generate the Nth comparison dark combination image ( Step S802).

  For example, in the case of N = 1, as shown in FIG. 9, the difference image extraction circuit 116m extracts the first long-time captured image 905 and the comparative dark combined image 919, which is the result of combining up to the second image. Then, an image 922 is generated for only the first star. The star-only image 922 is an image in which only the star in the superimposition region of the long-second captured image 905 remains as a difference image.

  Next, the value of the counter N is incremented (step S803).

  Thereafter, when the long-time photographed image used in step S802 satisfies N> n, that is, until the (N-1) th image becomes the nth (last) long-time photographed image (NO in step S804). The processing from S802 is repeated. Thereby, as shown in FIG. 9, when N = 2, only the second star image 923 is generated, and when N = 3, the third star only image 924 is generated.

  Thereafter, if it is determined in step S804 that the last long-second captured image has been reached, the process proceeds to step S805, and the counter N is initialized to 2.

  Subsequently, the blur amount calculating circuit 116c calculates the blur amount of the stars between the (N−1) th and N-th stars only images (the amount of blur between the stars only) (step S806). For example, when N = 2, as shown in FIG. 9, the blur amount is calculated by the blur amount calculating circuit 116c from the images 922 and 923 of only the first and second stars.

  Next, based on the blur amount calculated in step S806, only the (N-1) th and Nth stars are image-aligned by the alignment circuit 116d (step S807). For example, when N = 2, as shown in FIG. 9, the alignment of the images 922 and 923 is performed only on the first and second stars.

  In step S808, the comparatively bright composite image and the N-th star only image, which are the result of combining the (N-1) th star-only images, which have been aligned in step S807, are compared and brightly combined using the combining circuit 116j. , And a comparatively bright composite image, which is the result of combining up to the Nth sheet. For example, when N = 3, as shown in FIG. 9, the comparative bright composite image 925, which is the composite result up to the second image, and the image 924 of the third star only are comparatively bright composited using the composite circuit 116j. A comparatively bright synthesized image 926, which is the synthesis result of up to the third sheet, is generated. Note that only in the case of N = 2, the image 922 of only the first star is used as a comparatively bright composite image which is the composite result up to the (N-1) th page.

  Next, the value of the counter N is incremented (step S809), and when N> n, that is, until the (N-1) th image is an image of only the nth (last) star (NO in step S810), step The processing from S806 is repeated, and this processing ends.

  As described above, according to the processing of FIG. 8, by incrementing the value of the counter N in order from 2 to n, for each star-only image extracted from the difference between each long-time captured image and the comparison dark composition result, only the star-only image , And the comparatively bright composition is repeated. As a result, a panoramic image can be generated only for stars in which the regions of star portions are gradually connected as in the comparative bright composite image 926.

  Returning to FIG. 5, when the panorama image generation processing for only the stars in step S504 described in detail in FIG. 10 is completed, the processing shifts to the starry sky panorama image generation processing (step S505).

  Hereinafter, the starry sky panorama image generation processing in step S505 will be described in detail with reference to the conceptual diagram of the image processing in FIG. 9 and the flowchart in FIG.

  10, first, a panoramic image of only the background (for example, the comparative dark composite image 921 of FIG. 9) and a panoramic image of only the stars (for example, the comparative bright composite image 926 of FIG. 9) are added and composited by the composite circuit 116j (step S1001). Thus, a starry sky panoramic image 927 is generated as shown in FIG.

  Subsequently, the generated starry sky panoramic image 927 is compressed into a general-purpose format such as Jpeg by the compression / expansion circuit 116l (step S1002), stored in the memory 120 (step S1003), and the process ends. At this time, it is also preferable to perform the gamma correction by the gamma correction circuit 116b in order to make the dark part of the starry sky panoramic image 927 easy to see, or to perform the correction in order to unify the color tone of the entire image. Since the image thus obtained has a large size, the image may be scaled by the scaling circuit 116f to the size designated by the user in advance. Further, it is preferable that the image is cut out from the maximum inscribed rectangle or a predetermined area by the trimming circuit 116g and then stored.

  Returning to FIG. 5, when the starry sky panorama image generation processing in step S505 described in detail in FIG. 10 ends, the entire processing of the starry sky panorama shooting processing ends.

  In the present embodiment, an example is shown in which the image pickup apparatus 100 swings in the horizontal direction, but it may swing in the vertical direction.

  As described above, in the starry sky panorama shooting, even when the star serving as the subject moves between shootings having different angles of view, both the background and the star can be aligned.

  In the present embodiment, in the star-only panorama image generation process in step S504 of the starry sky panorama shooting process in FIG. 5, when the calculation of the blur amount between the star-only images fails, an appropriate warning is displayed to the user, and the comparison darkness is displayed. Perform alternatives to the compositing process. Hereinafter, this embodiment will be specifically described with reference to FIGS. 11, 12, and 13. FIG.

  The present embodiment is different from the first embodiment only in the panorama image generation process for only the stars, but the other processes and the hardware configuration are the same as those in the first embodiment, so the same configurations and steps are the same. The reference numerals are used, and the repeated description is omitted.

  In FIG. 11, first, after performing the processing of steps S801 to S806 in FIG. 8, the process proceeds to step S1101.

  In step S1101, whether the calculation of the amount of blur between the star-only images has succeeded or not, specifically, in the superimposition region in the M-1st and M-th star-only images 1101 and 1102 shown in FIG. It is determined whether the extraction of the feature point is successful (an integer of 2 ≦ M ≦ n). If the result of this determination is a success, the processing from step S807 in FIG. 8 is performed, and this processing ends.

  On the other hand, if the extraction of the feature points has failed (NO in step S1101), a warning is displayed to the user indicating that the alignment of the star-only image has failed (step S1102). The method of displaying the warning is not particularly limited, but is performed by, for example, displaying a notification screen shown in FIG.

  Subsequently, a combined image 1105 is generated from only the M-1 and M-th stars from the images 1101 and 1102 by using the combining circuit 116j (step S1103). At this time, in this embodiment, the image in the superimposed area of the star-only images 1101 and 1102 employs the image 1101 only for the (M−1) -th star, but the image 1102 only for the M-th star is employed. Good. Further, in such a combined image 1105, since a luminance difference is likely to occur at the boundary, processing such as filtering and blurring the boundary may be performed.

  Thereafter, the processing after step S809 is performed, and this processing ends.

  As described above, when the extraction of the feature point in calculating the blur amount between the star-only images fails, the combining process of the two images is performed. Accordingly, it is possible to prevent a situation in which a starry sky panoramic image cannot be generated even though the user has taken a plurality of times taking a long time in the parallel processing of the panorama image generation and the imaging in FIG.

  On the other hand, when such a combining process is performed, it may be assumed that the alignment of only the stars in the starry sky panoramic imaging fails. Therefore, a starry sky panorama image is generated while displaying a warning in advance immediately before performing the combining process. This makes it possible to improve user convenience.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist.

  As described above, the present invention has been described in detail based on the preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms that do not depart from the gist of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined. Further, when a software program for realizing the functions of the above-described embodiments is supplied to a system or an apparatus having a computer capable of executing the program directly from a recording medium or using wired / wireless communication, and the program is executed. Is also included in the present invention. Therefore, the program code itself supplied and installed in the computer to implement the functional processing of the present invention by the computer also implements the present invention. That is, the present invention includes the computer program itself for implementing the functional processing of the present invention. In this case, any form of the program, such as an object code, a program executed by an interpreter, and script data to be supplied to the OS, may be used as long as the program has the function of the program. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used. As a method of supplying the program, a method in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program is also conceivable.

REFERENCE SIGNS LIST 100 imaging device 116 video signal processing circuit 116 c shake amount calculation circuit 116 d alignment circuit 116 j synthesis circuit 116 m difference image extraction circuit 123 microcomputer 124 operating member 125 release switch 133 gyro

Claims (10)

In an imaging apparatus that generates a starry sky panoramic image using n long-time shot images obtained by long-time exposure shooting at different angles of view,
An acquisition unit configured to acquire first and second short-time shot images by performing short-time exposure shooting before and after the long-time exposure shooting at each of the different field angles;
A first shake amount for calculating a shake amount between view angles from the (N-1) -th second short-time photographed image (N is an integer from 2 to n) and the N-th first short-time photographed image Calculating means;
A first alignment unit configured to perform alignment of the N-th long-second captured image with respect to the (N-1) -th long-second captured image using the inter-field-angle blur amount;
The N-th long-shooted image that has been sequentially aligned and the comparative dark-synthesized image that is the result of synthesizing the long-shooted images up to the (N-1) -th image are compared and dark-synthesized, and only the background is obtained. First synthesizing means for generating a panoramic image;
Difference image extracting means for extracting a difference image between the (N-1) -th long-time photographed image and a comparative dark-synthesized image which is a synthesis result up to the N-th frame as an image of only the (N-1) -th star;
Second blur amount calculating means for calculating a star-only inter-image blur amount from the (N-1) th star-only image and the N-th star-only image;
A second alignment unit that performs alignment of the N-th star-only image with respect to the (N-1) -th star-only image using the star-only inter-image blur amount;
The N-th star-only image in which the alignment has been sequentially performed and the comparative bright composite image, which is a composite result of the (N-1) -th star-only image, are comparatively bright composited, and only the star panoramic image is obtained. Second synthesizing means for generating
An imaging apparatus comprising: a third combining unit that adds and combines the background-only panoramic image and the star-only panoramic image to generate a starry sky panoramic image. Extracting means for extracting feature points from respective superimposed areas of the (N-1) th and Nth star-only images,
2. The imaging apparatus according to claim 1, wherein the second shake amount calculation unit calculates an image shake amount between the stars only based on the extracted feature points. 3.   If the extraction unit fails to extract the feature points from the respective superimposed regions of the (M−1) th and Mth star-only images (2 ≦ M ≦ n), the second The synthesizing unit is configured to perform the M-1 th star-only image without performing comparative bright synthesizing with the comparatively bright synthesized image that is a result of synthesizing the M-1 th star-only image. 3. The imaging apparatus according to claim 2, wherein a combined image with the star-only image is generated.   4. The image pickup apparatus according to claim 3, further comprising: a notifying unit that notifies a user that the positioning of the M-th only star has failed when the extracting unit fails. .   4. The method according to claim 1, wherein the obtaining unit performs the short-time exposure shooting only after the long-time exposure shooting for the first field angle of the different field angle. 5. Imaging device.   The imaging device according to claim 1, wherein the sensitivity at the time of the short-second exposure shooting is higher than the sensitivity at the time of the long-second exposure shooting.   When N = 2, the first synthesizing unit uses the first long-shot image as a comparative dark synthetic image that is a synthesis result of the (N-1) -th long-shot image. The imaging device according to claim 1, wherein:   When N = 2, the second combining unit uses the first star-only image as a comparatively bright combined image that is a combined result of the (N-1) th star-only image. The imaging device according to claim 1. In a control method of an image pickup apparatus that generates a starry sky panoramic image using n long-time captured images obtained by long-time exposure shooting at different angles of view,
An obtaining step of obtaining first and second short-time shot images by performing short-time exposure shooting before and after the long-time exposure shooting at each of the different field angles;
A first shake amount for calculating a shake amount between view angles from the (N-1) -th second short-time photographed image (N is an integer from 2 to n) and the N-th first short-time photographed image A calculating step;
A first alignment step of aligning the N-th long-second captured image with respect to the (N-1) -th long-second captured image using the inter-view-angle blur amount;
The N-th long-shooted image that has been sequentially aligned and the comparative dark-synthesized image that is the result of synthesizing the long-shooted images up to the (N-1) -th image are compared and dark-synthesized, and only the background is obtained. A first combining step of generating a panoramic image;
A difference image extracting step of extracting a difference image between the (N-1) -th long-time captured image and a comparison dark combined image that is a synthesis result up to the N-th image as only the (N-1) -th star image;
A second blur amount calculating step of calculating a blur amount between the stars only from the (N-1) th star-only image and the N-th star-only image;
A second alignment step of aligning the N-th star-only image with respect to the (N-1) -th star-only image using the star-only inter-image blur amount;
The N-th star-only image in which the alignment has been sequentially performed and the comparative bright composite image, which is a composite result of the (N-1) -th star-only image, are comparatively bright composited, and only the star panoramic image is obtained. A second synthesis step of generating
A third combining step of adding and combining the background-only panoramic image and the star-only panoramic image to generate a starry sky panoramic image. A program for executing the control method according to claim 9.