JP6025342B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging technique for generating a composite image by connecting a plurality of images.

  There is a technique of taking a plurality of images while changing the angle of the camera, and obtaining a panoramic image by smoothly joining the left and right overlapping portions of each image by image processing.

  In addition, there is a method of generating an image having a wide dynamic range by photographing a plurality of images with few whiteouts and images with little blackout under different exposure conditions and combining them. This is called high dynamic range (HDR) synthesis. Patent Document 1 discloses a technique for generating a panoramic image by connecting the HDR images by continuously changing the area while generating an image without overexposure and blackout by the above-described HDR combining. There is disclosure.

JP 2004-180308 A

  However, in the above-mentioned Patent Document 1, since a plurality of images are taken at the same angle, when photographing a plurality of images, it is necessary to fix the camera body with a tripod or the like, and the plurality of images are taken and the camera angle is adjusted (moved).・ Fixing) must be repeated, and complicated procedures are required.

  In addition, images with a wide dynamic range are generated for each angle, and the brightness of the images is readjusted when the images are connected. Therefore, in the finished panoramic image, the fixed angle joint or fixed angle brightness is uncomfortable. May remain.

  The present invention has been made in view of the above problems, and an object of the present invention is to realize a technique that enables shooting that covers a wide dynamic range with a simple operation and that can obtain a panoramic image that does not cause a sense of incongruity in brightness at a joint. That is.

In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging unit that changes an exposure condition in accordance with a change in angle and sequentially changes a plurality of exposure conditions to image a subject. A first composition for generating a plurality of first composite images by joining together images having the same exposure condition among a plurality of images photographed by the imaging means as one group and connecting the images for each group. And second combining means for combining the plurality of first combined images generated by the first combining means to generate one second combined image with an expanded dynamic range; Have.

  According to the present invention, it is possible to perform photographing that covers a wide dynamic range with a simple operation, and it is possible to obtain a panoramic image that does not cause discomfort in brightness at a joint.

1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention. The figure which shows the concept of the panoramic photography of the digital camera of this embodiment. The figure which shows operation | movement of the angular velocity sensor of the digital camera of this embodiment. 6 is a flowchart showing a shooting operation in a panoramic mode by the digital camera of the present embodiment. The conceptual diagram which determines the exposure conditions of the digital camera of this embodiment. FIG. 3 is a conceptual diagram of panoramic synthesis and HDR synthesis of the digital camera of the present embodiment.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

  [Embodiment 1] Embodiment 1 according to the present invention will be described below.

  <Apparatus Configuration> First, with reference to FIG. 1, an outline of the configuration and functions of an imaging apparatus according to an embodiment of the present invention will be described.

  In FIG. 1, the photographic lens 101 captures a subject image and forms it on the imaging screen of the imaging element 103. The diaphragm 102 limits the amount of light of the subject image incident from the photographing lens 101 to a predetermined amount. In the digital camera of the present embodiment, the zoom position is fixed at a focal length corresponding to 28 mm in terms of a 35 mm film equivalent in the focal length of the taking lens 101 in a panoramic shooting mode to be described later. A subject image formed by the image sensor 103 is converted into an electric signal by a photoelectric conversion element constituting a CMOS image sensor or the like, converted into a digital signal by an A / D converter 104, and output to the image processing unit 105. The image processing unit 105 performs gamma correction, white balance correction, noise reduction processing, and the like on the input image signal, and outputs the result to the image data bus 106 as uncompressed image data.

  The photometric sensor 111 detects the luminance distribution in the range of the imaging screen and outputs EV value data to the control bus 116. The angular velocity sensor 112 detects the angular velocity generated when the user changes the angle of the camera body for each of the X (yaw) and Y (pitch) directions orthogonal to the camera optical axis, and detects the detected angular velocity data as X, The data is output to the control bus 116 every two channels of Y. The liquid crystal panel 107 is a display unit that displays images and various types of information.

  The first SDRAM 109 accumulates uncompressed image data for a plurality of frames, and provides a memory space that is a work area for performing panorama synthesis and HDR synthesis described later. The codec 110 compresses and encodes the uncompressed image data, which has been combined by the first SDRAM 109, as a still image, and generates JPEG or RAW data. JPEG is an 8-bit gradation general-purpose recording method that is compatible with reproduction. RAW is 14-bit gradation data that has not been subjected to image processing by the image processing unit 105 and is encoded using a lossless compression method. This is a recording method that is compatible with reproduction on a specific device.

  The second SDRAM 113 provides a buffer memory space for writing or reading the JPEG data generated by the codec 110 with respect to the recording medium 114 and arbitrating with the recording speed of the recording medium 114. The second SDRAM 113 also serves as an image display memory (video memory). The liquid crystal driver 108 converts the image display data stored in the second SDRAM 113 into a liquid crystal display signal and supplies it to the liquid crystal panel 107. In this way, the display image data written in the second SDRAM 113 is displayed on the liquid crystal panel 107 via the liquid crystal driver 108.

  A digital signal that has been A / D converted by the A / D converter 104 and accumulated in the second SDRAM 113 is converted into a liquid crystal display signal by the liquid crystal driver 108 and sequentially transferred to the liquid crystal panel 107 for display. Thus, the liquid crystal panel 107 functions as an electronic viewfinder that can display a through image. The recording medium 114 is a memory card configured with, for example, a NAND flash memory that can be attached to and detached from the camera body. JPEG or RAW data is created in accordance with a PC compatible FAT (File Allocation Table) file system that is recognized as a disk drive when connected to a PC, and is recorded on the recording medium 114.

  The operation key 115 is a switch for receiving various operations from the user, and selects either a shutter button for instructing to capture a still image, a panorama mode for performing panorama shooting, or a normal shooting mode for performing normal still image shooting. Includes a mode switch. The control bus 116 is a signal path for communicating control signals between the blocks. The control bus 116 transmits a control signal from the CPU 118 to each block, and transmits a response signal from each block and a data signal from each sensor to the CPU 118. The flash ROM 119 is an electrically erasable / recordable nonvolatile memory, and stores constants, programs, and the like for the operation of the CPU 118. Here, the program is a program for executing a flowchart described later in the present embodiment. The CPU 118 controls the entire camera. By executing a program recorded in a flash ROM 119, which will be described later, each process of the embodiment described later is realized. Reference numeral 117 denotes a system memory, which uses a RAM. In the RAM 117, constants and variables for the operation of the CPU 118, a program read from the flash ROM 119, and the like are expanded. The CPU 118 also controls display by controlling the liquid crystal driver 108 and the like.

  FIG. 2 shows a state in which a panoramic image that is horizontally long is photographed by the digital camera of the present embodiment. For example, when a photograph is taken while continuously moving the camera angle so that the photographer rotates from left to right while pressing the shutter button, a plurality of images with overlapping regions are obtained. The imaging apparatus connects the obtained images to generate a single panoramic image.

  <Shooting Operation> Next, with reference to FIGS. 3 and 4, the shooting operation in the panorama mode by the digital camera of this embodiment will be described.

  FIG. 3 shows the state of the digital camera of the present embodiment and the output of the angular velocity sensor 112 according to the state. The rotation speed in the yaw direction of the camera is detected as X output, and the rotation speed in the pitch direction of the camera is detected as Y output. Then, the amount of rotation in each direction can be calculated by taking the integral value of each output.

  FIG. 4 is a flowchart showing the operation in the panoramic shooting mode by the digital camera of the present embodiment. Note that the processing in FIG. 4 is realized by the CPU 118 developing and executing the control program stored in the ROM 119 in the work area of the RAM 117.

  In FIG. 4, the CPU 118 starts processing when the camera is turned on while the mode switch is in the panorama mode or when the mode switch is switched to the panorama mode by the user's operation.

  First, in S401, the CPU 118 determines whether the shutter button is half-pressed. If the shutter button is half-pressed, the luminance of the subject is measured by the photometric sensor 111 in S402. The photometric sensor is an external light type, is installed on the upper surface of the camera body, has a luminance measurement range equivalent to the field angle range for panoramic photography, and can measure the maximum luminance and the minimum luminance within the measurement range. The photometric sensor is not limited to this, and the luminance value may be measured from a captured image captured from the image sensor 103.

  In S403, the CPU 118 detects the maximum luminance and the minimum luminance of the subject in the shooting range, and in S404, the standard dynamic range image necessary for synthesizing the HDR image based on the luminance difference between the maximum luminance and the minimum luminance. The required number N is calculated. Here, the required number N is not limited to the luminance difference between the maximum luminance and the minimum luminance, but may be, for example, the luminance corresponding to the appropriate exposure, the maximum luminance, the luminance corresponding to the appropriate exposure, and the minimum luminance. The required number N may be set in advance by the user.

  In S405, the CPU 118 calculates the exposure condition shift amount set differently from the required number N. FIG. 5 is a conceptual diagram for determining the required number N and the shift amount of the exposure condition in S404 and S405. Note that EV is an abbreviation of Exposure Value (exposure index), which is the amount of light that can sensitize a ISO 100 film to 18% gray (provides proper exposure). When the aperture value is F1.0 and the shutter speed is 1 second, EV = 0, and the EV value increases by 1 each time the aperture value of the film is doubled or the shutter speed (exposure time) is halved. . There is no single combination of aperture value and shutter speed for the same EV. As shown in FIG. 5, for example, to obtain a proper exposure for a light quantity of 7 EV, F4.0 · 1/8 second, F2 There are exposure conditions such as .8 · 1/16 seconds and F2.0 · 1/30 seconds. The width of the dynamic range is also expressed by an EV value. For example, 10 EV indicates a dynamic range that can cover a luminance difference of 2 10 = 1024 times. The required number N and the exposure condition shift amount are calculated by the following equations.

N = ((maximum luminance−minimum luminance) + luminance overlap amount × (N−1)) / dynamic range of the image sensor (1)

N = (maximum luminance−minimum luminance−luminance overlap amount) / (dynamic range of image sensor−luminance overlap amount) (2)

Exposure shift amount = ((maximum luminance−minimum luminance) −dynamic range of image sensor) / (N−1) (3)

Assuming that the standard dynamic range by the image pickup device of this embodiment is 10 EV and the luminance overlap amount is 6 EV, in S403, when the maximum luminance is 20 EV and the minimum luminance is 2 EV (equation 2), N = 3 is calculated. . That is, the required number N is determined according to the result of dividing the luminance difference between the maximum luminance and the minimum luminance by the dynamic range of the image sensor. Note that the luminance overlap amount is an amount by which the range of luminance that can be covered by the image sensor is overlapped. Also, the exposure condition shift amount determined in S405 is 4EV from Equation 3. As a result, EV values for determining the exposure conditions for the three areas are determined. The standard exposure condition is 11 EV, the under exposure condition is 15 EV, and the over exposure amount is 7 EV.

  Next, in S406 and S407, the CPU 118 determines an aperture value and a shutter speed. In the present embodiment, the change of the exposure condition performed when obtaining a plurality of standard dynamic range images is performed by changing the aperture value. When the shutter speed is fixed between the exposure conditions, and the speed at which the photographer shakes the camera is almost constant, the degree of motion blur is substantially unified between images. In addition, N aperture values are determined for one shutter speed. In addition, conditions for obtaining the highest shutter speed are set within the limits of the aperture value and shutter speed that can be set on the camera side. That is, among the combination of the aperture value and the shutter speed, which is 7 EV that is the over-side exposure condition, the shutter speed 1/60 seconds in the case of F1.4 where the aperture value is full is the common shutter speed, The aperture value for realizing each exposure condition is F1.4 on the over side, F5.6 on the standard side, and F22 on the under side. In the present embodiment, the shutter speed is determined from the photographing conditions such as the photometric value. However, the present invention is not limited to this, and the speed at which the photographer's camera is shaken may be considered in addition to the photographing conditions. The speed at which the photographer shakes the camera may be measured by the angular velocity sensor 112, or the speed may be input and set in advance. When measuring the speed, the UI is instructed to shake the camera assuming the speed at the time of shooting before panoramic shooting is started, and a value measured in advance is used at the time of panoramic shooting. In S408, the shutter speed corresponding to the speed at which the photographer shakes the camera (allowing motion blur or the like) is set.

  In S408, the CPU 118 determines whether or not the shutter button is fully pressed. If the shutter button is fully pressed, the value of the variable P indicating the number of shots in a series of continuous shooting performed while the user continues to press the shutter button is 0. (S409).

  Next, in S410 and S411a to S411c, the CPU 118 sets the aperture value calculated in S406 to the aperture 102. In S410, the CPU 118 calculates a remainder obtained by dividing the variable P by the required number N. In the present embodiment, if N = 3 and the remainder is 0, the aperture 102 is set to F22 of the aperture value, which is the smallest exposure amount, in S411a. If the remainder is 1, in S411b, the aperture 102 is set to F5.6, which is an aperture value that is brighter than the aforementioned F22 by the shift amount 4EV. Further, if the remainder is 2, in S411c, the aperture 102 is an aperture value that becomes brighter by the shift amount 4EV than the aforementioned F5.6, and is set to F1.4 that is the open value.

  In step S412, exposure of the subject is started by the image sensor 103, exposure at the shutter speed calculated in step S407 ends, and the image pickup signal is transferred. The transferred imaging signal is A / D converted by the A / D converter 104, subjected to image processing such as gamma correction, white balance correction, noise reduction processing, etc. by the image processing unit 105, and then temporarily stored in the first SDRAM 109. Remembered.

  In S413, the value of the variable P is incremented. In S414, the CPU 118 determines whether the user has shaken the shooting angle by 10 degrees horizontally as a compositing unit for panoramic composition. It is determined whether the user has released the shutter button. If it is determined in S415 that the shutter button has not been released and the shutter button has been continuously pressed, the process proceeds to S416. In S416, the CPU 118 determines whether the shooting angle swing amount by the user after the start of shooting exceeds 270 degrees, which is the maximum angle of view that can be shot in the panorama mode of the present embodiment. Return.

  Note that if the user releases the shutter button in S415, or if it is determined in S416 that the shooting angle swing amount by the user after starting shooting has reached 270 degrees, the imaging operation ends. That is, panorama image generation processing is performed in S417 and a panorama image is generated using the captured image temporarily stored in the first SDRAM 109, HDR image synthesis processing is performed in S418, and finally the recording medium 114 is recorded. A recorded panoramic image is generated.

  Thereafter, in S419, the CPU 118 JPEG or RAW-encodes the panoramic image data generated by the codec 110, and the encoded panoramic image is recorded in the recording medium 114 in S420, and the series of operations ends in S421. To do.

  <Panorama Composition and HDR Composition> Next, the panorama composition in S417 (first composition) and the HDR composition in S418 (second composition) will be described.

  FIG. 6 is a conceptual diagram of panorama synthesis and HDR synthesis. Reference numeral 601 denotes a group of captured images temporarily stored in the first SDRAM 109. With the above-described configuration, shooting is performed every time the horizontal angle changes by 10 degrees. A total of 27 images have been taken. During a series of shootings, the aperture value set in the aperture 102 is changed for each shooting, and the same aperture value is set for each of N shots. In the photographed image group 601 shown in the figure, N = 3, as described above, the group of images photographed with the aperture value of F22 is group A, and the group of images photographed with the aperture value of F5.6 is group B, F1. A group of images taken with an aperture value of. In each group, shooting is performed under the corresponding conditions once every three frames. Therefore, shooting is performed with the same aperture value every time the group moves 30 degrees in the horizontal angle of view.

  On the other hand, as described above, since the focal length of the taking lens 101 is equivalent to 28 mm in terms of 35 mm film, the horizontal angle of view covered by one imaging screen is about 65 degrees. Therefore, images taken in each group are taken with an overlap of 17.55 degrees at both ends every 30 degrees. In the panorama synthesis, three sets of panorama composite images (first composite images) are generated by performing blurring processing such as filter processing so that the joints are not conspicuous using the overlap portion.

  Reference numeral 602 denotes a panoramic image obtained by synthesizing the images picked up in the group A, 603 denotes a panoramic image obtained by synthesizing the images picked up in the group B, and 604 denotes a panoramic image obtained by synthesizing the images picked up in the group C. Each panoramic image is composed of 10-bit gradation.

  In HDR composition performed after panorama composition, first, the brightness of the panorama image 602 obtained by compositing the images picked up in group A is increased by an exposure shift amount of 4 EV. Further, the brightness of the panoramic image 604 obtained by combining the images picked up in the group C is lowered by 4 EV which is the exposure shift amount. Thereafter, the panoramic image 603 obtained by combining the images picked up in the group B is joined to the portion without whiteout or blackout, and an HDR image (second composite image) having a gradation wider than 10 bits is obtained. Generate. The HDR image is subjected to contrast compression, rounded to 8-bit gradation in recording by JPEG encoding, and rounded to 14-bit gradation in recording by RAW encoding. Encoding is performed and recording is performed on the recording medium 114.

  As described above, in this embodiment, among the plurality of images captured by the imaging device for each exposure condition while moving the angle, images having the same exposure condition are grouped into images for each group. To create a plurality of panoramic images. Then, a plurality of generated panoramic images are combined to generate a single panoramic image with an expanded dynamic range. Accordingly, it is possible to perform shooting that covers a wide dynamic range by the same operation as conventional panoramic shooting, and it is possible to perform panoramic shooting that does not cause a difference in brightness at the joint of panoramic images. Even in groups with different exposure conditions, since the shutter speed is the same, the amount of lateral blur caused by shooting while swinging matches in each panoramic image with 10-bit gradation, so in S418. The HDR synthesis process becomes easy.

  In the digital camera of the present embodiment, the shutter speed is calculated with the aperture value opened in order to obtain the overexposure amount. However, when the subject is very dark, the shutter speed becomes very slow. (For example, 1/15 or less), signal amplification means for amplifying the imaging signal from the imaging element 103 is provided in the preceding stage of the A / D converter 104, and the same amplification amount and imaging are provided in all groups. A configuration for performing a process of increasing the sensitivity of the element may be added. In this case, since the noise level of the image pickup device also increases, if the noise reduction processing is performed stronger than usual in the image processing unit 105, the configuration is still effective. By sensitizing the image sensor, it is possible to set a higher shutter speed and to reduce blurring of the captured image.

  In the present embodiment, the HDR processing of the panoramic image has been described. However, the same problem occurs in the image composition processing in which a plurality of images are joined and combined, and the present invention can be applied. For example, the present invention can be applied not only to the left / right or upper / lower direction but also to a composition process for generating an image as if it was shot with a wider angle of view than a single image by connecting a plurality of images.

  [Other Embodiments] The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. It is processing to do. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (11)

An imaging means for imaging the subject by changing the exposure conditions in accordance with the change of the angle , and sequentially changing the plurality of exposure conditions;
A first image that generates a plurality of first composite images by joining together images having the same exposure condition among a plurality of images captured by the imaging unit as a group, and connecting the images for each group. Combining means;
Second combining means for combining the plurality of first combined images generated by the first combining means to generate one second combined image with an expanded dynamic range. An imaging apparatus characterized by that.   The imaging apparatus according to claim 1, further comprising a determining unit that determines the plurality of exposure conditions by changing an aperture value of the imaging unit.   The imaging apparatus according to claim 2, wherein the determining unit makes the exposure time of the imaging unit constant when changing the plurality of exposure conditions.   The imaging apparatus according to claim 3, wherein an exposure time of the imaging unit is set so that motion blur of a subject image due to the change in the angle is allowed.   5. The imaging apparatus according to claim 1, wherein the imaging unit changes the plurality of exposure conditions each time the angle changes by a predetermined angle. 6. A determining unit that determines the plurality of exposure conditions according to an aperture value and an exposure time of the imaging unit;
The imaging apparatus according to claim 1, wherein the determination unit sets an exposure condition with the same exposure time and a different aperture value for each group. A signal amplifying means for amplifying the image signal obtained by the imaging means;
The imaging apparatus according to claim 6, wherein the determination unit sets an exposure condition with the same exposure time and the same amplification amount for each of the groups, and with different aperture values. A photometric means for measuring the luminance of the subject;
The determining unit detects a difference between the maximum luminance and the minimum luminance of the shooting range from the luminance of the subject obtained by the photometry unit, and according to a result obtained by dividing the detected luminance difference by a dynamic range of the imaging unit. The imaging apparatus according to claim 6, wherein the number of the groups is determined. A control method for an image pickup apparatus having an image pickup means for picking up an image of a subject by changing exposure conditions in accordance with an angle change and sequentially changing a plurality of exposure conditions,
A first image that generates a plurality of first composite images by joining together images having the same exposure condition among a plurality of images captured by the imaging unit as a group, and connecting the images for each group. A synthesis process;
A second synthesis step of synthesizing the plurality of first synthesized images generated by the first synthesis step to generate one second synthesized image having an expanded dynamic range. A control method characterized by that.   The control method according to claim 9, further comprising a determining step of determining the plurality of exposure conditions by changing an aperture value of the imaging unit.   The control method according to claim 10, wherein, in the determining step, when changing the plurality of exposure conditions, an exposure time of the imaging unit is made constant.