JP2020120194A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

To provide a development method which prevents brightness of a composite image from varying remarkably in performing panoramic imaging of a subject whose luminance varies suddenly.SOLUTION: An imaging apparatus comprises: imaging means which takes plural images; adjustment means which applies a gradation gain to images adjacent to each other, out of plural images, on the basis of the difference in luminance between the images adjacent to each other; and composition means which generates a composite image by composing the images to which the adjustment means has applied the gradation gain.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image pickup apparatus for performing panoramic image pickup and generating a panoramic image, and more particularly to an image pickup apparatus for reducing a luminance step in a panoramic image.

Patent Document 1 and the like disclose a method of capturing a plurality of images while panning an image capturing device such as a digital camera, and joining the captured images to generate a panoramic image.

At this time, if the exposure is fixed and panoramic imaging is performed, uneven brightness on the composite image does not occur. However, depending on the subject, when the brightness greatly changes at the start point and the end point of the panoramic photograph, the exposure is made to follow the imaging.

However, in the case of capturing an image by following the exposure, there is a problem that a brightness difference occurs between images adjacent to each other and a brightness step is formed in a combined panoramic image.

To solve such a problem, Patent Document 2 proposes a method of capturing a plurality of images with different exposures, adjusting the luminance so that the entire image is properly exposed, and synthesizing a panoramic image.

JP, 2010-28764, A JP, 2014-230018, A

However, according to the method of Patent Document 2, it is necessary to capture a large amount of images that are not finally used for composition, and composition cannot be performed in real time.

The present invention has been made in view of the above problems, and reduces a luminance step in a panoramic image in order to apply a gradation gain to the captured image when performing panoramic imaging.

In order to solve the above problems, the present invention applies a gradation-like gain to an image based on a difference in luminance between adjacent images of the plurality of images and an image capturing unit that captures the plurality of images. There is provided an imaging device including: an adjusting unit; and a combining unit that combines the images after the adjusting unit applies the gradation-like gain to generate a combined image.

According to the configuration of the present invention, it is possible to provide an image pickup apparatus that creates a panoramic image with a small luminance level difference, without performing unnecessary image pickup even in a scene in which the brightness changes greatly.

It is a block diagram which shows the structure of the digital camera which concerns on embodiment of this invention. 6 is a flowchart for explaining generation of a composite image in the embodiment of the present invention. It is a figure for explaining gradation gain in an embodiment of the present invention. FIG. 6 is a diagram for explaining a composite image to which no gradation gain is applied according to the embodiment of the present invention. It is a figure for demonstrating the estimation of the amount of movement in a cylindrical shape in the embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing the structure of a digital camera as an image processing apparatus according to this embodiment.

The digital camera 100 is capable of capturing still images and combining images. Note that the digital camera 100 may be replaced with any electronic device having a camera function such as a mobile phone with a camera function, a computer with a camera function, as well as a camera such as a digital video camera. The optical system 101 includes a lens, a shutter, and a diaphragm, and forms light from a subject on the image sensor 102 under the control of the CPU 103. The image sensor 102 such as a CCD image sensor or a CMOS image sensor converts the light imaged through the optical system 101 into an image signal.

The CPU 103 realizes the function of the digital camera 100 by controlling each unit that constitutes the digital camera 100 according to an input signal or a program stored in advance. The primary storage device 104 is a volatile device such as a RAM, stores temporary data, and is used for the work of the CPU 103. The information stored in the primary storage device 104 may be used by the image processing unit 105, which will be described later, or recorded in the recording medium 106. The secondary storage device 107 is a non-volatile storage device such as an EEPROM, stores a program (firmware) for controlling the digital camera 100 and various setting information, and is used by the CPU 103.

The image processing unit 105 performs various image processing on the recorded image signal data, such as white balance adjustment, color interpolation, filtering, and compression processing according to a standard such as JPEG. The image processing unit 105 may be built in the CPU 103.

The recording medium 106 records image data and the like stored in the primary storage device 104 and obtained by imaging. The recording medium 106 can be detached from the digital camera 100, such as a semiconductor memory card, and the recorded data can be loaded into a personal computer or the like and read out. That is, the digital camera 100 has a mechanism for attaching/detaching the recording medium 106 and a read/write function. The display unit 108 displays a viewfinder image at the time of image capturing, a captured image, a GUI image for interactive operation, and the like. The operation unit 109 is an input device group that receives a user's operation and transmits input information to the CPU 103, and may be, for example, a button, a lever, a touch panel, or the like, or an input device using a voice, a line of sight, or the like.

The sensor 110 acquires movement information of the digital camera 100 itself. For example, an input device such as a gyro or an acceleration sensor can be used. The amount of movement and the angular velocity of the digital camera 100 itself can be obtained by the calculation by the CPU 103.

The digital camera 100 according to the present embodiment has a plurality of image processing patterns applied to the captured image by the image processing unit 105, and the patterns can be set as the imaging mode from the operation unit 109. The image processing unit 105 performs image processing called so-called development processing, and also adjusts color tones according to the imaging mode. Note that at least a part of the functions of the image processing unit 105 may be implemented by the CPU 103 as software.

FIG. 2 is a flowchart for explaining the generation of the synthetic image in this embodiment.

In step S201, the image sensor 102 captures the first image. The image pickup in step S201 is generally performed with the exposure set to the proper exposure.

In step S202, the image processing unit 105 develops the image captured by the image sensor 102 in step S201.

In step S203, the image sensor 102 captures the nth image. The variable n is given a value of 1 in step S201 and is incremented in step S205.

In step S204, the image processing unit 105 applies a gain to two adjacent images (nth image and n+1th image) to perform exposure adjustment so that the two images have the same brightness level. The exposure alignment in step S204 is for alignment, which will be described later. If the exposure is not adjusted, the accuracy of vector search, which will be described later, may deteriorate.

In step S205, the CPU 103 aligns two adjacent images (the n-th and n+1-th exposures that have been exposed in step S204). First, the CPU 103 calculates the amount of positional shift between the nth image and the (n+1)th image after exposure adjustment. First, the CPU 103 sets a plurality of blocks in the nth image. It is preferable to set the size of each block to be the same. Next, the CPU 103 sets a search range that is wider than the block of the nth image at the same position as each block of the nth image of the n+1th image. Finally, the CPU 103 corresponds to each search range of the (n+1)th image such that the sum of absolute differences between the brightness of the nth image and the block of the nth image (Sum of Absolute Difference, hereinafter, referred to as SAD) is minimum. Calculate the points. The CPU 103 calculates, as a vector, the displacement of the position described above from the center of the block of the reference image and the corresponding point described above. In the calculation of the corresponding points described above, the CPU 103 may use a sum of squared differences (hereinafter referred to as SSD) or a normalized cross correlation (hereinafter referred to as NCC) in addition to the SAD. Good.

Next, the CPU 103 calculates the conversion coefficient from the above-described shift amount. For example, a projective transformation coefficient is used as the transformation coefficient. However, the transform coefficient is not limited to the projective transform coefficient, and an affine transform coefficient or a simplified transform coefficient of only horizontal and vertical shifts may be used.

Finally, the CPU 103 transforms the (n+1)th image.

For example, the CPU 103 can perform the transformation using the formula shown in (Formula 1).

In (Equation 1), (x', y') indicates the coordinates after the deformation, and (x, y) indicates the coordinates before the deformation. The matrix A shows the deformation coefficient.

Through the above processing, the CPU 103 can perform alignment with respect to the nth image and the (n+1)th image.

In step S206, the image processing unit 105 calculates a gradation gain used for developing the image captured by the image sensor 102 in step S206. In step S207, the image processing unit 105 uses the gradation gain calculated in step S206 to develop the image captured by the image sensor 102 in step S203.

FIG. 3 is a diagram for explaining the gradation gain in this embodiment. FIG. 3A shows the relationship between the brightness and the position of three captured images having different brightness in panoramic imaging. When trying to connect images with different exposures as shown in FIG. 3A, a brightness difference occurs at the joints, so that a brightness step occurs on the composite image. Therefore, as shown in FIG. 3B, the CPU 103 determines a combination position (between n and the (n+1)th image) based on the movement amount in the image estimated in step S204, and a gradation-like image is formed for the (n+1)th image. By applying a gain and synthesizing, it is possible to make the luminance uniform at the synthesizing position (between the nth and (n+1)th sheet). By performing the same thing on the (n+2)th sheet, it is possible to make the luminance uniform at the combining position (between the (n+1)th and (n+2)th sheet). Further, as shown in FIG. 3C, the planned synthesis position (between the (n+1)th and (n+2)th) is calculated using the movement amount (between the nth and (n+1)th sheet) in the image, and the gradation gain up to that point There is also a way to apply. By adjusting the gradation gain curve of the (n+2)th sheet and the combining position (between the (n+1)th and (n+2)th sheet), the images can be connected more smoothly than that shown in FIG. 3B. The gradation-like gain in FIG. 3 shows a linear function, but the method is not limited to the above, and any method can be used as long as it is a method of applying the gradation gain in order to reduce the luminance step. In addition, in order to prevent a sudden change in brightness, a certain upper limit may be set for the slope of a straight line that means gradation gain.

FIG. 4 is a diagram for explaining a combined image to which no gradation gain is applied according to this embodiment. FIG. 4A shows an image in which a plurality of images with different exposures as shown in FIG. 3A are combined without gain adjustment. FIG. 4A shows that a luminance step is created in the composite image. On the other hand, FIG. 4B shows an image in which a plurality of images captured with the exposure fixed are combined. In the image shown in FIG. 4B, it can be seen that a large brightness difference is generated in the image, and the left end of the image is too bright and the right end is too dark, and the observer cannot see it well.

In step S208, the CPU 103 determines whether to end the image capturing. The criterion here is whether a predetermined number of captured images has been reached, a width of the composite image has been reached, a set position has been reached, the upper limit of the capacity of the recording medium 106 or the set capacity has been reached. , Etc. When the imaging is ended, the process proceeds to step S212, and when the imaging is not ended, the process returns to step S203.

In step S209, the image processing unit 105 combines the captured images to generate a panoramic image. In the combining process, the image processing unit 105 preferentially uses the part to which the gradation gain is applied in the combined image between the images in the combined image. For example, when creating a composite image using an image as shown in FIG. 3B, a gradation gain is applied when synthesizing the part from the composite position of the nth and (n+1)th images to the right end of the nth image. A corresponding portion of the composite image is created using the (n+1)th portion.

In order to perform the composition, it is necessary to align the images, but the deformation coefficient detected in step S205 may be used, or the deformation coefficient may be calculated anew.

In addition, in the image combination in step S209, when the exposure difference between the images is wide, the CPU 103 can more smoothly change the brightness of the combined image when performing the alpha blending.

Further, the above-mentioned information on the deformation coefficient can be estimated from the angular velocity acquired by the sensor 110 and the like. As an example, there is a case where image synthesis is performed by pasting on a cylinder.

FIG. 5 is a diagram for explaining the estimation of the movement amount in the cylindrical shape in the present embodiment. The description will be made assuming that the rotation angle is θ, the focal length is r, the sensor size is d, and the planned composition position is the center of the compositable area between the two sheets. In this case, φ=θ/2, where φ is the angle from the center of the image of the first sheet to the planned composition position. Furthermore, since the image movement amount to the planned composition position is 2*r*tanφ=2*r*tan(θ/2), if the rotation angle can be acquired by the sensor 110, the deformation coefficient can be estimated.

According to the present embodiment, when performing panoramic imaging, it is possible to create a composite image in which the brightness is not significantly changed for a subject whose brightness changes greatly without capturing a large amount.

(Other embodiments)
In the above embodiment, the description was given based on a personal digital camera, but if a depth stacking function is installed, a mobile device, a smartphone, or a network camera connected to a server, etc. It is also possible to apply. Alternatively, a part of the processing described above may be performed by a mobile device, a smartphone, a network camera connected to a server, or the like.

It should be noted that the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in a computer of the system or apparatus execute the program. Can also be realized by a process of reading out and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital Camera 101 Optical System 102 Image Sensor 103 Central Processing Unit (CPU)
104 primary storage device 105 image processing unit 106 recording medium 107 secondary storage device 108 display unit 109 operation unit 110 sensor

Claims (8)

An image capturing means for capturing a plurality of images,
Of the plurality of images, based on a difference in brightness between adjacent images, an adjusting unit that applies a gradational gain to the images,
An image pickup apparatus, comprising: a synthesizing unit that synthesizes the image after the adjusting unit applies the gradation-like gain to generate a synthesized image. The imaging apparatus according to claim 1, wherein the imaging unit captures the plurality of images while panning. The image pickup apparatus according to claim 1, wherein the synthesizing unit uses a portion of the image after the gradation-like gain is applied to a corresponding portion of the synthetic image. 4. The image pickup apparatus according to claim 1, wherein the adjusting unit applies the gradation-like gain having a linear function relationship with the position of the image. The adjusting unit applies the gradation-like gain based on a difference in brightness between the adjacent images, so that the connection between corresponding portions of the adjacent images in the composite image becomes smooth. The imaging device according to any one of claims 1 to 4. The image pickup apparatus according to claim 1, wherein the synthesizing unit performs alpha blending in the synthesizing. An imaging step of capturing a plurality of images,
Among the plurality of images, an adjusting step of applying a gradation gain to the images based on a difference in brightness between adjacent images,
A synthesizing step of synthesizing the image after applying the gradation-like gain in the adjusting step to generate a synthesized image. A computer program that causes a computer of an imaging device to operate,
An imaging step of capturing a plurality of images,
Among the plurality of images, an adjusting step of applying a gradation gain to the images based on a difference in brightness between adjacent images,
A program that causes the image after being subjected to the gradation-like gain in the adjusting step to be combined to generate a combined image.

Images