JP7051586B2 - Imaging equipment, imaging methods and programs - Google Patents

Description

The present invention relates to an image pickup device that creates a panoramic image by synthesizing a plurality of images. In particular, the present invention relates to an imaging device that creates a panoramic image in a scene where the exposure changes during imaging.

Panorama imaging is widely performed in which a plurality of images continuously captured while the image pickup device is panning are aligned and combined to form an image having a wider angle of view (Patent Document 1). In panoramic imaging, it is common to fix the exposure condition to the appropriate exposure of the main subject existing at the imaging angle of view and capture the entire angle of view so that the composite image does not have unnaturalness due to the step of exposure.

Japanese Unexamined Patent Publication No. 11-46317

However, the following problems may occur by using the method described in Patent Document 1. FIG. 9 is a diagram for explaining panoramic imaging in a scene in which the exposure of a subject changes significantly. FIG. 9A shows a scene in which the exposure of the subject changes significantly during panoramic imaging (a landscape scene with the setting sun as the main subject). In FIG. 9A, the panoramic image 900 is imaged with the exposure conditions obtained by metering the central region 901 fixed. However, when the region 902 with the proper exposure different from the proper exposure in the region 901 is imaged, the exposure is not the proper exposure, and when the region 902 is imaged with the proper exposure, the region 903 with the signal is very low signal ( Hereinafter, it is expressed as blackout). In the blackened area, the feature points for alignment cannot be obtained, which makes alignment difficult. Therefore, in a scene where the exposure of the subject changes significantly during imaging, it is difficult to align the subject, and it may not be possible to create a composite image. On the other hand, in a scene where a dark subject is the main subject (such as a landscape scene where the tunnel is the main subject), the signal is almost saturated when the image is taken with proper exposure (hereinafter referred to as overexposure). Therefore, it becomes difficult to align.

The present invention has been made in view of the above problems, and provides an imaging device capable of creating a panoramic image by reducing misalignment in a scene where the exposure changes significantly.

In order to solve the above problems, the present invention has at least an image sensor, an image pickup means for capturing a plurality of images, and a first region and a second region in each image of at least a part of the plurality of images. The setting means for setting the region and the third region, the adjusting means for adjusting the exposure between the second region and the third region, and the said for each of at least a part of the plurality of images. The image has an alignment means for aligning with respect to the first region and a synthesis means for synthesizing with respect to the first region after the alignment means has performed the alignment. When the exposure of the second region of the image and the exposure of the third region of the image of the image in the previous frame of the image are not the same, the adjusting means measures the exposure of the second region and the third region. After adjusting so that the exposure of the image is the same, the alignment means aligns the second region of the image with the third region of the image of the previous frame of the image. Provided is an image pickup device in which the alignment means performs alignment with respect to the first region using the coefficient.

According to the configuration of the present invention, it is possible to create a panoramic image even in a scene where the exposure changes greatly.

It is a rear perspective view which shows the schematic structure of the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the hardware composition of the digital camera which concerns on embodiment of this invention. It is a flowchart for demonstrating panoramic photography in 1st Embodiment. It is a figure for demonstrating the alignment in 1st Embodiment. It is a figure for demonstrating the modification example of the image sensor in 1st Embodiment. It is a flowchart for demonstrating panoramic photography in 2nd Embodiment. It is a figure for demonstrating image pickup and image processing in 2nd Embodiment. It is a figure for demonstrating the modification example of the image sensor in 2nd Embodiment. It is a figure for demonstrating panoramic imaging in a scene where the exposure of a subject changes greatly.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a rear perspective view showing a schematic configuration of a digital camera 100 according to an embodiment of the present invention.

On the back surface of the digital camera 100, a display unit 101 for displaying images and various information, and an operation unit 102 composed of operation members such as various switches and buttons for receiving various operations by the user are provided. Further, on the back surface of the digital camera 100, a mode changeover switch 104 for switching an imaging mode or the like and a controller wheel 103 capable of rotation operation are provided. On the upper surface of the digital camera 100, a shutter button 121 for instructing an image pickup, a power switch 122 for switching the power of the digital camera 100 on / off, and a strobe 141 for irradiating a subject with a flash of light are provided.

The digital camera 100 can be connected to an external device via wired or wireless communication, and can output image data (still image data, moving image data) or the like to the external device. A recording medium slot (not shown) that can be opened and closed by a lid 131 is provided on the lower surface of the digital camera 100 so that a recording medium 130 such as a memory card can be inserted and removed from the recording medium slot. There is.

The recording medium 130 stored in the recording medium slot can communicate with the system control unit 210 (see FIG. 2) of the digital camera 100. The recording medium 130 is not limited to a memory card or the like that can be inserted into and removed from the recording medium slot, but may be a magnetic disk such as an optical disk or a hard disk, and is further built in the main body of the digital camera 100. It may have been done.

FIG. 2 is a block diagram showing a hardware configuration of the digital camera 100. The digital camera 100 includes a barrier 201, an image pickup lens 202, a shutter 203, and an image pickup unit 204. The barrier 201 covers the image pickup optical system to prevent the image pickup optical system from becoming dirty or damaged. The image pickup lens 202 is composed of a lens group including a zoom lens and a focus lens, and constitutes an image pickup optical system. The shutter 203 has a diaphragm function and adjusts the exposure amount to the image pickup unit 204. The image pickup unit 204 is an image pickup element that converts an optical image into an electric signal (analog signal), and is, for example, an image sensor such as a CCD sensor or a CMOS sensor having a Bayer array structure in which RGB pixels are regularly arranged. .. The shutter 203 may be a mechanical shutter (hereinafter referred to as a mechanical shutter) or an electronic shutter whose accumulation time is controlled by controlling the reset timing of the image pickup device.

Alternatively, if the image pickup unit 204 has a structure provided in a plurality of photoelectric conversion units in one pixel capable of acquiring a stereo image, the automatic focus detection (AF) process described later can be performed more quickly.

The digital camera 100 includes an A / D converter 205, an image processing unit 206, a memory control unit 207, a D / A converter 208, a memory 209, and a system control unit 210. An analog signal is output from the image pickup unit 204 to the A / D converter 205, and the A / D converter 205 converts the acquired analog signal into image data consisting of a digital signal, and the image processing unit 206 or the memory control unit 207. Output to.

The image processing unit 206 performs correction processing such as pixel interpolation and shading correction, white balance processing, γ correction processing, and color conversion for the image data acquired from the A / D converter 205 or the data acquired from the memory control unit 207. Perform processing etc. Further, the image processing unit 206 realizes an electronic zoom function by cutting out (trimming) an image and performing scaling processing. Further, the image processing unit 206 performs a predetermined calculation process using the image data of the captured image, and the system control unit 210 performs exposure control and distance measurement control based on the calculation result obtained in this way. For example, the system control unit 210 performs TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (strobe pre-flash) processing. The image processing unit 206 performs a predetermined calculation process using the image data of the captured image, and the system control unit 210 performs a TTL method AWB (auto white balance) process using the obtained calculation result.

The image processing unit 206 has an image composition processing circuit that synthesizes a panoramic image from a plurality of images and determines the result of the composition. The image composition processing circuit is not limited to simple additive average composition, but also comparative bright composition and comparison in which pixels having the brightest value or the darkest value in each area of the image data to be synthesized are selected to generate one image data. Processing such as dark composition can be performed. In addition, the synthesis result is evaluated and judged based on a specific criterion. For example, if the number of combined images does not meet the predetermined number, or if the length of the combined images does not meet the reference value, it is determined that the composition has failed.

The image processing unit 206 also has a compression / decompression function for converting the image data obtained by imaging into an image format such as JPEG. Other than that, the image processing unit 206 also has a function for realizing template matching described later. The details of template matching will be described later. Instead of the configuration including the image processing unit 206, the configuration may be such that the function of the image composition processing is realized by software processing by the system control unit 210.

The image data output from the A / D converter 205 is written to the memory 209 via the image processing unit 206 and the memory control unit 207, or via the memory control unit 207. The memory 209 also serves as an image display memory (video memory) for storing image data to be displayed on the display unit 101. The memory 209 has a storage capacity capable of storing a predetermined number of still images, panoramic images (wide-angle images), and panoramic image composition results. The memory 209 can also be used as a work area for developing a program or the like read from the non-volatile memory 211 by the system control unit 210.

The image display data (digital data) stored in the memory 209 is transmitted to the D / A converter 208. The D / A converter 208 converts the received digital data into an analog signal and supplies it to the display unit 101, whereby the image is displayed on the display unit 101. The display unit 101 is a display device such as a liquid crystal display or an organic EL display, and displays an image based on an analog signal from the D / A converter 208. The on / off of the image display on the display unit 101 is switched by the system control unit 210, and the power consumption can be reduced by turning off the image display. The digital signal stored in the memory 209 from the image pickup unit 204 through the A / D converter 205 is converted into an analog signal by the D / A converter 208 and sequentially displayed on the display unit 101 to display a through image. The electronic viewfinder function can be realized.

The digital camera 100 includes a non-volatile memory 211, a system timer 212, a system memory 213, a detection unit 215, and a strobe control unit 217. The non-volatile memory 211 is a memory (for example, EEPROM or the like) that can be electrically erased and stored, and stores a program executed by the system control unit 210, a constant for operation, and the like. Further, the non-volatile memory 211 has an area for storing system information and an area for storing user setting information, and the system control unit 210 has various types of areas stored in the non-volatile memory 211 when the digital camera 100 is started. Read and restore information and settings.

The system control unit 210 includes a CPU and controls the overall operation of the digital camera 100 by executing various program codes stored in the non-volatile memory 211. The programs, operation constants, variables, and the like read from the non-volatile memory 211 by the system control unit 210 are expanded on the system memory 213. RAM is used for the system memory 213. Further, the system control unit 210 controls the display by controlling the memory 209, the D / A converter 208, the display unit 101, and the like. The system timer 212 measures the time used for various controls and the time of the built-in clock. The strobe control unit 217 controls the light emission of the strobe 141 according to the brightness of the subject. The detection unit 215 includes a gyro sensor and an acceleration sensor, and acquires angular velocity information, acceleration information, attitude information, and the like of the digital camera 100. The angular velocity information includes information on the angular velocity and the angular acceleration at the time of panoramic imaging by the digital camera 100. Further, the posture information includes information such as the inclination of the digital camera 100 with respect to the horizontal direction.

The display unit 101, the operation unit 102, the controller wheel 103, the shutter button 121, the mode changeover switch 104, the power switch 122, and the strobe 141 shown in FIG. 2 are the same as those described with reference to FIG.

The various operation members constituting the operation unit 102 are used, for example, for selecting various function icons displayed on the display unit 101, and when a predetermined function icon is selected, the function is assigned to each scene. That is, each operation member of the operation unit 102 acts as various function buttons. Examples of the function button include an end button, a back button, an image feed button, a jump button, a narrowing down button, an attribute change button, a DISP button, and the like. For example, when the menu button is pressed, a menu screen for making various settings is displayed on the display unit 101. The user can intuitively perform the setting operation by using the menu screen displayed on the display unit 101 and the up / down / left / right four-direction buttons and the SET button.

The controller wheel 103, which is an operating member capable of rotating, is used when designating a selection item together with a four-way button. When the controller wheel 103 is rotated, an electric pulse signal corresponding to the amount of operation (rotation angle, number of rotations, etc.) is generated. The system control unit 210 analyzes this pulse signal and controls each unit of the digital camera 100.

The shutter button 121 has a first switch SW1 and a second switch SW2. The first switch SW1 is turned on in a half-pressed state during the operation of the shutter button 121, whereby a signal instructing preparation for imaging is transmitted to the system control unit 210. Upon receiving the signal that the first switch SW1 is turned on, the system control unit 210 starts operations such as AF processing, AE processing, AWB processing, and EF processing. The second switch SW2 is turned on in a fully pressed state in which the operation of the shutter button 121 is completed, whereby a signal instructing the start of imaging is transmitted to the system control unit 210. When the system control unit 210 receives the signal that the second switch SW2 is turned on, the system control unit 210 performs a series of image pickup operations from reading the signal from the image pickup unit 204 to writing the image data to the recording medium 130.

The mode changeover switch 104 is a switch for switching the operation mode of the digital camera 100 between various modes such as a still image image pickup mode, a moving image image pickup mode, and a reproduction mode. The still image imaging mode includes a panoramic image imaging mode for synthesizing a panoramic image by panoramic imaging, in addition to an auto imaging mode and the like.

The digital camera 100 includes a power supply unit 214 and a power supply control unit 218. The power supply unit 214 is a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like, and supplies power to the power supply control unit 218. The power supply control unit 218 detects whether or not a battery is installed in the power supply unit 214, the type of battery, the remaining battery level, and the like, and based on the detection result and the instruction of the system control unit 210, a necessary voltage is applied for a necessary period. It is supplied to each part including the recording medium 130.

The digital camera 100 includes a recording medium I / F 216 for enabling communication between the recording medium 130 and the system control unit 210 when the recording medium 130 is installed in a recording medium slot (not shown). Since the details of the recording medium 130 have already been described with reference to FIG. 1, the description thereof will be omitted here.

FIG. 3 is a flowchart for explaining the panoramic image pickup in the present embodiment. It is assumed that the flow processing shown in FIG. 3 is performed while the digital camera 100 is panning. Hereinafter, the present embodiment will be described with reference to the flowchart shown in FIG.

When SW1 is pressed in step S301, the process proceeds to step S302.

In step S302, the user moves the digital camera 100 toward the main subject, and the image pickup unit 204 acquires luminance information and the like through the image pickup optical system and measures the light. Subsequently, in step S303, the system control unit 210 calculates the aperture, sensitivity, and shutter speed so that the main subject has an appropriate exposure. Since the main subject is easily noticed, the user is required to have an appropriate exposure of the main subject.

When SW2 is pressed in step S304, the process proceeds to step S305. At this time, it is assumed that the user has already started the panning operation for panoramic image pickup and the digital camera 100 is directed to the first image pickup position. It should be noted that the word "panning" here is not limited to moving horizontally, that is, it also includes tilting and the like.

In step S305, the image pickup unit 204 measures the light again. At this point, since the panning operation has already started, the digital camera 100 heads to a place different from the main subject and measures light at that place.

In step S306, the system control unit 210 calculates the appropriate exposure of the portion imaged in step S305 based on the photometric result of the image pickup unit 204 in step S305.

In step S307, the image pickup unit 204 takes an image. In the imaging here, the system control unit 210 is performed under the conditions calculated in step S303. The system control unit 210 does not take an image with the proper exposure calculated in step S306.

In step S308, the system control unit 210 determines whether or not the image captured by the image pickup unit 204 in step S307 is the first image. If it is the first image, the process proceeds to step S314, and if it is not the first image, the process proceeds to step S309.

In step S309, the image processing unit 206 develops the compositing area. The compositing area referred to here is a region used for compositing panoramic images. In order to prevent image distortion and the like, the image processing unit 206 cuts out the vicinity of the center of the image captured by the image pickup unit 204 in step S307 and uses it as a compositing region.

In the final panoramic image, the exposure of each area used for compositing must be constant in order to prevent discomfort due to the difference in exposure. Therefore, in the development in step S309, the main subject is developed so as to have an appropriate exposure. That is, the development is performed under the conditions obtained by the photometry in step S302.

In step S310, the image processing unit 206 develops the alignment area. The alignment area referred to here is an area for alignment of the synthesis area used for synthesis. Details of the development in step S310 will be described later.

In step S311, the image processing unit 206 develops the exposure tracking region with an appropriate exposure. The exposure tracking area referred to here is an area for aligning with the above-mentioned alignment area.

FIG. 4 is a diagram for explaining alignment in the present embodiment. The image processing unit 206 first develops the first image under the condition that the main subject has an appropriate exposure (the condition calculated in step S303), and obtains the compositing area 411. Next, the image processing unit 206 redevelops the compositing region 411 so that the compositing region 411 has an appropriate exposure, and creates an exposure tracking region 413.

Next, the image pickup unit 204 captures the second image, and the image processing unit 206 similarly develops under the condition that the main subject has an appropriate exposure (condition calculated in step S303), and the composite area 421 is formed. obtain. Next, the image processing unit 206 adjusts the brightness so that the compositing region 421 is similar to the exposure of the exposure tracking region 413 (gain is applied to the compositing region 412), and the alignment region 422 is created. do. Then, the image processing unit 206 adjusts the brightness of the compositing region 421 so that the compositing region 421 has an appropriate exposure (gain is applied to the compositing region 412), and creates an exposure tracking region 423. .. In this way, the exposure between the alignment area and the exposure tracking area is adjusted.

The image processing unit 206 performs the same processing on the third image. First, the image processing unit 206 similarly develops under the condition that the main subject has an appropriate exposure (condition calculated in step S303) to obtain the compositing area 431. Next, the image processing unit 206 develops the compositing region 431 so as to have the same exposure as the exposure tracking region 423, and creates the alignment region 432. Then, the image processing unit 206 redevelops the compositing region 431 so that the compositing region 431 has an appropriate exposure, and creates an exposure tracking region 433.

In step S312, the image processing unit 206 performs alignment using the alignment area created in step S310. Specifically, the image processing unit 206 aligns the exposure tracking region of the image of the previous frame with the alignment region of the current frame. The alignment area created in step S310 is created from the composition area created in step S309, and the positions of the composition area and the alignment area are the same. Therefore, if the image processing unit 206 can align with the alignment area, the image processing unit 206 can also align with the compositing area.

In the following, the alignment method will be briefly described using the affine transformation as an example. In addition to the affine transformation, there are Euclidean transformations that can be rotated and translated, and projective transformations that can be corrected for trapezoidal shapes, but linear transformations that cannot be translated cannot be used. The affine transformation can correct expansion, skew, etc. in addition to movement and rotation to the X-axis and Y-axis.

First, since the digital camera 100 often pans around the user, it is preferable to perform a cylindrical conversion for projecting an image captured on a fictitious cylinder before the alignment process described later.

First, the image processing unit 206 divides one region of the alignment targets (for example, the alignment region of the current frame) into small blocks of arbitrary size. If the number of blocks is small, the trapezoid and rotation in the optical axis direction due to the tilt of the digital camera 100 cannot be corrected accurately. It will be like. Therefore, the optimum number of blocks should be appropriately determined according to the ease of finding the feature points and the angle of view of the subject. The image processing unit 206 is preferably set so that the size of each block is the same. Next, the image processing unit 206 sets a search range wider than the block at the same position as each set block in another area (for example, an exposure tracking area of the previous frame). .. Finally, the image processing unit 206 calculates a corresponding point in each search range in which the sum of the absolute differences in brightness from the initially set block (Su of Absolute Difference, hereinafter referred to as SAD) is minimized. The image processing unit 206 calculates a motion vector from the center of the initially set block and the corresponding point described above. Further, when the contrast is lower than the threshold value set in a certain area of the image, the error of the detected motion vector becomes large, so the motion vector detected from the area is not used. In the calculation of the corresponding points described above, the image processing unit 206 performs, in addition to SAD, sum of differences (Sum of Squared Difference, hereinafter referred to as SSD), normalized cross-correlation (hereinafter referred to as NCC), and the like. You may use it.

Next, the image processing unit 206 calculates the affine coefficient using the detected motion vector. The image processing unit 206 can perform conversion using the equation shown in (Equation 1).

In (Equation 1), (x', y') indicates the coordinates after the conversion, and (x, y) indicates the coordinates before the conversion. The matrix A shows the above-mentioned conversion coefficients. In this way, the image processing unit 206 calculates the affine coefficient using the detected motion vector. Alignment can be performed by calculating the converted coordinates using (Equation 1) for the pixels at the respective coordinates in the region to be aligned with the conversion coefficient A.

In step S313, the image processing unit 206 performs compositing using the compositing region after the alignment.

On the other hand, if the system control unit 210 determines in step S308 that the image captured by the image pickup unit 204 is not the first image in step S307, the system control unit 210 proceeds to step S314. The image processing unit 206 develops the composition region in step S314, and develops the exposure tracking region in step S315. Each development method is the same as in step S309 and step S311.

In step S318, the system control unit 210 determines whether or not to end the imaging. The criterion for determination may be, for example, whether SW2 is released, the upper limit of the capacity of the recording medium 130 is reached, or whether a predetermined number of images is taken. If the imaging is finished, the process proceeds to step S319, and if the imaging is not finished, the process returns to step S304.

In the above description, the composition area, the alignment area, and the exposure tracking area are described so as to be created by developing the image obtained by capturing the image once three times by different methods. , Not limited to this. In particular, the above region can be obtained more conveniently by changing the structure of the image sensor provided in the image pickup unit 204.

FIG. 5 is a diagram for explaining a modified example of the image sensor in the present embodiment.

FIG. 5 shows an image sensor in which the pixels used for creating the composition area, the alignment area, and the exposure tracking area are separated. The image signals acquired in each of the regions 501, 502, and 503 are used to generate the compositing region, the alignment region, and the exposure tracking region. By using such an image sensor, the resolution of the composite image is lowered, but the bias of the feature points is weakened.

According to this embodiment, by developing multiple times to create image regions with different exposures and calculating a coefficient for alignment using two regions with similar exposures, there is a possibility that alignment may fail. Can be lowered.

(Second embodiment)
In the first embodiment, the composition region, the alignment region, and the exposure tracking region are located at the same position on the captured image. On the other hand, the second embodiment has a configuration in which the composition region, the alignment region, and the exposure tracking region are located at different positions of the captured image. Hereinafter, the details of the second embodiment will be described. The description of the same parts as those of the first embodiment will be omitted.

FIG. 6 is a flowchart for explaining the panoramic image pickup in the second embodiment. The processing in steps S601 to S608 is the same as in steps S301 to S308 of the first embodiment.

In step S609, the image processing unit 206 develops the compositing area. The details of the synthesis area will be described later.

In step S620, the image processing unit 206 sets the alignment area and the exposure tracking area. Specifically, first, the image processing unit 206 divides the image signal obtained by the image pickup unit 204 in step S607 into two regions. Next, in step S621, the system control unit 210 calculates the difference between the imaging exposure of the previous frame and the proper exposure of the current frame in each region. The imaging exposure here is, in other words, the exposure when imaging is performed under conditions such that the main subject has an appropriate exposure. Finally, in step S622, the image processing unit 206 sets the region having the smaller difference calculated in step S621 as the alignment region, and sets the region having the larger difference as the exposure tracking region.

The processing in steps S610 to S615 is substantially the same as in steps S310 to S315 of the first embodiment. Next, the alignment process in the present embodiment will be described with reference to the drawings.

FIG. 7 is a diagram for explaining alignment in the present embodiment. The image processing unit 206 divides the right half area 712 and the left half area 713 into the above-mentioned "two areas". First, the image processing unit 206 sets a region near the center of the image signal as a compositing region for the first image, as in the first embodiment, so that the main subject is properly exposed. Develop under the conditions. Next, the image processing unit 206 sets the regions 712 and 713 as exposure tracking regions, and develops each of them with an appropriate exposure. Similarly, for the second image, the image processing unit 206 first sets the area 721 near the center as the area for compositing, and develops the image under the condition that the main subject has an appropriate exposure. Next, the image processing unit 206 sets the left half region 722 as the alignment region and develops it with the same exposure as the region 713. Further, the image processing unit 206 sets the right half region 723 as an exposure tracking region, and develops the subject in which the region 723 is captured so that the exposure is appropriate. Similarly, for the third image, the image processing unit 206 first sets the region 731 near the center as the compositing region, and develops the third image under the condition that the main subject is properly exposed. Next, the image processing unit 206 sets the right half region 732 as the alignment region and develops it with the same exposure as the region 723. Further, the image processing unit 206 sets the right half region 723 as an exposure tracking region, and develops the subject in which the region 723 is captured so that the exposure is appropriate.

In FIG. 7, the areas 711, 721, and 731 are shown to be vertically displaced from the respective images, but the illustrations are changed only for the sake of clarity, and the areas are shown in the actual captured image. The top and bottom of 711, 721, and 731 are not offset.

Next, alignment will be described. As an example, the alignment method may be a method using the affine coefficient described in the first embodiment.

In step S612, the system control unit 210 calculates the affine coefficient for the alignment region of each of the second and subsequent frames and the exposure tracking region of the previous frame. That is, the system control unit 210 calculates the affine coefficient for the region 723 and the region 722 for the region 723 and the region 732. Further, the present invention is not limited to this, and the affine coefficient may be calculated for the region 722 and the region 733. The image processing unit 206 aligns the regions 711, 721, and 731 using the calculated affine coefficient. Finally, in step S613, the image processing unit 206 performs composition on the regions 711, 721, and 731 after the alignment.

When the alignment is performed in step S612, if sufficient feature points cannot be acquired and the alignment cannot be performed, as an example, the setting in step S622 described above may be reversed and the process may be performed again. That is, in step S622 described above, the image processing unit 206 sets the region where the difference in exposure is small as the exposure tracking region, and sets the region where the difference is large as the region for alignment.

In the method shown in FIG. 7, the overlapping area of the alignment area corresponds to the range 714 and the range 724, but in the method shown in FIG. 4, the overlapping area of the alignment area corresponds to the range 715 and the range 725. In general, the wider the overlapping region is, the better the alignment accuracy is, so that the method used in the second embodiment seems to be more accurate than the first embodiment.

Also in the second embodiment, by changing the structure of the image sensor, it is possible to more conveniently obtain signals in the composition region, the alignment region, and the exposure tracking region.

FIG. 8 is a diagram for explaining a modified example of the image sensor in the present embodiment.

FIG. 8 shows an image sensor having a structure in which an alignment region and an exposure tracking region are arranged line by line on both sides of a composition region. When such a structure is used, the bias of the feature points is weakened as in FIG. 5 described above.

As described above, in the second embodiment, the image processing unit 206 divides the area and sets the alignment area and the exposure tracking area on one screen, so that more accurate alignment can be realized. ..

(Other embodiments)
In the above embodiment, the description is based on a digital camera for individuals, but if it is equipped with panoramic imaging and compositing functions, it may be a mobile device, a smartphone, or a network camera connected to a server. It is also possible to apply to. Alternatively, a part of the above-mentioned processing may be performed on a mobile device, a smartphone, a network camera connected to a server, or the like.

It should be noted that the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or an apparatus via a network or a recording medium, and one or more processors in the computer of the system or the apparatus are a program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital camera 102 Operation unit 104 Mode changeover switch 130 Recording medium 204 Imaging unit 206 Image processing unit 207 Memory control unit 209 Memory 210 System control unit 215 Detection unit 216 I / F

Claims (16)

An imaging means that has at least an image sensor and captures a plurality of images,
A setting means for setting a first region, a second region, and a third region in each of at least a part of the plurality of images.
An adjusting means for adjusting the exposure of the second region and the third region, and
An alignment means for aligning with respect to the first region of each of at least a part of the plurality of images.
The alignment means has a synthesis means for synthesizing the first region after the alignment.
When the exposure of the second region of the image is not the same as the exposure of the third region of the image in the previous frame of the image, the adjusting means is the exposure of the second region and the third region. After adjusting the brightness of the image and the image of the frame in front of the image so that the exposure of the region is the same, the alignment means causes the second region of the image and the front of the image. Calculate the coefficient for alignment with respect to the third region of the image of the frame of
An image pickup apparatus in which the alignment means aligns with respect to the first region using the coefficient. The imaging device according to claim 1, wherein the imaging means captures the plurality of images with a fixed exposure. The adjustment means according to claim 1 or 2, wherein the adjustment means performs the adjustment by redeveloping the second region and the third region, or by applying a gain. Imaging device. The imaging device according to claim 1 or 2, wherein the adjusting means makes the adjustment so that the exposure of the third region becomes an appropriate exposure. The image pickup apparatus according to any one of claims 1 to 4, wherein the alignment means performs the alignment by detecting a feature point and calculating a motion vector. The imaging device according to claim 5, wherein the coefficient for alignment is an affine coefficient. The image pickup apparatus according to any one of claims 1 to 6, wherein the second region and the third region in one image do not overlap each other. The imaging device according to claim 7, wherein the setting means sets the third region on the first image of the plurality of images, and does not set the second region. The setting means compares the exposure of the image with the image of the frame before the image with respect to the second and subsequent images of the plurality of images, whereby the second region and the first image are compared. The image pickup apparatus according to claim 8, wherein the area of 3 is set. The setting means sets a first portion and a second portion in advance at the same positions for the second and subsequent images of the plurality of images, and appropriately exposes the image and the image. The ninth aspect of claim 9, wherein the difference between the exposure of the image of the previous frame and the first portion and the second portion is calculated, and the second region is set to the smaller of the difference. Imaging device. The image pickup apparatus according to any one of claims 1 to 6, wherein the first region, the second region, and the third region are located in the center of the image. The image pickup apparatus according to any one of claims 1 to 6, wherein the first region, the second region, and the third region are located at the same position in the image. The image pickup apparatus according to any one of claims 1 to 6, wherein the second region and the third region are arranged for each line. The imaging device according to any one of claims 1 to 6, wherein the first region is arranged for each line. An imaging step that captures multiple images and
A setting step for setting a first region, a second region, and a third region in each of at least a part of the plurality of images.
An adjustment step for adjusting the exposure of the second region and the third region,
An alignment step of aligning with respect to the first region of each of at least a part of the plurality of images.
It has a synthesis step of synthesizing with respect to the first region after the alignment is performed in the alignment step.
When the exposure of the second region of the image is not the same as the exposure of the third region of the image in the previous frame of the image, in the adjustment step, the exposure of the second region and the second region. After adjusting the brightness of the image and the image of the frame before the image so that the exposure of the region 3 is the same, in the alignment step, the second region of the image and the image A coefficient for alignment with respect to the third region of the image in the previous frame was calculated.
An imaging method characterized in that, in the alignment step, alignment is performed with respect to the first region using the coefficient. A computer program that runs on the computer of an image processing device.
An imaging step that captures multiple images and
A setting step for setting a first region, a second region, and a third region in each of at least a part of the plurality of images.
An adjustment step for adjusting the exposure of the second region and the third region,
An alignment step of aligning with respect to the first region of each of at least a part of the plurality of images.
In the alignment step, the synthesis step of synthesizing the first region after the alignment is performed is performed.
When the exposure of the second region of the image is not the same as the exposure of the third region of the image in the previous frame of the image, in the adjustment step, the exposure of the second region and the second region. After adjusting the brightness of the image and the image of the frame before the image so that the exposure of the region 3 is the same, in the alignment step, the second region of the image and the image A coefficient for alignment with respect to the third region of the image in the previous frame was calculated.
In the alignment step, a program characterized in that alignment is performed with respect to the first region using the coefficient.

Images