JP2016039597A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the number of images superposed at each position uniform, by suppressing generation of noise when creating a wide-angle composite image from a plurality of images.SOLUTION: Images 601-607 are captured continuously by means of a digital camera 100 while changing the shooting range, and then duplicate regions 701-706 where a subject duplicates between adjacent images are identified sequentially. Number-of-times-of synthesis maps 801-806 calculating the number of times of superposition of the duplicate regions 701-702, for each region of different number of times of superposition when the images 601-607 are superposed sequentially, are created so that the number of times of superposition of the duplicate regions 701-706 does not exceed a predetermined upper limit number of times of superposition in each region of the-number-of-times-of synthesis maps 801-806. Based on the-number-of-times-of synthesis maps 801-806 thus created, a wide-angle composite image 610 is created by sequentially synthesizing the images 601-607 while connecting the regions reaching the upper limit number of times of synthesis.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that synthesize a wide-angle image from a plurality of images.

  In recent years, a method has been proposed as a method of photographing with an imaging device such as a digital camera, in which a plurality of images are photographed in the same scene and the plurality of images are combined by calculation for the purpose of expanding the dynamic range or the angle of view. (See Patent Documents 1 and 2). For example, in the technique described in Patent Document 1, continuous imaging is performed while moving the imaging device in a specific direction, and a wide-angle image is generated by aligning and synthesizing the obtained images at specific positions. doing. In the technique described in Patent Document 2, a part of the entire photographing range is photographed by sequentially changing the photographing direction, and an image area of a predetermined size that constitutes a part of the image is mutually connected for the plurality of photographed images. Read out so that an overlapping area occurs. Then, the read image area is sequentially overlapped so that at least two images overlap each other at each image position, thereby generating an entire image representing the entire photographing range.

  Hereinafter, a technique for combining a plurality of images in such a manner that the angle of view is enlarged will be referred to as “wide-angle image composition”, and an image obtained by wide-angle image composition will be referred to as “wide-angle composite image”. And

JP 2013-58931 A Japanese Patent No. 4356621

  However, as in the technique described in the above-mentioned Patent Document 1, when a plurality of images are aligned and the images are connected, a noise difference is generated between the image joints (image overlapping regions) and the other regions. There is a problem that becomes conspicuous. Moreover, in the technique described in Patent Document 2, since a plurality of images are always overlapped at each position, the difference in noise between the joints of the images and the other portions becomes inconspicuous, but is overlapped depending on the positions. The number of images may not be the same. In this case, for example, unnaturalness such as a difference in brightness between the central portion and the end portion of the entire image may occur. Further, the technique described in Patent Document 2 is based on the premise that shooting is performed by panning the camera unit of the imaging apparatus at a constant motor speed. Therefore, there is a problem that application is difficult when the photographer performs panning photographing by operating the image pickup apparatus by hand.

  The present invention provides a technique capable of suppressing generation of noise and obtaining a good wide-angle composite image by making the number of images superimposed at each position of the wide-angle composite image uniform. For the purpose.

  An image processing apparatus according to the present invention includes an acquisition unit that acquires a plurality of images that are continuously shot while changing a shooting range, and an overlap that overlaps the plurality of images acquired by the acquisition unit so that subject positions are aligned. The image forming apparatus includes: an aligning unit; and a generating unit that generates a wide-angle composite image by sequentially joining regions where the number of images to be superimposed by the overlaying unit reaches a predetermined number.

  According to the present invention, it is possible to obtain a wide-angle synthesized image in which the number of synthesized images at each position is the same in the entire wide-angle synthesized image and noise is suppressed.

1 is a block diagram illustrating a schematic configuration of a digital camera according to an embodiment of the present invention. It is a figure which shows the coordinate system set with respect to the digital camera of FIG. It is a block diagram which shows the structure of the image process part with which the digital camera of FIG. 1 is provided. 2 is a flowchart illustrating an example of a processing procedure when performing wide-angle image composition with the digital camera of FIG. 1. It is a figure which shows typically the method of the panning imaging | photography of step S401 of FIG. It is a figure which shows the example of the actual picked-up image obtained by the panning imaging | photography of step S401 of FIG. 4, and the example of the wide-angle synthetic | combination image produced | generated. It is a figure explaining the identification method of the duplication area | region of step S403 of FIG. 4 using the picked-up image shown to Fig.6 (a). FIG. 7 is a diagram for explaining the composition number map creation processing in step S404 in FIG. 4 using the captured image shown in FIG. 5 is a detailed flowchart of a composition number map creation process in step S404 in FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, a so-called digital camera is taken up as the image processing apparatus according to the present invention. However, the image processing apparatus according to the present invention is not limited to a digital camera, and may be any apparatus that can process a captured image acquired by a camera (imaging means), for example, a personal computer ( PC) or the like. That is, the imaging unit and the image processing unit may be integrated like a digital camera, or may be separate bodies such as a combination of the imaging unit and a personal computer.

  When the imaging unit and the image processing unit are integrated, the image processing apparatus according to the present invention is not limited to a device specialized for photographing such as a digital camera or a digital video camera, and for example, a portable device having a camera function. A communication terminal, a personal computer, a game machine, etc. may be sufficient. In addition, when the imaging unit and the image processing unit are separate, the transmission of the captured image from the imaging unit to the image processing unit is not limited to wireless communication or wired communication. It doesn't matter if it is connected.

<Schematic structure of digital camera>
FIG. 1 is a block diagram showing a schematic configuration of a digital camera 100 according to an embodiment of the present invention. The digital camera 100 includes a system control unit 101 (hereinafter referred to as “control unit 101”), a ROM 102, a RAM 103, an optical system 104, an imaging unit 105, an A / D conversion unit 106, an image processing unit 107, a storage medium 108, and a detection unit. 110 and a display unit 111. Communication between these blocks is performed via a bus 109.

  The control unit 101 is, for example, a CPU, and comprehensively controls the operation of each block by reading out an operation program of each block constituting the digital camera 100 from the ROM 102 and developing it in the RAM 103. The ROM 102 is a rewritable nonvolatile memory, and stores an operation program for each block constituting the digital camera 100, parameters necessary for the operation of each block, and the like. The RAM 103 is a rewritable volatile memory, and is used as a work area for the CPU and as a temporary storage area for data output by the operation of each block constituting the digital camera 100.

  The optical system 104 takes in light from a subject and forms it as an optical image (subject image) on an imaging element such as a CCD sensor or a CMOS sensor provided in the imaging unit 105. The optical system 104 includes an aperture that adjusts the amount of light during shooting by adjusting the aperture diameter of the optical system 104. The imaging device included in the imaging unit 105 photoelectrically converts the optical image and outputs an analog electric signal to the A / D conversion unit 106. The A / D conversion unit 106 performs A / D conversion on the analog electrical signal supplied from the imaging unit 105, and outputs the generated digital signal image data to the RAM 103 for storage.

  The image processing unit 107 performs necessary image processing such as white balance processing and gamma correction processing on the image data stored in the RAM 103. The storage medium 108 is, for example, a detachable memory card or the like, and image data of an image processed by the image processing unit 107 stored in the RAM 103 or digital data that has been A / D converted by the A / D conversion unit 106. Signals (image signals) and the like are stored.

  The detection unit 110 includes a known gyro sensor, acceleration sensor, and the like, and an output signal from the detection unit 110 is transmitted to the control unit 101. The control unit 101 detects the posture and movement (angle, acceleration, movement amount, etc.) of the digital camera 100 based on the output signal acquired from the detection unit 110.

  FIG. 2 is a diagram showing a coordinate system set for the digital camera 100 in the present embodiment. The optical axis direction of the optical system 104 of the digital camera 100 is defined as the Z direction, and an X axis and a Y axis that are orthogonal to the Z axis are defined in the main body of the digital camera 100. Based on the output signal acquired from the detection unit 110, the control unit 101 calculates the angle (α, β, γ) and the position (X, Y, Z) shown in FIG. The position is calculated as a relative value from the reference value. For example, the control unit 101 determines the angle (0 °, 0 °, 0 °) and the position (0, 0, 0) using the detection result of the detection unit 110 synchronized with the first shooting as a reference value. After calculating the reference value, the control unit 101 calculates a relative value, which is a relative movement amount from the reference value, and detects whether panning photographing described later is performed as expected.

  The display unit 111 includes a display device such as a liquid crystal display (LCD) that displays images and various information, and a display control circuit that controls image display on the display device. The display image data stored in the RAM 103 is converted into an analog signal by a D / A converter included in the display control circuit and displayed on the display.

<Configuration of image processing unit>
FIG. 3 is a block diagram illustrating a configuration of the image processing unit 107. The image processing unit 107 includes an overlapping area extraction unit 301, a synthesis number map creation unit 302, and a wide-angle image synthesis unit 303. The overlapping area extraction unit 301 specifies the area where the subject overlaps between the N−1th captured image and the Nth captured image as the N−1th overlapping area. Here, “N” is an integer of 2 or more, and this definition is always applied in the following description.

  The synthesis number map creation unit 302 is based on a synthesis number map (details will be described later) when wide-angle image synthesis is performed N-1 times and the Nth overlap region output from the overlap region extraction unit 301. An Nth synthesis number map is created. A wide-angle image composition unit 303 generates a wide-angle composite image obtained by performing N-1 times of wide-angle image synthesis and a captured image to be synthesized for the N-th time (N + 1-th captured image) based on the output result of the synthesis number map creation unit 302. To generate a wide-angle composite image obtained by performing wide-angle image synthesis N times. Note that the first combination number map and the first wide-angle combined image are created from the first and second captured images.

<First Embodiment>
FIG. 4 is a flowchart illustrating an example of a processing procedure when the digital camera 100 performs wide-angle image composition. Here, a process for limiting the number of overlapping regions to a predetermined number (hereinafter referred to as “upper limit combining number”) when performing wide-angle image combining using a plurality of captured images will be described. Further, the photographer performs panning photographing while rotating the digital camera 100 in the horizontal direction from the left to the right (around the Y axis in FIG. 2) as viewed from the photographer, that is, the photographing range is changed. Continuous shooting. However, the panning shooting direction may be other directions, for example, the reverse horizontal direction (from right to left) or the vertical direction (from bottom to top or from top to bottom).

  Each process shown in the flowchart of FIG. 4 is realized by developing a program read from the ROM 102 by the control unit 101 in the RAM 103 and controlling the operation of each block constituting the digital camera 100. The processing shown in the flowchart of FIG. 4 is started when an instruction for photographing is given to the control unit 101 by pressing a release button provided in an operation unit (not shown) provided in the digital camera 100 or the like.

  In step S401, the control unit 101 first calculates an appropriate exposure value suitable for a scene to be shot. Then, a divided exposure value obtained by dividing the appropriate exposure value by a predetermined number is set, and various conditions necessary for photographing are determined. The division number of the appropriate exposure value is set to the upper limit synthesis number of the overlapping area in the synthesis number map. For example, when the upper limit synthesis number of the synthesis number map is “2”, the division number of the appropriate exposure value is also “2”. Set to In addition, the control unit 101 instructs the photographer in the shooting direction, so that the photographer performs shooting while panning the digital camera 100 in the direction instructed by the control unit 101. Details of the panning shooting in step S401 will be described later with reference to FIGS.

  At this time, the control unit 101 detects the posture and movement of the digital camera 100 based on the output signal from the detection unit 110, and determines whether the digital camera 100 is performing the expected panning shooting based on the detection result. judge. For example, when the detection unit 110 detects an angular velocity or acceleration, the control unit 101 calculates the angle and position of the digital camera 100 by performing an integral operation on the detected numerical value.

  In step S <b> 402, the control unit 101 determines whether or not a captured image for performing the processing of steps S <b> 403 to S <b> 405 (wide-angle image composition) remains in the captured image obtained by the panning imaging in step S <b> 401. If there is no captured image remaining (NO in S402), the control unit 101 terminates the present process. If the captured image remains (YES in S402), the control proceeds to step S403.

  In step S403, the control unit 101 causes the overlapping area extraction unit 301 to overlap the subject positions of the N-1 and N-th captured images adjacent to the captured image obtained by the panning shooting in step S401. The area is specified as the N-1th overlapping area. The overlapping area specifying process in step S403 will be described later with reference to FIG.

  In subsequent step S <b> 404, the control unit 101 causes the synthesis number map creation unit 302 to create an N−1th synthesis number map. Here, as described above, the first combination number map is created by combining the first and second captured images. The second and subsequent synthesis number maps are created using the (N-1) th synthesis number map and the Nth overlapping region (N + 1th captured image). In creating the synthesis number map, the synthesis area and the number of synthesis are controlled so that the number of synthesized images at the final output angle of view is constant, details of which will be described later with reference to FIG.

  In step S405, the control unit 101 generates a wide-angle composite image from the wide-angle image composition unit 303 while sequentially synthesizing temporally adjacent captured images based on the composition region and the composition number in the composition number map created in step S404. Generate. At that time, the composition ratio of the wide-angle composite image that has been subjected to the (N−1) -th wide-angle image composition and the captured image (N + 1-th captured image) that has been subjected to the N-th wide-angle image composition is represented in the N-th composition number map. It changes according to the frequency | count of the synthesis | combination for every synthetic | combination area | region.

  For example, the N-th wide-angle synthesized image is calculated by the following synthesis formula using the (N-1) -th wide-angle synthesized image, the captured image used for the N-th wide-angle image synthesis, and the N-th synthesized frequency map.

  Note that (x, y) indicates the position on the coordinates in the wide-angle composite image. Note that the region where the value of the Nth combination frequency map value is zero (0) is output from an image having an angle of view among the (N−1) th wide-angle combined image and the captured image used for the Nth wide-angle image combination. To control. This specific example will be described later when FIG. 8 is described later.

  After the end of step S405, the control unit 101 returns the process to step S402. In this way, the processing of steps S402 to S405 is repeated, and wide-angle image synthesis is performed using all the captured images obtained by panning imaging in step S401, thereby generating a wide-angle synthesized image to be finally output.

  FIG. 5 is a diagram schematically illustrating the panning shooting in step S401. As shown in FIG. 5A, shooting is started with the digital camera 100 facing a predetermined direction, and a shooting start image 501 is shot. Shooting is continued at a predetermined shutter speed while rotating the digital camera 100 about the rotation center 503 of the digital camera 100 in the rotation direction 504 (clockwise in FIG. 5 (around the Y axis in FIG. 2)), and a shooting end image 502 is captured. Then, panning shooting ends.

  As shown in FIG. 5B, the shooting start image 501 and the shooting end image 502 each have a shooting margin area. These photographing blank areas are not included in the angle of view of the wide-angle composite image that is finally output. Therefore, the angle of view 505 of the wide-angle composite image that is finally output is a range from a region that excludes the photographing margin region in the photographing start image 501 to a region that excludes the photographing margin region in the photographing end image 502.

  FIG. 6A is a diagram illustrating an example of a captured image obtained by panning imaging in step S401. Here, facing the building as a subject, start shooting with a shooting margin area on the left side of the building so that the left side of the building appears with a margin, so that the right side of the building appears with a margin. The shooting margin area is provided on the right side of the building to finish the shooting. Thereby, seven photographed images 601 to 607 are photographed. FIG. 6B is a diagram illustrating an example of a wide-angle composite image 610 generated using the captured images 601 to 607 in FIG. It can be seen that the wide-angle composite image 610 does not include the shooting margin areas of the shot image 601 that is the shooting start image and the shot image 607 that is the shooting end image.

  With reference to FIG.6 and FIG.7, the identification process of the duplication area | region of step S403 is demonstrated. FIG. 7 is a diagram for explaining a method for specifying an overlapping area between adjacent captured images for the captured images 601 to 607 shown in FIG. In the process of specifying an overlapping area between adjacent captured images, an area where the subject overlaps in the adjacent captured image is calculated. Therefore, in FIG. 7, the movement amount of the captured images 602 to 607 is taken on the horizontal axis with the captured image 601 as a reference so that the same portion of the subject overlaps the previous captured image in each of the captured images 602 to 607. It is aligned with.

  A general image alignment method can be used as a method of aligning two adjacent captured images. For example, a method of dividing a captured image into small blocks of arbitrary size, calculating corresponding points where the SAD (Sum of Absolute Difference) of the image signal (luminance, etc.) is minimum for each small block, and calculating a motion vector, etc. Use.

  In the alignment process between the captured images 601 and 602, the first overlap region 701 in which the start coordinate is SX1 and the end coordinate is EX1 is specified. Similarly, in the alignment process between the captured images 602 and 603, the second overlapping region 702 in which the start coordinate is SX2 and the end coordinate is EX2 is specified. As described above, by sequentially calculating the feature amount such as the movement amount (coordinates) for the captured images 601 to 607 by the alignment processing of two adjacent captured images, the first to sixth overlap regions 701 to 706 are performed. Can be specified. In step S403, the first to sixth overlap areas 701 to 706 are specified, and at the same time, the adjacent captured images subjected to the alignment process are matched with the position of the previous captured image in terms of time. Perform geometric transformation processing.

  Next, the composition number map creation processing in step S404 will be described with reference to FIGS. FIG. 8 is a diagram for explaining a method of creating a composite number map when generating the wide-angle composite image 610 using the captured images 601 to 607 shown in FIG.

  FIG. 8A is a diagram schematically illustrating a method of creating a first composition number map from the captured images 601 and 602. Here, in step S403, the first overlapping area 701 (the overlapping area between the first photographed image 601 and the second photographed image 602) has been specified, but the composition number map has not yet been created. . For this reason, in the first synthesis number map 801, the areas SX1 to EX1 in which the synthesis is performed once and the areas 0 to SX1 and EX1 in which the synthesis is not performed are specified.

  Here, for the areas SX1 to EX1 corresponding to the first overlapping area 701, the first photographed image 601 and the second photographed image 602 are combined at a ratio of 1/2: 1/2. In areas 0 to SX1 and EX1 that do not overlap between the first photographed image 601 and the second photographed image 602, areas 0 to SX1 of the first photographed image 601 and the second photographed image 602 are displayed. The regions EX1 to EX1 are used as they are.

  FIG. 8B schematically shows a method of creating the second synthesis number map 802 from the first synthesis number map 801 and the shot image (third shot image 603) to be subjected to the second wide-angle image synthesis. FIG. Of the regions SX1 to EX1 in the first combination number map 801, regions SX2 to EX1 overlap with the second overlapping region 702. Therefore, in the second synthesis number map 802, the regions SX2 to EX1 in which the second synthesis is performed, the regions SX1 to SX2 and EX1 to EX2 in which the first synthesis is performed, the synthesis is not performed (the number of synthesis) In which the areas 0 to SX1 and EX2 to zero (0) are specified are created.

  More specifically, in the second combination count map 802, the areas SX2 to EX1 that have been combined twice are the areas SX2 to EX1 of the first combination count map 801 and the area SX2 of the third photographed image 603. ~ EX1 is synthesized at a ratio of 2/3: 1/3. Thereby, in the areas SX2 to EX1 of the second combination number map 802, the captured images 601 to 603 are respectively combined by 1/3. That is, in the Nth number-of-combination map, the region where the number of times of composition is two is 2/3 of the region where the number of times of composition is one in the N-1th number-of-composite map and the region of the (N + 1) th captured image. : Controlled to be created by combining at a ratio of 1/3.

  Further, in the second combination number map 802, the regions EX1 to EX2 in which the first combination is performed are the regions EX1 to EX1 of the first combination number map 801 and the regions EX1 to EX2 of the third photographed image 603. Is created at a ratio of 1/2: 1/2. In other words, in the Nth number-of-compositions map, for the region where the number of times of composition is 1 by overlapping with the (N + 1) th captured image, the region where the number of times of composition is 0 in the N-1th number-of-compositions map It is controlled so as to be created by combining the captured image area of the eye with a ratio of 1/2: 1/2.

  Further, in the second combination number map 802, the areas SX1 to SX2 that have been combined one time do not overlap with the third photographed image 603, so the areas SX1 to SX2 of the first combination number map 801 are not overlapped. Is used as is. Further, the areas 0 to SX1 of the first combination number map 801 are used as they are for the areas 0 to SX1 where the composition is not performed, and the areas EX2 to EX3 of the third photographed image 603 are used as they are for the area EX2. It is done. In this way, in the N-th synthesis number map, the area where the number of synthesis is zero (0) is an image area where the angle of view exists in the N-1th synthesis number map and the N + 1th captured image. Controlled to use.

  FIG. 8C schematically shows a method of creating the third combination number map 803 from the second combination number map 802 and the photographed image (fourth photographed image 604) for performing the third wide-angle image composition. FIG. Here, in the present embodiment, the upper limit synthesis number in the region on the synthesis number map is set to two. That is, even if there is an overlapping area between the areas SX2 to EX1 that have already been combined twice and the third overlapping area 703 (fourth captured image 604), the area that is combined three times is determined. Do not ask. Then, an overlapping area is obtained between the area that has already been combined once and the overlapping area 703 for the third time, and is combined with the area that has already been combined twice. As a result, as the third synthesis number map 803, the regions SX2 to EX2 in which the second synthesis is performed, the regions SX1 to SX2, EX2 to EX3 in which the first synthesis is performed, and the region 0 in which the synthesis is not performed Those in which ~ SX1, EX3 are specified are created.

  More specifically, in the third combination frequency map 803, the areas that are not combined are areas 0 to SX1 and EX3. Here, the areas 0 to SX1 of the second combination map 802 are used for the areas 0 to SX1, and the areas EX3 to 3 of the fourth photographed image 604 are used for the areas EX3 to EX3, respectively. In addition, the region where the synthesis is performed once is a region SX1 to SX2, EX2 to EX3. Here, since the areas SX1 to SX2 do not overlap with the areas SX1 to SX2 of the second combination number map 802 and the fourth photographed image 604, the areas SX1 to SX2 of the second combination number map 802 are used as they are. It is done. On the other hand, the regions EX2 to EX3 overlap in the region EX2 and the fourth captured image 604 that are not combined in the second combination number map 802. Therefore, the regions EX2 to EX3 are created by combining the regions EX2 to EX3 of the second combination number map 802 and the regions EX2 to EX3 of the fourth photographed image 604 at a ratio of 1/2: 1/2. Has been.

  In the third combination number map 803, the regions SX2 to EX1 out of the regions SX2 to EX2 that have been combined twice have already been combined twice in the second combination number map 802. , No new synthesis. Therefore, the regions SX2 to EX1 of the second combination number map 802 are used as they are for the regions SX2 to EX1. On the other hand, the regions EX1 to EX2 of the regions SX2 to EX2 that have been combined twice are the regions EX1 to EX2 that have been combined once in the second combination number map 802 and the fourth photographed image. It is created by synthesizing 604 areas EX1 to EX2 at a ratio of 2/3: 1/3.

  Note that after the second synthesis number map 802 is created, a wide-angle synthesized image having the regions SX2 to EX1 in which the second synthesis has been performed is generated. Therefore, when the third combination frequency map 803 is created, the areas EX1 to EX2 that have been newly synthesized twice are joined to the areas SX2 to EX1 of the wide-angle synthesized image that have already been created. Regions SX <b> 2 to EX <b> 2 where the synthesis is performed once are generated.

  FIG. 8D is a diagram schematically showing a method of creating the sixth synthesis number map 806 from the fifth synthesis number map and the seventh photographed image 607, which is the final in this embodiment. Creates a composite number map. In accordance with the description in FIG. 8C, as the sixth synthesis number map 806, the regions SX <b> 2 to EX <b> 5 in which the second synthesis is performed, the regions SX <b> 1 to SX <b> 2, EX <b> 5 to EX <b> 6 in which the first synthesis is performed, Those in which the areas 0 to SX1 and EX6 to which synthesis is not performed are specified are created.

  In steps S404 and S405, a wide-angle composite image is generated as the N-th composite number map is created. As is clear from the final sixth composite number map 806 created as described above, the wide-angle composite image 610 is generated from the regions SX2 to EX5 in which the number of composites is uniformly two, Regions are not used for wide-angle composite images. That is, the wide-angle combined image is always generated by combining three photographed images. This is because the number of compositing does not reach the upper limit compositing number in the area (end side) outside the areas SX2 to EX5 where the compositing has been performed twice, and if these areas are included, the image quality becomes uniform. This is because there is a risk that it will not be possible. In this embodiment, at the time of shooting, a shooting blank area (see FIG. 5) is prepared in the shooting start image (shooted image 601) and shooting end image (shot image 607) in advance, and the angle of view more than the angle desired by the photographer is prepared. The angle of view is shot by panning shooting in step S401.

  FIG. 9 is a flowchart of the composition number map creation processing in step S404. In step S <b> 901, the control unit 101 determines whether there is an N−1th synthesis number map. If there is no N-1th synthesis number map, that is, if the first synthesis number map is created (NO in S901), the control unit 101 advances the process to step S902, and the N-1th synthesis number map. If there is any (YES in S901), the process proceeds to step S903.

  In step S902, the control unit 101 creates a composition number map based on the overlapping area of the two captured images specified in the immediately preceding step S403. The processing in step S902 corresponds to the first synthesis number map creation processing in FIG. That is, in the case of FIG. 8A, since there is no past composition number map, the first composition number map in which the regions SX1 to EX2 which are the first overlapping regions 701 are subjected to one composition. 801 is created. After the process of step S902, the control unit 101 ends this process.

  In step S <b> 903, the control unit 101 determines whether or not there is a region in which the number of synthesis is equal to or greater than the upper limit number of synthesis in the N−1th synthesis number map when creating the N-th synthesis number map. If there is no area equal to or greater than the upper limit number of synthesis (NO in S903), control unit 101 advances the process to step S904. If there is an area equal to or greater than the upper limit synthesis number (YES in S903), control proceeds to step S905.

  In step S904, the control unit 101 increases the number of synthesis by 1 for an area where the number of synthesis is not greater than or equal to the upper limit number of synthesis in the (N-1) th synthesis number map and overlaps with the N + 1th photographed image. An Nth synthesis number map is created. In the present embodiment, since the upper limit number of synthesis is set to 2, the processing in step 904 corresponds to the second creation frequency map 802 creation process in FIG. Since the method of creating the second synthesis number map 803 has already been described in detail, the description thereof is omitted here. After the process of step S904, the control unit 101 ends this process.

  In step S <b> 905, the control unit 101 sets the number of synthesis for an area in which the number of synthesis is equal to or greater than the upper limit synthesis number in the N−1th synthesis number map even if there is an area overlapping with the N + 1th captured image. The Nth synthesis number map is created without increasing. Also in this case, the number of compositing is increased by 1 for an area where the number of compositing is not greater than or equal to the upper limit compositing number in the N-1th compositing number map and overlaps with the N + 1th photographed image. This is because when there is no upper limit on the number of compositing, noise is added to the wide-angle composite image to be composited by mixing a region with a large number of compositing and a region with a small number of compositing in the compositing number map, and the number of compositing is large. This is because the boundary between the region and the small region becomes an unnatural image. In other words, by controlling so that the number of synthesizing times does not exceed a predetermined number, it is possible to suppress the occurrence of noise associated with wide-angle image synthesis and generate a wide-angle synthesized image having natural seams.

  In the present embodiment, since the upper limit number of synthesis is set to 2, the processing in step 905 corresponds to the creation processing of the third to sixth synthesis number maps 803 to 806 in FIG. Since the method of creating the third synthesis number map 803 has already been described in detail, the description thereof is omitted here. After the process of step S905, the control unit 101 ends this process.

Second Embodiment
In the first embodiment, the upper limit synthesis number of each region in the synthesis number map is set to 2, but the upper limit synthesis number is not limited to this. For example, if the upper limit number of synthesis is Nlim (Nlim: an integer equal to or greater than 2), the region that is the upper limit number of synthesis occurs for the first time when the Nlim-th synthesis number map is created. Here, assuming that the start coordinate of the overlapping area between adjacent images in the Nth composite number map is SX (N) and the end coordinate is EX (N), the composite number map up to Na times (1 ≦ Na ≦ Nlim). In the creation, the composition number map is updated in the overlapping areas SX (Na) to EX (Na) between adjacent images. Then, in the creation of the composite number map after the Nbth (Nb> Nlim), the composite number map is updated in the region EX (Nb-Nlim) to EX (Nb) for the overlapping region between adjacent images. change.

  For example, if the upper limit number of synthesis is set to 3, the region that becomes the upper limit number of synthesis occurs for the first time when the third synthesis number map is created. Therefore, when the seven captured images 601 to 607 shown in FIG. 6A are synthesized, the first, second, and third synthesis number maps are created by SX1 to EX1, SX2 to EX2, and SX3, respectively. The EX3 area is updated. For example, in the fourth, fifth, and sixth synthesis number maps, the areas EX1 to EX4, EX2 to EX5, and EX3 to EX6 are updated. By performing the processing as described above, the image processing according to the present invention can be performed even if the upper limit number of times of combining is changed or the upper limit of the number of sheets to be combined in the same region is changed.

In the description of the second embodiment, the case where the upper limit synthesis number is 1 is excluded, but this does not require the creation of the second synthesis number map when the upper limit synthesis number is 1. This is because it is not necessary to connect the regions that have reached the upper limit synthesis number. However, even if N _lim is one time, Na = 1, so that the areas SX1 to EX1 are updated in the first synthesis number map, and thus the above processing can be applied as it is.

<Third Embodiment>
In the first embodiment, an overlapping area between adjacent captured images is specified from the feature amount obtained by performing the alignment process on the entire captured image. Here, a process for changing the overlap area is performed on the overlap area obtained by the alignment process for the entire captured image according to the aberration amount of the lens constituting the optical system 104 and the focal length at the time of shooting. May be.

  For example, due to the amount of aberration of the lens and the focal length at the time of shooting, even if alignment processing is performed between adjacent shot images that have been panned, the image distortion may be caused The position may not be correct. In view of this, a process of setting the overlap area narrower in accordance with the magnitude of distortion of the captured image may be performed on the overlap area calculated from the feature amount obtained in the alignment process. Conversely, when processing to improve the synthesis accuracy of wide-angle synthesized images, such as distortion correction and cylindrical transformation, the overlapping area calculated from the feature values obtained in the alignment process can be used to the maximum It is also possible to set to a different area.

<Fourth embodiment>
In the first embodiment, a shooting margin area that is not used for the finally generated wide-angle composite image is set in the shot images at the start and end of shooting. However, the present invention is not limited to this, and processing for changing the photographing margin area may be performed according to the upper limit number of times of synthesis in the synthesis number map. As the upper limit synthesis number increases, the number of captured images to be synthesized increases until the upper limit synthesis number is reached, and the angle of view that can be used for wide-angle image synthesis becomes narrower. Therefore, when the upper limit synthesis number is large, the photographing blank area is controlled widely.

<Fifth Embodiment>
In the first embodiment, alignment processing is performed on adjacent captured images, and an overlapping area between adjacent captured images is specified. However, the present invention is not limited to this, and the panning position and angle of the digital camera 100 may be specified using the detection result of the detection unit 110 to obtain an overlapping area between captured images. That is, by calculating the angle and position of the digital camera 100 calculated from the detection signal of the detection unit 110 as a relative value from the reference value, it is possible to specify an overlapping region between adjacent captured images. By using this method, when shooting a subject that includes low contrast or a repetitive pattern, it is possible to specify the correct overlapping area rather than obtaining the overlapping area by image analysis such as the motion vector method. it can. Note that the process of specifying an overlapping area between captured images by image analysis and the process of specifying an overlapping area between captured images from the detection signal of the detection unit 110 may be used in combination. For example, an overlapping area between captured images obtained by image analysis may be corrected using a detection signal from the detection unit 110.

<Sixth Embodiment>
In the first embodiment, the upper limit synthesis number of the synthesis number map is fixed in advance. However, the present invention is not limited to this, and the upper limit synthesis number of the synthesis number map is changed according to the panning speed at the time of shooting. Also good. For example, when the panning speed is high, the number of photographed images for forming the angle of view that forms the desired wide-angle composite image decreases, and therefore the upper limit composition number of the composition number map decreases as the panning speed increases. Control processing is performed as follows. Further, since the panning speed is high, it is detected whether there is an overlapping area between adjacent captured images using a detection signal from the detection unit 110. If there is no overlapping area, a warning is issued or a wide-angle image is displayed. You may comprise so that the process of a synthesis | combination may be stopped.

  Furthermore, the upper limit number of times of composition in the composition number map and the exposure time of time-division exposure at the time of image capture may be changed according to the length of the appropriate exposure time at the time of photographing. For example, when shooting a night scene or the like, the appropriate exposure time becomes longer, so the longer the appropriate exposure time, the higher the upper limit number of times of combining in the composition number map is controlled, and the shorter the exposure time for each captured image. It is good also as a structure controlled to become.

<Other embodiments>
Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program code. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

100 Digital Camera 101 System Control Unit 102 ROM
DESCRIPTION OF SYMBOLS 104 Optical system 105 Image pick-up part 107 Image processing part 110 Detection part 801,802 The 1st time and the 2nd composition number map 803,806 The 3rd time and the 6th composition number map

Claims (9)

An acquisition means for acquiring a plurality of images continuously shot while changing the shooting range;
A superimposing unit that superimposes the plurality of images acquired by the acquiring unit so that the subject positions match;
An image processing apparatus comprising: a generating unit that generates a wide-angle composite image by sequentially joining regions in which the number of images to be superimposed by the superimposing unit reaches a predetermined number.   The image processing apparatus according to claim 1, wherein the acquisition unit acquires an image captured with a divided exposure value obtained by dividing a proper exposure value by a predetermined number as the plurality of images.   3. The image processing according to claim 2, wherein the superimposing unit includes a detecting unit that detects an overlapping region in which subjects overlap in adjacent images with respect to the plurality of images acquired by the acquiring unit. apparatus.   The superimposing means calculates the number of times that the overlapping area detected by the detecting means is superimposed for each of the areas where the number of times of superposition is different when the plurality of images are sequentially superimposed. 5. The image processing apparatus according to claim 4, further comprising a synthesis number map creating means for creating a map. The synthesis number map creating means creates the synthesis number map so that the number of times the overlapping area is overlapped does not exceed a predetermined upper limit number of synthesis times in each area of the synthesis number map,
The image processing apparatus according to claim 4, wherein the generation unit synthesizes the plurality of images while connecting the regions that have reached the upper limit synthesis number in the synthesis number map.   The said detection means correct | amends the said overlap area based on the focal distance and aberration amount of the lens of the imaging means which image | photographed these several images, The any one of Claim 3 thru | or 5 characterized by the above-mentioned. Image processing device.   7. The image according to claim 1, wherein the generation unit changes the predetermined number in accordance with a speed of movement of the imaging apparatus when capturing the plurality of images. Processing equipment. An image processing method for synthesizing a plurality of images continuously shot while changing a shooting range,
An obtaining step for obtaining the plurality of images;
For the plurality of images acquired in the acquisition step, a specifying step for sequentially identifying overlapping regions where subjects overlap between adjacent images;
When the plurality of images acquired in the acquisition step are sequentially overlapped, the number of times of synthesis that calculates the number of times the overlapping region specified in the specifying step is overlapped for each region having a different number of times of overlapping. A creation step to create a map;
A generating step of sequentially combining the plurality of images based on the combining number map generated in the generating step to generate a wide-angle combined image;
Repeating the identifying step, the creating step and the generating step for all of the plurality of images,
In the creation step, the synthesis number map is created such that the number of times the overlapping region is overlapped does not exceed a predetermined upper limit number of synthesis in each region of the synthesis number map,
In the generating step, the wide-angle combined image is generated while connecting the regions that have reached the upper limit combining number in the combining number map.   A program causing a computer to execute the image processing method according to claim 8.