JP2011188035A - Imaging device, panoramic image synthesis method, and program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device for synthesizing images to form a panoramic image with high accuracy without requiring any sensor on a camera, and to provide a synthesis method. <P>SOLUTION: The panoramic image synthesis method synthesizes a plurality of images captured continuously while moving an imaging device to form a panoramic image. The method includes:searching for areas to be synthesized in an N-th image and an N+1-th image (step S11); and when the amount of movement between the N-th image and the N+1-th image cannot be computed since the similarity between the areas to be synthesized in the N-th image and the N+1-th image is lower than a predetermined threshold (less than the threshold in step S13), panoramic synthesis is performed using the amount of movement computed from two continuous images before the N-th image or after the N+1-th image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging device, a panoramic image synthesis method, and a program thereof, and more particularly, to an imaging device, a panoramic image synthesis method, and a program thereof that can synthesize a panoramic image easily and with high accuracy.

  When shooting a still image of a subject with a camera (imaging device), there is a limit to the viewing angle of the camera, so a wide field of view cannot be shot with a single shot. Therefore, for example, a plurality of still images that are continuous in the horizontal direction (left-right direction) are taken, and these are combined into one to synthesize a panoramic image with a wide viewing angle.

  As digital cameras have become widespread, it has become easier to panoramicly synthesize digital still image data, and an increasing number of digital cameras have a panorama synthesizing function.

  When shooting a panoramic image with a digital camera, the user can hold the digital camera with his hand and pan the video in the horizontal direction to shoot a movie, or take a series of still images to continuously capture multiple images in the horizontal direction. Can be obtained.

  Then, as described in Patent Document 1 below, a feature point is detected from two adjacent images, a movement vector between the two images of the feature point is calculated, and this movement vector is used. An overlapping area is found out of the two pieces of image data, and the two pieces of image data are combined and combined so that the overlapping areas overlap.

  However, if the captured image has a low spatial frequency pattern with little change, it will be difficult to detect feature points and calculate movement vectors, it will be difficult to find overlapping areas, and only panoramic images with low synthesis accuracy can be generated. The problem arises.

  Therefore, if a gyro sensor is installed in the camera as in the digital camera described in Patent Document 2 below, the camera can be moved from the sensor output even if the overlapping area cannot be detected with high accuracy by comparing the two images. It is possible to calculate the overlapping area of two images by calculating the distance.

Japanese Patent No. 3421859 JP-A-6-105214

  In recent years, the price of digital cameras has been reduced, and it has become impossible to mount an expensive gyro sensor or the like. For this reason, there is an increasing demand for a digital camera capable of performing panoramic composition with high accuracy without mounting a sensor as hardware for detecting the moving distance of the camera.

  An object of the present invention is to provide an imaging apparatus, a panoramic image synthesizing method thereof, and a program thereof that enable high-accuracy panoramic synthesis without using a sensor that detects a camera moving distance.

  The panoramic image composition method of the present invention is a panoramic image composition method for performing panoramic composition of a plurality of images taken continuously while moving an imaging device, and searches for a synthesis location of the Nth and N + 1th images. Since the similarity between the synthesis location of the Nth image and the synthesis location of the (N + 1) th image is lower than a predetermined threshold, the amount of movement between the Nth image and the (N + 1) th image cannot be calculated. At this time, panorama synthesis is performed using the movement amount calculated from the image before the Nth image or after the N + 1th image.

  The panoramic image synthesis program of the present invention is characterized in that the panoramic image synthesis method described above is described in the order of processing steps.

  An imaging apparatus according to the present invention includes the panoramic image synthesis program described above and a control device that executes the panoramic image synthesis program.

  According to the present invention, it is possible to perform panorama synthesis with high accuracy without mounting expensive sensors in the imaging apparatus.

It is a functional block diagram of the imaging device concerning one embodiment of the present invention. 3 is a flowchart showing a panorama synthesis processing procedure executed by the imaging apparatus shown in FIG. 1. It is explanatory drawing of the panorama synthetic | combination process shown in FIG. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is explanatory drawing of the panorama synthetic | combination process of FIG. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is explanatory drawing of the panorama synthetic | combination process of FIG. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is explanatory drawing of the panorama synthetic | combination process of FIG. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is explanatory drawing of the panorama synthetic | combination process of FIG. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention. It is a flowchart which shows the panorama synthetic | combination processing procedure which concerns on another embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional configuration diagram of an imaging apparatus (in this example, a digital still camera) 20 according to an embodiment of the present invention. The imaging device 20 includes an imaging lens 21, an imaging element 22 provided on the back of the imaging lens, and analog image data output from the imaging element 22 such as automatic gain adjustment (AGC) and correlated double sampling processing. An analog signal processing unit 23 to process, an analog / digital conversion unit (A / D) 24 that converts analog image data output from the analog signal processing unit 23 into digital image data, and a system control unit (CPU) 25 to be described later And a drive unit 26 that controls the drive of the image sensor 22 in response to the instruction.

  The imaging device 20 of the present embodiment is further detachably connected to the digital signal processing unit 30, the memory control unit 36, the main memory 37 connected to the memory control unit 36, the media interface 38, and the media interface 38. A recording medium 39. The memory control unit 36, digital signal processing unit 30, and media interface 38 process data on the data bus 27 in response to an instruction from the CPU 25 through the control bus 40.

  The digital signal processing unit 30 includes an input / output processing circuit 31, an image composition location pattern frequency determination unit 32, an image processing unit 33, and a control unit 34, and a digital image output from the A / D 24 to the data bus 27. Data is captured, and well-known digital image processing (interpolation processing, white balance correction, RGB / YC conversion processing, synchronization processing, etc.) is performed, and panorama synthesis processing described later is performed.

  The system control unit (CPU) 25 has a memory 25a, and saves or updates a movement amount L, which will be described later, for performing panorama synthesis processing in the memory 25a. In this embodiment, the memory 25a is built in the system control unit 25. However, a partial area of the memory 37 may be used, or another memory connected to the buses 40 and 27 may be used.

  In the digital camera of the present embodiment, the user continuously captures still images while panning the camera by hand, or captures a moving image, and is well known for each of a plurality of continuous image data output from the image sensor 22. Image processing is performed, and each is stored in the recording medium 39 as JPEG image data. Thereafter, a series of image data is read from the recording medium 32, an overlapping area between adjacent images is found and panorama synthesized, and the combined panorama image data is stored in the recording medium 32.

  Alternatively, when the processing speed of the system control unit 25 or the digital image processing unit 33 is high, it is also possible to capture image data output from the image sensor 22 one after another and synthesize panoramas one after another during continuous shooting. is there.

  FIG. 2 is a flowchart showing a processing procedure of a panoramic image synthesis program performed by the system control unit 25 using the subordinate digital signal processing unit 30 when performing the above-described panoramic synthesis. When this program is started, a search is performed for a synthesis portion (overlapping area) of the first image data and the second image data (step S1), and the first image data and the second image data are obtained. Panorama composition is performed (step S2). Then, the difference between the position of the first sheet and the position of the second sheet, that is, the movement amount L is calculated and stored in the memory 25a of FIG. 1 (step S3). A series of processes is repeated, and the process is performed on all the continuous images.

  In the iterative process, first, in step S11, a search is made for a synthesis location of the Nth image data and the N + 1th image data, and in the next step S12, the synthesis location is determined to determine the similarity of this synthesis location. The spatial frequency F of the reflected pattern is calculated.

  Then, it is determined whether the spatial frequency F is equal to or greater than a predetermined threshold (step S13). If the spatial frequency F is equal to or greater than the threshold, a new movement amount L is obtained in step S14, and the movement amount L in the memory 25a is calculated. Update is performed, and the N-th image data and the (N + 1) -th image data are panorama synthesized using the new movement amount L (step S15), and the process returns from step B to step A.

  If the result of determination in step S13 is that the spatial frequency F determined in step S12 is less than the threshold, it is determined that the feature points determined from both image data and their movement vectors are inaccurate, and the process proceeds to step S16. Panorama synthesis of the Nth image data and the (N + 1) th image data is performed using the previous movement amount (hereinafter also referred to as movement distance) L calculated and stored in the memory 25a. move on. Step S16 is repeatedly executed when the panorama shooting is performed and the spatial frequency F becomes less than the threshold value continuously for several frames. Therefore, even if the movement amount cannot be calculated for a certain period, high-accuracy panorama composition is possible. Become.

  FIG. 3 is a diagram for explaining the processing contents in FIG. The N-2nd image shows a mountain and its base, and the next N-1st image shows a portion from the base of the mountain to the sea. For this reason, the spatial frequency of the image is high, and the panned moving distance L can be obtained with high accuracy.

  In the next Nth image, the image of the sea and the sky occupies most, and the spatial frequency of the image is low. Even if this Nth image is compared with the rightmost region (region where the sea and sky show the most part) of the N-1th image, the change in pattern is scarce. Cannot be obtained with high accuracy. Therefore, in the present embodiment, when the Nth image and the (N-1) th image are combined, a panorama is used by using the N-2th and N-1th movement distances as the movement distance between them. Synthesize.

  When a person pans the camera by hand, the change in the moving speed at each interval for each continuous shooting is small and can be regarded as constant. For this reason, in this embodiment, when it is determined that the calculation of the current movement distance is inaccurate, panorama synthesis is performed using the movement distance L obtained immediately before, and a high-accuracy panorama composite image is obtained. .

  FIG. 4 is a flowchart showing a panorama synthesis processing procedure according to another embodiment of the present invention. In the present embodiment, the process between the repeated processes (step A, step B) in FIG. 3 is different, and steps S1, S2, S3, S11, S12, and S13 are the same as in the embodiment in FIG. However, the movement distance obtained in step S3 is L1.

  In the present embodiment, as a result of the determination in step S13, panorama composition of the Nth and N + 1th sheets is performed in step S15, and in the next step S17, the movement amount of the Nth and N + 1th sheets is set to L1 in the memory 25a. Hold (update) and proceed to Step B.

  If the result of determination in step S13 is that the spatial frequency F is less than the threshold value, the number of shots at the time of calculating the moving distance is determined in step S18, and the moving distance is calculated from the Nth and N + 1th images. If it is found from the number of shots, the process proceeds from step S18 to step S16, and the Nth and N + 1th images are panorama synthesized using this moving distance, and the process proceeds to step B.

  If the result of determination in step S18 is negative, that is, if it is determined that the movement distance is not the data calculated immediately before but the data is calculated before that, the panorama composition is performed using the movement distance. Accuracy will deteriorate. Therefore, in this case, the process proceeds from step S18 to step S20, where the N + 1 and N + 2 synthesis locations are searched, and the spatial frequency F of the image at this synthesis location is calculated (step S21).

  Then, in the next step S22, it is determined whether or not the spatial frequency F is equal to or greater than a predetermined threshold value. If the spatial frequency F is equal to or greater than the threshold value, the process proceeds to step S23, and the moving distance from the (N + 1) th image and the (N + 2) th image. L2 is calculated. In the next step S24, the Nth image and the (N + 1) th image are panorama synthesized using the value of the movement distance L2.

  That is, in the present embodiment, as shown in FIG. 5, when the moving distance L1 calculated from the Nth image and the (N + 1) th image is unknown, it is obtained from the (N + 1) th image and the (N + 2) th image. The Nth image and the (N + 1) th image are panorama synthesized by using the movement distance L2. This improves the accuracy of panoramic synthesis of the Nth image and the (N + 1) th image.

  In step S25 following step S24 of FIG. 4, panoramic synthesis is performed using the movement distance L2 for the (N + 1) th image and the (N + 2) th image. In the next step S26, the variable N is replaced with N + 1, and the next step In S27, the movement distance L2 is stored in the memory 25a as the movement distance L1, and the previous data is updated. Then, step B is entered.

  If the result of determination in step 22 is that the spatial frequency F is less than the threshold value, the Nth image and the N + 1th image are panorama synthesized using the value of the movement amount L1 stored in the memory 25a (step S28), Step B is entered.

  As described above, in this embodiment, when the movement amount cannot be calculated between the immediately preceding images, the panorama synthesis is performed using the movement amount calculated between the immediately following images, so that a sensor is not used. It is possible to improve the panorama synthesis accuracy.

  FIG. 6 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 4, the same processing steps are given the same step numbers, and detailed descriptions thereof are omitted.

  The processing procedure shown in FIG. 6 is different in that the process proceeds to step S20 when steps S18 and S16 in FIG. 4 are not present and the determination result in step S13 is “spatial frequency F <threshold”. Further, step S24a is provided instead of step S24. In step S24a, “average value L = (L1 + L2) / 2” between the movement distance L2 obtained in step S23 and the movement distance L1 in the memory 25a is obtained, and the Nth and N + 1 sheets are obtained using this average value L. A panoramic composition of the eye image.

  In this embodiment, as shown in FIG. 7, the moving distance L1 between the (N-1) th image and the Nth image is known, and the moving distance L between the Nth image and the (N + 1) th image is determined. It is unknown because it is a low-frequency image, and when the moving distance L2 between the (N + 1) th image and the (N + 2) th image is known, the unknown moving distance L is assumed to be the average value of the moving distances L1, L2 before and after that Panorama composition. This makes it possible to improve the panorama synthesis accuracy without using a sensor.

  FIG. 8 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 6, the same processing steps are given the same step numbers, and detailed descriptions thereof are omitted.

  In the present embodiment, when the spatial frequency F of the Nth and N + 1th synthesis locations is less than the threshold value in step S13, the process proceeds to step S20a, and the previous N-1th and N + 1th synthesis locations are searched. Then, in the next step S21a, the spatial frequency F of the synthesis location is obtained, and in step S22, it is determined whether the frequency F is greater than or less than a threshold value.

  If the result of determination in step S22 is equal to or less than the threshold value, the movement amount L2 between the two images (N−1 and N + 1) is calculated, and in the next step S24b, N = L2−L1 The panorama composition of the Nth image and the N + 1th image is performed, and in the next step S27a, the movement amount L of the Nth and N + 1th images is stored as L1 in the memory 25a, and the process proceeds to Step B.

  In this embodiment, as shown in FIG. 9, when the movement amount L between the Nth and N + 1th images is unknown, the movement amount L2 between the N + 1th and N−1th images, The unknown movement amount L is calculated as L = L2−L1 from the movement amount L1 between the first and N−1th images, and the Nth and N + 1th images are panorama synthesized using this. This improves the synthesis accuracy without using a sensor.

  FIG. 10 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 8, the same step number is assigned to the same processing step, and detailed description thereof is omitted. In the process of FIG. 8, the movement distance L = L2−L1 of the Nth and N + 1th sheets in FIG. 9 is calculated, but in this embodiment, L2 is used and the N−1th and N + 1th images are used. The difference is that panorama composition is performed.

  That is, after calculating the (N−1) th and (N + 1) th moving distance L2 in step S23, the N−1th and N + 1th images are panorama synthesized in step S24c, and in the next step S27b, the moving distance is calculated. L = L2−L1 is held as L1 in the memory 25a, and in the next step S26, the variable N is set to N + 1 and the process proceeds to step B.

  According to the present embodiment, it is possible to omit the N-th and (N + 1) -th panorama composition processing and reduce the amount of panorama composition processing compared to the embodiment of FIG.

  FIG. 11 is a flowchart showing a panorama composition processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 2, the same step number is assigned to the same processing step, and detailed description thereof is omitted.

  When the spatial frequency F is less than the threshold value in step S13 in FIG. 2, panoramic synthesis is performed on the Nth and N + 1th images using the old movement distance L in step S16. In the present embodiment, the step preceding step S16 is performed. In S5, blur processing is performed on the Nth and N + 1th combined portions, and then panoramic combining is performed in step S16.

  According to this embodiment, when the image has a low-frequency pattern, panorama synthesis is performed after blurring the combined portion, so that the shift between the two images can be alleviated and the image quality of the panoramic image can be improved.

  FIG. 12 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 2, the same step number is assigned to the same processing step, and detailed description thereof is omitted.

  The amount of movement between the two images is unclear due to the low-frequency image and may be a repetitive pattern image. Therefore, in the present embodiment, when the determination result in step S13 is the spatial frequency F ≧ threshold value, it is next determined in step S6 whether or not there is a repetitive pattern in the combined portion. If there is a repetitive pattern, the process proceeds to step S16, and the Nth and N + 1th images are panorama synthesized using the old movement distance L in the memory 25a.

  When the panorama synthesis part includes a repeated pattern, there is a high possibility that the feature points between the Nth and N + 1th images are shifted, and the panorama synthesis accuracy may be lowered. Therefore, in this embodiment, when there is a repeated pattern, the Nth and N + 1th sheets are calculated using the old movement amount L calculated between the (N−1) th sheet and the Nth sheet without the repeated pattern. Panorama images are combined to improve the image quality of the panorama images.

  FIG. 13 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 2, the same step number is assigned to the same processing step, and detailed description thereof is omitted.

  In the processing procedure of FIG. 2, the spatial frequency at the synthesis location is determined in step S <b> 13, but in this embodiment, the spatial frequency is not used and the determination is based on the number of feature points. That is, in step S11, the Nth and N + 1th synthesis points are searched, and in the next step S7, the number of feature points at the synthesis point is determined. If the number of feature points is greater than or equal to the threshold, the process proceeds to step S14. If the number of feature points is less than the threshold, the process proceeds to step S16.

  In the present embodiment, when the number of feature points at the composition location is large, panorama composition is performed using the amount of movement calculated this time, and when the number of feature points is small, panorama composition is performed using the amount of movement calculated previously. . As a result, it is possible to improve the image quality of panorama synthesis.

  FIG. 14 is a flowchart showing a panorama composition processing procedure according to still another embodiment of the present invention. Since the panorama synthesis processing procedure of the present embodiment is basically the same as the processing procedure of FIG. 2, the same step number is assigned to the same processing step, and detailed description thereof is omitted.

  In steps S1 to S3 in the processing procedure of FIG. 2, the movement amount L is obtained on the assumption that the first and second images are not low-frequency images. However, in practice, the first and second images may be low-frequency images, and the movement amount L cannot be calculated.

  Therefore, in this embodiment, in order to deal with such a situation, for example, the movement amount L at the previous shooting performed on the previous day is stored, and the data is used at the time of shooting today. In other words, after searching for the first and second composite locations in step S1, the spatial frequency F of the composite location is calculated in the next step S31. In step S32, it is determined whether or not the spatial frequency F is equal to or higher than the threshold value. If the spatial frequency F is equal to or higher than the threshold value, the movement amount L is calculated (step S33). Panorama synthesis of the eye and second image.

  As a result of the determination in step S32, if the spatial frequency F is less than the threshold value, the process proceeds to step S36, and panorama synthesis of the first and second images is performed using the average moving distance Lave at the time of previous shooting.

  After step S2 or step S36, the process proceeds to step S35, and a variable Lave representing the movement amount average value is set to “0” (that is, initialization is performed). Then, the process proceeds to step A of the iterative process. In the iterative process, steps S11, S12, S13, S14, S15, and S16 in FIG. 2 are the same, but in this embodiment, after step S41, Lave = Lave + L is set in step S41 after step S15. In step S42, the average value of the movement amount L at the time of the current photographing is obtained as “Lave = Lave / movement amount calculation number” and stored in the memory 25a.

  Accordingly, it is possible to improve the panorama synthesis accuracy of the first and second images without using a sensor.

  FIG. 15 is a flowchart showing a processing procedure when, for example, a super wide-angle panoramic image having 360 degrees around the entire periphery is synthesized, and FIG. 16 is a conceptual explanatory diagram thereof. When a user pans the camera with his hand to shoot a panoramic image with a wide angle, the probability that the entire movement curve is on a predetermined curve is high even if the movement distance varies every time one image is taken with the camera.

  Therefore, as illustrated in FIG. 16, the movement distance from the previous time at the first, second,..., Ninth shooting is calculated, and the distribution with respect to time from the start of shooting is obtained. In FIG. 16, the movement distance from the previous time at the time of the fifth shooting cannot be calculated, but this movement distance can be obtained from the entire movement distribution curve. Note that the sixth movement distance is obtained from the movement distance from the previous time or the seventh movement distance, as in the above-described embodiment.

  Therefore, in the processing procedure of FIG. 15, the repetitive processing between step A and step B is repeated for the number of shots. In this iterative process, first, in step S51, the synthesis location of the Nth image and the (N + 1) th image is searched, the spatial frequency F of the synthesis location is calculated in step S52, and the spatial frequency F is calculated in step S53. It is determined whether or not the threshold value is exceeded. If it is determined in step S53 that the spatial frequency F is greater than or equal to the threshold value, the movement amount Ln is recorded in the next step S54, and the process returns from step B to step A. If it is determined in step S53 that the spatial frequency F <threshold, the process returns from step B to step A.

  After the repetition of step A-step B is completed, the process proceeds to step S55, and the movement distribution curve of FIG. 16 is obtained. Then, the iterative process between the next step C and step D is repeated as many times as there are places where the movement amount cannot be calculated.

  In the repetitive processing between Step C and Step D, first, the movement amount Ln of the part where the movement amount cannot be calculated from the movement distribution curve is calculated in Step S56, and the process proceeds to Step D.

  After this repetitive processing is completed, the repetitive processing between step E and step F is then repeated for the number of shots, and this processing ends. That is, in the repetitive processing between steps E and F, panorama synthesis is performed using the movement amount Ln for the Nth and N + 1th images in step S57, and the process proceeds to step F. Thereby, it is possible to improve the movement amount calculation accuracy for a portion where the movement amount greatly changes, and it is possible to synthesize an ultra-wide-angle panoramic image with high accuracy.

  FIG. 17 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. In this embodiment, when the amount of movement L cannot be calculated by comparing two low-frequency images, the amount of movement cannot be calculated at the time of this ultra-wide-angle shooting based on the movement distribution curve shown in FIG. It is characterized in that the amount of movement of the location is obtained, and panoramic composition with an ultra wide angle is performed based on the amount of movement.

  First, the iterative process between step A and step B is performed for the number of shots. In step S61, the Nth and N + 1th combined points are searched, and in step S62, the spatial frequency F of the combined point is searched. Is calculated. Then, it is determined whether or not the spatial frequency F is equal to or higher than a predetermined threshold (step S63). If the spatial frequency F is equal to or greater than the threshold, the movement amount L is obtained and recorded in the memory 25a in the next step S64, Panorama synthesis of the Nth and N + 1th images is performed (step S65), and the process returns from step B to step A.

  If the result of determination in step S63 is that the spatial frequency F <threshold, the process proceeds to step S66, where the amount of movement is calculated from the past movement distribution curve, and panorama synthesis of the Nth and N + 1th images is performed (step S66). S67), the process returns from step B to step A.

  After the above repetitive processing is completed, the process proceeds to step S68 to obtain a movement distribution curve at the time of the current super wide-angle panorama shooting, and at step S69, the average value of the movement distribution curve calculated previously and the current movement distribution curve. In the last step S70, the movement distribution curve of the average value is recorded in the memory 25a, and this process ends.

  If the photographer holds the camera by hand and rotates around itself to perform 360-degree super wide-angle panoramic shooting, it is considered that the same movement curve is drawn to some extent during one round. Therefore, even if the movement amount of the section where the current movement amount could not be calculated from the movement curve at the time of past shooting, it can be estimated that it was the same movement amount, and high accuracy panorama synthesis can be performed without using a sensor. It becomes possible.

  FIG. 18 is a flowchart showing a panorama synthesis processing procedure according to still another embodiment of the present invention. This embodiment is basically the same as the embodiment of FIG. In FIG. 17, a past movement distribution curve is used. However, this movement distribution curve is expected to be different for each camera user.

  When a camera is used by a plurality of users, it is possible to perform panoramic synthesis with high accuracy by recording and using a movement distribution curve for each user. Therefore, in this embodiment, step S45 and step S46 are performed before entering the processing procedure of FIG.

  In step S45, the camera acquires user (photographer) information. This acquisition method may be any method. A method in which the user himself designates from the menu screen may be used. Alternatively, the camera may have a recognition device (for example, a fingerprint recognition device) and make a determination. In the next step S46, the memory 25a is searched, and the past movement distribution curve of the corresponding user is extracted, and then step A is entered.

  Thereby, since panorama synthesis is performed in consideration of the habit of each user, it is possible to prevent a reduction in the accuracy of movement amount calculation and to perform high-accuracy panorama synthesis processing.

  In each of the above-described embodiments, the panoramic composition is described as being performed in the digital camera. However, the present invention can be similarly applied to a case where individual captured images are read out to a personal computer and combined in the personal computer.

  Although the embodiments described above have been described individually, it is also possible to perform panorama synthesis using a plurality of embodiments in combination. For example, when performing panoramic synthesis using a previous movement amount for a low-frequency image, as described in step S5 in FIG.

  As described above, the panoramic image synthesis method according to the embodiment is a panoramic image synthesis method for panoramic synthesis of a plurality of images taken continuously while moving the imaging apparatus, and the Nth and N + 1th images. Since the similarity between the synthesis location of the Nth image and the synthesis location of the (N + 1) th image is lower than a predetermined threshold value, the Nth image is between the N + 1th image and the N + 1th image. Panoramic composition is performed using the movement amount calculated from two consecutive images before the N-th image or after the N + 1-th image when the movement amount cannot be calculated.

  Further, the level of similarity in the panoramic image synthesis method of the embodiment is determined by determining whether or not the spatial frequency of the image at the synthesis location is equal to or higher than a threshold value.

  Further, the level of similarity in the panoramic image synthesis method of the embodiment is characterized in that it is determined by whether or not the number of feature points of the image at the synthesis location is greater than a predetermined number.

  In addition, when the movement amount (L) of the Nth image and the (N + 1) th image cannot be calculated, the panoramic image composition method of the embodiment uses the N−1th image as the movement amount (L). It is calculated as an average value (L = (L1 + L2) / 2) of the movement amount (L1) of the Nth image and the movement amount (L2) of the (N + 1) th image and the (N + 2) th image. To do.

  Further, when the panoramic image composition method of the embodiment cannot calculate the movement amount (L) of the (N−1) -th image and the N-th image, the movement amount (L) is set to the (N−2) -th image. It is calculated from the difference (L = L2−L1) between the moving distance (L2) between the image and the Nth image and the moving distance (L1) between the N−2th image and the N−1th image. Features.

  In addition, the panoramic image synthesis method according to the embodiment, when the movement amount (L) between the (N−1) th image and the Nth image cannot be calculated, The panorama synthesis is stopped, and the N-2th image and the Nth image are panorama synthesized using the moving distance between the N-2th image and the Nth image.

  The panoramic image synthesis method of the embodiment is characterized in that when synthesizing the low-similarity image, panoramic synthesis is performed after blurring the image of the synthesis portion of the low-similarity part.

  The panoramic image composition method of the embodiment performs panoramic composition using a movement amount calculated in a state where there is no common repetitive pattern between both images when there is a repetitive pattern at the composition location. And

  In addition, the panoramic image synthesis method of the embodiment uses the average value of the movement amount calculated at the previous panorama shooting when the similarity between the synthesis position of the first image and the second image is lower than a predetermined threshold. Instead, panoramic composition of the first and second images is performed.

  In the panoramic image synthesis method of the embodiment, a movement amount distribution curve of the calculated movement amount is calculated, and a movement amount that could not be calculated on the way from the movement amount distribution curve is estimated to perform panorama synthesis. It is characterized by performing.

  In addition, the panoramic image composition method of the embodiment retains data of the movement amount distribution curve calculated at the previous panorama shooting, and when the movement amount cannot be calculated at the current panorama shooting, the distribution curve is calculated from the distribution curve. The panoramic composition is performed according to the obtained movement amount.

  Further, the movement amount distribution curve data of the panoramic image synthesis method of the embodiment is stored and used for each photographer.

  The panoramic image synthesis program according to the embodiment is a panoramic image synthesis program in which the panoramic image synthesis method described above is described in the order of processing steps.

  In addition, the imaging apparatus according to the embodiment includes the panorama image synthesis program and a control device that executes the panorama image synthesis program.

  According to the embodiment described above, it is possible to synthesize a panoramic image with high accuracy even in an imaging apparatus that does not include a gyro sensor or a position sensor.

  The imaging apparatus and panoramic image synthesis method according to the present invention can be used for a compact digital camera or the like because panoramic image synthesis can be performed with high accuracy without using a sensor, and image data is taken out to an external computer. It is also suitable for panoramic composition on a computer.

20 Imaging device 21 Shooting lens 22 Imaging element 25 CPU
25a Movement amount memory 30 Digital signal processing unit

Claims (14)

  A panoramic image composition method for performing panorama composition of a plurality of images taken continuously while moving an imaging device, searching for a composition portion of an Nth image and an (N + 1) th image, and combining the Nth image If the amount of movement between the Nth image and the (N + 1) th image cannot be calculated because the similarity between the location and the N + 1th image is lower than a predetermined threshold, or before the Nth image A panoramic image synthesis method for performing panorama synthesis using the movement amount calculated from two consecutive images after an N + 1th image.   The panoramic image synthesis method according to claim 1, wherein the similarity level is determined based on whether a spatial frequency of an image at the synthesis location is equal to or higher than a threshold value.   The panoramic image synthesis method according to claim 1, wherein the similarity level is determined based on whether or not the number of feature points of the image at the synthesis location is greater than a predetermined number.   4. The panoramic image synthesis method according to claim 1, wherein when the movement amount (L) of the Nth image and the N + 1th image cannot be calculated, the movement amount (L). As an average value (L = (L1 + L2) of the movement amount (L1) of the (N-1) th image and the Nth image and the movement amount (L2) of the (N + 1) th image and the (N + 2) th image. / 2) Panorama image synthesis method.   4. The panoramic image composition method according to claim 1, wherein when the movement amount (L) between the N−1th image and the Nth image cannot be calculated, the movement amount ( L) is the difference (L1) between the movement distance (L2) of the (N-2) th image and the Nth image and the movement distance (L1) of the (N-2) th image and the (N-1) th image. = P2-L1) Panorama image synthesis method.   4. The panoramic image synthesis method according to claim 1, wherein when the movement amount (L) between the N−1th image and the Nth image cannot be calculated, N−1 images. Stop panorama synthesis of the Nth image and the Nth image, and panoramicly compose the N-2th image and the Nth image using the moving distance of the N-2th image and the Nth image Panorama image synthesis method.   The panoramic image synthesis method according to any one of claims 1 to 6, wherein when the image with low similarity is synthesized, the image at the synthesis portion of the low similarity portion is blurred. A panoramic image composition method for panoramic composition.   The panoramic image synthesis method according to any one of claims 1 to 6, wherein when there is a repetitive pattern at the synthesis location, a movement amount calculated in a state where there is no common repetitive pattern between both images. A panoramic image composition method for performing panorama composition using the.   9. The panoramic image synthesizing method according to claim 1, wherein when the similarity between the synthesis positions of the first image and the second image is lower than a predetermined threshold, the previous panorama shooting is performed. A panoramic image synthesis method for performing panoramic synthesis of the first image and the second image instead of using the calculated average value of movement amounts.   10. The panoramic image synthesis method according to claim 1, wherein a movement amount distribution curve of the calculated movement amount is calculated, and an intermediate movement amount cannot be calculated from the movement amount distribution curve. Panorama image composition method for performing panorama composition by estimating the amount of movement.   10. The panoramic image synthesis method according to claim 1, wherein movement amount distribution curve data calculated at the previous panorama shooting is stored, and the movement amount at the current panorama shooting is stored. A panoramic image composition method for performing panorama composition based on a movement amount obtained from the distribution curve when calculation cannot be performed.   13. The panoramic image synthesis method according to claim 12, wherein the movement amount distribution curve data is stored and used for each photographer.   A panoramic image synthesis program in which the panoramic image synthesis method according to any one of claims 1 to 12 is described in the order of processing steps.   An image pickup apparatus equipped with the panorama image composition program according to claim 13 and a control device that executes the panorama image composition program.

Images