JP2011199425A - Panoramic image generator, panoramic image generation method, panoramic image generation program, imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a digital camera for generating a panoramic image without sense of incongruity even when an image has parallax.SOLUTION: The digital camera includes a panoramic synthesis part 19 which acquires n (n is a natural number ≥2) pieces of image data obtained by continuous shooting, calculates a moving vector between the (k-1)-th (k is a natural number of 2 to n) image data and the k-th image data adjacent to each other in time series, and sets the boundary position between the (k-1)-th image data and the k-th image data in the middle between the center of the (k-1)-th image data and the center of the k-th image data, based on the moving vector, to perform panoramic synthesis of the (k-1)-th image data and the k-th image data.

Description

  The present invention relates to a panorama image generation apparatus, a panorama image generation method, a panorama image generation program, and an imaging apparatus.

  A method for generating panoramic image data by pasting together image data obtained by performing imaging a plurality of times is known (see, for example, Patent Documents 1 to 5).

  Patent Documents 3, 4, and 5 describe a method for determining a cutout range of image data in accordance with a movement vector between image data. However, since these methods do not consider the parallax between the image data, a step occurs at the joint between the images, and a good panoramic image cannot be obtained. Hereinafter, this reason will be described with a specific example.

  FIG. 9 is a diagram showing images A, B, and C obtained by each imaging when a certain building is divided and imaged in three times while changing the direction of the camera. As can be seen from the images A to C, if the imaging is performed a plurality of times while rotating the camera, the imaging direction of each imaging is different, so that the parallax between the images A and B and between the images B and C ( In this specification, the change in the distance between the camera and the subject occurs. For example, the ridgeline R1 of the building in the image A is a straight line rising to the right, while the ridgeline R2 of the image B is a horizontal straight line. Further, the ridge line R3 of the image C is a straight line descending to the right with respect to the ridge line R2 of the image B.

  An image obtained by superimposing these images A, B, and C by the method described in Patent Document 4 is shown in FIG. In FIG. 10, the image B is subjected to a transmission process so that the image A that should overlap with the image B and cannot be seen in the portion where the image A and the image B overlap (A + B) can be seen. Further, in the portion (B + C) where the image B and the image C overlap with each other, the image C is subjected to the transmission process so that the image B which is supposed to be invisible due to the overlap with the image C can be seen.

  A broken line position P1 illustrated in FIG. 10 is a position where the parallax between the image A and the image B becomes zero. The broken line position P2 is a position where the parallax between the image B and the image C becomes zero. At this position P1, there is an intersection between the ridge line R1 and the ridge line R2, and at the position P2, there is an intersection between the ridge line R2 and the ridge line R3. For this reason, if the image A is used up to the position P1, the image B is used from the position P1 to the position P2, and the image C is used from the position P2 onward, the difference between the images AB and the image BC is good. Panoramic images can be obtained.

  However, in the method described in Patent Document 4, since the images are not joined at the positions P1 and P2, there is a step between the ridge line R1 and the ridge line R2 at the boundary between the image A and the image B, and the boundary between the image B and the image C. Therefore, a step is formed between the ridgeline R2 and the ridgeline R3. Therefore, with this method, a good panoramic image cannot be generated for an image with parallax.

  FIG. 11 is a diagram for explaining the panoramic image generation method described in Patent Document 5. FIG. 11 schematically shows the images A, B, and C shown in FIG.

  In this method, first, a region A1 on the left side from the center of the image A is trimmed from the image A. Next, a region B1 having a length of a movement amount a between the images A and B is trimmed from the image B on the left side from the center of the image B. Next, a region C1 having a length of a movement amount b between the image B and the image C is trimmed from the image C on the left side from the center of the image C. Then, the region A1, the region B1, and the region C1 are connected in this order to generate a panoramic image D.

  Even in such a method, as shown in FIG. 11, the position P1 where the parallax is zero exists at the center of the region B1, and the position P2 where the parallax is zero exists at the center of the region C1. . For this reason, there is a step between the ridgelines at the boundary between the region A1 and the region B1, and an image with a step between the ridgelines at the boundary between the region B1 and the region C1. In this method, for example, when the movement amount b is larger than ½ of the horizontal direction of the image C, information between the region C1 and the region B1 is lost.

  12 is a diagram showing an image obtained by panoramic synthesis of the images A, B, and C shown in FIG. 9 by the method described in Patent Document 5. In FIG. As shown in FIG. 12, at the boundaries 120 and 121 between the images, a step occurs in the straight portion of the building, and a good panoramic image cannot be obtained by this method.

JP 2005-328497 A JP 2006-172163 A JP 2008-167092 A JP-A-6-326965 JP-A-6-284321

  The present invention has been made in view of the above circumstances, and is capable of generating a panoramic image that is comfortable even with parallax images, a panoramic image generating method, a panoramic image generating program, and an imaging device. The purpose is to provide.

  The panoramic image generation apparatus of the present invention is adjacent in time series to an image data acquisition unit that acquires n (n is a natural number of 2 or more) pieces of image data obtained by continuous imaging (k−1). A movement vector calculation unit for calculating a movement vector between the image data of the th (k is a natural number of 2 or more and n or less) and the kth image data, the (k−1) th image data and the kth A boundary position between the center of the (k−1) -th image data and the center of the k-th image data based on the movement vector, A panorama synthesizing unit that performs a first process of panorama synthesizing the k th image data and the k th image data.

  The imaging device of the present invention includes the panoramic image generation device and a solid-state imaging device for acquiring the n pieces of image data.

  The panoramic image generation method of the present invention is adjacent in time series to an image data acquisition step of acquiring n (n is a natural number of 2 or more) image data obtained by continuous imaging (k-1). A movement vector calculating step for calculating a movement vector between the image data (k is a natural number greater than or equal to 2 and less than or equal to n) and the kth image data, the (k−1) th image data and the kth image data A boundary position between the center of the (k−1) -th image data and the center of the k-th image data based on the movement vector, A panorama synthesizing step for performing a first process of panorama synthesizing the k th image data and the k th image data.

  The panorama image generation program of the present invention is a program for causing a computer to execute the panorama image generation method.

  According to the present invention, it is possible to provide a panorama image generation device, a panorama image generation method, a panorama image generation program, and an imaging device that can generate a panorama image that does not feel strange even with an image with parallax.

The figure which shows schematic structure of the digital camera which is an example of the imaging device for describing one Embodiment of this invention The figure explaining the usage method at the time of the panorama imaging mode of the digital camera shown in FIG. The figure for demonstrating the specific example of the process of the panorama synthetic | combination part in the digital camera shown in FIG. The figure which showed the panoramic image obtained by carrying out the panorama synthesis | combination of the image A, B, C shown in FIG. 9 by the method demonstrated in FIG. The flowchart for demonstrating operation | movement at the time of the panorama imaging mode of the digital camera shown in FIG. The flowchart for demonstrating the operation | movement at the time of the panoramic imaging mode in the 1st modification of the digital camera shown in FIG. The flowchart for demonstrating the operation | movement at the time of the panorama imaging mode in the 2nd modification of the digital camera shown in FIG. The flowchart for demonstrating operation | movement at the time of the panoramic imaging mode in the 3rd modification of the digital camera shown in FIG. The figure which shows the images A, B, and C obtained by each imaging when a certain building is divided and imaged in three times while changing the direction of the camera The figure which shows the image which shifted and overlap | superposed the image A, B, C shown in FIG. 9 by the movement amount between each image. The figure for demonstrating the panorama image generation method of patent document 5 The figure which shows the image obtained by carrying out panorama synthesis | combination of the image A, B, C shown in FIG. 9 by the method of patent document 5

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing a schematic configuration of a digital camera which is an example of an imaging apparatus for explaining an embodiment of the present invention.

  The imaging system of the illustrated digital camera 100 includes a photographic lens 1, a solid-state imaging device 5, a diaphragm 2 provided between them, an infrared cut filter 3, and an optical low-pass filter 4.

  A system control unit 11 that performs overall control of the entire electric control system of the digital camera 100 controls the lens driving unit 8 to adjust the position of the photographing lens 1 to a focus position or to perform zoom adjustment. Then, the exposure amount is adjusted by controlling the aperture amount of the aperture 2.

  In addition, the system control unit 11 drives the solid-state imaging device 5 via the imaging device driving unit 10 and outputs a subject image captured through the photographing lens 1 as an imaging signal. An instruction signal from the user is input to the system control unit 11 through the operation unit 14.

  The electrical control system of the digital camera 100 further outputs an analog signal processing unit 6 that performs analog signal processing such as correlated double sampling processing connected to the output of the solid-state imaging device 5 and the analog signal processing unit 6. And an A / D conversion circuit 7 for converting RGB color signals into digital signals, which are controlled by the system control unit 11.

  Furthermore, the electric control system of the digital camera 100 performs interpolation calculation, gamma correction calculation, RGB / YC conversion processing, and the like on the main memory 16, the memory control unit 15 connected to the main memory 16, and the imaging signal. A digital signal processing unit 17 for generating image data, a compression / decompression processing unit 18 for compressing the image data generated by the digital signal processing unit 17 into a JPEG format or decompressing the compressed image data, and time series. The external memory control to which the panorama synthesis unit 19 that generates panorama image data by synthesizing n (n is a natural number of 2 or more) image data obtained by imaging and the removable recording medium 21 is connected Unit 20 and a display control unit 22 to which a liquid crystal display unit 23 mounted on the back of the camera or the like is connected.

  The memory control unit 15, digital signal processing unit 17, compression / decompression processing unit 18, panorama synthesis unit 19, external memory control unit 20, and display control unit 22 are mutually connected by a control bus 24 and a data bus 25, and system control is performed. It is controlled by a command from the unit 11.

  In this digital camera 100, the user rotates the digital camera 100 main body as shown in FIG. 2 to change the orientation and continuously perform imaging a plurality of times, and generate panoramic image data from the image data obtained by each imaging. Thus, a panoramic imaging mode for recording on the recording medium 21 is provided.

  Below, the processing content of the panorama composition part 19 is explained in full detail. It is assumed that image data generated by the digital signal processing unit 17 by each imaging is temporarily stored in the main memory 16.

  The panorama synthesizing unit 19 acquires the n pieces of image data obtained by continuously capturing images from the main memory 16 and the (k−1) th (k) adjacent in time series from the acquired image data. Is a natural number between 2 and n) and the k-th image data, and the boundary position between the (k-1) -th image data and the k-th image data. Based on the moved vector, the center of the (k−1) th image data and the center of the kth image data is set, and the (k−1) th image data and the kth image data are panorama. And a function of performing a process of combining.

  These functions are realized by the system control unit 11 executing a panoramic image generation program stored in a ROM (not shown).

  The movement vector can be calculated, for example, by extracting the same feature point from the (k−1) th image data and the kth image data and obtaining a vector connecting them. Hereinafter, the processing content of the panorama composition unit 19 will be described with a specific example.

  FIG. 3 is a diagram for explaining a specific example of processing of the panorama synthesizing unit in the digital camera shown in FIG. In FIG. 3, n = 3, the first image data is image data A, the second image data is image data B, and the third image data is image data C. In the example of FIG. 3, description will be made assuming that imaging is performed while the digital camera 100 is rotated in the horizontal direction.

  First, the panorama composition unit 19 acquires the image data A, B, and C from the main memory 16. Next, the panorama composition unit 19 calculates a movement vector U1 between the image data A and the image data B and a movement vector U2 between the image data B and the image data C, respectively.

  The movement vector refers to information including the direction and the amount of movement. Since the image data may move not only horizontally but also vertically, the movement vector includes a horizontal component (component in a direction orthogonal to the vertical direction of the digital camera) and a vertical component orthogonal to the horizontal component.

  In this specification, the movement vector means the horizontal component. It is assumed that the direction of the movement vector U1 is the right direction and the movement amount is a. Further, it is assumed that the movement vector U2 has a right direction and a movement amount b.

  Next, the panorama composition unit 19 sets a straight line (center line) passing through the center point of the image data A and perpendicular to the direction of the movement vector U1 (right direction) to the image data A. A boundary position with the image data B is set at a distance of a / 2 from the center line toward the movement vector U1 (position indicated by a broken line in the figure).

  Further, the panorama composition unit 19 sets a straight line (center line) passing through the center point of the image data B and perpendicular to the direction of the movement vector U1 to the image data B, and the movement vector from the center line is set. A boundary position with the image data A is set at a distance a / 2 in the direction opposite to the direction U1 (position indicated by a broken line in the figure).

  Next, the panorama composition unit 19 trims, from the image data B, a region B1 that is in the direction of the movement vector U1 from the boundary position set in the image data B. Then, the region B1 is superimposed on the region A2 of the image data A that is in the direction of the movement vector U1 from the boundary position set in the image data A, and the panorama composition of the image data A and the image data B is completed.

  The data in area A2 is overwritten with the data in area B1.

  Next, the panorama composition unit 19 sets a boundary position with the image data C at a distance of b / 2 from the center line of the image data B in the direction of the movement vector U2 (right direction) (one point in the figure). Position indicated by a chain line).

  Further, the panorama composition unit 19 sets a straight line (center line) that passes through the center point of the image data C and is perpendicular to the direction of the movement vector U2 to the image data C, and the movement vector from the center line. A boundary position with the image data B is set at a distance of b / 2 in the direction opposite to the direction U2 (position indicated by a one-dot chain line in the figure).

  Next, the panorama composition unit 19 trims, from the image data C, a region C1 that is in the direction of the movement vector U2 from the boundary position set in the image data C.

  Then, the region C1 is overlaid on the region B2 of the image data B in the direction of the movement vector U2 from the boundary position set in the region B1, and the panorama synthesis of the image data B and the image data C is completed.

  The data in area B2 is overwritten with the data in area C1.

  Through the above processing, panoramic image data D is obtained.

  If there is an error in the amount of movement included in the movement vector, in the panoramic image data D, there may be a color difference or a positional shift at each of the boundary between the area A1 and the area B1 and the boundary between the area B1 and the area C1. . For this reason, it is preferable that the panorama composition unit 19 has an image quality adjustment function for adjusting the image quality at the boundary position.

  The image quality adjustment processing is performed on the ranges 30 and 31 corresponding to predetermined pixels from the boundary position between the (k−1) th image data and the kth image data.

  Examples of image quality adjustment processing include color correction processing for correcting the color differences in the regions 30 and 31 to be uniform, and blur processing for cutting the high-frequency components in the regions 30 and 31 with a high-frequency cut filter to blur the image. As mentioned. These treatments may be performed alone or in combination.

  In the description of FIG. 3, the image data A is not trimmed. However, the process of trimming the area A1 from the image data A and joining the area A1 and the area B1 at the boundary position set in both. , Panoramic synthesis of the image data A and the image data B may be performed.

  Alternatively, the region excluding the region B2 from the region B1 may be trimmed, and the right end of this region and the left end of the region C1 may be joined to perform panoramic synthesis of the image data B and the image data C.

  In the description of FIG. 3, the region B1 is trimmed from the image data B. However, without trimming the image data B, the address of each pixel in the region B1 is determined from the image data B, and each determined pixel is determined. The panorama composition of the image data A and the image data B may be performed by generating data designating that the data is arranged at the right end of the area A1.

  Similarly, without trimming the image data C, the address of each pixel in the area C1 is determined from the image data C, and data specifying that the determined pixel data is to be arranged from the right end of the area B2 is generated. Thus, panoramic synthesis of the image data B and the image data C may be performed.

  In the above description, the panorama composition of the image data B and the image data C is performed after the panorama composition of the image data A and the image data B is finished. However, the area A1, the area B1, the area B2, and the area After determining all of C1, region A1, region B1, and region C1 may be combined together. The movement vector U2 may be calculated after the panoramic synthesis of the image data A and the image data B is completed.

  The boundary position (dashed line) set for the image data A and B in FIG. 3 and the boundary position (dashed line) set for the image data B and C are positions where the parallax between the image data A and B is the smallest, respectively. This corresponds to the position where the parallax between the image data B and C is the smallest.

  That is, the panorama composition unit 19 uses the position where the parallax between the (k−1) th image data and the kth image data is the smallest as the boundary position, and the (k−1) th image data and the kth image. The data is panorama synthesized. For this reason, in the panorama image data D, the images are switched at a position where the parallax is the smallest, and the occurrence of a step at the boundary position can be prevented.

  4 is a view showing a panoramic image obtained by performing panoramic synthesis on the images A, B, and C shown in FIG. 9 by the method described with reference to FIG. As can be seen by comparing FIG. 4 and FIG. 12, in FIG. 4, there is no step at the boundaries 120 and 121 between images, and a good panoramic image can be obtained.

  Next, the operation of the digital camera 100 shown in FIG. 1 in the panoramic imaging mode will be described.

  FIG. 5 is a flowchart for explaining the operation of the digital camera shown in FIG. 1 in the panoramic imaging mode.

  First, after setting the panorama imaging mode, the user presses a shutter button included in the operation unit 14, for example, and issues an imaging start instruction. Then, in this state, the digital camera is rotated to a predetermined position, and the finger is released from the shutter button at the predetermined position to give an instruction to end imaging.

  By such an operation, continuous imaging or moving image imaging is performed by the solid-state imaging device 5 from the time point of the imaging start instruction to the time point of the imaging end instruction (step S51). A plurality of image data obtained by this continuous imaging or moving image imaging is sequentially stored in the main memory 16.

  Next, the panorama synthesis unit 19 calculates a movement vector between the image data stored in the main memory 16 (step S52).

  Next, the panorama synthesizing unit 19 determines a boundary position for each image data stored in the main memory 16 based on the calculated movement vector (step S53).

  Next, the panorama synthesizing unit 19 panorama synthesizes each image data stored in the main memory 16 according to the determined boundary position (step S54).

  Next, if there is an instruction to end the panoramic imaging mode, the process ends. If there is no instruction, the process returns to step S51 to enter an imaging standby state.

  As described above, according to the digital camera 100, it is possible to generate good panoramic image data without a step even between images having parallax.

  When imaging is performed a plurality of times by the method shown in FIG. 2, the distance between the digital camera 100 and the subject changes for each imaging. When imaging a subject that is not so far away, the change in the distance between the digital camera 100 and the subject in a plurality of imaging is particularly large. Therefore, the panoramic image data generation method described above is particularly effective when panoramic imaging is performed by the method shown in FIG.

  The processing performed by the panorama composition unit 19 can also be performed by a computer in which a panorama image generation program is installed.

  Next, a modified example of the digital camera 100 will be described.

(First modification)
In this modification, the panorama synthesizing unit 19 in the digital camera 100 shown in FIG. 1 performs panorama synthesis without using a movement vector in addition to the high-accuracy mode in which the processing described in FIG. 3 is performed as panorama synthesis processing. Implement high-speed mode. The panorama synthesizing unit 19 sets either the high accuracy mode or the high speed mode according to a manual operation.

  As a method of performing panoramic synthesis without using a movement vector, there is a method of superimposing image data by fixing the overlapping position of two adjacent image data in time series (the boundary position of two image data). . According to this method, it is not necessary to calculate a movement vector, and the processing speed can be increased.

  That is, when panoramic synthesizing the (k-1) th image data and the kth image data, the synthesizing process is performed assuming that the movement vector between these two image data is a constant value.

  For example, with the center line of the (k−1) th image data as the boundary position and the end of the kth image data in the direction opposite to the direction of the movement vector as the boundary position, these boundary positions are matched. Panorama composition is performed by superimposing the k-th image data on the (k-1) -th image data.

  Note that the information on the direction of the movement vector of the (k-1) -th image data and the k-th image data may be obtained by providing a gyro sensor in a digital camera, for example. Alternatively, the rotation direction of the digital camera may be designated by the user in the panorama imaging mode, and the panorama composition unit 19 may acquire information on the designated direction.

  FIG. 6 is a flowchart for explaining the operation in the panoramic imaging mode in the first modification of the digital camera shown in FIG. In FIG. 6, the same processes as those in FIG.

  After setting the panoramic imaging mode, the user presses a shutter button included in the operation unit 14, for example, and issues an imaging start instruction. Then, in this state, the digital camera is rotated to a predetermined position, and the finger is released from the shutter button at the predetermined position to give an instruction to end imaging.

  By such an operation, continuous imaging or moving image imaging is performed by the solid-state imaging device 5 from the time point of the imaging start instruction to the time point of the imaging end instruction (step S61). A plurality of image data obtained by this continuous imaging or moving image imaging is sequentially stored in the main memory 16.

  Next, the panorama composition unit 19 determines the set mode, and when the high accuracy mode is set (step S62: high accuracy mode), the panorama composition is performed in the processing after step S53, that is, the high accuracy mode. To implement.

  On the other hand, when the high-speed mode is set (step S62: high-speed mode), the panorama composition unit 19 omits the calculation of the movement vector and sets the boundary position of each image data stored in the main memory 16 to a fixed value. Set (step S63). After step S63, the panorama composition unit 19 performs panorama composition of each image data according to the determined boundary position (step S54).

  As described above, according to the first modification of the digital camera shown in FIG. 1, in a scene with few linear components in the direction in which the digital camera is moved, such as mountains and forests, the processing speed is given priority in the high-speed mode. Panorama scenes can be combined in a high-accuracy mode with priority on image quality in scenes with many linear components in the direction of moving a digital camera such as a building.

  In the high-speed mode, the amount of calculation can be reduced and the processing speed can be improved. For this reason, compared with a digital camera that always performs panorama composition in the high-accuracy mode, it is possible to reduce power consumption and perform high-speed panoramic image generation processing.

  Note that the process of step S52 of FIG. 6 may be performed between step S62 and step S61. In this case, when it is determined in step S62 that the high-accuracy mode is selected, the processing in step S53 may be performed using the calculated movement vector. If it is determined in step S62 that the high-speed mode is selected, the process in step S63 may be performed. Even in this case, in the high-speed mode, since the calculation for determining the boundary position using the movement vector can be omitted, it is possible to obtain the effects of reducing the calculation amount and improving the processing speed.

(Second modification)
In this modification, the panorama composition unit 19 in the digital camera shown in FIG. 1 performs the high-speed mode in addition to the high-accuracy mode, as in the first modification, as the panorama composition processing. The high-accuracy mode and the high-speed mode are automatically switched according to the subject.

  In the above description, it is assumed that panoramic imaging is performed by the method shown in FIG. However, even in the method shown in FIG. 2, when imaging a subject such as a distant mountain, the change in the distance between the digital camera 100 and the subject is not so large, and the parallax between image data is small. Therefore, in such a case, good panorama image data can be generated even if the panorama synthesis process is performed in the high-speed mode.

  Further, as a panoramic imaging method, there is a method of performing imaging a plurality of times by moving the digital camera 100 in parallel without rotating the digital camera 100. According to this method, the distance between the digital camera and the subject at the time of each imaging does not change, and no parallax occurs between the image data. Therefore, in such a case, good panorama image data can be generated even if the panorama synthesis process is performed in the high-speed mode.

  In this modification, the panorama synthesizing unit 19 determines whether or not a certain amount of parallax has occurred between the respective image data to be subjected to the panorama synthesizing process. The panorama synthesis process is performed in the high-accuracy mode, and the panorama synthesis process is performed in the high-speed mode when a certain amount of parallax has not occurred.

  The determination as to whether or not a certain amount of parallax occurs between the image data to be combined is performed as follows. First, the digital camera 100 is provided with a distance measuring sensor that measures the distance to the subject. The distance measuring sensor is connected to the bus 25, for example.

  Each time imaging for obtaining each image data to be panorama synthesis processing is performed, the system control unit 11 acquires distance information measured by the distance measuring sensor, and the image data obtained by the imaging is acquired. And the distance information is stored in the main memory 16 in association with each other.

  The panorama composition unit 19 reads the distance information corresponding to each image data in the main memory 16 and obtains the absolute value of the difference between the distance information corresponding to each of two adjacent image data in time series. If there is a set of image data in which the absolute value of the difference between the distance information exceeds the threshold value, it is determined that a certain amount of parallax has occurred. On the other hand, when there is no set of image data in which the absolute value of the distance information difference exceeds the threshold value, it is determined that a certain amount of parallax has not occurred.

  Note that the information on the distance to the subject may be acquired based on the information on the in-focus position when imaging for obtaining image data is performed.

  FIG. 7 is a flowchart for explaining the operation in the panoramic imaging mode in the second modification of the digital camera shown in FIG. In FIG. 7, the same processes as those in FIG. 6 are denoted by the same reference numerals.

  When imaging is performed in step S61, the system control unit 11 stores the distance information measured by the distance measuring sensor in the memory 16 in association with the image data obtained by the imaging. (Step S71).

  Next, the panorama composition unit 19 determines from the distance information whether or not a certain amount of parallax has occurred between the image data to be subjected to the panorama composition processing (step S72).

  If a certain amount of parallax has occurred (step S72: YES), panorama composition is performed in the high accuracy mode. That is, the process after step S52 is performed.

  On the other hand, when the parallax exceeding a certain level does not occur (step S72: NO), panorama composition is performed in the high speed mode. That is, the process after step S63 is performed.

  As described above, according to the second modification of the digital camera shown in FIG. 1, the camera can automatically determine and switch between the high-speed mode and the high-accuracy mode. Therefore, power consumption, processing speed, and image quality can be optimized without any special operation.

  In addition, you may perform the process of FIG.7 S52 between step S71 and step S72. In this case, when it is determined in step S72 that there is parallax, the process of step S53 may be performed using the calculated movement vector. If it is determined in step S72 that there is no parallax, the process of step S63 may be performed. Even in this case, in step S63, since the calculation for determining the boundary position using the movement vector can be omitted, it is possible to obtain the effects of reducing the calculation amount and improving the processing speed.

(Third modification)
In this modification, the panorama composition unit 19 in the digital camera shown in FIG. 1 performs the high-speed mode in addition to the high-accuracy mode, as in the first modification, as the panorama composition processing. The high accuracy mode and the high speed mode are automatically switched according to the rotation angle per frame of the digital camera.

  Further, in this digital camera, a sensor for detecting a rotation angle of the digital camera such as a gyro sensor is provided in the configuration shown in FIG.

  In this modified example, when a plurality of times of imaging is performed in the panorama imaging mode, the panorama composition unit 19 determines the rotation angle of the digital camera from the time when the imaging is started to the time when the imaging is ended, to the gyro sensor. Get from. Further, the panorama composition unit 19 acquires from the system control unit 11 the frame rate (the number of frames per second) of the solid-state imaging device 5 when the plurality of images are taken. The frame rate referred to here is information indicating how many frames from which image data used for panorama composition is obtained are included per second.

  The panorama synthesizing unit 19 calculates a rotation angle per unit frame (for example, one frame) at the time of panoramic imaging from the acquired rotation angle and frame rate, and compares the calculated value with a threshold value.

  The panorama synthesizing unit 19 sets the high accuracy mode when the calculated value is larger than the threshold value, and sets the high speed mode when the calculated value is equal to or less than the threshold value.

  Note that this threshold value is a value at which the parallax between adjacent image data increases when the rotation angle is larger than this, and in the high-speed mode, the step between the image data to be synthesized is noticeable.

  FIG. 8 is a flowchart for explaining the operation in the panoramic imaging mode in the third modification of the digital camera shown in FIG. In FIG. 8, the same processes as those in FIG. 6 are denoted by the same reference numerals.

  After step S61, the panorama synthesizing unit 19 obtains the rotation angle of the digital camera from the imaging start to the imaging end of step S61 from the gyro sensor, and also determines the frame rate of the solid-state imaging device 5 at this time as the system control unit 11. (Step S81).

  Next, the panorama composition unit 19 calculates a rotation angle per unit frame from the acquired rotation angle and frame rate, and compares this with a threshold value.

  If the calculated value is larger than the threshold value (step S82: Y), panorama synthesis is performed in the high accuracy mode for each image data stored in the main memory 16. That is, the process after step S52 is performed.

  On the other hand, if the calculated value is equal to or less than the threshold value (step S82: N), panorama synthesis is performed in the high speed mode for each image data stored in the main memory 16. That is, the process after step S63 is performed.

  As described above, according to the third modification of the digital camera shown in FIG. 1, the camera can automatically determine and switch between the high-speed mode and the high-accuracy mode. Therefore, power consumption, processing speed, and image quality can be optimized without any special operation.

  Note that the process of step S52 of FIG. 8 may be performed between step S81 and step S82. In this case, when it is determined in step S82 that the rotation angle is larger than the threshold value, the process of step S53 may be performed using the calculated movement vector. If it is determined in step S82 that the rotation angle is equal to or smaller than the threshold value, the process of step S63 may be performed. Even in this case, in step S63, since the calculation for determining the boundary position using the movement vector can be omitted, it is possible to obtain the effects of reducing the calculation amount and improving the processing speed.

  As described above, the following items are disclosed in this specification.

  The disclosed panoramic image generation apparatus is adjacent in time series to an image data acquisition unit that acquires n (n is a natural number of 2 or more) pieces of image data obtained by continuous imaging (k−1). A movement vector calculation unit for calculating a movement vector between the image data of the th (k is a natural number of 2 or more and n or less) and the kth image data, the (k−1) th image data and the kth A boundary position between the center of the (k−1) -th image data and the center of the k-th image data based on the movement vector, A panorama synthesizing unit for performing panorama synthesizing of the image data and the k-th image data.

  With this configuration, since the boundary position is set in the middle between the center of the (k−1) th image data and the center of the kth image data, the (k−1) th image data is at the position with the smallest parallax. And k-th image data are combined, and a good panoramic image can be generated.

  In the disclosed panoramic image generation apparatus, the panorama synthesis unit is configured to make the first of the movement vectors from the center of the (k−1) th image data toward the first direction that is the direction of the movement vector. The boundary position is set to a distance that is half the movement amount in the direction, and the boundary is set to a distance that is half the movement amount from the center of the k-th image data toward the second direction opposite to the first direction. A position is set, and the region in the first direction from the boundary position of the (k−1) th image data is in the first direction from the boundary position of the kth image data. The image processing apparatus includes an image quality adjustment unit that performs the first process by overwriting the area and performs image processing for adjusting the image quality at the boundary position of the image data after the first process.

  In the disclosed panoramic image generation apparatus, the image processing includes color correction processing.

  In the disclosed panoramic image generation apparatus, the image processing includes blurring processing.

  The disclosed imaging device includes the panoramic image generation device and a solid-state imaging device for acquiring the n pieces of image data.

  In the disclosed imaging device, the panorama synthesis unit performs the (k−1) th in a method other than the first mode in which the first process is performed and the first process in which the movement vector is not used. It has a second mode for panoramic synthesis of image data and the k-th image data, and includes a mode switching unit that automatically switches between the first mode and the second mode.

  In the disclosed imaging device, the mode switching unit determines whether or not a certain amount of parallax has occurred between the image data, and determines that the certain amount of parallax has occurred. One mode is set, and the second mode is set when it is determined that the above-mentioned parallax is not generated.

  The disclosed imaging apparatus has a panoramic imaging mode that continuously performs imaging, and obtains a rotation angle and a frame rate of the imaging apparatus from the start to the end of the imaging in the panoramic imaging mode The mode switching unit sets the first mode when a rotation angle per unit frame obtained from the rotation angle and the frame rate is greater than a threshold value, and the rotation angle is equal to or less than the threshold value. To set the second mode.

  In the disclosed imaging device, the panorama synthesis unit performs the (k−1) th in a method other than the first mode in which the first process is performed and the first process in which the movement vector is not used. It has a second mode for panoramic synthesizing the image data and the k-th image data, and includes a mode switching unit that switches between the first mode and the second mode according to manual settings.

  The disclosed panoramic image generation method is adjacent in time series to an image data acquisition step of acquiring n (n is a natural number of 2 or more) image data obtained by continuous imaging (k−1). A movement vector calculating step for calculating a movement vector between the image data (k is a natural number greater than or equal to 2 and less than or equal to n) and the kth image data, the (k−1) th image data and the kth image data A boundary position between the center of the (k−1) -th image data and the center of the k-th image data based on the movement vector, A panorama synthesizing step for performing a first process of panorama synthesizing the k th image data and the k th image data.

  In the disclosed panoramic image generation method, in the panorama synthesizing step, the first direction of the movement vector from the center of the (k−1) th image data toward the first direction that is the direction of the movement vector. The boundary position is set at a distance that is half of the movement amount, and the boundary position is set at a distance that is half the movement amount from the center of the k-th image data toward the second direction opposite to the first direction. And the region in the first direction from the boundary position of the kth image data in the region in the first direction from the boundary position of the (k-1) th image data Is overwritten, and includes an image quality adjustment step of performing the first process and performing an image process for adjusting the image quality at the boundary position of the image data after the first process.

  In the disclosed panoramic image generation method, the image processing includes color correction processing.

  In the disclosed panoramic image generation method, the image processing includes blurring processing.

  In the disclosed panorama image generation method, in the panorama synthesis step, the (k-1) is performed by a method other than the first mode in which the first process is performed and the first process in which the movement vector is not used. The second image data and the second mode for panoramic composition of the kth image data are automatically switched.

  In the disclosed panoramic image generation method, in the panorama synthesis step, it is determined whether or not a certain amount of parallax has occurred between the image data, and when it is determined that the certain amount of parallax has occurred or not The second mode is performed when the first mode is performed and when it is determined that the predetermined parallax has not occurred.

  The disclosed panoramic image generation method includes a step of acquiring a rotation angle and a frame rate of an imaging apparatus that has performed the imaging from the start to the end of imaging to obtain the n imaging data, In the panorama synthesis step, the first mode is performed when the rotation angle per unit frame obtained from the rotation angle and the frame rate is larger than a threshold value, and when the rotation angle is equal to or less than the threshold value, the second mode is performed. The mode is to be implemented.

  In the disclosed panorama image generation method, in the panorama synthesis step, the (k-1) is performed by a method other than the first mode in which the first process is performed and the first process in which the movement vector is not used. The second mode for panoramic composition of the kth image data and the kth image data is switched according to the manual setting.

  The disclosed panoramic image generation program is a program for causing a computer to execute the panoramic image generation method.

100 Digital Camera 19 Panorama Composition Unit A First Image Data B Second Image Data C Third Image Data D Panorama Image Data

Claims (18)

An image data acquisition unit for acquiring n (n is a natural number of 2 or more) image data obtained by continuously capturing images;
A movement vector calculation unit that calculates a movement vector between the (k−1) th (k is a natural number of 2 to n) adjacent in time series and the kth image data;
The boundary position between the (k−1) th image data and the kth image data is determined based on the movement vector, and the center of the (k−1) th image data and the center of the kth image data. And a panorama image generation apparatus including a panorama composition unit that performs a first process of panorama compositing the (k−1) th image data and the kth image data. The panoramic image generating apparatus according to claim 1,
The panorama synthesizing unit moves the distance from the center of the (k−1) -th image data to a first direction that is the direction of the movement vector by a half of the movement amount of the movement vector in the first direction. The boundary position is set, the boundary position is set at a distance half the movement amount from the center of the k-th image data in the second direction opposite to the first direction, and the (k− 1) By overwriting the region in the first direction with respect to the boundary position of the k-th image data in the region in the first direction with respect to the boundary position of the first image data, the first One process,
A panoramic image generation apparatus including an image quality adjustment unit that performs image processing for adjusting image quality at the boundary position of the image data after the first processing. The panoramic image generation apparatus according to claim 2,
A panoramic image generation apparatus in which the image processing includes color correction processing. The panoramic image generation device according to claim 2 or 3,
A panoramic image generation apparatus in which the image processing includes blurring processing. A panoramic image generation apparatus according to any one of claims 1 to 4,
An imaging apparatus comprising: a solid-state imaging device for acquiring the n pieces of image data. The imaging apparatus according to claim 5, wherein
The panorama synthesizing unit performs a method other than the first mode in which the first process is performed and the first process that does not use the movement vector, and the (k−1) th image data and the kth image data. A second mode for panoramic synthesis of image data,
An imaging apparatus comprising a mode switching unit that automatically switches between the first mode and the second mode. The imaging apparatus according to claim 6,
The mode switching unit determines whether or not a certain amount of parallax has occurred between the image data, and sets the first mode when it is determined that the certain amount or more of parallax has occurred, An imaging device that sets the second mode when it is determined that the predetermined parallax has not occurred. The imaging apparatus according to claim 6,
It has a panoramic imaging mode that performs continuous imaging,
An acquisition unit that acquires a rotation angle and a frame rate of the imaging apparatus between the start and end of the imaging in the panoramic imaging mode;
The mode switching unit sets the first mode when a rotation angle per unit frame obtained from the rotation angle and the frame rate is greater than a threshold value, and sets the second mode when the rotation angle is equal to or less than the threshold value. Imaging device for setting the mode. The imaging apparatus according to claim 5, wherein
The panorama synthesizing unit performs a method other than the first mode in which the first process is performed and the first process that does not use the movement vector, and the (k−1) th image data and the kth image data. A second mode for panoramic synthesis of image data,
An imaging apparatus comprising a mode switching unit that switches between the first mode and the second mode according to a manual setting. An image data acquisition step of acquiring n (n is a natural number of 2 or more) image data obtained by continuously imaging;
A movement vector calculation step of calculating a movement vector between the (k−1) th (k is a natural number of 2 to n) adjacent in time series and the kth image data;
The boundary position between the (k−1) th image data and the kth image data is determined based on the movement vector, and the center of the (k−1) th image data and the center of the kth image data. A panorama image generation method comprising: a panorama composition step for performing a first process for panorama composition of the (k-1) th image data and the kth image data. The panoramic image generation method according to claim 10,
In the panorama synthesizing step, the distance from the center of the (k−1) -th image data to a first direction which is the direction of the movement vector is a half of the movement amount of the movement vector in the first direction. The boundary position is set, and the boundary is set to a distance that is half of the movement amount of the movement vector in the first direction from the center of the k-th image data toward the second direction opposite to the first direction. A position is set, and the region in the first direction from the boundary position of the (k−1) th image data is in the first direction from the boundary position of the kth image data. Performing the first process by overwriting the area,
A panoramic image generation method comprising an image quality adjustment step of performing image processing for adjusting image quality at the boundary position of the image data after the first processing. The panoramic image generation method according to claim 11,
A panoramic image generation method in which the image processing includes color correction processing. A panoramic image generation method according to claim 11 or 12,
A panoramic image generation method in which the image processing includes blurring. A panoramic image generation method according to any one of claims 10 to 13,
In the panorama synthesis step, the (k−1) -th image data and the k-th image are obtained by a method other than the first mode in which the first process is performed and the first process that does not use the movement vector. A panoramic image generation method for automatically switching between a second mode for panoramic composition of image data. The panoramic image generation method according to claim 14,
In the panorama synthesis step, it is determined whether or not a certain amount of parallax has occurred between the image data, and when it is determined that the certain amount or more of parallax has occurred, the first mode is performed, A panoramic image generation method for executing the second mode when it is determined that the predetermined parallax has not occurred. The panoramic image generation method according to claim 14,
Obtaining a rotation angle and a frame rate of an imaging apparatus that has performed the imaging from the start to the end of imaging to obtain the n imaging data,
In the panorama synthesis step, the first mode is performed when a rotation angle per unit frame obtained from the rotation angle and the frame rate is larger than a threshold value, and when the rotation angle is equal to or less than the threshold value, the second mode is performed. Panorama image generating method for implementing the mode of the above. A panoramic image generation method according to any one of claims 10 to 13,
In the panorama synthesis step, the (k−1) -th image data and the k-th image are obtained by a method other than the first mode in which the first process is performed and the first process that does not use the movement vector. A panoramic image generation method for switching a second mode for panoramic synthesis of image data according to a manual setting.   A panorama image generation program for causing a computer to execute the panorama image generation method according to any one of claims 10 to 17.