JP2015181302A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of generating data of a wide-angle image with no discomfort.SOLUTION: An image processing device 1 includes: a positioning unit 55 that performs positioning of adjacent photographing images; and a panoramic image generating unit 56 that generates data of a panoramic image by combining the adjacent photographing images on the basis of a result of the positioning, in panoramic image generation processing. The positioning unit 55 includes: a feature point tracking unit 552 that calculates a vector of a plurality of feature points between the adjacent photographing images; and an image positioning unit 554 that performs positioning of the adjacent photographing images while avoiding dynamic bodies in the photographing images on the basis of the calculated vector of the plurality of feature points.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In a digital camera, a mobile phone having an imaging function, etc., the limit of the imaging angle of view depends on hardware specifications provided in the apparatus main body, such as the focal length of the lens and the size of the imaging element. Therefore, a technique such as panoramic imaging for obtaining a wide-angle image exceeding the hardware specifications has been conventionally known.

  In order to realize panoramic imaging, for example, the user rotates the digital camera in the horizontal direction while maintaining the state where the shutter switch is pressed and holding the body as an axis, and the digital camera is substantially fixed in the vertical direction. Move to. Then, the digital camera executes a plurality of imaging processes in the meantime, and obtains data of a plurality of images (hereinafter referred to as “captured images”) obtained as a result of each of the plurality of imaging processes in the horizontal direction (horizontal). Direction), wide-angle image data such as a panoramic image is generated.

  For example, in Patent Document 1, a feature point in a captured image is detected every time a plurality of image capturing processes are performed, and two such images are arranged so that the feature points of two adjacent captured images coincide with each other. A technique for generating panoramic image data by synthesizing captured image data is disclosed.

JP-A-11-282100

By the way, in the imaging of a wide-angle image typified by a panoramic image, when a moving object that moves regardless of the movement of the digital camera is included in the imaging target, the adjacent captured images are synthesized simply by looking at the matching of the feature points. With this method, there is a risk that panoramic image data with a sense of incongruity may be generated.
This point will be specifically described with reference to FIG. FIG. 6 is a diagram illustrating an example of a panoramic image P1 obtained as a result of panoramic imaging. The data of the panoramic image P1 shown in FIG. 6B is panoramic image data obtained by combining the data of the plurality of captured images F1, F2, F3, F4, and F5 shown in FIG. . Referring to FIG. 6A, moving bodies M1, M2, M3, and M4 are included in the plurality of captured images F1 to F5. When each data of the captured images F1 to F5 including such moving bodies M1 to M4 is synthesized by a conventional method, a part of the moving bodies M1 to M4 may be cut off, and as a result, FIG. ), There is a possibility that data of the panoramic image P1 having a sense of incongruity may be generated.

  The present invention has been made in view of such a problem, and an object of the present invention is to generate a composite image without a sense of incongruity.

  In order to achieve the above object, an image processing apparatus according to one embodiment of the present invention includes an acquisition unit that acquires a plurality of images that are continuously captured with the movement of an imaging unit, and a plurality of images that are acquired by the acquisition unit. Based on the feature points detected by the feature point detection means, the feature point detection means for detecting feature points in the plurality of regions divided by the note splitting means, respectively. A vector calculating unit that calculates a plurality of vectors corresponding to the plurality of regions between adjacent images of the image of the image, and based on the plurality of vectors calculated by the vector calculating unit, from the plurality of divided regions A region selection unit that selects a region and a raw image that generates a composite image by combining the adjacent images based on the specific region selected by the region selection unit. Characterized in that it comprises a means.

  According to the present invention, it is possible to generate a composite image without a sense of incongruity.

1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing a panorama image generation process among the functional structures of the image processing apparatus of FIG. It is a figure for demonstrating the function of the position alignment part of an image process part. It is a figure for demonstrating the function of the position alignment part and panorama image generation part of an image processing part. 3 is a flowchart for explaining a flow of panoramic image generation processing executed by the image processing apparatus of FIG. 1 having the functional configuration of FIG. 2. It is a figure which shows the panoramic image obtained as a result of panoramic imaging.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus 1 according to an embodiment of the present invention.
The image processing apparatus 1 is configured as a digital camera, for example.

  The image processing apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an imaging. A unit 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22 are provided.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 20 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.

  The CPU 11, ROM 12 and RAM 13 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

  The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject. The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range. The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like. The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17.
Such an output signal of the imaging unit 17 is hereinafter referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11, the image processing unit 14, and the like.

The input unit 18 includes various buttons such as a shutter switch, and inputs various information and commands in accordance with user instruction operations.
The output unit 19 includes a display, a speaker, and the like, and outputs images and sounds.
The storage unit 20 is configured by a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 22. The program read from the removable medium 31 by the drive 22 is installed in the storage unit 20 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

  FIG. 2 is a functional block diagram showing a functional configuration for executing a panoramic image generation process among the functional configurations of such an image processing apparatus 1. Here, “panoramic image generation processing” refers to processing for generating panoramic image data using data of a plurality of captured images acquired successively. “Panorama image” refers to a 2: 3 aspect ratio image taken with a 35 mm silver salt film or a 3: 4 aspect ratio image taken with a digital camera. It is an example of a wide-angle image.

  When the image processing apparatus 1 executes the panoramic image generation process, the CPU 11 functions as the imaging control unit 51 and the storage control unit 52. In this case, the storage unit 20 is provided with an image storage unit 53 as an area for storing various image data such as captured image data and panoramic image data.

The imaging control unit 51 controls various imaging operations by the imaging unit 17.
Specifically, when the user presses the shutter switch of the input unit 18 while holding the image processing apparatus 1 as a digital camera, the image processing apparatus 1 starts panoramic image generation processing. When the panorama image generation process starts, the imaging control unit 51 starts the continuous imaging operation of the imaging unit 17 and causes an image to be captured every time a predetermined time elapses or the image processing apparatus 1 moves a predetermined amount.
During this time, when a predetermined condition is satisfied, such as when the user remains in the state of pressing the shutter switch for a predetermined time or when the digital camera moves a predetermined amount, the imaging control unit 51 captures the imaging unit 17. The continuous imaging operation at is terminated, and the panoramic image generation process is terminated.

  The storage control unit 52 executes control for storing various image data such as panoramic image data generated as a result of the panoramic image generation process in the image storage unit 53.

  Further, when the image processing apparatus 1 executes the panoramic image generation process, the image processing unit 14 functions as the image acquisition unit 54, the alignment unit 55, and the panoramic image generation unit 56.

  The image acquisition unit 54 sequentially acquires data of a plurality of captured images output by the continuous imaging operation of the imaging unit 17, that is, data of a plurality of images captured continuously.

The alignment unit 55 performs alignment of adjacent captured images among the plurality of captured images, with each of the plurality of captured image data acquired by the image acquisition unit 54 as a processing target. The adjacent captured images are an n-th captured image (n is an integer value of 1 or more) and an (n + 1) -th captured image among a plurality of continuously captured images.
Here, in order to prevent panoramic image data having a sense of incongruity due to the movement of a moving object such as a person regardless of the movement of the digital camera, the alignment unit 55 of the present embodiment has a feature. A point detection unit 551, a feature point tracking unit 552, a similarity calculation unit 553, and an image registration unit 554 are included. At this time, the feature point detection unit 551 functions as a feature point detection unit, the feature point tracking unit 442 functions as a vector calculation unit, and the similarity calculation unit 553 and the image registration unit 554 function as a region selection unit.

The feature point detection unit 551 detects a plurality of feature points from the captured image using each of the plurality of captured image data acquired by the image acquisition unit 54 as a processing target. That is, the feature point detection unit 551 detects a plurality of feature points from the acquired captured image every time the image acquisition unit 54 acquires captured image data. The feature point detection unit 551 stores the detected feature points in the image storage unit 53 in association with the captured image data acquired by the image acquisition unit 54.
At this time, the feature point detection unit 551 divides the captured image into a plurality of continuous regions (areas), and detects a feature point from each of the plurality of regions. In addition, the area | region which divides a captured image can be made into arbitrary shapes, For example, a rectangular area | region can be employ | adopted. Further, as will be described later, the plurality of areas are continuously arranged in the moving direction of the digital camera (hereinafter referred to as “panoramic imaging direction”) and the direction substantially orthogonal to the direction. Note that the term “substantially orthogonal” means that the panorama imaging direction may not be linear due to camera shake or the like. For simplicity, the direction in which the vertical direction is substantially orthogonal to the horizontal panorama imaging direction. It becomes.
Here, as a method for detecting a feature point from a captured image, a publicly known method can be employed. For example, a SIFT (Scale Invariant Feature Transform) may be used, and Harris corner detection may be used. Alternatively, other methods may be used.

  The feature point tracking unit 552 uses a plurality of feature points in each of the adjacent captured images as a processing target, and for each of the plurality of feature points, a vector indicating how the feature points have moved in the adjacent captured images, so-called A movement vector is calculated. That is, the feature point tracking unit 552, when the feature point detection unit 551 detects a plurality of feature points from the (n + 1) th captured image, the image storage unit 53 from among the detected feature points of the (n + 1) th captured image. The corresponding point corresponding to the feature point of the n-th captured image stored in the n-th captured image is specified, and the vector from the feature point of the n-th captured image to the corresponding point of the n + 1-th captured image is determined as the feature point. Calculate as a vector. At this time, the feature point tracking unit 552 stores the calculated vector in the image storage unit 53 in association with the data of the nth captured image. In the present embodiment, the vector indicates the direction in which the feature point has moved and the magnitude (distance) of the movement.

The similarity calculation unit 553 calculates the vector similarity for each region in the image using the vector calculated by the feature point tracking unit 552 as a processing target. Since the moving object moves regardless of the movement of the digital camera, the similarity calculation unit 553 calculates the vector similarity, thereby specifying the position of the moving object in the n-th captured image. By aligning adjacent captured images while avoiding the position of the moving object specified in this way, it is possible to prevent generation of unnatural panoramic image data.
Here, in the panoramic image generation process, it is common to perform alignment of data of adjacent captured images along a straight line in a direction substantially orthogonal to the panoramic imaging direction. Therefore, the similarity calculation unit 553 calculates the vector similarity in each of a group of regions arranged adjacent to each other in a direction substantially orthogonal to the panoramic imaging direction among the plurality of regions. In the digital camera, since the captured image may be slightly corrected at the ends (upper end, lower end, left end, and right end) due to lens distortion and the like, the similarity calculation unit 553 may calculate the captured image. It is preferable to calculate the vector similarity only for the region near the center excluding the edge. At this time, the size of the area near the center may be set based on the lens performance, for example, and based on the moving distance between adjacent captured images such as the moving speed and imaging timing of the digital camera. It may be set (when the movement is large, the vicinity of the center is large, and when the movement is small, the vicinity of the center is also small), and can be set arbitrarily. The use of such a region near the center may be applied only when calculating the similarity of vectors, or may be applied when dividing the region when detecting feature points.
Note that the similarity between vectors means that the direction in which the feature point has moved and the magnitude of the movement are similar. In this embodiment, the similarity is calculated by SAD (Sum of Absolute Difference) calculation. That is, the similarity calculation unit 553 calculates the vector similarity by calculating and adding the difference between adjacent vectors in a group of regions. In such SAD calculation, the smaller the calculation result, the more similar the vectors in the group of regions, and the larger the calculation result, the less the vectors in the group of regions. At this time, when the feature points for which the corresponding points cannot be specified are included in the feature points of the plurality of regions included in the group of regions, the similarity calculation unit 553 causes the vectors in the group of regions to be similar. It is preferable to judge that it is not. This is because it is highly possible that the corresponding point cannot be specified as the moving object moves (for example, because the moving object has moved to the front of the (n + 1) th feature point). The calculation of the similarity is not limited to the SAD calculation, and may be performed by an arbitrary method. For example, the similarity may be calculated by a known method such as an SSD (Sum of Squared Intensity Difference) calculation or a standard deviation.

The image alignment unit 554 aligns adjacent captured images based on the feature point vector calculated by the feature point tracking unit 552. In other words, the image alignment unit 554 aligns adjacent captured images while avoiding the positions of feature points that show abnormal vectors compared to other feature points.
More specifically, the image alignment unit 554 aligns adjacent captured images based on the similarity of vectors in a group of regions calculated by the similarity calculation unit 553. Here, the image alignment unit 554 may perform alignment at the position of a group of regions where the vectors are similar to each other by a predetermined value or more, and the position of the group of regions where the vectors are most similar. It is good also as aligning by. Thereby, the adjacent captured images can be aligned with each other at a position avoiding the moving object that moves regardless of the movement of the digital camera.

  The panoramic image generation unit 56 combines the data of the captured images aligned by the alignment unit 55 to generate panoramic image data. Further, the panorama image generation unit 56 stores the generated panorama image data in the image storage unit 53.

Next, the execution function of the panoramic image generation process will be specifically described with reference to FIGS.
FIG. 3 is a diagram for explaining the functions of the alignment unit 55 of the image processing unit 14, and FIG. 4 is a diagram for explaining the functions of the alignment unit 55 and the panoramic image generation unit 56 of the image processing unit 14. FIG. Here, in FIGS. 3 and 4, the panoramic imaging direction is from the left to the right in the drawing.

  Referring to FIG. 3A, the image acquisition unit 54 sequentially acquires data of captured images that are continuously captured. In FIG. 3A, the data of the captured image 70 and the data of the captured image 80 are acquired. Here, the captured image 70 is an n-th captured image, and is a captured image in which a moving body 71 indicating a person running from the left to the right in the drawing and a still body 73 indicating a background are captured. The captured image 80 is the (n + 1) th captured image and is a captured image in which the moving body 81 and the still body 83 are captured.

  Referring to FIG. 3B, when the image acquisition unit 54 acquires captured image data, the feature point detection unit 551 detects feature points for each region from the sequentially acquired captured images, and the feature points. The tracking unit 552 identifies corresponding points in adjacent captured images. In FIG. 3B, a plurality of feature points including feature points A, B, C, D, E, F, G, H, and I are detected from the n-th captured image 70, and the (n + 1) -th captured image. Corresponding points A ′, B ′, C ′, D ′, E ′, F ′, G ′, H ′, I ′ corresponding to the feature points A to I are detected from 80. Note that the feature points A to C, the feature points D to F, and the feature points G to I are feature points of a group of regions adjacent to each other in a direction substantially orthogonal to the panoramic imaging direction. The feature points A to G and the corresponding points A ′ to G ′ are the feature points and corresponding points detected from the still bodies 73 and 83, and the feature points H and I and the corresponding points H ′ and I ′ are moving objects. These are feature points and corresponding points detected from 71 and 82.

Referring to FIG. 3C, when the feature point tracking unit 552 detects the feature point and the corresponding point in the adjacent captured image, the feature point tracking unit 552 detects the (n + 1) th captured image 80 from the feature point of the nth captured image 70. A vector up to the corresponding point is calculated. That is, the feature point tracking unit 552 calculates a vector from the feature point A to the corresponding point A ′, for example, as shown in FIG.
When the feature point tracking unit 552 calculates the vector from the feature point to the corresponding point in this way, the similarity calculation unit 553 is a group of regions arranged adjacent to each other in a direction substantially orthogonal to the panoramic imaging direction. Calculate the similarity of vectors. Here, in FIG. 3C, for the corresponding points A ′ to G ′ from the feature points A to G, a vector accompanying the movement of the digital camera in the panoramic imaging direction is calculated, while the feature point H , I from the corresponding points H ′, I ′, vectors that are not related to the movement of the digital camera are calculated.
Therefore, the image alignment unit 554 can determine that the vectors of the group of regions including the feature points A to C and the group of regions including the feature points D to F are similar based on the similarity calculated by the similarity calculation unit 553. It can be determined that the vectors of the group of regions including the feature points G to I are not similar.

Referring to FIG. 4A, image alignment unit 554 aligns adjacent captured images based on the result of vector similarity determination.
Here, as shown in FIG. 4B, the image alignment unit 554 determines that the vectors from the feature points A to C to the corresponding points A ′ to C ′ are the most similar. Positioning is performed at positions A to C.

  That is, as shown in FIG. 4B, the image alignment unit 554 matches the feature points A and corresponding points A ′, feature points B and corresponding points B ′, feature points C and corresponding points C ′. Then, the captured images 70 and 80 are aligned. Thereby, the alignment of the captured images 70 and 80 is performed at a position that does not include the moving object 71 included in the captured image 70.

With reference to FIG. 4C, when the image alignment unit 554 aligns the captured images 70 and 80, the panoramic image generation unit 56 converts the data of the captured images 70 and 80 based on the alignment result. The synthesized image 90 is generated by combining the images.
As described above, in the image processing apparatus 1 according to the present embodiment, the captured images 70 and 80 are aligned while avoiding the moving bodies 71 and 81. Therefore, the moving body 71 and 81 are partly cut out, and the composition is strange. Generation of image data can be prevented. That is, as shown in FIG. 4C, it is possible to generate data of a composite image 90 that directly includes the moving object 81 included in the captured image 80.

Moreover, since the image processing apparatus 1 according to the present embodiment aligns adjacent captured images while avoiding moving objects, the moving objects included in each captured image can be displayed in various modes. That is, it is possible to generate panoramic image data that does not include any moving object, to generate panoramic image data that includes only one moving object, and to see how the moving object continuously moves. It is also possible to generate panoramic image data.
Such a moving object display mode can be realized by considering the panoramic imaging direction and the position of the moving object in the captured image in addition to the similarity of the feature point vectors. For example, the moving object included in the nth captured image can be removed by performing alignment at a position rearward in the panoramic imaging direction from the position of the moving object in the nth captured image. The moving object included in the n-th captured image can be displayed by performing alignment at a position in front of the panoramic imaging direction with respect to the position of the moving object in the captured image of the eye.

Next, the panorama image generation process will be described with reference to FIG. FIG. 5 is a flowchart showing a flow of panoramic image generation processing executed by the image processing apparatus 1 having the functional configuration of FIG.
Note that the panorama image generation process is triggered by the user's operation to start the panorama image generation process on the input unit 18, that is, the shutter button pressing operation or the like.

  In step S1, the imaging control unit 51 controls the imaging unit 17 to perform continuous imaging.

  In step S <b> 2, the image acquisition unit 54 acquires captured image data every time the imaging unit 17 captures an image. At this time, the image acquisition unit 54 temporarily stores the acquired captured image data in the image storage unit 53.

  In step S <b> 3, the feature point detection unit 551 detects a plurality of feature points from the captured image data acquired in the process of step S <b> 2 and temporarily stores them in the image storage unit 53.

  In step S4, the feature point tracking unit 552 determines whether the captured image that has undergone the process in step S3 is the second or subsequent image. If it is the second and subsequent captured images, YES is determined in step S4, and the process proceeds to step S5. If it is not the second and subsequent images, that is, if it is the first captured image, step S5 is performed. If NO is determined in S4, the process returns to step S1.

  In step S5, the feature point tracking unit 552 extracts the feature points in the adjacent captured image (previously captured image) from the image storage unit 53, and then compares the corresponding feature points between the adjacent captured images. Thus, a vector of a plurality of feature points between adjacent captured images is calculated.

  In step S6, the similarity calculation unit 553 calculates the similarity of the vector of the plurality of feature points calculated in the process of step S5. Specifically, the similarity calculation unit 553 calculates the vector similarity in each of a group of areas adjacent in a direction orthogonal to the panorama imaging direction.

  In step S7, the image alignment unit 554 aligns adjacent captured images based on the vector similarity calculated in step S6. For example, the image alignment unit 554 selects a group of regions with the most similar vectors, and performs alignment at the position of the region.

  In step S8, the panoramic image generation unit 56 combines the data of the captured images that have been aligned in the process of step S7, and generates panoramic image data.

  In step S9, the CPU 11 determines whether the panoramic image generation process should be terminated. For example, the CPU 11 determines that the panoramic image generation process should be ended when the digital camera moves more than a predetermined amount or when a predetermined end operation is received by the user. If YES is determined in step S9, the storage control unit 52 stores the panoramic image data generated so far in the image storage unit 53, and ends the panoramic image generation process. If NO is determined in step S9, the process proceeds to step S10.

  In step S10, the CPU 11 or the image processing unit 14 determines whether an error has occurred. For example, the CPU 11 or the image processing unit 14 when the digital camera moves more than a predetermined amount in a direction orthogonal to the panoramic imaging direction (ie, when camera shake is large) or when a sufficient number of feature points cannot be detected from the captured image. Determines that an error has occurred. If YES is determined in step S10, the panorama image generation process is terminated. If NO is determined in step S11, the process proceeds to step S1.

The image processing apparatus 1 configured as described above includes a registration unit 55 that performs registration of adjacent captured images with the captured image data supplied from the imaging unit 17 as a processing target, and a registration unit 55. A panorama image generation unit 56 that generates panorama image data by combining adjacent captured image data based on the alignment result; At this time, the alignment unit 55 positions the adjacent captured images based on the feature point tracking unit 552 that calculates a vector of a plurality of feature points between adjacent captured images and the calculated vector of the plurality of feature points. An image positioning unit 554 for matching.
Thereby, it is possible to align adjacent captured images while avoiding the position of a feature point with an abnormal vector, that is, avoiding the position of a moving object that moves regardless of the movement of the digital camera. As a result, in the panoramic image obtained as a result of the synthesis, it is possible to prevent a part of the moving object from being cut out, and it is possible to generate panoramic image data without a sense of discomfort.
Further, in the image processing apparatus 1, in addition to the feature point vector, the panoramic imaging direction and the position of the moving object in the captured image are taken into consideration, so that the moving object can be displayed in various modes in the panoramic image obtained as a result of the synthesis. it can. That is, it is possible to generate panoramic image data that does not include any moving object, to generate panoramic image data that includes only one moving object, and to see how the moving object continuously moves. It is also possible to generate panoramic image data.

  At this time, in the panorama image generation process, it is common to perform alignment of data of captured images adjacent in a direction substantially orthogonal to the panorama imaging direction. Therefore, in order to specify a position to avoid moving objects, the panorama imaging direction It is preferable to use the similarity of vectors adjacent to each other in a direction substantially orthogonal to the direction. Thereby, adjacent captured images can be aligned in a straight line shape in a direction substantially orthogonal to the panoramic image capturing direction, and there is no need for complicated calculation as in the case of aligning in a wave shape.

  Note that, based on the degree of similarity, the alignment may be performed at the position of a group of areas where the vectors are similar to each other by a predetermined amount or more, and the position of the group of areas where the vectors are most similar. It is good also as performing alignment. When the alignment is performed with a predetermined similarity, the alignment can be arbitrarily performed according to the position of the moving object in the captured image, and the display mode of the moving object can be arbitrarily set. In addition, when the alignment is performed at the most similar group of positions, the data of adjacent captured images can be synthesized most naturally.

  In addition, in digital cameras, it is common that a slight correction is applied to the edge of the captured image due to the distortion of the lens, etc., so the area near the center excluding the edge of the captured image is targeted. It is preferable to identify a group of regions. Accordingly, the vector of feature points can be accurately calculated regardless of the distortion of the lens, and the data of adjacent captured images can be appropriately combined.

  The image processing apparatus 1 of the present embodiment has been described above. However, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within a scope that can achieve the object of the present invention are included in the present invention. It is what

  In the above-described embodiment, in the panorama image generation process, adjacent captured images are aligned and synthesized each time captured image data is acquired. However, all the imagings for generating panoramic image data are performed. After acquiring image data, alignment and composition may be performed together.

In the above-described embodiment, the image processing apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having a panoramic image generation function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the image processing apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. The recording medium etc. provided in The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 20 in FIG.

In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.
Further, in the present specification, the term “system” means an overall apparatus configured by a plurality of devices, a plurality of means, and the like.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Acquisition means for acquiring a plurality of images continuously captured with movement of the imaging unit;
A dividing unit that divides the image acquired by the acquiring unit into a plurality of regions;
Feature point detecting means for detecting feature points in a plurality of regions divided by the dividing means,
Vector calculation means for calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points respectively detected by the feature point detection means;
Area selecting means for selecting a specific area from the plurality of divided areas based on the plurality of vectors calculated by the vector calculating means;
Generating means for generating a synthesized image by synthesizing the adjacent images based on the specific area selected by the area selecting means;
An image processing apparatus comprising:
[Appendix 2]
The dividing unit divides the acquired image into a plurality of continuous regions,
The similarity for calculating the similarity between the calculated vectors for each of a group of regions arranged adjacent to each other in a direction substantially orthogonal to the direction of movement among the plurality of regions divided by the dividing unit. A calculation means;
Further comprising
The image processing apparatus according to appendix 1, wherein the region selection unit selects the group of regions as the specific region based on the similarity calculated by the similarity calculation unit.
[Appendix 3]
The similarity calculation means calculates the similarity of the vector for each of the group of regions in the central portion of the image;
The image processing apparatus according to Supplementary Note 2, wherein
[Appendix 4]
The region selecting means selects a position where the vectors of the group of regions are most similar as a region for aligning the adjacent images.
The image processing apparatus according to appendix 2 or 3, characterized by the above.
[Appendix 5]
An acquisition step of acquiring a plurality of images continuously imaged with movement of the imaging unit;
A dividing step of dividing the image acquired by the acquiring step into a plurality of regions;
A feature point detecting step for detecting feature points in a plurality of regions divided by the dividing step, respectively;
A vector calculation step of calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points detected by the feature point detection step;
A region selection step of selecting a specific region from the plurality of divided regions based on the plurality of vectors calculated by the vector calculation step;
Based on the specific region selected by the region selection step, a generation step for generating the composite image by combining the adjacent images;
An image processing method comprising:
[Appendix 6]
Computer
Acquisition means for acquiring a plurality of images continuously captured with movement of the imaging unit;
A dividing unit that divides the image acquired by the acquiring unit into a plurality of regions;
Feature point detecting means for detecting feature points in a plurality of regions divided by the dividing means,
Vector calculation means for calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points respectively detected by the feature point detection means;
A region selecting unit that selects a specific region from the plurality of divided regions based on the plurality of vectors calculated by the vector calculating unit;
Generating means for generating the synthesized image by synthesizing the adjacent images based on the specific area selected by the area selecting means;
A program characterized by functioning as

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... Bus, 16 ... Input / output interface, 17. ..Imaging section, 18 ... input section, 19 ... output section, 20 ... storage section, 21 ... communication section, 22 ... drive, 31 ... removable media, 51 ... Imaging control unit, 52... Storage control unit, 53... Image storage unit, 54... Image acquisition unit, 55 ... alignment unit, 551 ... feature point detection unit, 552. Point tracking unit, 553... Similarity calculation unit, 554... Image alignment unit, 56.

To achieve the above object, an image processing apparatus of one embodiment of the present invention, an acquisition unit configured to acquire a plurality of images are continuously captured with the moving of the imaging unit shooting, each of the plurality of images, a feature point detection means for detecting a feature point in a region-gun size which is determined based on the predetermined information, based on the feature point, area selecting means for selecting a specific area from the area-gun, of the specific And generating means for generating a composite image by combining adjacent images based on a region.

Claims (6)

Acquisition means for acquiring a plurality of images continuously captured with movement of the imaging unit;
A dividing unit that divides the image acquired by the acquiring unit into a plurality of regions;
Feature point detecting means for detecting feature points in a plurality of regions divided by the dividing means,
Vector calculation means for calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points respectively detected by the feature point detection means;
Area selecting means for selecting a specific area from the plurality of divided areas based on the plurality of vectors calculated by the vector calculating means;
Generating means for generating a synthesized image by synthesizing the adjacent images based on the specific area selected by the area selecting means;
An image processing apparatus comprising: The dividing unit divides the acquired image into a plurality of continuous regions,
The similarity for calculating the similarity between the calculated vectors for each of a group of regions arranged adjacent to each other in a direction substantially orthogonal to the direction of movement among the plurality of regions divided by the dividing unit. A calculation means;
Further comprising
The image processing apparatus according to claim 1, wherein the region selection unit selects the group of regions as the specific region based on the similarity calculated by the similarity calculation unit. The similarity calculation means calculates the similarity of the vector for each of the group of regions in the central portion of the image;
The image processing apparatus according to claim 2. The region selecting means selects a position where the vectors of the group of regions are most similar as a region for aligning the adjacent images.
The image processing apparatus according to claim 2, wherein the image processing apparatus is an image processing apparatus. An acquisition step of acquiring a plurality of images continuously imaged with movement of the imaging unit;
A dividing step of dividing the image acquired by the acquiring step into a plurality of regions;
A feature point detecting step for detecting feature points in a plurality of regions divided by the dividing step, respectively;
A vector calculation step of calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points detected by the feature point detection step;
A region selection step of selecting a specific region from the plurality of divided regions based on the plurality of vectors calculated by the vector calculation step;
Based on the specific region selected by the region selection step, a generation step of generating a composite image by combining the adjacent images;
An image processing method comprising: Computer
Acquisition means for acquiring a plurality of images continuously captured with movement of the imaging unit;
A dividing unit that divides the image acquired by the acquiring unit into a plurality of regions;
Feature point detecting means for detecting feature points in a plurality of regions divided by the dividing means,
Vector calculation means for calculating a plurality of vectors corresponding to the plurality of regions between adjacent images of the plurality of images based on the feature points respectively detected by the feature point detection means;
A region selecting unit that selects a specific region from the plurality of divided regions based on the plurality of vectors calculated by the vector calculating unit;
Generating means for generating a synthesized image by synthesizing the adjacent images based on the specific area selected by the area selecting means;
A program characterized by functioning as

Images