JP2014123916A - Image composition device, image composition method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable reduction of a waste of time for composition processing of images.SOLUTION: An imaging device 1 comprises a captured image selection unit 51, an image composition unit 55, an extraction candidate area determination unit 53, and an extraction area determination unit 54. The extraction candidate area determination unit 53 determines whether a feature amount in a captured image selected by the captured image selection unit 51 is equal to or smaller than a predetermined value. The extraction area determination unit 54 determines an extraction area of an image to be combined by the image composition unit 55 on the basis of the determination result of the extraction candidate area determination unit 53.

Description

  The present invention relates to an image composition apparatus, an image composition method, and a program for generating a panoramic image, for example.

  The following techniques are known as techniques for generating a conventional panoramic image. That is, the user moves the digital camera so as to rotate in the horizontal direction while being substantially fixed in the vertical direction while maintaining the state of pressing the shutter switch. The digital camera captures a plurality of images continuously during the movement, and generates a panoramic image by compositing so that a part of the regions of the images captured at the plurality of times is joined (for example, patent document). 1).

JP-A-6-303562

  However, in the technique of Patent Document 1 described above, for example, when there is no feature in the area between the images to be combined, it is highly possible that the alignment process between the images will fail, and the time for the combining process is wasted. There is a risk of becoming.

  Therefore, the present invention has been made in view of the above situation, and an object of the present invention is to reduce the waste of time for combining images.

In order to achieve the above object, an image composition device according to an aspect of the present invention includes:
Acquisition means for acquiring a plurality of images;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determining unit that determines an extraction region of an image to be combined by the combining unit based on a determination result by the determining unit;
It is characterized by providing.

In order to achieve the above object, an image composition device according to another aspect of the present invention includes:
An imaging control means for controlling the imaging means,
Obtaining means for obtaining an image captured by the imaging means;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determination unit that determines a next imaging range so as to avoid the processing avoidance region in the image based on a determination result by the determination unit;
With
The imaging control unit controls the imaging unit to change the imaging range based on a determination result by the determination unit;
It is characterized by that.

  According to the present invention, it is possible to reduce the waste of time for synthesizing images.

It is a schematic diagram for demonstrating the method of the production | generation of the panorama image in this embodiment. It is a block diagram which shows the structure of the hardware of the imaging device which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional structure for performing an image composition process among the functional structures of the imaging device of FIG. It is a schematic diagram for demonstrating the structure of the production | generation of the panoramic image in this embodiment. It is a schematic diagram for demonstrating the expansion of the extraction candidate area | region in a captured image. 3 is a flowchart illustrating a flow of imaging processing executed by the imaging device of FIG. 2. 4 is a flowchart for explaining a flow of image composition processing executed by the imaging apparatus of FIG. 2 having the functional configuration of FIG. 3.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, a method for generating a panoramic image in the present embodiment will be described.
FIG. 1 is a schematic diagram for explaining a method of generating a panoramic image in the present embodiment.

As shown in FIG. 1A, it is assumed that the imaging device 1 continues to display a so-called live view image on the display unit.
Therefore, the user first moves the imaging device 1 to a position (hereinafter referred to as “start point position”) where the start point area is included in the live view image while viewing the live view image. . Then, at this starting point position, the user performs an instruction operation to start imaging such as pressing the shutter switch.
The imaging device 1 performs the first imaging in response to this instruction operation. That is, the first imaging is performed at a place where the start point region is included in the angle of view, and is performed in response to an instruction operation by the user such as a shutter switch pressing operation.
Thereafter, the user moves the imaging apparatus 1 so as to rotate in the horizontal direction from the starting point position while being substantially fixed in the vertical direction while maintaining a state such as a pressing operation of the shutter switch.
During this time, the imaging apparatus 1 repeatedly performs imaging a plurality of times at a predetermined interval (in this embodiment, an interval at which the movement distance is constant) while moving. The number of times of imaging is determined according to the set size of the panoramic image area. By the final imaging, an image including at least a part of the end point region of the panoramic image is obtained.
In this way, a plurality of images (hereinafter referred to as “original images”) from which a panoramic image is generated can be obtained by performing imaging a plurality of times by the imaging apparatus 1 that is moving.

Then, the imaging apparatus 1 cuts out a part of a plurality of original images and synthesizes them, so that a panoramic image having a wider angle of view than the normal angle of view as shown in FIG. Generate. For example, when a half area of four original images is cut out and combined with each other, a panoramic image having a field angle twice as wide as that of a normal captured image is generated.
In the present embodiment, in consideration of image distortion, not all regions R1 included in the original image are used for generating a panoramic image in the vertical direction of the original image. The nearby region R2 is used for generating a panoramic image.

Here, when the panorama image is generated, with regard to the composition of the original image, for example, when an object including an edge serving as a mark of the combination is not in the region to be combined, a conventional panoramic image generation method (for example, If the method described in Patent Document 1 above is simply applied, it will be difficult to align the images, and image synthesis will fail. As a result, the time required for generating the panoramic image increases due to the failure of image synthesis.
Therefore, the imaging apparatus 1 according to the present embodiment reduces the possibility of failure even if the image alignment regions do not have a feature even if the image alignment processing is performed, and the time required for the combining process is wasted. It has a function to prevent it.

Next, the hardware configuration of the imaging apparatus 1 having the functions described above will be described.
FIG. 2 is a block diagram illustrating a hardware configuration of the imaging apparatus 1 according to an embodiment of the present invention.
The imaging device 1 is configured as a digital camera, for example.

  The imaging device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an imaging unit. 17, a movement detection unit 18, an input unit 19, an output unit 20, a storage unit 21, a communication unit 22, and a drive 23.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 21 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.

The CPU 11, ROM 12, RAM 13, and image processing unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, a movement detection unit 18, an input unit 19, an output unit 20, a storage unit 21, a communication unit 22, and a drive 23 are connected to the input / output interface 16.
Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

The optical lens unit is configured with a lens that collects light, such as a focus lens and a zoom lens, in order to capture an image of the subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17. The output signal of the imaging unit 17 is hereinafter referred to as “captured image data”. The captured image data is used as original image data used for generating a panoramic image.

The movement detection unit 18 is composed of, for example, a gyro sensor, and detects the speed and angular acceleration of the imaging device 1.
Here, as described above, the imaging apparatus 1 repeatedly performs imaging a plurality of times at a predetermined interval (in this embodiment, an interval at which the movement distance is constant) while moving. The detection result of the movement detector 18 is used for the calculation of the predetermined interval. That is, the CPU 11 calculates the movement amount of the imaging device 1 based on the detection result (speed or angular acceleration) of the movement detection unit 18 and performs multiple imaging (hereinafter referred to as “hereafter”) at a predetermined interval corresponding to the movement amount. “Panorama imaging” is repeated.

The input unit 19 is configured with various buttons and the like, and inputs various types of information according to user instruction operations.
The output unit 20 includes a display, a speaker, and the like, and outputs an image and sound.

The storage unit 21 is configured with a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 22 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 23. The program read from the removable medium 31 by the drive 23 is installed in the storage unit 21 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 21 in the same manner as the storage unit 21.

Heretofore, the hardware configuration of the imaging apparatus 1 according to an embodiment of the present invention has been described.
In the present embodiment, panorama image generation processing for generating a panorama image is performed. This panoramic image generation process includes an imaging process for acquiring a captured image for generating a panoramic image and an image composition process for combining the acquired captured images.
In view of this, a functional configuration for executing the image composition process among the panorama image generation processes will be described.

FIG. 3 is a functional block diagram showing a functional configuration for executing the image composition process among the functional configurations of the imaging apparatus 1 of FIG.
In detail, the “image composition processing” is processing performed after the imaging apparatus 1 performs panoramic image capture and generates original image data as a plurality of captured images, and is used for compositing each original image. This is a series of processes from adjusting the width (extraction width) of the region to be extracted to combining the data of a plurality of original images in the order of imaging, and as a result, generating panoramic image data.

  When the image composition process is executed, the image processing unit 14, as shown in FIG. 3, the captured image selection unit 51, the feature amount calculation unit 52, the extraction candidate region determination unit 53, and the extraction region determination unit. 54, the image composition unit 55, and the storage control unit 56 function.

A captured image storage unit 71 is provided as an area of the storage unit 21.
In the captured image storage unit 71, data of a plurality of captured images obtained by panoramic image capturing are stored in association with metadata indicating the capturing time and the like.

The captured image selection unit 51 sequentially selects original image data to be combined among a plurality of captured image data stored in the captured image storage unit 71 in the order of imaging. That is, the captured image selection unit 51 selects the captured image data as the original image data in the order of the imaging order that is the synthesis order. In this embodiment, as soon as the synthesis of the data of one original image is completed, the next captured image becomes the original image, and the data of the original image is synthesized. For this reason, the captured image selection unit 51 selects the next captured image data as the original image data after the process of combining the selected original image data ends.
In addition, when it is determined that the extraction width is to be expanded, the captured image selection unit 51 cancels and skips the captured image to be selected according to the expansion width, and selects the next captured image.

  The feature amount calculation unit 52 calculates the feature amount of a partial region for image synthesis (hereinafter referred to as “extraction candidate region”) from the original image data selected by the captured image selection unit 51. Here, the width of the region in which the feature amount is calculated is set to a predetermined width (hereinafter referred to as “standard width”).

  More specifically, the “feature amount” is for quantitatively determining the presence or absence of a subject whose edge can be detected in the extraction candidate region. The higher the feature value, the higher the probability of existence of a subject capable of detecting an edge in the extraction candidate area for which the feature value has been calculated. In the present embodiment, the feature amount calculation unit 52 calculates the variance of the luminance values of the pixels constituting the entire extraction determination as the feature amount.

  The extraction candidate region determination unit 53 compares the feature amount calculated by the feature amount calculation unit 52 with a predetermined threshold value, thereby determining a portion to be determined in the extraction candidate region (in this embodiment, the entire extraction candidate region ) And the like.

  Specifically, for example, when the feature amount is equal to or less than a predetermined threshold, the extraction candidate region determination unit 53 includes a sky or the like in the region to be determined, and the region has no edge (hereinafter, “ Non-textured region ”). In addition, when the feature amount exceeds a predetermined threshold, the extraction candidate region determination unit 53 includes a building or the like in the determination target region, and the region is referred to as an edged region (hereinafter referred to as “texture region”). ).

  The extraction region determination unit 54 determines the width of the extraction region based on the determination result of the extraction candidate region determination unit 53. More specifically, the extraction area determination unit 54 uses, as the extraction area, an area having the entire width (hereinafter referred to as “extraction width”) determined to have no edge based on the determination result of the extraction candidate area determination unit 53. decide.

  Specifically, when the extraction candidate region determination unit 53 determines that the texture region is the texture region in the first determination, the extraction region determination unit 54 determines the standard width as the width of the extraction region. If the extraction candidate region determination unit 53 determines that the region corresponding to the standard width is a non-texture region in the first determination, the region for determining the edge by the standard width is shifted for the second time. When the subsequent determination is performed and the extraction candidate region determination unit 53 determines that the region determined after the second time is a non-texture region, the extraction region determination unit 54 An area having a width (hereinafter referred to as “extended width”) is determined as an extraction area.

  The image compositing unit 55 is a process of cutting out part of the original image data to be combined in the extraction region determined by the extraction region determining unit 54 and combining it with the extracted region of the image data selected so far Is repeatedly executed for each of the selected plurality of original images, thereby generating panoramic image data.

  Here, when the extraction width of the extraction region determined by the extraction region determination unit 54 is an extended width, the image composition unit 55 uses the standard width of the captured image acquired next as an extraction region candidate. This data is skipped and excluded from composition. That is, the image compositing unit 55 does not adopt, as a compositing target, a captured image having a standard width that overlaps with the previously determined expansion width among captured images newly acquired for composition. As described above, in the present embodiment, among the extracted image of the extended width and the newly acquired captured image, the captured image whose standard width region overlaps the previously determined extended width is excluded from the synthesis target. Since the image synthesis process for the newly acquired captured image is eliminated, the time required for generating the panorama image can be shortened and the frequency of composition failure can be reduced.

  The storage control unit 56 controls the removable media 31 so as to store panoramic image data generated by the image composition by the image composition unit 55.

Next, a mechanism for generating a panoramic image in the present embodiment will be described.
FIG. 4 is a schematic diagram for explaining a mechanism for generating a panoramic image in the present embodiment. In the example of FIG. 4, a case where a panoramic image is captured of the same landscape as in FIG. 1A will be described as an example. That is, in the example of FIG. 4, a case where a panoramic image is generated in the present embodiment will be described for the case of FIG. 1A in which the “parking lot” is photographed against the background of “tower” and “facility”.
In addition, when it is not necessary to distinguish each of the captured images p1 to p5, these are collectively referred to as “captured image p”. Similarly, when there is no need to individually distinguish each of the extraction regions CR1 to CR3, these are collectively referred to as an “extraction region CR”.

In the present embodiment, as shown in FIG. 4, the panorama image is generated by cutting out a part of the captured images in the order of synthesis and combining them one after another to finally generate one panoramic image. To do.
In FIG. 4, captured images are shown in the order of imaging in order from the left in the figure. That is, in the captured images p1 to p5..., The imaging order of the leftmost captured image p1 is the earliest, and the imaging order becomes slower as going to the right. Among them, the imaging order is the slowest.

  In the generation of the panoramic image, specifically, when determining the extraction region CR1, the region corresponding to the standard width in the central portion of the captured image p1 is set as the extraction candidate region SHR1, and the characteristics of the entire extraction candidate region SHR1 are characterized. The amount is calculated, and it is determined whether or not the width of the extraction region CR1 is expanded from the value of the feature amount.

  Next, in the generation of the panoramic image, the extraction region CR1 is determined by determining the extraction candidate region. In the captured image p1, the feature amount exceeds a predetermined threshold, as can be seen from the fact that the “facility” is shown below in the extraction candidate region SHR1, and the extraction width of the extraction region CR1 becomes the standard width. It is determined.

  Next, in the generation of the panoramic image, when determining the extraction region CR2, the feature amount of the entire extraction candidate region SHR2-1 in the center portion of the captured image p2 is calculated, and the extraction region CR2 is extracted from the feature amount value. It is determined whether or not the candidate area SHR2-1 is expanded.

FIG. 5 is a schematic diagram for explaining the extension of the extraction candidate region in the captured image p2.
In the generation of the panoramic image, the determination is performed by calculating the feature amount of the extraction candidate region SHR2-1 corresponding to the standard width in the captured image p2.

  As shown in FIG. 5, in the captured image p2, as can be seen from the fact that “sky” occupies most, the feature amount of the extraction candidate region SHR2-1 is equal to or less than a predetermined threshold. Therefore, in the captured image p2, in order to expand the extraction region CR2, the feature amount of the extraction candidate region SHR2-2 adjacent to the extraction candidate region SHR2-1 is also calculated, and the extraction candidate region SHR2-2 is also included in the extraction region CR2. Judgment whether or not should be done. As can be seen from the fact that “sky” occupies most, the feature amount of the extraction candidate region SHR2-2 is equal to or less than a predetermined threshold value, and therefore the extraction candidate region SHR2-1 and the extraction candidate are included in the extraction region CR2. Region SHR2-2 will be included. In the captured image p2, such extension is repeatedly performed until the extraction candidate area exceeds a predetermined threshold, and the extraction area is expanded until it is determined that the feature amount of the extraction candidate area exceeds the predetermined threshold. That is, in the captured image p2, “another facility” appears in the extraction candidate region SHR2-3 that is expanded by two standard widths, and finally, the extraction candidate region SHR2-1 and the extraction candidate region SHR2- The region up to 3 is defined as an extraction region CR2. That is, the extraction region CR2 is a total of three extraction candidate regions SHR2-1 corresponding to the standard width for which the first determination has been finally made, and extraction candidate regions SHR2-2 and SHR2-3 corresponding to the two standard widths. This is an extraction area having a standard width of one.

  At this time, in the generation of the panoramic image, the extraction region CR2 is an extraction region having a standard width corresponding to three, so that the captured image p3 including the region corresponding to the expanded width as the standard width as shown in FIG. And it does not employ | adopt as an image which synthesize | combines the captured image p4.

  Next, in the generation of the panoramic image, the feature amount of the extraction candidate area corresponding to the standard width in the captured image p5 is calculated. As can be seen from the fact that “another facility” occupies most of the captured image p5, the feature amount exceeds the predetermined threshold, and the extraction region having the standard width is the extraction region CR3 in the captured image p5. As determined.

  Thereafter, in the generation of the panoramic image, an extraction region in the captured image sequentially acquired is determined.

  In the generation of a panoramic image, after completion of determination of extraction candidate regions and determination of extraction widths in all captured images, the images are combined. As a result, a panoramic image is generated.

The mechanism of image synthesis in the present embodiment has been described above.
Next, a flow of panorama image generation processing executed by the imaging device 1 will be described.
As described above, the panoramic image generation process includes the imaging process and the image composition process.
Such a panoramic image generation process is started by an operation for starting the imaging process, and ends when the image synthesis process to be started ends after the imaging process ends.

First, the flow of the imaging process in the panoramic image generation process will be described.
FIG. 6 is a flowchart for explaining the flow of the imaging process executed by the imaging apparatus 1 of FIG.
The imaging process is started by an operation for starting the imaging process on the input unit 19 by the user.
After the start of the imaging process, the user moves the imaging device 1 so that the images are connected in one direction. Thereby, imaging is performed when moving a predetermined distance.

In step S <b> 1, the CPU 11 determines whether the imaging device 1 has moved a certain distance based on the detection result of the movement detection unit 18.
If it has not moved for a certain distance, it is determined as NO in step S1, and it enters a standby state until it moves for a certain distance.
On the other hand, if the robot has moved a certain distance, YES is determined in step S1, and the process proceeds to step S2.

  In step S <b> 2, the CPU 11 causes the imaging unit 17 to perform imaging because the imaging device 1 has moved a certain distance, and as a result, the acquired captured image is stored in the captured image storage unit 71 of the storage unit 21 and stored. At this time, the acquired captured image is stored so that information on the imaging order is retained.

In step S3, the CPU 11 determines whether or not an end signal is generated by an operation by the user. For example, when the user generates an end signal for generating a panoramic image from a current captured image, or when an end signal for canceling panorama image generation is generated, the imaging processing ends.
If an end signal has been generated, YES is determined in step S3, and the imaging process ends.
On the other hand, if no end signal is generated, NO is determined in step S3, and the process proceeds to step S4.

In step S4, the CPU 11 determines whether or not an image acquisition error has been detected.
If an image acquisition error has occurred, YES is determined in step S4, and the imaging process ends.
On the other hand, if no image acquisition error has occurred, NO is determined in step S4, and the process proceeds to step S5.

In step S5, the CPU 11 determines whether or not the moving distance in the image acquisition direction has exceeded a threshold value.
If the moving distance in the image acquisition direction does not exceed the threshold value, NO is determined in step S5, the process returns to step S1, and the processes after step S1 are executed again.
On the other hand, when the moving distance in the image acquisition direction exceeds the threshold value, since the necessary number of captured images has been acquired, YES is determined in step S5, and the imaging process is terminated.

Next, the flow of the image composition process in the panorama image generation process will be described.
FIG. 7 is a flowchart for explaining the flow of image composition processing executed by the imaging apparatus 1 of FIG. 2 having the functional configuration of FIG.

  In step S <b> 31, the captured image selection unit 51 selects a captured image to be synthesized from the captured images stored in the captured image storage unit 71 in the order of imaging order.

  In step S32, the feature amount calculation unit 52 calculates the feature amount by calculating the variance of the luminance values of the pixels in the extraction candidate region in the selected captured image.

In step S <b> 33, the extraction candidate area determination unit 53 determines whether or not the variance value calculated as the feature amount is equal to or less than a predetermined threshold value.
If the variance value exceeds the predetermined threshold value, NO is determined in step S33, and the process proceeds to step S34.

  In step S34, the extraction area determination unit 54 determines an area extraction area (normal extraction area) having a standard width as a synthesis width.

In step S35, the image composition unit 55 performs image composition in an extraction region having a standard width (normal extraction region). That is, since it is determined in step S34 that the extraction width is the standard width, the image composition unit 55 uses the extraction region CR1 corresponding to the standard width from the captured image P1 as shown in the example of FIG. To combine the images.
Thereafter, the process proceeds to step S38. The processing after step S38 will be described later.

On the other hand, if the variance value is less than or equal to the predetermined threshold, YES is determined in step S33, and the process proceeds to step S36.
In step S36, the extraction area determination unit 54 extends the extraction width, and determines an area having the expanded width as an extraction area used for image synthesis. Note that the extraction region determination unit 54 sequentially determines new extraction candidate regions until an extraction candidate region whose feature amount is determined to exceed a predetermined threshold is found, and determines an extraction region for image synthesis.

  In step S <b> 37, the image composition unit 55 performs image composition in an extraction region having an extended width (extended extraction region). That is, the image composition unit 55 expands the extraction width in step S36, and since the region having the expanded extraction width (expansion width) is determined as the extraction region, as shown in the example of FIG. An extraction region CR2 having an extended width is extracted from the captured image p2, and the images are combined. Further, in the next composition, the image composition unit 55 considers the expansion width of the extraction region CR2 and selects the image areas corresponding to the expansion width so as not to be extracted. The image to be skipped is skipped, and the next image skipped is selected as a synthesis target. Specifically, as in the example of FIG. 4, when two standard widths are expanded, the image area for two images is not extracted and canceled as a selection target for image synthesis, and the cancel The next image (third captured image p3) of the obtained image is extracted. In this way, by not extracting the captured image area including the area corresponding to the extended width, wasteful synthesis processing does not occur, so that it is possible to reduce the processing load related to synthesis and the chance of composition failure.

In step S38, the image processing unit 14 determines whether the composition has failed.
If the composition has failed, YES is determined in step S37, and the process proceeds to step S42. The process of step S42 will be described later.

  If the composition has not failed, NO is determined in step S38, and the process proceeds to step S39.

In step S39, the image processing unit 14 determines whether or not the composite number has reached the acquired number.
If the composite number has not reached the acquired number, NO is determined in step S39, and the process proceeds to step S40.

In step S40, the image processing unit 14 determines whether or not the extraction area has been expanded.
If the extraction area is not expanded, NO is determined in step S40, the process returns to step S31, and the processes after step S31 are executed again.

  On the other hand, if the extraction area is widened, YES is determined in step S40, and the process proceeds to step S41.

In step S41, the captured image selection unit 51 selects an acquired image by skipping an image corresponding to the expanded width of the extraction region as a selection target. That is, the captured image selection unit 51 determines how many images the extraction area is to be extracted, skips not to select the determined number of images, and acquires the next image as an acquired image. Select as. Specifically, as in the example of FIG. 4, when two standard widths are expanded, two images (captured images p3 and p4) are canceled without being selected, and the next image after the canceled image. Image (third captured image p5) is selected.
Thereafter, the process returns to step S32, and the processes after step S32 are executed again.

  On the other hand, if the composite number has reached the acquired number, YES is determined in step S39, and the process proceeds to step S42.

  On the other hand, if the composition fails, YES is determined in step S38, and the process proceeds to step S42.

In step S42, the image composition unit 55 trims a portion necessary for the panoramic image (in the present embodiment, the region of the lower portion of the image). Thereafter, the storage control unit 56 stores the panoramic image in the removable medium 31.
Thereby, the image composition process is completed.

The imaging apparatus 1 configured as described above includes a captured image selection unit 51, an image composition unit 55, an extraction candidate region determination unit 53, and an extraction region determination unit 54.
The extraction candidate area determination unit 53 determines whether or not the feature amount in the captured image selected by the captured image selection unit 51 is equal to or less than a predetermined value.
The extraction area determination unit 54 determines an extraction area of an image to be synthesized by the image synthesis unit 55 based on the determination result by the extraction candidate area determination unit 53.
Thereby, in the imaging device 1, when the feature amount in the captured image is equal to or less than the predetermined value, it is possible to change the synthesis area, and so on, and thus it is possible to reduce waste of time for the synthesis process between the images. it can.

The extraction candidate area determination unit 53 determines whether or not the feature amount of the partial area in the acquired image is equal to or less than a predetermined threshold value.
Thereby, in the imaging device 1, when the feature amount in the captured image is equal to or less than the predetermined value, it is possible to change the synthesis area, and thus it is possible to reduce the waste of time for the synthesis process between the images. .

When it is determined that the feature amount is equal to or smaller than the predetermined threshold as a determination result, the extraction candidate region determination unit 53 repeatedly determines whether or not the feature amount in the region shifted by a predetermined width from the predetermined partial region is equal to or smaller than the predetermined threshold. .
The extraction area determination unit 54 integrates the areas determined by the extraction candidate area determination unit 53 to have a feature amount equal to or less than a predetermined threshold value, and determines as an extraction area.
Thereby, in the imaging device 1, since a plurality of areas are integrated into an extraction area, it is not necessary to perform synthesis in the area, and waste of time for synthesizing the images can be reduced.

When the extraction candidate region determination unit 53 determines that the feature amount is equal to or greater than a predetermined threshold, the extraction region determination unit 54 determines a predetermined part of the region as the extraction region.
Thereby, in the imaging device 1, since it is more than a threshold value and appropriate composition can be performed, composition processing of images can be surely performed.

The captured image selection unit 51 selects a plurality of captured images at intervals according to the extraction region determined by the extraction region determination unit 54.
The image synthesis unit 55 synthesizes a plurality of images acquired at intervals according to the extraction area.
Thereby, in the imaging device 1, in order to synthesize the images acquired at intervals corresponding to the extraction regions determined by the extraction region determination unit 54, an unnecessary combining operation does not occur, and the images are combined. Time waste can be reduced.

In addition, the imaging apparatus 1 includes a feature amount calculation unit 52.
The feature quantity calculation unit 52 calculates the variance of the feature points in the overlapping area between the captured images as the feature quantity.
The extraction candidate region determination unit 53 determines whether or not the feature amount calculated by the feature amount calculation unit 52 is equal to or less than a predetermined value.
Therefore, in the imaging device 1, when the feature amount in the captured image is equal to or smaller than the predetermined value, it is possible to change the synthesis region, and thus it is possible to reduce waste of time for synthesizing the images. .

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

  In the above-described embodiment, the panoramic image is generated by using the captured images that are continuously captured by moving a predetermined imaging region by one imaging device, but the present invention is not limited thereto. The panoramic image may include a plurality of captured images in a predetermined imaging range, and the captured images including adjacent imaging ranges may include the same imaging region. For example, an area where adjacent imaging regions overlap may be included. You may generate using the picked-up image imaged from the several imaging device arrange | positioned so that it may have, and the imaging device provided with the several imaging means.

  In the above-described embodiment, the determination is made based on the image sequence.

  The feature amount is configured by determining the variance constituting the region, but is not limited thereto. The feature amount may be determined based on, for example, frequency, high frequency, horizontal and vertical edge frequencies, edge absolute value histogram, and the like, as long as it can determine variations in pixels constituting the region.

  In the above-described embodiment, the extraction area is expanded by calculating the feature amount of the entire area (all pixels) of the extraction area having the standard width and determining the presence or absence of an edge. Not limited. For example, for a predetermined pixel column in an image captured in the panorama shooting direction (for example, when panoramic imaging is performed to move the imaging device 1 in the right direction, two pixel columns from the right end in the first extraction candidate region) ) Area, and the presence or absence of an edge of the area of the predetermined pixel column for which the characteristic quantity is calculated. If it is determined that there is no edge, the characteristic amount is calculated for the area shifted by the predetermined pixel line. The amount of expansion of the extraction region may be determined variably by determining the presence / absence of calculation / edge.

  In the above-described embodiment, the feature amount is calculated for all pixel rows in the extraction candidate region SHR, but the present invention is not limited to this. In the present embodiment, as shown in FIG. 1B, not all the regions R1 included in the original image are used for generating the panoramic image, but the region R2 near the center of the original image is panoramic. Used for image generation. For this reason, the feature amount may be calculated only from the region R2 that is also used for generating a panoramic image. By being configured in this way, for example, the edge is concentrated on a portion that is not used for generating a panoramic image, and the feature amount is increased, so that it is affected by a portion that is not used for generating a panoramic image. Judgment can be made. In the example of FIG. 4, in the case of the extraction area CR1, the area where “sky” other than the “building” below is an area used for generating a panoramic image, and thus the calculated feature amount is panorama. It is not affected by the area where the “building” that is not used for image generation is shown.

Further, in the above-described embodiment, the width of the extraction region is changed in consideration of the flatness, but is not limited thereto. For example, the extraction area may be narrowed for the subject of interest, and the extraction area may be widened for the subject not of interest. Further, the extraction area may be narrowed for a subject of interest, and the extraction area may be narrowed for a subject that is not of interest.
Further, since the importance of the subject increases as it approaches the end from the time when the imaging starts, the extraction area may be widened for those with a fast imaging order, and the extraction area may be narrowed as the imaging order goes backward. . In addition, when a complicated subject unsuitable for composition is shown, the extraction area may be widened and the subject portion may be withdrawn.
Further, in the above-described embodiment, the extraction region is variably determined with respect to substantially one direction for panorama synthesis, but is not limited thereto.
That is, for example, in the synthesis of a wide-angle image by imaging in two directions such as the vertical and horizontal directions, the extraction region can be determined variably in the two directions such as the vertical and horizontal directions.
In addition, when the imaging direction is three or more directions such as diagonally and vertically, the direction in which the extraction area is expanded can be determined according to the number of imaging directions.

  In the above-described embodiment, the image is determined at the time of extracting the synthesis area at the time of image synthesis. However, the present invention is not limited to this, and the determination may be made at the time of acquiring the captured image.

  Further, in the above-described embodiment, the imaging apparatus is assumed as an apparatus including the imaging unit itself. However, the present invention is not limited to this, and only the synthesis process is performed by the own apparatus using the data of the captured image acquired from another imaging apparatus. It may be configured. In this case, it is assumed that the acquired captured image is a captured image in which a predetermined imaging range includes a plurality of captured images, and captured images including adjacent imaging ranges include the same imaging region. is there.

  In the above-described embodiment, an example in which the imaging range (imaging field angle) is fixed for each imaging is shown, but the present invention is not limited to this. In the case of using an imaging apparatus equipped with an imaging unit that varies the imaging range (imaging angle of view) for each imaging, for example, a feature amount is determined from the acquired captured image, and the imaging range (imaging) is determined at the next imaging. The imaging means may be engineered to widen the angle of view).

Further, in the above-described embodiment, the change in the extraction width is, for example, a target (non-texture target) that increases the possibility of a failure in pasting in a pasting region (region where processing is to be avoided) in which image composition processing is performed. The object is to expand and change the extraction area while avoiding the area to be processed when there is a target such as a subject of interest.
Also, from the viewpoint of a region where processing is to be avoided, the acquired captured image is determined, for example, a target (non-textured target) that increases the possibility of failure of pasting to the end of the captured image, a subject of interest, etc. If the target is present, changing the next imaging range is also incorporated as the same idea.

In the above-described embodiment, the imaging apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having an image composition function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a smartphone, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 3 is merely an example, and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 2 in which the program is recorded, the hard disk included in the storage unit 21 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Acquisition means for acquiring a plurality of images;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determining unit that determines an extraction region of an image to be combined by the combining unit based on a determination result by the determining unit;
An image composition apparatus comprising:
[Appendix 2]
The image synthesizing apparatus according to claim 1, wherein the determination unit determines whether or not a feature amount of a partial region in the acquired image is equal to or less than a predetermined threshold value.
[Appendix 3]
When the determination unit determines that the feature amount is equal to or less than a predetermined threshold as the determination result, the determination unit repeatedly determines whether the feature amount in a region shifted by a predetermined width from the predetermined partial region is equal to or less than a predetermined threshold. ,
The image synthesizing apparatus according to appendix 2, wherein the determination unit integrates and determines the extraction region as a region where the determination unit determines that the feature amount is equal to or less than a predetermined threshold.
[Appendix 4]
The determination unit according to claim 2 or 3, wherein, when the determination unit determines that the feature amount is equal to or greater than a predetermined threshold, the determination unit determines the predetermined part of the region as the extraction region. Image composition device.
[Appendix 5]
The acquisition unit acquires the plurality of images at an interval according to the extraction region determined by the determination unit,
The image synthesizing device according to any one of appendices 1 to 4, wherein the synthesizing unit synthesizes the plurality of images acquired at intervals according to the extraction region.
[Appendix 6]
A calculating means for calculating a variance of the feature points of the image acquired by the acquiring means as the feature amount;
The determination unit determines whether or not the variance calculated by the calculation unit is equal to or less than a predetermined threshold;
The image synthesizing device according to any one of appendices 1 to 5, characterized in that:
[Appendix 7]
An imaging control means for controlling the imaging means,
Obtaining means for obtaining an image captured by the imaging means;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determination unit that determines a next imaging range so as to avoid a processing avoidance region in the image based on a determination result by the determination unit;
With
The imaging control unit controls the imaging unit to change the imaging range based on a determination result by the determination unit;
An image composition device characterized by the above.
[Appendix 8]
An image composition method using an image composition device,
An acquisition step of acquiring a plurality of images;
A combining step of combining a plurality of images acquired by the acquiring step;
A determination step of determining whether or not a feature amount in the image acquired by the acquisition step is a predetermined value or less;
A determination step for determining an extraction region of an image to be synthesized by the synthesis step based on a determination result by the determination step;
An image composition method comprising:
[Appendix 9]
Computer
An acquisition means for acquiring a plurality of images;
Combining means for combining a plurality of images acquired by the acquiring means;
Determining means for determining whether or not a feature amount in the image acquired by the acquiring means is equal to or less than a predetermined value;
A determining unit that determines an extraction region of an image to be combined by the combining unit based on a determination result by the determining unit;
A program characterized by functioning as

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... Bus, 16 ... Input / output interface, 17 ... Image pick-up unit, 18 ... movement detection unit, 19 ... input unit, 20 ... output unit, 21 ... storage unit, 22 ... communication unit, 23 ... drive, 31 ... Removable media, 51 ... picked-up image selection unit, 52 ... feature quantity calculation unit, 53 ... extraction candidate region determination unit, 54 ... region determination unit, 55 ... image composition unit, 56 ... .Storage control unit, 71 ... Captured image storage unit

Claims (9)

Acquisition means for acquiring a plurality of images;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determining unit that determines an extraction region of an image to be combined by the combining unit based on a determination result by the determining unit;
An image composition apparatus comprising:   The image synthesizing apparatus according to claim 1, wherein the determination unit determines whether or not a feature amount of a partial region in the acquired image is equal to or less than a predetermined threshold value. When the determination unit determines that the feature amount is equal to or less than a predetermined threshold as the determination result, the determination unit repeatedly determines whether the feature amount in a region shifted by a predetermined width from the predetermined partial region is equal to or less than a predetermined threshold. ,
The image synthesizing apparatus according to claim 2, wherein the determining unit integrates and determines the extraction region as a region in which the feature amount is determined to be equal to or less than a predetermined threshold.   The said determination means determines the said predetermined part area | region as the said extraction area | region, when it determines with the feature-value being more than a predetermined threshold value by the said determination means. Image synthesizer. The acquisition unit acquires the plurality of images at an interval according to the extraction region determined by the determination unit,
The image synthesizing apparatus according to any one of claims 1 to 4, wherein the synthesizing unit synthesizes the plurality of images acquired at intervals corresponding to the extraction regions. A calculating means for calculating a variance of the feature points of the image acquired by the acquiring means as the feature amount;
The determination unit determines whether or not the variance calculated by the calculation unit is equal to or less than a predetermined threshold;
The image synthesizing device according to claim 1, wherein the image synthesizing device is an image synthesizing device. An imaging control means for controlling the imaging means,
Obtaining means for obtaining an image captured by the imaging means;
Combining means for combining a plurality of images acquired by the acquiring means;
Determination means for determining whether or not a feature amount in the image acquired by the acquisition means is equal to or less than a predetermined value;
A determination unit that determines a next imaging range so as to avoid a processing avoidance region in the image based on a determination result by the determination unit;
With
The imaging control unit controls the imaging unit to change the imaging range based on a determination result by the determination unit;
An image composition device characterized by the above. An image composition method using an image composition device,
An acquisition step of acquiring a plurality of images;
A combining step of combining a plurality of images acquired by the acquiring step;
A determination step of determining whether or not a feature amount in the image acquired by the acquisition step is a predetermined value or less;
A determination step for determining an extraction region of an image to be synthesized by the synthesis step based on a determination result by the determination step;
An image composition method comprising: Computer
An acquisition means for acquiring a plurality of images;
Combining means for combining a plurality of images acquired by the acquiring means;
Determining means for determining whether or not a feature amount in the image acquired by the acquiring means is equal to or less than a predetermined value;
A determining unit that determines an extraction region of an image to be combined by the combining unit based on a determination result by the determining unit;
A program characterized by functioning as

Images