JP2012150826A - Image composition device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent waste of processing time, and to suppress the deterioration of composition precision or image quality.SOLUTION: An imaging apparatus 100 is provided with a CPU 71 for performing image composition processing for combining any one reference image among a plurality of images with an object image, processing of specifying shooting condition relevant information related to shooting conditions of the image, and processing of setting contents of the image composition processing on the basis of specified shooting conditions.

Description

  The present invention relates to an image composition apparatus and a program for synthesizing a plurality of images.

  In recent years, feature points are extracted and tracked from a plurality of images photographed in succession, a projection transformation matrix H is obtained from the amount of movement, and alignment is performed using the projection transformation matrix H to perform superposition. A method for creating a single composite image has been developed (see, for example, Patent Document 1).

Specifically, first, feature points are extracted from a composite reference image among a plurality of images photographed continuously, and these are tracked with other images. Here, tracking of feature points is performed using an iterative method in order from low resolution images after sub-sampling a continuous image a predetermined number of times to create a low resolution hierarchy (pyramidization).
When the tracking of feature points is completed, RANSAC is performed. Four sets are randomly selected from the correspondence between a plurality of feature points between images, and an H matrix used for projective transformation is calculated. With respect to the calculated H matrix, the number of feature points to be supported is used as the score of the H matrix, and the above operation is repeated until an H matrix having a certain score or more is obtained.
Then, projective transformation is performed using the H matrix finally obtained by RANSAC to synthesize two images. These steps are repeated for a predetermined number of times to add and synthesize all the images.
As a result, it is possible to create an image with good S / N in which noise that is randomly generated is averaged compared to a normal single-shot image.

JP 2005-94614 A

  However, the above-described image composition is performed under uniquely set conditions without taking into consideration the brightness of the environment in which the image is being taken, the amount of camera shake, the feature amount of the captured image, the shooting situation of the subject, and the like. It was. For this reason, there has been a problem that image composition processing suitable for the shooting situation cannot be performed, processing time is wasted, and composition accuracy and image quality are degraded.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image composition device and a program that can prevent waste of processing time and can suppress deterioration in composition accuracy and image quality.

The image synthesizing device of the invention according to claim 1
Image combining means for performing image combining processing for combining any one of the plurality of images and the target image;
Detecting means for detecting the brightness at the time of shooting the image as any one of the previously divided brightness;
Based on the detection result by the detection means, composition processing setting means for setting the content of the image composition processing by the image composition means;
With
The image composition means includes
Including feature point extraction means for extracting feature points from the reference image;
When the combination processing setting means detects the brightness that is classified as dark as the detection result by the detection means, the number of feature points extracted by the feature point extraction means is a larger number,
The image composition means includes
Including image creating means for creating a low-resolution image by gradually reducing the resolution of the plurality of images,
The composition processing setting means sets the number of steps related to the reduction of the resolution by the image creating means to a larger number when the detection result by the detection means is detected by the brightness that is classified as dark. It is said.

The program of the invention described in claim 2 is:
On the computer,
A combining function for performing image combining processing for combining any one of the plurality of images and the target image;
A detection function for detecting the brightness at the time of shooting the image as any one of the pre-divided brightness levels;
A setting function for setting the content of the image composition processing based on the detection result by the detection function;
Is a program for realizing
The synthesis function is
An extraction function for extracting feature points from the reference image;
When the setting function detects a brightness classified as dark as a detection result by the detection function, the number of feature points extracted by the extraction function is set to a larger number,
The synthesis function is
A creation function for creating a low-resolution image by gradually reducing the resolution of the plurality of images;
The setting function is characterized in that when the detection result by the detection function is detected at a brightness that is classified as dark, the number of steps related to the resolution reduction by the creation function is increased. .

  According to the present invention, wasteful processing is not performed according to shooting conditions, that is, processing time can be prevented from being wasted and processing speed can be improved by performing necessary processing according to shooting conditions. Therefore, it is possible to suppress a decrease in synthesis accuracy and image quality.

It is a block diagram which shows schematic structure of the imaging device of Embodiment 1 to which this invention is applied. It is the figure which showed typically the video signal processing part with which the imaging device of FIG. 1 is equipped. It is a figure which shows typically the correspondence of the processing content and brightness which concern on the image processing by the imaging device of FIG. It is a figure which shows typically the correspondence of the process content and brightness which concern on the image composition process by the imaging device of FIG. 3 is a flowchart illustrating an example of an operation related to an imaging process by the imaging apparatus of FIG. 1. It is a block diagram which shows schematic structure of the imaging device of Embodiment 2 to which this invention is applied. It is a figure which shows typically the correspondence of the processing content and the deviation | shift amount which concern on the image composition process by the imaging device of FIG. FIG. 7 is a diagram schematically illustrating a correspondence relationship between processing content and a shift amount related to image processing by the imaging apparatus of FIG. 6. 7 is a flowchart illustrating an example of an operation related to an imaging process by the imaging apparatus of FIG. 6. It is a block diagram which shows schematic structure of the imaging device of Embodiment 3 to which this invention is applied. It is a figure which shows typically the correspondence of the process content and feature-value which concern on the image composition process by the imaging device of FIG. It is a figure which shows typically the correspondence of the processing content and feature-value which concern on the image processing by the imaging device of FIG. 11 is a flowchart illustrating an example of an operation related to an imaging process by the imaging apparatus in FIG. 10.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[Embodiment 1]
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to the first embodiment to which the present invention is applied. FIG. 2 is a diagram schematically illustrating the video signal processing unit 13 of the imaging apparatus 100.

The imaging apparatus 100 according to the first embodiment sets the content of the image composition processing for composing the reference image and the target image and the content of the image processing for a plurality of composite images to be composited according to the brightness at the time of image capturing.
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes an imaging unit 1, an imaging auxiliary unit 2, a display unit 3, an operation unit 4, a recording medium 5, a USB terminal 6, and a control unit 7. Etc. are provided.

  The imaging unit 1, as an imaging unit, continuously captures a subject and generates a plurality of image frames. Specifically, the imaging unit 1 includes an imaging lens group 11, an electronic imaging unit 12, a video signal processing unit 13, an image memory 14, a shooting control unit 15, and the like.

The imaging lens group 11 includes a plurality of imaging lenses.
The electronic imaging unit 12 includes an imaging element such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) that converts a subject image that has passed through the imaging lens group 11 into a two-dimensional image signal.

  The video signal processing unit 13 performs predetermined image composition processing and image processing on the image signal output from the electronic imaging unit 12. Specifically, as shown in FIG. 2, the video signal processing unit 13 forms a pyramid hierarchy of image signals related to a plurality of images to be combined, extracts feature points from a predetermined image, and tracks the extracted feature points. A predetermined number of feature points are randomly selected from an optical flow detection unit 131 that performs (track) and the like and a feature point correspondence table in which feature points are associated with each other, and projective transformation (H calculation) using these feature points is performed. ) And a RANSAC unit 132 that performs support calculation using other feature points, and an image synthesis unit 133 that calculates positions corresponding to pixels and adds (synthesizes) image signals.

In addition, the video signal processing unit 13 performs predetermined image processing such as fixed noise erasure processing and color reproduction processing on a plurality of images as image processing means.
The fixed noise is erased by measuring a fixed noise value for each image pickup apparatus in advance and subtracting the value from the continuously shot images.
In the color reproduction process, correction is performed so that the color distribution is extended or the contrast is increased.

The image memory 14 temporarily stores the image signal after image processing.
The imaging control unit 15 controls the electronic imaging unit 12 and the video signal processing unit 13 under the control of the CPU 71. Specifically, the imaging control unit 15 controls the execution of the process of causing the electronic imaging unit 12 to image a subject with a predetermined exposure time and reading an image signal from the imaging area of the electronic imaging unit 12 at a predetermined frame rate. .

  The imaging assisting unit 2 is driven when the subject is imaged by the imaging unit 1, and includes a focus driving unit 21, a zoom driving unit 22, and the like.

The focus drive unit 21 drives a focus mechanism unit (not shown) connected to the imaging lens group 11.
The zoom drive unit 22 drives a zoom mechanism unit (not shown) connected to the imaging lens group 11.
The focus driving unit 21 and the zoom driving unit 22 are connected to the shooting control unit 15 and are driven under the control of the shooting control unit 15.

The display unit 3 displays an image captured by the imaging unit 1. Specifically, the display unit 3 includes a display control unit 31, an image display unit 32, and the like.
The display control unit 31 includes a video memory (not shown) that temporarily stores display data appropriately output from the CPU 71.
The image display unit 32 includes a liquid crystal monitor that displays a predetermined image based on an output signal from the display control unit 31.

  The operation unit 4 is for performing a predetermined operation of the imaging apparatus 100, and includes an operation input unit 41, an input circuit 42, and the like.

  The operation input unit 41 includes a shutter button 41 a that instructs the imaging unit 1 to image a subject. The shutter button 41a outputs an instruction to the CPU 71 to execute the image composition processing by the video signal processing unit 13 while being pressed by the user in the continuous photographing process of the subject.

  The input circuit 42 is for inputting an operation signal output from the operation input unit 41 to the CPU 71.

  The recording medium 5 is configured by, for example, a card-type nonvolatile memory (flash memory), a hard disk, and the like, and stores a plurality of image data of images captured by the imaging unit 1.

  The USB terminal 6 is a terminal for connection with an external device, and transmits and receives data via a USB cable (not shown).

  The control unit 7 controls each unit of the imaging apparatus 100, and includes, for example, a CPU 71, a program memory 72, a data memory 73, and the like.

  The CPU 71 performs various control operations according to various processing programs for the imaging apparatus 100 stored in the program memory 72.

  The data memory 73 is composed of, for example, a flash memory and temporarily stores data processed by the CPU 71.

  The program memory 72 stores various programs and data necessary for the operation of the CPU 71. Specifically, the program memory 72 stores an image composition program 72a, a brightness detection program 72b, a first image processing setting program 72c, a first composition processing setting program 72d, and the like.

Under the control of the CPU 71, the image composition program 72 a is one of a plurality of images that are continuously captured by the electronic image capturing unit 1 in the video signal processing unit 13 (for example, the first image). ) A program for tracking a feature point extracted from within a target image (for example, the second and subsequent images) and realizing a function related to an image composition process for compositing a reference image and the target image based on the tracking result It is.
Specifically, when the CPU 71 executes the image synthesis program 72a, the optical flow detection unit 131 of the video signal processing unit 13 serves as a low-resolution image creation unit based on a plurality of input image signals, for example, an image A low-resolution image in which the resolution is reduced stepwise by sequentially halving the vertical and horizontal pixels is created (pyramidization). Further, the optical flow detection unit 131 selects a predetermined number of feature points (block regions having high features) from the reference image by using any one of the plurality of images as a reference image as a feature point extraction unit. Extract. Then, the optical flow detection unit 131 sequentially tracks the feature points from the image with the lowest resolution to the higher resolution as a feature point tracking unit for the target image.
Then, the RANSAC unit 132 randomly selects four points from the correspondence between a plurality of feature points in the reference image and the target image, calculates a provisional projective transformation matrix H, and calculates the projective transformation matrix H obtained by the support number. Evaluation is repeated, and the operation is repeated to calculate a projective transformation matrix H with higher reliability.
Thereafter, the image composition unit 133 performs projective transformation of the pixels of the target image into the coordinate system of the reference image using the projective transformation matrix H, and synthesizes the target image and the reference image.
In the image composition process, the pixels of the reference image may be projectively transformed into the coordinate system of the target image.
Here, the CPU 71 and the video signal processing unit 13 function as an image synthesizing unit that performs an image synthesizing process for synthesizing the reference image and the target image.

The brightness detection program 72b causes the video signal processing unit 13 to measure the luminance of pixels in a predetermined area based on the image signal of the image captured by the electronic imaging unit 1 under the control of the CPU 71, and capture the image. It is a program for realizing a function relating to brightness detection processing for detecting brightness (photographing situation related information) at the time of shooting.
That is, when the CPU 71 executes the brightness detection program 72b, the video signal processing unit 13 measures the brightness of pixels in a predetermined area based on the image signal of the preview image (through image) of the subject, and the brightness of the image. Detect. Specifically, for example, detection is performed by classifying into four patterns of brightness of “sufficiently bright environment”, “slightly dark environment”, “dark environment”, and “dark environment”.
The “sufficiently bright environment” corresponds to, for example, an environment where shooting is performed outdoors during daytime or under bright lighting. The “slightly dark environment” is, for example, in the evening or in a cloudy day, in a room with low lighting. It corresponds to an environment where you shoot, and “dark environment” means, for example, an environment where you shoot using a flash, such as at night or in a room where the lighting is very dark, and “dark environment” means a night forest. This corresponds to an environment where there is no light source, such as in a room or indoors where there is no lighting.
Here, the CPU 71 and the video signal processing unit 13 function as brightness detection means for detecting the brightness at the time of shooting an image. Further, the CPU 71 and the video signal processing unit 13 function as a shooting situation specifying unit that specifies shooting situation related information related to the shooting situation of an image.

The first image processing setting program 72c causes the CPU 71 to function as brightness image processing setting means. That is, the first image processing setting program 72c, based on the brightness detection result by the brightness detection processing, according to the first image processing content setting table T1 (see FIG. 3), the reference image by the video signal processing unit 13. And a program for causing the CPU 71 to realize a function related to processing for setting the contents of image processing for a target image (composite image).
Specifically, when the brightness detection process detects that the environment is “slightly dark”, the CPU 71 executes the first image processing setting program 72c to cause the video signal processing unit 13 to erase fixed noise. And setting so that the color reproduction process is not executed. When the brightness detection process detects a “dark environment”, the CPU 71 executes the first image processing setting program 72c to cause the video signal processing unit 13 to erase fixed noise and reproduce colors. Set to process. That is, when the shooting environment is extremely dark, there arises a problem that the color distribution for each continuously shot image becomes narrow or changes. This is because an image in which the color component is out of order cannot be reproduced with color and contrast even if they are simply added and combined, and color reproduction processing must be performed. Furthermore, in a “dark environment”, since the image component is small, the influence of fixed noise that appears in a specific pixel that is unique to the camera cannot be ignored. In other words, the image signal has a small value compared with the fixed noise, and the fixed noise cannot be ignored in the image synthesis in such an environment, so it is necessary to remove the fixed noise.

The first synthesizing process setting program 72d causes the CPU 71 to function as a brightness synthesizing process setting unit. That is, the first composition processing setting program 72d performs image composition by the video signal processing unit 13 according to the first composition processing content setting table T2 (see FIG. 4) based on the brightness detection result by the brightness detection processing. This is a program for causing the CPU 71 to realize a function related to the process of setting the process content.
Specifically, when it is detected in the brightness detection process that the environment is “slightly dark”, the CPU 71 executes the first synthesis processing setting program 72d, whereby the optical flow detection unit of the video signal processing unit 13 is executed. The number of feature points extracted by 131 and the number of hierarchies of pyramids are set to be normal numbers. When the brightness detection process detects a “dark environment”, the CPU 71 executes the first synthesis process setting program 72d, so that the number of feature points extracted by the optical flow detection unit 131 and the pyramid are extracted. Set the number of hierarchies to be changed to a larger number than usual. That is, when the shooting environment is extremely dark, since the image signal of each pixel is extremely small and the feature points are buried in noise, the feature point extraction / tracking becomes very difficult. Therefore, in the “dark environment”, it is necessary to increase the accuracy by extracting more feature points and increasing tracking than in the case of image synthesis in a bright environment.

  The program memory 72 stores a first image processing content setting table T1 (see FIG. 3) and a first composition processing content setting table T2 (see FIG. 4).

  As shown in FIG. 3, the first image processing content setting table T1 includes image processing details (“fixed noise elimination processing”, “color reproduction processing”) and brightness (“sufficiently bright environment”), “Slightly dark environment”, “dark environment”, “dark environment”) are stored in association with each other. Specifically, “fixed noise elimination processing” is associated with “not performed” in “slightly dark environment” and “performed” in “dark environment” as the processing content. As the processing content, “color reproduction processing” is associated with “do not perform” in “slightly dark environment” and “execute” in “dark environment”.

  As shown in FIG. 4, the first composition processing content setting table T2 includes image composition processing content (“feature point extraction processing”, “pyramid layering processing”) and brightness (“sufficiently bright environment”). , “Slightly dark environment”, “dark environment”, “dark environment”) in association with each other. Specifically, “feature point extraction processing” is associated with “normal number” in “slightly dark environment” and “more than normal” in “dark environment” as the processing content. In addition, “pyramid hierarchization processing” is associated with “normal number” in “slightly dark environment” and “more than normal” in “dark environment” as the processing content.

Next, imaging processing performed by the imaging apparatus 100 according to Embodiment 1 will be described with reference to FIG.
FIG. 5 is a flowchart illustrating an example of an operation related to the imaging process.

As shown in FIG. 5, when the imaging of the subject by the imaging unit 1 is started, the CPU 71 executes a brightness detection program 72b in the program memory 72 and causes the video signal processing unit 13 to display a preview image (through-image of the subject). Based on the image signal of the image), the brightness of the pixels in the predetermined area is measured to detect the brightness of the image (step S1).
Subsequently, the CPU 71 performs branching to set the shooting mode to the normal shooting mode and the continuous shooting mode in accordance with the detection result of the brightness detection process (step S2). Specifically, when “brightly bright environment” or “dark environment” is detected in the brightness detection process, the CPU 71 sets the shooting mode to the normal shooting mode, while in the brightness detection process. When “slightly dark environment” or “dark environment” is detected, the CPU 71 sets the shooting mode to the continuous shooting mode.

  When “brightly bright environment” is detected in the brightness detection process and the shooting mode is set to the normal shooting mode in step S2, the CPU 71 sets the imaging unit 1 based on the pressing operation of the shutter button 41a by the user. Thus, the subject is photographed under normal photographing conditions (step S3).

  Further, when “dark environment” is detected in the brightness detection process, and the shooting mode is set to the normal shooting mode in step S2, the CPU 71 combines a plurality of continuously shot images. “Unsynthesizable” is displayed on the image display unit 32, indicating that the image composition process for obtaining the image is not executable (step S4). Thereafter, the CPU 71 causes the imaging unit 1 to shoot the subject under normal shooting conditions based on the pressing operation of the shutter button 41a by the user (step S5).

  On the other hand, when “slightly dark environment” or “dark environment” is detected in the brightness detection process, and the shooting mode is set to the continuous shooting mode in step S2, the CPU 71 sets the first in the program memory 72. The image processing setting program 72c is executed to set the content of the image processing by the video signal processing unit 13 according to the first image processing content setting table T1, and the first synthesis processing setting program 72d in the program memory 72 is executed. Then, the content of the image composition processing by the video signal processing unit 13 is set according to the first composition processing content setting table T2 (step S6).

  Subsequently, when continuous imaging of the subject by the imaging unit 1 is started based on the pressing operation of the shutter button 41a by the user (step S7), the “slightly dark environment” and “dark environment” of the detection result of the brightness detection process. ”(Step S8), and if the detection result of the brightness detection process is“ slightly dark environment ”, the CPU 71 executes the image synthesis program 72a in the program memory 72 to perform video signal processing. The optical flow detection unit 131 of the unit 13 performs pyramid hierarchization processing on the reference image and the target image with the number of hierarchies as a normal number, and extracts the feature points with the normal number as the extraction number for the reference image. The extracted feature points are tracked for the target image (step S9).

  On the other hand, if the detection result of the brightness detection process is “dark environment” in step S8, the CPU 71 causes the video signal processing unit 13 to erase fixed noise and reproduce colors for the reference image and the target image. After the processing (step S10), the image composition program 72a in the program memory 72 is executed, and the optical flow detection unit 131 of the video signal processing unit 13 sets the number of hierarchies for the reference image and the target image. Pyramid hierarchization processing is performed as a larger number, feature points are extracted as the number of extractions than the normal number for the reference image, and the extracted feature points are tracked for the target image (Step S11).

  Thereafter, the CPU 71 causes the RANSAC unit 132 of the video signal processing unit 13 to calculate a projective transformation matrix H based on a feature point correspondence table in which a plurality of feature points in the reference image and the target image are associated with each other, and image synthesis is performed. The unit 133 uses the projective transformation matrix H to projectively transform the pixels of the target image into the coordinate system of the reference image and synthesize the target image and the reference image (step S12). Of course, in order to perform the image composition in step S12, it is necessary to complete the photographing of the reference image and at least one target image at that time. Therefore, if the photographing is not completed, the image composition of step S12 is performed. Will be skipped.

  Next, the CPU 71 determines whether or not the end of continuous shooting of the subject is instructed based on the release of the pressing operation of the shutter button 41a by the user (step S13). Here, if it is determined that the end of continuous shooting of the subject has been instructed (step S13; YES), the CPU 71 performs predetermined image processing on the composite image (step S14), and performs the imaging processing. finish.

  If it is determined in step S13 that the end of continuous shooting of the subject has not been instructed (step S13; NO), the process proceeds to step S8, and video signal processing is performed on a plurality of input image signals. Image synthesis processing and image processing by the unit 13 are performed.

As described above, according to the imaging apparatus 100 of the first embodiment, the brightness at the time of image shooting is detected as the image shooting state related information, and based on the detection result of the brightness, The contents of the image processing for the target image and the contents of the image composition processing in the image composition processing can be set.
In other words, in a “sufficiently bright environment”, it is possible to capture a sufficiently clear image by normal shooting, so that there are many cases where disadvantages such as processing time for an image obtained by image synthesis become more conspicuous. Therefore, a normal single image is taken without performing the image composition process. Accordingly, the selection of whether or not to perform image composition is automatically switched depending on the brightness of the shooting environment, so that the user can save the trouble.
Also, in a “slightly dark environment”, it is possible to shoot to a certain degree of brightness even in normal shooting, so that sufficient accuracy can be obtained without performing so carefully in extracting and tracking feature points. In addition, since there are enough image signals for each pixel, it is not easily affected by fixed noise, and the color distribution is not so narrow, so that sufficient quality can be ensured without performing processing to correct them. , Processing time can be shortened.
Also, since image composition is likely to fail in a “dark environment”, it is necessary to perform composition more carefully than in a “slightly dark environment”, so the number of feature points extracted and the number of layers of pyramids are increased. In addition, since the image signal component is small, it is greatly affected by fixed noise, and the color distribution is narrow and the color reproducibility is poor. Therefore, processing for eliminating fixed noise and processing for reproducing colors are performed.
Also, in the “dark environment”, extraction and tracking of feature points for image composition becomes very difficult, and the possibility of successful image composition becomes very low, so without performing image composition processing, In consideration of the user's position, the notification of “composite is impossible” is made, and then normal one-shot shooting is performed.

In this way, image composition in a “dark environment” has many problems related to the accuracy and quality of the image, and a number of processes must be performed to solve it, but these processes are performed in a bright environment. This is not necessary when compositing, but rather it may waste processing time or reduce image quality.
Therefore, in the synthesis of continuously shot images, by performing synthesis processing and image processing appropriate for each brightness, it is possible to reduce waste of processing time and improve overall processing speed by not performing unnecessary processing. Can do. Further, if necessary, the composition success rate can be improved by carefully combining the images, and the image quality of the output image can be improved by using appropriate parameters.
In addition, since it is possible to automatically perform shooting optimal for the brightness of the shooting situation, usability can be improved.

  In the first embodiment, the video signal processing unit 13 performs the brightness of the four patterns detected in the brightness detection process (“sufficiently bright environment”, “slightly dark environment”, “dark environment”, “darkness”). However, the present invention is not limited to this. For example, the brightness detection pattern is divided more finely, and image processing is performed according to the division. The content may be controlled more finely. For example, an environment between “slightly dark environment” and “dark environment” may be set so that only one of fixed noise elimination or color reproduction processing is performed in the environment, or fixed noise is performed. The degree (intensity) of erasing and color reproduction processing may be changed.

In the first embodiment, the processing contents of both fixed noise elimination and color reproduction processing are set in the setting of the image processing contents. However, it is not always necessary to set both, and the fixed noise elimination is performed. In addition, at least one of the color reproduction processing and the color reproduction processing may be set.
Furthermore, the processing content of both the number of feature points extracted and the number of layers of the pyramid is set in the image composition processing content setting, but it is not always necessary to set both, the number of feature points extracted and the number of pyramids At least one of the number of layers may be set.
In addition, the set image processing content is not limited to the fixed noise elimination and color reproduction processing, and the set image composition processing content includes the number of feature points extracted and the number of pyramid layers. The present invention is not limited to this, and any process that can improve the image quality of the output image may be used.

Further, in the first embodiment, in the “sufficiently bright environment”, the normal one-shot shooting is performed without performing the image composition processing. However, the present invention is not limited to this. May be corrected by performing an image composition process.
In the first embodiment, the shutter button 41a is a two-stage push button that can be pressed halfway and fully, and brightness measurement is performed when the shutter button 41a is pressed halfway (step S1). Thereafter, when the shutter button 41a is fully pressed, normal shooting (steps S3 and S5) and continuous shooting (step S7) may be started.

[Embodiment 2]
Hereinafter, the imaging apparatus 200 according to the second embodiment will be described with reference to FIGS.
Here, FIG. 6 is a block diagram illustrating a schematic configuration of the imaging apparatus 200 according to the second embodiment to which the present invention is applied.
The imaging apparatus 200 according to the second embodiment sets the content of the image composition process and the content of the image process for the composite image based on the deviation amount between the reference image and the target image.
The imaging apparatus 200 according to the second embodiment is substantially the same as that according to the first embodiment except that the contents of the image composition process and the contents of the image process are changed, and detailed description thereof is omitted. To do.

  As shown in FIG. 6, the program memory 72 stores a deviation amount detection program 72e, a second composition processing setting program 72f, a second image processing setting program 72g, and the like in addition to the image composition program 72a.

The deviation amount detection program 72e is a program for causing the video signal processing unit 13 to realize a function related to processing for detecting a deviation amount (shooting situation related information) between the reference image and the target image under the control of the CPU 71. .
That is, the CPU 71 executes the shift amount detection program 72e to detect the entire amount of motion between the reference image and the target image without performing feature point tracking on the lowest resolution image that has been pyramidal, The value is set as the deviation amount. Specifically, for example, detection is performed by classifying into three patterns of shift amounts of “small shift amount”, “medium shift amount”, and “large shift amount”.
“Small amount of deviation” means, for example, a case where the amount of deviation between the reference image and the target image is smaller than a predetermined value t1, and “medium amount of deviation” means that the amount of deviation between the reference image and the target image is equal to or larger than the predetermined value t1. And is smaller than the predetermined value t2 (t1 <t2), and “large deviation amount” is a case where the deviation amount between the reference image and the target image is equal to or larger than the predetermined value t2.
Here, the CPU 71 and the video signal processing unit 13 function as a deviation amount detection unit that detects a deviation amount between the reference image and the target image. Further, the CPU 71 and the video signal processing unit 13 function as a shooting situation specifying unit that specifies shooting situation related information related to the shooting situation of an image.

The second synthesis processing setting program 72f causes the CPU 71 to function as a shift synthesis processing setting unit. That is, the second composition processing setting program 72f performs image composition by the video signal processing unit 13 in accordance with the second composition processing content setting table T3 (see FIG. 7) based on the detection result of the displacement amount by the displacement amount detection process. This is a program for causing the CPU 71 to realize a function related to the process of setting the process content.
Specifically, the CPU 71 executes the second synthesis processing setting program 72f, so that the target image in the feature point tracking by the optical flow detection unit 131 according to the magnitude of the deviation amount detected by the deviation amount detection process. The size of the search range (tracking range) is changed. For example, when the deviation amount detection process detects that the deviation amount is “small deviation amount”, the feature point search range is set to a normal size, and the deviation amount is detected to be “medium deviation amount”. Then, the feature point search range is set to a size larger than usual. In other words, the search range of feature points is usually determined from the image size, and the size of this search range directly affects the processing time, but considering the fact that a large shift occurs, The size must be increased to some extent. On the other hand, when the amount of deviation is small, the processing time for increasing the size of the search range is wasted. Therefore, it is necessary to perform tracking more carefully as the amount of deviation between the reference image and the target image increases.
In addition, when a shift that is beyond the search range occurs and the shift amount is detected to be “large shift amount”, similar pixels in the image are compared with the case where the shift amount is small. Since there is a high possibility that the image is erroneously tracked, the target image is discarded without being used for the image synthesis process at that time.

The second image processing setting program 72g causes the CPU 71 to function as a shifted image processing setting unit. That is, the second image processing setting program 72g, based on the detection result of the shift amount by the shift amount detection process, in accordance with the second image processing content setting table T4 (see FIG. 8), the composite image by the video signal processing unit 13. This is a program for causing the CPU 71 to realize a function related to processing for setting the content of image processing for.
Specifically, when the CPU 71 executes the second image processing setting program 72g, the sharpness processing of the synthesized image by the video signal processing unit 13 according to the magnitude of the deviation detected by the deviation detection processing. Change strength. For example, if the deviation amount detection process detects that the deviation amount is “small deviation amount”, the strength of the sharpness filter is normal, and the deviation amount is detected as “medium deviation amount”. If so, the strength of the sharpness filter is set to be stronger than usual. In other words, the greater the amount of deviation between the reference image and the target image, the greater the effect of parallax at the time of composition, and the combined accuracy decreases and the photographed image becomes blurred. The resulting failed image may occur. This is because, in a shooting environment where the amount of deviation between the reference image and the target image is large and the composite image is blurred, it is necessary to perform sharpness processing in order to eliminate this.
Note that the strength of the sharpness processing is preferably set according to the maximum value of the phase shift during continuous shooting because the image quality deteriorates if the sharpness filter is excessively applied.

  The program memory 72 stores a second composition processing content setting table T3 (see FIG. 7) and a second image processing content setting table T4 (see FIG. 8).

  As shown in FIG. 7, the second composition processing content setting table T3 includes a “search range size” and a deviation amount (“deviation amount small”, “deviation amount” in the feature point tracking as the processing contents of the image composition processing. “Medium” and “large deviation”) are stored in association with each other. Specifically, “the size of the search range” is associated with “normal” for “small deviation” and “larger than normal” for “medium deviation”.

  As shown in FIG. 8, the second image processing content setting table T4 includes “sharpness filter strength” and deviation amounts (“deviation amount small”, “medium deviation amount”) for the composite image as the processing content of the image processing. , “Large deviation amount”). Specifically, “the strength of the sharpness filter” is associated with “normal” for “small deviation” and “stronger than normal” for “medium deviation”.

Next, imaging processing by the imaging apparatus 200 of Embodiment 2 will be described with reference to FIG.
FIG. 9 is a flowchart illustrating an example of an operation related to the imaging process.

As shown in FIG. 9, when continuous shooting of the subject by the imaging unit 1 is started based on the pressing operation of the shutter button 41a by the user (step S21), the CPU 71 detects the shift amount detection program 72e in the program memory 72. To cause the video signal processing unit 13 to detect the amount of deviation between the reference image and the target image (step S22). Here, when the first image is the reference image and the second and subsequent images are the target images, after the continuous shooting is started in step S21, the shooting of the second image is completed and then step S22. The shift amount detection process is started and the shift amount detection process in step S22 is performed every time image capturing is completed.
Subsequently, the CPU 71 branches whether or not the target image to be synthesized is the target of the image synthesis process according to the detection result of the deviation amount detection process (step S23), and is detected as “large deviation amount”. Then, after discarding the target image to be synthesized (step S24), the CPU 71 proceeds to step S22.

  On the other hand, when “small deviation amount” or “medium deviation amount” is detected in step S23, the CPU 71 sets the target image as a target of image synthesis processing, and sets the second synthesis processing setting program 72f in the program memory 72. To set the content of the image composition processing by the video signal processing unit 13 in accordance with the second composition processing content setting table T3, and execute the second image processing setting program 72g in the program memory 72 to execute the second composition processing content setting table T3. The content of the image processing by the video signal processing unit 13 is set according to the image processing content setting table T4 of No. 2 (step S25). Here, when executing the second image processing setting program 72g and setting the content of the image processing by the video signal processing unit 13 according to the second image processing content setting table T4, the content of the image processing to be set is set. Only when the intensity (stronger than normal) is higher than the currently set image processing content (normal), the update setting is made. If the intensity of the image processing content is the same, update setting may be performed, or update setting may not be performed.

Subsequently, the CPU 71 executes the image synthesis program 72a in the program memory 72 to cause the optical flow detection unit 131 of the video signal processing unit 13 to perform pyramid hierarchization processing on the reference image and the target image. The feature points are extracted from the reference image, and the extracted feature points are tracked from the target image within the search range set in accordance with the shift amount (step S26).
Thereafter, the CPU 71 causes the RANSAC unit 132 of the video signal processing unit 13 to calculate a projective transformation matrix H based on a feature point correspondence table in which a plurality of feature points in the reference image and the target image are associated with each other, and image synthesis is performed. The unit 133 performs projective transformation of the pixels of the target image into the coordinate system of the reference image using the projective transformation matrix H to synthesize the target image and the reference image (step S27).

  Next, the CPU 71 determines whether or not the end of continuous shooting of the subject is instructed based on the release of the pressing operation of the shutter button 41a by the user (step S28). If it is determined that the end of continuous shooting of the subject is instructed (step S28; YES), the CPU 71 causes the video signal processing unit 13 to set the sharpness set according to the maximum shift amount during continuous shooting. Sharpness processing is performed on the composite image with the strength of the filter (step S29), and the imaging processing ends.

  If it is determined in step S28 that the end of continuous shooting of the subject has not been instructed (step S28; NO), the process proceeds to step S22, and video signal processing is performed on a plurality of input image signals. A setting is made as to whether or not the image is to be processed by the unit 13.

As described above, according to the imaging apparatus 200 of the second embodiment, the shift amount between the reference image and the target image is detected as the image capturing state related information, and based on the detection result of the shift amount, the image composition in the image composition processing is performed. It is possible to set the content of the processing and the content of the image processing for the composite image in the image processing.
In other words, when camera shake or the like occurs, by setting the size of the feature point search range at the time of feature point tracking according to the amount of deviation between the reference image and the target image, tracking by the size, Waste of unnecessary processing time can be suppressed. In addition, as the amount of displacement increases, the effect of parallax increases during image composition, resulting in a decrease in composition accuracy and a blurred photographed image. Sometimes. Therefore, in a shooting environment where the amount of deviation between the reference image and the target image is large and the composite image is blurred, sharpness processing is performed in order to eliminate this. Furthermore, if the amount of deviation between the reference image and the target image is remarkably large, there is a high possibility that image composition will fail, and the captured image is discarded at that time.

As described above, the amount of deviation between the reference image and the target image affects the image composition processing and image processing. Therefore, the unique image composition processing and image processing are not considered without considering the amount of deviation. Doing so would result in excessive or insufficient processing.
Therefore, in the synthesis of continuously shot images, by performing image synthesis processing and image processing appropriate for the amount of deviation between the reference image and the target image, the waste of processing time can be suppressed by not performing unnecessary processing. The processing speed can be improved. Further, if necessary, the composition success rate can be improved by carefully combining the images, and the image quality of the output image can be improved by using appropriate parameters.

  In the second embodiment, the amount of motion between the reference image and the target image is set as the amount of shift in the detection of the shift amount between the reference image and the target image. However, the present invention is not limited to this. An acceleration sensor may be used to measure the moving distance of the camera during continuous shooting, and the moving distance may be used as a shift amount.

  In the second embodiment, the content of the image processing to be set is the sharpness processing. However, the content of the image processing to be set is not limited to the sharpness processing, and further, the content of the image composition processing to be set. Is not limited to changing the size of the search range for feature points, and any process that can improve the image quality of the output image may be used.

[Embodiment 3]
Hereinafter, the imaging apparatus 300 according to the third embodiment will be described with reference to FIGS.
Here, FIG. 10 is a block diagram illustrating a schematic configuration of the imaging apparatus 300 according to the third embodiment to which the present invention is applied.
The imaging apparatus 300 according to the third embodiment sets the content of the image composition process and the content of the image process for the composite image based on the measurement result of the feature amount of the feature points.
The imaging apparatus 300 according to the third embodiment is substantially the same as that according to the first embodiment except for the contents of the image composition process and the change of the contents of the image process, and detailed description thereof will be omitted. .

  As shown in FIG. 10, the program memory 72 stores a feature amount detection program 72h, a third composition processing setting program 72i, a third image processing setting program 72j, and the like in addition to the image composition program 72a.

The feature amount detection program 72h allows the video signal processing unit 13 to realize a function related to processing for detecting feature amounts (shooting situation related information) of feature points extracted from the reference image under the control of the CPU 71. It is a program.
That is, when the CPU 71 executes the feature amount detection program 72h, the feature amount is detected based on the degree of spread and uniformity of feature points extracted from the reference image. Specifically, detection is performed by classifying into three pattern feature amounts of “small feature amount”, “medium feature amount”, and “large feature amount”.
“Large feature amount” means, for example, a case where feature points are distributed almost uniformly over a wide range in the image, and “medium feature amount” means that feature points are distributed over a wide range in the image. However, the distribution is not uniform, and “small feature amount” is a case where the feature points are not distributed over a wide range in the image and the distribution is not uniform. When a uniform white wall or an empty image is photographed with no object, the feature amount becomes small.
Here, the CPU 71 and the video signal processing unit 13 function as a feature amount measuring unit that measures a feature amount of a feature point extracted from the reference image. Further, the CPU 71 and the video signal processing unit 13 function as a shooting situation specifying unit that specifies shooting situation related information related to the shooting situation of an image.

The third synthesis process setting program 72i causes the CPU 71 to function as a feature synthesis process setting unit. That is, the third composition processing setting program 72i performs image composition by the video signal processing unit 13 according to the third composition processing content setting table T5 (see FIG. 11) based on the feature amount detection result by the feature amount detection processing. This is a program for causing the CPU 71 to realize a function related to the process of setting the process content.
Specifically, the CPU 71 executes the third synthesis processing setting program 72i, so that the number of feature points extracted by the optical flow detection unit 131 according to the size of the feature amount detected by the feature amount detection process, Change the number of iterations and the number of pyramid hierarchies. For example, if the feature amount detection process detects that the feature amount is “large feature amount”, the number of feature points extracted, the number of tracking iterations, and the number of hierarchy levels of the pyramid are set to the normal number, and the feature amount is When it is detected that the feature amount is “medium”, the number of feature points extracted, the number of tracking iterations, and the number of hierarchies of the pyramid are set higher than usual, and the feature amount is detected as “small feature amount”. Then, the number of feature points extracted, the number of tracking iterations, and the number of pyramid hierarchies are set to the largest number (a larger number than when the feature amount is “medium in feature amount”). In other words, the situation where the image can be best synthesized is when the feature amount is uniformly distributed over a wide range in the image and the feature amount is large. On the other hand, when the feature amount of the feature point is small, there is a problem that a pixel having a similar feature is mistracked and tracking tends to fail. Therefore, when the feature amount is small, it is necessary to add more processing such as increasing the number of feature points extracted, increasing the number of tracking iterations, or increasing the number of pyramid layers, and perform image synthesis processing more carefully. This is because.

The third image processing setting program 72j causes the CPU 71 to function as feature image processing setting means. That is, the third image processing setting program 72j, based on the feature amount detection result by the feature amount detection processing, in accordance with the third image processing content setting table T6 (see FIG. 12), the composite image by the video signal processing unit 13. This is a program for causing the CPU 71 to realize a function related to processing for setting the content of image processing for.
Specifically, the CPU 71 executes the third image processing setting program 72j so that the video signal processing unit 13 performs sharpness processing on the synthesized image according to the feature amount detected by the feature amount detection processing. Change strength. For example, if the feature amount detection process detects that the feature amount is “large feature amount”, the strength of the sharpness filter is normal, and the feature amount is detected as “medium feature amount”. The strength of the sharpness filter is made stronger than usual, and if the feature quantity is detected as “small feature quantity”, the strength of the sharpness filter is the strongest (the feature quantity is “feature quantity” Stronger than “medium”). That is, the smaller the feature amount, the lower the tracking accuracy, and the captured image becomes blurred. Therefore, if the sharpness filter to be applied to the composite image is strengthened as the feature amount is small, it is possible to apply the sharpness filter according to the composite accuracy, and as a result, the image quality can be improved. Because.

  The program memory 72 stores a third composition processing content setting table T5 (see FIG. 11) and a third image processing content setting table T6 (see FIG. 12).

  As shown in FIG. 11, the third composition processing content setting table T5 includes image composition processing content ("feature point extraction processing", "tracking iteration processing", "pyramid hierarchization processing") and features. Point feature values (“small feature value”, “medium feature value”, “large feature value”) are stored in association with each other. Specifically, “feature point extraction processing” as processing details is associated with “normal number” for “large feature amount”, “more than normal” for “medium feature amount”, “Small amount” is associated with “most often”. In addition, as the processing content, “tracking repetitive processing” is associated with “normal number” for “large feature amount”, “more than normal” for “medium feature amount”, and “small feature amount” for “medium amount”. “Most” is associated. In addition, “Pyramid hierarchization processing” is associated with “large feature amount” associated with “normal number”, “medium feature amount” associated with “more than normal”, and “small feature amount”. Is associated with “most common”.

  As shown in FIG. 12, the third image processing content setting table T6 includes “sharpness filter strength” and feature point feature amounts (“feature amount small”, “feature” for the composite image as the image processing content. “Medium” and “large feature”) are stored in association with each other. Specifically, the “strength of the sharpness filter” is associated with “normal” in “large feature amount”, “stronger than normal” in “medium feature amount”, and “small in feature amount”. “The strongest” is associated.

Next, imaging processing by the imaging apparatus 300 according to the third embodiment will be described with reference to FIG.
FIG. 13 is a flowchart illustrating an example of an operation related to the imaging process. Note that “normal” set when the feature amount is detected as “large feature amount” as the processing content of the image processing (the strength of the sharpness filter) at the time of starting the operation related to the imaging processing. It is assumed that it is set by default. Similarly, at the time of starting the operation related to the imaging process, as the processing contents of the image synthesis process (“feature point extraction process”, “tracking iterative process”, “pyramid hierarchization process”), the feature amount It is assumed that “normal number” that is set when it is detected that “the feature amount is large” is set by default.

  As illustrated in FIG. 13, when continuous shooting of the subject by the imaging unit 1 is started based on the pressing operation of the shutter button 41 a by the user (step S <b> 41), the CPU 71 loads the image composition program 72 a in the program memory 72. Then, the optical flow detection unit 131 of the video signal processing unit 13 performs a “normal number” pyramid hierarchization process and a feature point extraction process that are set as default for the reference image (step S42). . Here, when the first image is the reference image and the second and subsequent images are the target images, after the continuous shooting is started in step S41, the shooting of the first image is completed and then step S42. Then, the feature point extraction process is performed.

  Next, the CPU 71 determines whether or not the number of extracted feature points is equal to or greater than a predetermined value (step S43). Here, if it is determined that the number of feature points is not equal to or greater than the predetermined value (step S43; NO), the CPU 71 executes an image composition process for composing a plurality of continuously shot images to obtain one image. Is displayed on the image display unit 32 (step S44), and then the subject is photographed under normal photographing conditions (step S45). .

On the other hand, if it is determined in step S43 that the number of feature points is equal to or greater than the predetermined value (step S43; YES), the CPU 71 executes the feature amount detection program 72h in the program memory 72 to execute the reference image in the reference image. The feature amount is detected from the extent and uniformity of the feature points extracted from (step S46; feature amount detection process).
Subsequently, the CPU 71 branches whether or not to reset the set image composition processing and the contents of the image processing in accordance with the detection result of the feature amount detection processing (step S47). Or “small feature amount” is detected, the CPU 71 executes the third synthesis processing setting program 72i in the program memory 72, and the video signal processing unit 13 executes the third synthesis processing content setting table T5. The content of the image composition processing is reset, and the third image processing setting program 72j in the program memory 72 is executed, and the content of the image processing by the video signal processing unit 13 is determined according to the third image processing content setting table T6. Reset (step S48). In other words, the default setting “normal number” as the content of the image composition process is changed to “more than normal” (medium feature amount) or “most” (small feature amount), and the image processing content The default setting “normal” is changed to “stronger than normal” (medium feature amount) or “strongest” (small feature amount).
After that, the CPU 71 executes the image synthesis program 72a in the program memory 72, and resets the pyramid hierarchization processing for the reference image in accordance with the feature amount in the optical flow detection unit 131 of the video signal processing unit 13. The feature points are re-extracted with the number set according to the feature amount (step S49), and the number of pyramid hierarchization processes and tracking iterations set according to the feature amount are performed. The extracted feature points are tracked by performing (step S50).

  On the other hand, when “large feature amount” is detected in the feature amount detection process in step S47, the process proceeds to step S50, and the CPU 71 causes the optical flow detection unit 131 of the video signal processing unit 13 to apply the target image. The extracted feature points are tracked by performing the default “normal number” pyramid hierarchization process and the iterative tracking process (step S50).

  Thereafter, the CPU 71 causes the RANSAC unit 132 of the video signal processing unit 13 to calculate a projective transformation matrix H based on a feature point correspondence table in which a plurality of feature points in the reference image and the target image are associated with each other, and image synthesis is performed. The unit 133 uses the projective transformation matrix H to projectively transform the pixels of the target image into the reference image coordinate system to synthesize the target image and the reference image (step S51).

  Next, the CPU 71 determines whether or not the end of the continuous shooting of the subject is instructed based on the release of the pressing operation of the shutter button 41a by the user (step S52). Here, if it is determined that the end of the continuous shooting of the subject is instructed (step S52; YES), the CPU 71 sets the strength of the sharpness filter set as a default or the feature amount in the video signal processing unit 13. Then, the sharpness processing is performed on the composite image with the strength of the sharpness filter reset according to (step S53), and the imaging processing is terminated.

  If it is determined in step S52 that the end of continuous shooting of the subject has not been instructed (step S52; NO), the process proceeds to step S50, and video signal processing is performed on a plurality of input image signals. The extracted feature points are tracked by causing the unit 13 to perform pyramid hierarchization processing and tracking iterative processing.

As described above, according to the imaging apparatus 300 of the third embodiment, the feature amount of the feature point is detected as the image capturing state-related information, and the content of the image composition processing in the image composition processing based on the feature amount detection result. It is possible to set the content of image processing for a composite image in image processing.
That is, the smaller the feature amount, the more carefully the image composition processing is performed by adding processing such as increasing the number of extracted feature points, increasing the number of tracking iterations, or increasing the number of pyramid layers. If the sharpness filter applied to the composite image is strengthened as the feature amount is small, it is possible to apply a sharpness filter corresponding to the composite accuracy. Furthermore, if the feature amount is too small to extract a specified number of feature points from the reference image, there is a very high possibility that image composition will fail, so consider the user's position without performing image composition processing. After notifying “insufficient feature points”, normal one-shot shooting is performed.

As described above, there are many problems related to the accuracy and quality of the image, and many processes must be performed in order to solve the problem. This is not necessary when synthesizing images, but rather, processing time may be wasted and image quality may be lowered.
Therefore, in the synthesis of continuously shot images, by performing synthesis processing and image processing appropriate for each feature amount, it is possible to reduce waste of processing time and improve overall processing speed by not performing unnecessary processing. Can do. Further, if necessary, the composition success rate can be improved by carefully combining the images, and the image quality of the output image can be improved by using appropriate parameters.

In the third embodiment, all the processing contents of the number of feature points extracted, the number of tracking iterations, and the number of pyramid hierarchies are set in the setting of the image composition processing content. Instead, at least one of the number of feature point extractions, the number of tracking iterations, and the number of pyramid layers may be set.
In addition, the content of image processing that is set is not limited to sharpness processing, and the content of image composition processing that is set is limited to the number of feature points extracted, the number of tracking iterations, and the number of layers of pyramids. Any processing that can improve the image quality of the output image is possible.

  The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.

  In the first to third embodiments, the contents of the image composition processing and the image processing are set according to the shooting state of the image. However, the present invention is not limited to this. For example, shooting state related information After changing the exposure time, aperture, sensitivity, etc. according to the “brightness”, “reference image and target image deviation amount”, and “feature point feature amount” specified Even if the synthesis process is performed, the overall processing speed can be improved and the image quality of the output image can be improved.

  Furthermore, the configurations of the imaging apparatuses 100, 200, and 300 are merely examples as illustrated in the above-described embodiment, and are not limited thereto.

  In addition, the image capturing apparatuses 100, 200, and 300 are illustrated as the image synthesizing apparatus. However, the present invention is not limited to this. Shooting condition specifying process, brightness detection process, deviation amount detection process, feature amount detection process, low-resolution image creation process, feature point extraction process, feature point tracking process Further, a synthesis process setting process, an image process setting process, and the like may be performed.

  In addition, in the above-described embodiment, the shooting situation specifying unit, the brightness detection unit, the deviation amount detection unit, the feature amount detection unit, the low-resolution image creation unit, the feature point extraction unit, the feature point tracking unit, the synthesis processing setting unit, the brightness The CPU 71 has predetermined functions of the image composition processing setting means, the deviation composition processing setting means, the feature composition processing setting means, the image processing setting means, the brightness image processing setting means, the deviation image processing setting means, and the feature image processing setting means. The configuration is realized by executing a program or the like. However, the configuration is not limited to this. For example, the configuration may be configured by a logic circuit or the like for realizing various functions.

100, 200, 300 Imaging device (image composition device)
13 Video signal processing unit (image synthesis means, image processing means)
131 Optical flow detection unit (low-resolution image creation means, feature point extraction means, feature point tracking means)
132 RANSAC section 133 Image composition section 71 CPU (shooting condition specifying means, brightness detection means, deviation amount detection means, feature amount detection means, low resolution image creation means, feature point extraction means, feature point tracking means, composition processing setting means Brightness composition processing setting means, deviation composition processing setting means, feature composition processing setting means, image processing setting means, brightness image processing setting means, deviation image processing setting means, feature image processing setting means)

The image synthesizing device of the invention according to claim 1
Combining means for performing image combining processing for combining any one of the plurality of images with the target image ;
Detecting means for detecting the feature quantity of the feature point extracted from the reference image as any of the feature quantities divided in advance ;
Setting means for setting the content of the image composition processing by the composition means based on the detection result by the detection means;
It is characterized by providing.

The program of the invention described in claim 7 is:
On the computer,
A composition function for performing image composition processing for compositing any one of the plurality of images with the target image ;
A detection function for detecting a feature quantity of a feature point extracted from the reference image as any of the feature quantities that have been preliminarily classified;
A setting function for setting the content of the image composition processing based on the detection result of the feature amount
It is characterized by realizing.

Claims (2)

Image combining means for performing image combining processing for combining any one of the plurality of images and the target image;
Detecting means for detecting the brightness at the time of shooting the image as any one of the previously divided brightness;
Based on the detection result by the detection means, composition processing setting means for setting the content of the image composition processing by the image composition means;
With
The image composition means includes
Including feature point extraction means for extracting feature points from the reference image;
When the combination processing setting means detects the brightness that is classified as dark as the detection result by the detection means, the number of feature points extracted by the feature point extraction means is a larger number,
The image composition means includes
Including image creating means for creating a low-resolution image by gradually reducing the resolution of the plurality of images,
The composition processing setting means sets the number of steps related to the reduction of the resolution by the image creating means to a larger number when the detection result by the detection means is detected by the brightness that is classified as dark. An image composition device. On the computer,
A combining function for performing image combining processing for combining any one of the plurality of images and the target image;
A detection function for detecting the brightness at the time of shooting the image as any one of the pre-divided brightness levels;
A setting function for setting the content of the image composition processing based on the detection result by the detection function;
Is a program for realizing
The synthesis function is
An extraction function for extracting feature points from the reference image;
When the setting function detects a brightness classified as dark as a detection result by the detection function, the number of feature points extracted by the extraction function is set to a larger number,
The synthesis function is
A creation function for creating a low-resolution image by gradually reducing the resolution of the plurality of images;
The setting function is a program for increasing the number of steps related to the reduction of the resolution by the creation function when the brightness is classified as dark as a detection result by the detection function.

Images