JP5055571B2 - Image processing apparatus, electronic camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an electronic camera, and an image processing program.

2. Description of the Related Art Conventionally, a technique is known in which blurring of a captured image is repaired by performing multiple divided exposures using the same electronic camera and aligning and synthesizing the obtained images (for example, Patent Document 1) below. Pattern matching, block matching, and the like are known as processing for detecting such misalignment of a plurality of images.
The following Patent Document 2 is known as a technique for detecting a positional deviation of a pattern between images output from different imaging elements having different resolutions. In this prior art, first, a global search is performed between a high-resolution image and a low-resolution image to find a candidate for a matching region. Next, both images are subjected to pixel interpolation within the range of the matching region, and a region in which the pattern matches in detail is found by comparison between the interpolated pixels. By such a two-stage search process, a positional shift between images having different resolutions is detected.
JP 2002-107787 A International Publication WO95 / 04329 Pamphlet

In the prior art, as a result of the divided exposure, a high-resolution still image must be read out a plurality of times. In this case, since the frame rate of a still image is generally low, the time interval between images is greatly opened. For this reason, in a moving subject, the degree of coincidence of patterns between a plurality of images is low, and it is difficult to perform good image composition.
Note that a low-resolution image can be read out at a high frame rate by using the low-resolution reading mode (draft mode) of the image sensor. In this case, the time interval between images can be shortened. However, the detection accuracy of the positional deviation of the pattern is rough between the low resolution images. For this reason, when the patterns are aligned according to the low-precision positional deviation, the patterns are likely to fail, for example, the edges between images are slightly shifted. As a result, it is difficult to perform good image composition even with this method.
Therefore, an object of the present invention is to provide a technique for solving the above-described problems and performing satisfactory image composition.

<< 1 >> An image processing apparatus of the present invention includes an image input unit, a positional deviation detection unit, and an image composition unit.
The image input unit captures a plurality of first resolution images obtained by photographing the same subject and a second resolution image having a higher resolution than at least one first resolution image obtained by photographing the same subject.
Positional displacement detector detects a positional deviation between the first resolution image each second resolution image.
Image combining unit, in the case of an image where the second resolution image taken with a short exposure time than the exposure time determined by the exposure calculation, based on the positional deviation, the line physician pairs and aligned with a plurality of first resolution images The signal components of the plurality of first resolution images are rearranged to generate a rearranged image having a larger image size than the first resolution image, and a composite image is generated based on the second resolution image and the rearranged image .
<< 2 >> Preferably, the positional deviation detection unit includes a phase division unit and a precision detection unit.
The phase dividing unit generates a plurality of pieces of sampling information whose phases are shifted from each other by sub-sampling the second resolution image or its features while shifting the phase.
Precision detection unit, by detecting the best matching positional deviation between the first resolution image and a plurality of sampling information, than the pixel spacing of the first resolution image to detect the positional deviation of the small unit.
<< 3 >> Preferably, the positional deviation detection unit includes a rough detection unit and a precise detection unit.
Coarse detection unit detects a positional shift between a plurality of first resolution image.
The precision detection unit detects a positional shift between the second resolution image and the first resolution image, starting from the positional shift detected by the coarse detection unit.
<< 4 >> Preferably, the image composition unit generates a composite image by locally reducing the composition ratio of the rearranged image as the signal difference between the second resolution image and the rearranged image is larger.
<< 5 >> Preferably, the image synthesizing unit generates a synthesized image by lowering the synthesis ratio of the rearranged images locally at a location where the local signal change of the second resolution image is larger.
<< 6 >> Preferably, the image composition unit generally increases the composition ratio of the rearranged images as the time interval and / or the positional deviation between the second resolution image and the first resolution image is smaller. Generate.
<< 7 >> Preferably, the image composition unit generates a composite image for a signal component having a high visual sensitivity that forms a color image. On the other hand, the image composition unit generates rearranged images for the remaining signal components with low visual sensitivity.
<< 8 >> An electronic camera of the present invention includes the image processing device according to any one of << 1 >> to << 7 >> and an imaging unit that captures an object with at least two types of resolutions.
In this case, the second resolution image processed by the image processing apparatus is a high-resolution still image captured by the imaging unit. On the other hand, the first resolution image processed by the image processing apparatus is a low-resolution image captured by the imaging unit before and / or after the still image is captured.
<< 9 >> The image processing program of the present invention is a program for causing a computer to function as the image processing apparatus described in any one of << 1 >> to << 7 >>.

In the present invention, a low resolution image with a high frame rate is synthesized. Therefore, the degree of coincidence of the patterns between the images increases, and good image composition is possible.
Furthermore, in the present invention, a positional shift of each of a plurality of low resolution images is detected using a high resolution image having a high pixel density as a position reference. Therefore, it is possible to detect the relative positional deviation of the low resolution image with high accuracy. As a result, the positioning accuracy of the pattern of the low-resolution image is high, and it becomes possible to perform better image composition.

[Description of electronic camera configuration]
FIG. 1 is a block diagram showing an electronic camera 10 (including an image processing device 25) according to the present embodiment.
In FIG. 1, a photographing lens 12 is attached to the electronic camera 10. In the image space of the photographic lens 12, the imaging surface of the imaging element 11 is arranged. The imaging element 11 is controlled by the imaging control unit 14. The image pickup device 11 has a mode for reading a high-resolution image and a mode for reading a low-resolution image by performing pixel thinning and pixel addition inside the device. The image signal output from the image sensor 11 is processed via the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17.
This memory 17 is connected to a bus 18. The bus 18 is also connected with an imaging control unit 14, a microprocessor 19, a recording unit 22, an image compression unit 24, a monitor display unit 30, an image processing device 25, and the like. The microprocessor 19 is connected to an operation unit 19a such as a release button. A recording medium 22a is detachably attached to the recording unit 22.

[Description of Image Processing Device 25]
FIG. 2 is a block diagram schematically showing the configuration of the image processing device 25.
The high resolution image read from the memory 17 is supplied to the reduced image generation unit 32, the feature amount extraction unit 33, and the image composition unit 34 via the gain correction unit 31, respectively. The output data of the reduced image generation unit 32 is supplied to the rough detection unit 36 via the feature amount extraction unit 35. The output data of the feature quantity extraction unit 33 is supplied to the precision detection unit 38 via the phase division unit 37.
On the other hand, the plurality of low resolution images read from the memory 17 are supplied to the feature amount extraction unit 39 and the image composition unit 34, respectively. The output data of the feature quantity extraction unit 39 is supplied to the rough detection unit 36 and the precise detection unit 38, respectively.
The positional deviation roughly detected by the rough detection unit 36 is supplied to the precision detection unit 38. The positional deviation detected with high precision by the precision detection unit 38 is supplied to the image composition unit 34. The image synthesis unit 34 synthesizes a plurality of low resolution images and high resolution images based on the detection result of the positional deviation.

[Description of operation]
3 and 4 are flowcharts for explaining the operation of the electronic camera 10. Hereinafter, this operation will be described along the step numbers shown in FIG.

Step S1: When the main power supply of the electronic camera 10 is turned on, the microprocessor 19 instructs the imaging control unit 14 to read out a low resolution image. The imaging control unit 14 drives the imaging device 11 in a low resolution readout mode, and sequentially reads out low resolution images at, for example, 30 frames / second as shown in FIG.

Step S2: The low resolution image read from the image sensor 11 is processed through the signal processing unit 15 and the A / D conversion unit 16, and then temporarily stored in the memory 17. The microprocessor 19 sequentially deletes the low resolution images exceeding the predetermined number of frames in the memory 17 from the oldest one.
The predetermined number of frames here corresponds to the number of frames of a low-resolution image used for composition of a rearranged image described later, and is preferably set to be equal to or greater than (number of pixels of a high-resolution image / number of pixels of a low-resolution image). .
For example, when the number of vertical and horizontal pixels of the low-resolution image is 1/4 of the number of vertical and horizontal pixels of the high-resolution image, the predetermined number of frames is preferably set to 4 × 4 = 16 frames or more.

Step S3: The monitor display unit 30 displays a low resolution image (through image) on the monitor screen. On the other hand, the microprocessor 19 calculates exposure based on the photometric result of a photometric unit (not shown) and the brightness of the low resolution image, and determines the exposure time of the high resolution image.

Step S4: Here, the microprocessor 19 determines whether or not the release button has been fully pressed by the user.
When the release button is fully pressed, the microprocessor 19 proceeds to step S5.
On the other hand, if the release button has not been fully pressed, the microprocessor 19 returns the operation to step S1.

Step S5: Here, the microprocessor 19 determines whether or not the exposure time setting of the high-resolution image determined in Step S3 is equal to or less than an allowable upper limit where blur is not noticeable. For example, this allowable upper limit is set to about 1 / (35 mm equivalent focal length of the photographing lens 12) seconds.
If the exposure time setting is less than or equal to the allowable upper limit, the microprocessor 19 proceeds to step S6.
On the other hand, when the exposure time setting exceeds the allowable upper limit, the microprocessor 19 shifts the operation to step S7.

Step S6: The imaging control unit 14 performs shutter control of the imaging device 11 according to the set exposure time. Subsequently, the imaging control unit 14 drives the imaging device 11 in a high-resolution readout mode and reads out a high-resolution image. This high-resolution image (still image) is recorded and stored in the recording medium 22a after image processing and image compression as in the prior art.
With this operation, the electronic camera 10 completes the photographing operation.

Step S7: On the other hand, if it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 limits the exposure time to an allowable upper limit that does not cause blurring.
The imaging control unit 14 controls the shutter of the imaging device 11 according to the exposure time limited to a short time. In this state, the imaging control unit 14 drives the imaging device 11 in a high resolution readout mode and reads out a high resolution image. This high-resolution image is an image that has a low signal level due to insufficient exposure but is less likely to blur. This high resolution image is temporarily recorded in the memory 17.

Step S <b> 8: The gain correction unit 31 in the image processing device 25 takes in a high resolution image from the memory 17. The gain correction unit 31 adjusts the gain of this high resolution image to match the signal level of the low resolution image.

Step S9: The reduced image generating unit 32 converts the resolution of the high-resolution image after gain adjustment and matches the number of pixels of the low-resolution image.
For example, by extracting an average value of 4 × 4 pixels, the number of vertical and horizontal pixels of the high-resolution image can be converted to 1/4.
The high-resolution image (hereinafter referred to as a reduced image) whose resolution has been reduced in this way is transmitted to the feature amount extraction unit 35.

Step S10: FIG. 6 is a diagram for explaining image shift detection based on comparison of projected edges. Hereinafter, the process of detecting the image shift will be described with reference to FIG.
First, the feature extraction unit 35 uses the filter for vertical edge detection shown in the following formula (see FIG. 6 [A]), extracts a vertical edge components g _y from the reduced image f (x, y).
g _y (x, y) =-f (x, y-1) + f (x, y + 1)
Furthermore, the feature quantity extraction unit 35 extracts a horizontal edge component g _x from the reduced image f (x, y) using a horizontal edge extraction filter (see FIG. 6B) represented by the following equation.
g _x (x, y) =-f (x-1, y) + f (x + 1, y)
In order to reduce the influence of noise, the feature amount extraction unit 35 preferably replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.
Next, as shown in FIG. 6A, the feature quantity extraction unit 35 calculates the vertical projection waveform by cumulatively adding the vertical edge component _gy to the horizontal unit.
Further, as shown in FIG. 6B, the feature quantity extraction unit 35 calculates a horizontal projection waveform by accumulating the horizontal edge component g _x in units of vertical columns.
On the other hand, the feature amount extraction unit 39 captures a plurality of low resolution images from the memory 17. The feature amount extraction unit 39 performs the same processing as the feature amount extraction unit 35 on each low-resolution image, and obtains a vertical projection waveform and a horizontal projection waveform, respectively.
Here, as shown in FIG. 6A, the coarse detection unit 36 takes a difference while shifting the vertical projection waveform in the central area of the reduced image and the vertical projection waveform in the central area of the low-resolution image, and the difference The waveform shift that minimizes the absolute value sum of is detected. This waveform shift corresponds to the vertical position shift between the reduced image and the low resolution image.
Further, as shown in FIG. 6B, the coarse detection unit 36 calculates a difference while shifting the horizontal projection waveform in the central area of the reduced image and the horizontal projection waveform in the central area of the low-resolution image, and calculates the absolute difference. Detects the waveform shift that minimizes the sum of values. This waveform shift corresponds to a horizontal position shift between the reduced image and the low resolution image.
In this way, the coarse detection unit 36 obtains positional deviations (coarse detection results) of the plurality of low resolution images using the reduced image as a position reference, and outputs the obtained positional deviations to the precise detection unit 38.

Step S11: The feature amount extraction unit 33 takes in the high-resolution image that has been gain-corrected, and extracts the vertical edge component g _y and the horizontal edge component g _x using an edge extraction filter.
Note that the edge extraction filter here is preferably switched as follows according to the low-resolution image readout method.
・ When a low-resolution image is created by pixel addition or pixel averaging
g _y (x, y) = [-f (x, y-4) -f (x, y-3) -f (x, y-2) -f (x, y-1) + f (x, y + 4) + f (x, y + 5) + f (x, y + 6) + f (x, y + 7)] / 4
g _x (x, y) = [-f (x-4, y) -f (x-3, y) -f (x-2, y) -f (x-1, y) + f (x + 4, y) + f (x + 5, y) + f (x + 6, y) + f (x + 7, y)] / 4
・ When a low-resolution image is created by pixel decimation
g _y (x, y) =-f (x, y-4) + f (x, y + 4)
g _x (x, y) =-f (x-4, y) + f (x + 4, y)
In order to reduce the influence of noise, the feature amount extraction unit 33 preferably replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.
Next, the feature amount extraction unit 33 calculates the vertical projection waveform by cumulatively adding the vertical edge component _gy to the horizontal unit. Further, the feature amount extraction unit 33 calculates a horizontal projection waveform by cumulatively adding the horizontal edge component g _x in units of vertical columns.
The phase division unit 37 subsamples the vertical projection waveform of the high resolution image every four pixels. At this time, the phase splitting unit 37 generates four types of sampling information whose phases are shifted from each other as shown in FIG. 7 by shifting the sub-sampling phase.
Similarly, the phase division unit 37 subsamples the horizontal projection waveform of the high resolution image every four pixels. At this time, by shifting the sub-sampling phase, four types of sampling information whose phases are shifted from each other are generated.

Step S12: The precision detection unit 38 shifts the sampling information of the vertical projection waveform obtained from the high resolution image and the vertical projection waveform of the low resolution image from the coarse detection result of the positional deviation by the coarse detection unit 36 as a starting point. The difference is taken, and the waveform shift that minimizes the sum of the absolute values of the differences is detected.
The precision detection unit 38 performs the detection of the waveform deviation for each of the four types of sampling information, thereby obtaining the waveform deviation that most closely matches the feature of the pattern (in this case, the waveform). This waveform shift corresponds to a horizontal position shift in units smaller than the pixel interval of the low-resolution image.
Further, the precision detection unit 38 similarly detects the positional deviation in the vertical direction in units smaller than the pixel interval of the low resolution image (for example, the unit of the pixel interval of the high resolution image).
In this manner, the precision detection unit 38 obtains positional deviations (precision detection results) of the plurality of low resolution images using the high resolution image as a position reference, and outputs the positional deviation to the image composition unit 34.

Step S13: The image composition unit 34 performs high-pass filter processing on the low-resolution image, calculates the absolute value sum of the filter results, and obtains the amount of the high-frequency component. A plurality of low resolution images are classified according to the obtained amount of the high frequency component.
In addition, as shown in FIG. 8, it is good also as an image used for a synthesis | combination by giving an order to an image in order with many high frequency components, and classifying the image from a high rank to a predetermined order. Alternatively, an image in which the amount of the high frequency component exceeds a predetermined threshold may be selected and used as an image used for composition.
By such a selection process, an image whose high frequency component does not meet the standard, such as a large blur amount, can be appropriately excluded from the synthesis process. As a result, it is possible to reliably improve the image quality after synthesis.

Step S14: The image composition unit 34 performs high-pass filter processing on the high-resolution image, and divides the high-resolution image into an edge region and a flat region from the filter result.

Step S15: In the edge area obtained in step S14, the composition ratio is reduced as the image has fewer high frequency components. On the other hand, in the flat region obtained in step S14, the composition ratio of an image with a small high-frequency component is increased from that of the edge portion. In addition, about the area | region which is neither an edge area | region nor a flat area, it sets to the synthetic | combination ratio of the middle of both area | regions.

Step S16: The image composition unit 34 enlarges (4 × 4 times) each of the plurality of low resolution images. At this time, the image composition unit 34 does not perform pixel interpolation, and obtains an enlarged image with a large pixel interval.
Next, the image composition unit 34 shifts the pixel position of the enlarged image of the low-resolution image based on the precise detection result of the positional deviation obtained by the precise detection unit 38, and performs mapping (rearrangement) as shown in FIG. )I do.
In this way, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.

Step S17: The rearranged image that has been subjected to the mapping process includes pixels that are not filled with gaps, pixels that deviate from normal pixel positions, and overlapping pixels.
The image composition unit 34 picks up neighboring pixels for each normal pixel position of the rearranged image. The image composition unit 34 performs a weighted average on the signal components of these neighboring pixels according to the composition ratio set in step S15. The image composition unit 34 uses the weighted average value as a signal component (luminance / color difference component, etc.) at the normal pixel position. By such a composition process, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.
Instead of the above weighted average, the rearranged images may be synthesized by median calculation of signal components of neighboring pixels.

Step S18: The image composition unit 34 performs the filtering process on the luminance component of the high resolution image as follows.
First, the image composition unit 34 extracts a luminance component from the high-resolution image after gain correction, and performs filter processing that combines median processing and Gaussian filtering. For example, the Gaussian filter is implemented by setting the filter size to 3 × 3 pixels and extracting the central three values from nine pixels within the filter size. By this processing, it is possible to reduce in advance the noise included in the underexposed luminance component.

Step S19: Next, a composition ratio is set between the luminance component A of the high-resolution image after the filtering process and the luminance component B of the rearranged image.

Here, it is preferable to set the composition ratio according to the following rules.
(1) The portion where the signal difference between the luminance component A of the high-resolution image and the luminance component B of the rearranged image is larger reduces the composition ratio of the rearranged image locally.
(2) The portion where the local signal change of the luminance component A of the high-resolution image is larger reduces the composition ratio of the rearranged image locally.
(3) The synthesis ratio of the luminance component B of the rearranged image is generally increased as the time interval between the high resolution image and the low resolution image is smaller.
(4) The composition ratio of the luminance component B of the rearranged image is generally increased as the positional deviation between the high resolution image and the low resolution image is smaller.

上式中では、２段階の処理がなされる。すなわち、再配置画像の輝度成分Ｂに対して、例えばｍ＝５程度のガウシアンフィルタ（平滑化）が施される。次に、この輝度成分Ｂの平滑化結果が、高解像度画像の輝度成分Ａに画素単位に加重合成される。
このとき、輝度成分Ａ，Ｂの信号差が大きいほど、輝度成分Ｂの合成比率Ｇ(i,j,p)が下がる。その結果、高解像度画像と再配置画像との絵柄が大きく違う箇所については、高解像度画像の輝度成分Ａが局所的に優先される。 It should be noted that it is preferable to use a Gaussian filter of the following formula for calculating the specific luminance component g (i, j) of the composite image.

In the above equation, two stages of processing are performed. That is, for example, a Gaussian filter (smoothing) of about m = 5 is applied to the luminance component B of the rearranged image. Next, the smoothing result of the luminance component B is weighted and synthesized on the luminance component A of the high-resolution image in units of pixels.
At this time, the larger the signal difference between the luminance components A and B, the lower the synthesis ratio G (i, j, p) of the luminance component B. As a result, the luminance component A of the high-resolution image is locally prioritized at locations where the patterns of the high-resolution image and the rearranged image are greatly different.

Note that the composition ratio G (i, j, p) is preferably not changed depending on the reference distance of the pixel. As a result, the luminance component B within the filter size m can be reflected in the luminance component A even if the reference distance is long. As a result, it is possible to tolerate misalignment of the rearranged image to some extent.
Also, σ in the above formula is a numerical value for adjusting the value of the composition ratio and the amount of decrease. This σ is preferably set to be smaller locally as the variance value of 3 × 3 pixels around the luminance component A is larger. Thus, by locally reducing σ, the composition ratio of the rearranged image is locally reduced near the edge of the high-resolution image. As a result, the influence (smoothing or disturbance) on the image structure such as an edge can be suppressed.

  In addition, it is preferable that σ in the above equation is set to be larger as the time interval and / or the positional deviation between the high resolution image and the low resolution image is smaller. By varying σ in this manner, the composition ratio of the rearranged images is generally increased as it is estimated that the patterns of the high resolution image and the low resolution image are closer. As a result, information on a rearranged image (low-resolution image) with a similar pattern can be preferentially reflected on the composite image.

Step S20: The color difference component of the rearranged image generated in step S17 and the luminance component of the composite image generated in step S19 are combined to generate a high resolution color image reflecting the information of the low resolution image.
This color image is stored and recorded in the recording medium 22a via the image compression unit 24, the recording unit 22, and the like.

[Effects of this embodiment, etc.]
In the present embodiment, the through image (low resolution image) discarded after the monitor display is effectively utilized for improving the image quality of the still image (high resolution image). By effectively using this through image, the imaging performance of the electronic camera 10 can be enhanced.

  In this embodiment, a low-resolution image with a short imaging time interval is synthesized. Therefore, the difference in the design between the images is originally small, and a good image composition result can be obtained.

  Further, in the present embodiment, a plurality of low resolution images are aligned using a high resolution image having a high pixel density as a position reference. Therefore, the positioning accuracy of the pattern of the low resolution image is high, and a better image composition result can be obtained.

  In the present embodiment, a plurality of pieces of sampling information whose sampling phases are shifted from each other are generated from a high-resolution image. By performing position shift detection between the sampling information and the low resolution image, the position shift can be detected in units smaller than the pixel interval of the low resolution image. Therefore, it is possible to further improve the alignment accuracy of the pattern of the low-resolution image, and a better image composition result can be obtained.

  Furthermore, in this embodiment, the luminance components (signal components with high visual sensitivity) of a plurality of low resolution images can be aligned and mapped to obtain a luminance component having a higher resolution than that of the low resolution image.

  In the present embodiment, it is also possible to obtain a color difference component having a higher resolution than that of the low resolution image by aligning and mapping the color difference components (signal components having low visual sensitivity) of the plurality of low resolution images.

  Furthermore, in this embodiment, the high-resolution luminance component (rearranged image) created by the alignment and the luminance component of the original high-resolution image are weighted and synthesized. By this weighted synthesis, non-correlated luminance noise between both images can be suppressed. As a result, it is possible to improve the S / N of a high-resolution image in an underexposed state (see step S7).

  In the present embodiment, when the signal difference between the high-resolution image and the rearranged image is significantly large, the composite ratio of the rearranged image is locally reduced. As a result, when a misalignment occurs in the rearranged image, the signal difference between the two images is significantly increased, and the composite ratio of the rearranged image is reduced. Therefore, this misalignment is not so much reflected in the image composition result.

  Further, in the present embodiment, the composition ratio of the rearranged image is locally reduced at a location where the local signal change of the high resolution image is large. Therefore, priority is given to the image structure of a high-resolution image in the edge part etc. of an image. As a result, it is possible to avoid problems such as the edge becoming multi-line.

  In the present embodiment, the overall ratio of the rearranged images is increased as the time interval and / or positional deviation between the high resolution image and the low resolution image is smaller. When such a condition is satisfied, it can be estimated that the patterns of the high resolution image and the low resolution image are very close. Therefore, by increasing the composition ratio of the rearranged images, it is possible to further increase the image S / N without fear of pattern failure.

  In the present embodiment, as shown in FIG. 8, a plurality of low resolution images are ranked according to the amount of the high frequency component of the spatial frequency. At this time, an image with few high-frequency components is excluded from an image used for composition because blurring is conspicuous.

  Furthermore, even if the images used for the composition are selected, the composition ratio of the alignment composition at the edge portion of the pattern is adjusted to be small for an image that has been determined to have relatively few high-frequency components. As a result, alignment is performed with priority given to edge portions in an image with little camera shake or subject blur. As a result, adverse effects such as a dull edge of the rearranged image are suppressed.

  Further, in the present embodiment, for an image determined to have few high-frequency components, the synthesis ratio of the flat portion is increased compared to the edge portion. As a result, in the flat portion, a plurality of images are positioned and combined more evenly, and the image S / N after the positioning and combining can be increased.

<< Additional items of embodiment >>
In addition, this inventor has disclosed the procedure which further speeds up position shift detection in Japanese Patent Application No. 2005-345715. According to this procedure, the position shift detection of this embodiment may be speeded up.

  In step S10, absolute positional deviation is roughly detected between the reduced image of the high resolution image and the low resolution image. However, the present invention is not limited to this. A relative positional shift may be roughly detected between a plurality of low-resolution images. The precise detection unit 38 can roughly know the remaining absolute positional deviation based on the relative coarse detection result and the precise detection result of at least one positional deviation. The precision detection unit 38 can quickly detect a precise positional deviation by performing a positional deviation search using the absolute rough detection result as a starting point.

  In the above-described embodiment, the positional deviation of the image is detected by comparing the projected waveforms. However, the present invention is not limited to this. For example, the positional deviation may be detected by spatial comparison of the pixel arrangement of both images.

  Further, in the above-described embodiment, the case where the image processing apparatus is mounted on the electronic camera has been described. However, the present invention is not limited to this. An image processing program in which the above-described image processing is program-coded may be created. By executing this image processing program on a computer, it is possible to create a high-quality and high-resolution composition result on the computer by effectively utilizing the information of the low-resolution image.

  In the above-described embodiment, a plurality of low resolution images are acquired before capturing a high resolution image. However, the present invention is not limited to this. A plurality of low-resolution images may be acquired after capturing a high-resolution image. Further, a plurality of low resolution images may be acquired before and after the high resolution image is captured.

  In the above-described embodiment, the color difference component is obtained from the rearranged image, and the luminance component is obtained from the composite image. However, the present invention is not limited to this. The luminance component and the color difference component may be obtained from the rearranged image. Further, the luminance component and the color difference component may be obtained from the composite image.

  In the above-described embodiment, the case where the image signal of the luminance color difference is handled has been described. However, the present invention is not limited to this. In general, the present invention may be applied to cases where RGB, Lab and other image signals are handled. In the case of this RGB image signal, a signal component with high visual sensitivity is a G component, and the remaining signal components are RB components. In the case of a Lab image signal, a signal component with high visual sensitivity is an L component, and the remaining signal components are ab components.

  In the above-described embodiment, the positional deviation of the pattern is detected by image processing. However, the present invention is not limited to this. For example, an acceleration sensor or the like may be mounted on the camera side to determine the movement (vibration) of the shooting area of the camera, and the positional deviation of the pattern of the multiple images may be detected from the movement (vibration) of the shooting area.

  In the above-described embodiment, a case has been described in which a high-resolution image is used as a reference for alignment, and a low-resolution image is aligned and combined with the high-resolution image. However, the present invention is not limited to this. The present invention may be applied to a case where a plurality of images having the same resolution are synthesized. In this case, one of the plurality of images may be used as a reference for alignment, and the composition ratio of other images may be locally lowered at a location where the local signal change of the image is large.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiments are merely examples in all respects and should not be interpreted in a limited manner. The scope of the present invention is shown by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

  As described above, the present invention is a technique that can be used for an image processing apparatus or the like.

1 is a block diagram showing an electronic camera 10 (including an image processing device 25). 3 is a block diagram schematically showing a configuration of an image processing device 25. FIG. 3 is a flowchart for explaining the operation of the electronic camera 10. 3 is a flowchart for explaining the operation of the electronic camera 10. It is a figure explaining a low resolution image and a high resolution image. It is a figure which shows the mode of the image shift detection by the comparison of a projection edge. It is a figure which shows the mode of subsampling. It is a figure which shows adjustment operation | movement of a synthetic | combination ratio. It is a figure explaining the production | generation operation | movement of a rearranged image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Electronic camera, 11 ... Imaging device, 12 ... Shooting lens, 14 ... Imaging control part, 15 ... Signal processing part, 16 ... A / D conversion part, 17 ... Memory, 18 ... Bus, 19 ... Microprocessor, 19a ... Operation unit, 22 ... recording unit, 22a ... recording medium, 24 ... image compression unit, 25 ... image processing device, 30 ... monitor display unit, 31 ... gain correction unit, 32 ... reduced image generation unit, 33 ... feature amount extraction unit , 34... Image composition unit, 35... Feature amount extraction unit, 36... Coarse detection unit, 37.

Claims (9)

An image input unit for capturing a plurality of first resolution images obtained by photographing the same subject and at least one second resolution image obtained by photographing at least one first resolution image obtained by photographing the same subject;
A positional displacement detector for detecting a positional deviation between the second resolution image and the first Zodo image respectively,
For images taken in a short exposure time than the exposure time of the second resolution image is determined by the exposure calculation, based on the positional deviation, have rows aligned against the plurality of first resolution image, wherein A rearranged image having a larger image size than the first resolution image is generated by rearranging the signal components of the plurality of first resolution images, and a composite image is generated based on the second resolution image and the rearranged image An image processing apparatus comprising: an image synthesizing unit. The image processing apparatus according to claim 1.
The positional deviation detection unit is
A phase dividing unit for sub-sampling the second resolution image or the feature thereof by shifting the phase and generating a plurality of sampling information whose phases are shifted from each other;
By detecting the most matching the positional deviation between the first resolution image and the plurality of sampling information, and precision detector than the pixel spacing of said first resolution image for detecting the positional deviation of the small unit An image processing apparatus comprising: The image processing apparatus according to claim 1.
The positional deviation detection unit is
A coarse detection unit for detecting a positional deviation between said plurality of first resolution image,
An image processing apparatus comprising: a precision detection unit that detects a positional shift between the second resolution image and the first resolution image, starting from the positional shift detected by the rough detection unit. The image processing apparatus according to any one of claims 1 to 3,
The image synthesizing unit generates the synthesized image by locally lowering the synthesis ratio of the rearranged image as the signal difference between the second resolution image and the rearranged image is larger. Image processing device. It claims 1 image processing apparatus according to any one of claims 4,
The image processing device generates the composite image by locally lowering the composite ratio of the rearranged image as the local signal change of the second resolution image is larger. . It claims 1 image processing apparatus according to any one of claims 5,
The image synthesizing section, the higher the second time interval and / or the misalignment between resolution image and the first resolution image is small, on the mixing ratio of the rearranged images general Gaité, the composite image An image processing apparatus characterized by generating. It claims 1 image processing apparatus according to any one of claims 6,
The image synthesizing unit generates the synthesized image for a signal component with high visual sensitivity constituting a color image, and generates the rearranged image for the remaining signal component with low visual sensitivity. . An image processing apparatus according to any one of claims 1 to 7,
An electronic camera including an imaging unit that captures a subject with at least two types of resolutions,
The second resolution image processed by the image processing device is a high-resolution still image captured by the imaging unit,
Said first resolution image, electronic camera, wherein the Ru low-resolution image der of being captured by the imaging unit after shooting before and / or imaging of the still image by the image processing apparatus to process. The image processing program for a computer to function as an image processing apparatus according to any one of claims 1 to 7.

Images