JP5012805B2 - Image processing apparatus, electronic camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an electronic camera, and an image processing program.

2. Description of the Related Art Conventionally, there has been known a technique for correcting blur of a captured image by performing multiple divided exposures using an electronic camera and aligning and synthesizing the obtained images (for example, the following) Patent Document 1).
JP 2002-107787 A (Claim 1 etc.)

  By the way, in the prior art, there is a concern that the edge portion of the pattern may become dull after synthesis due to individual differences between a plurality of images to be aligned and synthesized.

  Therefore, an object of the present invention is to provide a technique for improving the image quality after composition in alignment composition of a plurality of images.

An image processing apparatus as an example of the present invention includes an image input unit, a positional deviation detection unit, and an image composition unit. The image input unit captures a plurality of images taken of the same subject. The positional deviation detection unit detects a positional deviation of the pattern between the plurality of images. The image composition unit performs composition alignment on a plurality of images based on the positional deviation. In such a configuration, the image composition unit determines the high frequency component of the spatial frequency for each of the plurality of images. At this time, among the plurality of determined images, an image determined to have a lower high-frequency component has a smaller composition ratio of the alignment composition at the edge portion of the pattern.

  In the present invention, a plurality of images are distinguished by high frequency components of spatial frequency. That is, for an image that is determined to have few high-frequency components, the composition ratio of the alignment composition is lowered at least at the edge portion of the pattern. As a result, in the edge portion of the pattern, alignment synthesis is mainly performed on an image with many high-frequency components. As a result, it is possible to suppress adverse effects such as a dull edge after synthesis.

Block diagram showing the electronic camera 10 (including the image processing device 25) The block diagram which showed typically the structure of the image processing apparatus 25 A flow chart explaining operation of electronic camera 10 in a 1st embodiment. A flow chart explaining operation of electronic camera 10 in a 1st embodiment. Diagram explaining low-resolution image and high-resolution image Diagram showing how image misalignment is detected by comparing projected edges Diagram showing sub-sampling Diagram showing adjustment of composite ratio The figure explaining the production | generation operation | movement of a rearranged image A flow chart explaining operation of electronic camera 10 in a 2nd embodiment. Schematic diagram showing image composition processing in the second embodiment

<Description of First Embodiment>
[Description of electronic camera configuration]
FIG. 1 is a block diagram showing an electronic camera 10 (including an image processing device 25) according to the present embodiment.

  In FIG. 1, a photographing lens 12 is attached to the electronic camera 10. In the image space of the photographic lens 12, the imaging surface of the imaging element 11 is arranged. The imaging element 11 is controlled by the imaging control unit 14. The image pickup device 11 has a mode for reading a high-resolution image and a mode for reading a low-resolution image by performing pixel thinning and pixel addition inside the device. The image signal output from the image sensor 11 is processed via the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17.

  This memory 17 is connected to a bus 18. The bus 18 is also connected with an imaging control unit 14, a microprocessor 19, a recording unit 22, an image compression unit 24, a monitor display unit 30, an image processing device 25, and the like. The microprocessor 19 is connected to an operation unit 19a such as a release button. A recording medium 22a is detachably attached to the recording unit 22.

[Description of Image Processing Device 25]
FIG. 2 is a block diagram schematically showing the configuration of the image processing device 25.

  The high resolution image read from the memory 17 is supplied to the reduced image generation unit 32, the feature amount extraction unit 33, and the image composition unit 34 via the gain correction unit 31, respectively. The output data of the reduced image generation unit 32 is supplied to the rough detection unit 36 via the feature amount extraction unit 35. The output data of the feature quantity extraction unit 33 is supplied to the precision detection unit 38 via the phase division unit 37.

  On the other hand, the plurality of low resolution images read from the memory 17 are supplied to the feature amount extraction unit 39 and the image composition unit 34, respectively. The output data of the feature quantity extraction unit 39 is supplied to the rough detection unit 36 and the precise detection unit 38, respectively.

  The positional deviation roughly detected by the rough detection unit 36 is supplied to the precision detection unit 38. The positional deviation detected with high precision by the precision detection unit 38 is supplied to the image composition unit 34. The image synthesis unit 34 synthesizes a plurality of low resolution images and high resolution images based on the detection result of the positional deviation.

[Description of operation]
3 and 4 are flowcharts for explaining the operation of the electronic camera 10. Hereinafter, this operation will be described along the step numbers shown in FIG.

  Step S1: When the main power supply of the electronic camera 10 is turned on, the microprocessor 19 instructs the imaging control unit 14 to read out a low resolution image. The imaging control unit 14 drives the imaging device 11 in a low resolution readout mode, and sequentially reads out low resolution images at, for example, 30 frames / second as shown in FIG.

  Step S2: The low resolution image read from the image sensor 11 is processed through the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17. The microprocessor 19 sequentially deletes the low resolution images exceeding the predetermined number of frames in the memory 17 from the oldest one.

  The predetermined number of frames here corresponds to the number of frames of a low-resolution image used for composition of a rearranged image described later, and is preferably set to be equal to or greater than (number of pixels of a high-resolution image / number of pixels of a low-resolution image). .

  For example, when the number of vertical and horizontal pixels of the low-resolution image is 1/4 of the number of vertical and horizontal pixels of the high-resolution image, the predetermined number of frames is preferably set to 4 × 4 = 16 frames or more.

  Step S3: The monitor display unit 30 displays a low resolution image (through image) on the monitor screen. On the other hand, the microprocessor 19 calculates exposure based on the photometric result of a photometric unit (not shown) and the brightness of the low resolution image, and determines the exposure time of the high resolution image.

  Step S4: Here, the microprocessor 19 determines whether or not the release button has been fully pressed by the user.

  When the release button is fully pressed, the microprocessor 19 proceeds to step S5. On the other hand, if the release button has not been fully pressed, the microprocessor 19 returns the operation to step S1.

  Step S5: Here, the microprocessor 19 determines whether or not the exposure time setting of the high-resolution image determined in step S3 is equal to or less than an allowable upper limit where blur is not noticeable. For example, this allowable upper limit is set to about 1 / (35 mm equivalent focal length of the photographing lens 12) seconds.

  If the exposure time setting is less than or equal to the allowable upper limit, the microprocessor 19 proceeds to step S6. On the other hand, when the exposure time setting exceeds the allowable upper limit, the microprocessor 19 shifts the operation to step S7.

  Step S6: The imaging control unit 14 performs shutter control of the imaging device 11 according to the set exposure time. Subsequently, the imaging control unit 14 drives the imaging device 11 in a high-resolution readout mode and reads out a high-resolution image. This high-resolution image (still image) is recorded and stored in the recording medium 22a after image processing and image compression as in the prior art. With this operation, the electronic camera 10 completes the photographing operation.

  Step S7: On the other hand, if it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 limits the exposure time to an allowable upper limit that does not cause blurring.

  The imaging control unit 14 controls the shutter of the imaging device 11 according to the exposure time limited to a short time. In this state, the imaging control unit 14 drives the imaging device 11 in a high resolution readout mode and reads out a high resolution image. This high-resolution image is an image that has a low signal level due to insufficient exposure but is less likely to blur. This high resolution image is temporarily recorded in the memory 17.

  Step S <b> 8: The gain correction unit 31 in the image processing device 25 captures a high resolution image from the memory 17. The gain correction unit 31 adjusts the gain of this high resolution image to match the signal level of the low resolution image.

  Step S9: The reduced image generating unit 32 converts the resolution of the high-resolution image after gain adjustment and matches the number of pixels of the low-resolution image.

  For example, by extracting an average value of 4 × 4 pixels, the number of vertical and horizontal pixels of the high-resolution image can be converted to 1/4.

  The high-resolution image (hereinafter referred to as a reduced image) whose resolution has been reduced in this way is transmitted to the feature amount extraction unit 35.

  Step S10: FIG. 6 is a diagram for explaining image shift detection based on comparison of projected edges. Hereinafter, the process of detecting the image shift will be described with reference to FIG.

First, the feature extraction unit 35 uses the filter for vertical edge detection shown in the following formula (see FIG. 6 [A]), extracts a vertical edge components g _y from the reduced image f (x, y).
g _y (x, y) =-f (x, y-1) + f (x, y + 1)
Furthermore, the feature quantity extraction unit 35 extracts a horizontal edge component g _x from the reduced image f (x, y) using a horizontal edge extraction filter (see FIG. 6B) represented by the following equation.
g _x (x, y) =-f (x-1, y) + f (x + 1, y)
In order to reduce the influence of noise, the feature amount extraction unit 35 preferably replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.

Next, as shown in FIG. 6A, the feature quantity extraction unit 35 calculates the vertical projection waveform by cumulatively adding the vertical edge component _gy to the horizontal unit.

Further, as shown in FIG. 6B, the feature quantity extraction unit 35 calculates a horizontal projection waveform by accumulating the horizontal edge component g _x in units of vertical columns.

  On the other hand, the feature amount extraction unit 39 captures a plurality of low resolution images from the memory 17. The feature amount extraction unit 39 performs the same processing as the feature amount extraction unit 35 on each low-resolution image, and obtains a vertical projection waveform and a horizontal projection waveform, respectively.

  Here, as shown in FIG. 6A, the coarse detection unit 36 takes a difference while shifting the vertical projection waveform in the central area of the reduced image and the vertical projection waveform in the central area of the low-resolution image, and the difference The waveform shift that minimizes the absolute value sum of is detected. This waveform shift corresponds to the vertical position shift between the reduced image and the low resolution image.

  Further, as shown in FIG. 6B, the coarse detection unit 36 calculates a difference while shifting the horizontal projection waveform in the central area of the reduced image and the horizontal projection waveform in the central area of the low-resolution image, and calculates the absolute difference. Detects the waveform shift that minimizes the sum of values. This waveform shift corresponds to a horizontal position shift between the reduced image and the low resolution image.

  In this way, the coarse detection unit 36 obtains positional deviations (coarse detection results) of the plurality of low resolution images using the reduced image as a position reference, and outputs the obtained positional deviations to the precise detection unit 38.

Step S11: The feature amount extraction unit 33 takes in a high-resolution image that has been gain-corrected, and extracts a vertical edge component g _y and a horizontal edge component g _x using an edge extraction filter.

Note that the edge extraction filter here is preferably switched as follows according to the low-resolution image readout method.
・ When a low-resolution image is created by pixel addition or pixel averaging
g _y (x, y) = [-f (x, y-4) -f (x, y-3) -f (x, y-2) -f (x, y-1) + f (x, y + 4) + f (x, y + 5) + f (x, y + 6) + f (x, y + 7)] / 4
g _x (x, y) = [-f (x-4, y) -f (x-3, y) -f (x-2, y) -f (x-1, y) + f (x + 4, y) + f (x + 5, y) + f (x + 6, y) + f (x + 7, y)] / 4
・ When a low-resolution image is created by pixel decimation
g _y (x, y) =-f (x, y-4) + f (x, y + 4)
g _x (x, y) =-f (x-4, y) + f (x + 4, y)
In order to reduce the influence of noise, the feature amount extraction unit 33 preferably replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.

Next, the feature amount extraction unit 33 calculates the vertical projection waveform by cumulatively adding the vertical edge component _gy to the horizontal unit. Further, the feature amount extraction unit 33 calculates a horizontal projection waveform by cumulatively adding the horizontal edge component g _x in units of vertical columns.

  The phase division unit 37 subsamples the vertical projection waveform of the high resolution image every four pixels. At this time, the phase splitting unit 37 generates four types of sampling information whose phases are shifted from each other as shown in FIG. 7 by shifting the sub-sampling phase.

  Similarly, the phase division unit 37 subsamples the horizontal projection waveform of the high resolution image every four pixels. At this time, by shifting the sub-sampling phase, four types of sampling information whose phases are shifted from each other are generated.

  Step S12: The precision detection unit 38 shifts the sampling information of the vertical projection waveform obtained from the high resolution image and the vertical projection waveform of the low resolution image from the coarse detection result of the positional deviation by the coarse detection unit 36 as a starting point. The difference is taken, and the waveform shift that minimizes the sum of the absolute values of the differences is detected.

  The precision detection unit 38 performs the detection of the waveform deviation for each of the four types of sampling information, thereby obtaining the waveform deviation that most closely matches the feature of the pattern (in this case, the waveform). This waveform shift corresponds to a horizontal position shift in units smaller than the pixel interval of the low-resolution image.

  Further, the precision detection unit 38 similarly detects the positional deviation in the vertical direction in units smaller than the pixel interval of the low resolution image (for example, the unit of the pixel interval of the high resolution image).

  In this manner, the precision detection unit 38 obtains positional deviations (precision detection results) of the plurality of low resolution images using the high resolution image as a position reference, and outputs the positional deviation to the image composition unit 34.

  Step S13: The image composition unit 34 performs a high-pass filter process on the low-resolution image, calculates the absolute value sum of the filter results, and obtains the amount of the high-frequency component. A plurality of low resolution images are classified according to the obtained amount of the high frequency component.

  In addition, as shown in FIG. 8, it is good also as an image used for a synthesis | combination by giving an order to an image in order with many high frequency components, and classifying the image from a high rank to a predetermined order. Alternatively, an image in which the amount of the high frequency component exceeds a predetermined threshold may be selected and used as an image used for composition.

  By such a selection process, an image whose high frequency component does not meet the standard, such as a large blur amount, can be appropriately excluded from the synthesis process. As a result, it is possible to reliably improve the image quality after synthesis.

  Step S14: The image composition unit 34 performs high-pass filter processing on the high-resolution image, and divides the high-resolution image into an edge region and a flat region from the filter result.

  Step S15: In the edge area obtained in step S14, the composition ratio is reduced as the image has fewer high frequency components. On the other hand, in the flat region obtained in step S14, the composition ratio of an image with a small high-frequency component is increased from that of the edge portion. In addition, about the area | region which is neither an edge area | region nor a flat area, it sets to the synthetic | combination ratio of the middle of both area | regions.

  Step S16: The image composition unit 34 enlarges (4 × 4 times) each of the plurality of low resolution images. At this time, the image composition unit 34 does not perform pixel interpolation, and obtains an enlarged image with a large pixel interval.

  Next, the image composition unit 34 shifts the pixel position of the enlarged image of the low-resolution image based on the precise detection result of the positional deviation obtained by the precise detection unit 38, and performs mapping (rearrangement) as shown in FIG. )I do.

  In this way, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.

  Step S17: The rearranged image that has been subjected to the mapping process includes pixels that are not filled with gaps, pixels that deviate from normal pixel positions, and overlapping pixels.

  The image composition unit 34 picks up neighboring pixels for each normal pixel position of the rearranged image. The image composition unit 34 performs a weighted average on the signal components of these neighboring pixels according to the composition ratio set in step S15. The image composition unit 34 uses the weighted average value as a signal component (luminance / color difference component, etc.) at the normal pixel position. By such a composition process, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.

  Instead of the above weighted average, the rearranged images may be synthesized by median calculation of signal components of neighboring pixels.

  Step S18: The image composition unit 34 performs the filtering process on the luminance component of the high resolution image as follows.

  First, the image composition unit 34 extracts a luminance component from the high-resolution image after gain correction, and performs filter processing that combines median processing and Gaussian filtering. For example, the Gaussian filter is implemented by setting the filter size to 3 × 3 pixels and extracting the central three values from nine pixels within the filter size. By this processing, it is possible to reduce in advance the noise included in the underexposed luminance component.

  Step S19: Next, a composition ratio is set between the luminance component A of the high-resolution image after the filter processing and the luminance component B of the rearranged image.

Here, it is preferable to set the composition ratio according to the following rules.
(1) The portion where the signal difference between the luminance component A of the high-resolution image and the luminance component B of the rearranged image is larger reduces the composition ratio of the rearranged image locally.
(2) The portion where the local signal change of the luminance component A of the high-resolution image is larger reduces the composition ratio of the rearranged image locally.
(3) The synthesis ratio of the luminance component B of the rearranged image is generally increased as the time interval between the high resolution image and the low resolution image is smaller.
(4) The composition ratio of the luminance component B of the rearranged image is generally increased as the positional deviation between the high resolution image and the low resolution image is smaller.

  It should be noted that it is preferable to use a Gaussian filter of the following formula for calculating the specific luminance component g (i, j) of the composite image.

上式中では、２段階の処理がなされる。すなわち、再配置画像の輝度成分Ｂに対して、例えばｍ＝５程度のガウシアンフィルタ（平滑化）が施される。次に、この輝度成分Ｂの平滑化結果が、高解像度画像の輝度成分Ａに画素単位に加重合成される。

In the above equation, two stages of processing are performed. That is, for example, a Gaussian filter (smoothing) of about m = 5 is applied to the luminance component B of the rearranged image. Next, the smoothing result of the luminance component B is weighted and synthesized on the luminance component A of the high-resolution image in units of pixels.

  At this time, the larger the signal difference between the luminance components A and B, the lower the synthesis ratio G (i, j, p) of the luminance component B. As a result, the luminance component A of the high-resolution image is locally prioritized at locations where the patterns of the high-resolution image and the rearranged image are greatly different.

  Note that the composition ratio G (i, j, p) is preferably not changed depending on the reference distance of the pixel. As a result, the luminance component B within the filter size m can be reflected in the luminance component A even if the reference distance is long. As a result, it is possible to tolerate misalignment of the rearranged image to some extent.

  Also, σ in the above formula is a numerical value for adjusting the value of the composition ratio and the amount of decrease. This σ is preferably set to be smaller locally as the variance value of 3 × 3 pixels around the luminance component A is larger. Thus, by locally reducing σ, the composition ratio of the rearranged image is locally reduced near the edge of the high-resolution image. As a result, the influence (smoothing or disturbance) on the image structure such as an edge can be suppressed.

  In addition, it is preferable that σ in the above equation is set to be larger as the time interval and / or the positional deviation between the high resolution image and the low resolution image is smaller. By varying σ in this manner, the composition ratio of the rearranged images is generally increased as it is estimated that the patterns of the high resolution image and the low resolution image are closer. As a result, information on a rearranged image (low-resolution image) with a similar pattern can be preferentially reflected on the composite image.

  Step S20: The color difference component of the rearranged image generated in step S17 and the luminance component of the composite image generated in step S19 are combined to generate a high resolution color image reflecting the information of the low resolution image.

  This color image is stored and recorded in the recording medium 22a via the image compression unit 24, the recording unit 22, and the like.

[Effects of this embodiment, etc.]
In the present embodiment, the through image (low resolution image) discarded after the monitor display is effectively utilized for improving the image quality of the still image (high resolution image). By effectively using this through image, the imaging performance of the electronic camera 10 can be enhanced.

  In this embodiment, a low-resolution image with a short imaging time interval is synthesized. Therefore, the difference in the design between the images is originally small, and a good image composition result can be obtained.

  Further, in the present embodiment, a plurality of low resolution images are aligned using a high resolution image having a high pixel density as a position reference. Therefore, the positioning accuracy of the pattern of the low resolution image is high, and a better image composition result can be obtained.

  In the present embodiment, a plurality of pieces of sampling information whose sampling phases are shifted from each other are generated from a high-resolution image. By performing position shift detection between the sampling information and the low resolution image, the position shift can be detected in units smaller than the pixel interval of the low resolution image. Therefore, it is possible to further improve the alignment accuracy of the pattern of the low-resolution image, and a better image composition result can be obtained.

  Furthermore, in this embodiment, the luminance components (signal components with high visual sensitivity) of a plurality of low resolution images can be aligned and mapped to obtain a luminance component having a higher resolution than that of the low resolution image.

  In the present embodiment, it is also possible to obtain a color difference component having a higher resolution than that of the low resolution image by aligning and mapping the color difference components (signal components having low visual sensitivity) of the plurality of low resolution images.

  Furthermore, in this embodiment, the high-resolution luminance component (rearranged image) created by the alignment and the luminance component of the original high-resolution image are weighted and synthesized. By this weighted synthesis, non-correlated luminance noise between both images can be suppressed. As a result, it is possible to improve the S / N of a high-resolution image in an underexposed state (see step S7).

  In the present embodiment, when the signal difference between the high-resolution image and the rearranged image is significantly large, the composite ratio of the rearranged image is locally reduced. As a result, when a misalignment occurs in the rearranged image, the signal difference between the two images is significantly increased, and the composite ratio of the rearranged image is reduced. Therefore, this misalignment is not so much reflected in the image composition result.

  Further, in the present embodiment, the composition ratio of the rearranged image is locally reduced at a location where the local signal change of the high resolution image is large. Therefore, priority is given to the image structure of a high-resolution image in the edge part etc. of an image. As a result, it is possible to avoid problems such as the edge becoming multi-line.

  In the present embodiment, the overall ratio of the rearranged images is increased as the time interval and / or positional deviation between the high resolution image and the low resolution image is smaller. When such a condition is satisfied, it can be estimated that the patterns of the high resolution image and the low resolution image are very close. Therefore, by increasing the composition ratio of the rearranged images, it is possible to further increase the image S / N without fear of pattern failure.

  In the present embodiment, as shown in FIG. 8, a plurality of low resolution images are ranked according to the amount of the high frequency component of the spatial frequency. At this time, an image with few high-frequency components is excluded from an image used for composition because blurring is conspicuous.

  Furthermore, even if the images used for the composition are selected, the composition ratio of the alignment composition at the edge portion of the pattern is adjusted to be small for an image that has been determined to have relatively few high-frequency components. As a result, alignment is performed with priority given to edge portions in an image with little camera shake or subject blur. As a result, adverse effects such as a dull edge of the rearranged image are suppressed.

  Further, in the present embodiment, for an image determined to have few high-frequency components, the synthesis ratio of the flat portion is increased compared to the edge portion. As a result, in the flat portion, a plurality of images are positioned and combined more evenly, and the image S / N after the positioning and combining can be increased.

<Description of Second Embodiment>
FIG. 10 is a flowchart for explaining the operation of the electronic camera 10 in the second embodiment. FIG. 11 is a schematic diagram showing image composition processing in the second embodiment.

  Here, the second embodiment is a modification of the first embodiment, and the electronic camera combines a plurality of images having the same resolution. In addition, since the configuration of the electronic camera in the second embodiment is the same as that of the electronic camera in the first embodiment (FIG. 1), repeated description is omitted.

  Furthermore, S101 to S103 in FIG. 10 correspond to S1 to S3 in FIG. Hereinafter, the operation of the electronic camera will be described along the step numbers shown in FIG.

  Step S104: Here, the microprocessor 19 determines whether or not the exposure time setting of the high-resolution image determined in step S103 is equal to or less than an allowable upper limit where blur is not noticeable. The allowable upper limit is set in the same manner as in step S5.

  If the exposure time setting is less than the allowable upper limit, the microprocessor 19 proceeds to step S105. On the other hand, when the exposure time setting exceeds the allowable upper limit, the microprocessor 19 shifts the operation to step S106.

  Step S105: The imaging control unit 14 performs shutter control of the imaging device 11 according to the set exposure time, and images a high-resolution image (still image). In addition, since the operation | movement of this step respond | corresponds to said step S6, the overlapping description is abbreviate | omitted.

  Step S106: On the other hand, if it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 switches the operation state of the electronic camera to the continuous shooting mode. Then, the microprocessor 19 limits the exposure time of each frame in the continuous shooting mode to an allowable upper limit that does not cause blurring.

  The imaging control unit 14 in the continuous shooting mode drives the imaging device 11 in a high-resolution reading mode according to a short limited exposure time. As a result, a plurality of high-resolution images are continuously captured by the image sensor 11. Each high-resolution image is an image that has a low signal level due to underexposure but is less likely to be blurred. Each of the above high-resolution images is temporarily recorded in the memory 17.

  Here, in the continuous shooting mode described above, for example, the shooting operation is performed in the following manner (1) or (2).

  (1) The imaging control unit 14 starts continuous shooting of high-resolution images from the time when the mode is switched to the continuous shooting mode, and accumulates high-resolution images for a certain period in the memory 17. At this time, the microprocessor 19 deletes the high resolution images exceeding the predetermined number of frames in the memory 17 in order from the oldest one.

  When the microprocessor 19 receives a user instruction (such as pressing a release button) during continuous shooting, the microprocessor 19 designates an image corresponding to the timing of the instruction among the high-resolution images in the memory 17 as a reference image. . In addition, the microprocessor 19 designates a predetermined number of high-resolution images that move back and forth in the time axis direction with respect to the reference image as targets for the synthesis process described later. In the second embodiment, a high-resolution image designated as a target for synthesis processing is referred to as a synthesized image. The reference image and the combined image are held in the memory 17 until at least the combining process is completed.

  (2) The microprocessor 19 causes the imaging control unit 14 to start continuous shooting in response to a user's continuous shooting start instruction (such as pressing a release button). Then, the microprocessor 19 accepts a user's continuous shooting end instruction (such as release of the release button) or ends the continuous shooting when a predetermined number of frames have been shot.

  Thereafter, the microprocessor 19 displays the captured images on the monitor display unit 30 and causes the user to select a reference image. Then, the microprocessor 19 uses, as a recorded image, an image designated by the user among the high-resolution images taken continuously. Further, the microprocessor 19 sets an image other than the reference image among the high-resolution images taken continuously as a synthesized image.

  In the case of (2) above, the microprocessor 19 may designate an image corresponding to the timing of the continuous shooting start instruction (the first high-resolution image captured) as the reference image. In this case, it is possible to omit the step of allowing the user to specify the reference image.

  Step S107: The image processing device 25 detects a shift of the image of the combined image (S106) with respect to the reference image (S106) by the same process as step S10 of the first embodiment. Note that this image shift detection is performed for all the combined images.

  Step S108: The image composition unit 34 of the image processing device 25 performs high-pass filter processing on the composite image (S106), calculates the sum of absolute values of the filter results, and obtains the amount of the high-frequency component. Then, the image composition unit 34 classifies the plurality of composite images according to the obtained amount of the high frequency component.

  At this time, as in the first embodiment, the image composition unit 34 assigns the order to the images in the descending order of the high frequency components, selects the images from the top to the predetermined order, and actually You may narrow down the image used for composition. In addition, the image composition unit 34 may select an image in which the amount of the high frequency component exceeds a predetermined threshold, and narrow down images to be actually used for composition from among the composite images.

  Step S109: The image composition unit 34 performs high-pass filter processing on the reference image. Then, the reference image is divided into an edge area and a flat area from the filter result. Then, the image composition unit 34 reduces the composition ratio in the edge region of the reference image as the composite image with fewer high-frequency components. On the other hand, the image composition unit 34 raises the composition ratio of an image with less high frequency components in the flat region of the reference image than in the edge portion. In addition, about the area | region which is neither an edge area | region nor a flat area, it sets to the synthetic | combination ratio of the middle of both area | regions.

  Step S110: The image composition unit 34 performs mapping (rearrangement) by displacing the pixel positions of the composite image based on the position shift detection result obtained in S107, using the reference image as a position reference (FIG. 11). reference). In the second embodiment, the combined image after mapping is referred to as a rearranged image.

  In the second embodiment, the resolution (number of pixels) of the reference image and each synthesized image are the same. Therefore, as shown in FIG. 11, the rearranged image group is treated as a three-dimensional image having a time axis t in addition to the x-axis and y-axis indicating the position in the same image.

  Step S111: The image composition unit 34 performs composition processing of the reference image (S106) and the rearranged image (S110) to generate a final composite image.

  Here, the image synthesizing unit 34 sequentially synthesizes the rearranged images with the reference image, and repeats the synthesizing process for the number of the rearranged images. Note that the image synthesis unit 34 may synthesize a plurality of rearranged images in advance by median calculation or the like, and then synthesize the rearranged image after the synthesis and the reference image.

  Hereinafter, the synthesizing process in S111 will be specifically described. First, the image composition unit 34 reads the initial value of the composition ratio obtained in S109. Then, the image composition unit 34 further adjusts the composition ratio of the reference image and the rearranged image by paying attention to the gradation value of the reference image and the gradation value of the rearranged image.

  For example, when the difference in gradation value between the target pixel of the reference image and the target pixel of the rearranged image is equal to or greater than the threshold value (when the gradation difference is large), the image composition unit 34 Lower the synthesis ratio locally. On the other hand, when the difference in gradation value between the target pixel of the reference image and the target pixel of the rearranged image is less than the threshold value (when the gradation difference is small), the image composition unit 34 Increase the composition ratio locally. Thereby, the influence by the misalignment between images etc. is suppressed significantly.

  In addition, the image composition unit 34 adjusts the composition ratio according to a local signal change on the reference image. For example, the image synthesizing unit 34 locally lowers the synthesis ratio with the rearranged image at a location where the local signal change on the reference image is large. Further, the image composition unit 34 locally increases the composition ratio with the rearranged image at a portion where the local signal change on the reference image is small. As a result, it is possible to suppress adverse effects such as edges becoming multi-line on the synthesized image.

  Second, the image composition unit 34 adds and synthesizes corresponding pixels of the reference image and the rearranged image according to the composition ratio. In S111, the image composition unit 34 obtains the luminance component and the color difference component of each pixel by the composition processing. As a result, a high-resolution color image reflecting information of a plurality of combined images is generated.

  Step S112: The microprocessor 19 records the final composite image (generated in S111) on the recording medium 22a via the image compression unit 24, the recording unit 22, and the like. Above, description of FIG. 10 is complete | finished.

  The same effect as that of the first embodiment can be obtained by the configuration of the second embodiment. In particular, in the case of the second embodiment, since the resolution of the rearranged image is high and the amount of information is high, a higher-definition color image can be acquired.

<< Additional items of embodiment >>
(1) In the Japanese Patent Application No. 2005-345715, the present inventor discloses a procedure for further speeding up the position shift detection. According to this procedure, the position shift detection in each of the above embodiments may be speeded up.

  (2) In step S10, absolute positional deviation is roughly detected between the reduced image of the high resolution image and the low resolution image. However, the present invention is not limited to this. A relative positional shift may be roughly detected between a plurality of low-resolution images. The precise detection unit 38 can roughly know the remaining absolute positional deviation based on the relative coarse detection result and the precise detection result of at least one positional deviation. The precision detection unit 38 can quickly detect a precise positional deviation by performing a positional deviation search using the absolute rough detection result as a starting point.

  (3) In the first embodiment and the second embodiment described above, the positional deviation of the image is detected by comparing the projected waveforms. However, the present invention is not limited to this. For example, the positional deviation may be detected by spatial comparison of the pixel arrangement of both images.

  (4) In the first embodiment and the second embodiment described above, the case where the image processing apparatus is mounted on the electronic camera has been described. However, the present invention is not limited to this. An image processing program in which the above-described image processing is program-coded may be created. By executing this image processing program on a computer, it is possible to create a high-quality and high-resolution composition result on the computer by effectively utilizing the information of the low-resolution image. Similarly, when the image processing of the second embodiment is executed on a computer, a high-quality and high-resolution composite result can be created on the computer by effectively utilizing other images taken continuously.

  When the image processing according to the second embodiment is executed on a computer, the user can specify an arbitrary image as a reference image from the continuous shot images.

  (5) In the first embodiment described above, a plurality of low resolution images are acquired before capturing a high resolution image. However, the present invention is not limited to this. A plurality of low-resolution images may be acquired after capturing a high-resolution image. Further, a plurality of low resolution images may be acquired before and after the high resolution image is captured.

  (6) In the first embodiment described above, the color difference component is obtained from the rearranged image, and the luminance component is obtained from the synthesized image. However, the present invention is not limited to this. The luminance component and the color difference component may be obtained from the rearranged image. Further, the luminance component and the color difference component may be obtained from the composite image. In the second embodiment, as in the first embodiment, the luminance component may be obtained from the reference image and the rearranged image, and the color difference component may be obtained from the rearranged image.

  (7) In the first embodiment and the second embodiment described above, the case of handling an image signal of luminance color difference has been described. However, the present invention is not limited to this. In general, the present invention may be applied to cases where RGB, Lab and other image signals are handled. In the case of this RGB image signal, a signal component with high visual sensitivity is a G component, and the remaining signal components are RB components. In the case of a Lab image signal, a signal component with high visual sensitivity is an L component, and the remaining signal components are ab components.

  (8) In the first embodiment and the second embodiment described above, the positional deviation of the pattern is detected by image processing. However, the present invention is not limited to this. For example, an acceleration sensor or the like may be mounted on the camera side to determine the movement (vibration) of the shooting area of the camera, and the positional deviation of the pattern of the multiple images may be detected from the movement (vibration) of the shooting area.

  (9) In the first embodiment and the second embodiment described above, the example in which the image processing apparatus is mounted on a general electronic camera has been described. However, the image processing apparatus of the present invention can of course be applied in combination with, for example, a surveillance camera or a drive recorder system including an in-vehicle camera that records accident images.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiments are merely examples in all respects and should not be interpreted in a limited manner. The scope of the present invention is shown by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

As described above, the present invention is a technique that can be used for an image processing apparatus or the like.

Claims (9)

An image input unit that captures a plurality of images of the same subject;
A position shift detection unit for detecting a position shift of the pattern between the plurality of images;
An image composition unit configured to perform composition of the images on the plurality of images based on the positional deviation;
The image synthesizing unit determines a high frequency component of a spatial frequency for each of the plurality of images, and among the determined plurality of images, an image having fewer high frequency components has a composition of alignment synthesis at an edge portion of the pattern. An image processing apparatus characterized in that the ratio is reduced. The image processing apparatus according to claim 1.
The image synthesizing unit raises a composition ratio of alignment composition in the flat part of the pattern higher than a composition ratio of the edge part for an image determined to have a small amount of the high frequency component. . The image processing apparatus according to claim 1 or 2,
The image processing device, wherein the image composition unit selects an image to be used for composition according to the amount of the high frequency component. The image processing apparatus according to any one of claims 1 to 3,
The image input unit captures a plurality of first resolution images and a second resolution image having a higher resolution than the first resolution image,
The positional deviation detection unit obtains positional deviations of a plurality of the first resolution images at intervals smaller than the first resolution image using the second resolution image as a position reference.
An image processing apparatus. The image processing apparatus according to claim 4.
The positional deviation detection unit generates a plurality of types of sampling information having different phases from the second resolution image, and detects the positional deviation by comparing the plurality of types of sampling information and the first resolution image. Do
An image processing apparatus. The image processing apparatus according to claim 4 or 5,
The image synthesizing unit generates a rearranged image obtained by mapping information on each pixel of the first resolution image according to the resolution of the second resolution image using the positional shift information, and A luminance component and a luminance component of the second resolution image are synthesized.
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 3,
The image input unit captures a plurality of first resolution images and a second resolution image having a higher resolution than the first resolution image,
A control unit for performing exposure calculation when acquiring the second resolution image;
  The control unit sets the exposure time to be smaller than the allowable upper limit when the exposure time of the second resolution image obtained by the exposure calculation exceeds an allowable upper limit set according to image blurring. , Causing the image composition unit to perform image composition
An image processing apparatus. An image processing apparatus according to any one of claims 1 to 7,
An imaging unit that continuously shoots a subject and generates a plurality of images,
A function of aligning and synthesizing the plurality of images by the image processing apparatus;
An electronic camera characterized by that. An image processing program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 7.

Images