WO2008032441A1

WO2008032441A1 - Image processing device, electronic camera and image processing program

Info

Publication number: WO2008032441A1
Application number: PCT/JP2007/000978
Authority: WO
Inventors: Shigeru Doida; Tsukasa Murata
Original assignee: Nikon Corporation
Priority date: 2006-09-14
Filing date: 2007-09-07
Publication date: 2008-03-20
Also published as: JPWO2008032441A1; JP5012805B2

Abstract

An image processing device is provided with an image input unit, a position discrepancy detecting unit and an image synthesizing unit. The image input unit receives a plurality of images formed from the same object. The position discrepancy detecting unit detects position discrepancies of patterns among a plurality of the images. The image synthesizing unit makes the positions of the patterns consistent with each other in accordance with the position discrepancies and synthesizes the patterns. In this structure, the image synthesizing unit judges high frequency components of spatial frequencies of a plurality of images. As an image is judged to be less in the high frequency components in this case, a synthesizing ratio for a position adjustment at edge portions of the patterns is set to be smaller.

Description

Specification

Image processing apparatus, electronic camera, and image processing program

Technical field

TECHNICAL FIELD [0001] The present invention relates to an image processing apparatus, an electronic camera, and an image processing program.

[0002] Conventionally, there has been known a technique for repairing a blur of a captured image by performing a plurality of divided exposures using an electronic camera and aligning and synthesizing the obtained images. For example, the following patent document 1).

Patent Document 1: Japanese Patent Laid-Open No. 2 00 2 _ 1 0 7 7 8 7 (Claim 1 etc.)

Disclosure of the invention

Problems to be solved by the invention

[0003] By the way, in the prior art, there is a concern that the edge portion of the pattern may become dull after synthesis due to individual differences in the multiple images to be aligned and synthesized.

[0004] Therefore, an object of the present invention is to provide a technique for improving the image quality after composition in the alignment composition of a plurality of images.

Means for solving the problem

<< 1 >> An image processing apparatus of the present invention includes an image input unit, a positional deviation detection unit, and an image composition unit. The image input unit captures multiple images of the same subject. The positional deviation detection unit detects a positional deviation of the pattern between a plurality of images. The image compositing unit performs pattern alignment on a plurality of images based on the positional deviation and combines them. In such a configuration, the image composition unit determines the high frequency component of the spatial frequency for a plurality of images. At this time, the composition ratio of the alignment composition at the edge portion of the pattern is reduced as the image is determined to have few high-frequency components.

[0006] << 2 >> Preferably, for an image determined to have few high-frequency components, the image composition unit sets the composition ratio of the alignment composition in the flat part of the pattern to the edge. Higher than the composite ratio.

<3> Further preferably, the image composition unit selects an image to be used for composition according to the amount of the high frequency component.

<< 4 >> An electronic camera according to the present invention includes an image processing apparatus according to any one of << 1 >> to << 3 >>, and an imaging unit that continuously captures a subject and generates a plurality of images. And has a function of aligning and synthesizing a plurality of images by the image processing apparatus.

<< 5 >> The image processing program of the present invention is a program for causing a computer to function as the image processing apparatus according to any one of << 1 >> to << 3 >>.

In the present invention, a plurality of images are distinguished by a high frequency component of spatial frequency. In other words, for images that are judged to have few high-frequency components, the composition ratio of the registration composition is lowered at least at the edge of the picture. As a result, at the edge portion of the pattern, the image is composed mainly of high-frequency components. As a result, it is possible to suppress adverse effects such as a dull edge after synthesis.

Brief Description of Drawings

[001 1] [Fig. 1] Block diagram showing electronic camera 10 (including image processing device 25)

FIG. 2 is a block diagram schematically showing the configuration of the image processing device 25.

FIG. 3 is a flowchart for explaining the operation of the electronic force mela 10 in the first embodiment.

FIG. 4 is a flowchart for explaining the operation of the electronic force mela 10 in the first embodiment.

[Fig.5] Diagram explaining low resolution image and high resolution image

[Fig.6] Image misalignment detection by comparing projected edges

[Figure 7] Diagram showing sub-sampling

[Fig.8] Diagram showing adjustment of composite ratio

[Fig. 9] Diagram for explaining the operation of generating rearranged images

FIG. 10 is a flowchart for explaining the operation of the electronic camera 10 in the second embodiment.

[FIG. 1 1] Schematic diagram showing image composition processing in the second embodiment. BEST MODE FOR CARRYING OUT THE INVENTION

[Description of electronic camera configuration]

FIG. 1 is a block diagram showing an electronic camera 10 (including an image processing device 25) of the present embodiment.

In FIG. 1, a photographing lens 12 is attached to the electronic camera 10. In the image space of the photographic lens 12, the imaging surface of the imaging device 11 is arranged. The imaging element 11 is controlled by the imaging control unit 14. The imaging device 11 has a mode for reading out a high resolution image and a mode for reading out a low resolution image by performing pixel thinning and pixel addition inside the device. The image signal output from the image sensor 11 is processed through the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17.

The memory 17 is connected to the bus 18. An imaging control unit 14, a microprocessor 19, a recording unit 22, an image compression unit 24, a monitor display unit 30, and an image processing device 25 are connected to the bus 18. An operation unit 19 a such as a release button is connected to the microprocessor 19. In addition, a recording medium 2 2 a is detachably attached to the recording unit 22.

[Description of Image Processing Device 2 5]

The high resolution image read from the memory 17 is supplied to the reduced image generation unit 3 2, the feature amount extraction unit 3 3, and the image synthesis unit 3 4 via the gain correction unit 3 1. . The output data of the reduced image generation unit 32 is supplied to the rough detection unit 36 via the feature amount extraction unit 35. The output data of the feature quantity extraction unit 33 is supplied to the precision detection unit 38 via the phase division unit 37.

On the other hand, a plurality of low-resolution images read from the memory 17 are supplied to the feature amount extraction unit 39 and the image synthesis unit 34, respectively. The output data of the feature quantity extraction unit 39 is supplied to the rough detection unit 36 and the precise detection unit 38, respectively.

[0018] The positional deviation roughly detected by the rough detection unit 36 is supplied to the fine detection unit 38. . The positional deviation detected with high precision by the precision detection unit 38 is supplied to the image composition unit 34. The image synthesis unit 34 synthesizes a plurality of low-resolution images and high-resolution images based on the detection result of the positional deviation.

[001 9] [Description of operation]

3 and 4 are flowcharts for explaining the operation of the electronic camera 10. Less than

This operation will be described along the step numbers shown in FIG.

Step S 1: When the main power supply of the electronic camera 10 is turned on, the microprocessor 19 instructs the imaging control unit 14 to read out a low resolution image. The imaging control unit 14 drives the imaging device 11 in the low-resolution readout mode, and sequentially reads out the low-resolution images at, for example, 30 frames / second as shown in FIG.

[0021] Step S2: The low-resolution image read from the image sensor 11 is processed through the signal processing sound ΙΠ5 and the A / D converter 16 and then temporarily stored in the memory 17 The The microprocessor 19 deletes the low resolution images exceeding the predetermined number of frames in the memory 17 in order from the oldest one.

[0022] The predetermined number of frames here corresponds to the number of frames of a low-resolution image used for composition of a rearranged image, which will be described later, and the number of pixels of a high-resolution image / number of pixels of a low-resolution image It is preferable to set.

[0023] For example, the number of vertical and horizontal pixels of a low-resolution image is 1 each of the number of vertical and horizontal pixels of a high-resolution image.

In the case of / 4, it is preferable to set the predetermined number of frames to 4 X 4 = 1 6 frames or more.

Step S 3: The monitor display unit 30 displays a low-resolution image (through image) on the monitor screen. On the other hand, the microprocessor 19 calculates the exposure based on the photometry result of the photometry unit (not shown) and the brightness of the low resolution image, and determines the exposure time of the high resolution image.

Step S 4: Here, the microprocessor 19 determines whether or not the release button has been fully pressed by the user.

[0026] When the release button is fully pressed, the microprocessor 19 shifts the operation to step S5. On the other hand, when the release button is not fully pressed The microprocessor 19 returns to step S1.

Step S 5: Here, the microprocessor 19 determines whether or not the exposure time setting of the high-resolution image determined in Step S 3 is equal to or less than an allowable upper limit where blur is not noticeable. For example, the allowable upper limit is set to about 1 / (the focal length in terms of 35 mm size of the taking lens 1 2) second.

[0028] If the exposure time setting is less than or equal to the allowable upper limit, the microprocessor 19 moves to step S6. On the other hand, if the exposure time setting exceeds the allowable upper limit, the microprocessor 19 moves to step S7.

[0029] Step S6: The imaging control unit 14 uses the imaging device according to the set exposure time.

1 Control 1 Subsequently, the imaging control unit 14 drives the imaging device 11 in a high-resolution reading mode, and reads a high-resolution image. This high-resolution image (still image) is recorded and stored in the recording medium 22 a after undergoing image processing and image compression as in the conventional case. With this operation, the electronic camera 10 completes the shooting operation.

Step S 7: On the other hand, if it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 limits the exposure time to an allowable upper limit that does not cause blurring.

[0031] The imaging control unit 14 performs shutter control of the imaging device 11 according to a short limited exposure time. In this state, the imaging control unit 14 drives the imaging device 11 in the high resolution readout mode to read out the high resolution image. This high-resolution image has a low signal level due to underexposure but is less likely to blur. This high resolution image is temporarily recorded in the memory 17.

Step S 8: The gain correction unit 31 in the image processing device 25 fetches a high resolution image from the memory 17. The gain correction unit 31 adjusts the gain of this high resolution image to match the signal level of the low resolution image.

Step S 9: The reduced image generation unit 32 performs resolution conversion on the high-resolution image after gain adjustment and matches the number of pixels of the low-resolution image.

[0034] For example, by extracting the average value of 4 X 4 pixels, the vertical and horizontal dimensions of the high-resolution image The resolution can be converted to 1/4 of the number of pixels.

[0035] The high resolution image (hereinafter referred to as a reduced image) reduced in resolution in this way is transmitted to the feature quantity extraction unit 35.

Step S 10: FIG. 6 is a diagram for explaining image shift detection based on comparison of projected edges. Hereinafter, the image misalignment detection process will be described with reference to FIG.

[0037] First, the feature quantity extraction unit 35 performs the vertical edge extraction filter shown in the following equation (Fig. 6).

(See [A]), the vertical edge component g _y is extracted from the reduced image f (x, y) force.

g _y (X, y) =-f (x, y-D + f (χ, y + 1)

In addition, the feature extraction unit 35 uses a horizontal edge extraction filter (Fig.

6 [B]) is used to extract the reduced image f (x, y) force and horizontal edge component g _x .

g _x (X, y) = -f (χ-1, y) + f (χ + 1, y)

In order to reduce the influence of noise, the feature quantity extraction unit 35 preferably replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.

Next, as shown in FIG. 6A, the feature quantity extraction unit 35 calculates the vertical projection waveform by accumulating the vertical edge component g _y in a horizontal unit.

Furthermore, as shown in FIG. 6B, the feature quantity extraction unit 35 calculates a horizontal projection waveform by cumulatively adding the horizontal edge component g _x to the unit of the vertical column.

On the other hand, the feature quantity extraction unit 39 fetches a plurality of low resolution images from the memory 17. The feature quantity extraction unit 39 performs the same processing as the feature quantity extraction unit 35 on each low-resolution image, and obtains a vertical projection waveform and a horizontal projection waveform, respectively.

Here, as shown in FIG. 6 [A], the coarse detection unit 36 takes a difference while shifting the vertical projection waveform in the central area of the reduced image and the vertical projection waveform in the central area of the low-resolution image. Detect the waveform deviation that minimizes the sum of the absolute values of the differences. This waveform shift corresponds to the vertical position shift between the reduced image and the low resolution image.

[0042] Further, as shown in Fig. 6 [B], the coarse detection unit 36 is arranged in the horizontal direction of the central area of the reduced image. The difference between the projected waveform and the horizontal projected waveform in the center area of the low-resolution image is taken to detect the waveform shift that minimizes the sum of the absolute values of the differences. This waveform shift corresponds to a positional shift in the horizontal direction between the reduced image and the low resolution image.

In this way, the coarse detection unit 36 obtains the positional deviations (coarse detection results) of the plurality of low resolution images using the reduced image as a position reference, and outputs it to the fine detection unit 38.

[0044] Step S 1 1: The feature amount extraction unit 3 3 takes in the high-resolution image that has been gain-corrected, and extracts the vertical edge component g _y and the horizontal edge component g _x using an edge extraction filter. .

Note that the edge extraction filter here is preferably switched as follows according to the low-resolution image readout method.

■ When a low-resolution image is created by pixel addition or pixel averaging

g _y (X, y) = (-f (x, y-4)-f (x, y-3)-f (x, y-2)-f (x, y-1) + f (x, y + 4) + f (x, y + 5) + f (x, y + 6) + f (x, y + 7)] / 4

g _x (x, y) = (-f (x- 4, y)-f (x-3, y)-f (x-2, y)-f (x-1, y) + f (x + 4, y) + f (x + 5, y) + f (x + 6, y) + f (x + 7, y)] / 4

• When a low-resolution image is created by pixel decimation

g _y (X, y) = -f (χ, y-4) + f (x, y + 4)

g _x (x, y) = -f (x— 4, y) + f (x + 4, y)

In order to reduce the influence of noise, it is preferable that the feature quantity extraction unit 33 replaces the vertical edge component g _y and the horizontal edge component g _x that fall within a predetermined minute amplitude with zero.

Next, the feature quantity extraction unit 33 calculates a vertical projection waveform by cumulatively adding the vertical edge component g _y to a horizontal unit. Also, the feature quantity extraction unit 33 calculates the horizontal projection waveform by cumulatively adding the horizontal edge component g _x to the unit of the vertical column.

[0047] The phase division unit 37 sub-samples the vertical projection waveform of the high-resolution image every four pixels. At this time, the phase division unit 37 shifts the phase of subsampling. By doing so, as shown in Fig. 7, four types of sampling information whose phases are shifted from each other are generated.

[0048] Similarly, the phase division unit 37 subsamples the horizontal projection waveform of the high-resolution image every four pixels. At this time, by shifting the phase of sub-sampling, four types of sampling information whose phases are shifted from each other are generated.

[0049] Step S12: The precision detection unit 3 8 uses the coarse detection result of the positional deviation by the coarse detection unit 36 as a starting point, sampling information of the longitudinal projection waveform obtained from the high resolution image, and the low resolution image Differences are taken while shifting from the vertical projection waveform of, and the waveform deviation that minimizes the absolute sum of the differences is detected.

[0050] The precise detection unit 38 performs the detection of the waveform deviation for each of the four types of sampling information, thereby obtaining the waveform deviation that most closely matches the feature of the pattern (in this case, the waveform). This waveform shift corresponds to a horizontal position shift in units smaller than the pixel interval of the low resolution image.

[0051] Further, the precision detection unit 38 similarly detects the positional deviation in the vertical direction in a unit smaller than the pixel interval of the low resolution image (for example, the unit of the pixel interval of the high resolution image).

In this manner, the precision detection unit 38 obtains positional shifts (precision detection results) of a plurality of low resolution images using the high resolution image as a position reference, and outputs them to the image composition unit 34.

Step S 1 3: The image composition unit 34 performs high-pass filter processing on the low-resolution image, calculates the absolute value sum of the filter result, and obtains the amount of the high-frequency component. Classify multiple low-resolution images according to the amount of high-frequency components found.

Note that, as shown in FIG. 8, an image may be given an order in descending order of high frequency components, and images from higher order to a predetermined order may be selected to be used for composition. In addition, an image in which the amount of the high frequency component exceeds a predetermined threshold may be selected and used as an image to be combined.

[0055] By such a selection process, an image whose high frequency component does not meet the standard, such as a large amount of blurring, can be appropriately excluded from the synthesis process. As a result, synthesis It becomes possible to improve the image quality later.

Step S 14: The image composition unit 34 performs high-pass filter processing on the high-resolution image, and divides the high-resolution image into an edge area and a flat area from the filter result.

Step S 15: In the edge area obtained in step S 14, the composition ratio is reduced as the image has fewer high frequency components. On the other hand, in the flat region obtained in step S 14, the composition ratio of the image with few high-frequency components is increased from the edge portion. For regions that are neither the edge region nor the flat region, the composite ratio is set to an intermediate value between the two regions.

[0058] Step S 1 6: The image composition unit 3 4 enlarges each of the plurality of low resolution images.

(4 x 4 times) At this time, the image composition unit 34 does not perform pixel interpolation, and obtains an enlarged image with a large pixel interval.

Next, the image compositing unit 34 shifts the pixel position of the enlarged image of the low resolution image based on the precise detection result of the positional shift obtained by the precise detection unit 38, respectively.

Perform mapping (relocation) as shown in Fig. 9.

In this way, a rearranged image having the same number of vertical and horizontal pixels as a high-resolution image can be obtained.

[0061] Step S 17: The rearranged image that has been subjected to the mapping process includes pixels that are not filled with gaps, pixels that deviate from the normal pixel position, and overlapping pixels.

[0062] The image composition unit 34 picks up neighboring pixels for each regular pixel position of the rearranged image. The image composition unit 34 performs a weighted average on the signal components of these neighboring pixels according to the composition ratio set in step S15. The image composition unit 34 uses the weighted average value as the signal component (luminance / color difference component, etc.) at the normal pixel position. By such a composition process, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.

[0063] Instead of the above weighted average, rearranged images may be synthesized by median calculation of signal components of neighboring pixels. Step S 1 8: The image composition unit 34 performs the filtering process on the luminance component of the high resolution image as follows.

First, the image composition unit 34 extracts a luminance component from the high-resolution image after gain correction, and performs filter processing that combines median processing and Gaussian filtering. For example, the filter size is set to 3 X 3 pixels, and the three values in the center are extracted from 9 pixels within this filter size, and the Gaussian filter is executed. With this process, it is possible to reduce in advance the noise contained in the underexposed luminance component.

Step S 19: Next, a composition ratio is set between the luminance component A of the high-resolution image after the filtering process and the luminance component B of the rearranged image.

[0067] Here, it is preferable to set the composition ratio according to the following rules.

(1) Lower the composition ratio of the rearranged image locally as the signal difference between the luminance component A of the high-resolution image and the luminance component B of the rearranged image increases.

(2) The ratio of the rearranged image is locally reduced at locations where the local signal change of the luminance component A of the high-resolution image is large.

(3) The smaller the time interval between the high-resolution image and the low-resolution image, the higher the overall composition ratio of the luminance component B of the rearranged image.

(4) As the positional deviation between the high-resolution image and the low-resolution image is small, the composition ratio of the luminance component B of the rearranged image is generally increased.

[0068] It is preferable to use a Gaussian filter of the following formula for calculating the luminance component g (i, j) of a specific composite image.

[0069] [Equation 1]

∑∑ {G (k, I, A (i, j)) B (i _ — 1) / 2 + 1 1, 1 (w _ 1) / 2 + / _ 1)} + A (i, j ) g J) =

∑∑G (J ()) + 1

2σ In the above formula, a two-step process is performed. That is, a Gaussian filter (smoothing) of about m = 5 is applied to the luminance component B of the rearranged image. Next, the smoothing result of the luminance component B is weighted and combined with the luminance component A of the high-resolution image in units of pixels.

[0070] At this time, the larger the signal difference between the luminance components A and B, the lower the synthesis ratio G (i, j, p) of the luminance component B. As a result, the luminance component A of the high-resolution image is locally given priority in places where the pattern of the high-resolution image and the rearranged image are significantly different.

[0071] Note that it is preferable that the composition ratio G (i, j, p) is not particularly changed depending on the reference distance of the pixel. Thus, if the luminance component B is within the filter size m, it can be reflected in the luminance component A even if the reference distance is long. As a result, it is possible to tolerate misalignment of the rearranged image to some extent.

[0072] Also, σ in the above equation is a numerical value for adjusting the value of the composition ratio and the amount of decrease. This σ is preferably set to be locally smaller at locations where the variance of 3 × 3 pixels around the luminance component Α is larger. By reducing σ locally in this way, the composition ratio of the rearranged image is locally reduced near the edge of the high-resolution image. As a result, it is possible to suppress the influence (smoothing and disturbance) on the image structure such as edges.

[0073] Note that it is preferable that σ in the above equation is set to be larger as the time interval and / or positional deviation between the high resolution image and the low resolution image is smaller. By varying σ in this way, the overall ratio of the rearranged images increases as the images of the high-resolution image and the low-resolution image are estimated to be closer. As a result, information on rearranged images (low-resolution images) with similar patterns can be preferentially reflected in the composite image.

Step S 2 0: Combines the color difference component of the rearranged image generated in step S 17 and the luminance component of the composite image generated in step S 19 to obtain information on the low-resolution image. A reflected high-resolution color image is generated.

The color image is stored and recorded in the recording medium 2 2 a via the image compression unit 24 and the recording unit 22. [Effects of this embodiment, etc.]

In this embodiment, the through image (low resolution image) discarded after the monitor display is effectively utilized for improving the image quality of the still image (high resolution image). By effectively using this through image, the imaging performance of the electronic camera 10 can be improved.

In this embodiment, a low-resolution image with a short imaging time interval is synthesized. Therefore, the difference in the pattern between images is originally small, and a good image composition result can be obtained.

Furthermore, in this embodiment, a plurality of low resolution images are aligned using a high resolution image with a high pixel density as a position reference. Therefore, the image alignment accuracy of the low-resolution image is high, and a better image composition result can be obtained.

[0079] In the present embodiment, a plurality of pieces of sampling information whose sampling phases are shifted from each other are generated from a high-resolution image. By performing position shift detection between the sampling information and the low resolution image, the position shift can be detected in units smaller than the pixel interval of the low resolution image. Therefore, it is possible to further improve the alignment accuracy of the image of the low resolution image, and obtain a better image composition result.

[0080] Furthermore, in this embodiment, luminance components (signal components with high visual sensitivity) of a plurality of low resolution images are aligned and mapped to obtain luminance components with higher resolution than the low resolution images. be able to.

[0081] In the present embodiment, the color difference components (signal components having low visual sensitivity) of a plurality of low resolution images may be registered and mapped to obtain a color difference component having a higher resolution than that of the low resolution image. it can.

Furthermore, in this embodiment, the high-resolution luminance component (rearranged image) created by the alignment and the luminance component of the original high-resolution image are weighted and synthesized. This weighted composition can suppress uncorrelated luminance noise between the two images. As a result, it is possible to improve the S / N of the high-resolution image in the underexposure state (see Step S7). In this embodiment, when the signal difference between the high-resolution image and the rearranged image is significantly large, the composition ratio of the rearranged image is locally reduced. As a result, if a misalignment occurs in the rearranged image, the signal difference between the two images opens significantly, and the composite ratio of the rearranged image decreases. Therefore, this misalignment is not reflected much in the image composition result.

Furthermore, in the present embodiment, the composition ratio of the rearranged image is locally lowered at a location where the local signal change of the high resolution image is large. Therefore, the image structure of the high-resolution image is given priority in the edge portion of the image. As a result, it is possible to avoid adverse effects such as multi-line edges.

Further, in the present embodiment, as the time interval and / or positional deviation between the high resolution image and the low resolution image is small, the rearranged image synthesis ratio is generally increased. If this condition is met, it can be estimated that the pattern of the high-resolution image and the low-resolution image are very close. Therefore, by increasing the composition ratio of the rearranged images, it is possible to further increase the image S / N without the risk of pattern failure.

In the present embodiment, as shown in FIG. 8, a plurality of low resolution images are ranked according to the amount of the high frequency component of the spatial frequency. At this time, images with few high-frequency components are excluded from the images used for composition because blurring is noticeable.

[0087] Furthermore, even if the images used for the composition are selected, the composition ratio of the alignment composition at the edge portion of the pattern is adjusted to be small for an image that is determined to have relatively few high-frequency components. As a result, if there is a camera shake, the edge part in the image with little subject blur is given priority, and the alignment composition is performed. As a result, adverse effects such as a dull edge in the rearranged image are suppressed.

[0088] In the present embodiment, for the image determined to have few high-frequency components, the synthesis ratio of the flat portion is increased compared to the edge portion. As a result, in the flat portion, multiple images are positioned and combined more evenly, and the image S / N after the position combining can be increased. <Explanation of Second Embodiment>

FIG. 10 is a flowchart for explaining the operation of the electronic camera 10 in the second embodiment. FIG. 11 is a schematic diagram showing the image composition processing in the second embodiment.

Here, the second embodiment is a modification of the first embodiment, and the electronic camera combines a plurality of images having the same resolution. In addition, since the configuration of the electronic camera in the second embodiment is the same as that of the electronic camera in the first embodiment (FIG. 1), a duplicate description is omitted.

Furthermore, S 1 0 1 to S 1 0 3 in FIG. 10 correspond to S 1 to S 3 in FIG. The operation of the electronic camera will be described below along the step numbers shown in FIG.

Step S 1 0 4: Here, the microprocessor 19 determines whether or not the exposure time setting of the high-resolution image determined in Step S 1 0 3 is below an allowable upper limit where blur is not noticeable. The allowable upper limit is set in the same manner as in step S5 above.

If the exposure time setting is less than or equal to the allowable upper limit, the microprocessor 19 moves to step S 1 0 5. On the other hand, if the exposure time setting exceeds the allowable upper limit, the microprocessor 19 moves to step S 1 06.

Step S 1 0 5: The imaging control unit 14 controls the image sensor 11 according to the set exposure time and captures a high-resolution image (still image). Note that the operation of this step corresponds to the above step S 6, and thus redundant description is omitted.

Step S 1 0 6: On the other hand, when it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 switches the operation state of the electronic camera to the continuous shooting mode. The microprocessor 19 limits the exposure time of each frame in the continuous shooting mode to an allowable upper limit that does not cause blurring.

In this continuous shooting mode, the imaging control unit 14 drives the imaging device 11 in a high-resolution readout mode according to a short limited exposure time. This Multiple high-resolution images will be continuously captured by the image sensor 1 1

. Each high-resolution image has a low signal level due to underexposure, but is less likely to blur. Each of the above high-resolution images is temporarily recorded in memory 17.

Here, in the continuous shooting mode described above, for example, the shooting operation is performed in the following manners (1) and (2).

(1) The imaging control unit 14 starts continuous shooting of high-resolution images from the time when the mode is switched to the continuous shooting mode, and stores high-resolution images for a certain period in the memory 17. At this time, the microprocessor 19 deletes the high-resolution images exceeding the predetermined number of frames in the memory 17 in order from the oldest one.

[0099] When the microprocessor 19 receives an instruction from the user (such as pressing the release button) during continuous shooting, the microprocessor 19 responds to the timing of the instruction in the high-resolution image in the memory 17. The image to be specified is designated as the reference image. Further, the microphone mouth processor 19 designates a predetermined number of high-resolution images that move back and forth in the time axis direction with respect to the reference image as targets of a synthesis process described later. In the second embodiment, a high-resolution image designated as a target for the synthesis process is referred to as a synthesized image. Note that the reference image and the composite image are held in the memory 17 at least until the compositing process is completed.

(2) The microprocessor 19 causes the imaging control unit 14 to start continuous shooting in response to a user's continuous shooting start instruction (such as pressing a release button). Then, the microprocessor 19 receives the user's one continuous shooting end instruction (such as releasing the release button) or when the continuous shooting for a predetermined number of frames is completed, Stop shooting.

[0101] After that, the microprocessor 19 displays the captured images on the monitor display unit 30 and allows the user to select the reference image. The microphone processor 19 uses, as a recorded image, an image designated by the user among the high-resolution images taken continuously. In addition, the microprocessor 19 uses, as a composite image, an image other than the reference image among high-resolution images taken continuously. The

[0102] Note that in the case of (2) above, the microprocessor 19 may designate an image corresponding to the timing of the continuous shooting start instruction (the first high-resolution image shot) as the reference image. In this case, it is possible to omit the step of letting the user specify the reference image.

Step S 1 0 7: The image processing device 25 performs processing of the composite image (S 1 0 6) with respect to the reference image (S 1 0 6) by the same processing as Step S 1 0 of the first embodiment. Detect image misalignment. Note that this image shift is detected for all the synthesized images.

Step S 1 0 8: The image compositing unit 3 4 of the image processing device 25 performs high-pass filtering on the composite image (S 1 0 6), calculates the absolute value sum of the filter results, and calculates the high frequency Determine the amount of ingredients. Then, the image composition unit 34 classifies a plurality of composite images according to the obtained amount of the high frequency component.

At this time, as in the first embodiment, the image compositing unit 34 assigns the order to the images in the descending order of the high frequency components, selects the images from the higher order to the predetermined order, and You may narrow down the image actually used for composition from among the images. In addition, the image composition unit 34 may select an image in which the amount of the high-frequency component exceeds a predetermined threshold, and narrow down images to be actually used for composition from among the composite images.

Step S 1 0 9: The image composition unit 34 performs high-pass filter processing on the reference image. Then, the reference image is divided into an edge area and a flat area from the filter result. Then, the image composition unit 34 reduces the composition ratio in the edge region of the reference image as the composite image with fewer high-frequency components. On the other hand, in the flat region of the reference image, the image composition unit 34 increases the composition ratio of an image with less high-frequency components than the edge portion. For areas that are neither edge areas nor flat areas, the composition ratio is set to an intermediate value between the two areas.

Step S 1 1 0: The image composition unit 3 4 shifts the pixel position of the composition image based on the position shift detection result obtained in S 1 0 7 with the reference image as the position reference. Mapping (rearrangement) is performed (see Fig. 11). Second embodiment Then, the synthesized image after matbing is called a rearranged image.

In this second embodiment, the resolution (number of pixels) of the reference image and each synthesized image are the same. For this reason, as shown in Fig. 11, the rearranged image group is treated as a three-dimensional image having a time axis t in addition to the X axis y axis indicating the position in the same image.

Step S 1 1 1: The image composition unit 3 4 performs composition processing of the reference image (S 1 0 6) and the rearranged image (S 1 1 0) to generate a final composite image. .

[0110] Here, the image composition unit 34 sequentially composes each rearranged image with respect to the reference image, and repeats the compositing process for the number of rearranged images. Note that the image composition unit 34 may synthesize a plurality of rearranged images in advance by median calculation or the like, and then synthesize the rearranged image after synthesis and the reference image.

[01 1 1] Hereinafter, the synthesis process in S 1 1 1 will be described in detail. First, the image composition unit 34 reads out the initial value of the composition ratio obtained in S 1 09. Then, the image composition unit 34 further adjusts the composition ratio of the reference image and the rearranged image by paying attention to the gradation value of the reference image and the gradation value of the rearranged image.

[01 12] For example, the image composition unit 3 4 determines that the gradation value difference between the target pixel of the reference image and the target pixel of the rearranged image is greater than or equal to the threshold value (when the gradation difference is large). The composition ratio at the target pixel is locally lowered. On the other hand, when the difference in gradation value between the target pixel of the reference image and the target pixel of the rearranged image is less than the threshold value (when the gradation difference is small), the image composition unit 3 4 Increase the composition ratio at. This greatly suppresses the effects of misalignment between images.

[0113] Further, the image composition unit 34 adjusts the composition ratio in accordance with local signal changes on the reference image. For example, the image synthesizing unit 34 locally lowers the synthesis ratio with the rearranged image at a location where the local signal change on the reference image is larger. In addition, the image composition unit 34 increases locally the composition ratio with the rearranged image at a location where the local signal change is small on the reference image. As a result, it is possible to suppress adverse effects such as multi-line edges on the synthesized image. [0114] Second, the image composition unit 34 adds and composes the corresponding pixels of the reference image and the rearranged image according to the composition ratio. In S 1 1 1, it is assumed that the image composition unit 34 obtains the luminance component and the color difference component of each pixel by the composition processing. As a result, a high-resolution power error image reflecting the information of a plurality of synthesized images is generated.

[01 15] Step S 1 1 2: The microprocessor 19 records the final composite image (generated in S 1 1 1) via the image compression unit 24 and the recording unit 22. Record on medium 2 2 a. This is the end of the description of FIG.

[0116] The configuration of the second embodiment can provide the same type of effect as the first embodiment. In particular, in the case of the second embodiment, since the resolution of the rearranged image is high and the amount of I blue light is high, a higher-definition color image can be acquired.

[01 17] <Supplementary items of the embodiment>

(1) The present inventor has disclosed a procedure for further speeding up the position shift detection in Japanese Patent Application No. 2 0 0 5-3 4 5 7 15. According to this procedure, the position shift detection in each of the above embodiments may be speeded up.

[01 18] (2) In step S 10, absolute positional deviation is roughly detected between the reduced image of the high resolution image and the low resolution image. However, the present invention is not limited to this. Relative positional deviations between multiple low-resolution images may be detected roughly. The precision detection unit 38 can roughly know the remaining absolute positional deviation based on the relative coarse detection result and the precision detection result of at least one positional deviation. The precision detector 38 can detect a precise positional deviation quickly by performing a positional deviation search using the absolute rough detection result as a starting point.

[0119] (3) In the first and second embodiments described above, the positional deviation of the image is detected by comparing the projected waveforms. However, the present invention is not limited to this. For example, the positional deviation may be detected by spatial comparison of the pixel arrangement of both images.

[0120] (4) In the first embodiment and the second embodiment described above, an image is added to the electronic camera. The case where the processing apparatus is mounted has been described. However, the present invention is not limited to this. An image processing program in which the above-described image processing is converted into a program code may be created. By executing this image processing program with a computer, it is possible to create a high-quality and high-resolution composite result on a computer by effectively utilizing low-resolution image information. Similarly, when the image processing of the second embodiment is executed on a computer, a high-quality and high-resolution composite result can be created on the computer by effectively using other images taken continuously.

[0121] When the image processing of the second embodiment is executed on a computer, it is possible for the user to designate an arbitrary image as a reference image from among the continuously shot images.

(5) In the first embodiment described above, a plurality of low-resolution images are acquired before capturing a high-resolution image. However, the present invention is not limited to this. A plurality of low-resolution images may be acquired after capturing a high-resolution image. In addition, a plurality of low resolution images may be acquired before and after capturing a high resolution image.

(6) In the first embodiment described above, the color difference component is obtained from the rearranged image, and the luminance component is obtained from the synthesized image. However, the present invention is not limited to this. The luminance component and the color difference component may be obtained from the rearranged image. Further, the luminance component and the color difference component may be obtained from the composite image. In the second embodiment, as in the first embodiment, the luminance component may be obtained from the reference image and the rearranged image, and the color difference component may be obtained from the rearranged image.

(7) In the first embodiment and the second embodiment described above, the case of handling an image signal of luminance color difference has been described. However, the present invention is not limited to this. In general, the present invention may be applied to cases where RGB, Lab, or other image signals are handled. In the case of this RGB image signal, the signal component with high visual sensitivity is the G component, and the remaining signal component is the RB component. In the case of a Lab image signal, the signal component with high visual sensitivity is the L component, and the remaining signal component is the ab component. (8) In the first and second embodiments described above, the positional deviation of the pattern is detected by image processing. However, the present invention is not limited to this. For example, an acceleration sensor or the like may be mounted on the camera side to determine the movement (vibration) of the shooting area of the camera, and the displacement of the pattern of multiple images may be detected from the movement (vibration) of this shooting area.

(9) In the first embodiment and the second embodiment described above, the example in which the image processing apparatus is mounted on a general electronic camera has been described. However, the image processing apparatus of the present invention can of course be applied in combination with, for example, a surveillance camera or a drive recorder system equipped with an in-vehicle camera that records accident images.

[0127] It should be noted that the present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The scope of the present invention is indicated by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

Industrial applicability

As described above, the present invention is a technique that can be used for an image processing apparatus or the like.

Claims

The scope of the claims

[1] An image input unit that captures multiple images of the same subject,

A positional deviation detection unit that detects a positional deviation of a pattern between the plurality of images, and an image synthesis unit that performs synthesis by aligning the pattern on the plurality of images based on the positional deviation,

The image synthesizing unit determines a high frequency component of a spatial frequency for the plurality of images, and an image having fewer high frequency components reduces a composition ratio of alignment synthesis at an edge portion of the pattern.

An image processing apparatus.

[2] In the image processing device according to claim 1,

The image composition unit raises the composition ratio of the registration composition in the flat part of the pattern above the composition ratio of the edge part for the image determined to have a low high-frequency component.

An image processing apparatus.

[3] In the image processing device according to claim 1 or 2,

The image composition unit selects an image to be used for composition according to the amount of the high frequency component.

An image processing apparatus.

[4] The image processing device according to any one of claims 1 to 3,

An electronic camera comprising: an imaging unit that continuously shoots a subject to generate a plurality of images, and having a function of aligning and synthesizing the plurality of images by the image processing device.

[5] An image processing program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 3.