JP2010200178A - Image processing device, image processing method, image processing program, and program storage medium storing image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress a variation of an SN (signal to noise ratio) in an image generated by combining by optimizing the number of sheets to be combined for each pixel or for each area when the combined image is generated. <P>SOLUTION: By a motion detection unit 105, when one of a plurality of sheets of image is determined as a reference image and the other image is determined as a target image, an amount of the motion of the target image to the reference image is detected. By a combined area discrimination unit 106, a correlation value indicating a correlation of the target image to the reference image is calculated in a pixel unit or in an area unit based on the amount of motion, the correlation of the pixel or the area is determined based on the correlation value, the pixel or the area that has a high correlation is selected as the combined area, and the pixel or the area that has a low correlation is selected as the non-combined area. By a combined frame number selection unit 107, the number of sheets combined in the combined area is determined according to the state of the non-combined area and a combined image 113 is generated by combining the images according to the determined number of sheets to be combined. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, an image processing program, and a storage medium storing the image processing program, and in particular, an image processing apparatus that synthesizes an image using a plurality of images taken in time series, and image processing The present invention relates to a program storage medium storing a program and an image processing program.

  Conventionally, in an imaging device such as a digital camera or digital video, a process of capturing a plurality of temporally continuous images and detecting a motion vector of the entire image due to camera shake by taking a correlation between the plurality of images. It has been known. Japanese Patent Laid-Open No. 7-43647 (Patent Document 1) describes that image degradation due to camera shake is improved by matching corresponding points between a plurality of images based on a detected motion vector. A so-called electronic camera shake correction process is described.

  On the other hand, in Japanese Patent Application Laid-Open No. 2007-208925 (Patent Document 2), a physical motion detection mechanism such as a gyro sensor is provided inside an image pickup apparatus, and components of an optical system (for example, an image pickup element, a lens, etc.) are output from the output information. ) Is mechanically displaced to correct the camera shake.

  Since the above-described electronic camera shake correction process does not require a mechanical shake detection mechanism, it is advantageous in terms of reduction in size and weight of the image pickup apparatus and cost compared to the optical camera shake correction method, and a plurality of sheets. There is also an advantage that random noise caused by the device characteristics of the image sensor can be reduced by synthesizing these images. On the other hand, it is difficult to discriminate shake due to movement of the imaging device (hand shake) and blur due to movement of the subject (subject blur), and in some cases, there is a problem that a deteriorated image unintended by the photographer is generated.

  To solve this problem, for example, in Japanese Patent Laid-Open No. 2008-42659 (Patent Document 3), a reference image serving as a reference among a plurality of images, and two target images that are temporally before and after the reference image, Discloses a method of classifying an image into three regions, ie, a shielding region, an exposure region, and a region outside the shielding region, and processing a captured image more naturally and clearly.

Japanese Unexamined Patent Publication No. 7-43647 JP 2007-208925 A JP 2008-42659 A

However, in the method disclosed in Patent Document 3, there is a large difference between the cumulative addition number of pixels belonging to the shielding region and the exposure region and the cumulative addition number of pixels outside the shielding region, and the amount of noise included in the combined image is small. It will fluctuate greatly locally. In such a case, unnatural unevenness is generated in the image, or noise around the edge of the moving body is noticeable, resulting in deterioration in image quality.
In view of the above points, an object of the present invention is to provide an image processing apparatus capable of suppressing fluctuations in the amount of noise in the entire corrected image as much as possible when an image is synthesized for the purpose of camera shake correction.

In order to solve the above problems, the present invention employs the following means.
The present invention is an image processing apparatus that generates a composite image by combining a plurality of images, and when any one of the plurality of images is a reference image and another image is a target image. A motion detection unit that detects a motion amount of the target image with respect to the reference image; and after aligning the target image with the reference image based on the motion amount, the correlation of the target image with the reference image is determined. The calculated correlation value is calculated in units of pixels or areas, and the correlation value is compared with a preset threshold value, so that the composite area determined to have high correlation and the non-correlation determined to be low in correlation. Determined by area classification means for generating area classification information indicating classification with the composite area, composite number determination means for determining the composite number of the composite area based on the area classification information, and the composite number determination means Together To provide an image processing apparatus characterized by comprising an image synthesizing means for generating a composite image by synthesizing an image according to the number.

  The present invention is an image processing method for generating a composite image by combining a plurality of images, wherein one of the plurality of images is used as a reference image and another image is used as a target image. A motion detecting step for detecting a motion amount of the target image with respect to the reference image; and after aligning the target image with the reference image based on the motion amount, the correlation of the target image with the reference image is determined. The calculated correlation value is calculated in units of pixels or areas, and the correlation value is compared with a preset threshold value, so that the composite area determined to have high correlation and the non-correlation determined to be low in correlation. A region separation step for generating region separation information indicating separation from a composite region, a composite number determination step for determining a composite number of the composite region based on the region separation information, and a composite number determination unit. To provide an image processing method characterized by comprising an image combining step of generating a synthesized image by synthesizing an image according to a constant synthesis number, the.

  According to the present invention, an image processing program for photographing a subject to acquire a plurality of images of the subject and combining the acquired plurality of images to generate a composite image combines the plurality of images. An image processing program for generating a composite image, wherein when one of the plurality of images is set as a reference image and another image is set as a target image, a motion amount of the target image with respect to the reference image is detected A motion detection step, and after aligning the target image with the reference image based on the amount of motion, a correlation value indicating the correlation of the target image with the reference image is calculated in pixel units or region units. Then, by comparing the correlation value with a preset threshold value, region classification information indicating the separation between the synthesis region determined to have high correlation and the non-synthesis region determined to have low correlation is generated. An area classification step, a composite number determination step for determining a composite number of the composite areas based on the region classification information, and a composite image by combining the images according to the composite number determined by the composite number determination means An image processing program for causing a computer to execute an image synthesis step to be generated is provided.

  In order to solve the above problems, in the image processing apparatus according to the present invention, in a system that obtains a camera shake correction effect by combining a plurality of images, by optimizing the number of frames used for combining, Thus, there is an effect that it is possible to suppress fluctuations in the SN ratio in the composite image when there is a region unfavorable for composition due to subject blurring.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. In the 1st Embodiment of this invention, it is a figure which shows the example in case a moving body exists between several sheets of images. FIG. 3 is a diagram illustrating a state when a plurality of images in FIG. 2 are shift-combined. It is a figure which shows the relationship with the synthetic pixel number of each area | region in FIG. It is a figure which shows the relationship between a synthetic | combination number and the area rate which cannot ensure the number of synthetic | combination pixels for the number of synthetic | combination. In the 2nd Embodiment of this invention, it is a figure which shows the example in case a rotary body exists between several sheets of images. In FIG. 6, it is a figure which shows the relationship between a synthetic | combination number and the area ratio which cannot ensure the synthetic | combination pixel number for a synthetic | combination number. In FIG. 7, it is a histogram which shows the relationship between the number of composition and the area | region which can ensure the number of synthetic | combination pixels for a synthetic | combination number. It is a conceptual diagram which shows the example which restrict | limits the synthetic | combination number of sheets so that dispersion | distribution of the histogram of FIG. 8 may be suppressed.

  Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the image processing apparatus according to the present embodiment includes an optical system 100, an image sensor 101, an A / D conversion unit 102, an image processing unit 103, a frame memory 104, a motion detection unit 105, and a combined area classification. A unit 106, a composite frame number selection unit 107, and a synthesis processing unit 108.

  The optical system 100 is composed of a lens or the like and forms an image of a subject, and is disposed on the image sensor 101 so as to form an image. The imaging element 101 generates imaging data 110 that is electrical image information based on the subject image formed by the optical system 100, and outputs the imaging data 110 to the A / D conversion unit 102. In the A / D conversion unit 102, the image data 110 generated by the image sensor 101 is converted into discrete digital data 111 that can be subjected to predetermined processing in the image processing unit 103, and is output to the image processing unit 103. The image processing unit 103 performs predetermined image processing such as white balance adjustment, Y / C signal generation processing, electronic zoom processing, and noise suppression processing on the input digital data 111 to generate image data 112. If the image sensor 101 is a so-called single-plate type, processing such as demosaicing processing can also be performed. The frame memory 104 stores image data 112 relating to an image that has been subjected to predetermined image processing by the image processing unit 103. The above process is repeated for a plurality of sheets, and as a result, a plurality of pieces of image data 112 are stored in the frame memory 104.

  The motion detection unit 105 (motion detection means) uses, as a reference image, an image at a moment when a shutter button (not shown) provided in the image processing apparatus is pressed among a plurality of pieces of image data 112 stored in the frame memory 104. The movement (the amount of camera shake) of the entire image of another image (hereinafter referred to as a target image) with respect to the reference image is estimated. As the motion detection means 105, a block matching method, an optical flow method, and the like are known, but the means is not limited in the present invention.

  The composite area classification unit 106 compares the reference image stored in the frame memory 104 with the target image, and has a motion that is different from the motion of the entire image detected by the low-correlation portion, for example, the motion detection unit 105. The part shielded by the moving body is detected in units of pixels or predetermined regions. Specifically, the synthesis region classification unit 106 performs pixel unit or region unit alignment in a state where the target image and the reference image are aligned based on the movement of the target image with respect to the reference image transmitted from the motion detection unit 105. To calculate a correlation value indicating the correlation of the target image with the reference image. Here, “alignment” refers to a process of specifying pixels or regions having a correspondence relationship between a plurality of images having the same subject, and in this embodiment, the corresponding image is compared with the reference image. In other words, it is a process for specifying a pixel or a region having a corresponding relationship.

  Subsequently, the correlation of the target image with respect to the reference image is determined for each pixel or each region by comparing the calculated correlation value with a preset threshold value. When the correlation value is lower than the threshold value, that is, when it is determined that the target pixel or the target region of the target image is different from the reference image, a flag indicating a non-composite region is set. Further, when it is determined that the correlation value is higher than the threshold value, that is, the target pixel or the target region of the target image is substantially the same as the reference image, no flag is set as the synthesis region. The above processing is performed on the target images for the acquired number of images, and composite area classification information including a composite area and a non-composite area is generated.

  The criteria for determining the correlation include the sum of absolute differences (SAD), sum of square differences (SSD), or normalized cross-correlation (NCC: Values such as Normalized Cross Correlation) can be evaluated. In the present invention, any of these criteria may be used, and the means is not limited. In addition, regarding the determination of the level of correlation, a threshold is set in advance, and the level of correlation is determined by comparing this threshold with the correlation value. However, the present invention is not limited to this, and should be determined appropriately. Can do.

The composite frame number selection unit 107 sets a composite number determination threshold value as a reference when determining the composite number, and based on the composite number determination threshold value and the composite region classification information generated by the composite region classification unit 106. The number of images to be combined in the combining area is determined from the plurality of images, and the image to be combined is selected based on the determined number.
Here, according to the synthesis area classification information, the correlation between the plurality of images is high in the synthesis area, and all of the plurality of images can be synthesized. In the non-synthesis area, at least one of the plurality of images is The number of images that can be combined varies from region to region, such as being unfavorable for combining. However, if all images are synthesized in the synthesis area based on correlation alone and the number of synthesis is reduced in the non-synthesis area, the difference in the number of synthesis between the synthesis area and the non-synthesis area may increase. As a result, there is a possibility that the amount of noise is different between the synthesis area and the non-synthesis area, and the image quality of the entire image is degraded.

  Therefore, first, the composite frame number selection unit 107, when compositing an image, based on the composite region classification information, an area in which the number of pixels to be composited is less than the composite number, and the composite number (composite pixel number) at that time. Calculate the relationship. Subsequently, the ratio of this area to the entire image is calculated as an area ratio (non-composite area ratio) where the number of synthesized pixels corresponding to the number of synthesized pictures cannot be secured for each number of synthesized pictures. Then, by determining the upper limit of the composite number in the composite area based on the calculated non-composite area ratio and the composite number determination threshold, the composite number in the composite area is limited, and the difference from the composite number in the non-composite area is Prevent it from growing.

  Next, an image to be a target when the images are combined is selected. The determination criterion for selecting an image may be, for example, preferentially selecting an image close to the reference image in terms of time, or preferentially selecting an image having a high correlation with the reference image, and may be set as appropriate. Is possible. Note that, when combining, even if the number of combined images is the same, it is possible to vary the image selected for each pixel or region.

  The image composition unit 108 generates a composite image 113 by compositing the images selected by the composite frame selection unit 107 among the plurality of images stored in the frame memory 104.

  Next, an image processing method by the image processing apparatus having the above configuration will be described. In the present embodiment, an example will be described in which five divided exposures, that is, five imaging processes are performed in one shooting, and a combined image is generated using a maximum of five images as a target of the combining process.

  When the subject is photographed, an image of the subject is converted into imaging data by the image sensor 101 by the optical system 100 and output to the A / D converter 102. The imaging data is converted into digital data 111 by the A / D conversion unit 102 and output to the image processing unit 103. The input digital data is subjected to predetermined image processing in the image processing unit 103 and stored in the frame memory 104 as image data. The above-described imaging processing by the optical system 100, the imaging device 101, the A / D conversion unit 102, and the image processing unit 103 is repeated five times, and five image data (frames 1 to 5) that have undergone predetermined image processing shown in FIG. ) Is stored in the frame memory 104.

  As shown in FIG. 2, there are moving objects in the five frames 1 to 5 that can be image synthesis targets. The motion detection unit 105 detects the motion (camera shake amount) of the entire image of the frames 2 to 5 with respect to the frame 1 using the frame 1 as a reference image. The synthesized region classification unit 106 calculates the correlation of the frames 2 to 5 with respect to the frame 1 in units of pixels in a state where the images of the frames 1 and 5 are aligned based on the detected amount of motion. A combined region classification information is generated by setting a flag as a non-composite region to a pixel or region determined to be low. When the frames 1 to 5 are overlapped, the result is as shown in FIG.

  Subsequently, in the composite frame number selection unit 107, based on the composite region separation information for a plurality of images, a region (region a to region e) in which the number of pixels to be combined is less than the number of frames and a composite pixel when combining images. The relationship with the number (1-5) is calculated. As shown in FIG. 4, in FIG. 3, all the five frames can be combined in the region a, the number of combined pixels is 5, the region b is 4, the region c is 3, the region d is 2, and the region e Becomes 1.

  The ratio of this area to the entire image is calculated as a non-composite area ratio for which the number of synthesized pixels corresponding to the number of synthesized images cannot be secured for each number of synthesized images. As shown in FIG. 5, when the number of combined images is 1, 0, and when the number of combined images is 2, the ratio of non-composite areas where the number of combined pixels for two images cannot be secured is 0.054 with the entire image being 1, and similarly 3 Is 0.148 for four, 0.291 for four, and 0.387 for five. On the other hand, the composite frame number selection unit 107 sets a composite number determination threshold value in advance to 0.3, and determines the composite number based on the calculated non-composite area ratio and the composite number determination threshold value. That is, the non-composite area ratio shown in FIG. 5 is 0.3 or less when the number of composites is four, so the composite number is four.

  In addition, the minimum value of the composite number may be set in advance together with the composite number determination threshold. In this case, for example, when the composite number determination threshold is 0.1 and the minimum composite number is 3, the composite number is 2 according to the composite number determination threshold alone, but the composite number is less than the minimum composite number. Will be three.

  The composite frame number selection unit 107 preferentially selects, for example, a frame having a high correlation with the reference image frame 1 by setting the composite number to four. In this way, the selected frames are combined in the image combining unit 108, and a single combined image 113 is generated.

  According to the first embodiment of the present invention, when a composite image is generated through the above-described process, the number of composites for each pixel or region is optimized, and fluctuations in the SN ratio in the generated image due to the synthesis are suppressed. I can do it.

  Hereinafter, a second embodiment of the present invention will be described with reference to FIGS. The image processing apparatus according to the present embodiment is different from the first embodiment only in the composite frame number selection unit 107, and other configurations are the same as those in the first embodiment, and thus the description of the same configurations is omitted. The composite frame number selection unit 107 will be described.

  In the second embodiment of the present invention, it is difficult to obtain an optimal composite number when there are subjects that change at the same position between a plurality of images, such as a rotating body, as shown in FIG. . In the present embodiment, when the relationship between the composite number and the area ratio in which the number of pixels corresponding to the composite number cannot be ensured is represented, the area ratio of the rotating body with respect to the entire image becomes an area in which the number of pixels corresponding to the composite number cannot be ensured. As shown in FIG. 7, the area ratios are all the same for 2 to 5 composite sheets. Accordingly, the determination based on the composite number determination threshold value becomes almost invalid, and as a result, the composite number in this case is uniquely determined by the preset minimum composite number.

  Therefore, the composite frame number selection unit 107 superimposes (combines) each image in a state where the images are shifted based on the amount of motion detected by the motion detection unit 105, and the composite number and the composite number. An area where the number of synthesis target pixels can be secured is calculated as a synthesis area ratio. Then, as a degree of dispersion of the composite area ratio between the plurality of images, a histogram is created that represents the relationship between the number of composites and an area where the composition target pixels for the composite number can be secured.

  In the case shown in FIG. 6, a histogram as shown in FIG. 8 is created. According to the histogram shown in FIG. 7, the combined number of 5 sheets is the maximum frequency, but the intermediate number of combined sheet numbers of 2 to 4 is extremely small, and the minimum class (1 sheet) and the maximum class (5 sheets). It is greatly biased.

  In such a case, in the present embodiment, as shown in FIG. 9, the number of combined images is limited so as to suppress the dispersion of the entire histogram. The amount of movement of the class having the maximum frequency, that is, the combined number limit, may be calculated strictly by determining the maximum variance value of the histogram in advance, or is the center of the minimum class and the maximum class as shown in FIG. It may be determined simply as follows.

  As described above, according to the second embodiment of the present invention, it is possible to obtain an optimal composite number even when there is a pixel that cannot be correlated at the same location between a plurality of images. I can do it.

  Note that the first embodiment and the second embodiment according to the present invention may be applied by switching as appropriate according to the tendency of the acquired plurality of images.

  As described above, according to the image processing apparatus according to the present invention, when electronic image stabilization using a plurality of images is performed, the number of frames used for the synthesis is optimized, so that the Since the fluctuation of the SN ratio can be suppressed, it is possible to obtain a high quality image in which the fluctuation of the noise amount in the entire composite image is suppressed.

  The series of image processing for generating the above-described composite image can be executed by hardware, but can also be executed by software. In this case, a program for executing this series of image processing as software is stored in a recording medium in advance, and various kinds of computers such as a computer incorporated in predetermined hardware or a general-purpose personal computer are recorded from the recording medium. A predetermined process can be executed by installing the program.

DESCRIPTION OF SYMBOLS 100 Optical system 101 Image pick-up element 102 A / D conversion part 103 Image processing part 104 Frame memory 105 Motion detection part 106 Synthesis area classification part 107 Composite frame number selection part 108 Image composition part 110 Imaging data 111 Digital data 112 Image data 113 Composite image

Claims (8)

An image processing apparatus that generates a composite image by combining a plurality of images,
A motion detection means for detecting a motion amount of the target image with respect to the reference image when any one of the plurality of images is a reference image and another image is a target image;
After aligning the target image with the reference image based on the amount of movement, a correlation value indicating the correlation of the target image with the reference image is calculated in pixel units or region units, and the correlation value is calculated in advance. A region separation unit that generates region separation information indicating separation between a composite region determined to have high correlation and a non-composite region determined to have low correlation by comparing with a set threshold;
A composite number determination means for determining the composite number of the composite region based on the region classification information;
Image combining means for generating a composite image by combining images according to the composite number determined by the composite number determination means;
An image processing apparatus comprising: The composite number determination means calculates a non-composite area ratio that is a ratio of the non-composite area to the entire image, and determines the composite number of the composite area based on the non-composite area ratio;
The image processing apparatus according to claim 1. The composite number determining means calculates a ratio of the composite area in the entire image as a composite area ratio, and determines the composite number of the composite area based on a degree of dispersion of the composite area ratio between the plurality of images. thing,
The image processing apparatus according to claim 1. The image synthesizing unit generates a synthesized image by preferentially synthesizing an image having a higher correlation with the reference image among the target images according to the synthesized number determined by the synthesized number determining unit;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The image synthesizing unit generates a synthesized image by preferentially synthesizing from the image temporally closest to the reference image according to the synthesized number determined by the synthesized number determining unit;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image processing method for generating a composite image by combining a plurality of images,
A motion detection step of detecting a motion amount of the target image with respect to the reference image when any one of the plurality of images is a reference image and another image is the target image;
After aligning the target image with the reference image based on the amount of movement, a correlation value indicating the correlation of the target image with the reference image is calculated in pixel units or region units, and the correlation value is calculated in advance. An area classification step for generating area classification information indicating classification between a synthesis area determined to have high correlation and a non-synthesis area determined to have low correlation by comparing with a set threshold;
A composite number determination step for determining the composite number of the composite region based on the region classification information;
An image compositing step for generating a composite image by compositing images according to the composite number determined by the composite number determination means;
An image processing method comprising: In an image processing program for photographing a subject to acquire a plurality of images of the subject and generating a composite image by combining the acquired plurality of images,
An image processing program for generating a composite image by combining a plurality of images,
A motion detection step of detecting a motion amount of the target image with respect to the reference image when any one of the plurality of images is a reference image and another image is the target image;
After aligning the target image with the reference image based on the amount of movement, a correlation value indicating the correlation of the target image with the reference image is calculated in pixel units or region units, and the correlation value is calculated in advance. An area classification step for generating area classification information indicating classification between a synthesis area determined to have high correlation and a non-synthesis area determined to have low correlation by comparing with a set threshold;
A composite number determination step for determining the composite number of the composite region based on the region classification information;
An image compositing step for generating a composite image by compositing images according to the composite number determined by the composite number determination means;
An image processing program for causing a computer to execute.   A program recording medium on which the program according to claim 7 is recorded.

Images