JP2013258445A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform sufficient noise reduction processing while suppressing an afterimage based on the movement of a subject.SOLUTION: The image processing apparatus includes reliability calculation means 13 for calculating the reliability indicating correlation of the object pixel of a reference frame image and a corresponding pixel at a pixel position corresponding to the object pixel in a reference frame image, provisional cyclic coefficient calculation means 14 for calculating the provisional cyclic coefficient becoming the candidate value of cyclic coefficient when compounding the object pixel and the corresponding pixel, for each pixel, on the basis of the reliability, cyclic coefficient determination means 15 for determining the cyclic coefficient at each pixel position by performing expansion processing for a region consisting of pixels having a provisional cyclic coefficient lower than a predetermined value, and image composition means 17 for compounding the predetermined pixel and the corresponding pixel on the basis of the cyclic coefficient determined by the cyclic coefficient determination means.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, a so-called cyclic noise reduction method is known as a noise removal technique for image data. For example, motion detection is performed between the processed previous frame image and the current frame image, a cyclic coefficient indicating a combination ratio of these images is calculated based on the detection result, and the calculated cyclic coefficient is used. By combining, noise is reduced. As an example of an image processing apparatus that performs such cyclic noise reduction, Patent Document 1 discloses an apparatus that determines a current cyclic coefficient using a motion detection result and a stored previous cyclic coefficient. Yes.

International Publication No. 2006/025396

By the way, in general, in the cyclic noise reduction process, the cyclic coefficient is determined based on the motion compensation reliability such as the motion detection result and the inter-frame difference value so that the afterimage of the moving subject does not occur.
However, in practice, it is difficult to evaluate only the movement of the subject due to the influence of noise or the like. For example, there is an image between a region where motion compensation is correct but noisy (hereinafter referred to as “region 1”) and a region where a subject having a similar pixel value but not motion compensation is moved (hereinafter referred to as “region 2”). An evaluation value such as a difference value between them becomes an equivalent value. Therefore, if noise reduction processing is performed on these areas by applying the same cyclic coefficient, for example, noise is reduced in area 1, but afterimages occur in area 2. Further, there is a problem that the noise is not sufficiently reduced in the region 1 if the strength of the noise reduction processing is weakened so that no afterimage is generated, that is, if the cyclic coefficient is reduced.

  The present invention has been made in view of the above problem, and provides an image processing apparatus, an image processing method, and a program capable of performing sufficient noise reduction processing while suppressing an afterimage based on the motion of a subject. With the goal.

In order to achieve the above object, the present invention provides the following means.
The present invention provides an image acquisition unit that continuously acquires frame images, a processing unit that generates an output frame image by performing predetermined processing on the frame image acquired by the image acquisition unit, and the processing unit. An image storage unit that stores the generated output frame image, and the processing unit sets the frame image acquired by the image acquisition unit as a reference frame image, and is stored in the image storage unit and output immediately before A reliability calculation means for calculating a reliability indicating correlation between a target pixel of the base frame image and a corresponding pixel at a pixel position corresponding to the target pixel in the reference frame image, using the output frame image as a reference frame image; Based on the reliability, a temporary cycle for calculating, for each pixel, a temporary cyclic coefficient that is a candidate value of a cyclic coefficient when the target pixel and the corresponding pixel are combined. Coefficient calculation means and cyclic coefficient determination means for determining a cyclic coefficient at each pixel position by performing expansion processing on an area composed of pixels having the temporary cyclic coefficient lower than a predetermined value based on the temporary cyclic coefficient And an image synthesizing unit that synthesizes the predetermined pixel and the corresponding pixel based on the cyclic coefficient determined by the cyclic coefficient determining unit.

  According to the present invention, the reliability indicating the correlation between the target pixel of the base frame image and the corresponding pixel at the pixel position of the reference frame image corresponding to the target pixel is calculated, and the provisional cyclic coefficient is calculated based on the reliability. In addition to calculating for each pixel and performing an expansion process on an area composed of pixels having a provisional cyclic coefficient lower than a predetermined value based on the provisional cyclic coefficient, an appropriate value is finally obtained at each pixel position. Since the cyclic coefficient is determined, an appropriate cyclic coefficient can be determined for each pixel position. Accordingly, since an area having a small cyclic coefficient is prevented from becoming excessively large while suppressing afterimages, an image with sufficiently reduced noise can be acquired. Further, by performing the expansion process, a particularly high afterimage reduction effect can be obtained in the case of a subject that reciprocates.

In the above invention, it is preferable that the cyclic coefficient determining means determines, as the cyclic coefficient for each pixel position, a value indicating the lowest value among the temporary cyclic coefficients of a predetermined region including the pixel position for each pixel position.
For example, when the pixel values of the moving object and the background are close, the value of the provisional cyclic coefficient is high, and the correlation between the reference frame image and the reference frame image may actually be low even in an area where the apparent reliability is high. is there. In such a case, there is a possibility that an afterimage may occur in the synthesized image, but expansion processing, that is, for each pixel position, the one showing the lowest value among the provisional cyclic coefficients of the predetermined area including the pixel position. By determining the cyclic coefficient for each pixel position, the area where the provisional cyclic coefficient indicating the lowest value is set as the cyclic coefficient (that is, the moving object and the surrounding area) is widened. Also, the occurrence of afterimages can be suppressed.

In the above invention, the cyclic coefficient calculation means generates a temporary cyclic coefficient image indicating the distribution of the temporary cyclic coefficients, reduces the temporary cyclic coefficient image, performs expansion processing on the reduced temporary cyclic coefficient image, and performs expansion processing. It is preferable to enlarge the provisional cyclic coefficient image that has been subjected to and determine the cyclic coefficient based on the enlarged provisional cyclic coefficient image.
By doing in this way, the amount of calculation of expansion processing can be reduced.

  The present invention also includes a step of continuously acquiring frame images, a step of generating an output frame image by performing predetermined processing on the frame image, and a step of accumulating the generated output frame image And the step of generating the output frame image includes the acquired frame image as a reference frame image, the output frame image output immediately before as a reference frame image, and target pixels of the reference frame image; Calculating the reliability indicating the correlation between the target pixel and the corresponding pixel at the pixel position corresponding to the target pixel in the reference frame image, and combining the target pixel and the corresponding pixel based on the reliability Calculating a provisional cyclic coefficient that is a candidate value for the cyclic coefficient for each pixel, and based on the provisional cyclic coefficient, the provisional cyclic coefficient A step of determining a cyclic coefficient at each pixel position by performing expansion processing on an area composed of pixels lower than a predetermined value, and the predetermined pixel based on the cyclic coefficient determined by the cyclic coefficient determination means And an image processing program for causing a computer to synthesize the corresponding pixel and the corresponding pixel.

  The present invention also includes a step of continuously acquiring frame images, a step of generating an output frame image by performing predetermined processing on the frame image, and a step of accumulating the generated output frame image And the step of generating the output frame image includes the acquired frame image as a reference frame image, the output frame image output immediately before as a reference frame image, and target pixels of the reference frame image; Calculating the reliability indicating the correlation between the target pixel and the corresponding pixel at the pixel position corresponding to the target pixel in the reference frame image, and combining the target pixel and the corresponding pixel based on the reliability Calculating a provisional cyclic coefficient that is a candidate value for the cyclic coefficient for each pixel, and based on the provisional cyclic coefficient, the provisional cyclic coefficient A step of determining a cyclic coefficient at each pixel position by performing expansion processing on an area composed of pixels lower than a predetermined value, and the predetermined pixel based on the cyclic coefficient determined by the cyclic coefficient determination means And a step of combining the corresponding pixels.

  According to the present invention, it is possible to perform sufficient noise reduction processing while suppressing afterimages based on the movement of the subject.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is explanatory drawing which showed a mode that the input frame image was synthesize | combined sequentially in the image processing apparatus which concerns on one Embodiment of this invention. It is a flowchart in the case of reliability calculation in the image processing apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which showed the mode at the time of the reliability calculation in the image processing apparatus which concerns on one Embodiment of this invention. It is a graph which shows the relationship between the reliability in the image processing apparatus which concerns on one Embodiment of this invention, and temporary cyclic coefficient (beta) N. It is explanatory drawing which shows the mode of the image process in the image processing apparatus which concerns on one Embodiment of this invention. As a comparative example of FIG. 6, it is explanatory drawing which shows an example of the image process in the conventional image processing apparatus. It is explanatory drawing which shows the mode of the image process in the image processing apparatus which concerns on the modification of one Embodiment of this invention. It is explanatory drawing which shows an example of an image process in the image processing apparatus which concerns on the modification of one Embodiment of this invention.

  An image processing apparatus according to an embodiment of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a schematic configuration of the image processing apparatus. As shown in FIG. 2, the image processing apparatus appropriately combines the input frame image and the previously output frame image to generate a frame image, sequentially outputs the generated frame image, and cyclically outputs the frame image. That is, a moving image is output by repeating.

Therefore, as shown in FIG. 1, the image processing apparatus includes an image acquisition unit 2, a processing unit 3 that generates an output frame image by performing predetermined processing on the frame image acquired by the image acquisition unit, The image storage unit 4 stores an output frame image generated by the processing unit, that is, processed image data (hereinafter referred to as “previous frame image (reference frame image)”).
Further, the processing unit 3 performs a predetermined process on the acquired frame image, and includes a motion compensation unit 12, a reliability calculation unit 13, a provisional cyclic coefficient calculation unit 14, a cyclic coefficient determination unit 15, and a spatial filter unit 16. And an image composition unit 17.
The image acquisition unit 2 continuously acquires frame images (hereinafter referred to as “current frame image (reference frame image)”) captured by an imaging device (not shown), and the acquired current frame image is used as the motion compensation unit 12 and the space. Output to the filter unit 16.

The motion compensation unit 12 includes a motion vector calculation unit 11 that calculates a motion vector between the current frame image acquired by the image acquisition unit 2 and the previous frame image stored in the memory 18. Then, the motion compensation unit 12 aligns the previous frame image and the current frame image based on the motion vector calculated by the motion vector calculation unit 11. That is, the cutout position of the previous frame image is determined so as to cancel the motion vector calculated by the motion vector calculation unit 11, and the small area of the previous frame image is determined from the arbitrary cutout area of the current frame image and the determined cutout position. Is cut out, the previous frame image and the current frame image are aligned, and the center pixel of the small area of the previous frame image is output to the image composition unit 17.
Here, the motion compensation may be performed for each small area based on the motion vector calculated by the block matching method or the like for each small area between the current frame image and the previous frame image. Based on the motion vector statistics for each region, motion compensation may be performed using the most frequent motion vector as a global motion vector for the entire image.

The reliability calculation unit 13 uses the region motion-compensated by the motion compensation unit 12 to determine the reliability of motion compensation, that is, the corresponding pixel at the pixel position corresponding to the target pixel of the current frame image and the target pixel of the previous frame image. The reliability indicating the correlation with is calculated. More specifically, the reliability calculation unit 13 calculates the reliability according to the flowchart of FIG.
That is, as shown in FIG. 3, the reliability calculation unit 13 uses the small region of the current frame image as a reference region in the small region of the current frame image and the previous frame image that are aligned and extracted by the motion compensation unit 12, and A small area of the previous frame image is set as a reference area (step S101). Subsequently, an absolute value sum (SAD) of differences between the reference area and the reference area is calculated (step S102).

  In the reliability calculation example shown in FIG. 4, a small area of the current frame image contains a lot of noise, and a small area of the previous frame image has reduced noise. SAD is calculated between these areas, and the reliability is calculated according to the value of SAD. It is not always necessary to calculate SAD in order to calculate the reliability. For example, the sum of squares of differences (SSD) or normalized cross-correlation (NCC) that can evaluate the degree of difference or similarity is calculated. Also good.

  In the next step S103, the reliability is calculated based on the calculated SAD. When evaluating the dissimilarity as in SAD or SSD, the smaller the evaluation value, the higher the reliability, and the higher the evaluation value, the lower the reliability. Further, when the evaluation value for calculating the reliability is already calculated by the motion vector calculation unit 12, the evaluation value in the motion vector calculation unit may be used as it is without calculating the evaluation value again. .

  The temporary cyclic coefficient calculation unit 14 calculates the temporary cyclic coefficient βN based on the reliability calculated by the reliability calculation unit 13. Here, the provisional cyclic coefficient βN is used to calculate a cyclic coefficient that is a combination ratio when combining the target pixel of the current frame image and the corresponding pixel at the corresponding pixel position in the previous frame image corresponding to the target pixel. The provisional cyclic coefficient βN, which is a candidate value, is output to the cyclic coefficient determination unit 15.

  FIG. 5 shows the relationship between the reliability and the provisional cyclic coefficient βN. As shown in FIG. 5, the higher the reliability, the higher the provisional cyclic coefficient βN of the target pixel of the previous frame image. Note that the calculated temporary cyclic coefficients for all the pixels may be reduced by a near neighbor method or bilinear processing in a reduction unit (not shown) and temporarily stored in a memory (not shown). In this case, the reduced provisional cyclic coefficient can be used after the area is enlarged by an enlargement unit (not shown).

  Based on the temporary cyclic coefficient βN input from the temporary cyclic coefficient calculation unit 14, the cyclic coefficient determination unit 15 performs an expansion process on an area including pixels whose temporary cyclic coefficient is lower than a predetermined value. A position cyclic coefficient αN is determined. The determined cyclic coefficient αN is output to the spatial filter unit 16 and the image composition unit 17. Note that the “predetermined value” and the range where the expansion process is performed are appropriately determined in consideration of the relationship between noise and an afterimage.

The spatial filter unit 16 applies the spatial filter by controlling the strength of the spatial filter applied to the current frame image according to the determined cyclic coefficient αN, and outputs the result to the image composition unit 17.
The image synthesizing unit 17 synthesizes the spatial filter output pixel of the input image of the current frame and the corresponding pixel of the previous frame image input from the motion compensation unit 12 using the cyclic coefficient αN, and displays it as the current frame image in the display system. Are stored in the image storage unit 4.

Hereinafter, the operation of the above-described image processing apparatus will be described with reference to FIG.
Here, it is assumed that the frame image is acquired under the following conditions. That is, it is assumed that the image is taken with a camera fixed to a tripod, the background B is not moving, and the moving image scene is when the ball O as a subject rolls from left to right.
In addition, since the input frame image includes a lot of noise N, and the ball O rolls and moves to the right, a blur G due to movement occurs at the left edge of the ball O due to the relationship with the shutter speed. Yes. It is assumed that the luminance value of the blurred area at the left edge of the subject is similar to the luminance value of the background. Further, it is assumed that a motion vector between images is not obtained in any region (motion vector is “0”).

In this case, image processing on the (N-1) th frame image (hereinafter referred to as “N frame”) is performed as follows.
The motion compensation unit 12 performs alignment between the N-1 frame and the N-2th frame image (hereinafter referred to as “N-2 frame”), and based on this, the reliability calculation unit 13 performs the reliability. Is calculated. Based on this reliability, the temporary cyclic coefficient calculation unit 14 calculates the temporary cyclic coefficient βN−1 and outputs the calculated temporary cyclic coefficient βN−1 to the cyclic coefficient determination unit 15.

  In the cyclic coefficient determination unit 15, based on the previously calculated temporary cyclic coefficient βN−1 for each pixel, an area composed of pixels whose temporary cyclic coefficient β−1 in the N−1 frame is lower than a predetermined threshold value. Is expanded and determined as the cyclic coefficient (second from the right in the figure) αN−1 of the N−1 frame. In this way, by expanding the region having a low value of the temporary cyclic coefficient, the cyclic coefficient becomes lower in a region larger than the moving subject region that can be determined by calculating the temporary cyclic coefficient. That is, an afterimage does not occur in the vicinity of a moving subject because the composition ratio with the corresponding pixel of the previous frame is low. Thereafter, the same processing is performed in the N frame processing.

On the other hand, as a reference example, a case where image processing is performed on image data captured under the same conditions without using a provisional cyclic coefficient will be described with reference to FIG.
When processing the N-1 frame, the cyclic coefficient (middle in the figure) is calculated from the reliability between the input image (left in the figure) of the N-1 frame and the output image (right in the figure) of the N-2 frame. ) Is calculated, a value obtained by multiplying the corresponding pixel of the output image of the N-2 frame by the cyclic coefficient (αN) using the calculated cyclic coefficient (αN), and (1-αN) of the N-1 frame. The value multiplied by the output pixel of the target pixel from the spatial filter unit of the input image is added.

  The pixels of the addition result are processed for all pixels as pixels of the output image of N-1. At this time, the reliability is calculated between the left edge region of the subject of the output image of the N-2 frame and the background portion slightly left of the left edge of the subject of the input image of the N-1 frame. Since the SAD calculation result for calculating the reliability is similar in luminance value, the SAD value becomes a small value to some extent, and the reliability is not lowered.

  Therefore, when the cyclic coefficient is calculated as shown in FIG. 5, the cyclic coefficient does not become a completely low value. Using this cyclic coefficient, the synthesized result is an output image of N-1 frame (right in the figure). The same processing is performed in processing of N frames. As a result, in the output images of the N-1 frame and the N frame, the edge of the subject blurred in the output image of the previous frame is generated in the image on the left side of the moving subject.

  As described above, according to the present embodiment, an appropriate composition ratio control is performed in the vicinity of a moving subject area by setting and synthesizing a cyclic coefficient obtained by expanding an area having a low value of a temporary cyclic coefficient around the area. Thus, while suppressing the afterimage, it is possible to prevent a region having a small cyclic coefficient from being excessively widened and obtain a sufficient noise reduction effect.

(Modification)
In the above-described embodiment, an area composed of pixels whose provisional cyclic coefficient is lower than a predetermined threshold value is expanded and determined as the cyclic coefficient. However, the cyclic coefficient determination method is not limited to this.
As shown in FIG. 8, the cyclic coefficient determination unit 15 replaces the lowest temporary cyclic coefficient in the predetermined range including the target pixel with the cyclic coefficient of the target pixel, so that the temporary cyclic coefficient has a low value. The region can be expanded.

  In the above-described embodiment, the provisional cyclic coefficient may be stored in the memory and expanded as it is, or may be temporarily reduced and stored in the memory, and then expanded and expanded. Further, as shown in FIG. 9, after the temporary cyclic coefficient is reduced and stored in the memory, the expansion coefficient obtained by the expansion process by the cyclic coefficient determination unit can be used as the cyclic coefficient.

DESCRIPTION OF SYMBOLS 2 Image acquisition part 3 Processing part 4 Image storage part 11 Motion vector calculation part 12 Motion compensation part 13 Reliability calculation part 14 Temporary cyclic coefficient calculation part 15 Cyclic coefficient determination part 16 Spatial filter part 17 Image composition part

Claims (5)

An image acquisition unit for continuously acquiring frame images;
A processing unit that generates an output frame image by performing predetermined processing on the frame image acquired by the image acquisition unit;
An image storage unit for storing the output frame image generated by the processing unit,
The processing unit is
The frame image acquired by the image acquisition unit is set as a reference frame image, the output frame image output immediately before is set as a reference frame image, and the target pixel of the reference frame image and the target pixel in the reference frame image are corresponded A reliability calculation means for calculating a reliability indicating the correlation with the corresponding pixel at the pixel position;
Temporary cyclic coefficient calculating means for calculating, for each pixel, a temporary cyclic coefficient that is a candidate value of a cyclic coefficient when combining the target pixel and the corresponding pixel based on the reliability;
Cyclic coefficient determining means for determining a cyclic coefficient at each pixel position by performing an expansion process on a region composed of pixels having the temporary cyclic coefficient lower than a predetermined value based on the temporary cyclic coefficient;
Image combining means for combining the predetermined pixel and the corresponding pixel based on the cyclic coefficient determined by the cyclic coefficient determination means;
An image processing apparatus comprising:   2. The image according to claim 1, wherein the cyclic coefficient determination unit determines, for each pixel position, a value indicating the lowest value among the provisional cyclic coefficients of a predetermined region including each pixel position as a cyclic coefficient for each pixel position. Processing equipment.   The cyclic coefficient calculation means generates a temporary cyclic coefficient image indicating the distribution of the temporary cyclic coefficients, reduces the temporary cyclic coefficient image, performs expansion processing on the reduced temporary cyclic coefficient image, and performs the temporary processing on which the expansion processing has been performed. The image processing apparatus according to claim 1, wherein the cyclic coefficient image is enlarged and the cyclic coefficient is determined based on the enlarged provisional cyclic coefficient image. Continuously acquiring frame images;
Generating an output frame image by performing predetermined processing on the frame image;
Accumulating the generated output frame image, and
Generating the output frame image comprises:
Using the acquired frame image as a reference frame image, and the output frame image output immediately before as a reference frame image, a target pixel of the base frame image, and a corresponding pixel at a pixel position corresponding to the target pixel in the reference frame image Calculating a confidence level indicating the correlation of
Calculating, based on the reliability, a provisional cyclic coefficient that is a candidate value of a cyclic coefficient when combining the target pixel and the corresponding pixel for each pixel;
Determining a cyclic coefficient at each pixel position by performing an expansion process on a region composed of pixels having the temporary cyclic coefficient lower than a predetermined value based on the temporary cyclic coefficient;
Combining the predetermined pixel and the corresponding pixel based on the cyclic coefficient determined by the cyclic coefficient determination means;
An image processing program for causing a computer to execute. Continuously acquiring frame images;
Generating an output frame image by performing predetermined processing on the frame image;
Accumulating the generated output frame image, and
Generating the output frame image comprises:
Using the acquired frame image as a reference frame image, and the output frame image output immediately before as a reference frame image, a target pixel of the base frame image, and a corresponding pixel at a pixel position corresponding to the target pixel in the reference frame image Calculating a confidence level indicating the correlation of
Calculating, based on the reliability, a provisional cyclic coefficient that is a candidate value of a cyclic coefficient when combining the target pixel and the corresponding pixel for each pixel;
Determining a cyclic coefficient at each pixel position by performing an expansion process on a region composed of pixels having the temporary cyclic coefficient lower than a predetermined value based on the temporary cyclic coefficient;
Combining the predetermined pixel and the corresponding pixel based on the cyclic coefficient determined by the cyclic coefficient determination means;
An image processing method comprising:

Images