JP2019067268A - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Abstract

To provide an image processing apparatus capable of performing noise suppression with high resolution at an edge portion of an image.SOLUTION: The image processing apparatus includes: a first detection unit that detects first edge information which is information on an edge portion of a subject from first image data; a generation unit that generates second image data by reducing the first image data; a second detection unit that detects second edge information which is information on an edge portion of the subject from the second image data; a first suppression unit that suppresses noise of predetermined pixel data in the first image data by referring to pixel data of a first reference region around the predetermined pixel data; and a setting unit that sets the first reference region based on the first edge information and the second edge information.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for performing noise correction of an image signal.

  In recent years, the number of pixels of the imaging device has been increased, and the pixels tend to be miniaturized. Due to the miniaturization of the pixels, noise of the image signal obtained from the imaging device increases. Therefore, as a method of suppressing this noise by signal processing, a method of using multi-rate signal processing, a method of suppressing a noise component according to the shape of an object, and the like are known.

  Patent Document 1 discloses a method of frequency separating images so that frequency bands overlap. For example, when using a Gaussian pyramid, a plurality of reduced images with different reduction ratios are generated from the original image. When the Gaussian pyramid is used, DC components exist in all band images. A noise suppression process is performed on each of the reduced images, and a band image is synthesized using a synthesis ratio obtained based on the extracted edge signal to obtain a synthesized image.

  According to the technology described in Patent Document 1 described above, by adjusting the composition ratio, for example, high frequency noise can be reduced by using many images in which band limitation is performed in the low frequency region in the flat portion of the image. The amount of noise can be relatively easily controlled, such as effectively reduced.

  Further, Patent Document 2 discloses a method of suppressing a noise component according to the shape of a subject. In this method, an image is not blurred more than necessary by controlling an area to be referred to by noise suppression. It is possible to perform noise suppression.

JP, 2009-199104, A JP, 2009-20605, A

  In the technique described in Patent Document 1, in order to suppress noise in the edge portion of the image, it is necessary to use many images band-limited to the low band, and there is a problem that the edge portion is blurred. On the other hand, in the technique described in Patent Document 2, since noise is suppressed according to the shape of the edge portion, it is possible to prevent the edge portion from being blurred more than necessary, but in a high frequency image having a large amount of noise There is a problem that the accuracy of the shape determination of the image is deteriorated and the resolution feeling is deteriorated.

  The present invention has been made in view of the above-described problem, and an object thereof is to provide an image processing apparatus capable of performing noise suppression with a high sense of resolution at an edge portion of an image.

  An image processing apparatus according to the present invention comprises first detection means for detecting first edge information which is information on an edge portion of a subject from first image data, and reducing the first image data. A generation unit configured to generate second image data; a second detection unit configured to detect second edge information, which is information on an edge portion of a subject, from the second image data; First suppressing means for suppressing noise of predetermined one of the pixel data with reference to pixel data of a first reference region therearound, and based on the first edge information and the second edge information And setting means for setting the first reference area.

  According to the present invention, it is possible to perform noise suppression with a high sense of resolution at an edge portion of an image.

FIG. 1 is a block diagram showing a configuration of a digital camera which is a first embodiment of an image processing apparatus of the present invention. FIG. 2 is a block diagram showing the configuration of a noise suppression processing circuit. The flowchart which shows the flow of operation of noise suppression processing. The figure which shows the example of the pattern used for edge detection. The figure which shows the example of the reference area | region used for noise suppression. The figure which shows the example of the pattern used for edge detection. 6 is a flowchart showing a process of determining a noise suppression reference area in the first embodiment. The figure which shows the example of the reference area | region used for noise suppression. The figure which shows the example of the reference area | region used for noise suppression. The figure which shows the example of a synthetic | combination ratio. The flowchart which shows the process which determines the reference area | region of noise suppression in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

First Embodiment
FIG. 1 is a block diagram showing the configuration of a digital camera 100 which is a first embodiment of the image processing apparatus of the present invention.

  In FIG. 1, a digital camera 100 includes an imaging unit 103 including a zoom lens, a photographing optical system 101 including a focus lens, a shutter 102 having an aperture function, and a CCD or CMOS element converting an optical image into an electric signal. Equipped with An A / D converter 104 connected to the imaging unit 103 converts an analog signal from the imaging unit 103 into a digital signal. An image processing unit 105 performs various types of image processing such as white balance processing, γ processing, and color correction processing on the image data output from the A / D converter 104. The image memory 106 temporarily stores the image data output from the image processing unit 105, and the memory control unit 107 controls storage of the image data to the image memory 106 and control of reading of the image data from the image memory 106. The D / A converter 108 converts the input digital signal into an analog signal. A display 109 such as an LCD displays the image signal converted into an analog signal by the D / A converter 108. The codec unit 110 compresses and encodes image data. An interface I / F 111 is an interface with a recording medium 112 such as a memory card or a hard disk. The system control unit 50 controls the entire system of the digital camera 100.

  The operation unit 120 has a plurality of operation members for the user to input various operation instructions. The power switch 121 is an operation member for turning on / off the power of the digital camera 100 by the user. The power supply control unit 122 detects, for example, whether or not the battery is attached to the power supply unit 123, the type of the battery, and the remaining amount of the battery. The non-volatile memory 124 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. A system timer 125 measures the time used for various controls and the time of a built-in clock. The system memory 126 is used to expand constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 124, and the like.

  Next, the flow of the basic operation at the time of photographing an object in the digital camera 100 configured as described above will be described.

  The imaging unit 103 photoelectrically converts light incident through the photographing optical system 101 and the shutter 102, and outputs the light as an input image signal to the A / D converter 104. The A / D converter 104 converts an analog image signal output from the imaging unit 103 into a digital image signal and outputs the digital image signal to the image processing unit 105. The image processing unit 105 performs color conversion processing such as white balance, noise suppression processing, edge enhancement correction processing, and γ processing on the image data from the A / D converter 104 or the image data read from the memory control unit 107. Etc. Then, it outputs image data of Bayer RGB data and any one of luminance / color difference signals Y, R-Y, and B-Y. The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. Further, in the image processing unit 105, predetermined operation processing is performed using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation result. As a result, TTL (through the lens) AF (Auto Focus) processing, AE (Auto Exposure) processing, etc. are performed. The image processing unit 105 further analyzes the captured image data, estimates a light source, and performs AWB (auto white balance) processing based on the estimated light source.

  The image memory 106 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 109. Also, the D / A converter 108 converts data for image display stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display according to the analog signal from the D / A converter 108 on a display such as an LCD. The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on a standard such as MPEG.

  In addition to the above-described basic operation, the system control unit 50 implements each process of the present embodiment described later by executing the program stored in the nonvolatile memory 124 described above. The program referred to here is a program for executing various flowcharts to be described later in the present embodiment. At this time, constants and variables for the operation of the system control unit 50, programs read out from the non-volatile memory 124 and the like are expanded in the system memory 126. The above is the block configuration of the digital camera 100 and the basic operation.

  Next, noise suppression processing performed by the image processing unit 105 will be described.

  In the noise suppression process, a plurality of images with different frequency bands are used. In this process, a plurality of reduced images of different frequency bands are generated from an input image, which is an equal-magnification image. Then, band-by-band noise suppression processing using a reduced image of a specific reduction ratio and a reduced image of a larger one-step reduction ratio is repeatedly performed in order from the largest reduction ratio.

  Here, the above operation will be more specifically described with reference to FIG. FIG. 2 is a block diagram of the noise suppression processing circuit 200 included in the image processing unit 105. As shown in FIG.

  As shown in FIG. 2, after the reduction unit 201 applies pre-filters in the horizontal and vertical directions to the input image, the reduction unit 201 reduces the size to 1⁄2 in the horizontal and vertical directions to 1⁄2 reduced image. Generate In addition, the reduction unit 202 performs the same process on the 1⁄2 reduced image, generates a 1⁄4 reduced image, and generates images of different frequency bands. Next, using the 1⁄2 reduced image and the 1⁄4 reduced image having a one-step reduction rate larger than that image, a band-wise noise-suppressed image (output of the image combining unit 214) for the 1⁄2 reduced image is generated Do. Next, using the equal-magnification image and the image (output of the enlargement unit 215) of the result of the band-wise noise suppression processing for the 1/2 reduced image, the band-wise noise suppression processed image (output of the image combining unit 216 for the equal-magnification image ) Are output as the output of the noise suppression processing circuit 200. In addition, although noise suppression processing in the frequency band is performed on an image with the largest reduction ratio, band-wise noise suppression processing using images in different frequency bands is not performed. In this example, the 1⁄4 reduced image is the reduced image with the largest reduction rate, and there is no further reduced image, so the band-wise noise suppression process is not performed on the 1⁄4 reduced image. Further, although this example shows the case of using up to 1⁄4 reduced images, the same processing is performed using an image with a larger reduction ratio, such as 1⁄8 reduced images or 1/16 reduced images. May be

  Next, band-wise noise suppression processing using the 1/2 reduced image and the 1/4 reduced image will be described using the flowchart of FIG.

  In step S301, the edge detection unit 205 performs edge detection for detecting information on the edge portion of the subject on the 1⁄4 reduced image, and obtains two types of edge direction and edge intensity. This process is performed, for example, by pattern matching using a 5 tap pattern shown in FIG. 4 for each pixel of interest. For each search area around the pixel of interest shown in FIGS. 4 (a) to 4 (d), the difference absolute value between the pixel data of the pixel of interest and the pixel data of each pixel included in the search area is calculated; Calculate the average value. Next, a pattern in which the average value of the four calculated difference absolute values is minimum is taken as the edge direction of the pixel of interest, and the average value of the difference absolute values of patterns orthogonal to the pattern in the edge direction is taken as the edge intensity of the pixel of interest . For example, when the average value of the absolute values of differences in the pattern of FIG. 4A is minimum, it is determined that the edge is extending in the horizontal direction, and the horizontal direction is taken as the edge direction. The average value of the difference absolute values using the pattern of FIG. 4B is taken as the edge strength of the pixel of interest. In addition, although the average value process of the difference absolute value is performed for normalization, not only the average value process but also each reduction is performed in view of the fact that the noise amount is different in each reduced image due to the difference in the frequency band. Normalization according to the noise amount of the image may be performed.

  In step S302, the noise suppression unit 210 performs, for example, the following noise suppression processing on the 1⁄4 reduced image. First, an area to be referred to in noise suppression is determined based on the edge strength calculated in step S301. If the edge strength is larger than the predetermined threshold TH1, the pixel of interest is determined to be an edge area, and a search area in the edge direction of 5 taps detected in step S301 is set as a reference area for noise suppression. On the other hand, when the edge strength is TH1 or less, it is determined that the target pixel is not an edge area, and the area shown in FIG. 5 is set as a reference area. Thereafter, among the pixels in the reference area, the values of the pixels whose difference with the value of the pixel of interest is within a predetermined value are averaged, and the calculation to replace with the value of the pixel of interest is sequentially performed while changing the pixel of interest. Noise suppression processing is performed by such a method.

  In step S303, the edge detection unit 204 performs edge detection on the 1/2 reduced image. The processing content is the same as the edge detection for the 1/4 reduced image performed in step S301, but the pattern used for the edge detection is different. The pattern of 5 tap shown in FIG. 4 is used for the 1⁄4 reduced image, but the pattern of 7 tap shown in FIG. 6 is used for the 1⁄2 reduced image. This is because in an image with a large reduction ratio, the edges become thinner than in an image with a small reduction ratio, and it becomes difficult to detect an edge. Therefore, as the reduction ratio increases, the width of the number of taps used for edge detection is narrowed to appropriately detect the edge.

  In step S304, the noise suppression unit 209 performs noise suppression processing on the 1⁄2 reduced image. As in the noise suppression process of step S302, an area to be referred to for each pixel of interest is determined, and among the pixel values in the reference area, values having differences with the value of the pixel of interest within a predetermined value are averaged. An operation is performed to replace the value of the pixel of interest. However, how to determine the reference area (how to set) is different from the noise suppression process of step S302, and is determined as shown in the flowchart of FIG. 7, for example.

  In step S701, it is determined whether the edge strength detected from the 1/2 reduced image by the edge detection unit 204 is larger than a predetermined threshold TH2. If it is larger than TH2, it is determined that the area is an edge area, and the process proceeds to step S702. In step S702, it is detected whether or not the edge strength detected from the 1⁄4 reduced image by the edge detection unit 205 is larger than a predetermined threshold TH3, and from the edge direction detected from the 1⁄2 reduced image and the 1⁄4 reduced image. It is determined whether the detected edge direction is equal. If the edge strength detected from the 1/4 reduced image is greater than TH3 and the edge direction detected from the 1/2 reduced image and the edge direction detected from the 1/4 reduced image are equal, It is determined that the reliability of the number of taps used in edge detection is high. Then, the process proceeds to step S703. In step S703, noise suppression of the 1/2 reduced image is performed using the 9 tap reference area (FIG. 8) corresponding to the edge direction common to the 1/2 reduced image detected from the 1/4 reduced image. For example, when the edge direction detected from the 1⁄4 reduced image corresponds to FIG. 4B, the area shown in FIG. 8B is used as the reference area. In addition, since the processing of 5 tap in the 1/4 reduced image corresponds to 9 tap in the 1/2 reduced image, the size of the reference area is 9 tap in the noise suppression of the 1/2 reduced image.

  When the condition of step S702 is not satisfied (when the edge strength of the 1/4 reduced image is equal to or less than a predetermined threshold or when the edge direction detected from the 1/2 reduced image is different from the edge direction detected from the 1/4 reduced image ) Determines that the reliability of the number of taps used in the edge detection of the 1/2 reduced image is high, and performs noise suppression with the search region in the edge direction detected from the 1/2 reduced image as a reference region. If the edge strength of the 1/2 reduced image is less than or equal to TH2 (less than a predetermined value) in step S701, it is determined that the region is not an edge region, and the process proceeds to step S705 to perform noise suppression using the region shown in FIG. Conduct.

  Compared with a 1⁄2 reduced image, the 1⁄4 reduced image has a limited band and a small amount of noise, so the edge detection accuracy is high. On the other hand, the 1/2 reduced image has less blurring of the image than the 1/4 reduced image. Therefore, by using the edge information detected in the 1⁄4 reduced image also in the 1⁄2 reduced image, it is possible to perform the noise suppression process applying the high-accuracy edge detection result to the image with less blur. . Note that the edge strength signal of the 1⁄4 reduced image is enlarged by, for example, bilinear enlargement processing by the enlargement unit 207 so as to have the same size as the edge intensity signal of the 1⁄2 reduced image. The noise suppression process is performed by determining the reference area as described above.

  Returning to the explanation of FIG. 3, in step S305, the enlargement unit 213 enlarges the noise-suppressed 1/4 reduced image to the same size as the noise-suppressed 1/2 reduced image. For example, a bilinear enlargement process may be used for the enlargement process.

  In step S306, the combining ratio calculation unit 212 uses the edge intensity calculated by the edge detecting unit 204 to calculate the combining ratio as shown in FIG. As shown in FIG. 10, the larger the edge strength is, the more the signal of the noise-suppressed 1⁄2 reduced image is used (the combination ratio is increased), and the signal of the 1⁄4 reduced image is used less. Also, conversely, the smaller the edge strength, the more the signal of the noise-suppressed 1/4 reduced image is used, and the smaller the signal of the 1/2 reduced image is used. The thresholds TH4 and TH5 in FIG. 10 can be determined experimentally, for example.

  In step S307, the image combining unit 214 combines the noise-suppressed 1⁄2 reduced image and the image enlarged in step S305 based on the combining ratio calculated in step S306, and the band-specific noise for the 1⁄2 reduced image. Generate a suppression processed image. As described above, band-wise noise suppression processing is performed on the 1⁄2 reduced image using the 1⁄2 reduced image and the 1⁄4 reduced image.

  Also, although the band-wise noise suppression processed image for the equal-magnification image is generated using the equal-magnification image and the image of the band-wise noise suppression result for the 1⁄2 reduced image, also in this case, the same processing can be performed It is. The processes of the edge detecting unit 203, the noise suppressing unit 208, and the combining ratio calculating unit 211 for the same size image are the same as the processes of the edge detecting unit 204, the noise suppressing unit 209, and the combining ratio calculating unit 212 for the 1/2 reduced image. It is. The noise suppression unit 208 uses the edge information of the equal-magnification image detected by the edge detection unit 203 and the edge information of the 1⁄2 reduced image detected by the edge detection unit 204 enlarged by the enlargement unit 206. A noise suppression process is performed by determining a suppression reference area. Further, in the processing of the image combining unit 216, the noise-suppressed equal-magnification image and the image in which the band-wise noise suppression processing image for the 1⁄2 reduced image is enlarged by the enlargement unit 215 are synthesized A noise suppression processed image is generated. Then, this is used as the output of the noise suppression processing circuit 200. The above is the noise suppression processing in the first embodiment.

Second Embodiment
In the first embodiment, when performing noise suppression of an image of a specific size, noise is detected using edge information detected from the image and edge information detected from a reduced image having a larger reduction ratio by one step than the image. The case of controlling the number of taps in the region to be referred to in the suppression has been described. In the second embodiment, the case of controlling the number of taps of a region to be referred to in noise suppression and the direction to be referred to will be described.

  The second embodiment is the same as the first embodiment except the noise suppression process (step S304) in the first embodiment, so only different parts will be described. In the present embodiment, the noise suppression process of step S304 is performed according to the procedure of the flowchart of FIG.

  In step S1101, it is determined whether the edge strength of the 1/2 reduced image detected in step S303 is larger than a predetermined threshold TH6. If larger than TH6, it is determined that the edge area can be detected from the 1/2 reduced image, and the process advances to step S1102, and noise suppression is performed using the search area in the edge direction detected from the 1/2 reduced image as a reference area. If the edge strength of the 1/2 reduced image is equal to or less than TH6, the process advances to step S1103, and it is determined whether the edge strength of the 1/4 reduced image is larger than a predetermined threshold TH7. If larger than TH7, it is determined that the edge area can be detected from the 1⁄4 reduced image, and the process advances to step S1104. Then, noise suppression of the 1⁄2 reduced image is performed with the region of 9 taps corresponding to the edge direction detected from the 1⁄4 reduced image as a reference region. If the edge strength of the 1⁄4 reduced image is equal to or less than TH7, it is determined that the region is not an edge region, and the process advances to step S1105 to perform noise suppression of the 1⁄2 reduced image using the region shown in FIG. The above is the noise suppression processing in the second embodiment.

  In the above embodiment, edge detection is performed using the patterns shown in FIG. 4 and FIG. 6, but the type of pattern is not limited to this, and the number of taps and the search area may be changed as appropriate. Alternatively, the edge strength may be calculated by filter processing such as a laplacin filter without using a pattern.

  Further, in the above embodiment, when performing band-by-band noise suppression processing on an image of a specific size, it is determined that the reliability of edge information detected from a reduced image having a one-step reduction rate larger than that image is high. In the above, the case has been described in which noise suppression is performed using the number of taps expanded in order to match the number of taps used in a reduced image having a large one-stage reduction rate to the size of the image. However, for example, in order to prevent an increase in processing amount, the noise suppression processing may be performed without expanding the number of taps.

  Further, the method of determining the reference region when performing noise suppression is not limited to the methods described in the first and second embodiments, and edge information detected from an image of a specific size and a one-step reduction rate than that image Any method may be used as long as edge information detected from a large reduced image is used.

(Other embodiments)
Furthermore, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by the process to be executed. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

50: system control unit, 100: digital camera, 101: photographing optical system, 102: shutter, 103: imaging unit, 104: A / D conversion unit, 105: image processing unit, 106 ... image memory, 107: memory Control unit, 200: noise suppression processing circuit

Claims (14)

First detection means for detecting first edge information, which is information of an edge portion of a subject, from the first image data;
Generation means for reducing the first image data to generate second image data;
Second detection means for detecting second edge information which is information of an edge portion of a subject from the second image data;
First suppressing means for suppressing noise of predetermined pixel data of the first image data with reference to pixel data of a first reference region therearound;
Setting means for setting the first reference area based on the first edge information and the second edge information;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the first edge information and the second edge information include at least one of an edge direction and an edge strength of an edge portion of a subject.   3. The image processing apparatus according to claim 2, wherein the setting unit sets the first reference area along an edge direction included in the first edge information and the second edge information. .   The setting is performed when the edge strength included in the second edge information is larger than a predetermined threshold and the edge direction included in the first edge information is the same as the edge direction included in the second edge information. 4. The image processing apparatus according to claim 3, wherein the means sets the first reference area along an edge direction common to the first edge information and the second edge information.   When the edge strength included in the second edge information is equal to or less than the predetermined threshold or the edge direction included in the first edge information is not the same as the edge direction included in the second edge information The image processing apparatus according to claim 3, wherein the setting unit sets the first reference area along an edge direction included in the first edge information.   When the edge intensity included in the first edge information is equal to or less than a predetermined value, the setting means determines that the edge of the subject does not exist, and sets the first reference area. The image processing apparatus according to claim 2.   7. The apparatus according to any one of claims 1 to 6, wherein the setting unit sets the size of the first reference area based on the first edge information and the second edge information. The image processing apparatus according to claim 1.   The image processing apparatus may further include second suppression means for suppressing noise of a predetermined pixel data of the second image data with reference to pixel data of a second reference area around the second image data. The image processing apparatus according to any one of items 1 to 7.   The apparatus further comprises combining means for combining the second image data whose noise has been suppressed by the second suppressing means and the first image data whose noise has been suppressed by the first suppressing means. The image processing apparatus according to claim 8.   It further comprises calculation means for calculating a combination ratio for combining the second image data with noise suppressed and the first image data with noise suppressed based on the first edge information. The image processing apparatus according to claim 9, characterized in that:   11. The apparatus according to claim 10, further comprising: combining means for combining the second image data in which the noise is suppressed and the first image data in which the noise is suppressed based on the combining ratio. Image processing apparatus as described. A first detection step of detecting first edge information which is information of an edge portion of a subject from the first image data;
Generating the second image data by reducing the first image data;
A second detection step of detecting second edge information which is information of an edge portion of a subject from the second image data;
A first suppression step of suppressing noise of predetermined pixel data of the first image data with reference to pixel data of a first reference region therearound;
A setting step of setting the first reference area based on the first edge information and the second edge information;
An image processing method comprising:   The program for making a computer perform each process of the image processing method of Claim 12.   A storage medium storing a program for causing a computer to execute the steps of the image processing method according to claim 12.

Images