JP2011049696A - Image processing device, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent blur of an edge and achieve smooth gradation. <P>SOLUTION: A device is provided with: a resolution means which decomposes an original image into an edge preserving low frequency image which consists of smoothing components which preserve the edge, and a difference image between the original image and the edge preserving low frequency image; a first image enlargement means which applies a first image enlargement technique to the edge preserving low frequency image obtained by the resolution means, and creates a first enlarged image; a second image enlargement means which applies a second image enlargement technique to the difference image obtained by the resolution means, and creates a second enlarged image; and an image composition means which composes the first enlarged image and the second enlarged image. It is possible to prevent blur of the edge part and obtain also smooth gradation by applying the image enlargement technique which preserves the edge to the first image enlargement means, and applying the smoothing enlargement technique to the second image enlargement means. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique suitable for use in enlarging an image.

  When an image having a predetermined resolution is reproduced on a device having a higher resolution, an image enlargement process is required. For example, when an image acquired by a digital camera is displayed on a display or printed by a printer, the resolution of the original image and the reproduction destination device often do not match, so the image must be enlarged.

  As a technique for enlarging an image, for example, a nearest neighbor method or a bilinear method is used. Also, as disclosed in Patent Document 1, there is known a method of enlarging an image by converting the image to a spatial frequency domain, compensating for insufficient pixels in the area, and converting the image to a real space area again. In the technique disclosed in Patent Document 2, pixel interpolation is performed on two images, an original image and an image obtained by extracting high-frequency components of the original image, and then both images are added.

Japanese Patent No. 2522357 JP 2003-37733 A

  However, the enlargement method described above has the following problems. That is, in the nearest neighbor method shown in FIG. 9A, the enlarged pixel is copied from the adjacent pixel. For this reason, the edge portion can be clearly enlarged, but the smooth gradation portion of the original image is enlarged, and as shown by reference numeral 91 in FIG. Sometimes. On the other hand, in the bilinear method, the pixels in the enlarged portion are calculated by averaging four adjacent pixels. For this reason, although the gradation can be smoothly enlarged, the edge portion is linearly interpolated, so that the steep rise of the original image is lost as indicated by reference numeral 92 in FIG. 9B.

In Patent Document 1, the original image is converted into a spatial frequency domain, the size is enlarged in the area, the enlarged area, that is, the high frequency area is filled with 0, and is inversely transformed into the real space area, The image is enlarged. In this method, since the high frequency region whose value is not known due to the size expansion is filled with 0, edge information is lost, and there is a problem that the edge portion is blurred when inversely transformed into the real space region.
In Patent Document 2, since the original image is enlarged by pixel interpolation, a sharp rise in luminance at the edge portion cannot be stored. For this reason, there is a problem that the edge portion is blurred in the enlarged image.
SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to prevent edges from being blurred and to reproduce a smooth gradation in image enlargement processing.

  The image processing apparatus according to the present invention is an image processing apparatus for enlarging an image, wherein an original image is an edge-preserving low-frequency image composed of a smoothed component storing edges, and a difference between the original image and the edge-preserving low-frequency image Decomposition means for decomposing into images, first image enlargement means for generating a first enlarged image by applying a first image enlargement method to the edge-preserving low-frequency image obtained by the decomposition means, and the decomposition means A second image enlargement unit that generates a second enlarged image by applying the second image enlargement method to the obtained difference image, and an image composition that combines the first enlarged image and the second enlarged image. Means.

  According to the present invention, edge blurring can be suppressed and a smooth gradation can be reproduced in an image enlargement process.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. It is a figure which shows the flow of a process in an image processing apparatus. It is a figure which shows the data structure of a R, G, B color image. It is a figure which shows the processing result at the time of applying to a sample image. It is a figure explaining UI which concerns on 2nd Embodiment, and the flow of a process. It is a figure explaining UI which concerns on 3rd Embodiment, and the flow of a process. It is a figure explaining the 1st background art. It is a figure explaining the 2nd background art. It is a figure which shows the processing result by the nearest neighbor method and the bilinear method.

First, the background art of the present invention will be described.
FIG. 7 is a diagram for explaining the first background art of the present invention. FIG. 7A is a diagram showing an original image, and FIG. 7B is a diagram showing spatial frequency information obtained by subjecting the original image to Fourier transform. In FIG. 7B, the upper left indicates a low frequency component and the lower right indicates a high frequency component. Moreover, the brightness | luminance in FIG.7 (b) shows intensity | strength. FIG. 7C shows data in which the image size is enlarged in the spatial frequency domain and the high frequency domain is filled with zeros. Similarly to FIG. 7B, the upper left of FIG. 7C shows the low frequency component and the lower right shows the high frequency component. FIG. 7D is an image obtained by inversely converting the data in FIG. 7C into a real region. In this inversely transformed image, since the high frequency component is approximated to zero in FIG. 7C, the original edge information is not stored, and the edge is blurred.

  FIG. 8 is a diagram for explaining the second background art. As shown in FIG. 8, the two images are added after subjecting the original image and the image obtained by extracting the high-frequency component of the original image to image expansion by pixel interpolation. For example, when a luminance signal (a horizontal axis indicates a pixel position and a vertical axis indicates luminance intensity) as illustrated in FIG. 8A is included in the original image, the original signal is expanded by image interpolation by pixel interpolation. As shown in (b), it is stretched in the horizontal axis direction.

  Further, a high-frequency component as shown in FIG. 8C is extracted from the original signal by the high-pass filter. Then, the extracted high-frequency component is stretched by the image enlargement by the pixel interpolation, and becomes a high-frequency component as shown in FIG. Thereafter, correction processing by multiplication of coefficients and addition of constants is performed, and then an estimated value of the high frequency component as shown in FIG. 8E is calculated. By adding the estimated value and the signal shown in FIG. 8B, a final enlarged signal as shown in FIG. 8F is obtained. However, in this method, since the original image is enlarged by pixel interpolation, a sharp rise in luminance at the edge portion cannot be preserved as in the portion surrounded by a circle in FIG. For this reason, there is a disadvantage that the edge portion is blurred in the enlarged image.

(First embodiment)
Next, a first embodiment of the present invention that can eliminate the disadvantages described in the background art will be described.
FIG. 1 is a block diagram showing a hardware configuration for realizing the image processing apparatus according to the first embodiment of the present invention. In FIG. 1, reference numeral 1 denotes an image processing apparatus. Reference numeral 101 denotes an edge-preserving smoothing unit that applies a smoothing filter that preserves edges to the original image 11 to generate an edge-preserving low-frequency image 12 composed of smoothing components that preserve edges. Reference numeral 102 denotes a difference image generation unit that generates a difference image 13 by subtracting the edge-preserving low-frequency image 12 from the original image 11.

  Reference numeral 103 denotes a first image enlarging unit that applies the first image enlarging method to the edge-preserving low-frequency image 12 generated by the edge preserving / smoothing unit 101 and generates a first enlarged image 14. The image enlarging method for preserving edges performed in the first image enlarging unit 103 can be realized using a bilateral filter, a trilateral filter, or a Susan filter.

  A second image enlargement unit 104 generates the second enlarged image 15 by applying the second image enlargement method to the difference image 13 generated by the difference image generation unit 102. In the present embodiment, as the second image enlargement method, a bilinear method is used in which the pixels in the enlarged portion are calculated based on the average value of adjacent pixels. In addition, as a 2nd image expansion method, as long as it is the method of calculating | requiring by interpolating from an adjacent pixel, you may use other methods, such as a trilinear method.

  An image 105 generates the final enlarged image 16 by adding the first enlarged image 14 generated by the first image enlargement unit 103 and the second enlarged image 15 generated by the second image enlargement unit 104. It is a synthesis unit. Reference numeral 106 denotes a data holding unit for holding data necessary for calculation, and 107 denotes a buffer memory for temporarily holding data being calculated. Reference numeral 108 denotes a control unit that controls the overall operation of the image processing apparatus 1.

<Operation in Image Processing Apparatus 1>
Next, the operation of the image processing apparatus 1 will be described with reference to FIG.
FIG. 2 is a flowchart for explaining the operation of the image processing apparatus 1 according to the first embodiment of the present invention. In the following description, in order to simplify the description, the original image 11 will be described as a 1-channel 8-bit grayscale image. FIG. 3 shows a data format of a gray scale image of horizontal Nx pixels and vertical Ny pixels. In FIG. 3, each cell represents one pixel, and 8 bits are assigned to each pixel. Note that this method can also be applied to images other than 1-channel 8-bit images, such as 1-channel 16-bit images, 3-channel 24-bit images, or 4-channel 32-bit images.

  In S <b> 1, the edge preserving smoothing unit 101 performs edge preserving smoothing filter processing on the original image 11 to generate an edge preserving low-frequency image 12. In this embodiment, a bilateral filter is used as the edge preserving smoothing filter. The bilateral filter is a Gaussian filter whose weight changes according to the pixel position and the luminance difference between pixels. Expressions of the bilateral filter are shown in Expressions (1) to (4).

Here, x _j represents the position of the pixel j, and d _j represents the luminance at the pixel j.
Next, in S <b> 2, the difference image generation unit 102 generates a difference image 13 by subtracting the edge-preserving low-frequency image 12 from the original image 11. If the luminance at the position i, j of the original image 11 is d (i, j) and the luminance at the position i, j of the edge-preserving low-frequency image 12 is e (i, j), the luminance at the position i, j of the difference image 13 is The luminance f (i, j) is expressed by equation (5).

  Next, in S3, the first image enlargement unit 103 generates the first enlarged image 14 by using the first image enlargement method for the edge-preserving low-frequency image 12 generated in S1. In the present embodiment, the nearest neighbor method is used as the first image enlargement method. As the processing of the nearest neighbor method, for example, when the enlargement ratio is double, the luminance at the position i, j of the image before processing is P (i, j), and the luminance Q at the position i, j of the image after processing is (i, j) is expressed by equation (6).

Here, int (•) is an operation for truncating the decimal part.
Next, in S <b> 4, the second image enlargement unit 104 generates a second enlarged image 15 using the second image enlargement method for the difference image 13 generated in S <b> 2. In the present embodiment, the bilinear method is used as the second image enlargement method.
Next, in S5, the image composition unit 105 adds the two enlarged images generated in S3 and S4 to generate a final enlarged image 16.

FIG. 4 shows the state of conversion when this method is applied to a sample image.
FIG. 4A shows the original image 11, and its luminance distribution gradually decreases as it goes from the left side to the right side, rises stepwise at the center of the image, and decreases as it goes right again.

  FIG. 4B is an edge-preserving low-frequency image 12 generated by performing bilateral filter processing on the original image 11 in S1. FIG. 4C is a difference image 13 generated by subtracting the edge-preserving low-frequency image 12 from the original image 11 in S2. FIG. 4D is a first enlarged image 14 generated by performing processing by the nearest neighbor method on the edge-preserving low-frequency image 12 in S3. FIG. 4E shows a second enlarged image 15 that is generated by performing processing by the bilinear method on the difference image 13 in S4. FIG. 4F is a composite image generated by adding the first enlarged image 14 and the second enlarged image 15 in S5, that is, the final enlarged image 16.

  As described above, according to the technology described above, when the enlarged image 15 is generated from the original image 11, the original image 11 is decomposed into the edge-preserving low-frequency image 12 and the difference image 13, and appropriate for each image. Processing by a simple image enlargement method. Thereafter, by combining the two images, it is possible to generate an enlarged image 16 that does not blur the edge and realizes a smooth gradation.

(Second Embodiment)
Hereinafter, an image processing apparatus according to a second embodiment of the present invention will be described in detail with reference to the drawings. The hardware configuration for realizing the image processing apparatus according to the second embodiment is the same as that of the image processing apparatus 1 according to the first embodiment described above. Omitted.
In FIG. 5A, reference numeral 201 denotes a user interface (hereinafter abbreviated as UI) for allowing the user to select parameters relating to image enlargement. A menu 202 allows the user to select a desired filter from several edge-preserving smoothing filters prepared in advance. Reference numeral 203 denotes a first parameter that controls the shape of the filter selected in 202. As a parameter type, for example, when a bilateral filter is selected as a filter, a parameter for controlling the degree of smoothing in the distance direction is input.

  Reference numeral 204 denotes a second parameter that controls the shape of the filter selected in 202. As a parameter type, for example, when a bilateral filter is selected as a filter, a parameter for controlling a luminance difference recognized as an edge (level difference for R, G, and B) is input.

  Reference numeral 205 denotes a menu that allows the user to select a desired image enlargement method from several edge-preserved image enlargement methods. Reference numeral 206 denotes a menu that allows the user to select a desired image enlargement method from several smoothed image enlargement methods prepared in advance. Reference numeral 207 denotes an OK button for applying the settings at the time of image enlargement processing after each item from 202 to 206 is set. When the user clicks this OK button 207, the image enlargement process starts. Reference numeral 208 denotes a button for canceling the setting item.

Next, the operation of the image processing apparatus according to the present embodiment will be described with reference to FIG.
FIG. 5B is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment.
In S <b> 21, parameter acquisition processing is performed for acquiring information related to the edge preserving smoothing filter, the first parameter, the second parameter, the first expansion method, and the second expansion method from the UI 201. This parameter acquisition process is performed by the control unit 108 described in the image processing apparatus 1 of FIG. As the information related to the edge-preserving smoothing filter, for example, several types of processing procedures are prepared in advance as functions, and information indicating which function is selected from them is shown.

  That is, some processing procedures are programmed with functions, and address information (on the memory) for the function selected by the user is shown. The contents of the first parameter 203 and the second parameter 204 indicate the values of one or two parameters that control the characteristics of the selected edge-preserving smoothing filter. It should be noted that if there is no edge-preserving smoothing filter parameter or if there is only one parameter, no user input is accepted.

  Further, when there are three or more parameters as function characteristics, a parameter input field may be added, or three or more parameters may not be accepted. Further, the information relating to the first image enlargement technique and the second image enlargement technique may be obtained by converting the enlargement procedure into a function and indicating the address information thereof, similarly to the information relating to the edge preserving smoothing filter described above.

When the parameter acquisition process of S21 is completed, the process proceeds to S1, and the process described with reference to FIG. 2 is sequentially performed in the first embodiment. Since the processing performed in S1 to S5 has been described above, detailed description and illustration are omitted.
As described above, according to the second embodiment described above, a user can easily obtain a desired enlarged image by a simple operation in which the user selects and inputs a parameter related to image enlargement with the UI 201.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described in detail with reference to the drawings. The hardware configuration for realizing the image processing apparatus according to the third embodiment is the same as that of the image processing apparatus 1 according to the first embodiment described above. Omitted.
In FIG. 6A, reference numeral 301 denotes a UI for allowing the user to select a parameter related to image enlargement and confirming a preview when the image is converted using the parameter. Since 202, 203, and 204 are the same as those described in the second embodiment, description thereof is omitted.

  Reference numeral 302 denotes an original image display unit for displaying the original image (original image 11). Reference numeral 303 denotes a selection area selected by the user. Reference numeral 304 denotes a preview button. Reference numeral 305 denotes a preview display unit. The user selects a part of the original image as indicated by a selection area 303 on the original image display unit 302 using a pointing device such as a mouse or a touch panel. Thereafter, when a preview button 304 is pressed, an enlarged image of the selection area 303 can be confirmed on the preview display unit 305.

  Reference numeral 306 denotes an OK button for applying the settings selected and input by the user during image enlargement processing. When the user clicks the OK button 306, the image enlargement process starts. Reference numeral 307 denotes a button for canceling the setting item.

Next, the operation of the image processing apparatus according to the present embodiment will be described with reference to FIG.
FIG. 6B is a flowchart showing the operation in the image processing apparatus of the present embodiment.
In S <b> 31, a parameter acquisition process for acquiring information related to the edge preserving smoothing filter, the first parameter, and the second parameter from the UI 301 is performed. This parameter acquisition process is performed by the control unit 108 described in the image processing apparatus 1 of FIG.

  The information regarding the edge preserving smoothing filter is as described in the second embodiment. When the preview button 304 is pressed after the parameter is selected and input by the user, an image enlargement process is performed on the image in the selection area 303 using the information. As a result, the enlarged image is displayed on the preview display unit 305.

  When the user confirms the preview image, determines that the selected or input value is acceptable, and presses an OK button 306, image enlargement processing is performed on the original image. With such a configuration, the image enlargement process can be performed only on a part of the original image. Therefore, the parameters can be easily changed and the result can be confirmed. For this reason, it is possible to quickly find the optimum parameter for the image.

When the parameter acquisition process of S31 ends, the process proceeds to S1, and the process described with reference to FIG. 2 is sequentially performed in the first embodiment. Since the processing performed in S1 to S5 has been described above, detailed description and illustration are omitted.
As described above, according to the technology described above, it is possible to confirm in advance a preview of an enlarged image with settings made by the user selecting or inputting a parameter related to image enlargement on the UI. In addition, since the enlargement process is performed only on the area selected by the user, a preview image can be generated at a higher speed than when the enlargement process is performed on the original image. Therefore, the user can easily obtain a desired enlarged image. it can.

(Other embodiments)
The present invention is also achieved by a software program that implements the functions of the above-described embodiments. Specifically, when the computer of the system or apparatus reads and executes the computer program corresponding to each configuration in FIG. 1, each step in FIG. 2, each step in FIG. 5 and FIG. Are also included in the present invention.

Claims (11)

In an image processing apparatus for enlarging an image,
Decomposing means for decomposing the original image into an edge-preserving low-frequency image composed of a smoothing component that preserves edges, and a difference image between the original image and the edge-preserving low-frequency image;
First image enlarging means for applying the first image enlarging method to the edge-preserving low-frequency image obtained by the decomposing means to generate a first enlarged image;
A second image enlarging unit that applies a second image enlarging method to the difference image obtained by the decomposing unit to generate a second enlarged image;
Image synthesizing means for synthesizing the first enlarged image and the second enlarged image;
An image processing apparatus comprising:   The decomposition means includes an edge preserving smoothing unit that applies a smoothing filter that preserves edges in the original image, and a difference image generating unit that subtracts the edge preserving low frequency image from the original image. The image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the first image enlarging unit applies an image enlarging method that preserves edges.   The image processing apparatus according to claim 1, wherein the second image enlargement unit applies a bilinear method.   The image processing apparatus according to claim 3, wherein the image enlarging method for storing the edge is any one of a bilateral filter, a trilateral filter, and a Susan filter. In an image processing method for enlarging an image,
A decomposing step of decomposing the original image into an edge-preserving low-frequency image composed of smoothing components that preserve edges, and a difference image between the original image and the edge-preserving low-frequency image;
A first image enlargement step of generating a first enlarged image by applying a first image enlargement method to the edge-preserving low-frequency image obtained in the decomposition step;
A second image enlargement step of generating a second enlarged image by applying a second image enlargement method to the difference image obtained in the decomposition step;
An image synthesis step of synthesizing the first enlarged image and the second enlarged image;
An image processing method comprising:   The decomposing step comprises an edge preserving smoothing step for applying a smoothing filter for preserving edges to the original image, and a difference image generating step for subtracting the edge preserving low frequency image from the original image. 6. The image processing method according to 6.   The image processing method according to claim 6, wherein the first image enlargement step applies an image enlargement method that preserves edges.   The image processing method according to claim 6, wherein a bilinear method is applied in the second image enlargement step.   The image processing method according to claim 8, wherein the image enlargement method for storing the edge is any one of a bilateral filter, a trilateral filter, and a Susan filter. In a computer program for causing a computer to execute an image processing method for enlarging an image,
A decomposing step of decomposing the original image into an edge-preserving low-frequency image composed of smoothing components that preserve edges, and a difference image between the original image and the edge-preserving low-frequency image;
A first image enlargement step of generating a first enlarged image by applying a first image enlargement method to the edge-preserving low-frequency image obtained in the decomposition step;
A second image enlargement step of generating a second enlarged image by applying a second image enlargement method to the difference image obtained in the decomposition step;
An image synthesis step of synthesizing the first enlarged image and the second enlarged image;
A computer program for causing a computer to execute.

Images