JP2001043357A - Image processing method, device thereof and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain the interpolation with reduced aliasing and blur and at a low calculation cost. SOLUTION: A bilinear interpolation part 104 of this image processor fetches 4-pixel value corresponding to the pixel positions designated at a pixel position generation part 102 to the images of three hierarchies selected at a hierarchy selection part 103 from an original image/reduced image storage part 101 and performs a bilinear interpolation operation. These obtained three interpolation pixel value undergo the weighted sum via a weighted sum calculation part 105 by means of the weighting including negative value. The sum total of three weightings is always equal to 1 and also gradually changes as the enlargement/ reduction ratio varies at the periphery of an attentional pixel. Thus, a deformed image that is obtained via the perspective conversion and has the smoothly continuous parts of different scale factors and reduced aliasing and blur is obtained at a deformed image storage part 106.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to the field of digital image processing, and more particularly, to the field of digital image deformation processing.

[0002]

2. Description of the Related Art In the field of image processing, digital images often need to be deformed. For example, in order to accurately combine a plurality of images photographed under conditions such as different positions and angles, it is necessary to perform deformation by perspective transformation on the images. In the deformation corresponding to the perspective transformation, a portion to be enlarged and a portion to be reduced occur in one image. In such a case, it is desirable that both enlargement and reduction be performed with high quality, and that portions having different magnifications be smoothly continuous.

In order to transform a digital image, it is necessary to consider a new grid on the original image which is different from the pixel grid of the original image, and to obtain the pixel value at the grid point position. However, since the pixel position of the deformed image generally does not match the pixel position of the original image, the pixel value of the deformed image is obtained by interpolating the pixel value of the original image. Interpolation processing refers to the position (i, j) (where i, j
Is a process of estimating a pixel value B (x, y) at an arbitrary position (x, y) from A (i, j), where A (i, j) is a pixel value of (integer).
The following three methods are known as pixel value interpolation methods for digital images.

<< Nearest Neighbor Method (Nearest Neighbor Method) >>
This is a method in which an original image pixel (i, j) closest to (x, y) is selected, and the pixel value is directly used as an interpolation pixel value B (x, y). That is,

(Equation 1) This method has the worst image quality of the interpolation result among the three interpolation methods described here, but has the lowest calculation cost.

[0005] << Bilinear method >> This is a method of selecting four original image pixels surrounding (x, y) and linearly interpolating from the pixel values. That is,

(Equation 2)

<< Cubic Spline Method >> In this method, 4 × 4 = 16 original image pixels surrounding (x, y) are selected, and a weighted average of the pixel values is obtained. That is,

(Equation 3) Since this method uses a negative weight, the effect of edge enhancement is produced as compared with the bilinear method, and an interpolation result with the highest resolution among the three interpolation methods can be obtained. On the other hand, the calculation cost is the highest because 16 pixel values are used.

The following processing method is known as a technique for improving image quality or preventing image quality deterioration related to enlargement and reduction of digital images.

<< Technique for Preventing Blur: Unsharp Mask Processing >> When an image is enlarged, fine information exceeding the detail level of the original image cannot be obtained, and the image tends to be blurred. As a method of enhancing the outline and reducing blur, there is known a processing method called an unsharp mask that subtracts an image obtained by blurring an original image from an original image, as disclosed in, for example, JP-A-10-23267. I have. Hereinafter, the unsharp mask processing will be described. For example,
It is assumed that the original image (I) is filtered using the coefficients shown in FIG. This filter is a smoothing filter, and the result (Is) of the filter processing is a blur of the original image.

Next, consider the use of the original image (I) and the smoothed image (Is) to obtain a weighted sum for each pixel as in the following equation.

(Equation 4)

In equation (4), when considering the smoothed image (Is) retroactively to the filter processing in FIG.
It can be seen that the result is the same as the result of the filter processing using the coefficient shown in FIG. 5 in which the coefficient of .times. As shown in FIG. 5, a filter having a coefficient other than the center pixel having a negative value generally has an edge enhancement effect. From the coefficients (1.5, 0.5), the result of the weighted sum is the same as the original image (I) on average. From this,
As can be seen from equation (4), by subtracting the smoothed image from the original image, an image with enhanced edges can be obtained.

<< Method for Preventing Aliasing: Mipmap Method >> In image reduction processing, image quality deterioration called aliasing becomes a problem. FIG. 3 shows an example of aliasing. FIG. 3A shows an original image in which oblique lines are drawn,
(B) is an image obtained by reducing the original image to 1/4. As in the example of the reduced image, the image quality may be degraded due to the reduction, for example, a gentle oblique line may be changed to a broken line. This is called aliasing. In this example, since the pixel size is reduced to 1/4, the pixel position of the converted image matches the pixel position of the original image. Therefore, the results are the same no matter which of the three interpolation methods is used. In general, aliasing is not resolved by using high-precision interpolation methods such as cubic splines. As a processing method for preventing aliasing, a method called a mipmap is known. In the mipmap method, a deformed image is obtained by a series of processes described below. (1) Pyramid reduction: An image reduced to 1/2 is generated by taking an average of a total of 4 pixels each of 2 pixels vertically and horizontally of the original image. Such 1/2 reduction is repeated, and 1/4, 1 /
8,. . . Then, a series of reduced images whose level of detail gradually decreases is prepared. (2) Optimal hierarchy selection: An image having an appropriate reduction ratio with respect to the magnification around the pixel position to be subjected to the interpolation processing is selected. If the requested magnification is not equal to 1 / n, a reduced image of the upper and lower two layers having the same magnification and a reduced ratio is selected. For example, when the peripheral magnification is 1/6, a 1/4 reduced image and a 1/8 reduced image are selected. (3) Intra-hierarchical interpolation: A pixel value at a position corresponding to a target pixel position on each selected reduced image is obtained by an interpolation process. In this interpolation processing, aliasing is unlikely to occur because a reduced image having substantially the same size as the deformed image is used. (4) Inter-layer interpolation: When the magnification at the periphery of the pixel position to be interpolated is between the magnifications of the reduced image of the selected upper and lower two layers, it is determined according to the proximity between the peripheral magnification and the magnification of each layer. A weighted average of the two intra-layer interpolation results is obtained using the calculated weights.
For example, when the magnification is ６, the one-to-one average of the interpolation result on the ４ reduced image and the interpolation result on the ８ reduced image is calculated. With this processing, even when the magnification gradually changes depending on the position, the processing result smoothly continues.

The advantage of the mipmap method is that aliasing is less likely to occur because the reduction process is performed before sampling. For example, in the case of the image shown in FIG. 3A, a 縮小 reduced image as shown in FIG. 6A and a 縮小 reduced image as shown in FIG.

On the other hand, a disadvantage of the mipmap is that a bilinear method which generally requires only eight pixels is used as an intra-layer interpolation processing method. In this case, the resolution is lower than that of the cubic spline method. Using the cubic spline method in the hierarchy to increase the resolution, a total of 3
Since it is necessary to refer to two pixel values, the amount of calculation becomes very large.

[0014]

SUMMARY OF THE INVENTION It is an object of the present invention to achieve low aliasing and low blur interpolation at low computational costs. Another object of the present invention is to enable generation of a high quality image deformed by perspective transformation.

[0015]

According to the present invention, a plurality of images are selected from an original image and a plurality of images obtained by reducing the original image and having different levels of detail, and interpolation processing is performed on each of the selected images. The interpolation pixel value is obtained by performing a weighted sum operation of a plurality of interpolation results obtained, but by including a negative value in the weight used in the weighted sum operation, aliasing and blurring are reduced at a low calculation cost. To achieve interpolation.

In addition, a pixel position on the original image corresponding to each pixel position on the image obtained by subjecting the original image to perspective transformation based on the transformation parameter is generated, and an enlargement / reduction magnification around the pixel position is determined. Then, the original image is transformed by perspective transformation by selecting an image based on the reduction / enlargement magnification and performing an interpolation operation using a plurality of pixel values corresponding to the pixel positions of each of the selected images. Produces images with less aliasing and blur. Further, the weight used in the weighted sum calculation is set such that the sum thereof is always 1 and gradually changes as the enlargement / reduction magnification changes. Generate.

[0017]

Embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a block diagram showing an example of an image processing apparatus according to the present invention. This image processing apparatus generates an image in which an original image is transformed by perspective transformation based on transformation parameters, and includes a reduced image generation unit 100, an original image / reduced image storage unit 101, a pixel position generation unit 102, a hierarchical selection It comprises a unit 103, a bilinear interpolation unit 104, a weighted sum calculation unit 105, a deformed image storage unit 106, a result output unit 107, an original image input unit 108, and a parameter input unit 109. FIG. 2 is a flowchart showing a processing flow of the image processing apparatus. Hereinafter, the processing content will be described along the flow of this flowchart.

<Step S1> The original image data is input to the reduced image generation unit 100 by the original image input unit 108.

<< Step S2 >> Parameter input unit 109
The transformation parameter of the perspective transformation is
02 is input. At this time, the size (height, width) of the output image (target deformed image) is also input to the pixel position generation unit 109.

<Step S3> The reduced image generation section 100 performs a plurality of stages of reduction processing on the input original image data to generate a plurality of reduced image data having different levels of detail. In this case, the average value of 2 × 2 pixels in the vicinity of the original image is calculated to generate image data reduced to 縦 in height and width, and the reduced image is further reduced to generate image data. The process is repeated until the size in the vertical or horizontal direction becomes one pixel. The generated reduced image data is stored in the original image / reduced image storage unit 101 together with the original image data. In other words, hierarchical data composed of a plurality of reduced image data of which the degree of detail is sequentially reduced and original image data of the highest degree of detail are stored in the original image / reduced image storage unit 101 (a reduced image having a lower degree of detail has a higher hierarchy). The structure is obtained. The original image / reduced image storage unit 101 is referred to by the bilinear interpolation unit 104 by designating the hierarchy (the number of reduction stages) and the (x, y) coordinates on the image (original image or reduced image) of the hierarchy. Then, the pixel value at the corresponding position in the corresponding hierarchical image is output.

<Step S4> The pixel position generator 102 generates pixel positions on the original image corresponding to the respective pixel positions on the output image resulting from the perspective transformation based on the transformation parameters. Is output to the bilinear interpolation unit 104. That is, assuming that the position on the output image is (i, j) and the position on the original image is (x, y), for all combinations of (i, j) within the determined range of the output image, The pixel position (x, y) is calculated in the scan line order according to the following equation. Here, since the sequential processing for each pixel is considered in the processing loop of steps S4 to S8, calculation is performed for one (i, j) combination of interest in each pass of this processing loop, and as a result, Is output.

(Equation 5) However, a ₇ from a ₀ is a modified parameter given from the outside.

The pixel position generation unit 102 calculates the enlargement / reduction ratio around the pixel position (x, y), and outputs the result to the hierarchy selection unit 103. This magnification is obtained as follows. First, on the output image,
Consider a square having a size of 1 in the horizontal and vertical directions with (i, j) as one corner, and find the corresponding position of each vertex on the original image by the following equation.
(See FIG. 7).

(Equation 6)

Next, the range (w, h) occupied by the corresponding rectangle on the original image is determined.

(Equation 7) However, max (x ...) and min (x ...) are functions that return the maximum and minimum values of the arguments. The larger one of w and h is considered as the length on the original image corresponding to the length 1 on the output image, and the magnification m around the pixel position of interest is calculated by the following equation.

(Equation 8)

<Step S5> In the hierarchy selection unit 103, the hierarchy (L) of the image to be used is determined according to the following equation based on the magnification m given by the pixel position generation unit 102.

(Equation 9)

(Equation 10)

This layer (L) and three layers (L + 1) and (L + 2)
The hierarchical image (reduced image or original image) is to be processed by the next bilinear interpolation operation. That is, these three hierarchies are substantially selected. Further, the logarithm multiple 1 calculated from the peripheral magnification m by the equation (9) is also instructed to the weighted sum calculation unit 105 (however, the magnification m may be instructed to the weighted sum calculation unit 105).

<< Step S6 >> Bilinear interpolation section 104
, The pixel position (x _L , y _L ) on the image of the layer (L) is obtained from the pixel position (x, y) on the original image input from the pixel position generation unit 102 by the following equation.

[Equation 11]

Next, with reference to the original image / reduced image recording unit 101, the pixel values of four pixels surrounding the pixel (x _L , y _L ) on the image of the hierarchy (L) are read, and bilinear interpolation calculation is performed. . That is, the interpolation calculation of the following equation is performed using AL (i, j) as the pixel value at the position (i, j) on the image of the layer (L).

(Equation 12)

Similarly, for the images of the layers (L + 1) and (L + 2), the pixel positions are calculated, the pixel values of the four pixels surrounding the positions are read, and the bilinear interpolation operation is performed. Pixel values ( _{IL + 1} ) and ( _{IL + 2} ) are obtained.

<Step S7> In the weighted sum calculation unit 105, an output pixel value (Iout) is obtained by calculating the weighted sum of the bilinear interpolation results in each layer according to the following equation, and the output pixel value (Iout) is written to the deformed image storage unit 106.

(Equation 13)

Here, the weight w _i is determined by the pixel position generator 1
Logarithmic magnification (l) around the position of the pixel of interest given from 02
Is determined according to the following equation.

[Equation 14]

(Equation 15) However, it is also possible to determine the weight by a similar method based on the magnification m instead of the logarithmic magnification.

FIGS. 8 and 9 show the relationship between the magnification and the weight.
The horizontal axis represents the logarithmic magnification 1, the vertical axis represents the weight w, and the thick solid line represents the weight w _L for the interpolation result of the L layer. For reference, w _L-1 , w
_{L + 1 is} also indicated by a dashed line. For comparison, FIG.
Fig. 2 shows the weight in the conventional mipmap method.

Here, the features of the weighting as shown in FIGS. 8 and 9 and the resulting advantages of the present invention will be described.
(A) First, of the three weights, the weight of the higher hierarchy (small and reduced image) takes a negative value. By including the negative weight in this manner, the effect of the unsharp mask is generated, and a high-resolution interpolation result with less aliasing and blurring can be obtained. (B) The sum of the three weights is always 1
Thus, the weight gradually changes as the magnification changes. By the perspective transformation, even if the magnification gradually changes depending on the position, the processing content does not suddenly change, so that a result in which portions having different magnifications are smoothly continuous can be obtained. (C) Intra-hierarchical interpolation is a bilinear method using four pixel values in the same way as the normal mipmap method, and therefore refers to a total of 12 pixel values in three hierarchies. This is more than 8 pixels in the normal mipmap method, but less than half of 32 pixels when using the cubic spline method in each layer,
The computation cost is much lower.

<< Step S8 >> It is determined whether the generation of the target deformed image has been completed. For example, the pixel position generation unit 102 determines whether all pixel positions to be generated have been generated based on the size (height, width) of the input output image (deformed image). If there is a pixel position to be generated (determination result No), the process returns to step S4. If there is no pixel position to be generated (determination result Ye
s) The entire target deformed image data is stored in the deformed image storage unit 10
6, the process proceeds to the next step S9.

<Step S9> The result output unit 107 outputs the deformed image data obtained in the deformed image storage unit 106 to the outside.

In the first embodiment described above, three hierarchical images are used for the interpolation processing, and four pixels are used for the intra-hierarchical interpolation. Therefore, a total of 12 weighted sums are obtained. However, in general, when the magnification m is smaller than 1, that is, when the magnification is reduced,
Even if the resolution is slightly lowered, it is often inconspicuous. Therefore,
In the second embodiment, in order to generate an unsharp mask effect only in the case of enlargement, only images of two layers are used. Even in this case, a visually high effect can be expected.

That is, in the second embodiment, the bilinear interpolation unit 104 (step S6)
Interpolation is performed only on the images (L) and (L + 1),
The weighted sum calculation unit 105 (step S7) performs the weighted sum calculation of the following equation.

(Equation 16)

Here, the weight w _i is determined according to the following equation.

[Equation 17]

(Equation 18)

[Equation 19] FIG. 11 shows such a relationship between the magnification and the weight. In the second embodiment, since only two layers of images are used, only eight pixels need to be referenced as in the normal mipmap method, and the calculation cost can be further reduced.
For a portion where the blur is conspicuous and which is from the same size to the enlarged size, the same edge enhancement effect as in the unsharp mask processing can be obtained.

According to the present invention described above, for example, a CPU 200, a memory 201, a hard disk device 202, a display 203, an input device 204 such as a keyboard and a mouse, a printer 205, a communication device 2 as shown in FIG.
06, floppy (registered trademark) disk or CD-ROM
Drive 2 for reading and writing of recording medium 207 such as
08 and the like can be implemented by software using a general computer system connected by a bus 209.
In this case, a program for causing a computer to execute each processing step as shown in the flowchart of FIG. 2 or a program for realizing each functional unit as shown in FIG. From the recording medium 207 on which the program is recorded via the medium driving device 208 or from an external device, the communication device 206
Is read into the hard disk drive 202 via the hard disk drive 202, and is executed when the processing is executed.
Loaded into Alternatively, the program is stored in the memory 20
1 is implemented in a form stored in a ROM constituting a part of the ROM.

[0040]

As described above, claims 1 to 9 are described.
According to the present invention, it is possible to realize interpolation with less aliasing and blurring at a low calculation cost, and according to claims 4, 5, 8 and 9, projection with less aliasing and blurring A transformed image can be generated by the conversion, and in particular, according to the invention described in claim 5 or 9, a transformed image in which portions having different magnifications are smoothly continuous can be generated. According to the sixth aspect of the present invention, it is possible to obtain the effect that the above-described interpolation and generation of the deformed image can be easily performed by using a general computer.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an example of an image processing apparatus according to the present invention.

FIG. 2 is a flowchart for explaining processing in the image processing apparatus of FIG. 1;

FIG. 3 is a diagram showing an original image and a 1/4 reduced image in which aliasing has occurred.

FIG. 4 is a diagram illustrating coefficients of a smoothing filter for generating a blurred image.

FIG. 5 is a diagram illustrating coefficients of an outline emphasis filter.

6 is a diagram showing a 1/2 reduced image and a 1/4 reduced image without aliasing obtained by reducing the original image shown in FIG. 3;

FIG. 7 is a diagram for explaining calculation of a peripheral magnification of a target pixel position.

FIG. 8 is a diagram illustrating a relationship between weights for a weighted sum operation on an intermediate layer and logarithmic magnification.

FIG. 9 is a diagram illustrating a relationship between weights and a logarithmic magnification for a weighted sum operation with respect to the lowest layer.

FIG. 10 is a diagram showing the relationship between weight and logarithmic magnification used in conventional mipmap processing.

FIG. 11 is a diagram illustrating a relationship between weights and a logarithmic scaling factor for a weighted sum operation when two layers are used.

FIG. 12 is a block diagram showing an example of a computer for implementing the present invention by software.

[Explanation of symbols]

 Reference Signs List 100 Reduced image generation unit 101 Original image / reduced image storage unit 102 Pixel position generation unit 103 Hierarchical selection unit 104 Binary interpolation unit 105 Weighted sum calculation unit 106 Deformed image storage unit 107 Result output unit 108 Original image input unit 109 Parameter input unit

Claims (9)

[Claims] 1. A first step of selecting a plurality of images from an original image and a plurality of images obtained by reducing the original image and having different levels of detail, and a second step of performing an interpolation process on each of the selected images.
And a third step of performing a weighted sum operation on the obtained plurality of interpolation results, wherein a negative value is included in the weight used in the weighted sum operation. Method. 2. The image processing method according to claim 1, wherein interpolation processing is performed on three or more images. 3. The image processing method according to claim 1, wherein a negative weight is used in a weighted sum operation for an interpolation result for an image having a specific level of detail. 4. Generating a pixel position on the original image corresponding to each pixel position on the image obtained by subjecting the original image to perspective transformation based on the transformation parameter, and setting an enlargement / reduction magnification around the pixel position. The method further includes a fourth step of obtaining, wherein the first step selects an image based on the enlargement / reduction magnification obtained in the fourth step, and the second step generates the image in the fourth step. And performing an interpolation operation using a plurality of pixel values of each selected image corresponding to the pixel position thus selected, and generating a pixel value of an image obtained by transforming the original image by perspective transformation as a result of the weighted sum operation. The image processing method according to claim 1, 2, or 3, wherein 5. The weight used in the weighted sum calculation, the sum of the weights is always 1, and the weight gradually changes as the enlargement / reduction magnification obtained in the fourth step changes. Item 5. The image processing method according to Item 4. 6. A computer-readable recording medium on which a program for causing a computer to execute each step of the image processing method according to claim 1, 2, 3, 4, or 5 is recorded. 7. A first means for generating a plurality of images having different levels of detail obtained by reducing an input original image, and a second means for storing an image of a plurality of layers including the input original image and the generated image. Means, third means for selecting a plurality of images from among the images stored in the second means, fourth means for performing an interpolation process on each of the selected images, An image processing apparatus comprising: fifth means for performing a weighted sum operation, wherein the weight used in the weighted sum operation includes a negative value. 8. Generating a pixel position on the original image corresponding to each pixel position on the image obtained by transforming the original image by perspective transformation based on the input deformation parameters, and enlarging / enlarging the periphery of the pixel position. The image processing apparatus further includes a sixth unit for calculating a reduction ratio, wherein the third unit selects an image based on the enlargement / reduction ratio obtained by the sixth unit, and wherein the fourth unit is generated by the sixth unit. Interpolation operation is performed using a plurality of pixel values of each selected image corresponding to the pixel position thus selected, and as a result of the weighted sum operation, a pixel value of an image obtained by transforming the original image by perspective transformation is generated. The image processing apparatus according to claim 7, wherein: 9. The weight used for the weighted sum calculation, the sum of the weights is always 1, and the weight gradually changes as the enlargement / reduction ratio obtained by the sixth means changes. Item 10. The image processing device according to Item 8.