WO2000075865A1

WO2000075865A1 - Image processing method

Info

Publication number: WO2000075865A1
Application number: PCT/JP2000/003640
Authority: WO
Inventors: Kazuo Toraichi; Kouichi Wada; Masakazu Ohhira
Original assignee: Fluency Research & Development Co., Ltd.
Priority date: 1999-06-03
Filing date: 2000-06-05
Publication date: 2000-12-14

Abstract

An image processing method for forming a high-definition restructured image even if image magnification is conducted by processing a little amount of data. A magnified image is created by performing concentration value interpolation using a two-dimensional sampling function so produced that the function values of the points equidistant from a sampling point and constituting a two-dimensional image are made equal to each other. An edge included in an image is detected, the direction is determined, a two-dimensional sampling function is transformed by adapting the function to the direction thereby to create an adaptive sampling function, and interpolation about the edge part is performed using the adaptive sampling function.

Description

Description Image processing method Technical field

The present invention relates to an image processing method for displaying an image reconstructed on the basis of image data in a high-quality scale. Background art

In recent years, the development of the network infrastructure centered on the Internet has progressed on a global scale, and it has become easier to acquire and provide image data via the network. Was. At the same time, the use of digital images has expanded, and the need for image processing technology to process them has increased. Magnified reconstruction of digital images is one such image processing technology. On the other hand, enlarged reconstruction of digital images also has an aspect as resolution conversion technology that connects media with different resolutions. In a network environment where media of various resolutions are mixed, its importance is high.

Generally, in geometric transformation such as enlargement, reduction, and rotation of an image, image reconstruction is performed by interpolating density value information of the original image. General image reconstruction methods include (1) Nearest Neighbor interpolation, (2) Bi-linear interpolation (Bi-linear interpolation), and (3) Third-order convolution. Interpolation (Cubic Convo lution). These are all based on density value interpolation.

In recent years, research on new methods such as the super-resolution method and the high-quality image enlargement method using a neural network has been conducted. These methods aim at high-quality enlargement by restoring the high-frequency components lost by sampling, which cannot be reproduced by the above-described interpolation-based image reconstruction method. By the way, in the above-described image reconstruction method mainly based on interpolation, a sinc function derived from the sampling theorem is used as a sampling function in order to perform highly accurate interpolation. The conventional interpolation method using the sinc function as the sampling function Since it is guaranteed that the interpolated signal can be completely interpolated, it has been generally used as the most accurate interpolation method. Interpolation by the sinc function is also performed in the third-order convolution interpolation, which is said to provide the sharpest enlarged image in the above-described conventional method. However, since the sinc function is not a local table, it is necessary to terminate the sinc function in a finite interval in actual applications, which causes an error. Also, since the sampling theorem covers signals in continuous and differentiable finite time intervals, if the target signal contains discontinuous points, the original signal is completely restored. It is not possible. In general, there are many discontinuities in an image, and such points are important in determining image quality. Specifically, for example, there is a problem that the vicinity of an edge where the density value changes rapidly corresponds to a discontinuous point, and in such a point, the distortion of reconstruction becomes remarkable.

In addition, digital images have a problem that jaggies appear at edges in an oblique direction when the image is enlarged at a high magnification, and the image quality deteriorates. In addition, the newly proposed method such as the above-mentioned super-resolution method requires a repetitive process, so the time required for the process is longer than that of the conventional method that mainly uses interpolation. There is a problem. Disclosure of the invention

The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing method capable of obtaining a high-quality reconstructed image even when an enlargement process is performed with a small amount of data processing. To provide.

The image processing method according to the present invention, when enlarging a two-dimensional image using a two-dimensional sampling function in which the influence of sample values on points equidistant from the sample point In the second step, the edge contained in the image is detected, and in the second step, an elliptic two-dimensional sampling function that is transformed by adapting the two-dimensional sample function to the shape of the edge is generated. For the pixels along the edge included in, perform the interpolation operation to increase the number of pixels using the elliptic 2D sampling function, and for the other pixels, increase the number of pixels using the 2D sampling function An interpolation operation is being performed. Adapt to the edge shape when enlarging the edge part included in the 2D image Interpolation processing using a two-dimensional sampling function is performed, which enables smooth interpolation processing along the edge shape, and high-quality reconstructed images when performing 2D image enlargement processing Can be obtained.

The two-dimensional sampling function described above is composed of a (m-1) -th order piecewise polynomial that can be continuously differentiated (m-2) times, and desirably has a function value of a local table. By using a continuously differentiable two-dimensional sampling function, a high-quality enlarged image without unnatural steps can be obtained. In addition, by using a two-dimensional sampling function represented by a piecewise polynomial having local platform values, it is possible to reduce the amount of data processing when performing enlargement processing and to prevent the occurrence of truncation errors. it can. The elliptic two-dimensional sampling function described above desirably has an elliptical cross-section obtained by deforming a circular cross-section passing through a point having the same function value along the edge direction while maintaining the cross-sectional area. By using an elliptic two-dimensional sampling function having an elliptical cross section, sharp changes in the density value along the edge direction are eliminated, and jaggies in the edge direction can be reduced.

In addition, the ratio of the major axis to the minor axis of the elliptical cross section described above is set to a large value when another edge exists along the edge of interest, and another edge along the edge of interest is set. It is desirable to set it to a small value when does not exist. The longer the edge, the greater the degree to which the two-dimensional sampling function is deformed, so that jaggies that are particularly noticeable when the edge is long can be reduced.

Further, the first step described above includes a process of determining an edge direction based on local information corresponding to a minute region including a plurality of pixels included in the two-dimensional image, and a process of determining the determined edge. It is desirable to include a process of optimizing a region wider than the minute region in the direction. In order to reduce the amount of data processing, it is preferable to detect edges and determine their directions in as small a region as possible, but it is difficult to reflect the global edge shape. Therefore, after deciding the edge direction for each small area, the edge direction is optimized in an area wider than the small area, so that the global edge shape can be reduced while maintaining a small amount of data processing. The reflected edge detection can be performed accurately. BRIEF DESCRIPTION OF THE FIGURES

Figure 1 shows a schematic diagram of the two-dimensional sampling function.

FIG. 2 is a diagram illustrating a modification of the two-dimensional sampling function,

FIG. 3 is a diagram for explaining an enlarged result by an ellipse two-dimensional sampling function, FIG. 4 is a diagram for explaining a method of calculating a direction vector,

FIG. 5 is a configuration diagram of an image processing apparatus according to an embodiment,

FIG. 6 is a flowchart showing an operation procedure of the image processing apparatus shown in FIG. BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, the image processing method of the present invention will be described in detail with reference to the drawings. According to the image processing method of the present invention, a sampling function more suitable for interpolation of image density values is selected from a function system called a fluency function system for classifying a function with differentiability as a parameter. The image is enlarged and reconstructed by performing interpolation using a sampling function.

Next, we explain the definition of the function space spanned by the fluency function and the fact that the function system corresponding to the impulse response characterizing this function space is derived by the biorthogonal sampling theorem.

(Definition of function space):

Rectangle function above

): = 2

1 (1)

0. The function space consisting of the step-like functions generated by> is 2 "-"), {"}: (2)

Then, the function space ^m S is defined as follows. S

(> 2)

The function space ^m S (m = l, 2, ……, oo) consists of at most (m−1) -order piecewise polynomials that can be continuously differentiated (m−2) times, and has a local platform in the time domain. The system is constructed on the basis of. These function spaces are a series of function spaces that link the step-like function space (when m = l) to the Fourier band-limited function space (when m → oo), with the continuous differentiability m as a parameter. Has become. The functional system corresponding to the impulse response characterizing this series of function spaces is derived by the following theorem.

(Biorthogonal sampling theorem):

Any function

f≡ ^m s is, for a sample value f _n = f (n),

/ (= ∑ _[5 (^-«) (4 a)

Satisfy the relationship. At this time,

-1

[Σ (-i ( ^卜 ) β ^ίωί άω

ω (5) [S *] Plara

Is

1, p = n

_[S ^ (tn) _[s ^m _] (tp) dt =

p ≠ n (6

[,

Is a function that satisfies

The above theorem states that if the original signal is in the function space ^ms , then the function

_[s] Indicates that ^m can be reproduced using the sampling function. Where the function

_{[s] As in} the case of ^m , in this specification, the character “1” written at the lower left of a certain character “A” and the character “2” written at the upper left of “A” Originally, they should be listed in the same column, but since such expressions are not possible, they should be described in a staggered place like “8”. In the function system _[s] ^m ^ (m≥1) derived as described above, the smaller the parameter m, the higher the locality of the functions included in the function system. On the other hand, taking into account the smoothness of the interpolated signal, a sampling function of m = 3 class that can be continuously differentiated at least once was selected. Hereafter, this function will be referred to as “m3 function”.

Next, a two-dimensional sampling function based on the above-described m 3 function is defined as a sampling function suitable for interpolating density values distributed in two dimensions.

(Definition of 2D sampling function):

When the one-dimensional sampling function is _[s] ³ ^ (t), the two-dimensional m 3 function for the variable (X, y) is defined as follows.

(7) By defining in this way, the influence of the sample value on points equidistant from the sample point on the X-y plane is equalized. Here, the distance between pixels is calculated by a geometric method. This is because the quality of the image obtained when the reconstruction was actually performed was higher than that obtained when the distance was obtained by other methods such as chessboard distance. This is probably because the spatial spectrum distribution of a general image is isotropically restricted. Figure 1 shows an outline of the two-dimensional sampling function. Next, a procedure for enlarging an image using the above-described two-dimensional sampling function will be described. Here, in order to simplify the explanation, it is assumed that the image is enlarged to an integral multiple. However, the same concept can be applied to the case of a real multiple.

First, a sampling matrix is generated by expressing the two-dimensional sampling function as a matrix according to the magnification. In addition, we use a magnified image matrix to hold the pixel values after convolution and magnification.

The original image is a color image represented by the density values of the three RGB primary colors. When its size is NxXNy, the density value for one color element is represented by the following matrix.

Also, the two-dimensional sampling function is represented by a matrix corresponding to the magnification, and is referred to as a sampling matrix. The sampling matrix when the image is enlarged n times can be expressed as follows. The size of the matrix is 4 n.

[ー一 2+; 7). Is] ψ-2 + 2 * η-2)… [-2) Oka

…-[S

so

Next, a matrix [Gn] is prepared by expanding the matrix of the original image vertically and horizontally by η times. Elements that are not in the original image are set to 0. ) g (l +-, l) = 0 g (l +-, l) = 0

n n

g (U +-) = 0 g (l ₊ -, l +-) = 0

n n n

(10) g (U +-) = 0

n

… Λ) — The convolution of the original image and the sampling matrix is realized by filtering the enlarged image matrix [G n] using the sampling matrix [M 3] described above.

By the way, one of the issues that must be considered when implementing image enlargement is the emergence of jaggy in edges. Normal digital images are sampled in a square grid, so diagonal lines always contain jagged elements. Since the shape of the edge greatly affects the quality of the image, this problem cannot be overlooked when considering improvement in image quality. Therefore, in the image processing method of the present invention, jaggy is reduced by adapting and deforming the two-dimensional sampling function to the edge shape.

As mentioned above, a two-dimensional sampling function is defined to be circular when viewed in a section passing through points of equal function value. By transforming this two-dimensional sampling function so that the cut surface becomes elliptical, a two-dimensional sampling function with directionality can be generated (see Fig. 2). Here, the two-dimensional sampling function that has directionality by being deformed is called “elliptic two-dimensional sampling function”. When this deformation direction matches the edge direction, the edge shape can be smoothly interpolated.

FIG. 3 is a diagram for explaining an enlarged result by an elliptic two-dimensional sampling function. Fig. 3 (a) shows the result of enlargement by the conventional method of nearest neighbor interpolation, Fig. 3 (b) shows the result of enlargement by the conventional method of third-order convolution, and Fig. 3 (c) shows the result of enlargement by the two-dimensional sampling function. As a result, Fig. 3 (d) shows the results of the enlargement by the elliptic two-dimensional sampling function, respectively (when both are 10 times larger).

In order to apply the modified sampling function (elliptical two-dimensional sampling function) to general image enlargement, it is necessary to appropriately determine the edge direction in the image. Next, a method for determining the edge direction will be described. The edges of natural images have complex structures such as partial edge breaks and intersections of multiple edges. Determining the direction using only local information causes distortion in the image. In the present invention, an edge direction is first determined based on local information. Then, by acquiring the general information step by step, a more appropriate edge direction is obtained. The procedure is described below.

Eight directions of Laplacian fill are used for edge detection. An edge in a digital image refers to a point at which the density value changes abruptly between two pixels, and thus the edge is actually between two pixels. In normal edge detection, either a pixel with a high density value or a pixel with a low density value is expressed as an edge from two pixels sandwiching this edge. In the present embodiment, since the purpose is to adapt the sampling function to the edge shape when performing interpolation around the edge, two pixels sandwiching the edge are considered as edges. In other words, two types of edge data are prepared: edge data with high density pixels in contact with the original edge as the first edge, and edge data with low density pixels as the second edge. The direction is determined every time. Also, the appearance of jaggies is a problem in gently continuous edges, and in areas where the edge shape changes finely, or in areas where edges are dense, jaggies may be imaged due to the complexity of the edge shape. The impact on quality is small. Therefore, from the detected edge data, only slowly continuous edges that cause jaggies are selected and used for direction detection.

In order to determine the direction at each edge, the least squares method using principal component analysis is used. The least squares method is a technique for finding a straight line that minimizes the distance between two-dimensionally distributed points. The straight line obtained in this way is called a regression line. Here, the least squares method is applied to the edge pattern, and the direction of the obtained regression line is set as the edge direction at the center of the region. First, a regression line is determined for each small area of 3 × 3 pixels. At this time, there are some patterns where the regression line is not uniquely determined, such as intersection of edges. In such a case, the direction is calculated based on the surrounding edge information whose direction is determined. For this calculation, the direction vector of the target pixel is used. That is, the direction vectors of the edges whose directions have been obtained by the least squares method in the target area are added to obtain the direction vector of the target pixel. However, let's decide here Is the direction of the straight line represented by the target point and the direction vector. Therefore, here, when the direction vector is a, it is considered that a =-a.

For the reasons described above, it is necessary to determine which vector to use before calculating the sum of the vectors. In the present invention, the determination is made based on the magnitude of the inner product. In other words, since the straight line closer to each of the two straight lines is said to be the straight line indicating the desired direction, a vector having a small angle formed by the direction vectors and thus having a large inner product is selected. In the example shown in Fig. 4, a vector b 'having a larger inner product with the vector a is selected. In this way, the sum of each direction vector is obtained, and the direction of the edge that cannot be determined by the least squares method can be determined.

By the way, as described above, since the direction is determined based on extremely local information of 3 × 3 pixels, it is difficult to reflect a more general change in the edge shape as it is. Therefore, direction optimization is performed based on the obtained local direction information. Next, the method will be described. The direction optimization is realized by adding the direction vectors of the edges in the 5 × 5 pixel area centered on the edge. However, if the edge in the target area is not continuous with the edge at the center of the area, the direction information is not added. When calculating the sum of the vectors, weighting is performed in inverse proportion to the distance from the center of the area. The above processing is performed for all edges to optimize the direction.

Next, the adaptive sampling matrix will be described. In order to transform the above two-dimensional sampling function into an ellipse, the lengths of the major axis and the minor axis are required. However, since the sampling function is deformed so that the cross-sectional area at a certain function value is constant, the ratio of the length of the major axis to the minor axis may be determined in practice. Hereinafter, this ratio is referred to as “shape factor”. The shape factor is calculated based on information on edges existing in the direction of the target edge. In other words, if an edge exists ahead of the edge direction, and if the directions match, the shape is greatly changed, and conversely, if there is no other edge in the edge direction, deformation is suppressed. Determine the shape factor.

Based on the edge direction and shape factor obtained as described above, a sampling matrix suitable for the situation of the target pixel is generated. Now, assuming that the shape factor is a "and the edge direction is angle 0, the mapping (χ ', y') of the point (X, y) is expressed by the following equation. ,

(11)

Where a =, β =

The two-dimensional sampling function on this mapping plane is the adaptive sampling function to be found. Convolution is performed using the adaptive sampling matrix obtained for each pixel. However, a standard two-dimensional sampling function is used for pixels other than edges. The convolution is performed in the same manner as the convolution using the two-dimensional sampling function described above.

As described above, in the image processing method of the present invention, by using the two-dimensional sampling function, a sharp enlarged image can be obtained as compared with the conventional method. Also, by using an enlargement method that performs interpolation by deforming the two-dimensional sampling function according to the edge shape, it is possible to prevent jaggies from appearing.

FIG. 5 is a diagram showing a schematic configuration of an image processing apparatus to which the image processing method of the present invention using an elliptic two-dimensional sampling function is applied. The image processing device shown in FIG. 5 includes an image data storage unit 10, an edge determination unit 20, a sampling matrix setting unit 30, an interpolation operation unit 40, and an enlarged data storage unit 50. I have.

The image data storage unit 10 stores a two-dimensional image data consisting of density values corresponding to each pixel constituting a two-dimensional image to be interpolated. The edge determination unit 20 determines whether the pixel of interest (target pixel) to be interpolated is included in an edge in the two-dimensional image, and if so, determines the status of the pixel of interest. I do. As described above, the status of the target pixel, the edge direction of the angle 0 of edges included the target pixel includes a shape factor a _r when deforming the elliptical two-dimensional sampling function.

The sampling matrix setting unit 30 generates a sampling matrix used for interpolation processing of a target pixel. For example, when the target pixel is not included in the edge, the sampling matrix setting unit 30 generates the sampling matrix [M 3] shown in the above equation (9) for the interpolation processing. When the target pixel is included in the edge, the sampling matrix setting unit 30 sets the edge judgment Based on the angle Θ of the edge direction obtained by the part 20 and the shape factor a _r , the contents of the above-mentioned relational expression of the mapping expressed by the expression (11) are determined, and the expression shown in the expression (9) is obtained. The sample matrix is transformed to generate an adaptive sampling matrix corresponding to the situation of the target pixel. The interpolation calculation unit 40 reads the density values of a plurality of pixels located in a predetermined range around the target pixel from the pixel data storage unit 10 and calculates the enlarged image matrix [G n] shown in the above equation (10). By generating and filtering this enlarged image matrix [G n] using the sampling matrix function generated by the sampling matrix setting unit 30, the interpolation operation by convolution of the original image and the sample matrix is performed. To calculate the density value (interpolation value) of the target pixel. The density values of the target pixel thus calculated are sequentially stored in the enlarged data storage unit 50.

FIG. 6 is a flowchart showing an operation procedure of the image processing apparatus shown in FIG. First, the interpolation calculation unit 40 sets a pixel of interest (pixel of interest) to be interpolated (step 100). When the target pixel is determined, the sampling matrix generation unit 30 generates a sampling matrix according to the situation of the target pixel (step 101). If the target pixel is not included in the edge, the effect is notified from the edge judgment unit 20 and the sampling matrix generation unit 30 sends the sampling matrix (10) Generate a sampling matrix. On the other hand, when the target pixel is included in the edge, the effect, the angle 0 in the edge direction, and the shape coefficient _ar are notified from the edge determination unit 20.The sampling matrix generation unit 30 deforms in the edge direction. Generate an adaptive sampling matrix.

Next, the interpolation calculation unit 40 reads out the density values of a predetermined number of pixels included in a predetermined range around the target pixel (step 102), and reads out the density values of the read out pixels and the sample matrix generation. An interpolation operation using the sampling matrix generated by the unit 30 is performed to obtain the density value of the target pixel (step 103).

In this way, when the density value of a certain target pixel is obtained by the interpolation calculation, the interpolation calculation unit 40 determines whether or not the calculation of the density value has been completed for all the pixels (step 104). If there is another pixel for which the density value has not been calculated, a negative determination is made and the processing from step 100 onward is repeated. When the density values have been completed for all the pixels, an affirmative determination is made in the determination in step 104, and a series of enlargement processing ends. As described above, in the image processing apparatus of the present embodiment, when the edge portion included in the two-dimensional image is enlarged, the interpolation process using the elliptic two-dimensional sampling function adapted to the edge shape is performed. It is possible to perform a smooth interpolation process along the edge shape, and to obtain a high-quality reconstructed image when performing a 2D image enlargement process. Also, by using a continuously differentiable two-dimensional sampling function, it is possible to obtain high-quality enlarged images without unnatural steps. In particular, by using a two-dimensional sampling function expressed by a piecewise polynomial having local platform values, it is possible to reduce the amount of data processing when performing the enlarging process and prevent the occurrence of truncation errors. can do. In addition, by using an elliptic two-dimensional sampling function having an elliptical cross section, a sharp change in density value along the edge direction is eliminated, so that jaggies in the edge direction can be reduced. Also, the longer the edge, the greater the degree to which the two-dimensional sampling function is deformed, so that jaggies that are particularly noticeable when the edge is long can be reduced.

In addition, after determining the edge direction for each minute area of the two-dimensional image, the edge direction is optimized in an area wider than the minute area, so that a small amount of data processing is maintained while maintaining a large amount of global processing. This makes it possible to perform accurate edge detection that reflects various edge shapes. Industrial applicability

As described above, according to the present invention, when enlarging an edge portion included in a two-dimensional image, interpolation processing using an ellipse two-dimensional sampling function adapted to the edge shape is performed. Interpolation processing can be performed according to the above, and a high-quality reconstructed image can be obtained when the two-dimensional image is enlarged.

Claims

The scope of the claims

1. In an image processing method that enlarges a two-dimensional image using a two-dimensional sampling function in which the effects of sample values on points equidistant from the sample point are equal,

A first step of detecting edges included in the two-dimensional image;

A second step of generating a deformed elliptic two-dimensional sample function by adapting the two-dimensional sampling function to the shape of the edge;

For the pixels along the edge included in the two-dimensional image, an interpolation operation for increasing the number of pixels is performed using the elliptic two-dimensional sampling function, and for the other pixels, the two-dimensional sample is used. An image processing method comprising: performing an interpolation operation to increase the number of pixels by using a conversion function.

2. The said two-dimensional sampling function is composed of a (m-1) -th order piecewise polynomial that can be continuously differentiated (m-2) times, and has a function value of a local table. The image processing method described.

3. The elliptic two-dimensional sampling function is characterized in that the elliptical two-dimensional sampling function has an elliptical cross section obtained by deforming a circular cross section passing through a point having the same function value along the direction of the edge while maintaining a cross sectional area. 2. The image processing method according to claim 1, wherein

4. The ratio of the major axis to the minor axis of the elliptical cross section is set to a large value when another edge exists along the edge, and a small value when no other edge exists along the edge. 4. The image processing method according to claim 3, wherein the image processing method is set to:

5. The first step includes a step of determining a direction of the edge based on local information corresponding to a minute region including a plurality of pixels included in the two-dimensional image; and a step of determining the determined edge. 2. The image processing method according to claim 1, further comprising a process of optimizing a region wider than the minute region in the direction of.